Struct swh_graph::utils::dataset_writer::ParquetTableWriter

pub struct ParquetTableWriter<Builder: Default + StructArrayBuilder> {
    pub flush_threshold: usize,
    /* private fields */
}

Writer to a .parquet file, usable with ParallelDatasetWriter

Builder should follow the pattern documented by arrow::builder

Fields

flush_threshold: usize

Automatically flushes the builder to disk when it length reaches the value. To avoid uneven row group sizes, this value plus the number of values added to the builder between calls to Self::builder should be either equal or equal to max_row_group_size (or a multiple of it).

Implementations

impl<Builder: Default + StructArrayBuilder> ParquetTableWriter<Builder>

pub fn builder(&mut self) -> Result<&mut Builder>

Flushes the internal buffer is too large, then returns the array builder.

Trait Implementations

impl<Builder: Default + StructArrayBuilder> Drop for ParquetTableWriter<Builder>

fn drop(&mut self)

Executes the destructor for this type.

impl<Builder: Default + StructArrayBuilder> TableWriter for ParquetTableWriter<Builder>

type Schema = (Arc<Schema>, WriterProperties)

type CloseResult = FileMetaData

fn new( path: PathBuf, (schema, properties): Self::Schema, flush_threshold: Option<usize> ) -> Result<Self>

fn flush(&mut self) -> Result<()>

Calls .into() on the internal builder, and writes its result to disk.

fn close(self) -> Result<FileMetaData>