Struct swh_graph::utils::dataset_writer::PartitionedTableWriter
source · pub struct PartitionedTableWriter<PartitionWriter: TableWriter + Send> { /* private fields */ }Expand description
Wraps N TableWriter in such a way that they each write to base/0/x.parquet,
…, base/N-1/x.parquet instead of base/x.parquet.
This allows Hive partitioning while writing with multiple threads (x is the
thread id in the example above).
If num_partitions is None, disables partitioning.
Implementations§
source§impl<PartitionWriter: TableWriter + Send> PartitionedTableWriter<PartitionWriter>
impl<PartitionWriter: TableWriter + Send> PartitionedTableWriter<PartitionWriter>
pub fn partitions(&mut self) -> &mut [PartitionWriter]
Trait Implementations§
source§impl<PartitionWriter: TableWriter + Send> TableWriter for PartitionedTableWriter<PartitionWriter>
impl<PartitionWriter: TableWriter + Send> TableWriter for PartitionedTableWriter<PartitionWriter>
§type Schema = (String, Option<NonZeroU16>, <PartitionWriter as TableWriter>::Schema)
type Schema = (String, Option<NonZeroU16>, <PartitionWriter as TableWriter>::Schema)
(partition_column, num_partitions, underlying_schema)
type CloseResult = Vec<<PartitionWriter as TableWriter>::CloseResult>
fn new( path: PathBuf, (partition_column, num_partitions, schema): Self::Schema, flush_threshold: Option<usize> ) -> Result<Self>
source§fn flush(&mut self) -> Result<()>
fn flush(&mut self) -> Result<()>
Calls
.into() on the internal builder, and writes its result to disk.fn close(self) -> Result<Self::CloseResult>
Auto Trait Implementations§
impl<PartitionWriter> RefUnwindSafe for PartitionedTableWriter<PartitionWriter>where PartitionWriter: RefUnwindSafe,
impl<PartitionWriter> Send for PartitionedTableWriter<PartitionWriter>
impl<PartitionWriter> Sync for PartitionedTableWriter<PartitionWriter>where PartitionWriter: Sync,
impl<PartitionWriter> Unpin for PartitionedTableWriter<PartitionWriter>where PartitionWriter: Unpin,
impl<PartitionWriter> UnwindSafe for PartitionedTableWriter<PartitionWriter>where PartitionWriter: UnwindSafe,
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
§impl<T, U> CastableInto<U> for Twhere
U: CastableFrom<T>,
impl<T, U> CastableInto<U> for Twhere U: CastableFrom<T>,
§impl<T> DowncastableFrom<T> for T
impl<T> DowncastableFrom<T> for T
§fn downcast_from(value: T) -> T
fn downcast_from(value: T) -> T
Truncate the current UnsignedInt to a possibly smaller size
§impl<T, U> DowncastableInto<U> for Twhere
U: DowncastableFrom<T>,
impl<T, U> DowncastableInto<U> for Twhere U: DowncastableFrom<T>,
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere T: ?Sized,
§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> UpcastableFrom<T> for T
impl<T> UpcastableFrom<T> for T
§fn upcast_from(value: T) -> T
fn upcast_from(value: T) -> T
Extend the current UnsignedInt to a possibly bigger size.