Trait swh_graph::utils::sort::Sortable

pub trait Sortable<Line: IntoIterator<Item = u8>>: ParallelIterator<Item = Line>where
    <Line as IntoIterator>::IntoIter: ExactSizeIterator,{
    // Provided method
    fn unique_sort_to_dir(
        self,
        target_dir: PathBuf,
        target_prefix: &str,
        temp_dir: &Path,
        pl: ProgressLogger<'_>,
        args: &[&str],
        buffer_size: usize,
        expected_lines: usize
    ) -> Result<()> { ... }
}

Provides a unique_sort_to_dir method to deduplicate, sort, and write to disk

Panics

If files cannot be created in temp_dir.

Provided Methods

fn unique_sort_to_dir( self, target_dir: PathBuf, target_prefix: &str, temp_dir: &Path, pl: ProgressLogger<'_>, args: &[&str], buffer_size: usize, expected_lines: usize ) -> Result<()>

Drains a [ParallelIterator], sorts its values, deduplicates them, and write them to multiple newline-separated ZSTD-compressed files in the given directory.

buffer_size is the RAM used by each of this process’ threads before flushing to sort.

Object Safety

This trait is not object safe.

Implementors

impl<Line: IntoIterator<Item = u8>, T: ParallelIterator<Item = Line>> Sortable<Line> for Twhere <Line as IntoIterator>::IntoIter: ExactSizeIterator,