datarust

Scikit-Learn Preprocessing in Rust — a modular, dependency-free machine-learning preprocessing and modeling library built on a lightweight Matrix type.

#![allow(unused)]
fn main() {
use datarust::scaler::StandardScaler;
use datarust::traits::Transformer;
use datarust::Matrix;

let x = Matrix::new(vec![
    vec![1.0, 10.0],
    vec![2.0, 20.0],
    vec![3.0, 30.0],
    vec![4.0, 40.0],
])?;

// Standardize: (x - mean) / std (population, ddof=0)
let mut scaler = StandardScaler::new();
let standardized = scaler.fit_transform(&x)?;
}

Default build has zero external dependencies. All linear algebra (eigenvalue decomposition, covariance, Cholesky factorization, coordinate descent) is implemented in pure Rust — no system BLAS/LAPACK, no Python runtime, no GIL.

What’s included

Why datarust?

Zero dependencies by default

cargo add datarust pulls in nothing. No BLAS, no LAPACK, no numpy, no Python runtime. The default build has a completely empty dependency tree — every algorithm (Jacobi eigendecomposition, Cholesky factorization, one-sided Jacobi SVD, coordinate descent, IRLS) is pure Rust. Opt in to serde, rayon, or a tuned pure-Rust GEMM via feature flags when you need them.

Rust-native type safety

sklearn’s single duck-typed fit/transform API becomes four separate traits enforced by the type system:

• Transformer — numeric Matrix → Matrix
• Predictor — supervised fit(X, y) + predict(X)
• Regressor / Classifier — regression or classification semantics
• CategoricalTransformer — StrMatrix → Matrix
• LabelTransformer — 1-D &[String] ↔ Vec<usize>

You cannot accidentally pass categorical strings to a numeric scaler — the compiler catches it.

Panic-free public API

Every fallible public method returns Result<T, DatarustError>. No hidden panics on bad input, no unwrap() in the API surface.

JSON serialization (not pickle)

Fitted models serialize to plain JSON via the serde feature — human-readable, language-agnostic, and diff-friendly. No binary pickle blobs.

Measured speedups vs scikit-learn

On heterogeneous ColumnTransformer composition, categorical encoding, and numeric scaling, datarust is 1.5–620× faster than scikit-learn. See the Performance page for the full benchmark tables and methodology.

A taste: end-to-end pipeline

#![allow(unused)]
fn main() {
use datarust::compose::{ColumnTransformer, Table};
use datarust::encoder::OneHotEncoder;
use datarust::categorical_kind::CategoricalTransformerKind;
use datarust::scaler::StandardScaler;
use datarust::transformer_kind::TransformerKind;
use datarust::Matrix;

// Mixed numeric + categorical table
let numeric = Matrix::new(vec![vec![1.0, 2.0], vec![3.0, 4.0]])?;
let table = Table::new(numeric, /* categorical StrMatrix */ /* ... */ /* ) */;

// Build a column transformer: scale numeric, one-hot encode categorical
let mut ct = ColumnTransformer::new()
    .add_numeric("num", vec![0, 1], TransformerKind::StandardScaler(StandardScaler::new()));
    // .add_categorical("cat", vec![2, 3], CategoricalTransformerKind::OneHotEncoder(OneHotEncoder::new()));

let out = ct.fit_transform_to_table(&table)?;
// out.numeric and out.categorical hold the transformed columns
}

Next steps

• New to datarust? Start with the Quick Start.
• Want the full feature list vs sklearn? See the Feature Comparison.
• Looking for a specific module? Browse the Module Guide.
• Need the API reference? It’s on docs.rs.

Quick Start

Get from cargo add to a fitted, transformed pipeline in five minutes.

Free Tier, Self-Serve Keys & Sandbox Access

datarust is built for zero onboarding friction:

• Free Tier: 100% free and open source under the MIT license with zero paid tiers.
• Self-Serve Keys: No API key or registration required to use the library, public REST endpoints, or the MCP server.
• Interactive Sandbox: Run algorithms locally in Rust or test remote endpoints in an instant sandbox at https://datarust.dev/mcp.

1. Add the dependency

[dependencies]
datarust = "0.7"

Or with the most useful feature flags:

[dependencies]
datarust = { version = "0.7", features = ["serde", "rayon"] }

See Installation for what each feature does.

2. Create a matrix

All numeric data flows through Matrix — a row-major dense matrix backed by a single contiguous Vec<f64>:

#![allow(unused)]
fn main() {
use datarust::Matrix;

let x = Matrix::new(vec![
    vec![1.0, 10.0],
    vec![2.0, 20.0],
    vec![3.0, 30.0],
    vec![4.0, 40.0],
])?;
}

3. Fit and transform

Every numeric transformer implements the Transformer trait: call fit to learn parameters, then transform to apply them.

#![allow(unused)]
fn main() {
use datarust::scaler::StandardScaler;
use datarust::traits::Transformer;

let mut scaler = StandardScaler::new();
let standardized = scaler.fit_transform(&x)?;  // fit + transform in one call

// mean=0, variance=1 per column
}

4. Train a model

Regression and classification models implement Predictor: call fit(X, y) then predict(X).

#![allow(unused)]
fn main() {
use datarust::linear_model::LinearRegression;
use datarust::traits::Predictor;

let y = vec![3.0, 5.0, 7.0, 9.0]; // y = 2x + 1 (for feature 0)
let features = x.select_columns(&[0])?; // pick one feature
let mut model = LinearRegression::new();
model.fit(&features, &y)?;
let pred = model.predict(&features)?;
}

5. Evaluate

#![allow(unused)]
fn main() {
use datarust::metrics::regression::r2_score;

let r2 = r2_score(&y, &pred)?;
println!("R² = {r2:.4}"); // ≈ 1.0 for a clean linear signal
}

6. Split and cross-validate

#![allow(unused)]
fn main() {
use datarust::model_selection::{train_test_split, cross_val_score, KFold};
use datarust::metrics::regression::r2_score;

let (x_tr, x_te, y_tr, y_te) = train_test_split(&x, &y)?;

let cv = KFold::new().with_n_splits(3);
let scores = cross_val_score(&LinearRegression::new(), &x, &y, &cv, r2_score)?;
// scores.len() == 3, one R² per fold
}

Where to go next

• All transformers at a glance: Module Guide
• Mixing numeric + categorical columns: Compose
• Why datarust is fast: Performance

Installation

Cargo

[dependencies]
datarust = "0.7"

The default build has zero external dependencies — cargo add datarust pulls in nothing but the standard library.

Optional features

All features are opt-in and independent. Enable the ones you need:

[dependencies]
datarust = { version = "0.7", features = ["serde", "rayon", "matrixmultiply"] }

serde

Enables JSON serialization of fitted transformers via datarust::serialize::{save_json, load_json, to_json, from_json}. Models serialize to human-readable JSON (not pickle).

#![allow(unused)]
fn main() {
use datarust::serialize::{save_json, load_json};
use datarust::scaler::StandardScaler;
use datarust::traits::Transformer;

let mut scaler = StandardScaler::new();
scaler.fit(&x)?;

// Save
save_json(&scaler, "model.json")?;

// Load into a fresh instance
let restored: StandardScaler = load_json("model.json")?;
let out = restored.transform(&x)?;
}

rayon

Enables parallel column/row operations for large datasets. Speeds up scaler transforms, encoders, and the KNN imputer on inputs above ~4 000 rows.

matrixmultiply

Enables a tuned pure-Rust GEMM (via the matrixmultiply crate, no system BLAS) for Matrix::matmul and centered-covariance computation. Significantly speeds up PCA, TruncatedSVD, and the linear models on large dense inputs. The default build remains zero-external-dependency; opt in with this feature.

datasets

Embeds four classic toy datasets (Iris, Breast Cancer, Wine, Diabetes) as const arrays for examples, tests, and onboarding. No file I/O, no network access — the data is compiled into the binary. See the Datasets guide.

Rust version

datarust targets Rust 1.70+ (the rust-version field in Cargo.toml). It is tested against stable and the MSRV in CI.

Platform support

CI runs on Ubuntu and macOS (both x86-64 and arm64). WASM should work for the default build (no system calls, no threading without rayon), though it is not formally tested.

Core Concepts

Understanding seven concepts unlocks the entire crate.

1. Matrix — the data container

All numeric data flows through Matrix: a row-major dense matrix backed by a single contiguous Vec<f64> buffer. The flat layout keeps every numeric hot loop cache-friendly and auto-vectorizable.

#![allow(unused)]
fn main() {
use datarust::Matrix;

let m = Matrix::new(vec![
    vec![1.0, 2.0, 3.0],
    vec![4.0, 5.0, 6.0],
])?;
assert_eq!(m.nrows(), 2);
assert_eq!(m.ncols(), 3);
assert_eq!(m.get(0, 1), 2.0);
}

Companion types:

• StrMatrix — categorical string input for encoders.
• SparseMatrix — CSR format for memory-efficient one-hot output.

2. The Transformer trait

All numeric transformers implement Transformer:

#![allow(unused)]
fn main() {
pub trait Transformer {
    fn name(&self) -> &'static str;
    fn fit(&mut self, x: &Matrix) -> Result<()>;
    fn transform(&self, x: &Matrix) -> Result<Matrix>;
    fn fit_transform(&mut self, x: &Matrix) -> Result<Matrix>; // default: fit + transform
    fn inverse_transform(&self, _x: &Matrix) -> Result<Matrix>; // optional
    fn is_fitted(&self) -> bool;
}
}

• fit learns parameters from training data (takes &mut self).
• transform applies the learned transformation (takes &self).
• fit_transform is a convenience that calls both.
• inverse_transform reverses the transformation where supported.

3. Supervised estimator traits

Regression and classification estimators implement Predictor:

#![allow(unused)]
fn main() {
pub trait Predictor: Estimator {
    fn fit(&mut self, x: &Matrix, y: &[f64]) -> Result<()>;
    fn predict(&self, x: &Matrix) -> Result<Vec<f64>>;
    fn fit_predict(&mut self, x: &Matrix, y: &[f64]) -> Result<Vec<f64>>; // default
    fn is_fitted(&self) -> bool;
}
}

• LinearRegression, Ridge, Lasso — predict returns continuous predictions.
• LogisticRegression — implements Classifier; predict returns the original hard labels used during fitting. predict_proba returns one column per class in classes() order.
• DecisionTreeClassifier / RandomForestClassifier — implement Classifier and PredictProba; tree probabilities are leaf class frequencies, forest probabilities are averaged across trees.
• DecisionTreeRegressor / RandomForestRegressor — implement Regressor; trees predict leaf means, forests average the tree predictions.

4. The Clusterer trait (unsupervised)

Clustering estimators implement Clusterer — the unsupervised counterpart to Predictor. fit takes only X (no target y), and predict returns cluster indices as Vec<usize> rather than regression targets or class labels:

#![allow(unused)]
fn main() {
pub trait Clusterer: Estimator {
    fn fit(&mut self, x: &Matrix) -> Result<()>;
    fn predict(&self, x: &Matrix) -> Result<Vec<usize>>;
    fn fit_predict(&mut self, x: &Matrix) -> Result<Vec<usize>>; // default
    fn fit_transform(&mut self, x: &Matrix) -> Result<Matrix>;   // default: one-hot labels
    fn n_clusters(&self) -> usize;
    fn is_fitted(&self) -> bool;
}
}

• KMeans — fit_predict returns one cluster index per row; cluster_centers() exposes the learned centroids and inertia() the minimized within-cluster sum of squares.

5. The Params trait (hyperparameter introspection)

Estimators whose hyperparameters should be searchable implement the opt-in Params trait:

#![allow(unused)]
fn main() {
pub trait Params {
    fn get_params(&self) -> Vec<(&'static str, ParamValue)>;
    fn set_params(&mut self, name: &str, value: ParamValue) -> Result<()>;
}
}

KMeans (n_clusters, max_iter, tol, n_init), LogisticRegression (max_iter, tol, fit_intercept), the decision trees (criterion, max_depth, min_samples_*, max_features), and both random forests implement it. Not every estimator needs Params — only those whose hyperparameters should be tuned by an automated search (the foundation for future GridSearchCV).

6. The categorical traits

Categorical data is kept separate at the type level:

• CategoricalTransformer — StrMatrix → Matrix (OneHot, Ordinal, Frequency encoders).
• TargetTransformer — supervised encoders needing y during fit (TargetEncoder).
• LabelTransformer — 1-D &[String] ↔ Vec<usize> (LabelEncoder).

This separation means the compiler prevents you from accidentally passing strings to a numeric scaler.

7. Error handling

Every fallible public method returns Result<T, DatarustError>. No hidden panics on bad input. The error variants are ML-domain-specific:

#![allow(unused)]
fn main() {
pub enum DatarustError {
    NotFitted(String),
    InvalidInput(String),
    ShapeMismatch { expected: String, actual: String },
    EmptyInput(String),
    InvalidConfig(String),
    Singular(String),       // e.g. rank-deficient matrix in Cholesky
    UnknownCategory(String),
    // ...
}
}

Typical usage:

#![allow(unused)]
fn main() {
match scaler.transform(&x) {
    Ok(out) => { /* use out */ }
    Err(DatarustError::NotFitted(name)) => eprintln!("call fit() first on {name}"),
    Err(DatarustError::ShapeMismatch { expected, actual }) => eprintln!("{expected} vs {actual}"),
    Err(e) => eprintln!("error: {e}"),
}
}

Scalers

Feature scaling and distribution-shaping transformers. All implement Transformer (Matrix → Matrix) and live in datarust::scaler.

Scaler inputs and floating-point configuration values must be finite. NaN and infinity return DatarustError before fitted state is changed.

StandardScaler

Standardize by removing the mean and scaling to unit variance. Population standard deviation (ddof = 0), matching sklearn.

#![allow(unused)]
fn main() {
use datarust::scaler::StandardScaler;
use datarust::traits::Transformer;

let mut s = StandardScaler::new()
    .with_mean(true)   // default: center
    .with_std(true);   // default: scale to unit variance
let out = s.fit_transform(&x)?;
}

MinMaxScaler

Scale each feature to a given range (default [0, 1]).

#![allow(unused)]
fn main() {
use datarust::scaler::MinMaxScaler;

let mut s = MinMaxScaler::new().feature_range(-1.0, 1.0);
let out = s.fit_transform(&x)?;
}

RobustScaler

Outlier-robust scaling using the median and interquartile range.

#![allow(unused)]
fn main() {
use datarust::scaler::RobustScaler;

let mut s = RobustScaler::new()
    .with_centering(true)
    .with_scaling(true);
let out = s.fit_transform(&x)?;
}

MaxAbsScaler

Scale by dividing by the maximum absolute value per feature. Preserves sparsity.

Normalizer

Row-wise normalization (not column-wise). Each sample is scaled to unit norm.

#![allow(unused)]
fn main() {
use datarust::scaler::{Normalizer, Norm};

let mut n = Normalizer::new().norm(Norm::L2); // or Norm::L1, Norm::Max
}

Binarizer

Threshold features to 0/1.

#![allow(unused)]
fn main() {
use datarust::scaler::Binarizer;

let mut b = Binarizer::new().threshold(0.5);
}

KBinsDiscretizer

Continuous-to-discrete bin discretization.

#![allow(unused)]
fn main() {
use datarust::scaler::{KBinsDiscretizer, BinStrategy, KBinsEncode};

let mut k = KBinsDiscretizer::new()
    .strategy(BinStrategy::Quantile) // or Uniform, KMeans
    .encode(KBinsEncode::Ordinal)    // or OneHotDense
    .n_bins(5);
}

QuantileTransformer

Transform features to follow a uniform or normal distribution. Robust to outliers.

#![allow(unused)]
fn main() {
use datarust::scaler::{QuantileTransformer, OutputDistribution};

let mut q = QuantileTransformer::new()
    .output(OutputDistribution::Normal); // or Uniform
}

PowerTransformer

Gaussianize features via Yeo-Johnson or Box-Cox, with automatic lambda estimation.

#![allow(unused)]
fn main() {
use datarust::scaler::{PowerTransformer, PowerMethod};

let mut p = PowerTransformer::new().method(PowerMethod::YeoJohnson); // or BoxCox
}

When to use which?

Encoders

Categorical encoding transformers. Live in datarust::encoder.

LabelEncoder

1-D string ↔ int mapping for target labels. Implements LabelTransformer.

#![allow(unused)]
fn main() {
use datarust::encoder::LabelEncoder;
use datarust::traits::LabelTransformer;

let labels = vec!["cat".to_string(), "dog".to_string(), "cat".to_string()];
let mut le = LabelEncoder::new();
let codes = le.fit_transform(&labels)?;   // [0, 1, 0]
let back = le.inverse_transform(&codes)?; // ["cat", "dog", "cat"]
}

OneHotEncoder

Encode categorical features as a one-hot numeric matrix. Supports dense and CSR sparse output.

#![allow(unused)]
fn main() {
use datarust::encoder::{OneHotEncoder, HandleUnknown};

let mut ohe = OneHotEncoder::new()
    .handle_unknown(HandleUnknown::Ignore) // or Error
    .sparse_output(true);                  // CSR SparseMatrix output
let out = ohe.fit_transform(&str_matrix)?;
}

The CSR SparseMatrix output stores only the 1.0 positions, saving significant memory for high-cardinality columns.

OrdinalEncoder

Encode categorical features as integer codes with optional user-defined ordering.

#![allow(unused)]
fn main() {
use datarust::encoder::{OrdinalEncoder, OrdinalCategories};

// Auto-detect categories from data:
let mut oe = OrdinalEncoder::new();

// Or specify explicit ordering (e.g. "low" < "medium" < "high"):
let mut oe = OrdinalEncoder::new()
    .categories(OrdinalCategories::Manual(vec![vec!["low".into(), "medium".into(), "high".into()]]));
}

TargetEncoder

Supervised encoder: replaces categories with the smoothed mean of the target. Useful for high-cardinality features. Implements TargetTransformer — requires y during fit.

#![allow(unused)]
fn main() {
use datarust::encoder::{TargetEncoder, UnknownTarget};

let mut te = TargetEncoder::new()
    .smoothing(1.0)
    .handle_unknown(UnknownTarget::GlobalMean); // or NaN, Error
let out = te.fit_transform(&str_matrix, &y)?;
}

FrequencyEncoder

Replace categories with their count or proportion in the training data.

#![allow(unused)]
fn main() {
use datarust::encoder::{FrequencyEncoder, UnknownFrequency};

let mut fe = FrequencyEncoder::new()
    .proportion(true)                          // proportion vs raw count
    .handle_unknown(UnknownFrequency::Zero);   // or Error
}

Choosing an encoder

Imputers

Missing-value completion. Live in datarust::imputer. Both implement Transformer.

Missing values are represented as f64::NAN in the Matrix.

SimpleImputer

Fill missing values with a per-column statistic or a constant.

#![allow(unused)]
fn main() {
use datarust::imputer::{SimpleImputer, ImputeStrategy};

// Mean (default), median, most_frequent, or constant
let mut imp = SimpleImputer::new()
    .strategy(ImputeStrategy::Median); // or Mean, MostFrequent, Constant(0.0)
let out = imp.fit_transform(&x_with_nans)?;
}

Strategies:

• Mean — column mean (default)
• Median — column median, robust to outliers
• MostFrequent — most common value
• Constant(v) — a fixed fill value

KnnImputer

Impute using k-Nearest Neighbors. Distance is computed over co-observed features only (features where neither sample is missing).

#![allow(unused)]
fn main() {
use datarust::imputer::{KnnImputer, KnnWeights};

let mut imp = KnnImputer::new()
    .n_neighbors(5)
    .weights(KnnWeights::Distance); // or Uniform
let out = imp.fit_transform(&x_with_nans)?;
}

KnnImputer is more accurate than SimpleImputer when features are correlated, but slower — it computes pairwise distances over all samples.

When to use which?

Decomposition

Dimensionality reduction. Live in datarust::decomposition. Both implement Transformer.

PCA

Principal Component Analysis via Jacobi eigenvalue decomposition of the covariance matrix.

#![allow(unused)]
fn main() {
use datarust::decomposition::{PCA, PCAComponents};

// Select components by count, variance ratio, or keep all:
let mut pca = PCA::new(PCAComponents::Variance(0.95)) // keep 95% of variance
    .whiten(true);                                    // optional: whiten components
let projected = pca.fit_transform(&x)?;

// After fit:
pca.components();                // [[f64; n_features]; n_components]
pca.explained_variance_ratio();  // how much variance each component captures
pca.noise_variance();            // estimated noise variance
let reconstructed = pca.inverse_transform(&projected)?; // approximate recovery
}

Component selection

• PCAComponents::Count(k) — request k components, capped at min(n_samples, n_features).
• PCAComponents::Variance(0.95) — keep enough to explain 95% of variance.
• PCAComponents::All — keep all components.

Solver

#![allow(unused)]
fn main() {
use datarust::decomposition::PCASolver;

let mut pca = PCA::new(PCAComponents::Count(10))
    .solver(PCASolver::Randomized); // Halko–Martinsson–Tropp randomized SVD
}

• Auto (default) — exact eigensolver.
• Full — full Jacobi eigendecomposition.
• Randomized — randomized SVD, much faster for tall-and-wide, low-rank data.

Performance tip: enabling the matrixmultiply feature speeds up PCA significantly on large dense inputs by dispatching covariance and transform matmuls to a tuned pure-Rust GEMM.

TruncatedSVD

Dimensionality reduction via truncated SVD. Does not center the data, making it suitable for sparse or TF-IDF inputs.

#![allow(unused)]
fn main() {
use datarust::decomposition::{TruncatedSVD, SVDComponents};

// By count, variance threshold, or all:
let mut svd = TruncatedSVD::new(5).unwrap();                 // 5 components
let mut svd = TruncatedSVD::new(0.95).unwrap();              // 95% variance
let mut svd = TruncatedSVD::new(SVDComponents::All).unwrap();
let out = svd.fit_transform(&x)?;
}

PCA vs TruncatedSVD

Linear Models

Regression and classification estimators. Live in datarust::linear_model. All implement Predictor; regression models also implement Regressor and classifiers implement Classifier.

LinearRegression

Ordinary least squares. Estimates y ≈ Xβ + b by minimising ‖Xβ − y‖².

#![allow(unused)]
fn main() {
use datarust::linear_model::{LinearRegression, LinearSolver};
use datarust::traits::Predictor;

let mut model = LinearRegression::new()
    .with_fit_intercept(true)             // default
    .with_solver(LinearSolver::Cholesky); // default; or LinearSolver::Svd

model.fit(&x, &y)?;
let pred = model.predict(&new_x)?;

model.coef();          // coefficients β
model.intercept();     // intercept b
model.n_features_in(); // feature count
let r2 = model.score(&x, &y)?;  // R²
}

Solvers:

• Cholesky (default) — solves XᵀX β = Xᵀy via pure-Rust Cholesky. Fast; requires full column rank.
• Svd — eigendecomposition pseudo-inverse. Stable for rank-deficient / collinear inputs.

Ridge

L2-regularized regression. Minimises ‖Xβ − y‖² + α‖β‖². The penalty guarantees the system matrix is positive-definite — Ridge succeeds on collinear inputs where LinearRegression fails.

#![allow(unused)]
fn main() {
use datarust::linear_model::{Ridge, RidgeSolver};

let mut model = Ridge::new()
    .with_alpha(1.0)                      // regularization strength (default 1.0)
    .with_solver(RidgeSolver::Cholesky);  // or Svd
model.fit(&x, &y)?;
}

Larger alpha → more shrinkage (coefficients shrink toward zero). alpha must be finite and non-negative.

Lasso

L1-regularized regression. Minimises (1/(2n))‖Xβ − y‖² + α‖β‖₁. Solved by coordinate descent with soft-thresholding.

The L1 penalty drives irrelevant coefficients to exactly zero, producing a sparse model that performs implicit feature selection.

#![allow(unused)]
fn main() {
use datarust::linear_model::Lasso;

let mut model = Lasso::new()
    .with_alpha(0.1)          // larger alpha → more sparsity
    .with_max_iter(1000)      // default 1000
    .with_tol(1e-4);          // convergence tolerance
model.fit(&x, &y)?;

model.coef();   // some entries may be exactly 0.0 (sparsity)
model.n_iter(); // iterations actually run
}

For iterative solvers, max_iter must be positive and tol must be finite and non-negative. Invalid configurations return InvalidConfig before the optimization starts.

LogisticRegression

Binary classification via IRLS and multiclass classification via multinomial softmax regression. Labels can be any non-negative integers; the model compacts them internally and maps predictions back to their original values.

#![allow(unused)]
fn main() {
use datarust::linear_model::{LogisticRegression, LogisticSolver};
use datarust::traits::Predictor;

let mut model = LogisticRegression::new()
    .with_solver(LogisticSolver::Cholesky) // or Svd
    .with_max_iter(100)                    // default 100
    .with_tol(1e-4);

model.fit(&x, &y)?;               // for example, labels can be 2.0 / 5.0 / 9.0
let classes = model.predict(&x)?; // returns the original labels
let probabilities = model.predict_proba(&x)?;
// Probability column i always corresponds to model.classes()[i].
let class_five_probability = model.predict_proba_for_class(&x, 5.0)?;
let acc = model.score(&x, &y)?; // mean accuracy
}

LogisticRegression applies the same iteration and tolerance validation as Lasso. Invalid values supplied through Params::set_params are rejected without changing the estimator.

Choosing a model

Trees

Decision trees are the first supervised learners in datarust beyond linear models, and the foundation the ensembles build on. The tree module ships a CART implementation whose splitter is a line-by-line transliteration of scikit-learn 1.6.1’s BestSplitter, so a fitted tree matches sklearn node-for-node, bit-for-bit when given the same data, parameters, and integer seed — split features, thresholds, impurities, predictions, probabilities, and feature_importances_ (verified by tests/tree_parity.rs against seeded sklearn fixtures).

#![allow(unused)]
fn main() {
use datarust::tree::{Criterion, DecisionTreeClassifier, MaxFeatures};
use datarust::traits::{Predictor, PredictProba};

let mut clf = DecisionTreeClassifier::new()
    .with_criterion(Criterion::Gini)      // default; or Criterion::Entropy
    .with_max_depth(None)                 // None = unlimited (default)
    .with_min_samples_split(2)            // default
    .with_min_samples_leaf(1)             // default
    .with_min_impurity_decrease(0.0)      // default
    .with_max_features(MaxFeatures::All)  // default; Sqrt, Log2, Count(n)
    .with_random_state(7);                // seed for per-split feature draws

clf.fit(&x, &y)?;
let pred = clf.predict(&new_x)?;
let proba = clf.predict_proba(&new_x)?;
let importances = clf.feature_importances()?;
}

Why a seed matters for trees

Even with max_features = All, sklearn’s tree draws features through its RNG at every split; ties in split quality therefore resolve through the seeded stream, not feature order. datarust reproduces this exactly — the same random_state yields the same tree as sklearn, and different seeds may yield different (equally valid) trees on tied data. For reproducible models, always set with_random_state.

DecisionTreeClassifier

CART classification tree (Gini by default, or entropy — both use sklearn’s exact base-2 definitions). Implements Predictor and PredictProba; predict returns the majority class of the reached leaf (ties break to the smallest class label, like sklearn), predict_proba returns the leaf’s class frequencies.

#![allow(unused)]
fn main() {
use datarust::tree::DecisionTreeClassifier;
use datarust::traits::{Classifier, PredictProba, Predictor};

let mut clf = DecisionTreeClassifier::new().with_random_state(7);
clf.fit(&x_train, &y_train)?;

let pred = clf.predict(&x_test)?;
let proba = clf.predict_proba(&x_test)?;   // n_rows × n_classes
let r2_like_acc = clf.score(&x_test, &y_test)?;
}

DecisionTreeRegressor

CART regression tree with the squared-error (variance-reduction) criterion. Leaves predict the mean target of their training samples.

#![allow(unused)]
fn main() {
use datarust::tree::DecisionTreeRegressor;
use datarust::traits::{Predictor, Regressor};

let mut reg = DecisionTreeRegressor::new()
    .with_max_depth(Some(5))
    .with_random_state(7);

reg.fit(&x, &y)?;
let pred = reg.predict(&new_x)?;
let r2 = reg.score(&x_test, &y_test)?;
}

With the default min_samples_leaf = 1 the tree grows to pure leaves and reproduces the training targets exactly; constrain leaf size or depth to regularize.

Feature importances

feature_importances() computes sklearn’s normalized impurity reduction:

$$\text{importance}j = \frac{\sum{t \in \text{splits on } j} \frac{N_t}{N},\Delta i(t)}{\sum_{t \in \text{all splits}} \frac{N_t}{N},\Delta i(t)}$$

The vector sums to 1 (or is all-zero for a pure root). The summation and per-split arithmetic reproduce sklearn’s compiled float behavior exactly, so importances match feature_importances_ bit-for-bit.

Inspecting the tree

tree() exposes the fitted TreeStructure:

#![allow(unused)]
fn main() {
let tree = clf.tree().unwrap();
tree.n_nodes();        // arena size
tree.get_depth();      // maximum depth
tree.n_leaves();       // leaf count
tree.classes();        // sorted class labels (classification)

// Per-node introspection (preorder arena, root = 0):
tree.node_feature(0);        // Some(feature) for internal nodes, None for leaves
tree.node_threshold(0);      // split threshold
tree.node_impurity(0);       // impurity at fit time
tree.node_improvement(0);    // impurity improvement over the parent
tree.node_n_samples(0);      // training samples reaching the node
tree.node_regression_value(0); // leaf/regression mean value

// Route one feature row to its leaf:
let leaf = tree.leaf_for(&row)?;
}

Parameters

Divergences from sklearn (by design): no sample_weight, ccp_alpha pruning, missing-value routing, or sparse input; X must be finite and single-output. The Params trait exposes the numeric subset for hyperparameter introspection, ready for v0.8’s grid search.

The shared RNG

The per-split feature draws use datarust’s public, seedable Rng (datarust::rng), promoted to a top-level module in v0.7 so ensembles can build deterministic bootstrap samples from the same utility. See Model Selection for its use in splitting, and Ensembles for the forests built on this kernel.

Ensembles

Random forests were the reason the tree kernel was built the way it was: not “a CART implementation”, but a transcription of scikit-learn’s, so that a datarust forest and a sklearn forest trained on the same data with the same integer random_state produce the same trees — node for node, bit for bit. src/ensemble.rs ships the two flagship ensembles on top of that kernel.

#![allow(unused)]
fn main() {
use datarust::ensemble::RandomForestClassifier;
use datarust::traits::{Classifier, PredictProba, Predictor};

let mut forest = RandomForestClassifier::new()
    .with_n_estimators(100)               // default
    .with_criterion(Criterion::Gini)      // default; or Entropy
    .with_max_features(MaxFeatures::Sqrt) // default; All, Log2, Count(n)
    .with_bootstrap(true)                 // default
    .with_random_state(42);

forest.fit(&x, &y)?;
let pred = forest.predict(&new_x)?;
let proba = forest.predict_proba(&new_x)?;   // averaged tree probabilities
let importances = forest.feature_importances()?;
}

How the parity works

A forest’s randomness has three layers, and all three are reproduced exactly:

1. The seed stream. sklearn’s forest draws one seed per tree from a legacy numpy RandomState (MT19937). datarust embeds a minimal clone of that generator, so the master stream splits into the same per-tree seeds.
2. The bootstrap samples. each tree’s row indices are drawn from the same per-tree stream, with the same masked-rejection arithmetic. The indices are applied as sample-weight multiplicities, so the tree kernel sees exactly the bootstrap resample sklearn’s trees see.
3. The splits. each tree’s splitter consumes its seed through the same rand_r chain and Fisher–Yates feature draws as sklearn’s BestSplitter (see Trees).

The result is pinned by tests/integration_forest.rs: predictions, probabilities, feature_importances, and bootstrap samples all match a reference sklearn forest via to_bits().

Aggregation

• Classifier: per-tree class probabilities are averaged, then argmax is taken (ties break to the smallest class label, like sklearn). This is probability averaging, not majority voting of hard predictions.
• Regressor: predictions are averaged arithmetically.
• feature_importances(): per-tree normalized importances are averaged and re-normalized, matching sklearn’s forest-level feature_importances_.

estimators_samples()

Each tree’s bootstrap indices are derived, not stored: the fitted model keeps only the per-tree seeds (8 bytes each) and regenerates the index lists on demand — the same on-demand scheme sklearn’s estimators_samples_ uses. A 100-tree forest on a million rows therefore doesn’t carry ~800 MB of indices around:

#![allow(unused)]
fn main() {
let samples = forest.estimators_samples();  // Vec<Vec<usize>>, n_estimators lists
let first_tree_rows = &samples[0];
}

RandomForestRegressor

The regressor shares every builder except with_criterion — regression trees are squared-error only:

#![allow(unused)]
fn main() {
use datarust::ensemble::RandomForestRegressor;
use datarust::traits::{Predictor, Regressor};

let mut forest = RandomForestRegressor::new()
    .with_n_estimators(200)
    .with_max_features(MaxFeatures::All)  // sklearn's regressor default
    .with_random_state(7);

forest.fit(&x, &y)?;
let r2 = forest.score(&x_test, &y_test)?;
}

Defaults follow sklearn: 100 trees, bootstrap on, max_features = Sqrt for the classifier and All for the regressor.

Parameters

Both forests implement the Params trait (get_params/set_params) for hyperparameter introspection.

Scope (this first release)

Out of bag scoring, weighted fits, and parallel tree fitting are intentionally outside the first forest release; ExtraTrees and a generic Bagging meta-estimator are on the roadmap. Fitting is sequential — embarrassingly parallel, and a natural future rayon win.

Clustering

Unsupervised clustering estimators. Live in datarust::cluster. All implement the Clusterer trait — the unsupervised counterpart to Predictor. fit takes only X (no target y); predict returns cluster indices as Vec<usize>.

KMeans

k-means clustering via Lloyd’s algorithm with k-means++ initialization. Partitions n observations into k clusters by minimizing within-cluster sum of squares (inertia). Each iteration assigns every point to its nearest centroid, then recomputes centroids as the mean of their members.

#![allow(unused)]
fn main() {
use datarust::cluster::{KMeans, KMeansInit};
use datarust::traits::Clusterer;

let mut km = KMeans::new()
    .with_n_clusters(3)                         // default 8
    .with_init(KMeansInit::KMeansPlusPlus)      // default; or KMeansInit::Random
    .with_n_init(10)                            // restarts, keep best (default 10)
    .with_max_iter(300)                         // default 300
    .with_tol(1e-4)                             // default 1e-4
    .with_random_state(42);                     // deterministic seed

let labels = km.fit_predict(&x)?;               // Vec<usize>, one cluster per row
let centers = km.cluster_centers();             // &[Vec<f64>], one centroid per cluster
let inertia = km.inertia();                     // f64, sum of squared distances
let iters = km.n_iter();                        // Lloyd's iterations of best run

let new_labels = km.predict(&new_x)?;           // assign new points to nearest centroid
}

n_clusters, n_init, and max_iter must be positive. tol must be finite and non-negative. Invalid values return InvalidConfig before clustering; the same checks apply immediately when values are supplied through Params.

Initialization strategies (KMeansInit):

• KMeansPlusPlus (default) — first centroid is uniform random; each subsequent centroid is sampled with probability proportional to its squared distance from the nearest already-chosen centroid. Produces well-spread initial centroids and is the scikit-learn default.
• Random — choose n_clusters distinct points uniformly at random from the data.

How it works:

1. Initialize k centroids using the chosen strategy.
2. Assignment step — each point is assigned to its nearest centroid (lowest squared Euclidean distance).
3. Update step — each centroid is recomputed as the mean of its members. Empty clusters keep their previous centroid.
4. Repeat until centroid movement drops below tol or max_iter is reached.
5. Steps 1–4 run n_init times with different seeds; the lowest-inertia result is kept.

Reproducibility: with_random_state(seed) makes results fully deterministic — the same seed yields identical labels, centroids, and inertia. KMeans uses the crate’s internal xorshift64 PRNG, preserving the zero-dependency ethos (no rand crate).

Serialization: KMeans derives Serialize/Deserialize under the serde feature — fitted centroids, labels, and inertia round-trip through JSON, and the restored model serves predictions without refitting.

Choosing a clustering algorithm

Evaluating clusters

Without ground-truth labels, the silhouette score assesses clustering quality from the data alone:

#![allow(unused)]
fn main() {
use datarust::cluster::metrics::silhouette_score;

let s = silhouette_score(&x, &labels)?;  // f64 in [-1, 1]
}

For each sample, the silhouette coefficient is (b − a) / max(a, b) where a is the mean intra-cluster distance and b is the mean distance to the nearest other cluster. Values near 1 indicate well-separated clusters; near 0 indicate overlapping clusters. The inertia() (within-cluster sum of squares, lower is better) is also useful for comparing runs with different n_clusters or random_state values.

Metrics

Model evaluation. Mirrors sklearn.metrics. Live in datarust::metrics.

Regression metrics

In datarust::metrics::regression. Each takes y_true and y_pred as &[f64].

#![allow(unused)]
fn main() {
use datarust::metrics::regression::*;

let mse  = mean_squared_error(&y_true, &y_pred, true)?;   // squared=true → MSE
let rmse = mean_squared_error(&y_true, &y_pred, false)?;  // squared=false → RMSE
let mae  = mean_absolute_error(&y_true, &y_pred)?;
let r2   = r2_score(&y_true, &y_pred)?;
let me   = max_error(&y_true, &y_pred)?;
let ev   = explained_variance_score(&y_true, &y_pred)?;
}

Classification metrics

In datarust::metrics::classification. Hard-label metrics accept arbitrary non-negative integer labels such as {2, 5, 9} and compact them internally. Fractional, negative, or non-finite labels are rejected. The compatibility precision, recall, and F1 functions use macro averaging; the *_with variants make the averaging strategy explicit.

#![allow(unused)]
fn main() {
use datarust::metrics::classification::*;

// Hard-label metrics (work for binary and multiclass):
let acc  = accuracy_score(&y_true, &y_pred)?;
let prec = precision_score(&y_true, &y_pred)?;     // macro-avg for multiclass
let rec  = recall_score(&y_true, &y_pred)?;
let f1   = f1_score(&y_true, &y_pred)?;
let cm   = confusion_matrix(&y_true, &y_pred)?;    // compact Vec<Vec<usize>>, n×n
let labeled = confusion_matrix_labeled(&y_true, &y_pred)?;
// labeled.labels[i] identifies row and column i

let weighted_f1 = f1_score_with(&y_true, &y_pred, Average::Weighted)?;
let positive_f1 = f1_score_with(
    &y_true,
    &y_pred,
    Average::Binary { positive_label: 5.0 },
)?;
let per_class = classification_report(&y_true, &y_pred)?;
let ll   = log_loss(&y_true, &y_proba, 1e-15)?;     // {0, 1}; P(label = 1)

// Ranking metrics for {0, 1}; scores refer to label 1:
let auc  = roc_auc_score(&y_true, &y_score)?;        // ROC-AUC (Mann–Whitney U)
let ap   = average_precision_score(&y_true, &y_score)?; // average precision

// For another binary label space, name the positive class explicitly:
let ll_custom = log_loss_with_positive_label(&y_true, &y_proba, 5.0, 1e-15)?;
let auc_custom = roc_auc_score_with_positive_label(&y_true, &y_score, 5.0)?;
let ap_custom = average_precision_score_with_positive_label(&y_true, &y_score, 5.0)?;

// Agreement & correlation (binary + multiclass):
let kap  = cohen_kappa_score(&y_true, &y_pred)?;     // chance-corrected agreement
let mcc  = matthews_corrcoef(&y_true, &y_pred)?;     // Matthews correlation
}

precision_score_with_options, recall_score_with_options, and f1_score_with_options additionally accept ZeroDivision::Zero (the default) or ZeroDivision::Error for undefined class metrics.

Clustering metrics

In datarust::cluster::metrics. Evaluate clustering quality without ground-truth labels.

#![allow(unused)]
fn main() {
use datarust::cluster::metrics::silhouette_score;

let s = silhouette_score(&x, &labels)?;  // [-1, 1], higher is better
}

Cluster IDs are compacted internally rather than used as vector indices. At least two clusters and fewer clusters than samples are required, and a sample in a singleton cluster receives coefficient zero.

Estimator .score() shorthand

Every estimator has a built-in score method:

• Regression models (LinearRegression, Ridge, Lasso) → R².
• LogisticRegression → accuracy.

#![allow(unused)]
fn main() {
let r2 = ridge.score(&x, &y)?;       // R²
let acc = logistic.score(&x, &y)?;   // accuracy
}

Model Selection

Train/test splitting and cross-validation. Mirrors sklearn.model_selection. Live in datarust::model_selection.

train_test_split

Split X and y into train and test subsets.

#![allow(unused)]
fn main() {
use datarust::model_selection::{train_test_split, TrainTestSplit};

// Quick split with defaults (25% test, shuffled):
let (x_tr, x_te, y_tr, y_te) = train_test_split(&x, &y)?;

// Or configure via the builder:
let (x_tr, x_te, y_tr, y_te) = TrainTestSplit::new()
    .with_test_size(0.2)        // fraction in (0, 1)
    .with_shuffle(true)
    .with_random_state(42)      // deterministic seed
    .split(&x, &y)?;
}

The split requires at least two samples. Fractional test sizes round up, so a 25% test share uses two rows when the dataset contains five rows.

KFold

K-fold cross-validation. Each sample serves as the test set exactly once.

#![allow(unused)]
fn main() {
use datarust::model_selection::KFold;

let cv = KFold::new()
    .with_n_splits(5)
    .with_shuffle(true)
    .with_random_state(42);

for (train_idx, test_idx) in cv.split(n_samples)? {
    let x_train = x.select_rows(&train_idx)?;
    let x_test = x.select_rows(&test_idx)?;
    // ...
}
}

StratifiedKFold

Approximately preserves class balance across folds. It accepts two or more non-negative integer-valued classes, including gapped labels such as {2, 5, 9}. Essential for imbalanced classification.

#![allow(unused)]
fn main() {
use datarust::model_selection::StratifiedKFold;

let cv = StratifiedKFold::new().with_n_splits(5);

for (train_idx, test_idx) in cv.split(&y)? {  // note: takes y, not n_samples
    // each fold has roughly the same class ratio as the full dataset
}
}

cross_val_score

Evaluate any Predictor + Clone estimator by K-fold cross-validation with a user-supplied scorer.

#![allow(unused)]
fn main() {
use datarust::model_selection::{cross_val_score, KFold};
use datarust::linear_model::LinearRegression;
use datarust::metrics::regression::r2_score;

let cv = KFold::new().with_n_splits(5);
let scores = cross_val_score(&LinearRegression::new(), &x, &y, &cv, r2_score)?;
// scores.len() == 5, one R² per fold
}

For classification, pass accuracy_score instead:

#![allow(unused)]
fn main() {
use datarust::linear_model::LogisticRegression;
use datarust::metrics::classification::accuracy_score;

let scores = cross_val_score(&LogisticRegression::new(), &x, &y, &cv, accuracy_score)?;
}

The scorer is any closure Fn(&[f64], &[f64]) -> Result<f64>, so you can pass custom metrics too.

Compose: Pipeline & ColumnTransformer

Chain transformers and dispatch different columns to different transformers. Live in datarust::pipeline and datarust::compose.

Pipeline

Chain multiple transformers sequentially. Each step fits on the output of the previous one.

#![allow(unused)]
fn main() {
use datarust::pipeline::Pipeline;
use datarust::scaler::{StandardScaler, MinMaxScaler, RobustScaler};
use datarust::transformer_kind::TransformerKind;

let mut pipe = Pipeline::new()
    .push("standard", TransformerKind::StandardScaler(StandardScaler::new()))
    .push("minmax",   TransformerKind::MinMaxScaler(MinMaxScaler::new()))
    .push("robust",   TransformerKind::RobustScaler(RobustScaler::new()));

let out = pipe.fit_transform(&x)?;
}

Pipelines are serializable under the serde feature — fit, save to JSON, load and transform in a different process. Before use, leaf transformers validate the dimensions and required fields of loaded fitted state, so inconsistent JSON returns an error instead of panicking.

Supervised pipeline

Attach a final estimator with with_estimator to fit preprocessing and a model together. During fit(X, y), supervised selectors such as SelectKBest receive the training targets before the final estimator is fitted.

#![allow(unused)]
fn main() {
use datarust::linear_model::LogisticRegression;
use datarust::pipeline::Pipeline;
use datarust::selection::{ScoreFunc, SelectKBest};
use datarust::traits::Predictor;
use datarust::transformer_kind::TransformerKind;

let mut model = Pipeline::new()
    .push("select", TransformerKind::SelectKBest(SelectKBest::new(ScoreFunc::FClassif, 5)?))
    .with_estimator(LogisticRegression::new());
model.fit(&x, &y)?;
let labels = model.predict(&x)?;
}

Runtime step inspection

#![allow(unused)]
fn main() {
pipe.names();           // ["standard", "minmax", "robust"]
pipe.get_step("minmax");  // Option<&TransformerKind>
pipe.set_step("robust", TransformerKind::StandardScaler(StandardScaler::new()));
pipe.insert_step(0, "impute", /* ... */);
pipe.remove_step("standard");
}

ColumnTransformer

Dispatch different columns to different transformers — the workhorse for mixed numeric + categorical data.

#![allow(unused)]
fn main() {
use datarust::compose::{ColumnTransformer, Remainder, Table};
use datarust::encoder::OneHotEncoder;
use datarust::scaler::StandardScaler;
use datarust::categorical_kind::CategoricalTransformerKind;
use datarust::transformer_kind::TransformerKind;

let mut ct = ColumnTransformer::new()
    .add_numeric(
        "nums",
        vec![0, 1, 2],           // column indices
        TransformerKind::StandardScaler(StandardScaler::new()),
    )
    .add_categorical(
        "cats",
        vec![3, 4],
        CategoricalTransformerKind::OneHotEncoder(OneHotEncoder::new()),
    )
    .remainder(Remainder::Passthrough); // keep un-specified columns

// Table bundles numeric (Matrix) + categorical (StrMatrix) of equal row count
let out = ct.fit_transform_to_table(&table)?;
// out.numeric    — the scaled + one-hot-encoded numeric block
// out.categorical — the transformed categorical block
}

Supervised encoders

For TargetEncoder, use fit_with_target / fit_transform_with_target:

#![allow(unused)]
fn main() {
ct.fit_transform_with_target(&table, &y)?;
}

When to use what

Datasets

Classic toy datasets for examples, tests, and onboarding. Live in datarust::datasets. Enable with the datasets feature:

[dependencies]
datarust = { version = "*", features = ["datasets"] }

The data is compiled into the binary as const arrays — no file I/O, no network access, no external dependencies. Each loader returns a Dataset struct with features(), targets(), feature_names(), and target_names().

Available datasets

Usage

#![allow(unused)]
fn main() {
use datarust::datasets;

let iris = datasets::iris::load();

let x = iris.features();           // Matrix 150×4
let y = iris.targets();            // &[f64], values {0, 1, 2}
let names = iris.feature_names();  // &["sepal_length", "sepal_width", ...]
let classes = iris.target_names(); // &["setosa", "versicolor", "virginica"]

assert_eq!(iris.n_samples(), 150);
assert_eq!(iris.n_features(), 4);
assert_eq!(iris.n_classes(), 3);
}

Feed directly into a model:

#![allow(unused)]
fn main() {
use datarust::datasets::iris;
use datarust::linear_model::LogisticRegression;
use datarust::traits::Predictor;

let data = iris::load();
let x = data.features();
let y = data.targets().to_vec();

let mut model = LogisticRegression::new().with_max_iter(200);
model.fit(&x, &y)?;
let accuracy = model.score(&x, &y)?;
println!("Iris accuracy: {:.1}%", accuracy * 100.0);
}

Normalized Diabetes

diabetes::load() returns the raw feature values. Since v0.7, diabetes::load_normalized() returns the layout sklearn’s load_diabetes(scaled=True) produces — each feature centered by its column mean and divided by its population standard deviation (and by sqrt(442)), so the columns have zero mean and unit-ish scale:

#![allow(unused)]
fn main() {
use datarust::datasets::diabetes;

let raw = diabetes::load();               // original units
let scaled = diabetes::load_normalized(); // sklearn scaled=True layout
}

The raw literals were re-precisioned in v0.7 to sklearn’s exact f64 values, so load() output matches sklearn bit-for-bit (previously it could differ at the last-ULP level).

Choosing a dataset

datarust-profile

One-call data profiling and data-quality reports for the datarust ecosystem.

#![allow(unused)]
fn main() {
use datarust::Matrix;
use datarust_profile::{profile_matrix, report};

let m = Matrix::from_rows(vec![
    vec![1.0, 10.0],
    vec![2.0, 12.0],
    vec![3.0, f64::NAN], // missing income in row 3
])?;

let profile = profile_matrix(&m, Some(&["age".into(), "income".into()]))?;
println!("{}x{}, {} duplicate rows",
    profile.n_rows, profile.n_columns, profile.duplicate_rows);

// Self-contained HTML report — no dependencies, no JavaScript.
std::fs::write("profile.html", report::to_html(&profile))?;
}

Since v0.4, profiling can also start straight at the file — a pure-Rust RFC-4180 CSV reader (csv feature), a single-pass streaming profiler for large files (streaming feature), a terminal summary (pretty feature), and a datarust-profile CLI binary (cli feature):

#![allow(unused)]
fn main() {
use datarust_profile::DatasetProfile;

let profile = DatasetProfile::from_csv("customers.csv")?;      // csv feature
let big = datarust_profile::profile_csv_streaming("logs.csv")?; // streaming feature
}

datarust-profile customers.csv --pretty --target churn

Default build has zero external dependencies beyond datarust itself. Statistics flow through datarust::stats; the profiling crate owns the interpretation (summaries, quality flags, reports). JSON output is opt-in via the serde feature; CSV, streaming, terminal rendering, and the CLI are opt-in via their own features.

What it computes

For each column, depending on its inferred type:

Pairwise Relationships (Relationships block):

• Pearson correlation matrix over numeric columns (CorrelationMatrix).
• Cramér’s V matrix over categorical columns (pure-Rust χ² association).
• Point-biserial correlation between binary categorical and numeric columns (PointBiserialEntry).

Dataset-wide: n_rows, n_columns, estimated memory_bytes, exact duplicate_rows and duplicate_fraction, optional target_column.

Data-quality findings

quality::run_checks scans the profile against configurable Thresholds and emits QualityIssues across 8 categories:

• HighMissing — missing fraction at/above threshold.
• ConstantColumn — numeric column with near-zero variance.
• NearUnique — categorical column whose cardinality ≈ row count (likely an identifier).
• DuplicateRows — exact-duplicate rows present.
• Outliers — values outside Tukey IQR fences.
• Imbalance — categorical column dominated by a single value.
• HighCorrelation — pair of numeric columns exceeding correlation threshold (|r| >= 0.95).
• TargetLeakage — feature column strongly correlated (|r| >= 0.90 or V >= 0.90) with designated target column.

Each finding carries a Severity (Info / Warning / Critical) and an optional column name. The HTML and JSON renderers include findings and correlation heatmaps by default.

Output formats

#![allow(unused)]
fn main() {
#[cfg(feature = "serde")]
{
    use datarust_profile::report::{to_json, JsonReport};
    let json = to_json(&JsonReport::from_profile(&profile))?;
    std::fs::write("profile.json", json)?;
}
}

Relationship to datarust

datarust-profile is a sibling crate in the datarust workspace. It reuses datarust’s Matrix / StrMatrix containers and its stats module (mean, std, quantile, median_sorted) rather than reimplementing them. Column-type inference, missing-value handling, cardinality counting, and the report renderers live in this crate.

Next steps

• New to profiling? Start with the Quick Start.
• Want the full API surface? It’s on docs.rs.

Quick Start

A 60-second tour: profile a CSV file or a numeric matrix, read the results, and render a report. Everything here works with the default feature set — no serde required for the HTML report.

Install

[dependencies]
datarust = "0.7"
datarust-profile = "0.4"

For JSON output, enable the serde feature (this also pulls in datarust/serde):

[dependencies]
datarust-profile = { version = "0.4", features = ["serde"] }

Meet the user at the file: from_csv

Since v0.4, the fastest path starts at the CSV file itself (csv feature) — no manual parsing, quoting rules handled for you:

#![allow(unused)]
fn main() {
use datarust_profile::DatasetProfile;

let p = DatasetProfile::from_csv("customers.csv")?;
}

For files too large to hold in memory, the streaming feature profiles in a single forward pass with bounded memory:

#![allow(unused)]
fn main() {
use datarust_profile::{profile_csv_streaming, StreamingOptions};

let p = profile_csv_streaming("customers.csv")?;
}

Or skip Rust entirely with the CLI (cargo install datarust-profile --features cli):

datarust-profile customers.csv --pretty --target churn
datarust-profile customers.csv --streaming -o report.html

Profile a numeric matrix

profile_matrix takes a datarust::Matrix and returns a DatasetProfile. Missing values are encoded as NaN and excluded from every statistic.

#![allow(unused)]
fn main() {
use datarust::Matrix;
use datarust_profile::profile_matrix;

let m = Matrix::from_rows(vec![
    vec![1.0, 10.0],
    vec![2.0, 20.0],
    vec![3.0, 30.0],
    vec![4.0, 40.0],
    vec![5.0, f64::NAN], // missing in column 1
])?;

let p = profile_matrix(&m, Some(&["x".into(), "y".into()]))?;
}

Column names are optional — pass None and they default to x0..x{n-1}.

Read the numeric profile

Each column carries a NumericStats block: central tendency, spread, shape, and outliers.

#![allow(unused)]
fn main() {
let col = &p.columns[0];
let n = col.numeric.as_ref().unwrap();

println!("mean={:.2}  std={:.2}", n.mean, n.std);
println!("min={:.1}  Q1={:.1}  median={:.1}  Q3={:.1}  max={:.1}",
    n.five.min, n.five.q1, n.five.median, n.five.q3, n.five.max);

// v0.2: distributional shape.
println!("skewness={:.3}  kurtosis={:.3}", n.skewness, n.kurtosis);
println!("histogram bins: {} (counts: {:?})", n.histogram.nbins(), n.histogram.counts);
println!("outliers: {} ({:.1}%)", n.outlier_count, n.outlier_fraction * 100.0);
}

Skewness near zero means a symmetric column; large positive values indicate a right tail. Excess kurtosis near zero matches a normal distribution; positive values indicate heavier tails, negative values a flatter peak.

Pairwise relationships & target leakage

v0.3 computes pairwise column relationships: Pearson correlation for numerics, Cramér’s V for categoricals, and point-biserial correlation for binary categorical ⇄ numeric pairs. You can also specify a target column with profile_table_with_target.

#![allow(unused)]
fn main() {
use datarust_profile::profile_table_with_target;

let p = profile_table_with_target(
    Some(&numeric_matrix),
    Some(&categorical_matrix),
    &["age".into(), "income".into(), "churn".into()],
    "churn",
)?;

if let Some(rels) = &p.relationships {
    if let Some(pearson) = &rels.pearson {
        println!("Pearson correlation between {} and {}: {:.3}",
            pearson.labels[0], pearson.labels[1], pearson.values[0][1]);
    }
}
}

Profile categorical data

profile_str_matrix infers each column’s type: if every non-empty cell parses as f64 it is treated as numeric, otherwise categorical.

#![allow(unused)]
fn main() {
use datarust::StrMatrix;
use datarust_profile::profile_str_matrix;

let s = StrMatrix::from_strings(vec![
    vec!["25", "Istanbul", "basic"],
    vec!["40", "Ankara", "premium"],
    vec!["31", "Izmir", "basic"],
])?;

let p = profile_str_matrix(&s, Some(&["age".into(), "city".into(), "tier".into()]))?;

let city = &p.columns[1];
let c = city.categorical.as_ref().unwrap();
println!("{} unique values; top is '{}' ({} rows, {:.0}% of data)",
    c.unique, c.top, c.freq, c.imbalance_ratio * 100.0);
}

imbalance_ratio is the share of the most frequent value: 1.0 means a single value dominates the whole column — a data-quality smell.

Mixed tables

profile_table takes a numeric block and a categorical block side by side, sharing the same row count. This is the natural fit for real-world CSV data.

#![allow(unused)]
fn main() {
use datarust_profile::profile_table;

let p = profile_table(
    Some(&numeric_matrix),
    Some(&categorical_str_matrix),
    &["age".into(), "income".into(), "city".into(), "tier".into()],
)?;
}

Render a report

The HTML renderer produces a single self-contained document — inline CSS, no JavaScript, no external assets, now with correlation heatmaps. The JSON renderer needs the serde feature, and the pretty feature adds a terminal-table summary (report::to_terminal_auto).

#![allow(unused)]
fn main() {
use datarust_profile::report;

// Always available:
let html = report::to_html(&p);
std::fs::write("profile.html", html)?;

#[cfg(feature = "pretty")]
println!("{}", report::to_terminal_auto(&p));

#[cfg(feature = "serde")]
{
    let json = report::to_json(&report::JsonReport::from_profile(&p))?;
    std::fs::write("profile.json", json)?;
}
}

The HTML report lays columns out as a responsive card grid and adds a Relationships section with interactive-feel correlation heatmaps (Pearson & Cramér’s V) and point-biserial tables.

Next steps

• The Profiling Guide covers numeric, categorical, relationships, and the full set of data-quality checks.
• The API reference documents every type and function.

Profiling Guide

How to profile numeric, categorical, and mixed data, and how to act on the data-quality findings. Types and functions live in datarust_profile; the statistics they report flow from datarust::stats.

All public APIs return Result; invalid input is a recoverable ProfileError, never a panic.

Profiling numeric data

Pass a Matrix to profile_matrix. Non-finite values (NaN, ±inf) are treated as missing and counted separately — they never poison the mean or standard deviation.

#![allow(unused)]
fn main() {
use datarust::Matrix;
use datarust_profile::profile_matrix;

let m = Matrix::from_rows(vec![
    vec![1.0, 100.0],
    vec![2.0, 110.0],
    vec![3.0, 105.0],
    vec![4.0, 95.0],
    vec![5.0, f64::NAN], // missing reading in column 1
])?;

let p = profile_matrix(&m, Some(&["index".into(), "reading".into()]))?;
}

Each numeric column yields a NumericStats:

The histogram bin count follows Sturges’ rule (ceil(log2(n) + 1), floored at 1). Read histogram.counts directly, or use histogram.nbins() / histogram.max_count() for rendering.

Fast path

profile_matrix routes the mean, variance, and five-number summary through datarust’s fused flat-buffer helpers (column_mean_var_flat, column_quantiles_many_flat) over Matrix::as_slice(), avoiding per-column Vec allocation on wide tables. Columns containing NaN fall back to the per-column path, since the flat helpers are not NaN-aware.

Profiling categorical data

profile_str_matrix infers each column’s type. A column is Numeric when every non-empty cell parses as f64; otherwise it is Categorical. Empty markers recognised as missing: "", NA, N/A, null, NaN, None, -, ?.

#![allow(unused)]
fn main() {
use datarust::StrMatrix;
use datarust_profile::profile_str_matrix;

let s = StrMatrix::from_strings(vec![
    vec!["Istanbul", "basic"],
    vec!["Ankara", "premium"],
    vec!["Izmir", "basic"],
    vec!["Istanbul", "basic"],
])?;

let p = profile_str_matrix(&s, Some(&["city".into(), "tier".into()]))?;
}

A CategoricalStats reports unique (cardinality), top / freq (most frequent value and its count), imbalance_ratio (freq ÷ non-missing cells), and top_values — the top-N (value, count) pairs, sorted descending, for frequency charts.

Mixed tables

Real datasets mix numeric and categorical columns. profile_table takes an optional numeric [Matrix] and an optional categorical StrMatrix side by side, with one shared row count. names must list every column across both blocks (numerics first, then categoricals).

#![allow(unused)]
fn main() {
use datarust_profile::profile_table;

let p = profile_table(
    Some(&numeric),
    Some(&categorical),
    &["age".into(), "income".into(), "city".into(), "tier".into()],
)?;
}

Duplicate-row detection works across both blocks: a row is a duplicate only if its numeric and categorical cells match an earlier row exactly.

Profiling CSV files (v0.4, csv feature)

DatasetProfile::from_csv (and the profile_csv* free functions) parse the file with the crate’s own pure-Rust RFC-4180 reader — quoted fields with "" escapes, embedded delimiters and newlines inside quotes, LF and CRLF line endings, UTF-8 BOM stripping, configurable delimiter/quote bytes, optional trimming, and strict ragged-row rejection that reports the physical line number:

#![allow(unused)]
fn main() {
use datarust_profile::{DatasetProfile, CsvReadOptions};

let p = DatasetProfile::from_csv("customers.csv")?;

// Or with options (delimiter, headerless, trimming):
let opts = CsvReadOptions { delimiter: b';', has_header: false, ..Default::default() };
let p = DatasetProfile::from_csv_with("customers.csv", &opts)?;
}

Forcing types per column is also supported — schema-forced Numeric columns turn non-parseable cells into missing values instead of flipping the column type:

#![allow(unused)]
fn main() {
use datarust_profile::types::ColumnType;

let schema = [ColumnType::Numeric, ColumnType::Categorical];
let p = DatasetProfile::from_csv_with_types("data.csv", &schema)?;
}

Streaming large files (v0.4, streaming feature)

profile_csv_streaming profiles a CSV in a single forward pass with bounded memory — the natural fit for files that don’t fit in RAM:

#![allow(unused)]
fn main() {
use datarust_profile::{profile_csv_streaming_with, StreamingOptions};

let opts = StreamingOptions {
    reservoir_capacity: 10_000, // default; quantile/histogram sample size
    compute_duplicates: true,   // opt-in exact duplicate tracking
    ..StreamingOptions::default()
};
let p = profile_csv_streaming_with("huge.csv", &opts)?;
}

The fidelity contract, briefly:

• Exact: row/cell counts, missing counts, mean, variance/std, min/max, categorical frequencies, duplicate rows (when enabled).
• Approximate: quantiles, histogram, and outlier counts (reservoir sample — exact whenever the table fits within reservoir_capacity rows), and unique beyond distinct_cap distinct values.
• Column types are decided online: a numeric column permanently converts to categorical at its first non-parseable cell (a documented one-pass trade-off).

Terminal reports & the CLI (v0.4, pretty / cli features)

report::to_terminal_auto renders a fixed-width summary — per-column type, missing percentage, μ/σ or top values — plus the quality findings with CRIT/WARN/INFO tags, with ANSI color only on a TTY and NO_COLOR support:

#![allow(unused)]
fn main() {
#[cfg(feature = "pretty")]
println!("{}", datarust_profile::report::to_terminal_auto(&p));
}

The cli feature builds the datarust-profile binary that wraps all of it:

datarust-profile data.csv                       # terminal summary
datarust-profile data.csv -o report.html        # self-contained HTML
datarust-profile data.csv --json report.json    # JSON (serde)
datarust-profile data.csv --streaming --pretty  # single-pass, bounded memory
datarust-profile data.csv --target churn --delimiter ';' --types numeric,categorical

Exit codes: 0 success, 1 runtime error (I/O or parsing), 2 usage error (unknown --target column, bad --types token, and the like — validated against the header before the file is profiled).

Pairwise relationships & target leakage

v0.3 computes pairwise interactions across columns:

• Pearson correlation matrix (rels.pearson): computed over numeric columns using datarust::stats::correlation_matrix.
• Cramér’s V matrix (rels.cramers_v): computes categorical association (V ∈ [0.0, 1.0]) via a pure-Rust contingency table.
• Point-biserial correlation (rels.point_biserial): measures correlation between binary categorical columns and continuous numeric columns.

Target-leakage detection

To check for feature leakage against a target column, construct the profile with profile_matrix_with_target or profile_table_with_target:

#![allow(unused)]
fn main() {
use datarust_profile::profile_table_with_target;

let p = profile_table_with_target(
    Some(&numeric),
    Some(&categorical),
    &["age".into(), "income".into(), "city".into(), "churn".into()],
    "churn",
)?;
}

If a feature column has high correlation or association (|r| >= 0.90 or V >= 0.90) with the target column, run_checks fires a QualityKind::TargetLeakage issue.

Data quality checks

run_checks turns a profile into a list of QualityIssues. The thresholds are conservative defaults; tune them via Thresholds.

#![allow(unused)]
fn main() {
use datarust_profile::quality::{Thresholds, QualityKind};
use datarust_profile::quality::checks::run_checks;

let mut t = Thresholds::default();
t.outlier_fraction = 0.02;  // flag columns with ≥2% outliers
t.high_correlation = 0.90;  // flag numeric pairs with |r| >= 0.90
t.target_leakage = 0.85;    // flag feature-target correlation >= 0.85

for issue in run_checks(&p, &t) {
    println!("{:?} [{}] {}: {}",
        issue.kind, issue.severity,
        issue.column.as_deref().unwrap_or("(dataset)"),
        issue.message);
}
}

The eight checks

Each QualityIssue carries a Severity (Info, Warning, Critical) and an optional column name (dataset-wide findings like DuplicateRows set column to None).

Rendering reports

The report module renders a profile plus its findings. HTML is always available; the terminal table needs the pretty feature; JSON needs the serde feature.

#![allow(unused)]
fn main() {
use datarust_profile::report;

// HTML: self-contained, no dependencies.
let html = report::to_html(&p);

// Bring your own findings (e.g. with custom thresholds):
let findings = run_checks(&p, &t);
let html = report::to_html_with(&p, &findings);

// Terminal (pretty feature):
#[cfg(feature = "pretty")]
let text = report::to_terminal_auto(&p);

// JSON (serde feature):
#[cfg(feature = "serde")]
let json = report::to_json(&report::JsonReport::from_profile(&p))?;
}

The HTML report uses a responsive card grid: numeric cards carry summary statistics, CSS mini-histograms, and outlier counts; categorical cards carry top values, imbalance ratios, and frequency bar charts. A Relationships section presents correlation heatmaps for numeric (Pearson) and categorical (Cramér’s V) pairs, plus point-biserial tables. Findings appear in a severity-coloured list at the top.

Roadmap

The canonical, detailed roadmap lives in ROADMAP.md inside the datarust-profile crate. This page gives a higher-level tour of why each phase is ordered the way it is, and what it unlocks.

The destination

datarust-profile is heading toward v1.0: a complete one-line data-profiling and data-quality toolkit for the datarust ecosystem — zero external dependencies by default, reusing datarust::stats for the numerical kernels, owning the interpretation layer (summaries, flags, reports).

Where we are — v0.4.0

Data loading & ergonomics landed: a pure-Rust RFC-4180 CSV reader behind the csv feature, one-call profiling entry points (from_csv, from_csv_with_types), a streaming single-pass profiler with bounded memory (Welford moments, reservoir-sampled quantiles) behind the streaming feature, a terminal-table renderer behind pretty, and the datarust-profile CLI binary (CSV → HTML/JSON/terminal) behind cli. The default build still has zero external dependencies. Eight quality checks run out of the box.

The release track

v0.3 — Relationships & interaction (Shipped)

Columns in isolation miss collinearity, redundant features, and leakage risk. v0.3 introduced pairwise and dataset-wide relationship analysis, reusing datarust::stats’s correlation matrix, plus a pure-Rust Cramér’s V for categorical pairs and point-biserial correlation for binary categorical ⇄ numeric pairs. A target-leakage hint flags features highly correlated with a designated target column, and correlation heatmaps are rendered in the HTML report.

v0.4 — Data loading & ergonomics (Shipped)

Meet the user at the file, not the in-memory Matrix. A csv feature adds one-call profiling from a path, a cli binary generates reports from the shell, and a streaming single-pass mode profiles large files without holding them in memory. All four landed behind optional feature flags — the default build stays zero-dependency.

v0.5 — Missingness & comparison

Promote missing values from a count to a structured analysis (which columns go missing together), and add dataset comparison: schema diff plus per-column distributional drift (population stability index), the backbone of production data monitoring.

v0.6 — Text & semantics

Richer categorical profiling: string-length distributions, casing/format flags that catch silent duplicates ("USA" vs "usa "), and identifier-vs-feature scoring beyond the current binary NearUnique check.

v0.7 — Performance (In progress; streaming groundwork shipped in v0.4)

The hot paths are already competitive with the reference tools: row deduplication is hash-based instead of an O(n²) scan, categorical columns are gathered as borrowed &str slices (no per-cell String clones) with all infer/profile/relationships paths generic over T: AsRef<str>, missing detection no longer allocates, and Cramér’s V runs over pre-encoded integer codes. A criterion benchmark suite now guards throughput (profile_str_matrix 100 000 × 20: 269 ms → 132 ms). Remaining: a rayon feature for parallel per-column statistics.

v0.8 — Time series

Temporal structure is invisible to column-wise profiling. v0.8 adds datetime type inference, monotonicity and gap detection, autocorrelation at configurable lags, and seasonal-strength hints.

v0.9 — Extension & quality rules

A QualityCheck trait lets teams codify their own data contracts (uniqueness, allowed-value sets, ranges, regex patterns), serialised as a schema file and checked in CI with a non-zero exit code on failure.

v1.0 — Stability

Freeze the public API, version the JSON output schema, publish the SemVer stability statement, and document a full parity matrix against pandas describe / ydata-profiling.

For the granular deliverables and the explicit out-of-scope list, see the canonical roadmap on GitHub.

Performance vs scikit-learn

The numbers below are measured, not estimated. The same deterministic synthetic dataset (xorshift64, seed 42, values in [-100, 100)) is fed to both libraries, and the median fit_transform time over 15 runs (after one warmup) is reported.

Test setup: Apple M5 Pro (18 cores, arm64), Rust 1.96.0 (release), Python 3.9.6, scikit-learn 1.6.1, numpy 2.0.2, scipy 1.13.1. Times are in milliseconds. The Ratio column is sklearn_ms / datarust_ms — values > 1 mean datarust is faster.

Benchmark table

Feature flags make a difference

matrixmultiply. Enabling this feature dispatches covariance and matmuls to a tuned pure-Rust GEMM (no system BLAS). On 50 000 × 200, PCA drops from 206 ms → 111 ms (1.9× faster), and LinearRegression fit+predict from 263 ms → 91 ms (2.9× faster).

rayon. Parallel column/row processing. RobustScaler at 50 000 × 200 drops from 68.7 ms (default) to 14.7 ms (4.7× faster with rayon).

Scalar vs matrixmultiply kernels

The table below isolates what the optional matrixmultiply feature buys inside the numeric hot paths — medians from the criterion suite (cargo bench -p datarust --bench benchmarks -- --warm-up-time 0.5 --measurement-time 1.5 --sample-size 10), single-threaded release build on the same Apple M5 Pro hardware as above. +GEMM is the same build with features = ["matrixmultiply"]; Speedup is scalar / +GEMM. Each time carries its own unit (ms / µs / ns).

The GEMM pays off most on matmul-heavy kernels — Matrix::matmul 1.8–2.8×, LinearRegression fit 2.6–2.7×, TruncatedSVD ~2.0× — while predict paths are memory-bound matvecs and barely move (~1.0×). Pearson no longer needs GEMM for wide tables (the lower-triangle scalar covariance puts the kernel at 6.0 ms for 10 000 × 100); PCA barely moves at these small sizes and shines on larger matrices.

Where datarust wins

• Mixed numeric + categorical composition. ColumnTransformer is 153–380× faster than scikit-learn’s on large inputs. This is the headline result — it reflects the cost of sklearn’s per-column Python dispatch, dtype coercion, and object-array marshalling.
• String / categorical encoding. OneHotEncoder is ~2.9–4.0× faster because datarust operates on a native StrMatrix directly — no Python object-array overhead, no GIL.
• Numeric scalers with rayon. StandardScaler/RobustScaler/Pipeline beat sklearn by 2.1–13× at 50 000 × 200.
• Small data and startup latency. At 1 000 × 10, datarust is faster on every workload — up to 8.9× on StandardScaler and 153× on ColumnTransformer. No Python interpreter to spin up, no numpy import cost.

Where scikit-learn still wins

• PCA on tall-and-wide data (without the matrixmultiply feature). sklearn calls into LAPACK’s full SVD via shared-library BLAS; datarust uses a from-scratch Jacobi sweep. With matrixmultiply the gap narrows from ~17× to ~9×, and PCASolver::Randomized closes it further for low-rank inputs.

Criterion microbenchmark suite

Alongside the sklearn comparison, the criterion suite (cargo bench -p datarust --bench benchmarks and cargo bench -p datarust-profile --bench benchmarks) covers the whole API surface. The medians below are from the default (zero-dependency) build with --warm-up-time 0.5 --measurement-time 1.5 --sample-size 10 on the same Apple M5 Pro, and document the operations added most recently. Each time carries its own unit.

Feature-build comparisons: Criterion identifies a benchmark by group and ID, not Cargo features. Use a distinct CARGO_TARGET_DIR for the default and matrixmultiply runs; otherwise its change output compares unlike builds and falsely reports the feature switch as a regression.

CARGO_TARGET_DIR=target/criterion-default cargo bench -p datarust --bench benchmarks
CARGO_TARGET_DIR=target/criterion-matrixmultiply cargo bench -p datarust \
  --bench benchmarks --features matrixmultiply

datarust

The stats_nested group mirrors the flat-storage kernels over Vec<Vec<f64>> inputs; mode_column shares mode’s sort-then-scan, so a 100 000 × 20 mostly-distinct table modes in 27.7 ms. matrix_ops/from_flat (49 µs at 10 000 × 50) shows why flat construction is the fast path over the nested from_nested (~289 µs for the same shape).

The next pass added eight groups covering the production-time paths that fit_transform-only benchmarks hid — transform/inverse_transform on fitted transformers, predict_proba, fitted-encoder transforms, sparse one-hot, the remaining classification metrics, the splitters, and string-column gathering:

Three rows are the current bottlenecks. QuantileTransformer.transform is still the per-value outlier — ~7.3 ns per value (14.6 ms at 100 000 × 20, down from 23.4 ms) against ~0.85 ns for the linear scalers: each value maps into one of 512 pre-partitioned value spans over the 1 000-point reference and interpolates inside that span’s handful of entries — O(1) plus a few compares instead of a full log₂(1000) binary search, with bit-identical output. The remaining cost is the per-value interpolation itself (the nested-Vec transpose round-trips are gone; the transform now streams the flat buffer row-major). StrMatrix::column (clone) is ~8× slower than the borrowing column_refs (72.0 ms vs 8.12 ms at 100 000 × 20) — all four categorical encoders used to pay this clone tax in their fit paths, and switching them to column_refs made fit_transform 60–83% faster (ordinal/frequency fit at 10 000 × 20: ~21 ms → ~3.6–3.9 ms, with transform unchanged). StratifiedKFold.split used to rebuild each fold’s train set with a per-fold HashSet scan of every sample; a reusable boolean mask (mark → scan → reset) cut it from 6.31 ms to 1.86 ms at 100 000 rows (−71%), leaving the stratification setup itself as the remaining cost. onehot_sparse transform_sparse was 2× slower than the dense transform because it round-tripped through per-row triplet Vecs and SparseMatrix::from_triplets’ per-row re-sort; it now builds the CSR arrays directly — 18.6 ms vs 14.3 ms dense at 50 000 × 20 (down from 27.6 ms), and fit_transform_sparse dropped from 78.3 ms to 32.9 ms.

datarust-profile

The high-cardinality groups pin down the worst cases of wide-table profiling: near-unique identifier columns (10 000 × 20) take ~83 ms end-to-end — dominated by duplicate detection and per-column categorical tallies — and Cramér’s V over 1 000-level columns costs ~6 ms per 10-column table. report_json scales with the column count, not the row count (the profile is column-summarized), so 10 000 × 20 and 100 000 × 20 serialize in ~36–39 µs.

Reproduce the benchmarks

The harness lives in examples/bench_compare_rust.rs (Rust side) and benches/compare_sklearn.py (Python side). Run on your own hardware:

# Rust (all feature combos)
cargo run --release --features matrixmultiply --example bench_compare_rust 15

# Python (requires numpy, scikit-learn)
python3 benches/compare_sklearn.py 15

Feature Comparison: datarust vs sklearn

A side-by-side view of what’s implemented. ✓ = supported, — = no equivalent.

Preprocessing

Encoders

Imputation & Selection

Models & Decomposition

Model Selection & Metrics

Composition & Infrastructure

Where datarust goes beyond sklearn

• FrequencyEncoder — count/proportion encoding not in sklearn.
• Pipeline runtime editing — get_step, set_step, insert_step, remove_step for live pipeline surgery.
• JSON serialization — human-readable, language-agnostic fitted-model persistence (vs sklearn’s binary joblib/pickle).
• Type-safe categorical/numeric separation — the compiler prevents mixing StrMatrix and Matrix.

Rust ecosystem comparison

datarust is a preprocessing-first classical ML library. Within the Rust ecosystem its direct peers are smartcore and linfa. Deep-learning frameworks (candle, burn, tch-rs) solve a different problem and are out of scope here; rusty-machine has been archived and is omitted.

Design philosophies

The three libraries were built around different priorities, which shows up in every design decision:

• datarust optimizes for preprocessing depth and zero-dependency deployability. All linear algebra is pure Rust (Jacobi eigensolver, Cholesky, coordinate descent), so the default build has no external C libraries — it links cleanly into WASM, embedded targets, and CLI tools. Since v0.7 it also ships CART decision trees and random forests whose splitter reproduces scikit-learn’s bit-for-bit. The remaining trade-off is algorithm breadth: SVM, DBSCAN, and boosting are not implemented yet.
• smartcore optimizes for single-crate algorithm breadth. One dependency gives you SVM, RandomForest, DecisionTree, KMeans, DBSCAN, KNN, NaiveBayes, and more, plus model selection and metrics. The trade-off is preprocessing: only StandardScaler and OneHotEncoder are provided, and the crate depends on ndarray plus a BLAS backend.
• linfa optimizes for modularity. Each algorithm family lives in its own crate (linfa-svm, linfa-trees, linfa-clustering, linfa-ensemble, …) behind a shared linfa-core trait surface, so you only compile what you use. linfa-preprocessing additionally offers scalers and text vectorizers (Count / TF-IDF). The trade-off: categorical encoders, imputers, and feature selectors are sparse or undocumented, and Ridge/Lasso exist only through linfa-elasticnet’s l1_ratio parameter.

Feature matrix

Verified against the July 2026 releases: smartcore 0.5.3, linfa 0.8.1. Legend: ✓ present, ✗ confirmed absent, ? not clearly documented at the time of writing — please open an issue or PR if a cell goes stale.

Preprocessing & Encoders

Models & Decomposition

Infrastructure

When to choose which

Complementary use

These libraries are not mutually exclusive. Because all three expose plain Vec/matrix in/out interfaces, you can mix them in a single pipeline: use datarust for preprocessing (where its coverage is unique), then hand the transformed features to a smartcore or linfa estimator for an algorithm datarust doesn’t implement yet — SVM and DBSCAN are the current examples.

Examples

Runnable examples from the examples/ directory. Each can be run with cargo run --example <name>.

Basic preprocessing

cargo run --example basic_preprocessing

Standardize, scale, and encode a small mixed-type dataset. Demonstrates StandardScaler, MinMaxScaler, OneHotEncoder, and the Transformer trait.

Pipeline workflow

cargo run --example pipeline_workflow

Build a 3-step pipeline (StandardScaler → MinMaxScaler → RobustScaler), fit and transform, then inspect individual steps via the runtime ergonomics API.

Target encoding

cargo run --example target_encoding

Use TargetEncoder with smoothed mean encoding on a high-cardinality categorical feature, including the TargetTransformer trait and fit_with_target.

Regression workflow

cargo run --example regression_workflow

End-to-end regression on synthetic house-price data: generate data, train_test_split, fit a StandardScaler on train only (data-leakage prevention), train a Ridge, score test R², then run 5-fold cross_val_score. Shows the full preprocess → train → evaluate → cross-validate loop on a single realistic dataset.

Classification workflow

cargo run --example classification_workflow

Customer-churn binary classification with mixed numeric + categorical features. A ColumnTransformer applies StandardScaler to numeric columns and OneHotEncoder to the categorical column, feeding a LogisticRegression. Reports the full metric suite (accuracy, precision, recall, f1, confusion_matrix, log_loss), compares decision thresholds (0.3 / 0.5 / 0.7) to show the precision–recall trade-off, and finishes with stratified 5-fold cross-validation driven manually (StratifiedKFold::split, since cross_val_score only accepts KFold).

Regularization comparison

cargo run --example regularization_comparison

Compares Ridge (L2) and Lasso (L1) across multiple alpha values on an 8-feature dataset where only 3 features carry signal, one feature is collinear, and the rest are pure noise. Ridge shrinks all coefficients but zeros none; Lasso’s soft-thresholding drives the noise features to exactly zero (automatic feature selection). The collinear feature would make LinearRegression singular — both regularized solvers handle it.

Model persistence

cargo run --example model_persistence --features serde

Production-style model persistence. Trains a SupervisedPipeline<Ridge> (StandardScaler → PCA → Ridge), writes it to a JSON file with save_json, reloads it with load_json, and confirms the restored model is still is_fitted() and produces bit-identical predictions without refitting. Requires the serde feature.

KMeans clustering

cargo run --example kmeans_clustering

Unsupervised clustering on synthetic 2-D point data. Generates three well-separated blobs, fits a KMeans with k-means++ initialization, inspects the learned centroids against the true blob centers, predicts cluster assignments for new points, compares k-means++ vs random initialization by inertia, and (with --features serde) round-trips the fitted model through JSON. Demonstrates the Clusterer trait.

Decision trees

cargo run --example decision_tree

Seeded CART classification on a synthetic two-feature rectangle problem: a depth-1 stump vs an unlimited tree, Gini vs entropy, and DecisionTreeRegressor reconstructing a feature from pure leaves. Shows with_random_state reproducibility, TreeStructure inspection (depth, leaves, per-node thresholds), feature_importances, and the Sqrt per-split feature strategy — the same seeded-fits-sklearn-exactly kernel the forests build on.

A minimal forest looks like:

#![allow(unused)]
fn main() {
use datarust::ensemble::RandomForestClassifier;
use datarust::traits::{Classifier, Predictor};

let mut forest = RandomForestClassifier::new()
    .with_n_estimators(100)
    .with_random_state(42);
forest.fit(&x_train, &y_train)?;
println!("accuracy: {:.1}%", forest.score(&x_test, &y_test)? * 100.0);
}

Benchmark comparison

cargo run --release --example bench_compare_rust

The deterministic benchmark harness that produces the numbers on the Performance page. Uses a xorshift64 PRNG (seed 42) so Rust and Python generate identical data. Pass a repetition count as an argument:

cargo run --release --features matrixmultiply --example bench_compare_rust 15

Real World Examples

We provide several real-world examples to demonstrate the capabilities of datarust.

• cargo run --example iris --features datasets: Logistic regression on the classic Iris dataset.
• cargo run --example wine --features datasets: Logistic regression on the Wine quality dataset.
• cargo run --example mnist: Digits classification on a synthetic MNIST-like dataset using Logistic Regression.
• cargo run --example housing: Linear regression on a synthetic California-housing-like dataset, predicting median house value.
• cargo run --example titanic: Survival classification using a synthetic Titanic-like dataset with ColumnTransformer (scaling numeric features and one-hot encoding categorical features).
• cargo run --example spam: Email spam classification using synthetic TF-IDF text features.

Using embedded datasets

The datasets feature provides classic toy datasets (Iris, Breast Cancer, Wine, Diabetes) as const arrays — no file I/O needed. See the Datasets guide for the full API.

#![allow(unused)]
fn main() {
use datarust::datasets::iris;
use datarust::linear_model::LogisticRegression;
use datarust::traits::Predictor;

let data = iris::load();
let x = data.features();
let y = data.targets().to_vec();

let mut model = LogisticRegression::new().with_max_iter(200);
model.fit(&x, &y)?;
println!("Iris accuracy: {:.1}%", model.score(&x, &y)? * 100.0);
}

Architecture

How datarust is organized and the design decisions behind it.

Design philosophy

1. Zero external dependencies by default. cargo add datarust pulls in nothing. All linear algebra — Jacobi eigendecomposition, one-sided Jacobi SVD, Cholesky factorization, coordinate descent, IRLS — is pure Rust. Feature flags (serde, rayon, matrixmultiply) opt in to extras.

2. sklearn-inspired but Rust-native. The familiar fit/transform/predict API, but using Result (panic-free), builder-pattern config, and enum-based parameters (Norm::L2, ImputeStrategy::Median) instead of strings.

3. Type-safe modality separation. Four traits enforce that numeric, categorical, target, and label data go to the right transformer. The compiler catches misuse.

4. Composability via type-erased enums. TransformerKind, CategoricalTransformerKind, and TargetTransformerKind allow heterogeneous pipelines that are still serializable.

Module layout

src/
├── lib.rs                 # Crate root: re-exports + module declarations
├── error.rs               # DatarustError enum + Result alias
├── traits.rs              # Estimator, Predictor, Regressor, Classifier, transformers, FeatureNames
├── matrix.rs              # Matrix (f64 flat), StrMatrix, SparseMatrix (CSR)
├── rng.rs                 # public xorshift64* Rng (foundation module)
├── stats.rs               # Column statistics, covariance/correlation
├── pipeline.rs            # Sequential and supervised Pipeline
├── transformer_kind.rs    # TransformerKind enum (type erasure)
├── categorical_kind.rs    # CategoricalTransformerKind enum
├── target_kind.rs         # TargetTransformerKind enum
├── function_transformer.rs
├── polynomial.rs          # PolynomialFeatures
├── serialize.rs           # JSON save/load (serde feature)
├── label_space.rs         # Label vocabulary (fit/encode/decode)
├── datasets/              # Iris, Breast Cancer, Wine, Diabetes (datasets feature)
├── linalg/
│   └── cholesky.rs        # Cholesky decomposition + SPD solver
├── scaler/                # 9 transformers (standard, minmax, robust, ...)
├── encoder/               # 5 encoders (onehot, ordinal, label, target, frequency)
├── imputer/               # SimpleImputer, KnnImputer
├── selection/             # VarianceThreshold, SelectKBest
├── decomposition/         # PCA, TruncatedSVD, Jacobi, randomized_svd
├── linear_model/          # LinearRegression, Ridge, Lasso, LogisticRegression
├── tree/                  # CART kernel: estimators, splitter, node arena
├── ensemble.rs            # RandomForestClassifier / RandomForestRegressor
├── cluster/               # KMeans
├── metrics/
│   ├── regression.rs      # MSE, MAE, R², max_error, explained_variance
│   └── classification.rs  # accuracy, precision, recall, F1, confusion_matrix, log_loss
├── model_selection/
│   ├── split.rs           # train_test_split
│   ├── kfold.rs           # KFold, StratifiedKFold
│   └── cross_val.rs       # cross_val_score (rng re-exported from src/rng.rs)
└── compose/
    ├── column_transformer.rs
    └── output.rs

Trait hierarchy

Solver infrastructure

Four distinct solver families, all pure-Rust:

1. Cholesky (linalg::cholesky) — symmetric positive-definite system solver. Used by LinearRegression, Ridge, and LogisticRegression (per IRLS iteration).
2. Coordinate descent (Lasso) — soft-thresholding iteration for L1-regularized problems.
3. IRLS (LogisticRegression) — Newton-Raphson on the logistic loss, solving a weighted least-squares system each iteration.
4. CART splitter + forest seeding (tree/splitter.rs, ensemble.rs) — an exact transliteration of sklearn’s BestSplitter/DensePartitioner criterion accumulation, plus a local MT19937 “legacy RandomState” clone that seeds bootstrap samples identically to sklearn’s forests.

The first three are backed by the shared Matrix::matmul for forming Gram matrices, which dispatches to a tuned GEMM under the matrixmultiply feature; the tree kernel is allocation-light comparison sort + prefix accumulation and needs no BLAS at all.

Error handling

Hand-rolled DatarustError enum (no anyhow/thiserror, consistent with the zero-dependency ethos). Every fallible public API returns Result<T, DatarustError>. The variant set is ML-domain-specific (NotFitted, UnknownCategory, Singular, etc.), more informative than a generic error blob.

DNS for AI Discovery (DNS-AID)

datarust.dev supports agent discovery via DNS for AI Discovery (DNS-AID) per draft-mozleywilliams-dnsop-dnsaid and RFC 9460.

Entrypoint Records

Agents can query the _agents namespace via DNS-over-HTTPS (DoH) or standard DNS resolvers to discover available agent endpoints:

• Agent Index Discovery: _index._agents.datarust.dev (HTTPS record)
• Agent-to-Agent (A2A): _a2a._agents.datarust.dev (SVCB record)
• Root Agent Endpoint: _agents.datarust.dev (HTTPS record)

DNSSEC Authentication

The datarust.dev zone is signed with DNSSEC so validating resolvers receive authenticated cryptographic proof of DNS-AID records.

datarust MCP Server (Model Context Protocol)

datarust.dev provides an official datarust Model Context Protocol (MCP) server enabling AI agents, LLMs, and coding assistants (Cursor, Claude Desktop, Antigravity, Cline) to directly invoke datarust machine learning, preprocessing, and dataset profiling tools.

MCP Server Endpoint & Manifest

• MCP Server Endpoint: https://datarust.dev/mcp
• In-Repository Implementation: Built in Rust in crates/datarust-mcp
• MCP Server Card (SEP-1649): https://datarust.dev/.well-known/mcp/server-card.json
• Agentic Resource Discovery (ARD): Indexed in https://datarust.dev/.well-known/ard.json under urn:air:datarust.dev:mcp:datarust
• Link Header: Every response advertises rel="ard" and rel="api-catalog" discovery headers.

Available Capabilities & Tools

The datarust MCP server exposes tools across classical machine learning and data profiling:

1. profile_dataset

Generates a comprehensive statistical profile of a dataset from CSV or raw tabular data.

• Inputs: CSV text, table rows, or file reference.
• Outputs: Column statistics (mean, std, median, min, max, missingness, cardinality), distribution shape, quality issues, target-leakage hints, and pairwise relationships (Pearson, Cramér’s V, point-biserial).

2. preprocess

Applies tabular preprocessing transformations with zero external dependencies.

• Encoders: OneHotEncoder, OrdinalEncoder, TargetEncoder, FrequencyEncoder.
• Scalers: StandardScaler, MinMaxScaler, RobustScaler, Normalizer.
• Imputers: SimpleImputer (mean, median, most_frequent, constant).

3. train_model

Trains classical estimators on feature matrix X and target vector y with exact scikit-learn parity.

• Models: LinearRegression, Ridge, Lasso, LogisticRegression, DecisionTreeClassifier, DecisionTreeRegressor, RandomForestClassifier, RandomForestRegressor, KMeans.
• Outputs: Model coefficients, tree structure, training metrics, and serialization payloads.

4. evaluate_metrics

Computes regression metrics (MSE, RMSE, MAE, R², max error) or classification metrics (accuracy, F1, precision, recall, confusion matrix, ROC-AUC, log loss).

Client Configuration

Registry Listings & Direct Installation

The datarust MCP server is listed in major MCP directories:

• Smithery Registry: https://smithery.ai/server/@genc-murat/datarust

  npx -y @smithery/cli install @genc-murat/datarust --client claude
  

• Agentic Resource Discovery (ARD): https://datarust.dev/.well-known/ard.json

Claude Desktop / Cursor

Add the datarust MCP server to your configuration (e.g. claude_desktop_config.json or Cursor settings):

{
  "mcpServers": {
    "datarust": {
      "url": "https://datarust.dev/mcp"
    }
  }
}

In-Browser WebMCP

On the datarust.dev website, client-side WebMCP is registered via navigator.modelContext so browser agents can invoke datarust estimators directly in-page.

Roadmap

The canonical, detailed roadmap lives in ROADMAP.md inside the datarust crate. This page gives a higher-level tour of why each phase is ordered the way it is, and what it unlocks for users.

The destination

datarust aims to be the scikit-learn of Rust for classical, CPU-bound machine learning: the library you reach for when you need preprocessing, trees, clustering, SVM, cross-validation, and text features — without pulling in BLAS, a Python runtime, or a GPU stack.

The principles that shape every decision:

• Zero dependencies by default. Every algorithm ships as pure Rust.
• CPU-first. GPU and deep learning are served by candle and burn.
• scikit-learn API parity where Rust allows it; type-safe improvements where Rust enables them.
• No panics — public APIs return Result; missing_docs is enforced.

Where we are (v0.7.0)

The preprocessing, linear-model, clustering, and tree-based foundations are solid:

• 18 transformers/encoders/imputers/selectors — the deepest preprocessing coverage of any Rust ML library.
• 4 linear models (Linear / Ridge / Lasso / Logistic regression) with binary IRLS and multiclass softmax support.
• Decision trees and random forests (v0.7) — a CART kernel and both RandomForest estimators that reproduce scikit-learn’s fits bit-for-bit under the same seed.
• KMeans clustering (Lloyd’s algorithm, k-means++ initialization) + silhouette score.
• 15 metrics: regression (MSE, MAE, R², …), classification (accuracy, precision, recall, F1, ROC-AUC, PR-AUC, kappa, Matthews, …), clustering (silhouette).
• Pipeline, ColumnTransformer, cross-validation, Params trait, and a public seedable Rng.
• Zero external dependencies by default; serde / rayon / matrixmultiply are opt-in.

What is conspicuously absent: hyperparameter search, text features, and SVM. The roadmap below addresses each, in order of impact.

The release track

v0.7 — Tree-based learning (Shipped)

Trees and ensembles are the backbone of tabular ML, and v0.7 delivered the core of that stack with an unusual guarantee: bit-exact scikit-learn parity. The CART splitter is a transliteration of sklearn 1.6.1’s BestSplitter + DensePartitioner, and the forests seed their bootstrap samples through the same MT19937 “legacy RandomState” stream sklearn uses — the same integer random_state yields the same trees, node for node, verified bit-for-bit by the parity test suite.

Shipped: src/tree/ (DecisionTreeClassifier / DecisionTreeRegressor, TreeStructure introspection, feature_importances), src/ensemble.rs (RandomForestClassifier / RandomForestRegressor with on-demand estimators_samples()), and the public Rng promoted from model_selection. ExtraTrees and a generic Bagging meta-estimator remain planned for a later phase, as do out-of-bag scoring and parallel forest fitting.

v0.8 — Model selection & text

Why this comes second. Once the algorithm catalog is broader, the next bottleneck is tuning and NLP. GridSearchCV builds directly on the Params trait (get_params/set_params) shipped in v0.6. Text vectorizers depend on sparse matrix arithmetic, which today does not exist — so this phase widens SparseMatrix from read-only storage into a real linear-algebra type.

Highlights: GridSearchCV/RandomizedSearchCV, validation_curve/ learning_curve, CountVectorizer/TfidfVectorizer, KNeighbors, and Naive Bayes — the last three together forming a complete document-classification stack.

v0.9 — Depth & breadth

Why this is late. These are high-value but lower-leverage than the foundational phases. Many users will be productive with v0.7–0.8 alone.

Highlights: GradientBoosting/AdaBoost/Voting/Stacking, ExtraTrees and the Bagging meta-estimator carried over from the tree phase, LinearSVC/SVC (SMO solver, pure Rust), DBSCAN/AgglomerativeClustering, ElasticNet/SGDClassifier, NMF/FactorAnalysis.

v1.0 — Stability

Why v1.0 is mostly cleanup. By this point the API surface has been exercised across many estimators and is ready to be frozen. v1.0 removes legacy #[doc(hidden)] APIs, refreshes ARCHITECTURE.md, runs a full public-API audit, and publishes the SemVer stability commitment.

What is deliberately out of scope

The roadmap is as much about what datarust will not become:

• GPU compute and deep learning — served by candle and burn.
• Distributed training — a networking project, not a classical-ML one.
• pickle/joblib compatibility — technically impossible across languages.
• SHAP/LIME — permutation importance (planned) covers the common need.

A few items are under consideration for post-1.0 but not committed: f32 generics, manifold learning (TSNE), HistGradientBoosting, ONNX export/import, NumPy .npy interop, and a minimal MLPClassifier. See ROADMAP.md for the full list and rationale.

Contributing

The checkboxes in ROADMAP.md double as a contribution guide. Pick an unchecked item, open an issue to align on API shape, and land it with doc comments and sklearn-parity tests. Progress is recorded in CHANGELOG.md under each release.

Changelog

All notable changes to datarust are documented in the project’s CHANGELOG.md on GitHub.

This page provides a summary. For the full, detailed changelog (including internal refactors and performance tables), see the canonical source.

0.7.0

The tree-based learning release.

• Tree module (src/tree/) — DecisionTreeClassifier (Gini / entropy) and DecisionTreeRegressor (squared error). The CART splitter is a line-by-line transliteration of scikit-learn 1.6.1’s BestSplitter + DensePartitioner: with the same data, parameters, and integer random_state, the fitted tree matches sklearn node-for-node, bit-for-bit — split features, thresholds, impurities, node values, predictions, probabilities, and feature_importances_ (pinned by tests/tree_parity.rs against seeded sklearn fixtures). Includes the with_* builder options, TreeStructure introspection (per-node feature/threshold/impurity/improvement/sample count/value, classes(), leaf_for routing), sklearn-spelling Criterion/MaxFeatures parsing, and fit-time rejection of the regression-only criterion on classifiers.
• Random forest ensembles (src/ensemble.rs) — RandomForestClassifier / RandomForestRegressor with bootstrap aggregation seeded through a local MT19937 “legacy RandomState” clone, matching sklearn’s forests bit-for-bit under the same seed. Classifiers average per-tree probabilities (lowest-class tie breaks), regressors average predictions; estimators_samples() regenerates each tree’s bootstrap indices from stored per-tree seeds on demand. Out-of-bag scoring, sample weights, and parallel fitting are explicitly out of scope for this first release.
• Public RNG (src/rng.rs) — the internal xorshift64* PRNG promoted from model_selection to a top-level module (next_u64, next_unit, next_range, next_usize/next_index, next_normal, Fisher-Yates shuffle); model_selection::rng remains as a re-export shim.
• datasets::diabetes::load_normalized() — the sklearn load_diabetes(scaled=True) layout (mean-centred, population-std scaled) computed from the embedded raw data.
• Twelve DIABETES_DATA literals were re-precisioned to sklearn’s exact f64 values; load() output can differ from previous releases at the ULP level.

0.6.6

• PowerTransformer: the lambda-search likelihood now uses a precomputed jacobian term and a single fused Welford mean+variance pass — fit_transform 1 000 × 20 drops from 36.4 ms to 20.3 ms (1.8×).
• QuantileTransformer: column-major transform with a single trailing transpose — 10 000 × 100 goes from 27.7 ms to 23.1 ms.
• Quantile statistics: introselect replaces the full sort in column_quantiles_many — 100 000 × 100 goes from 144.7 ms to 86.0 ms (1.7×).
• KnnImputer: precomputed observed features, borrow-based neighbor search (no per-row reference-matrix copy), and partial neighbor selection — fit_transform 1 000 × 20 drops from 72.9 ms to 18.1 ms (4.0×).
• KMeans: nearest-centroid assignment via precomputed squared norms (FMA dot product) — fit_predict 5 000 × 20 × 10 is ~7% faster.
• LogisticRegression: the IRLS weighted Gram matrix and multinomial Hessian are accumulated in place (rank-1 sums, lower triangle only) instead of via a materialised weighted design matrix and general matmul — fit 50 000 × 100 drops from 3.86 s to 1.56 s (2.5×).
• stats::mode: sort-then-scan replaces hash-map counting (one heap entry per distinct value) — SimpleImputer MostFrequent 50 000 × 20 drops from 32.9 ms to 17.0 ms (1.9×).
• Wide-table Pearson: the new stats::correlation_matrix_flat streams the mean/center/covariance passes over contiguous memory, and the scalar covariance accumulates only the lower triangle — correlation_matrix_flat 10 000 × 100 drops from 38.7 ms to 6.7 ms (5.8×); datarust-profile’s wide profile_matrix 10 000 × 100 drops from ~63 ms to ~31 ms.
• Performance docs: the Performance page and README now include a “Scalar vs matrixmultiply kernels” table with stable criterion medians (--warm-up-time 0.5 --measurement-time 1.5 --sample-size 10) isolating the GEMM feature’s effect on matmul-heavy kernels.
• datarust-profile 0.3.1: hash-based row deduplication, borrow-based categorical access (T: AsRef<str>), allocation-free missing detection, and integer-coded Cramér’s V — profile_str_matrix 100 000 × 20 drops from 269 ms to 132 ms.
• Encoder fit paths: the four categorical encoders now gather columns through the borrowing StrMatrix::column_refs instead of cloning every cell — fit_transform 10 000 × 20 drops from ~20.7 ms to 3.60 ms (ordinal, 5.7×).
• QuantileTransformer transform: streams the flat buffer row-major (no nested-Vec transpose round-trip) — 100 000 × 20 drops from 28.3 ms to 22.9 ms; the reference lookup is then pre-partitioned into 512 uniform value spans so each value skips the full binary search — 23.4 ms to 14.6 ms (1.6×), bit-identical output.
• OneHotEncoder sparse transform: builds the CSR arrays directly instead of per-row triplets plus a re-sort — transform_sparse 50 000 × 20 drops from 27.6 ms to 18.3 ms; fit_transform_sparse from 78.3 ms to 33.2 ms.
• StratifiedKFold: each fold’s training complement uses a reusable boolean mask instead of a per-fold HashSet scan — split 100 000 drops from 6.31 ms to 1.82 ms (3.5×).
• Fresh sklearn comparison: the “datarust vs scikit-learn” table was re-run end-to-end (scikit-learn 1.6.1 / numpy 2.0.2 / scipy 1.13.1) — OneHotEncoder 50 000 × 20 default 89 → 54.7 ms, PCA 50 000 × 200 838 → 206 ms, and the LinearRegression column is now filled in (263 ms default / 91 ms with matrixmultiply vs sklearn 118 ms).

For the full details and performance tables, see the canonical changelog.

0.6.5

• Restored enforceable Rust 1.70 compatibility by pinning MSRV-compatible Rayon and serde_json dependency lines.
• Hardened deserialized categorical encoder state against mismatched category lists, index maps, and output widths.
• Rejected fractional ordinal codes during inverse transformation.
• Completed fitted-state checks for KNN/Simple imputers, MaxAbsScaler, and QuantileTransformer.
• Aligned PCA solver and component-count documentation with the implemented behavior.

0.6.4

• Silhouette scoring compacts arbitrary cluster IDs, handles singleton clusters correctly, and rejects invalid one-cluster-per-sample input.
• Numerical estimators, decompositions, scalers, selectors, and regression metrics reject non-finite observations; imputers retain NaN missing-value support while rejecting infinity.
• Non-finite scaler, selector, encoder, and SVD configuration values fail before fitting.
• Inconsistent fitted state loaded from JSON returns a recoverable error instead of panicking during transform or prediction.
• CI now builds and validates the docs/blog site and denies rustdoc warnings explicitly.

0.6.3

• Ridge, Lasso, logistic regression, and KMeans now reject non-finite or otherwise invalid solver hyperparameters before optimization.
• Params::set_params validation no longer mutates KMeans or logistic regression when a candidate is invalid.
• CI now executes the intended OS × feature matrix, including datasets and an all-features build, and verifies the crates.io package.
• The published crate excludes the blog, book, website, npm, and Cloudflare deployment sources, reducing the compressed archive from about 3.0 MiB to 289 KiB.

0.6.2

• Safe dense access now checks bounds in release builds, while dense allocations reject dimension overflow.
• Raw and deserialized CSR matrices validate pointer ranges and sorted column indices; duplicate triplets are summed.
• Train/test and cross-validation reject undersized or mismatched inputs, and fractional test sizes now round up consistently.
• StratifiedKFold supports validated, non-contiguous multiclass labels.
• Binary log loss, ROC-AUC, and average precision validate labels and numeric inputs. New *_with_positive_label functions support label spaces such as {2, 5}, and average-precision ties are threshold-grouped.

0.6.1

• Classification labels are now compacted safely, so metrics and LogisticRegression support non-contiguous labels such as {2, 5, 9} while preserving original predictions and probability-column mappings.
• Precision, recall, and F1 now expose explicit binary, macro, weighted, and micro averaging, configurable zero-division handling, and per-class reports.
• The optional datasets feature adds embedded Iris, Breast Cancer, Wine, and Diabetes datasets with no runtime file or network dependency.

0.5.0

• Estimator hierarchy — Estimator, Predictor, Classifier, and PredictProba now define common supervised flows. Regressor is reserved for regression semantics.
• SupervisedPipeline — Pipeline::with_estimator combines preprocessing, target-aware feature selection, and a final predictor in a cloneable, serde-serializable pipeline.
• BREAKING: LogisticRegression classification semantics — predict returns hard labels; predict_proba returns a two-column probability matrix; predict_positive_proba exposes the positive-class vector.
• BREAKING: Predictor imports — import Predictor to call shared fit/predict methods on supervised models.
• BREAKING: Custom trait implementors — custom transformers and encoders must implement Estimator; custom regressors must implement Estimator and Predictor before Regressor.

0.4.0

Added

• model_selection module — train_test_split, KFold, StratifiedKFold, cross_val_score. Shared deterministic xorshift64 PRNG, now used by both model_selection and decomposition::randomized_svd.
• LogisticRegression — binary classification via IRLS (Cholesky/SVD). The crate’s first classifier.
• metrics::classification — accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, log_loss.
• Ridge — L2-regularized regression (Cholesky/SVD). Succeeds on collinear inputs.
• Lasso — L1-regularized regression via coordinate descent. Produces sparse models (feature selection).
• Regressor trait — supervised counterpart of Transformer (fit(X, y) + predict).
• linalg::cholesky module — shared SPD solver foundation.
• LinearRegression — OLS regression, the crate’s first estimator.
• metrics::regression — MSE/RMSE, MAE, R², max_error, explained_variance.
• PCASolver enum (Auto / Full / Randomized) on PCA.
• jacobi::eigh_topk_flat — power-iteration + deflation for top-k eigenpairs.
• Flat-storage Jacobi eigensolver, flat matmul helpers, flat covariance.

Performance

• PCA 50 000 × 200 dropped from ~320 ms to ~104 ms with matrixmultiply.
• LinearRegression fit at 50 000 × 200: 258 ms → 84 ms with matrixmultiply.
• Scaler transform rayon threshold: scalar loop below 4 096 rows, parallel above.

0.3.0

• matrixmultiply feature: optional tuned pure-Rust GEMM (no system BLAS).
• BREAKING: Matrix internal storage switched from Vec<Vec<f64>> to a single contiguous Vec<f64>. ~13× on RobustScaler, ~5× on StandardScaler at 50 000 × 200.
• Flat-storage scalers, fused Welford statistics, NaN validation fused into transform loops.

Earlier versions

See the full changelog on GitHub for 0.1.x and 0.2.x history.

datarust API Documentation & Reference

The complete API reference for datarust and datarust-profile is hosted on docs.rs/datarust and docs.rs/datarust-profile — the official standard home for Rust crate documentation.

→ Open the full datarust API reference on docs.rs → Open the datarust-profile API reference on docs.rs

What’s in the API Documentation

docs.rs builds the rustdoc documentation with all features enabled (serde, rayon, matrixmultiply, datasets), so you see the complete API surface including:

• Every pub struct, pub enum, pub trait, and pub fn with doc comments.
• Type signatures, trait implementations, and method docs.
• Inline code examples from doc comments (runnable via cargo test --doc).
• Cross-links between types and modules.

Direct links to key modules

Direct links to key traits

Building locally

You can generate the same docs offline:

cargo doc --all-features --no-deps --open

The --open flag opens the generated HTML in your browser.