API Reference

Read CSV-like input into an ArFrame. Supports extension-flexible paths, TSV delimiter inference, explicit dtypes, decoding policy, and configurable malformed-row handling. Key options: trim_headers strips whitespace from column names; null_values specifies additional tokens to treat as null; thousands_separator enables numeric parsing with grouping separators; mode controls parser strictness ("strict" or "lax").

Returns: ArFrame. Raises CsvReadError for read and parse failures.

Yield ArFrame chunks from large files. delimiter=None infers tab delimiters for .tsv paths and comma otherwise. on_bad_lines may error, warn, or skip malformed row-width records. Supports the same parser controls as read_csv(), including dtype, usecols, null_values, thousands_separator, trim_headers, and mode.

Infer column names and dtypes without loading the full file. Like read_csv, omitted delimiters use tab for .tsv paths and comma otherwise.

Example: ar.scan_csv("raw.csv", sample_size=500, null_values=["NA", "MISSING"]) keeps sampling and null-token handling aligned with a later read_csv() call.

Read JSON Lines or NDJSON into an ArFrame; mixed object columns are coerced to strings. The encoding and encoding_errors parameters control how file bytes are decoded into text.

Yield JSON Lines or NDJSON records as ArFrame chunks without buffering the whole file. Uses the same validation, line-numbered errors, decoding controls, and nrows limit as read_jsonl().

Write an ArFrame to CSV or TSV with delimiter and line terminator validation. Set escape_formulas=True to prefix string cells that start with spreadsheet formula trigger characters before CSV quoting. encoding defaults to "utf-8" (native fast path); any Python codec is accepted and transcoded in bounded chunks. encoding_errors controls unencodable character handling: "strict" (default), "replace", or "ignore".

Export an ArFrame to JSON without pandas conversion. Supported orient formats: "records", "list", and "split".

Export an ArFrame to JSON Lines or NDJSON. Emits one compact JSON object per row, preserves JSON null, and supports .jsonl and .ndjson output paths. encoding_errors controls unencodable character handling: "strict", "replace", or "ignore".

Read a Parquet file into an ArFrame via the optional pyarrow dependency. Install with pip install "arnio[parquet]". Use usecols or columns for column subset selection.

Write an ArFrame to Parquet via the optional pyarrow dependency. Install with pip install "arnio[parquet]". preserve_attrs=True writes JSON-serializable DataFrame.attrs into Parquet metadata; set to False to skip that export. Accepted compression values: "snappy" (default), "gzip", "brotli", "zstd", and "none".

Detect a likely delimiter for CSV-like input before reading or scanning. sample_size controls how many characters are read from the start of the file for detection; the default of 2048 is sufficient for most files. Increase it when the file has an unusual delimiter that only appears further into the content.

Python wrapper over the native C++ frame. Core properties include shape, columns, dtypes, is_empty, memory_usage(), head(), tail(), and preview().

Build an ArFrame from record dictionaries or row-like records. Added in v1.17.0.

Jupyter/IPython automatically calls this to render a bounded HTML table preview showing up to 10 rows, column names, dtypes, and shape summary.

Select one column as a Python list or multiple columns as an ArFrame.

Return True when the item is a valid column name in the frame.

Return a new ArFrame containing only the requested columns in original order.

Return a new ArFrame filtered by dtype names like int64, float64, string, bool, or null.

Cast one or more columns to a specified dtype or per-column type mapping and return a new ArFrame.

Return a dict[str, list] representation for serialization or tests.

Convenience wrapper to write the frame to a CSV file. Delegates to arnio.write_csv.

Current-main method equivalent to the top-level drop_columns helper.

Summary statistics and column summaries. schema_summary returns ColumnSummary objects with name, dtype, and nullability.

Convert to pandas, preserving the fast zero-copy path where supported. Use copy=True for defensive isolation.

Convert a pandas DataFrame to ArFrame with dtype validation, duplicate-column checks, attrs preservation, and zero-column row-count preservation.

Build an ArFrame from a mapping of column names to column values.

Export to pyarrow.Table. Install with pip install "arnio[arrow]". Added in v1.18.0.

Convert a PyArrow Table to an ArFrame. Install with pip install "arnio[arrow]".

Convert an ArFrame to a Polars DataFrame via Arrow. Requires optional Polars dependency; install with pip install "arnio[polars]".

Convert a Polars DataFrame to ArFrame with inferred types and null values preserved. Requires optional Polars dependency; install with pip install "arnio[polars]".

Replace values in string columns with their lowercase hex-encoded hash digest. Uses the standard-library hashlib module — no custom digest code. Null cells are preserved as null; empty strings are hashed normally. algorithm accepts "sha256" (default, recommended) or "md5" (for deduplication workloads only — not cryptographically strong). Raises ValueError for missing columns; raises TypeError for non-string columns. Available as a pipeline step: ("hash_columns", {"subset": ["email"], "algorithm": "sha256"}).

Warning: hashing is deterministic pseudonymization, not encryption. This does not constitute anonymization under GDPR or equivalent regulations.

Returned by cast_types(..., errors="report"). CastReport holds the cast frame and a failures list of CastFailure records. Each CastFailure has column, row, value, and target_dtype attributes. bool(report) is True when there is at least one failure.

Run named cleaning steps or registered Python callables. verbose=True enables lightweight diagnostics through the arnio logger and optional PipelineContext. Pass track_lineage=True to receive a (ArFrame, LineageReport) tuple that maps every dropped row back to its original index and the step that removed it.

Returned by pipeline(..., track_lineage=True). Attributes: dropped_by_step (dict mapping step name → sorted list of original row indices dropped by that step; steps that dropped no rows have an empty list) and total_dropped (int). Method: to_pandas() returns a flat DataFrame with columns original_index and step.

Manage the step registry and discover available built-in signatures.

Return DataQualityReport with row/column counts, duplicate metrics, column profiles, warnings, quality score, score components, and cleaning suggestions.

Jupyter/IPython automatically calls this to render the report dashboard HTML directly in the notebook output area.

Export reports for notebooks, CI, docs, and automation.

Compare quality over time, generate pipeline-compatible steps, or run automatic cleaning with optional reports and audit output.

Additional options:

• return_report – return the pre-cleaning quality report alongside cleaned output.
• allow_lossy_casts – opt in to strict-mode casts that may lose information.
• confirmed_casts – explicit column-to-dtype confirmations required before strict-mode casts are applied.

Return values:

• auto_clean(...) → ArFrame
• auto_clean(..., return_report=True) → (ArFrame, DataQualityReport)
• auto_clean(..., dry_run=True) → DataQualityReport
• auto_clean(..., explain=True) → (ArFrame, CleanExplanation)
• auto_clean(..., return_report=True, explain=True) → (ArFrame, DataQualityReport, CleanExplanation)

# Preview proposed changes
report = ar.auto_clean(frame, mode="strict", dry_run=True)

# Clean and return report
cleaned, report = ar.auto_clean(
    frame,
    return_report=True,
)

# Clean with explanation
cleaned, explanation = ar.auto_clean(
    frame,
    explain=True,
)

Compare two DataQualityReport objects and return a QualityGateResult suitable for CI and release checks.

baseline = ar.profile(ar.read_csv("baseline.csv"))
current = ar.profile(ar.read_csv("current.csv"))

result = ar.check_quality_gates(
    baseline,
    current,
    max_null_ratio_delta=0.02,
    allow_new_columns=True,
)

if not result.passed:
    print(result.to_markdown())
    result.raise_for_failures()

ColumnProfile, CleaningSuggestion, CleanStepRecord, CleanExplanation, DriftReport, ProfileComparison, QualityGateIssue, and QualityGateResult describe structured profiling, cleaning, comparison, and gate output.

Compare two ArFrame datasets and return a DriftReport describing structural and statistical changes. Unlike compare_profiles, this function handles added or removed columns without raising an error.

DriftReport fields:

• added_columns – columns present in new but absent from old.
• removed_columns – columns present in old but absent from new.
• dtype_changes – shared columns whose dtype changed: {col: (old_dtype, new_dtype)}.
• null_ratio_changes – shared columns whose null ratio changed: {col: (old_ratio, new_ratio)}.
• row_count – row counts as (old_rows, new_rows).
• has_drift – True when any of the above are non-empty or row counts differ.
• to_dict() – JSON-safe export of the report.

old_frame = ar.read_csv("data_v1.csv")
new_frame = ar.read_csv("data_v2.csv")
report = ar.detect_drift(old_frame, new_frame)

if report.has_drift:
    print("Added columns:", report.added_columns)
    print("Removed columns:", report.removed_columns)
    print("Dtype changes:", report.dtype_changes)
    print("Null ratio changes:", report.null_ratio_changes)

import json
print(json.dumps(report.to_dict(), indent=2))

Validate frames against field definitions. max_errors was added in v1.18.0.

unique: list or tuple of column names that must contain no duplicate values in the frame.

rules: list of callables (pd.DataFrame) -> list[ValidationIssue] for cross-field or custom contract checks.

schema = ar.Schema(
    fields={"email": ar.Email(), "user_id": ar.Int64()},
    unique=["email"],
)
result = schema.validate(frame)

Validate an iterable of ArFrame chunks against a Schema and return a single ValidationResult. Row indices in all ValidationIssue objects are adjusted to reflect global positions across chunk boundaries. Supports max_errors for early termination. Does not modify validate().

Each builder returns a Field for use in Schema(fields={...}). All builders accept keyword-only arguments unless noted otherwise.

Common parameters: nullable — allow null values; unique — require all non-null values to be distinct; severity — "error" (default) or "warning"; required_if — a (column, value) tuple that makes the field mandatory when another column equals a given value.

Compare schema contracts, export schema definitions, and register custom semantic validators. Structured results include SchemaDiff, SchemaDiffEntry, ValidationIssue, and ValidationResult.

Run Arnio workflows directly from pandas DataFrames.

Clean a DataFrame with Arnio and return a pandas.DataFrame.

When steps is provided, it is passed directly to ar.pipeline() and the keyword flags are ignored. When steps is omitted, the convenience-clean path runs with the keyword flags controlling which operations apply.

• strip_whitespace – strip leading and trailing whitespace from string columns (default True).
• drop_nulls – drop rows that contain any null value (default False).
• drop_duplicates – drop duplicate rows (default False).

import pandas as pd
import arnio as ar

df = pd.DataFrame({"name": [" Alice ", " Alice "], "score": [10, 10]})
cleaned = df.arnio.clean(drop_duplicates=True)
# cleaned: one row, name stripped to "Alice"

Returns: pandas.DataFrame

Register an ArFrame as a DuckDB relation through pandas interop.

Optional scikit-learn transformer for pipeline-safe data preparation. Install with pip install "arnio[sklearn]".

ArnioError is the package-level base exception. Public specialized exceptions include CsvReadError, JsonlReadError, RemoteReadError, TypeCastError, PipelineStepError, UnknownStepError, and SchemaValidationError.