Anish Raj
@im-anishraj
Repository owner and maintainer. View GitHub profile
for contributions and activity.
About Arnio
A production-ready preparation layer for messy data that keeps pandas, Arrow, DuckDB, and scikit-learn workflows safe and explicit.
What is Arnio?
Arnio is a fast, safety-first library for preparing raw tabular data before it reaches analysis or machine learning. It handles brittle CSV and JSONL ingestion, applies reusable cleaning primitives, profiles data quality, and validates schema contracts so downstream tools can trust the input.
Hardened ingestion
Fast C++ CSV reading with explicit encoding policy, delimiter sniffing, chunked reads, and controlled bad-line handling.
Reusable cleaning
Composable pipeline steps make it easy to standardize missing values, strip whitespace, drop unwanted columns, and keep transformations predictable.
Contract validation
Schemas and quality reports catch malformed rows, invalid types, and bad values before data is handed to pandas or production workflows.
Design philosophy
Arnio is built around explicit contracts, safe I/O, and reusable primitives. It was designed to prepare data safely at the boundary, not to replace the analysis tools that come after it.
Built for
- fast C++-backed ingestion
- reusable cleaning primitives
- safe schema validation
- operational quality reporting
Designed to
- make data preparation explicit
- avoid silent pandas failures
- surface data issues early
- deliver clean data to the ecosystem
How Arnio fits in the ecosystem
Arnio is a preparation layer for the Python data stack: it prepares data for pandas, Arrow, DuckDB, and machine learning tooling without pretending to replace them.
Pandas
Use Arnio to prepare and validate data before analysis. Keep pandas for exploration, modeling, and final transformations.
Pandera
Arnio adds ingestion, data quality reports, and a contract-first preparation layer. Pandera remains a strong choice for schema validation inside pandas workflows.
Great Expectations
Great Expectations is a validation and observability platform. Arnio focuses on the ingestion and cleaning boundary, where raw files become trustworthy inputs.
Maintainers & Community
Arnio is maintained in public on GitHub, with contributor support through Discord and a transparent issue-driven process.
Open contributors
open-source contributors
All community contributions are tracked and visible in the
GitHub contributor graph.
Community support
Join Discord for onboarding, quick questions, and collaboration. Use GitHub issues for reproducible bugs, feature requests, and tracked work.
Project history
Arnio was created to address data preparation challenges. The project focuses on making ingestion, cleaning, and validation explicit and trustworthy.
Founding goal
Prepare messy data before it reaches pandas, reducing silent coercion, bad-line failures, and hidden errors.
Early milestones
Added schema validation, max_errors, URL scheme checks, and
quality profiling to catch bad data before it becomes a
downstream problem.
Current focus
Deliver safe interoperability, reproducible benchmarks, and stable primitives for the Python data stack.
How to get involved
New contributors can start with the guide, ask help in Discord, and open GitHub issues for work that should be tracked and reviewed.