Reproducible Python data and ML work: typed pipelines, pinned environments, data contracts, and testable transforms.
Data and ML projects fail quietly when environments drift, transforms go untested, and notebooks become the source of truth. TailorMD's Python Data recipe encodes reproducibility-first conventions — pinned environments, typed pipelines, data contracts, and testable transforms — into CLAUDE.md and Cursor rules your assistant reads before every change.
Configure your workflow (ETL, training, analytics, or research), orchestrator (Airflow, Dagster, Prefect, or scripts-only), data store, Python version, and packaging tool (uv, Poetry, conda, pip-tools). The generator produces a project brief with folder maps, non-negotiables for schema validation, deterministic runs, and ship checklists suited to production data work.
This is one of two free TailorMD recipes — no account required. Use the web form for a live preview and zip download, or run npx tailormd init python-data from the terminal. The output is plain Markdown you own: edit commands, bucket names, and team conventions freely. When your stack shifts from local Parquet to BigQuery or from scripts to Dagster, regenerate in about 60 seconds instead of rewriting context by hand.
Answer a few questions about your stack. Copy CLAUDE.md or download a zip — no API key required.
npx tailormd init python-data# Data Pipeline TailorMD recipe: **Python Data**. ETL pipelines in Python 3.12. The bar is reproducibility: the same inputs and code produce the same outputs on another machine, next quarter. ## Stack - **Python:** 3.12 (pinned) - **Workflow:** ETL pipelines - **Orchestrator:** none - **Data store:** Parquet on S3 - **Env / packaging:** uv with a committed lockfile - **Core libs:** pandas/polars, pyarrow, pydantic for schemas, pytest for tests ## Folder map ``` src/Data Pipeline/ # importable package — all reusable logic lives here io/ # readers/writers for Parquet on S3 transforms/ # pure, testable functions (input df -> output df) pipelines/ # composition of transforms into runnable jobs schemas/ # pydantic / pandera data contracts config/ # typed settings, no hardcoded paths or creds notebooks/ # exploration only; import from src, don't define logic here tests/ # unit tests for transforms + schema checks data/ # gitignored; raw/ interim/ processed/ ``` ## Non-negotiables Reproducibility - Pin Python 3.12 and commit the uv lockfile. No `pip install` that isn't recorded. - Set and record a random seed anywhere randomness affects results (splits, sampling, model init). - Treat raw data as immutable and read-only; every derived dataset is regenerated from code, never hand-edited. - Parameterize runs (dates, paths, config) — no hardcoded absolute paths, no notebook-only state that can't be re-run top to bottom. Data safety and quality - Never commit data, credentials, or `.env`. Load secrets from env/secret manager only. - Validate data at every boundary with a schema (pydantic/pandera): types, nullability, ranges, uniqueness. Fail loud on contract violation. - Log row counts and key stats at each stage so silent data loss is visible. - No PII in logs, samples, or committed fixtures; anonymize test data. ## Code style - Logic lives in `src/`, imported by notebooks and pipelines. Notebooks orchestrate; they don't define. - Transforms are pure functions with type hints: given a frame, return a frame; no hidden global state. - Prefer vectorized pandas/polars over row-wise loops; avoid chained inplace mutation. - Format with ruff/black; type-check with mypy or pyright on `src/`. ## Data contracts and lineage - Define an explicit schema per dataset (columns, dtypes, constraints); version it when it changes. - Record provenance for every output: source, code version (git sha), params, and run timestamp. - Partition Parquet on S3 outputs by a stable key (e.g. date) and write atomically (temp path -> rename) so readers never see partial data. ## Pipelines and orchestration (none) - Each step is idempotent and re-runnable: rerunning with the same params yields the same output. - Steps declare inputs and outputs explicitly; downstream steps depend on artifacts, not side effects. - Make backfills first-class (parameterized by date range); keep steps small enough to retry cheaply. - Isolate slow/expensive steps (network, large scans) behind caching where safe. ## Commands ```bash # uv — adjust to your toolchain uv sync # install from lockfile uv run pytest # run tests uv run ruff check . # lint uv run mypy src # type-check uv run python -m Data Pipeline.pipelines.main # run a pipeline ``` ## Testing strategy - Unit-test transforms on small, hand-built frames covering nulls, empties, duplicates, and boundary values. - Assert schemas hold on representative fixtures; treat a schema break as a failing test. - Test idempotency: running a step twice equals running it once. - Keep notebooks out of CI logic; if a notebook matters, extract its code into `src/` and test that. ## Ship checklist - [ ] Lockfile updated; environment reproduces from scratch - [ ] Lint, type-check, and tests pass - [ ] Schemas validate on sample data; row-count/stat logging in place - [ ] Seeds set; run is parameterized and re-runnable top to bottom - [ ] No data, secrets, or PII in the diff - [ ] Outputs written atomically and partitioned; provenance recorded