Python Data

Your project

Live preview — CLAUDE.md

# Data Pipeline

TailorMD recipe: **Python Data**

## Stack

- **Python:** 3.12
- **Workflow:** ETL pipelines
- **Orchestrator:** none
- **Data store:** Parquet on S3

## Folder map

```
src/ or Data Pipeline/
  pipelines/       # ETL / transform steps
  models/          # ML models and training code
  utils/           # shared helpers
notebooks/         # exploratory analysis only — promote stable code to src/
tests/
  unit/
  data/            # schema and quality checks
data/
  raw/             # gitignored — never commit raw data
  processed/       # gitignored or DVC-tracked
configs/           # pipeline and environment configs
```

## Non-negotiables

- Pin dependencies (`requirements.txt`, `pyproject.toml`, or `uv.lock`)
- Never commit raw data, credentials, or large artifacts
- Reproducible runs: fixed random seeds for ML, logged parameters
- Data quality checks on every pipeline stage
- Notebooks are scratch space — production logic lives in `src/`
- Document data lineage: source → transform → output

## Naming and style

- Modules: `snake_case.py`
- Functions: `snake_case`
- Constants: `SCREAMING_SNAKE_CASE`
- Follow PEP 8; use type hints on public functions
- Config files: `kebab-case.yaml` or `snake_case.yaml`

## Commands

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"          # or uv sync
pytest                           # run tests
ruff check .                     # lint
mypy src/                        # type check (if configured)
python -m src.pipelines.run      # run pipeline locally
```

## Ship checklist

Before merging or deploying:

- [ ] Tests pass (`pytest`)
- [ ] Lint and type checks clean
- [ ] No data files or secrets in diff
- [ ] Pipeline runs end-to-end on sample data
- [ ] Data quality checks pass
- [ ] Dependencies pinned and documented
- [ ] README updated for new pipelines or configs
- [ ] Orchestrator DAG/task registered (if none ≠ none)

## Workflow (ETL pipelines)

- **ETL:** idempotent stages, partition-aware writes to Parquet on S3
- **ML training:** versioned datasets, logged hyperparameters, reproducible seeds
- **Analytics:** parameterized queries, no hardcoded date ranges in prod
- **Research notebooks:** clear outputs, export stable functions to `src/`

## Orchestrator (none)

- **none:** CLI entry points with `--dry-run` support
- **Airflow / Dagster / Prefect:** retries on transient failures, alerts on hard failures