Baby Names Explorer

Baby Names Explorer is an interactive data application for exploring 145 years of US baby name trends from the Social Security Administration (1880–2024). You can browse the most popular names for any year, pull up the full history of any single name with a 5-year machine-learning popularity forecast with measured uncertainty bands and projected ranks, and ask questions about the entire dataset in plain English.

The project began as a single-file Streamlit app. It has since been rebuilt as a decoupled, production-grade system: a Next.js 16 frontend over a Python FastAPI backend, with reproducible data ingestion, an empirical ML forecasting pipeline, and automated CI/CD validation gates.

The SSA publishes baby name records as annual text files behind anti-bot protections. Ingestion is fully automated and reproducible via backend/scripts/build_db.py: it handles download automation, aggregates and cleans ~145 annual files, computes relative metrics (popularity_percent share within sex/year, popularity_rank), and builds a composite index (name, sex, year) for zero-overhead streaming.

Relative metrics matter more than raw counts: a mid-century name given to 30,000 babies was a national phenomenon; the same count today in a much larger and more diverse birth pool represents a far smaller share.

The database is treated as a published build artifact rather than being checked into git:

• Observed records, canonical indexes, and precomputed forecast tables are generated via automated batch scripts.
• The artifact is validated by an automated deployment gate (backend/scripts/verify_db.py) before publication.
• At runtime, the FastAPI backend resolves names.db locally or downloads it from Hugging Face Hub, opening all connections in read-only mode (mode=ro).

The core forecasting challenge is projecting a name's popularity trajectory over a 5-year horizon ($h \in {1..5}$).

Why Per-Name ARIMA Was Replaced

The original prototype explored per-name ARIMA modeling, running ADF and KPSS stationarity tests to determine differencing orders and selecting orders via AICc. However, rigorous rolling-origin backtesting across historical decades revealed a structural limitation: with only 30 to 100 annual observations per series, individual series lack the statistical power to reliably satisfy stationarity or identify true ARIMA orders. While ARIMA performed moderately in the top 100 names (+0.161 skill), it produced negative skill outside the top 1,000 (–0.211 on ranks 1001–5000, –0.413 beyond 5000) — meaning visitors were receiving forecasts demonstrably worse than a flat-line naive baseline.

The Pooled Gradient-Boosted Model

The solution was shifting to a pooled model: training across every eligible name's history simultaneously. A single LightGBM booster is trained per horizon ($h \in {1..5}$) over a rolling 40-origin training window:

• Target: Scale-free relative log growth $\log(y_ / y_t)$.
• Feature engineering: 13-feature base block including multi-horizon momentum (1/2/3/5/10-year growth), acceleration, historical volatility, popularity level, and distance/years from historical peaks and troughs.
• Sample weighting: $\text^$ weighting to prioritize names visitors search for without letting a few historical giants dominate the split criteria.

Post-Processing Stack

Direct booster outputs undergo a three-step sequential correction pipeline:

1. Growth capping: Path implied log-growth is clipped at the empirical 99.9th percentile of historical 5-year moves, preventing unbounded extrapolation on volatile low-volume names.
2. Path smoothing: An endpoint-preserving 3-point moving average over the log steps eliminates erratic year-over-year oscillations without altering the 5-year terminal projection.
3. Global reconciliation: Because popularity is a zero-sum share of total births within a sex, multiplicative scaling reconciles the slice total onto the origin year's observed sum.

Conformal Uncertainty Bands & Projected Ranks

• Conformal prediction bands: Rather than assuming theoretical Gaussian residuals, the 80% and 95% uncertainty intervals are derived from empirical residual quantiles stratified by popularity tier and volatility bin. A volatile breakout name receives an appropriately wider band than an entrenched classic at the same rank.
• Projected ranks: Each forecast point includes a projected rank computed against a frozen field of all recorded names at that origin, preventing rank distortions in the long tail.
• Deploy gate: verify_db.py enforces that every published artifact beats the naive baseline across a 26-window backtest span, scoring positive skill across every tier (+0.374 in the top 100).

The chat feature translates natural-language questions into SQL rather than relying on hallucinated model memory:

1. Query generation: Groq translates the user's question and recent history into a SQLite query against the names schema.
2. Resource-budgeted execution: Generated SQL is validated against strict keyword guards (allowing only SELECT or WITH CTEs) and executed on a dedicated read-only connection with a 5-second deadline and 1 MB memory cap, wrapped with a LIMIT 1000 boundary.
3. Response composition: A second LLM pass synthesizes the query results into a concise explanation, displaying the raw SQL alongside the answer for complete transparency.

This project demonstrates end-to-end machine learning and data systems engineering:

• Empirical ML rigor: Replacing an established model (ARIMA) when backtesting proved negative skill, designing a pooled gradient-boosted architecture that achieved positive skill across all popularity tiers.
• Conformal prediction: Quantifying uncertainty through empirical, stratified calibration rather than fragile distribution assumptions.
• Data engineering & CI/CD: Reproducible ETL, published database build artifacts, automated release gates, and sub-second full-stack query performance.