fredgar-ai

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

Fredgar AI (FRED + EDGAR, with AI) — a Django + DRF backend serving SEC EDGAR company data (filings, XBRL facts) and optional FRED macro series, with a Vite/React frontend in frontend/ and AI-assisted analysis layered on top. Python lives under src/; tests under tests/. Database is SQLite by default, PostgreSQL when DATABASE_URL is set (Docker/CI Postgres defaults to user/password/db fredgar; the CI ephemeral test database is test_fredgar).

Commands

All manage.py commands run from src/ — being in that directory plus manage.py is what puts config settings and the apps on the path. Lint/test run from the repo root.

# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
cp src/.env.example src/.env        # set USER_AGENT_EMAIL (SEC requires it)

# Backend
cd src && python manage.py migrate && python manage.py runserver   # http://127.0.0.1:8000/api/v1/

# Tests (from repo root)
pytest -q
pytest tests/api/test_filings_api.py                 # one file
pytest tests/api/test_filings_api.py::test_name      # one test
pytest -k edgar_db_first                             # by keyword

# Lint / format (must pass for CI; ruff line-length 100, rules E,F,I,W)
ruff check src tests
ruff format src tests

# Enrichment (from src/; see docs/enrichment.md)
python manage.py enrich_init --dataset X --csv ../data/list.csv --mapping manufacturing_v1
python manage.py enrich_plan --dataset X            # what WOULD run, before spending anything
python manage.py enrich_run  --dataset X --no-web   # free + DB-first providers; resumable
python manage.py enrich_resolve --dataset X         # claims -> golden record
python manage.py enrich_link    --dataset X         # verified CIK -> warehouse Company
python manage.py enrich_export  --dataset X --out ../data/enrichment/master.csv
python manage.py enrich_report  --dataset X
python manage.py enrich_evidence --dataset X --name "Some Company"

# Migrations after model changes
cd src && python manage.py makemigrations
cd src && python manage.py makemigrations --check --dry-run   # CI gate; CI fails on missing migrations

# Frontend (from frontend/)
npm ci && npm run dev      # Vite :5173, proxies /api -> :8000
npm run lint && npm run build

# Docker (from repo root)
docker compose up -d --build                    # Postgres + API on :8000
docker compose --profile dev up -d --build      # + Vite dev UI on :5173
docker compose --profile prod up -d --build     # + nginx static UI on :8080
docker compose --profile ci run --rm test       # full CI suite vs Postgres (ruff, migration check, pytest w/ coverage gate)

Test settings

pyproject.toml [tool.pytest.ini_options] is the single pytest config: it sets DJANGO_SETTINGS_MODULE = config.settings_test, pythonpath = ["src"], testpaths, and the markers (slow, integration, requires_network). A bare local pytest, CI, and Docker all run against config.settings_test. (There used to be a pytest.ini that overrode this with config.settings; it was removed so local == CI.)

config/settings_test imports settings.py and flips on DEBUG + the fast MD5 password hasher, restricts ALLOWED_HOSTS, and neutralizes DRF throttles so tests never hit 429s. The coverage gate (--cov-fail-under, currently 50) is applied in CI/Docker (docker/ci.sh, ci.yml) and via make test, not a bare pytest.

Architecture

Apps (under src/)

Services hold the logic (thin views/commands, fat services)

Business logic lives in */services/ modules. Both the DRF viewsets (api/v1/views.py) and the manage.py commands (*/management/commands/) are thin wrappers that call the same service functions — e.g. POST /companies/{id}/sync-submissions/ and manage.py sync_submissions both call sync_submissions_for_company. When changing ingest/sync behavior, edit the service, not the two callers.

Key subsystems

DB-first SEC reads (the core caching pattern)

Raw SEC JSON is cached in the EdgarSecPayload table keyed by (cik, kind). Read paths go DB → SEC API → save: see edgar_sec_payload.py (get_submissions_payload, get_company_facts_payload). They return the cached row unless force_refresh=True. This is deliberate — SEC enforces fair-access rate limits, so avoid live calls when a payload exists. To bypass the cache, thread force_refresh through (services), or pass ?force_refresh=true (API) / --force-refresh-style flags (commands). ListedIssuer similarly caches company_tickers.json so ticker/name search never hits SEC. Proposed evolution (not built): content-addressed extract store, incremental revalidate, Kaggle/local clean snapshots — docs/distributed-storage.md. Two frontends share that plane: GitHub Pages is offline (--skip-sync); the local/cloud app is the only runtime with API + AI.

EDGAR data source adapter

sec_edgar/adapters/ defines an EdgarDataSource Protocol (base.py). DirectEdgarAdapter (direct.py) is the real implementation wrapping SecEdgarClient.

SEC client and User-Agent

sec_edgar/client.py (SecEdgarClient) does all HTTP to data.sec.gov / www.sec.gov, with tenacity retries and explicit 429 → RuntimeError. SEC requires a contact email in User-Agent; it resolves from the USER_AGENT_EMAIL env var, and API requests may override it per-call via the X-Sec-User-Agent-Email header (api/sec_user_agent.py).

API versioning

/api/v1/ is canonical; /api/ is a legacy alias that includes the exact same urlconf (api/urls.py). Add routes in api/v1/ only. OpenAPI 3 schema at /api/v1/schema/ + Swagger UI at /api/v1/docs/ (drf-spectacular); docs/api-and-cli.md is the human-readable route inventory.

Data directory (data/)

Conventions

This site is open source. Improve this page.