fredgar-ai

Static site — the public mirror on GitHub Pages

Fredgar AI ships two front ends over the same warehouse:

  Interactive app (frontend/) Static mirror (this doc)
Audience Account-based users (token/session auth; reads public, writes/sync staff-only) Anyone — no account, no backend
Stack Vite/React SPA → Django/DRF API Plain HTML/CSS/JS generated at build time
Freshness Live (on-demand SEC/FRED syncs) Rebuilt on a schedule (weekly + manual)
Hosting Docker / nginx (see docker-compose.yml) GitHub Pages

Both are rendered from the same services (build_company_context reuses the profile/statements/analytics code), so the mirror shows exactly what the API does. The proposed extract store sits behind both as a shared cache: Pages still builds offline from the Release warehouse; only the interactive backend (and refresh-data.yml) may pull new EDGAR/FRED bytes.

What gets published

One page per company (infobox, headline snapshot, financial statements with per-line SEC source links, derived metrics, leadership, stakeholder index, filings, documents, XBRL fact preview) plus:

The landing page’s live demo

generate_site() collects a small “featured companies” list while it’s already looping over every company to render their pages — zero additional DB queries. Featured companies are preferred by whether they have any of the three demo-shown concepts (Revenue, Net Income, Total Assets) and then by richest data (counts.facts), capped at FEATURED_LIMIT (8). Their identity, headline figures, and fact/filing counts are rendered as ordinary server-side HTML (.example-card elements) — visible and fully readable with JavaScript disabled. A small vanilla-JS enhancement (the block script template block in index.html) adds a .js class to <html>, which is what actually collapses the cards to one-at-a-time with pill-switching and enables the search-filter box; without JS, every example simply renders stacked. Company data with no matching headline concept still appears in the demo with a graceful fallback message instead of blank figures.

The pipeline

publish_static_site → warehouse/services/static_site_publish.py:

  1. For each ticker: resolve/create the Company, sync submissions + XBRL facts from SEC EDGAR (DB-first payload cache, paced with --delay), compute derived metrics, and best-effort sync leadership (Forms 3/4/5). A failing ticker is reported and skipped — never aborts the publish.
  2. If a FRED_API_KEY is present (and --skip-macro isn’t passed), refresh the FRED series bundles (refresh_series_bundles) so the macro section publishes with fresh data. No key → the sync is skipped and the macro section renders only from already-warehoused data (or is omitted). A failing FRED sync never blocks the company publish.
  3. Render the site with generate_site (warehouse/services/static_site.py); macro context comes from public_data/services/static_export.py.
cd src
# Full build (SEC network) — default curated cohort:
python manage.py publish_static_site --output ../site

# Custom cohort, with sitemap/robots for a known public URL:
python manage.py publish_static_site --tickers AAPL,MSFT,NVDA \
  --base-url https://bamr87.github.io/fredgar-ai --output ../site

# Offline re-render from already-warehoused data (no SEC calls):
python manage.py publish_static_site --skip-sync --output ../site

# Render EVERYTHING in the warehouse, offline (the dataset defines the site):
python manage.py publish_static_site --all-warehouse --output ../site

# Sync the warehouse without rendering (the dataset-refresh path):
python manage.py publish_static_site --sync-only --force-refresh

generate_static_site still exists for rendering arbitrary warehouse companies (--ticker/--cik/--all) without any syncing.

The serverless dataset (no re-downloading from EDGAR/FRED)

The extracted data lives in one compressed SQLite file — fredgar-warehouse.sqlite3.zst, attached to the rolling dataset GitHub Release. That file is the warehouse: companies, filings, XBRL facts, derived metrics, leadership, the raw SEC JSON payload cache (EdgarSecPayload), and FRED series observations.

Two workflows with strictly separated responsibilities:

The published site itself remains the presentation-layer dataset (per-company company.json/facts.csv, site-wide companies.json, per-bundle observations.csv / bundle.json) — the release asset is the build input, not the display format.

GitHub Pages deployment

.github/workflows/pages.yml builds and deploys on:

One-time repository setup:

  1. Settings → Pages → Build and deployment → Source: “GitHub Actions”.
  2. Add a USER_AGENT_EMAIL repository variable or secret — SEC requires a contact email in the User-Agent; the refresh workflow (and the publish’s bootstrap fallback) fails fast without it.
  3. Run the “Refresh warehouse dataset” workflow once — the one-time load that creates the dataset release. All publishes are offline from then on.
  4. Optional: a STATIC_SITE_APP_URL repository variable with the public URL of the interactive app; every static page then shows a “Live app ↗” cross-link.
  5. Optional: a FRED_API_KEY repository secret (free key) so the refresh workflow also syncs the macro section. Without it the site simply omits the macro pages.

The published URL is https://<owner>.github.io/<repo>/ (the workflow derives it automatically for the sitemap). All intra-site links are relative, so the site works at any base path — project pages, a custom domain, or file:// on disk.

Cross-linking the two front ends

Settings

Env var Used by Purpose
STATIC_SITE_BASE_URL generate_site Canonical public URL → sitemap.xml + robots.txt
STATIC_SITE_APP_URL templates “Live app” cross-link on the mirror
STATIC_SITE_SOURCE_URL templates “Source” link (defaults to this repo)
USER_AGENT_EMAIL SEC client Required contact email for SEC fair access
FRED_API_KEY FRED provider Optional — enables the macro section’s FRED sync at build time

CLI flags (--base-url, --app-url) override the env vars per run.

This site is open source. Improve this page.