Skip to content
Heterodata An Arcanum Research project Vlad
Vlad

Code

Everything you see on this site — the Explore charts and every Data download — is produced by the small FastAPI + Jinja + Plotly app below, from the JSON/CSV inventory artifacts shipped under app/data/. Stock-only stack, no pandas.

Run it, get identical output. Every JSON API response and every download is computed deterministically from the shipped inventory files — no hidden state, no network call, no random seed. Install the pinned deps, run uvicorn app.main:app, and the charts and download bytes regenerate byte-for-byte.

Download repro bundle (.zip) Data bundle (.zip)

1 · The JSON API over the real corpus inventory

The charts fetch these endpoints. Each returns a real on-disk inventory structure (per-cluster document counts, year spans, extracted-table counts) — genuine corpus measurements, no transformation beyond sort/slice.

app/main.py
@app.get("/api/panels")
def api_panels() -> JSONResponse:
    """Top panel-capable series clusters — real per-cluster measurements."""
    return JSONResponse(INVENTORY["panels"])


@app.get("/api/richest")
def api_richest() -> JSONResponse:
    """Series clusters with the most extracted tables — real counts."""
    return JSONResponse(INVENTORY["richest_by_tables"])


@app.get("/api/totals")
def api_totals() -> JSONResponse:
    return JSONResponse({
        "project": HEADLINES["project"],
        "inventory": INVENTORY["totals"],
        "provenance": INVENTORY["_provenance"],
    })

2 · The charts (vendored Plotly + Arcanum Site Kit theme, offline)

The explorer draws the fetched clusters with the vendored Plotly build (no CDN), routed through the kit's ArkPlotly so every chart shares one transparent, deep-red-accent, dark/light-aware theme and a modebar with PNG + CSV export. Labels are de-underscored/de-truncated for legibility. The bars are document counts and extracted-table counts per real cluster — not a time series.

app/templates/explorer.html
fetch('/api/panels').then(r => r.json()).then(rows => {
  const top = rows.slice(0, 18).reverse();
  // ArkPlotly = the kit's shared Plotly theme: transparent bg, --ark-* accent +
  // colorway, dark/light-aware, trimmed modebar (PNG + "Download data (CSV)").
  ArkPlotly.plot('panels-chart', [{
    type: 'bar', orientation: 'h',
    x: top.map(r => r.n_docs),
    y: top.map(r => r.display_name || r.label),   // English label (display_name)
    // r.original_label carries the Russian source title (shown in the hover/table)
    hovertext: top.map(r => `${r.display_name}<br>${r.n_docs} docs (${r.span})` +
                            (r.original_label ? `<br><i>${r.original_label}</i>` : '')),
    hovertemplate: '%{hovertext}<extra></extra>'
  }], {
    title: 'Panel-capable clusters by document count (top 18)',
    margin: {l: 300, r: 20, t: 40, b: 40},
    height: 600,
    xaxis: {title: 'documents in cluster'}
  }, {filename: 'vlad_panel_capable_clusters'});

  // Inline CSV download of the same table (real panels_csv artifact, text/csv).
  ArkDownloads.buttons({ csv: '/api/download/panels_csv' });
});

3 · The downloads (correct content-types)

The Data page serves files verbatim or as faithful CSV serializations of an existing real JSON structure. Each format carries its true media type — the "CSV-served-as-JSON" failure is avoided by deriving the content-type from the artifact's declared format.

app/downloads.py
MEDIA = {
    "csv": "text/csv; charset=utf-8",
    "json": "application/json",
    "zip": "application/zip",
}

def _clusters_csv(key: str) -> bytes:
    """Serialize a real cluster list from inventory.json (panels / richest)."""
    inv = _load("inventory.json")
    buf = io.StringIO()
    w = csv.writer(buf)
    w.writerow(["series_cluster", "example_document", "n_docs", "n_years",
                "span", "extracted_tables", "body_chunks"])
    for r in inv[key]:
        w.writerow([r["series"], r["label"], r["n_docs"], r["n_years"],
                    r["span"], r["tables"], r["body"]])
    return buf.getvalue().encode("utf-8")

def render_artifact(art: dict) -> tuple[bytes, str, str]:
    raw = _bytes_for(art)
    return raw, MEDIA[art["fmt"]], _download_filename(art)

4 · How the inventory is built (and translated)

The 266-cluster inventory is measured off the validated HDARP knowledge base by the project clusterer, then summarized into the shipped inventory.json by build_data.py. It only counts and groups real extractions — it never invents a value. The same build step applies the curated English-translation map (translations.json): every displayed Russian source-document title becomes an English display_name, with the Russian original kept as original_label. Untranslated Russian and dropped garbled labels are reported on every build.

deploy/vlad/build_data.py
with open(INV, newline="", encoding="utf-8-sig") as fh:
    for r in csv.DictReader(fh):
        rows.append({
            "series": (r.get("series") or "").strip(),
            "label":  (r.get("sample") or "").strip() or (r.get("series") or ""),
            "n_docs": _int(r.get("n_docs")), "n_years": _int(r.get("years")),
            "span": (r.get("span") or "").strip(),
            "tables": _int(r.get("tables")), "body": _int(r.get("body")),
            "sample": (r.get("sample") or "").strip(),
        })
# panel-capable = a real cluster spanning >= 2 distinct years
panel = [x for x in rows if x["span"] and x["n_years"] >= 2]

# English-first display: translate the Russian source title, keep the original.
def _localize(row, tmap):
    s = row["sample"]
    if _is_cyrillic(s) and s in tmap:
        row["display_name"]   = tmap[s]   # English (primary)
        row["original_label"] = _clean(s) # Russian (secondary, shown in parens)
    else:
        row["display_name"]   = _clean(s) # already-English slug
        row["original_label"] = None

5 · Series-construction code Roadmap — not yet written

The harmonized economic series do not exist yet, so neither do their construction scripts. The plan is fixed: each panel-capable cluster will be built with the uniform Anu pipeline (a single source of truth per series, series_registry.json, with full provenance). These scripts land in the project repository as clusters are built, starting with the Cluster A fiscal pilot (1856–1990) — see Methodology and the construction roadmap.

Planned stage (Anu pipeline)Will produceStatus
anu-researchVariable + source map from each cluster's KB dirsscoped
anu-ingestionParse per-chunk table CSVs into a raw long table (year × category × value)scoped
anu-extension / extenbookStitch multi-part / multi-year editions into one seriesscoped
anu-variantRegime / currency / orthography variants (1897 + 1922/1947/1961 reforms)scoped
anu-review / anu-adequacyCoverage + quality gate; register in series_registry.jsonscoped

None of these scripts are shipped in the repro bundle because none are written yet — labeling them "available" would be fabrication. When the pilot lands, this section gains real, runnable construction code.

A validated knowledge base of Imperial Russian & Soviet official statistics, extracted via HDARP. Russian source-document titles are translated into English (original preserved). Figures reconciled 2026-06-07. All numbers trace to canonical project artifacts — nothing on this site is fabricated; harmonized time-series panels are scoped but not yet constructed (see the Roadmap). Data is reconstructed for research and education; defer to the original archival sources.