How the data got into SPS V4

01 Lifecycle at a glance

SPS V4 was never fed by a live pipeline at launch. It was bootstrapped once from Excel, and financials are now kept current by a gold-layer sync. Three phases:

The Excel import ran through dev and staging repeatedly during build, and a real production replace was executed on 2026-06-22 (~140 projects, identity preserved).

02 The sources

Two Excel inputs feed the bootstrap, plus one gold source for the ongoing sync.

Central workbook — DUOMENYS_PVT.xlsx

One workbook, eight sheets. The source of truth for projects, people, and the financial / status baseline.

Weekly submissions — SPS FAILAI 3/

A folder of per-week, per-project macro workbooks. Filename encodes everything the walker needs; three form versions are supported.

2026-05-12_08_30 … forma V3.1 … _2114.xlsm
                   └ version         └ project code
versions: V3 · V3.1 (primary/additional hours split) · V3.2 (adds daily inventory)
sheets read: SPSPF (D3 = project code) · IVESTIS (hours) · SDU (work volume m²) · KRITINES (critical activities)

Gold layer — Databricks sps_table2

Per-Objektas monthly aggregate view in the Databricks Hive gold schema, read via the 1.2 Command Execution API on cluster adf, or from ADLS as an abfss:// export. Feeds the ongoing financials/inventory sync (§06).

03 The import machine

scripts/import-excel.ts orchestrates everything in two batches. Phase A reads the central workbook; Phase B reads the submission folder; then three materialization passes project the submissions into the normalized read tables.

A3 (materials catalog rebuild) was removed — materials_catalog is now app-owned and migration-seeded, so the import no longer touches it.

Idempotency — read this before assuming re-runnable

• Phase A is idempotent. Checksum-gated via import_manifest; per-project wipe-and-rewrite for financials/status. Safe to re-run.
• Phase B is not. weekly_submissions are immutable (ADR-021), versions restart at 1 per week; a second insert into an existing bucket violates the unique constraint and rolls the whole import back. Re-importing submissions means reset-then-rebuild, not incremental.

Entry points

04 Sheet → schema mapping

The centerpiece: what each Excel column becomes in Postgres, and what transform is applied on the way. This is the spec a replacement pipeline must reproduce. Exact cell references live in the phase parsers under scripts/import/excel/; the authoritative column-by-column list is docs/spec-gates/field-mapping.csv.

Projektai → projects · users · project_assignments

Projektu duomenys → projects

Pajamos_sanaudos · Papildomos_pajamos · Valandos_men → project_financials

All three aggregate to one row per (project, month-end). key: project_id, month

Projektu_stebesena · SPS_formai → project_status_weekly

Weekly rows keyed by ISO week. key: project_id, year, week_number

Submission .xlsm → weekly_submissions.snapshot_data (v1 JSONB) → read models

Each submission parses into an immutable v1 JSON snapshot, then materializes into the normalized tables. Shape contract: docs/spec-gates/snapshot-v1-schema.md.

05 Transforms that bite

The non-obvious conversions. Any pipeline that replaces the Excel import has to get these right or the numbers drift.

• Margin is a percent, not a cost. The workbook stores contract profitability as a margin ratio. Cost is derived: contract_costs = revenue × (1 − margin). Feed the raw margin as if it were EUR and every contract cost is wrong.
• Papildomos double-count. The earlier gold table (sps_table1) summed additional costs into the primary figure. sps_table2 fixed it — costs_primary = Sanaudos − Papildomos. If you re-derive financials from an older source, check which convention it follows.
• Financial month is month-end. Every financial row is keyed to the last day of the month, not the first. Match that grain or joins miss.
• Critical "done": flag → ISO date. Excel used + / √ for done; the schema wants an ISO completion date. Conversion takes the earliest submission where the task was marked done; - / X / blank → null. Not a boolean.
• Work volumes union, never latest-only. The m² series is replayed across all submissions (highest version per week, oldest-submitted-first) so a blank newest SDU grid can't erase weeks an earlier submission carried — the bug that left 2085 / 2090 / 2119 with no apimtys.
• work_volumes has no Phase-A baseline. Valandos_sav is weekly hours, not m². Work volume comes only from submission SDU sheets.
• Materials dedup by catalog, not by row. Snapshots can carry duplicate (type, name) rows; they're merged by catalog FK with quantities summed, else you double-count.

06 The seam — where the gold sync takes over

The Excel import set the financial baseline. From there, the Databricks sps_table2 sync keeps financials current — this is the boundary between "how it started" and "how it runs now."

• Overlay, not replace. The sync upserts by (project, month) onto the Phase-A baseline. It touches rev/costs/hours + accruals and writes daily project_inventory_snapshot rows for atsargos. It never wipes a project.
• financial_close_date is the Fakt/Planas boundary. Months on or before it are Fakt — import/sync-owned, read-only in the app. Months after it are Planas — PV-editable. The sync advances the close date as new actuals land. Full semantics: docs/design/2026-05-21-marza-faktine-semantics.md.
• Money fail-hard, hours best-effort. The parser aborts on bad money/dates but tolerates missing hours. _ADM codes are dropped; only known project codes are synced.
• Full re-read each run. The sync re-reads the whole gold export daily, so historical recalculations upstream are picked up. It's idempotent by (project, month) / (project, date).

07 Known gaps & caveats

Stated plainly so you can reconcile against them rather than discover them.

• A9 hours semantics undefined. hours_primary / hours_additional are not direct/indirect until Diana's A9 contract signs. Don't repurpose them into that meaning.
• Catalog misses are skipped, not fatal. A submission material whose (type, name) isn't in materials_catalog is logged to scripts/out/sps-import-skipped.json and dropped — the import still commits.
• AV emails are guessed. Resolved best-effort via az ad user list; unresolved AVs get a placeholder domain. Treat AV identity as low-confidence until reviewed.
• Backfilled submissions bypass KONTROLE. Imported weekly_submissions are written status='submitted' but don't necessarily satisfy the live-submit invariants (work-volume presence, 5/5 touch coverage). Read models stay populated because they source from Phase A + the union, not from any single snapshot's completeness.
• Apimtys version-dedup ~1% edge. Highest version per week wins; a rare empty top version can drop m² a lower version carried (2 of 218 weeks). Accepted, not fixed.

08 What could be owned next

The manual Excel import was a bootstrap, not a pipeline. The sustainable version is an automated source → SPS ingestion (Rivile / gold → the same Postgres tables). Sections 04–06 are its functional spec — reproduce those transforms and the boundary rules and the read models stay correct.

Open items to scope if that's the direction:

• A9 hours taxonomy. Blocked on Diana's contract; until then don't split direct/indirect.
• Materials catalog governance. Catalog is app-owned now; an automated feed needs a rule for unknown (type, name) beyond "skip + log".
• Incremental vs full-refresh. Today's import is reset-then-rebuild once submissions exist; the gold sync is full re-read. A real pipeline should decide its own change-capture strategy per table.
• Identity resolution. PV/PS/AV name→user matching is string + best-effort Graph lookup. A durable pipeline wants a stable person key upstream.

09 Reference & contact

Everything named on this page, in the SPS V4 repo.