Data Ingestion & Organization — The Plan
v1.2 · 2026-07-11 (v1.1 transport-agnostic connectors; v1.2 aligned to Allan's implemented field-event pattern, self-serve connectors — no flip order, FE/BE seam drawn) · implements AC-CANON ING-1…9 · companion to the demo DB artifact + schema issue
#764The model — one pipeline, both directions
sources → connectors (any transport: MCP adapter · REST/GraphQL API · CLI/runner · webhook · file drop/upload · email) → source-tagged field events (Allan's implemented pattern: e.g. po.po_field_events, source ∈ collmex · jira · excel · manual · d365) → OKF normalization → HIL conflict resolution (the VP Data surface) → confirmed truth → every lens
↑ provenance on every record (ING-1) · trust % visible (ING-4) · human answers cascade (ING-3)
VP-created & corrected data (ING-5/7) → staged write-back contracts → client systems (D365 / Collmex / ERP) — queued · audited · reversible
The FE/BE seam (Clay 2026-07-11): the pipeline — runners, adapters, field events, OKF processing — is Allan's side. We understand it (verified against vervana-backend 2026-07-11: Collmex/Jira/email×2/flowzz/B-Bot/Excel-import live; D365 & Odoo staged plumbing) and give feedback via GitHub issues; we do not build it. Frontend owns: the Data surface (conflict queue, cascade, trust meter), connector status + self-serve auth UI (clients connect sources themselves, like adding a connector to Claude/ChatGPT — no central rollout order), upload/triage, write-back review UI, and every lens that renders confirmed truth.
Transport-agnostic by design (Clay 2026-07-11): the connector layer never dictates the mechanism — D365 arrives via Allan's CLI runner, bridge orders via API, flowzz via API, email via MCP, order-desk truth via file upload. A connector = whatever transport the source speaks + a versioned mapping into OKF. Adding a source is configuration + mapping, never architecture.
OKF is the interchange (ING-9): one knowledge file per entity. Everything ingested normalizes into OKF; everything confirmed exports as OKF. The backend OKF pipeline is Allan's lane (working concept in hand). VP never blocks on a source: staged connectors + the HIL queue carry any gap.
Source inventory — state today, owner, what it unlocks
| Source | State | Owner | Feeds / unlocks |
|---|
| D365 (ERP) | STAGED | Allan (runner ~1-min exists; hard gate for catalog go-live) | Products, batches, POs, cost truth; first write-back target |
| Bridge / shop orders | LIVE | Claude/codex (bridge API in prod) | Orders, order lifecycle, pharmacy demand |
| Flowzz market feed | LIVE | Claude/codex (feed + snapshot API running) | Market prices by strain → catalog market columns, demand signals |
| Collmex (finance/ERP) | LIVE | Allan (full adapter suite in backend, verified) | POs, deliveries, products, invoices — the primary live source today |
| Excel / manual uploads | LIVE (as reality) | Claude/codex (upload UI = ING-7/8) | Order desk truth today (Lieferschein flow); docs born-tagged |
| Email ingest | LIVE | Allan (two paths verified: Gmail poll + inbound webhook with AI permit extraction → po_email_signals) | Permit/PO signals today; full thread fabric per Messages #140 next |
| WhatsApp | CONCEPT | Claude/codex (webhook per #140) | Same fabric, biggest counterparty channel |
| Trade register / regulator lists | CONCEPT | Claude/codex | Connect-layer pre-population (the 36k funnel) + KYB |
| Jira (regulatory checklists) | LIVE | Allan (DB mirror queried by pipeline; sync runner external) | 16-step regulatory status per PO |
| B-Bot knowledge graph | LIVE | Allan (5-min product/CoA poll + DAG storage) | CoA enrichment, lot/event documents |
Organization — how data stays clean (the VP Data surface)
- Provenance on arrival (ING-1/7): every record — system-ingested or human-entered — carries source, time, method, mapping version.
- Conflicts route to their human (ING-2): the Data queue knows the section owner; VP drafts the message with source links; item parks until answered.
- Answers cascade (ING-3): one confirmation collapses flags across the graph; cascade count shown, audited.
- Trust meter to cutover (ING-4): gold-tier % per section; ERP-cutover threshold is a number, not a debate.
- Docs are data (ING-8): every upload attaches to its entity with an RBAC category; unattached → triage queue, never a void.
Go-live sequence — "just needs ingestion," literally
- Schema settles — #764 accumulates a proposal per slice; demo DB artifact renders it as DDL + seed (today's slice 5).
- Allan migrates — demo schema → real vervana_app; OKF pipeline connects (his lane, working concept).
- Clients self-connect — each client connects their own sources via the auth-step connector UI, any order; our side is surface + verification, never a rollout schedule.
- Trust threshold — per-section gold % passes the cutover bar → that section shows real data, write-back arms.
- Live — /demo world remains as the permanent fixture/test bed; prod lenses read gold.
No rollout order (Clay 2026-07-11): connectors are self-serve — each client connects their own sources through an auth step, in whatever order they choose. What we build is the connector surface (status, auth flow, "Connect X → unlock Y"); who flips what when is the client's call.
DATA GATE: ingest-completeness ✓ (per-source counts vs. source-of-truth) · unit-normalization ✓ (OKF mapping versioned) · anomaly-detection ✓ (HIL queue thresholds) — enforced per ING ACs, evidenced in vp-verifications.json.
Owners: Allan = OKF pipeline, D365 runner, migrations, Collmex · Claude/codex = connector UI, Data queue, upload/triage, write-back UI, MCP adapters · Clay = flip order, trust threshold, free/paid line. · Clay ⇄ Claude · 2026-07-11