Bart Lind

Stoa: sovereign consultation analysis

Role
Sole engineer and technical lead. Paid engagement through Fridai, Oct 2025, ongoing.
Scope
Analysis pipeline, backend, review frontend, GPU and cloud infrastructure, validation, client-facing RFP work, documentation.
Stack
Python · Django · Astro/Svelte · Postgres + pgvector · Docker · Terraform · AWS (SageMaker, Bedrock) · self-hosted Llama 3.3 70B
Status
In pilot on real consultation data, against an agreed forward roadmap.

An agency runs a public consultation, gets tens of thousands of submissions back (one contentious bill drew over 300,000), and has weeks to tell ministers what the public actually said, defensibly enough to survive public scrutiny. Coding open text at that volume by hand does not hit the deadline, and submission counts keep climbing.

The obvious fix was unavailable twice over. The New Zealand qualitative-research firm I built this for works for New Zealand and Australian government bodies, under data-sovereignty rules that bar sending citizen submissions to a cloud SaaS product. And the commercial AI-assisted analysis tools that do exist were costing on the order of tens of thousands of dollars per consultation run, a large share of it model API charges passed straight through.

Stoa answers both: an AI analysis pipeline the firm runs on infrastructure it controls, turning raw submissions into reviewed, audit-trailed theme analysis. I built it solo: analysis pipeline, Django backend, review frontend, GPU and cloud infrastructure, evaluation, and the documentation the client runs it from. It is a paid engagement through Fridai, my consultancy, in pilot on real consultation data against an agreed roadmap.

1. The platform

The pipeline. Nine configurable phases take raw submissions to a hierarchical theme tree, from broad categories down to leaf themes, so a reviewer can move from a one-screen summary to the individual submissions behind any theme. Django backend, Astro/Svelte review frontend, Postgres with pgvector, containerised with Docker and deployable onto the client's own infrastructure. Inference runs on open-weight models the firm hosts itself, so no submission text reaches a third-party model API.

Provenance and scope. Stoa began as a fork of UK i.AI's open-source Consult, which put a working Django core, Terraform and CI in place and turned a from-scratch build into an adaptation against a November deadline. The pipeline extensions, the sovereign inference port, the retrieval layer, the review frontend and the AWS migration are mine. Initial scope was narrow by agreement: open-weight hosting, batched rather than bulk classification, augmentation rather than replacement of analysts, and inaccuracy mitigated by a visible human check.

2. Why self-hosting was the commercial case

Two constraints, one answer. Sovereignty made hosted models impossible. Cost made self-hosting attractive rather than merely necessary. An off-the-shelf tool meant tens of thousands of dollars per consultation, with model API charges the largest line and no lever to move them. Self-hosting turns that into a metered GPU cost the firm controls, and buys what the SaaS products do not sell: the prompts, the theme hierarchy and every pipeline stage are the client's to configure, so the tool follows the firm's analysis method instead of the vendor's.

Cost drove the engineering too. Operating cost is a function of GPU rental time, so run times were measured net of throttling and retries rather than naively, and development moved off rented clusters onto Terraform-provisioned AWS, then SageMaker and Bedrock as a cheaper surface for performance work. Development data only; no citizen submissions went through it.

3. Making open-weight models carry the work

Depth, matched to the client's method. Upstream's theme-finding topped out around twenty flat themes, built for smaller local-government questions. The firm's manual method at legislative scale produces hierarchical theme lists 50 to 100 deep. I learned that methodology, encoded it into per-stage prompts, and hit the inverse problem: open-weight context windows force small batches at theme generation, so the same themes regenerate batch after batch. I added condensation and reduction stages, then hierarchical clustering to group leaf themes under parents.

Degrading without dropping data. The hierarchy also gave mapping somewhere to go when it fails. A response that fits no leaf theme climbs to the parent, and to an explicit OTHER if nothing fits, rather than being forced into a bad match. Every response is accounted for, and a degraded run stays legible instead of quietly dropping data (Exhibit 16).

Guardrails a frontier model would not need. Smaller models against long, messy submissions ran behind three of them:

Bedrock throttling added a retry ladder on top of that (Exhibit 01, Exhibit 02).

Model selection. A bake-off across Llama, Qwen and Mistral at several parameter sizes, scored on analysis quality against the firm's own coding and on licence terms that survive a sovereignty review. That second screen does real work, since "open source" and "fully self-hostable" are not the same claim. Llama won on both, and on a third criterion that mattered for a government bid: it was the only open-weight family then approved for US government use. The RFP named Llama 3.3 70B Instruct.

Grounding in the legislation. Consultations respond to a bill, so I built retrieval over the bill itself: parsed into a section-level corpus in pgvector, with every theme mapped to the sections it concerns, so a reviewer checks a citation instead of trusting the model. Retrieval is hierarchical, shaped to how legislation is actually structured, and backed by server-side hybrid search (Postgres full-text plus vector embeddings) rather than one naive index. Sanitized and public as Exhibit 13.

4. What the validation showed

84.3% recall against the analysts' own coding. The firm supplied a legislative consultation it had already analysed by hand, and I matched its analysts' coding against Stoa's programmatically in Python. Recall is the number that matters, because the design goal was to surface everything that might matter and let an analyst decide what does.

The gap is over-assignment, not error. Where the two disagreed, Stoa had assigned themes the analysts had not. The analysts were shaping their final report as they coded and stopped where the report needed them to, while the system kept going. Qualitative coding has no gold standard to be precise against, which is why the review step below is load-bearing.

5. Review, audit, and a second product

Nothing ships unseen. No theme mapping leaves the system without a human able to check it. The review frontend puts AI-proposed mappings beside the source submission, lets a reviewer filter and correct in bulk, and keeps a timestamped audit trail of every change. Each mapping carries its provenance, AI-proposed or human-corrected, so the trail reflects who actually decided what. Around it sits the working surface analysts need: hierarchical theme views, stacked filters, respondent-ID lookup, and tags for confirming or extending the model's mappings, plus the login, MFA and encryption a government-adjacent deployment has to have.

A second product fell out of it. Once those screens existed, so did an adjacency. The firm spends roughly NZ$20-30k a year on per-seat NVivo licences, the incumbent qualitative coding tool, whose interface has aged into something close to a bare spreadsheet. Stoa's review workspace already does the core of that job, on the firm's own infrastructure and without a per-seat licence, so displacing that spend is now a roadmap goal in its own right. Same argument as the rest of the engagement: replace per-run and per-seat SaaS cost with something the client owns and can configure.

Explainability, designed rather than assumed. Stoa writes a reason alongside each theme mapping, and I was explicit with the client about what that reason is: text generated after the decision. Interpretability work on frontier models keeps finding that stated reasoning can diverge from the computation that actually produced the answer, so treating it as a window into the model would be a governance error when the output has to survive public scrutiny. It is specified as supporting text for a reviewer, the human step carries the accountability, and the stronger version is on the roadmap: capture the justification at mapping time, in a scratchpad written before the decision rather than after. Self-hosting is what puts that within reach, since with the weights in-house the model's own reasoning tokens can be captured rather than left behind someone else's API.

6. Procurement and delivery

The bid. Alongside the build I co-authored a two-envelope RFP response for a New Zealand government agency, against pre-conditions drawn from NZISM and Protective Security requirements, with data-residency guarantees written into the response. The architecture I proposed put processing on an air-gapped GPU cluster at a New Zealand national research and education network, with only submission text and identifiers ever reaching the model and results uploaded into a New Zealand-hosted platform running the dashboard. It was worked through in detail with the network operator and exercised on an architecturally equivalent AWS deployment.

The requirements pivot. Mid-process the requirements moved to close to the inverse of the original brief: not an internal tool whose process nobody would ask about, but a collaborative platform the agency could inspect itself, with accuracy reporting, a penetration-tested application, NZISM alignment, and mapping to a pre-defined taxonomy of around 100 topics agreed in advance. That last point met a finding from validation, that the firm's own theme definitions were not specific enough for a model to work from, and the fix, sitting down with their analysts and defining themes properly, became the consultative element of what we proposed and costed together.

Outcome. The bid was not selected. What it left is first-hand knowledge of how government AI procurement is structured and evaluated, including what a public-sector buyer screens for beyond the technology: evidence of the tool in daily use, and a delivery model that does not rest on one person.

What shipped. The engagement continued past the bid. I got the application into a deliverable state, containerised and runnable by the client, cleared the remaining functionality bugs, and wrote the user guide covering data schemas, the nine-phase pipeline, the dashboard and evaluation workflows, with a complete prompt reference. Alongside it, a commercialisation roadmap: a maturity assessment separating production-ready from groundwork, enhancements sized near, medium and long term, and five commercialisation pathways, recommending the consulting model first and sovereign AI deployment as the highest-value next investment.

7. Evidence