01 / Regulated AI

Stoa: sovereign consultation analysis

Role
Sole engineer and technical lead. Paid engagement through Fridai, Oct 2025, ongoing.
Scope
Analysis pipeline, backend, review frontend, GPU and cloud infrastructure, sovereign deployment and operator run sheets, validation, client-facing RFP work, documentation.
Stack
Python · Django · Astro/Svelte · Postgres + pgvector · Docker · Terraform · AWS (SageMaker, Bedrock) · Slurm · Ollama · self-hosted Llama 3.3 70B
Status
Delivered and in use. In 2026 it ran on self-hosted open-weight models inside New Zealand for a government consultation, and the firm's analysts used the reviewed output in the report they delivered.
~1 hour
AI mapping stage per 1,000 submissions
84.3%
theme recall against one analyst team's coding
2,000+
public comments analysed inside New Zealand for a government client
NZ$20-30k
incumbent annual licence spend

An agency runs a public consultation, gets tens of thousands of submissions back (one contentious bill drew over 300,000), and has weeks to tell ministers what the public actually said, defensibly enough to survive public scrutiny. Coding open text at that volume by hand does not hit the deadline, and submission counts keep climbing.

A conventional cloud SaaS product was not an option. The New Zealand qualitative-research firm I built this for works with government bodies under data-sovereignty rules that barred sending citizen submissions to such a service. Cost also mattered. By my estimate, commercial alternatives can cost tens of thousands of dollars per run, with model API charges making up much of that bill.

I built Stoa to handle both constraints. It combines a configurable analysis pipeline of up to nine steps with a review workspace. I built the pipeline, Django backend, review frontend, infrastructure, evaluation and documentation. It started as a paid pilot on real consultation data. In 2026 it went into a government consultation on a draft technical standard: more than 2,000 public comments, analysed on self-hosted open-weight models inside New Zealand.

1. The platform, and what's actually deployed

Nine configurable phases take raw submissions to a hierarchical theme tree, from broad categories down to leaf themes, so a reviewer can move from a one-screen summary to the individual submissions behind any theme. Django backend, Astro/Svelte review frontend, Postgres with pgvector. The delivered application was containerised and runnable by the client. Inference runs on self-hosted open-weight models, so citizen submissions never reach a third-party model API. That boundary started as a design and in 2026 carried a government consultation (section 6).

Provenance and scope. Stoa began as a fork of UK i.AI's open-source Consult, which put a working Django core, Terraform and CI in place and turned a from-scratch build into an adaptation against a November deadline. The pipeline extensions, the sovereign inference port, the retrieval layer, the review frontend and the AWS migration are mine. Initial scope was narrow by agreement: open-weight hosting, batched rather than bulk classification, augmentation rather than replacement of analysts, and inaccuracy mitigated by a visible human check.

2. From submissions to an accountable result

Nothing ships unseen. No theme mapping leaves the system without a human able to check it. The review frontend puts AI-proposed mappings beside the source submission, lets a reviewer filter and correct in bulk, and keeps a timestamped audit trail of every change. Each mapping carries its provenance, AI-proposed or human-corrected, so the trail reflects who actually decided what. Around it sits the working surface analysts need: hierarchical theme views, stacked filters, respondent-ID lookup, and tags for confirming or extending the model's mappings, plus the login, MFA and encryption a government-adjacent deployment has to have.

The pipeline behind that workspace runs up to nine configured analysis steps: sentiment analysis, initial theme generation, theme condensation, theme refinement, an optional target-alignment step, hierarchical clustering, response-to-theme mapping, and two further optional steps, detail detection and response analysis, switched on when a client's brief calls for them. Fallback lives inside response-to-theme mapping itself: a response that fits no leaf theme climbs to its parent, and to an explicit OTHER if nothing fits, so nothing is silently dropped. Analyst review and the audit-trailed export come after the analysis pipeline, not inside it. When configured, a targeted legislation-retrieval step grounds sentiment analysis, response mapping and detail detection in the bill's actual text rather than the model's memory of it.

SYSTEM MAP / 01

From raw submissions to an accountable result

How raw submissions move through the configurable analysis pipeline to analyst review and an auditable export.

  • automation
  • human review
flowchart TD
  accTitle: Stoa consultation analysis and analyst review path
  accDescr: In the delivered application, raw submissions can pass through nine configurable analysis steps: sentiment, theme generation, condensation, refinement, optional target alignment, hierarchical clustering, response mapping, optional detail detection and optional response analysis. Mapping falls back from leaf to parent or OTHER. Analyst confirmation or correction and the audit-trailed export follow the analysis pipeline. A separate note identifies the self-hosted open-weight inference that serves the model-assisted steps, which ran on New Zealand research compute in the 2026 government delivery.
  subgraph APP["Delivered application"]
    A["Raw submissions · schema checked and batched"] --> B["01 · Sentiment analysis"]
    B --> C["02 · Initial theme generation"]
    C --> D["03 · Theme condensation"]
    D --> E["04 · Theme refinement"]
    E --> F["05 · Target alignment · when configured"]
    F --> G["06 · Hierarchical clustering"]
    G --> H["07 · Response-to-theme mapping"]
    H --> H1{"Leaf match?"}
    H1 -->|No| H2{"Parent match?"}
    H2 -->|No| H3["Explicit OTHER"]
    H1 -->|Yes| I["08 · Detail detection · when enabled"]
    H2 -->|Yes| I
    H3 --> I
    I --> J["09 · Response analysis · when enabled"]
    J --> K{{"Analyst reviews source and proposed mapping"}}
    K --> L["Correction or confirmation · actor + timestamp"]
    L --> M["Auditable theme export"]
  end
  N["Targeted legislation retrieval · optional grounding path"] -.-> B
  N -.-> H
  N -.-> I
  P["Self-hosted open-weight inference · New Zealand research compute · ran in the 2026 government delivery"] -. "serves the model-assisted steps" .-> B
  classDef human fill:#f6ad55,stroke:#c05621,color:#1a1a1a;
  class K human;
The full application path. The 2026 government delivery ran its discovery and summary steps as batch jobs on the self-hosted inference shown, with analysts reviewing the output before use.

3. Why self-hosting, and how the models carry the work

Two constraints, one answer. Sovereignty made hosted models impossible. Cost made self-hosting attractive rather than merely necessary: it turns a per-run vendor bill into a metered GPU cost the client can control, and buys what the SaaS products do not sell, since the prompts, the theme hierarchy and every pipeline stage are the client's to configure, so the tool follows the firm's analysis method instead of the vendor's. The 2026 government run was self-hosted in exactly this way.

Cost drove the engineering too. Operating cost is a function of GPU rental time, so run times were measured net of throttling and retries rather than naively, and development moved off rented clusters onto Terraform-provisioned AWS, then SageMaker and Bedrock as a cheaper surface for performance work. Development data only; no citizen submissions went through it.

Depth, matched to the client's method. Upstream's theme-finding topped out around twenty flat themes, built for smaller local-government questions. The firm's manual method at legislative scale produces hierarchical theme lists 50 to 100 deep. I learned that methodology, encoded it into per-stage prompts, and hit the inverse problem: open-weight context windows force small batches at theme generation, so the same themes regenerate batch after batch. I added condensation and reduction stages, then hierarchical clustering to group leaf themes under parents.

Degrading without dropping data. The hierarchy also gave mapping somewhere to go when it fails. A response that fits no leaf theme climbs to the parent, and to an explicit OTHER if nothing fits, rather than being forced into a bad match. Every response is accounted for, and a degraded run stays legible instead of quietly dropping data (Exhibit 16).

Guardrails a frontier model would not need. Smaller models against long, messy submissions ran behind three of them:

  • enforced JSON structured output, with validity checks on the returned shape
  • token-limit handling, against mid-answer truncation and context bloat
  • prompt hardening, because some genuine submissions were being read by the model as instructions

Bedrock throttling added a retry ladder on top of that (Exhibit 01, Exhibit 02).

Model selection. A bake-off across Llama, Qwen and Mistral at several parameter sizes, scored on analysis quality against the firm's own coding and on licence terms that survive a sovereignty review. That second screen does real work, since "open source" and "fully self-hostable" are not the same claim. Llama won on both, and on a third criterion that mattered for a government bid: it was the only open-weight family then approved for US government use. The RFP named Llama 3.3 70B Instruct.

Grounding in the legislation. Consultations respond to a bill, so I built retrieval over the bill itself: parsed into a section-level corpus in pgvector, with every theme mapped to the sections it concerns, so a reviewer checks a citation instead of trusting the model. Retrieval is hierarchical, shaped to how legislation is actually structured, and backed by server-side hybrid search (Postgres full-text plus vector embeddings) rather than one naive index. Sanitized and public as Exhibit 13.

4. What the validation showed

84.3% recall against the analysts' own coding. I matched the firm's analyst coding against Stoa's theme assignments programmatically in Python. That analyst coding was completed independently before Stoa saw the data, so the comparison leads with theme recall rather than any measure Stoa could have shaped in advance. Recall is the number that matters, because the design goal was to surface everything that might matter and let an analyst decide what does.

The gap is over-assignment, not error. Where the two disagreed, Stoa had assigned themes the analysts had not. The analysts were shaping their final report as they coded and stopped where the report needed them to, while the system kept going. Qualitative coding has no single gold standard, so agreement between the two is not accuracy against some independent truth, which is why the review step below is load-bearing.

5. Review, audit, and a second product

A second product fell out of it. Once those screens existed, so did an adjacency. The firm spends roughly NZ$20-30k a year on per-seat NVivo licences, the incumbent qualitative coding tool, whose interface has aged into something close to a bare spreadsheet. Stoa's review workspace already does the core of that job without a per-seat licence, so displacing that spend is a roadmap goal in its own right. Same argument as the rest of the engagement: replace per-run and per-seat SaaS cost with something the client owns and can configure.

Explainability, designed rather than assumed. Stoa writes a reason alongside each theme mapping, generated after the decision rather than a trace of it. Interpretability work on frontier models keeps finding that stated reasoning can diverge from the computation that actually produced the answer, so treating it as a window into the model would be a governance error when the output has to survive public scrutiny. It is specified as supporting text for a reviewer; the human step carries the accountability. The stronger version is on the roadmap: capture the justification in a scratchpad at mapping time, before the decision rather than after, which self-hosted weights put within reach.

6. Sovereign delivery for a government client

It ran where the data had to stay. In 2026 the firm was engaged to analyse a government consultation: more than 2,000 public comments on a draft technical standard, written in the industry's specialist language. Stoa processed them on self-hosted Llama 3.3 70B, as Slurm batch jobs on a single GPU at a New Zealand national research and education network. No comment left New Zealand or reached a third-party model API.

The pivot: a summary for every part of the draft. Stoa was built to find themes in free-form submissions. These comments arrived already tagged to the clause, table or figure they addressed, and the firm's client first needed a view of opinion on each of those parts, which the analysts were building by hand. Within a day of the request I added a summary step, built on the batch jobs, model checks and input contract already in place and on what I knew of how the firm's analysts work. For each group of comments on the same part of the draft, it reports where submitters agree on a problem, where they agree on a change and where they differ. Counts come from code, and it reports agreement only when separate submitters express it. The summaries covered fourteen sections. The firm's analysts reviewed them and leaned on them heavily, against a tight deadline, in the report delivered to the government client.

Themes shaped to the document. Theme discovery changed with the data too. It ran separately for each of nine sections of the draft, with that section's context in view. The most precise themes stayed at the part of the draft they concerned, grouped under parents that cut across locations, and the final run seeded its themes from the analysts' own groupings.

Built without seeing the data. The submissions were restricted to the firm and the cluster, so I never held them. I developed on synthetic fixtures and pinned every release by commit. The firm's director ran each job from command sheets I wrote, and each job checked the pinned model, its context window and its GPU placement before going further. Real files still disagreed with the fixtures: matching the analysts' regrouped workbooks to the dataset on comment text passed every synthetic test and missed about one comment in five, so I rebuilt the release to match on submission ID.

Failing closed, then degrading visibly. The first real run passed every identity check and then stopped at a strict content check. The runner logged nothing that could contain submission text, which also hid the cause. Within two days I shipped a release that counts each fallback against a ceiling instead of aborting on the first imperfect row, and reports stage and error type without prompts or responses. It is tested end to end against a fake model, and every later run was built on it.

The route there. I first proposed inference at that network in a two-envelope RFP response, co-authored with the firm for a New Zealand government agency against NZISM and Protective Security pre-conditions, with data-residency guarantees written in. I worked the design through with the network operator and exercised it on an equivalent AWS deployment. Mid-process the agency's requirements moved from an internal tool to a platform it could inspect itself, with accuracy reporting, penetration testing and mapping to a taxonomy of around 100 topics agreed in advance, which made defining the firm's themes with its analysts the consultative part of what we costed. The bid was not selected. It left first-hand knowledge of what a public-sector buyer screens for beyond the technology: evidence of the tool in daily use, and a delivery model that does not rest on one person.

What shipped in between. I got the application into a deliverable state, containerised and runnable by the client, with a user guide covering data schemas, the nine-phase pipeline, the dashboard, evaluation workflows and a complete prompt reference. The commercialisation roadmap I wrote alongside it set out five pathways and recommended the consulting model first, with sovereign AI deployment as the highest-value next investment. The 2026 delivery was both.

7. Evidence