Bart Lind

Agora: feedback-analysis product

Role
Sole engineer. Self-directed product build through Fridai, on the analysis engine from the Stoa engagement.
Scope
Analysis pipeline, dashboard and human-review surface, onboarding wizard, public demo, deployment and hosting.
Stack
Python · Django · Astro/Svelte · Docker Compose · Caddy · Hetzner VPS
Status
In productisation. Public demo and one-command runnable bundle both live.

A business sitting on tens of thousands of customer verbatims (survey responses, reviews, open-text comments) has no route from that pile to an answer it can filter, act on, and defend. Reading them takes weeks the decision doesn't have. Handing them to a model returns a summary nobody can check, which is the worse of the two failures: a themed answer you can't audit isn't an answer a business can act on.

Agora takes a client's raw feedback through a preset analysis pipeline into a live, login-protected dashboard: hierarchical themes, stance and sentiment, with every AI-proposed coding decision visible and correctable rather than taken on faith. I built and deployed the product end to end.

The point of it is access. Thorough, checkable analysis of what people say about a business has been the preserve of organisations that can afford a research firm and wait weeks for the answer. Agora is built to put the same rigour inside a decision window, at a cost that lets a business run it routinely rather than once a year.

1. Where it came from

The engine is Stoa's. Agora forks the analysis engine behind Stoa, the sovereign consultation-analysis platform I built for a New Zealand qualitative-research firm, and aims it at commercial feedback instead of government consultation review. The analytical commitments carry over intact: batched analysis rather than one bulk classification pass, hierarchical themes deep enough to be useful, mapping that climbs leaf → parent → an explicit OTHER instead of forcing a bad fit, and a human check as the mitigation for inaccuracy. The Stoa case study covers how each of those was built.

The sovereignty stack does not carry over. Self-hosted open-weight inference existed to meet a government data-residency requirement. Agora has no such requirement and does not run that way.

2. Why orchestration beats a bigger model

The naive version doesn't work. Handing the whole pile to a frontier model and asking for themes fails in ways the business can't see: themes drift between runs, long inputs get skimmed, nothing traces back to a specific response, and the bill scales with the pile. Structure is what closes that gap: nine stages with defined inputs and outputs, prompts encoding a documented analysis method, batches sized to the model's context, and de-duplication and clustering as explicit steps rather than something you hope happens inside the model. Run the same corpus twice and you get the same shape of answer, with every theme traceable to the responses behind it.

It runs on small, fast models, by design. The engine had to work on open-weight models a government client could host itself in late 2025, well below the capability ceiling of a frontier API. The structural guardrails exist because the model could not be trusted to hold the whole job in its head. The by-product is that the pipeline still runs well on cheap, fast models now that it doesn't have to: the class that includes Gemini Flash, Claude Haiku and the OpenAI mini tier, where a million input tokens costs about a dollar.

What that costs in practice. The demo run put 1,643 reviews through six LLM stages, four of them touching every row, roughly 6,600 row-level coding decisions, in 353 model calls. On the benchmark run, a full pipeline pass over 800 texts came to about $0.24 in model spend, priced conservatively. Off-the-shelf AI-assisted analysis tooling in this space runs to the order of tens of thousands of dollars per engagement, a large share of it model API cost passed straight through. That gap is not a better model. It is orchestration: batching, staging, and spending tokens only where a decision actually has to be made.

3. What the accuracy record actually is

The engagement validation is the evidence. Because Agora forks the Stoa engine, its accuracy record is Stoa's, and that record comes from consultation submissions rather than customer reviews: 84.3% recall against a research firm's own analysts, on a legislative consultation they had already coded by hand before the engine saw it. Where the two disagreed the engine had assigned themes the analysts had not, so the gap is over-assignment rather than error, and qualitative coding has no gold standard to be precise against. The Stoa case study carries the full account and the companion figure.

Nothing above was measured on the demo data. The Emirates review run behind section 5 is a real traced run that demonstrates the product surface. No quality figure attaches to it.

A component regression check, on a third dataset. I run the unmodified production pipeline over SemEval-2014 Task 4, a public academic set of restaurant reviews, and score it in-house. Run as a client install runs, discovering its own themes rather than being handed the answer key, topic detection comes to 0.83 F1, after aligning its 35 discovered themes to the benchmark's five categories, roughly a third of those alignments being judgment calls. Given the five categories up front, the setting the 2014 entrants competed in, it edges the best of that field on topic detection and lands level with it on sentiment polarity.

Which of those to believe. The benchmark is a regression check: a public dataset scored by the person who built the system, on a task and a vintage that are not the product's. It exists so a pipeline change that quietly degrades coding gets caught. The engagement is the evidence: a paying client whose analysts had independently coded the same data first.

4. Human-verifiable by design

Every call is correctable. A reviewer can overturn any theme, stance or sentiment decision, and the hierarchy means a high-level summary claim can be audited down to the specific verbatims behind it. That review discipline is inherited from Stoa, where it was a hard requirement rather than a nice-to-have.

What the stated reason is. Each mapping is saved with the model's reason for it, generated after the decision rather than traced from it. Interpretability work on frontier models keeps finding that stated reasoning can diverge from the computation that produced the answer, so it is never presented as introspection. It ships because a reviewer who disagrees with a stated reason has something concrete to push against, which beats arguing with an unexplained label. The Stoa case study covers how that is specified and what replaces it on the roadmap.

5. Portfolio demo

Built to be inspected, not just watched. Agora is a login-gated product, so the public demo is a separate, deliberately open artifact: a static case-study page replaying a real traced pipeline run over 1,643 public Emirates reviews from Trustpilot and Skytrax, a captured walkthrough of the LLM-assisted onboarding wizard, and a dashboard tour. A restricted build of the application is also downloadable as a docker compose up bundle; the real dashboard, filtering and human-review flow against that same pre-analysed dataset, with the proprietary pipeline code excluded and a disclosed stub in its place.

Deployed solo. A Hetzner VPS behind a Caddy edge with automatic TLS, per-engagement Docker Compose stacks, a hardened host (key-only SSH, ufw, fail2ban), uptime monitoring, and cost-capped LLM runs so a demo cannot run up an open-ended bill.

See it live

6. Evidence