01Enterprise LLM platform · 2026
NeuroAudit
first prototyped as NeuroValidate
An LLM platform that reads unstructured clinical notes against study criteria, quotes the exact sentence behind every answer, and sends anything it cannot prove to a human reviewer.
My roleProduct owner from the first line of code
Problem
Research analysts were answering questions like "which patients permanently stopped this therapy, and can you prove it?" by reading charts by hand. The answer is rarely in a structured field. It lives in a doctor's note, where stopped, held, switched and missed a dose look alike and mean very different things.
The company sells real world evidence studies and trial recruitment to life sciences sponsors. Manual review ran on spreadsheets at roughly three hours per hundred patients. It was not reproducible and not auditable, so every new question started the work again.
My role
I owned NeuroAudit from the first line of code. I built the first clickable prototype myself in Claude Code (113 commits between February and September 2026), wrote the specs and ran the roadmap with a matrixed pod across backend, frontend, data science and QA.
- Two journeys, not one: build your own evidence study, or validate an existing trial cohort.
- Hard gates: keywords must pass an agent's check before a run, and full runs lock their prompts and strip personal data first.
- Cost before compute: every run shows an estimate, and runs above a set limit need an approver.
- A prototype flow only counts once production ships it. Every flow in my prototype has shipped except the agent builder.
What shipped
- Create a projectName it, pick the data source and choose the journey: evidence study or trial cohort.
- Define criteriaEach criterion gets keywords. A validation agent blocks the run until irrelevant keywords are removed.
- Configure prompts and modelsOne prompt pulls out the evidence; a second checks that the evidence really says it. The model is picked per criterion from a lineup ordered by cost.
- RunSample, specific records or a full run, with the cost shown first. Full runs lock prompts and remove personal data.
- ReviewThe reviewer sees the evidence, the full note and the model's verdict side by side, and marks Accept, Reject or Unsure with a required note.
- ExportA patient by criterion matrix with evidence, model output, reviewer decision and notes.
Iterations
| When | What changed | Why |
|---|---|---|
| Feb 2026 | First prototype, under the working name NeuroValidate | Prove the review loop before spending engineering time |
| Mar to Apr | Production foundations: auth, projects, criteria to prompts to run, review, roles | Handed to engineering; 44 tickets under the old name |
| Jun | Trial cohort journey. Structured criteria pass automatically, so reviewers only see free text | Review effort was the expensive step |
| Jun to Jul | Locked full runs, then a full personal data removal flow | Quality checks found clinician names in run outputs |
| Jul | Clear run states and re-running skipped records | "Pending" looked the same whether a job was healthy or had crashed |
| Aug to Sep | Run log, incremental processing, model comparison, an auto prompter, approvals above a cost limit, access control | Cost and governance asks from users, several prototyped first inside the real app |
Technical choices
- Retrieval: keyword search over an Elasticsearch index of encounters, with a validation agent in front. Hybrid keyword and semantic retrieval is on the roadmap, not shipped.
- Models: multi model by design. An open source small language model is the default for score extraction, alongside other open models and a fast frontier model, ordered from cheapest to most capable. In the flagship study the model was pinned per criterion so results could be reproduced.
- Two pass pattern: one model extracts evidence, a second checks that every quote appears word for word in the source note. Anything that fails goes to a human. Nothing passes silently.
- Architecture: Go services on AWS, a React frontend, MongoDB for projects and run state, Elasticsearch for encounters, queue based workers, reviewer decisions that cannot be edited afterwards and per run token and cost tracking.
Evaluation
Three things are evaluated: whether every extracted claim is grounded in the note, whether reviewers agree with the model on a reviewed sample, and whether the product works end to end on scripted test scenarios. The reviewer's verdict is the ground truth.
What I do not claim: platform accuracy, precision or recall. 84.9% of records needed no human check; that is not an accuracy score. Model and reviewer totals exist but are not yet paired decision by decision, so no honest accuracy number can be computed from them. Pairing them is the next piece of work.
Failure modes we found and fixed: ambiguous run status, a silent hang when a websocket dropped, prompts not locked before a run (caught by the test pass), long data fetches hanging for hours, and feature requests raised in chat that never became tickets.
Impact
Lessons
- Grounding beats confidence scores. Requiring a word for word quote made every answer checkable and turned "send it to a human" into a rule instead of a judgment call.
- Reliability, not intelligence, was the top user complaint. The biggest trust win was a clear job status, not a better model.
- Prototype and production drift. A feature built twice diverged in its columns, so I now treat the prototype as the spec and track what production actually shipped.
