ORION
How this actually got built

ORION wasn’t built to sound smart.
It was built to never fake it.

Most RAG demos have a dirty secret: nobody’s actually checking if the citations are real. The model says “according to the document,” you believe it, and everyone moves on.

ORION exists because that’s not good enough. Every citation gets checked against the actual source text by plain code — not another model, not a vibe — before anything reaches a user. If it can’t be verified, the answer doesn’t ship. That’s the whole bet this project makes.

Here’s how it actually got built.

Start here

The pipeline, in one breath

Upload a document. It gets parsed and chunked at real structural boundaries — not arbitrary character windows. Ask a question. Hybrid retrieval pulls candidate chunks using both keyword search and semantic similarity. One model call drafts an answer with citations.

Then the part that matters: deterministic code checks every citation through three separate gates — was this chunk actually retrieved this run, is it still valid in the database, does the quoted text actually appear verbatim in the source.

Fail any gate, the whole answer gets thrown out. Not trimmed. Thrown out.

A refusal and a rejected answer are two different outcomes for a reason — one means “the evidence wasn’t there,” the other means “the model said something it couldn’t back up,” and users deserve to know which one happened.

That’s the architecture. It didn’t arrive clean. Here’s what it took to get here for real.

Seven things that actually happened

The build arc

  1. Shipped

    Shipping the pipeline was the easy part

    Ingestion, chunking, retrieval, generation, verification — end to end, deployed across Vercel, Render, and Neon. That’s table stakes. The real work started after.
  2. Found

    The analytics were quietly lying

    Two documents that happened to share a title were getting their stats merged into one row — a title isn’t a real identity, an ID is. Fixed by keying everything on document ID instead. Small bug. Real lesson: anything that looks like an identifier probably isn’t, until you check.
  3. Found

    The live demo wasn't actually live

    render.yaml had both the LLM and embedding providers set to fake — deliberately, for cost reasons, but that meant the deployed instance wasn’t doing real semantic retrieval or real generation. It was a working simulation of itself. Every other open task got dropped to fix this first, because none of it means anything if the URL people click doesn’t demonstrate the thing it claims to.
  4. Built

    Fixing it meant not breaking it a different way

    Flipping on a real LLM on a public URL with no safeguard is just a different flavour of irresponsible. So the fix was a real per-tenant daily cap — not an in-memory counter, because the free-tier host spins down when idle and memory doesn’t survive that. Backed by the persisted run history instead. And the retry math got checked properly: a step can retry twice, so a cap of 20 runs a day is actually a ceiling of 60 provider calls, not 20. Get that wrong and “bounded cost” is just a guess.
    3×the cap, not 1×
  5. Checked

    Going free wasn't free

    Before spending anything real, the free-tier options got a real look — Groq, Gemini, the actual published limits, not the marketing numbers. The finding: free-tier rate limits are usually shared across the whole project, not per visitor. A slow Tuesday and a busy demo day hit the same wall. A small, bounded real cost beat a free tier that could 429 an interviewer mid-demo. That’s not a cop-out, it’s just math.
  6. Proved

    The moment that actually mattered

    A script got built that doesn’t trust the server’s word for it — it independently re-verifies every citation against the raw wire payload, client-side, checking for the fake provider’s literal fingerprint to prove a real model was actually running. It ran a real investigation on the real production API. Four claims came back. All four independently verified as verbatim quotes in their cited chunks.

    Here’s the part that actually matters: the citation verifier had never been tested against a real model before that moment. The fake provider is grounded by construction — it literally can’t fail its own check. A real model writing real prose and still landing 4-for-4 was the first actual evidence the verification layer does the job it exists to do.
    4 / 4citations verified
  7. Measured

    Then the eval, and the twist nobody expected

    A quality bar had been sitting unmeasured for months, because measuring it meant spending real money. Once the real model was live, that excuse was gone. So: 18 real generations, real embeddings, three cents for the whole run. Two of three bar criteria passed clean — zero failed verifications, latency well under the limit. The third didn’t hit its number.

    Digging into why turned up something better than a pass would have been: the bar itself was broken. It came from an earlier baseline that counted wrong answers as “answered.” Fix the counting, and the model got every single case with a defined right answer correct — 9 for 9. The one real gap that survived a full manual review: on one question category, it played it too safe and refused when it could have usefully cited a partial answer. Real finding. Recorded. Not smoothed over.
    9 / 9cases with a defined answer
The edges

What’s still honestly not finished

Embeddings are real now — but rate-limited to 3 requests a minuteThis one used to say “embeddings are still fake, and there’s no paid drop-in.” The second half was never checked, and it was wrong: a hosted provider existed the whole time. It’s live. The honest catch is that the account has no payment method, so the free tier allows three embedding calls per minute — quoted from the provider’s own error, not guessed. Past that, search falls back to keyword-only and says so rather than failing.
Fusion weighting is untouchedThe semantic channel is weighted twice as heavily as the keyword one. That was set while the semantic side was hash noise, where it was actively harmful. It carries real signal now, so it’s no longer wrong the same way — but it was never deliberately right either, and it stays untouched until there’s real relevance data to tune against.
One content-safety pattern over-triggersIt still fires on ordinary sentences. Tracked, not suppressed.

None of this is hidden. All of it’s tracked. A system that claims to be finished usually isn’t being honest — this one would rather tell you exactly where the edges are.

Want the receipts?

Every claim traces

Every claim above traces to a real commit, a real ADR, or a real measured number. The architecture page renders the design documents straight out of the repository, so what you read there is what is actually committed.

See the full architecture