The Inference TimesDemo deskPublic demo · files up to 1.5 MB

Voice Insight

An open-source pipeline: speech-to-text, an LLM, and a verifier that checks every quote against the recording.

  1. Transcribe
  2. Measure
  3. Infer
  4. Verify

Open source · MIT licensed · reference implementation

Audio in, grounded insight out.
A pipeline you can read and fork.

A working, end-to-end example of speech-to-text followed by LLM inference — with every model claim checked against the transcript before it is shown. The worked example is call-centre quality review; the pattern carries over to any recorded conversation.

Run the demo →Read the source
3
stages: transcribe, infer, verify
0
databases or storage buckets
~50s
per 10-minute call, measured
~$0.05
API cost per 10-minute call

The pipeline

Four stages, each in its own file

Three short API routes drive it from the browser: upload, poll for the transcript, then analyse. No queue, no webhook, no database.

  1. 01

    Upload and transcribe

    Audio is relayed to the speech-to-text provider. The language is detected, the two speakers are separated, and every word gets a timestamp and a confidence score.

    src/lib/providers/asr-assemblyai.ts

  2. 02

    Measure without a model

    Talk share, silence and interruptions are plain arithmetic over the word timings. Keeping these out of the LLM makes them repeatable.

    src/lib/metrics.ts

  3. 03

    Infer with structured output

    The transcript goes to the LLM in its original language with a rubric and a JSON schema. Every observation must cite a verbatim quote, a timestamp and a speaker.

    src/lib/analysis/prompt.ts

  4. 04

    Verify before showing

    Each quote is located in the real transcript. Anything that cannot be found, is at the wrong time, or is attributed to the wrong speaker is dropped and counted.

    src/lib/verify.ts

Browser
 ├─ POST /api/transcribe   audio bytes → ASR provider, returns a job id   ~2s
 ├─ GET  /api/status?id=   poll every 3s                                 30-60s
 └─ POST /api/insight      transcript → LLM → verify quotes              ~20s

Concepts

Techniques worth borrowing

Each of these is a design decision you will face in any voice-to-insight system. Each one links to where it is implemented.

नमस्ते सरவணக்கம்¿Bueno?مرحباGood morning

Language detection, not declaration

Nobody tags the recording. The ASR provider detects the language per file, so mixed-language datasets need no preprocessing.

src/lib/providers/asr-assemblyai.ts

AgentCustomer

Speaker diarization

Two speakers are split automatically. Diarization gives labels (A/B), not roles — the template guesses the agent and says that it is a guess.

src/lib/metrics.ts

0:14 · Speaker A“I am really sorry, that is a long time to be without a connection.”

Evidence-carrying output

The JSON schema makes a quote, timestamp and speaker mandatory on every observation. In the UI, clicking a quote seeks the audio to that moment.

src/lib/analysis/schema.ts

measured73% talk · 5.2s silence
judgedempathy, ownership, close

Measured kept apart from judged

Deterministic metrics and model opinions are computed separately and labelled separately, so a reader always knows which is which.

src/lib/metrics.ts

आपformalतूfamiliar

Both become “you” in English. One is a politeness violation.

No pre-translation

Translating before inference destroys what is being assessed — आप versus तू, usted versus tú, a switch out of English. The model reads the original.

src/lib/analysis/prompt.ts

Ask permission before a wait, rather than only telling the customer to bear with you.

A rubric as data

One list of categories drives the prompt, the output schema and the chart. Change the list and you have a different analyser.

src/lib/quality-guide.ts

Verification

Treat model output as untrusted input

In quality review, an invented quote becomes a false claim about a named person. So the pipeline does not trust the LLM: every quote is checked against the transcript before it reaches the page.

Three checks on every claim

  • 1

    Is the quote really in the transcript?

    Fuzzy-matched against the word stream, tolerant of punctuation and reformatting.

  • 2

    Was it said when the model says it was?

    Checked against the nearest real occurrence — which is what makes short, repeated phrases verifiable.

  • 3

    Did the right person say it?

    Compared against the diarized speaker at that moment.

Anything that fails is dropped and counted. The count is shown with every result, and if nothing survives, no score is shown. See src/lib/verify.ts and its tests.

KEPT

Owned the failure by stating the prior tickets were never assigned.

0:31 · Speaker A · conf 1.00“I can see 2 tickets were raised, but neither one was assigned…”

DISCARDED — QUOTE NOT FOUND

“I have issued a full refund to your card.”

Plausible, well-formed, and never said on the call. This is what the verification step exists to catch — and the reader is told it happened.

Build your own

Where to make it your own

The call-centre framing lives in a rubric and a prompt. The pipeline around them is generic, and each provider sits behind a small seam.

Swap the speech-to-text provider

Implement the four-method AsrProvider interface — upload, submit, status, fetchResult — for Whisper, Deepgram or anything else.

src/lib/providers/asr.ts

Swap the LLM

The client speaks the OpenAI-compatible protocol, so another hosted model or a self-hosted vLLM endpoint is a base-URL and model-name change.

src/lib/providers/llm.ts

Change the use case

Replace the rubric and the system prompt to review sales calls, interviews, clinic visits or lectures. The verifier does not care what the domain is.

src/lib/quality-guide.ts

git clone https://github.com/nitingoswami-io/voice-insight-template.git
cd voice-insight-template
npm install
cp .env.example .env.local   # add two API keys
npm run dev

You need an AssemblyAI key for transcription and an xAI key for analysis; both offer free credit. Tests run with no keys and no network.

Architecture guideContributing