Why I spent 3 weeks obsessing over voice latency for my interview prep SaaS

Most early-stage founders are obsessed with adding new features, tweaking their pricing tiers, or optimizing their landing page conversions. I just spent the last three weeks pulling my hair out over 800 milliseconds of silence.

Here is why that tiny gap almost killed my product, and what it took to fix it.

The "Rambling" Problem

I am building Akalan AI, a flight simulator for AI-led tech interviews. The core premise is that candidates need to practice speaking under actual pressure, not just read text-based question banks.

During our early beta tests, I noticed a massive drop-off in user engagement during the mock interviews. When I reviewed the session data, the issue was glaringly obvious: the AI was taking about 2.5 seconds to respond after the user finished speaking.

In a normal text-based chat window, 2.5 seconds is incredibly fast. In a live voice conversation, it is an eternity.

That silence caused candidates to panic. They assumed the AI had not heard them, or that their answer was insufficient. To fill the awkward void, they would start rambling, completely destroying the structured, concise answers they had just delivered. The latency was ruining the exact muscle memory I was trying to help them build.

Dissecting the Pipeline

To fix this, I had to tear down the entire voice processing architecture. A conversational AI pipeline has three distinct bottlenecks:

• Speech-to-Text (STT): Waiting for the user to stop speaking, capturing the audio, and transcribing it.

• LLM Processing: Feeding the transcript into the model to generate the interviewer's next question or reaction.

• Text-to-Speech (TTS): Converting that generated text back into a human-sounding voice.

Waiting for all three steps to complete sequentially was what caused the lag. It felt like talking to someone on a satellite phone in the 1990s.

How We Shaved Off the Milliseconds

I stopped building new features and dedicated three full weeks to optimizing this exact flow. Here is what actually moved the needle:

• Chunk Streaming: Instead of waiting for the LLM to generate the entire response, we started streaming the first few generated tokens directly into the TTS engine. By the time the AI finishes speaking its first sentence, the second sentence is already generated.

• Conversational Fillers: We optimized the system prompt to start responses with natural filler words ("Right," "I see," "Interesting point"). These tiny words render almost instantly, buying the system precious milliseconds to generate the heavy lifting of the actual follow-up question.

• Aggressive Endpointing: We tightened the silence detection threshold. The system now recognizes the end of a candidate's sentence much faster, cutting out dead air before the processing even begins.

The Result

We got the conversational latency down to a level where the AI interviewer actually feels like a sharp, fast-paced corporate screener. The dynamic shifted entirely. Users stopped rambling to fill the silence and started delivering confident, executive-level answers.

It was a brutal three weeks of backend optimization that no user will ever explicitly "see," but it is the exact feature that makes a flight simulator actually work.

Has anyone else here had to completely halt frontend feature development to fix a hidden infrastructure bottleneck?