The challenge
The idea sounded simple. A candidate clicks a link, and the AI says hello. The candidate has a short screening chat. In practice, a voice call falls apart quickly if it does not feel like talking to a person. If the AI waits too long to reply, the candidate starts repeating themselves. If the AI talks over the candidate, the candidate becomes annoyed. If the audio stutters, many candidates simply close the tab. Then NexJob has nothing to score.
We also set ourselves some rules. Recordings of a candidate's voice count as personal data, so we did not want that audio passing through any other company's servers. We did not want a bill that grows by the minute either. And we had to assume the worst about networks: candidates call in from home routers, locked-down office Wi-Fi and phone hotspots, and many of those will not allow two machines to talk directly.
There was another problem underneath all this. The browser records at 48 kHz. Our speech detector wants 16 kHz. The AI talks back at 24 kHz. If one of those conversions is slightly wrong, the candidate hears it immediately: voices come out slowed down, crackly or repeating the same fragment.
Our approach
A call has two jobs to do, so we gave it two routes. Setup messages go over a WebSocket, through Nginx, into the FastAPI backend. The sound itself is encrypted (SRTP). It goes from the browser to our own coturn relay, then into an aiortc connection that lives inside the backend. From there, we clean it up, check whether anyone is speaking and pass it to the AI. The AI's reply makes the same trip in reverse.
Setting up a call
Before anything else, the browser asks the API for a TURN login that is valid for only one hour. Then the browser opens the WebSocket and sends an SDP offer, which is basically a note saying which audio formats it can handle. aiortc replies with an answer. Browsers tend to send their network route suggestions (ICE candidates) quickly, sometimes before the API has read the offer. We hold those suggestions in a queue and apply them once the offer is received, so no route is lost.
Running our relay
coturn sits in a container on the host's network. Port 3478 handles UDP and TCP; port 5349 handles TLS. That second port is the one that saves calls on corporate networks, because to a firewall it looks a lot like normal HTTPS. We point it at the certificate that the domain already has from Let's Encrypt and give it ports 49152 to 65535 for relaying audio.
There is no shared password anywhere. coturn checks each login against a secret it shares with the API. The username is an expiry time plus an ID, and the password is that username signed with HMAC-SHA1, which is the usual TURN REST approach. After one hour, the login expires. On the server side, we stamp the call's ID into the username, so if something odd appears in the relay logs, we can tell which interview it came from.
Making the backend a caller
We skipped a separate media server. Instead, the backend joins the call itself through aiortc. It takes the candidate's Opus audio and converts it into 16 kHz mono PCM. Going the other way, a custom track takes the AI's 24 kHz voice and plays it to the browser at 48 kHz, one 20 ms frame at a time.
Honestly, this is where most of the debugging time was spent. Early on, the AI's voice sounded slow and distorted. We discovered that PyAV pads its buffers, and we were copying that padding together with the real audio samples; now we copy only the real audio. Next, we encountered crackling. We had been creating a new resampler for every audio chunk, which discarded a few samples at each seam. Now each stream uses a single resampler for the entire call. The strangest bug was a loop where the AI repeated its last syllable endlessly. When our audio queue was empty, aiortc simply resent the previous frame. We now give it 20 ms of silence instead. We also time every frame against a monotonic clock, so playback does not slowly drift out of sync.
Deciding who's talking
This part decides whether the interview feels human or robotic, so we wanted to see and change every number. Our detector is small: it measures how loud each chunk of audio is (its RMS energy) and steps through the states shown in Figure 3.
| Setting | Value | Why |
|---|---|---|
| Start threshold | RMS 3850 | Clears normal room noise, even after we boost quiet Bluetooth mics by 1.8x |
| End threshold | RMS 3800 | Set a touch lower, so someone trailing off mid-word doesn't lose their turn |
| Start time | 40 ms | Filters out clicks and taps but still catches the first syllable |
| Preroll | 200 ms | We also send the bit of audio just before speech was detected, so nothing gets chopped |
| End silence | 800 ms | Gives people a moment to think before the AI jumps in |
| Max turn | 45 s | Stops a noisy room from keeping one turn open forever |
| Barge-in level | RMS 4500 | To interrupt the AI, you have to be clearly louder than background chatter |
If the candidate starts talking while the AI is mid-sentence, we discard whatever AI audio is still queued, and the AI goes quiet, just like a person would. We also have a safety check. If a callback ever gets lost, the microphone reopens on its own three seconds after the AI should have finished, so no one stays muted for the rest of their interview. Little noises such as "hmm" or "uh" that appear in transcripts are ignored.
Keeping the server safe
A single worker handles at most 20 calls. If a call does not start within 20 seconds, we drop it. Each call ends at 180 seconds plus a little grace time. The audio queues are capped at 500 and 1,000 chunks, so a hung AI stream cannot slowly consume the server's memory. If the live AI connection fails partway through, we switch to a slower REST mode that starts from the last 5 seconds of buffered audio.
Tradeoffs we made on purpose
Owning every piece meant we kept running into choices. Here is what we picked and the price we paid for it.
| Decision | What we got | What it cost us |
|---|---|---|
| coturn on the same host as the API | Audio never leaves our server, and there is no per-minute relay fee | Certificates, ports and monitoring are our job now, and users far from our region experience slightly more latency |
| aiortc inside the API process | One Python codebase, nothing extra to deploy | Audio work competes with API requests for CPU, so we cap calls per worker |
| Loudness-based speech detection | Easy to read, easy to tune, close to free to run | A loud room can fool it, and the numbers require tuning on real calls |
| 800 ms of silence to end a turn | People get room to think | The AI answers a little later than it would with a 300 to 500 ms setting |
| Padding gaps with 20 ms of silence | No loops or drifting audio | You may hear a few milliseconds of nothing when the AI is slow |
| TURN logins that expire after an hour | A leaked login stops working by itself, and coturn keeps no user list | The API machine and the relay need their clocks in sync |
| A hard cap on calls per worker | CPU and memory stay predictable | Handling more calls means adding workers or machines |
Results
Once a call connects, the AI begins speaking within three seconds. After that, the AI keeps pace with the candidate the way a person would. Nobody has to install anything. Even candidates on networks that block direct UDP can get through, thanks to the TLS relay on port 5349.
Everything between the candidate's microphone and the AI is owned by us: the signaling, the relay, the resampling, the turn detection and the playback. The interviewer waits through real pauses, ignores clicks and filler words, and stops when someone interrupts. Because every one of those behaviours is a value in a single config file, adjusting the settings for a new country or a different headset is simply a settings change and a redeploy.
What we learned
Live voice turned out to be a game of details. Buffer padding, resampler state, frame timing and a few loudness numbers made the difference between a call that felt broken and a call that felt normal. Since we owned every layer, each of those problems appeared in our logs, and we could fix it in our own code. On a product where the conversation is the thing people pay for, we would make that call again.