Overview
The Live Speech API turns a live audio stream into text in real time over a single, authenticated WebSocket. You open one connection, send a short JSON config handshake, then stream raw audio frames. The server replies with transcript frames as each segment of speech is finalized — each one carrying precise word-level timings and, optionally, per-speaker turns. The API is designed for clinical dictation and conversation capture, so it is built around three guarantees: medical-grade accuracy, a continuous absolute timeline that survives reconnects, and strict privacy (audio is never written to disk and transcript text is never logged).One socket, one session. Everything — config, audio, control, transcripts, acknowledgements and errors — flows over a single WebSocket. There is nothing else to poll.
How it works
Connect & authenticate
Open a WebSocket to the endpoint with a valid JWT passed as the
token query parameter. The token is verified during the handshake.Send a config handshake
Your first message is a single JSON
config frame that declares your audio format, an optional language hint, and how you want speech chunked.Receive session_ready
The server echoes the effective (validated and clamped) configuration and a
session_id. You may now start streaming.Stream audio
Send raw audio as binary frames of any length. Use JSON control frames to flush (
end_utterance) or end (close) the session.Key features
Real-time captions
Finalized segments arrive as you speak. Tune latency vs. density with a single
chunk.strategy setting.Word-level timings
Every transcript carries per-word start/end times on one absolute timeline.
Speaker separation
Enable diarization to receive per-speaker turns (for example clinician vs. patient).
Resilient reconnects
processed_until and timeline_offset_seconds let a dropped connection resume with no lost transcript and one continuous timeline.Multilingual
Automatic language detection across a wide set of European languages, including mid-conversation switches.
JWT-secured
Every connection is authenticated at the handshake with your JWT.
Connection at a glance
Endpoint| Property | Value |
|---|---|
| Transport | WebSocket (WSS, TLS) |
| Authentication | JWT in the token query parameter (verified at handshake) |
| First message | JSON config frame |
| Audio | Binary frames — raw 16-bit PCM (pcm_s16le) or wav |
| Server messages | JSON: session_ready, transcript, ack, backpressure, status, error |
| Protocol version | 2.0 |
Message flow
One session per user
Each user may have one active live session at a time. If a second connection is opened while a session is still active, it is rejected with anerror frame of code: "session_conflict" and the socket is closed. Close the previous socket before reconnecting.
This rule is what makes reconnection safe: because only one session can be live, a reconnecting client always resumes a single, unambiguous timeline. See Reconnection.
Privacy
Next steps
Connection
Endpoint, JWT auth via query parameter, handshake sequence, and close codes.
Session config
Every config field, defaults, clamping, and the
session_ready echo.Messages
Audio frames, control frames, and the full transcript schema.
Reconnection
Resume mid-recording with no lost transcript and one continuous timeline.
Audio requirements
Formats, sample rate, channels, and chunking guidance.
Errors
Error frame schema, the full code table, and fatal/close semantics.