> ## Documentation Index
> Fetch the complete documentation index at: https://docs.medisync.me/llms.txt
> Use this file to discover all available pages before exploring further.

# Messages

> Streaming audio frames, control frames, and the transcript, ack, backpressure, and status frames

## Overview

After [`session_ready`](/websocket/session-config#session-ready), the wire is a two-way stream:

* **Client → server:** binary audio frames, plus JSON **control** frames (`end_utterance`, `close`).
* **Server → client:** JSON `transcript` frames, plus `ack`, `backpressure`, `status`, and `error` frames.

Every JSON message has a `type` field. Branch on it and ignore any message type or field you do not recognize.

## Streaming audio (binary frames)

Send raw audio as **binary** WebSocket frames. Frames may be **any length** — the server re-frames the stream internally, so you do not need to align to segment boundaries. Send the encoding, sample rate, and channel count you declared in your config.

<CodeGroup>
  ```javascript JavaScript (browser) theme={null}
  // After session_ready, capture mic audio, convert to 16-bit PCM, and send.
  function startMicrophone(ws) {
    navigator.mediaDevices.getUserMedia({
      audio: { sampleRate: 16000, channelCount: 1, echoCancellation: true, noiseSuppression: true }
    }).then((stream) => {
      const ctx = new AudioContext({ sampleRate: 16000 });
      const source = ctx.createMediaStreamSource(stream);
      const node = ctx.createScriptProcessor(4096, 1, 1);
      node.onaudioprocess = (e) => {
        const f32 = e.inputBuffer.getChannelData(0);
        const pcm = new Int16Array(f32.length);
        for (let i = 0; i < f32.length; i++) {
          pcm[i] = Math.max(-1, Math.min(1, f32[i])) * 0x7fff; // float32 -> int16
        }
        if (ws.readyState === WebSocket.OPEN) ws.send(pcm.buffer); // binary frame
      };
      source.connect(node);
      node.connect(ctx.destination);
    });
  }
  ```

  ```python Python (asyncio) theme={null}
  import asyncio, json, websockets

  async def stream(ws, pcm_source):
      # pcm_source yields raw pcm_s16le bytes (mono, 16 kHz).
      async for chunk in pcm_source:
          await ws.send(chunk)               # binary frame
      await ws.send(json.dumps({"type": "end_utterance"}))  # flush the tail
      await ws.send(json.dumps({"type": "close"}))          # end the session
  ```
</CodeGroup>

## Control frames (client → server)

Two JSON text frames let you control the session:

```json theme={null}
{ "type": "end_utterance" }
```

Forces the server to finalize (flush) whatever audio is currently buffered, without ending the session. The server responds with an [`ack`](#ack) confirming the durable high-water mark. Useful for push-to-talk, or before you pause.

```json theme={null}
{ "type": "close" }
```

Ends the session cleanly. The server finalizes any remaining audio and closes the socket. You may also simply close the WebSocket.

## transcript (server → client)

The server sends one `transcript` frame per **finalized segment** of speech.

```json theme={null}
{
  "type": "transcript",
  "seq": 12,
  "segment": { "start": 30.20, "end": 34.85 },
  "language_detected": "de",
  "diarization_enabled": false,
  "text": "patient reports intermittent chest pain",
  "words": [
    { "w": "patient", "start": 30.20, "end": 30.55 },
    { "w": "reports", "start": 30.55, "end": 31.02 }
  ],
  "processed_until": 34.85
}
```

<ResponseField name="type" type="string">
  Always `"transcript"`.
</ResponseField>

<ResponseField name="seq" type="integer">
  Monotonically increasing sequence number for this session. Use it to order and de-duplicate segments (helpful across a reconnect).
</ResponseField>

<ResponseField name="segment" type="object">
  The finalized segment's absolute time span: `{ "start": <seconds>, "end": <seconds> }`. Includes any `timeline_offset_seconds` from your config.
</ResponseField>

<ResponseField name="language_detected" type="string | null">
  The ISO 639-1 language detected for this segment (for example `"de"`), or `null` if undetermined.
</ResponseField>

<ResponseField name="diarization_enabled" type="boolean">
  Whether speaker separation is on for this session. When `true`, a `speakers` array is also present (see below).
</ResponseField>

<ResponseField name="text" type="string">
  The transcribed text for the segment.
</ResponseField>

<ResponseField name="words" type="array">
  Per-word timings. Each entry is `{ "w": <word>, "start": <seconds>, "end": <seconds> }`, on the same absolute timeline as `segment`.
</ResponseField>

<ResponseField name="speakers" type="array">
  **Only present when `diarization_enabled` is `true`.** Per-speaker turns within the segment — see [Diarization](#diarization-speakers).
</ResponseField>

<ResponseField name="processed_until" type="number">
  The absolute second up to which the server has consumed audio. This is your reconnection high-water mark — see [Reconnection](/websocket/reconnection).
</ResponseField>

<Note>
  All times (`segment`, `words`, and speaker turns) are on one **absolute** timeline that already includes your `timeline_offset_seconds`. You never need to add the offset yourself.
</Note>

<Tip>
  The server may include additional advisory fields on a `transcript` frame. Treat any field not documented here as opaque and ignore it, so your integration stays forward-compatible.
</Tip>

### Handling transcripts

<CodeGroup>
  ```javascript JavaScript theme={null}
  ws.onmessage = (event) => {
    // Ignore any binary frames the server does not send in normal operation.
    if (typeof event.data !== "string") return;
    const msg = JSON.parse(event.data);

    switch (msg.type) {
      case "transcript":
        lastProcessedUntil = msg.processed_until;      // track for reconnection
        appendCaption(msg.seq, msg.segment, msg.text); // render
        break;
      case "ack":
        lastProcessedUntil = msg.processed_until;
        break;
      case "backpressure":
        throttleSendRate();                            // slow down
        break;
      case "status":
        console.log("status:", msg.state);
        break;
      case "error":
        console.error(msg.code, msg.message);
        if (msg.fatal) ws.close();
        break;
    }
  };
  ```

  ```python Python (asyncio) theme={null}
  async def receive(ws, state):
      async for message in ws:
          if isinstance(message, (bytes, bytearray)):
              continue
          msg = json.loads(message)
          t = msg.get("type")
          if t == "transcript":
              state["processed_until"] = msg["processed_until"]
              print(f"[{msg['segment']['start']:.2f}] {msg['text']}")
          elif t == "ack":
              state["processed_until"] = msg["processed_until"]
          elif t == "backpressure":
              state["throttle"] = True
          elif t == "status":
              print("status:", msg["state"])
          elif t == "error":
              print("error:", msg["code"], msg["message"])
              if msg.get("fatal"):
                  break
  ```
</CodeGroup>

## Diarization (speakers)

When you enable diarization, each `transcript` frame carries a `speakers` array of per-speaker turns for that segment:

```json theme={null}
{
  "type": "transcript",
  "seq": 7,
  "segment": { "start": 12.00, "end": 18.40 },
  "language_detected": "de",
  "diarization_enabled": true,
  "text": "how long has the pain lasted about three days",
  "words": [ { "w": "how", "start": 12.00, "end": 12.18 } ],
  "speakers": [
    { "speaker": "S1", "start": 12.00, "end": 14.60, "text": "how long has the pain lasted" },
    { "speaker": "S2", "start": 14.90, "end": 18.40, "text": "about three days" }
  ],
  "processed_until": 18.40
}
```

<ResponseField name="speakers[].speaker" type="string">
  A stable label for a distinct speaker within the session (for example `"S1"`, `"S2"`).
</ResponseField>

<ResponseField name="speakers[].start" type="number">
  Absolute start time of the turn.
</ResponseField>

<ResponseField name="speakers[].end" type="number">
  Absolute end time of the turn.
</ResponseField>

<ResponseField name="speakers[].text" type="string">
  The text attributed to that speaker for this turn.
</ResponseField>

<Warning>
  Speaker labels are stable **within a single session**. If you reconnect mid-recording, speakers recovered after the boundary may be relabeled — treat labels as session-scoped. See the [reconnection caveat](/websocket/reconnection#diarization-caveat).
</Warning>

## ack (server → client)

Sent in response to an `end_utterance` control frame (and may be sent periodically in future). It confirms the **durable high-water mark** — the same meaning as `processed_until` on a transcript frame.

```json theme={null}
{ "type": "ack", "processed_until": 34.85 }
```

<ResponseField name="processed_until" type="number">
  The absolute second up to which all audio has been consumed. A client can confirm its tail was processed before stopping or reconnecting. Content-free — clients that don't need it can ignore it.
</ResponseField>

## backpressure (server → client)

Sent when you are sending audio faster than the session can finalize it. Slow your send rate toward real time; if the pressure persists the server may drop some buffered segments to stay live.

```json theme={null}
{ "type": "backpressure", "level": "warn", "dropped_segments": 1 }
```

<ResponseField name="level" type="string">
  Severity of the signal (for example `"warn"`).
</ResponseField>

<ResponseField name="dropped_segments" type="integer">
  How many buffered segments were dropped to keep the session live (`0` when this is only an early warning).
</ResponseField>

<Note>
  A `backpressure` frame may carry additional advisory numeric fields; treat them as opaque. The actionable response is always the same: **send audio closer to real time.** See [Errors → Backpressure](/websocket/errors#backpressure-handling).
</Note>

## status (server → client)

Informational session-state notifications.

```json theme={null}
{ "type": "status", "state": "idle_timeout" }
```

<ResponseField name="state" type="string">
  A short state label. For example `"idle_timeout"` — the session received no audio for an extended period (roughly two minutes) and is being closed. Send audio, or `end_utterance`, periodically to keep a session alive.
</ResponseField>

## Next steps

<CardGroup cols={2}>
  <Card title="Reconnection" icon="rotate" href="/websocket/reconnection">
    Resume mid-recording with `processed_until` and `timeline_offset_seconds`.
  </Card>

  <Card title="Errors" icon="triangle-exclamation" href="/websocket/errors">
    Error frames, the full code table, and backpressure handling.
  </Card>
</CardGroup>