Voice Translation Actions

Overview

A complete list of all actions available under the voice-translation type. For connection and authentication, see Connection and Authentication; for response event formats, see Response Events.

Table of Contents

1. start - Start Voice Translation
2. config - Configure Terminology / Correction Rules
3. audio - Send Audio
4. pause - Pause Translation
5. resume - Resume Translation
6. stop - Stop Translation
7. retranslate - Retranslate a Single Sentence
8. switch_language - Switch Language
9. set_name - Set Recording Name
10. rename_speaker - Globally Rename a Speaker
11. reassign_speaker - Change the Speaker of a Single Sentence
12. merge_speakers - Merge Speakers
13. tts_play - Play TTS
14. tts_stop - Stop TTS
15. tts_mode - Switch TTS Mode
16. set_tts - Two-Way Translation TTS Settings
17. start_speaking - Start Speaking (Manual Mode)
18. stop_speaking - Stop Speaking (Manual Mode)
19. switch_conversation_mode - Switch Conversation Mode
20. set_speaker_language - Set Speaker Language
21. set_speaking_speed - Change Speaking Speed During Recording
22. add_channel - Add a Channel (Multi-Channel)
23. remove_channel - Disable a Channel (Multi-Channel)
24. set_channel_language - Change a Channel's Language (Multi-Channel)
25. set_summary - Change Summary Settings During Recording
26. broadcast_go_live - Switch to the Live Phase
27. broadcast_announcement - Send an Announcement
28. set_standby_message - Set the Standby Phase Message

start - Start Voice Translation

Note: Do not send glossaries inside start. If the start payload carries terminology, fuzzy_correction, or translation_dict, those fields are ignored (start itself still succeeds) and a config_ignored_in_start warning is returned, with details.ignored_fields listing what was dropped. Always send glossaries through the config action instead.

Description

Start a new voice translation session and begin processing audio according to the configured parameters.

Request Parameters

options Sub-fields

options is the speech recognition options object; all fields are optional and use their respective defaults when omitted.

Note: These options apply to STT sentence segmentation; multi-speaker mode (multi_speaker) does not currently apply speaking_speed.

speaking_speed levels

A sentence ends once silence lasts longer than the threshold. A longer threshold tolerates pauses within a sentence (a sentence is less likely to be split in two), but each sentence appears later.

Request Example (Basic)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": false,
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "options": {
      "speaking_speed": "normal",
      "profanity_handling": "mask"
    }
  }
}

Request Example (Multi-Language Translation, v1.6.7)

Specify multiple languages in translation_languages (up to 12) to translate into several languages at once. This applies to transcribe and broadcast (for two-way translation the language is set by the server and is always a single language; record does not support translation as of v1.7.0):

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US", "ja-JP", "ko-KR"],
    "realtime_translation": true,
    "type": "transcribe",
    "summary_template": "meeting"
  }
}

How results arrive: each language returns its own independent result event (same sid, with a single language key inside translations). Multiple languages are never merged into one event. Clients must accumulate translations by "sid + language code" instead of overwriting. Completion order across languages is not fixed (translations run in parallel), and if one language fails the remaining languages are still delivered (the failed language additionally receives an error event whose details.translation_language identifies it).

Billing reminder: translation is billed per language starting from the second language (see the Pricing Guide). Specifying N languages is billed as N languages.

Request Example (Initial Default Name)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "name": "Product Planning Meeting"
  }
}

Recording Name Rules

Note: The name in start is an initial default name; the system may still override it when the session ends. If you need a fixed name, use set_name.

Default name formats (fixed English):

N is the sequential number of recordings of the same type for that user. Name priority: user > llm > default. Once the user sets a name, the system will not override it when the session ends.

Request Example (with TTS)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": true,
    "type": "transcribe",
    "tts_enabled": true,
    "tts_language": "en-US",
    "tts_voice": "en-US-JennyNeural",
    "tts_mode": "sync"
  }
}

Request Example (Two-Way Translation Mode - Automatic Detection)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "active_language": "zh-TW",
    "audio_format": "pcm",
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Request Example (Two-Way Translation Mode - Manual Mode)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "conversation_mode": "manual",
    "audio_format": "pcm",
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Special rules for two-way translation mode:

speakers field description:

conversation_mode description:

Successful Response

After a successful start, a session_started event is returned, containing the complete initial session information. For real-time recording, the first minute is deducted first, and the event is returned once that deduction completes (see the Pricing Guide).

General recording (transcribe / conversation / record):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "task_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "transcribe",
    "recognition_mode": "single",
    "message": "Speech recognition started",
    "resume_token": "L0VBAwIy... (43 chars)",
    "resume_grace_seconds": 45,
    "server_time": 1749550000000
  }
}

Broadcast mode (broadcast):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "task_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "broadcast",
    "recognition_mode": "multi_speaker",
    "phase": "standby",
    "viewer_count": 0,
    "queue_count": 0,
    "peak_viewers": 0,
    "total_viewers": 0,
    "message": "Speech recognition started",
    "resume_token": "L0VBAwIy... (43 chars)",
    "resume_grace_seconds": 45,
    "server_time": 1749550000000
  }
}

For response field descriptions, see the session_started event.

Recording Type Descriptions

Broadcast Mode Description (type: "broadcast")

In broadcast mode, the language settings are obtained automatically from the broadcast channel settings and do not need to be sent in the WebSocket message.

Required parameters:

Optional parameters (override broadcast channel settings):

Automatically configured parameters (can be omitted):

• transcription_languages: read automatically from the broadcast settings
• translation_languages: read automatically from the broadcast settings
• realtime_translation: always enabled in broadcast mode, and false is treated as true; translation is always billed at the real-time translation rate
• summary_template: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)
• summary_language: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)

Both the host and viewers receive interim translations: a sentence's translation first arrives with is_final: false, followed by the finalized version with is_final: true. Overwrite the display by sid plus language; to show only finalized translations, skip is_final: false.

The recording name does not reuse the channel name. If start omits name, a name such as Broadcast #1 is generated; naming otherwise works the same as for other types, see Recording Name Rules.

Broadcast phase description:

Purpose of the standby phase: Lets the host run STT/translation warm-up tests before going live, confirming the equipment works before switching to the live phase.

The standby phase has a time limit (30 minutes by default): when the accumulated standby time reaches the limit, the session ends automatically, with a warning about 2 minutes before. See Standby Time Limit.

broadcast_phase accepts only lowercase standby and live: an empty string is treated as live, and any other value is rejected (invalid_parameter).

Broadcast mode request example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm"
  }
}

Broadcast mode request example (standby phase + override summary template):

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "broadcast_phase": "standby",
    "standby_message": "The talk is about to begin, please wait...",
    "summary_template": "lecture"
  }
}

Summary template priority: the value passed in the WebSocket start > the default set when creating the broadcast channel. If neither is set, no summary is generated automatically.

Broadcast mode TTS settings (tts_config):

Use the tts_config parameter to specify which translation languages should produce TTS audio for viewers.

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "tts_config": {
      "en-US": {
        "voice": "en-US-JennyNeural",
        "speaking_rate": 1.0
      },
      "ja-JP": {
        "voice": "ja-JP-NanamiNeural",
        "speaking_rate": 1.0
      }
    }
  }
}

Note:

• The TTS language must be a valid language in translation_languages; invalid languages are ignored automatically
• The host (WebSocket) does not receive TTS audio; only SSE viewers receive the tts_ready event
• TTS is sent only during the live phase; it is not sent during the standby phase

Multi-Channel Mode Description (recognition_mode: "multi_channel")

A single recording session takes input from multiple physical microphones at the same time (one per person), and speaker identity is determined by the channel — whichever microphone captured the audio identifies the speaker; the system performs no speaker inference. Suitable for meetings, round-table discussions, and other settings where every participant has a dedicated microphone (v1.10.0).

There are two sub-modes (channel_mode):

Scope and hard limits:

channels field description:

Language consistency rule (per_channel): the union of all channel languages (deduplicated) must exactly match the session-level transcription_languages; otherwise channel_language_mismatch is returned (details lists both language sets so you can reconcile them).

Multi-channel request example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "transcribe",
    "recognition_mode": "multi_channel",
    "channel_mode": "per_channel",
    "transcription_languages": ["zh-TW", "ja-JP"],
    "translation_languages": ["en-US"],
    "audio_format": "pcm",
    "summary_template": "meeting",
    "channels": [
      { "channel_id": 1, "speaker_name": "Main Speaker", "transcription_languages": ["zh-TW"] },
      { "channel_id": 2, "speaker_name": "Manager Wang", "transcription_languages": ["zh-TW"] },
      { "channel_id": 3, "speaker_name": "Sato", "transcription_languages": ["ja-JP"] }
    ]
  }
}

Successful response: the data of session_started includes channel_mode and channels[] (each entry with channel_id, speaker_name, transcription_languages, status; transcription_languages is not present in shared mode). Each channel starts with status preparing and transitions to ready via a channel_status event once that channel recognizes its first sentence.

Note: Always check that session_started includes channel_mode: this is the signal that the server actually started in multi-channel mode. If the response has no channel_mode, the server does not support multi-channel — channels[] was ignored and the session is actually running single-channel.

Other multi-channel behavior:

• Every audio frame must include channel_id (see audio)
• During recording you can add and disable channels dynamically via add_channel / remove_channel, and change a single channel's language via set_channel_language; set_speaking_speed also supports multi-channel (see that section)
• rename_speaker works normally (a channel can be renamed before it is first used). The new name must not duplicate another channel's name (including the names set for each channel in start); duplicates return speaker_name_duplicate. reassign_speaker and merge_speakers do not apply to multi-channel recordings (speaker identity is determined by the channel, so there is nothing to reassign) and return speaker_op_not_allowed_multi_channel
• pause / resume: while paused, the recording file is still saved but no transcript is produced; after resuming, speech from the paused period is transcribed retroactively (timestamps reflect the actual speaking time). Retroactive transcription has a limit — a trailing 60 seconds shared across the whole session (under per_channel, split evenly across channels when there are several; under shared, the whole last 60 seconds, for all channels together, with speakers labeled); anything beyond that is kept only in the audio file. A sentence cut off at the moment of pausing may not appear in the transcript (same as single-channel recording); when it is indeed not kept, resuming also sends segment_discarded (reason: "resumed") to identify it
• The origin of result events includes channel_id and speaker_id (format channel_{N}); every transcript sentence also carries channel_id (see Response Events)
• Unlimited plans must include the multi-channel feature; a plan may also cap the number of channels — exceeding the cap returns plan_feature_not_allowed on the spot at start / add_channel (details.field is max_stt_streams)
• Billing: speech recognition plus speaker diarization as the base; per_channel adds a surcharge based on the number of active channels at the time, while shared is always counted as 1 channel — see the Pricing Guide

Shared mode (v1.21.0):

Request example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "transcribe",
    "recognition_mode": "multi_channel",
    "channel_mode": "shared",
    "transcription_languages": ["zh-TW", "en-US"],
    "audio_format": "pcm",
    "channels": [
      { "channel_id": 1, "speaker_name": "Host" },
      { "channel_id": 2, "speaker_name": "Guest A" },
      { "channel_id": 3, "speaker_name": "Guest B" }
    ]
  }
}

Client requirements:

• Send only one channel at a time: send audio only on the current speaker's channel. If several channels are sent at once, their audio is queued into the same recognition stream in the order received and the transcript becomes garbled (billing is not affected).
• Keep sending silence: when nobody is speaking, keep sending silence on the current channel (one frame every 100ms recommended); do not stop sending.
• pcm only (same as the common multi-channel rule).
• Languages are shared across the session: channels must not specify transcription_languages (returns channel_language_not_allowed); the language cannot be changed during the recording (set_channel_language returns channel_language_not_allowed).
• The first channel cannot be removed: the first entry in channels[] carries the recognition for the whole session, and remove_channel returns channel_remove_not_allowed; other channels can be removed.

Channel status:

• The status of the other channels follows the first channel: when the first channel turns ready, all channels turn ready together; when the first channel prepares recognition again (resume after pause, resume after disconnect, automatic reconnection), all channels return to preparing together, and each channel receives its own channel_status event with the same reason as the first channel. After resuming from a disconnect, preparing is reported in the resume_ok snapshot rather than by a separate event.
• A channel added with add_channel takes on the first channel's current status directly; removed follows each channel's own status.
• When the first channel turns error, the other channels turn error as well (with the same reason); when the first channel recovers, they recover together.
• The channels[] snapshots in session_started and resume_ok follow the same rules; channel entries do not carry transcription_languages.

Known limitations:

• When the gap between speakers is shorter than about 0.8 seconds, the words of the two people may be merged into one sentence labeled with only one speaker.
• Speakers are determined by channel labeling, which is highly accurate when people take turns; reassign_speaker and merge_speakers do not apply, so this cannot be corrected afterward.

TTS Playback Mode Description

Automatic End After a Long Silence

A recording that goes a certain length of time without recognizing any text (including interim results that are not final yet) ends automatically, so a recording someone forgot to stop does not keep being billed.

• The default threshold is 900 seconds (15 minutes) and can be changed per session with silenceTimeoutSeconds. About 2 minutes before the end, a warning is sent once.
• Any recognized text restarts the count from 0; resume and start_speaking also restart it.
• The count does not run in these cases:
  • While paused: the count restarts from 0 after resuming. A paused recording is still billed; to stop billing, end the recording.
  • While a disconnected session waits to be resumed: after resuming, the count continues from where it was.
  • Broadcasts (type: "broadcast"): never end for lack of speech. The standby phase has its own time limit; see the Broadcast Guide.
• Multi-channel: counts only when no channel recognizes any text; a single silent channel does not end the recording.
• Conversation manual mode: speech recognized while the speak button is not pressed also restarts the count.

Values of silenceTimeoutSeconds (top level of the start data, optional)

• The value must be a JSON integer: strings (for example "900"), decimal notations (for example 900.0 or 1e3), and booleans are all rejected.
• The parameter name is silenceTimeoutSeconds; sending silence_timeout_seconds is rejected (details.field is silence_timeout_seconds), not ignored.
• A valid value has no effect on broadcasts, but an invalid value is still rejected.
• With a threshold of 120 seconds or less, no warning is sent; the recording simply ends when the time is up.
• Resuming a disconnected session keeps the original setting; send it again when starting a new recording.

Events you receive

1. stt_silence_warning (an error event with severity: "warning"): a warning; the recording continues. details.silenceSeconds is how long the silence has lasted and details.remainingSeconds is how many seconds remain. Do not treat it as the end of the recording; recognized text or resuming the recording restarts the count.
2. stt_silence_timeout (severity: "fatal"): the recording has ended automatically. details.silence_seconds is the threshold in seconds.
3. Then status: "ended" and task_complete, in that order, just as when the client sends stop: the recording is saved and summarized as usual, and billing runs until the end.

{
  "type": "error",
  "data": {
    "error_code": "stt_silence_warning",
    "severity": "warning",
    "message": "No speech detected for a while; the recording will end automatically soon",
    "context": "stt",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-09-25T10:28:00.000Z",
    "details": {
      "silenceSeconds": 780,
      "remainingSeconds": 120
    }
  }
}

After stt_silence_timeout, stop sending audio. Each audio message that arrives afterwards gets a session_not_started reply, which you can ignore.

Error Codes

config - Configure Terminology / Correction Rules

Description

Send terminology, fuzzy-word correction rules, and translation dictionary settings before or during a recording. These settings can improve STT accuracy, fix homophone errors, and ensure translation consistency.

Terminology also drives homophone correction: When terminology is provided, the terms become the reference for homophone matching — any span in the transcript that sounds the same but is written differently is corrected back to the spelling of the term. Providing terminology alone is therefore enough to get correction; you do not need to list possible misspellings by hand.

For the full description of homophone correction (the reference table, the fact that it applies to Chinese only, common-word protection, and the pronunciation limits), see Terminology Guide → How Terminology Participates in Homophone Correction.

Limits

Per-block entry limits, length limits, and the matching error codes are collected in Terminology Guide → Limits.

The numbers are defaults; the limit actually in force can be tuned per environment — always treat max in the error response's details as authoritative rather than hard-coding the numbers.

Request Parameters

Note: At least one setting item must be provided.

Note: All or nothing: all three blocks are validated before any of them is applied. If any block fails validation, an error is returned and none of the three blocks is applied — the settings stay as they were.

For example, sending a valid terminology list together with more than 3000 translation-dictionary entries returns config_too_many_dict_entries, and the terminology does not take effect either. Fix the problem and resend the complete config; all three blocks replace their previous value wholesale, so resending does not stack on top of earlier settings.

Terminology Format (terminology)

Keyed by language code, with an array of terms as the value:

{
  "zh-TW": [
    { "term": "語者分離" },
    { "term": "WebSocket" }
  ],
  "en-US": [
    { "term": "diarization" }
  ]
}

Limit: Up to 500 terms across all languages combined in one config — not 500 per language. Exceeding the limit returns config_too_many_entries (with count and max in details).

Language scope: Terms apply only to recognition in the language they are registered under. Single-language situations — each multi-channel track, speaker diarization, and file import — use only that language's terms; multi-language transcription and conversation mode use the terms for the languages declared for the session. Terms registered under a language not used in this session have no effect, but still count toward the 500 combined total above.

These numbers are defaults: the limit actually in force can be tuned per environment; always treat max in the error response's details as authoritative.

Fuzzy-Word Correction Format (fuzzy_correction)

Note: This field usually does not need to be set manually — terminology already corrects misspellings that sound the same or nearly the same. Use it in these three cases:

1. The misspelling is itself an ordinary word and is therefore blocked by common-word protection — 晶圓 heard as 金元, for example
2. The misspelling sounds very different from the correct term, such as a foreign brand name recognized as a phonetically unrelated word
3. Misspellings in Japanese, Korean or English, which do not participate in homophone matching

When correct is Chinese, it also becomes a reference for homophone matching — homophone misspellings not listed in incorrect are corrected to correct as well. case_insensitive applies only to the literal matching of incorrect; it does not affect homophone matching.

Keyed by language code, with an array of correction rules as the value:

{
  "zh-TW": [
    { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] },
    { "correct": "IPEVO", "incorrect": ["ltfo"], "case_insensitive": true }
  ]
}

Supplying only the correct term: when correct is Chinese (contains Han characters), incorrect may be omitted entirely — the system matches by pronunciation, and spellings in the transcript that sound the same or nearly the same are corrected back to correct.

{ "fuzzy_correction": { "zh-TW": [{ "correct": "艾思通" }] } }

No misspellings need to be listed above: 愛思通, 愛時通, 愛司東 and 愛似通 are all corrected. Only spellings that sound quite different (愛自動, say) or have a different number of syllables (愛松) still need to be listed in incorrect.

Note: Both conditions must hold: the language must be Chinese (zh-TW, zh-CN, zh-HK and so on) and correct must contain Han characters. Otherwise incorrect remains required — omitting it in those cases would have no effect at all, and accepting it would leave you believing the setting took. List misspellings explicitly for Japanese, Korean and English.

Case sensitivity: case_insensitive is optional and defaults to false (exact-case matching). When set to true, every incorrect variant in that rule matches regardless of case. The flag is per rule — the same correct term can be split across several rules with different settings, for example making variants that cannot collide with ordinary words case-insensitive while keeping variants that could hit a personal name exact. It has no effect on Chinese rules (Chinese has no letter case).

Note: Enabling it widens the false-positive surface: if ivo is case-insensitive, the personal name Ivo is replaced too.

When the same incorrect variant appears in more than one rule: collisions are resolved on incorrect (the variant), not on correct.

• When several rules point at different correct terms, which one actually takes effect is not guaranteed; do not rely on any ordering (registration order included)
• The case flag resolves to strict wins — if any rule leaves case_insensitive off, that variant is matched with exact case

"Strict wins" is a deliberately conservative choice: it prevents a permissive rule elsewhere in your vocabulary from silently loosening a brand-name rule you explicitly set to strict.

Splitting one correct term across several rules is therefore safe, as long as their incorrect variants do not overlap. But if your data contains the same incorrect variant mapped to different correct terms, the later one is dropped with no warning — check for duplicate variants before sending.

Limit: Up to 4000 rules across all languages combined in one config. Exceeding it returns config_too_many_entries (with field: "fuzzy_correction", count and max in details).

This number is a default: the limit actually in force can be tuned per environment, so always treat max in the error response's details as authoritative. Do not hard-code the numbers in your integration — if you want to check before sending, read max and fill it back in. When you do hit a limit, details carries both count (what you sent) and max (the limit in force).

Language scope: A correction rule applies only to sentences in the language it is registered under — a rule under zh-TW will not alter an English sentence. When the sentence language cannot be determined, all rules are applied as a fallback (better to over-apply than to skip the sentence entirely). Homophone matching driven by terminology likewise follows the language each term is registered under.

Translation Dictionary Format (translation_dict)

Group entries by language code, giving each language its own dictionary:

{
  "en-US": [
    { "source": "語者分離", "target": "Speaker Diarization" },
    { "source": "晶圓", "target": "wafer", "case_sensitive": true }
  ],
  "ja-JP": [
    { "source": "語者分離", "target": "話者分離" }
  ]
}

Limit: Up to 3000 entries per language. Exceeding the limit returns config_too_many_dict_entries; the details in the response identify which language exceeded it.

Note: The entry count directly affects translation workload and cost. A single translation only carries entries whose source word actually appears in that piece of text, up to 100 of them; beyond that, longer source words are kept first. Also, the more entries there are, the smaller the share that is reliably honored — an inherent limit that a higher cap does not change.

Case sensitivity: case_sensitive is optional and defaults to false (case-insensitive). When set to true, the entry applies only where the source text matches source exactly, including case. The flag is per entry.

Note: The translation dictionary guides the model through prompting rather than literal substitution, so it is best-effort, not deterministic — the case flag is likewise a hint and is not guaranteed to be honored. Use fuzzy_correction when you need deterministic replacement.

Case-Flag Comparison

fuzzy_correction and translation_dict each have a case switch. Their field names are opposites, and so is the behavior their default value produces:

Both default to false, yet one means strict and the other means permissive. Do not share a single variable between them, and do not mirror one value onto the other — getting it wrong produces no error at all, only matching behavior opposite to what you intended.

Request Example (Recommended: terminology only)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離" },
        { "term": "CVD製程" },
        { "term": "wafer良率" }
      ]
    }
  }
}

Request Example (Full settings, with manual correction rules)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離" },
        { "term": "即時轉錄" }
      ]
    },
    "fuzzy_correction": {
      "zh-TW": [
        { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] }
      ]
    },
    "translation_dict": {
      "en-US": [{ "source": "語者分離", "target": "Speaker Diarization" }]
    }
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "config_updated",
    "updated": ["terminology", "fuzzy_correction", "translation_dict"],
    "message": "Settings updated"
  }
}

For response field descriptions, see the config_updated event.

Error Codes

Important: Your client must also listen for type: "error" messages — do not wait only for config_updated.

When the server rejects a configuration it sends a type: "error" message and notconfig_updated. An integration that waits only for config_updated will hang until its own timeout and appear as "the server never responded", even though the error was delivered and data.error_code already states the reason.

audio - Send Audio

Description

Send audio data to the server for speech recognition. The audio must be Base64-encoded before sending.

Request Parameters

Audio Format Requirements

PCM format (default):

WebM/Opus format:

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "audio",
    "payload": "Base64-encoded PCM audio data"
  }
}

Request Example (Multi-Channel, v1.10.0)

In multi-channel mode every frame must include channel_id, identifying which microphone the audio came from:

{
  "type": "voice-translation",
  "data": {
    "action": "audio",
    "channel_id": 2,
    "payload": "Base64-encoded PCM audio data"
  }
}

Multi-channel sending tips: send frames of roughly 100ms; keep sending on every channel even while nobody is speaking (silent audio). A single silent channel does not end the recording; see Automatic End After a Long Silence.

Error Codes

pause - Pause Translation

Description

Pause speech recognition processing. Audio received while paused is cached and processing resumes afterward.

Two-way translation is an exception: audio received while paused is not kept. For a sentence in progress at the moment of pausing, the server first waits for the recognizer to finish the last part (usually about 1 second, at most about 3 seconds), sends it with is_final: true, and only then returns status: "paused"; in manual mode, if the user is speaking, speaking is ended automatically and that sentence's final result is sent.

Billing continues while paused. See Pricing.

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "pause"
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "paused",
    "message": "Speech recognition paused"
  }
}

Error Codes

resume - Resume Translation

Description

Resume paused speech recognition processing.

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "resume"
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "live",
    "message": "Speech recognition resumed"
  }
}

Error Codes

If a sentence is in progress when this happens, it cannot be kept and is reported separately via segment_discarded (reason: "resumed"). In multi-channel mode each channel reports separately.

stop - Stop Translation

Description

Stop speech recognition and end the session. The server first waits for the recognizer to finish the last sentence (usually about 1 second, at most about 3 seconds) so that it is included in the transcript; if it does not arrive in time, the last interim result shown is used as that sentence. In two-way translation manual mode, if the user is speaking, that sentence's final result and translation are sent first. The system automatically uploads the audio file and transcript and generates a summary (if configured; when the available credits cannot cover the summary fee, no summary is generated and summary_error is sent instead).

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "stop"
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "ended",
    "message": "Speech recognition stopped"
  }
}

After stopping, once the audio file and transcript have finished uploading, you receive a task_complete event containing task_id (the Recording UUID).

Error Codes

Sending stop a second time returns this error rather than another success response. task_complete is sent only once, after the first successful stop, when the audio and transcript have finished uploading.

retranslate - Retranslate a Single Sentence

Description

Retranslate a specified sentence. This is useful when the source text has been corrected and the translation needs to be updated.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "retranslate",
    "sid": 1,
    "translation_languages": ["en-US"],
    "text": "The user's corrected source text"
  }
}

Successful Response

A translation event is returned (sharing the same schema as normal translation results), and the translation result includes is_retranslation: true:

{
  "type": "voice-translation",
  "data": {
    "action": "translation",
    "sid": 1,
    "translations": {
      "en-US": {
        "sid": 1,
        "text": "The new translation result",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

The data object also carries sid, identical to the sid inside each language of translations.

v1.5.6 documentation correction: Earlier documentation described retranslate returning action: "result", but on the wire it is actually action: "translation". If your client's dispatcher originally handled the result action, add a handler branch for the translation action.

Error Codes

If the translation service returns a specific failure code (for example llm_content_filtered when the content cannot be translated), that code is returned as-is instead of being wrapped in retranslate_llm_failed.

switch_language - Switch Language

Description

Switch or adjust translation languages during real-time translation. The behavior depends on the recording type and the number of translation languages:

• General mode, single language (translation_languages has 1 entry): replaces the translation target language and automatically batch-retranslates all already-translated sentences
• General mode, multiple languages (2 or more entries, v1.6.7): redefined as "add or remove a single language". The op parameter is required; omitting it returns a switch_language_op_required error
• Two-way translation mode (conversation): switches the STT source language (the spoken language); the translation target switches automatically to the other language
• Multi-channel mode (multi_channel, v1.10.0): disabled. Languages are bound to channels, so every form of switch_language (including op: "add" / op: "remove") returns multichannel_switch_language_not_allowed; to change a single channel's language, use set_channel_language instead

Request Parameters

Request Example (General Mode, Single-Language Replace)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "translation_languages": ["ja-JP"]
  }
}

Request Example (Multi-Language, Add a Language, v1.6.7)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "op": "add",
    "translation_languages": ["de-DE"]
  }
}

After a successful add: existing sentences are automatically backfilled with the new language (the response sequence is the same as single-language replace: language_switch_start → multiple batch_retranslation → language_switch_done), and subsequent sentences include the new language in real-time translation. The limit is 12 languages; exceeding it returns too_many_languages.

Billing reminder: after adding a language, billing reflects the new language count starting from the next billed minute.

Request Example (Multi-Language, Remove a Language, v1.6.7)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "op": "remove",
    "translation_languages": ["ko-KR"]
  }
}

After a successful remove, a translation_language_removed event is returned. Existing translations for that language are kept; subsequent sentences are no longer translated into it. At least 1 translation language must remain (removing the last one returns a switch_language_last_language error):

{
  "type": "voice-translation",
  "data": {
    "action": "translation_language_removed",
    "translation_language": "ko-KR",
    "translation_languages": ["en-US", "ja-JP"]
  }
}

translation_languages is an authoritative snapshot of the full translation-language set after removal; overwrite your local language set with it directly (see events.md).

Request Example (Two-Way Translation Mode)

Specify the switch target:

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "transcription_languages": ["en-US"]
  }
}

Automatic toggle (no parameters):

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language"
  }
}

Special behavior in two-way translation mode:

• Two-way translation mode uses automatic language detection, so you usually don't need to switch the language manually
• switch_language only updates the internal preference state
• After a successful switch, a language_switched event is returned (not the language_switch_start/done sequence)
• Switching to the same language returns a conversation_same_language warning

Response Sequence (General Mode)

After switching the language, you receive the following events in order:

1. language_switch_start: notifies that the switch has started

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_start",
    "translation_language": "ja-JP",
    "translation_languages": ["en-US", "ja-JP"],
    "total_segments": 15
  }
}

2. batch_retranslation (multiple): returns retranslation results sentence by sentence

{
  "type": "voice-translation",
  "data": {
    "action": "batch_retranslation",
    "sid": 3,
    "translations": {
      "ja-JP": {
        "sid": 3,
        "text": "今日はプロジェクトの進捗について話し合いましょう",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

3. language_switch_done: notifies that the switch is complete

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_done",
    "translation_language": "ja-JP",
    "translation_languages": ["en-US", "ja-JP"],
    "success_count": 15,
    "failed_count": 0
  }
}

Language-set sync: op:add and single-language replace emit identical events (both go through language_switch_start/done). These events carry translation_languages (the current full set of translation languages) — overwrite your local language set with it directly; do not infer append vs replace from the single translation_language (when only 1 language exists, an op:add expanding to a 2nd would be misread as a replace and drop the existing language).

Error Codes

set_name - Set Recording Name

Description

Set the name during a recording. After it is set, name_source flips to user, and the system will not override it when the recording ends (even if the LLM generates a summary name, it yields to the user-set name). For the full semantics and priority of name_source, see § Recording Name Rules above.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "set_name",
    "name": "Product Planning Meeting"
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "event": "name_set",
    "name": "Product Planning Meeting",
    "message": "Recording name set"
  }
}

Compatibility note: For backward compatibility, the set_name success response keeps action: "status" (unchanged). New clients should identify a successful set_name via event: "name_set" (together with the name field). Relying on action: "status" to detect set_name success is deprecated and may be removed in a future version.

Error Codes

rename_speaker - Globally Rename a Speaker

Description

In multi-speaker diarization mode (multi_speaker), globally rename a speaker. All sentences that use that speaker ID are updated in sync.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "rename_speaker",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang"
  }
}

Successful Response

Returns the speaker_renamed event:

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_renamed",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang",
    "affected_sids": [1, 3, 5, 8]
  }
}

Error Codes

reassign_speaker - Change the Speaker of a Single Sentence

Description

Change the speaker identity of a specific sentence, assigning the sentence to an existing speaker.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "reassign_speaker",
    "sid": 5,
    "target_speaker_id": "Guest-2"
  }
}

Successful Response

Returns the speaker_reassigned event:

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_reassigned",
    "sid": 5,
    "old_speaker_id": "Guest-1",
    "new_speaker_id": "Guest-2",
    "new_speaker_label": "Lisa Lee"
  }
}

Error Codes

merge_speakers - Merge Speakers

Description

Merge all sentences from one speaker into another. After merging, future recognition results from that speaker are also automatically converted to the target speaker. This applies within the current recognition pass only: after a connection recovers, speakers are re-identified and you need to merge again.

Difference from reassign_speaker

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "merge_speakers",
    "source_speaker_id": "Guest-2",
    "target_speaker_id": "Guest-1"
  }
}

Successful Response

Returns the speakers_merged event:

{
  "type": "voice-translation",
  "data": {
    "action": "speakers_merged",
    "source_speaker_id": "Guest-2",
    "target_speaker_id": "Guest-1",
    "affected_sids": [3, 5, 7]
  }
}

Error Codes

tts_play - Play TTS

Description

In async mode, manually play the TTS audio of a specified sentence. Repeated requests for the same sid are supported (replay).

Two-way translation mode (conversation): tts_play automatically synthesizes the translation in the appropriate language based on the voice settings in tts_config; you don't need to specify tts_language separately.

Request Parameters

Request Example (Single Sentence)

{
  "type": "voice-translation",
  "data": {
    "action": "tts_play",
    "sid": 5
  }
}

Request Example (Multiple Sentences)

{
  "type": "voice-translation",
  "data": {
    "action": "tts_play",
    "sid": 5,
    "length": 3
  }
}

Error Codes

A missing sentence or translation does not return error: If the starting sid does not exist, or the sentence has no translation in the target language, no error message is returned. A tts_error event is sent instead, with error set to sentence_not_found and translation_not_found respectively. When playing multiple sentences, a failing sentence is skipped and the rest still play.

tts_stop - Stop TTS

Description

Stop the currently playing TTS audio.

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "tts_stop"
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "TTS stopped"
  }
}

tts_mode - Switch TTS Mode

Description

Switch the TTS playback mode (synchronous/asynchronous) during a recording.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "tts_mode",
    "tts_mode": "async"
  }
}

Successful Response

Returns the tts_mode_changed event:

{
  "type": "voice-translation",
  "data": {
    "action": "tts_mode_changed",
    "tts_mode": "async"
  }
}

Error Codes

set_tts - Two-Way Translation TTS Settings

Description

During two-way translation mode (conversation), toggle TTS on/off or update the TTS voice settings mid-session. Available only under the conversation type.

Request Parameters

Request Example (Disable TTS)

{
  "type": "voice-translation",
  "data": {
    "action": "set_tts",
    "tts_enabled": false
  }
}

Request Example (Update TTS Voice)

{
  "type": "voice-translation",
  "data": {
    "action": "set_tts",
    "tts_enabled": true,
    "tts_config": {
      "en-US": { "voice": "en-US-GuyNeural", "speaking_rate": 1.2 }
    }
  }
}

Successful Response

Returns the tts_updated event:

{
  "type": "voice-translation",
  "data": {
    "action": "tts_updated",
    "tts_enabled": true,
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-GuyNeural", "speaking_rate": 1.2 }
    }
  }
}

Error Codes

start_speaking - Start Speaking (Manual Mode)

Description

In two-way translation manual mode (conversation_mode: "manual"), notify the system that the user has started speaking. From this moment, audio is sent to STT for recognition, and all recognition results accumulate into the same sentence (no automatic segmentation). If it is called again while already speaking, the system first ends the previous sentence (waiting for the last part to finish, then sending its final result) and starts a new one.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "start_speaking",
    "speaker": 1
  }
}

Successful Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Speaking started"
  }
}

If it is called again while already speaking, the final result of the previous sentence is sent as usual, followed by this same status.

Error Codes

stop_speaking - Stop Speaking (Manual Mode)

Description

In two-way translation manual mode, notify the system that the user has stopped speaking. The system first waits for the recognizer to finish the last part (usually about 1 second, at most about 3 seconds), then merges the recognition results accumulated during this period into one complete sentence, and translates it and synthesizes TTS. For languages that separate words with spaces (such as English), a space is added between parts automatically.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "stop_speaking"
  }
}

Successful Response

After speaking stops, the system sends a complete result event (containing origin and translations):

{
  "type": "voice-translation",
  "data": {
    "action": "result",
    "origin": {
      "sid": 1,
      "language": "zh-TW",
      "text": "The complete sentence merged from all recognition during this period",
      "is_final": true,
      "speaker_id": "Speaker-1",
      "start_time": "00:05"
    },
    "translations": {
      "en-US": {
        "sid": 1,
        "text": "The complete merged sentence from this speaking period",
        "is_final": true
      }
    }
  }
}

Error Codes

switch_conversation_mode - Switch Conversation Mode

Description

During two-way translation mode, switch between automatic detection mode (auto) and manual mode (manual). If the user is speaking during the switch, speaking ends automatically.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "switch_conversation_mode",
    "conversation_mode": "manual"
  }
}

Successful Response

Returns the conversation_mode_changed event:

{
  "type": "voice-translation",
  "data": {
    "action": "conversation_mode_changed",
    "conversation_mode": "manual"
  }
}

Error Codes

set_speaker_language - Set Speaker Language

Description

During two-way translation mode, change a specified user's language in real time. The system rebuilds the STT connection to adapt to the new language, and the translation target is updated automatically. Transcript content before the change keeps its original language, and timestamps continue counting without resetting.

Request Parameters

Request Example