Text-to-Translated-Speech

The tts_task command generates speech from text, with optional translation. Depending on your output_stream setting, generated audio is delivered in output_audio_data messages over WebSockets or as a WebRTC audio track. With WebRTC, command responses and text messages use the data channel. Follow the general API connection flow described here.

Refer to the Recommended settings for exact option values in the examples below.

The tts_task command

Send a WebSocket / WebRTC data channel message with the following structure:

{
  "message_type": "tts_task",
  "data": {
    "text": "Hello, how are you?",
    "language": "en",
    "translate_text": true
  }
}

Command acknowledgement

First, send set_task and wait for its task_status response with data.task_status: "running". You can then send tts_task. When the server accepts the text for processing, it sends the following acknowledgement:

{
  "message_type": "task_status",
  "data": {
    "command": "tts_task",
    "event": "queued",
    "task_status": "running",
    "transcription_id": "190983855fe3404e"
  }
}

The queued event confirms that the text is queued for processing. Translation and speech generation may still be in progress. Use data.transcription_id to match subsequent translation and audio messages to this command. The data.task_status field reports the current task state: running or paused.

If the command is rejected, the server sends an error response instead. These command responses are sent even when pipeline.allowed_message_types is empty.

Required fields

Optional fields

Text-to-Translated-Speech

Set translate_text to true in tts_task. The text will be translated into every target_language configured in the translations section, and TTS will be generated for each of them.

All output_stream and translations options are supported. Use the following set_task command structure:

{
  "input_stream": null, // set to `null`
  "output_stream": {/*...*/},
  "pipeline": {
    "transcription": null, // set to `null`
    "translations": [{/*...*/}, {/*...*/}], // translation and speech generation options for each language
    "allowed_message_types": [
      // Request translated text events; audio and command responses are sent separately.
      "translated_transcription"
    ]
  }
}

Multiple target languages

When multiple languages are configured in the translations section, the system returns translations and TTS for all configured languages simultaneously.

Text-to-Speech without translation (deprecated)

Set translate_text to false in tts_task or omit the field. All standard task options are supported — see the Recommended settings.

{
  "input_stream": null, // set to `null`
  "output_stream": {/*...*/},
  "pipeline": {
    "transcription": null, // set to `null`
    "translations": [{/*...*/}, {/*...*/}], // speech generation options for each language
    "allowed_message_types": [] // Receive audio and command responses without optional text events.
  }
}

Requirements in this mode:

• language must be one of the supported languages.
• language in tts_task must equal one of the target_language values in translations to apply speech settings (this will change in future versions).
• You can configure multiple languages in the translations section, but a separate tts_task message with the corresponding language is required for each one.