Skip to main content

Speech input and voice WebSocket endpoints

warning

The former bidirectional Playground voice mode is deprecated. The Playground microphone now provides speech-to-text input only.

The Microphone button uses the browser's speech recognition to fill the chat input. It does not start a voice dialog or speak responses aloud.

The separate /api/v1/voice/ws/ WebSocket handlers described below remain in the backend for applications with their own audio clients. Their presence in code is not a live-service availability check.

info

The former bidirectional Playground voice mode is not available in Orbis Studio Desktop. Browser speech input requires browser support and microphone permission.

Use the Playground microphone to dictate chat input, or build a separate client for the retained voice WebSocket endpoints if your application needs spoken responses.

Dictate a Playground message​

The Playground microphone requires a browser with SpeechRecognition or webkitSpeechRecognition support and permission to use your microphone. To run the flow after dictating, configure any credentials your flow requires.

  1. Open a flow with Chat Input and Chat Output, then open its Playground.
  2. Click the Microphone button. If prompted, allow microphone access.
  3. Speak, then click the microphone button again to stop recording. Review the transcription inserted into the chat input.
  4. Send the message to run the flow. The Playground displays the flow's response as text; it does not play it as audio.

Build a voice application with WebSocket endpoints​

The backend retains two WebSocket routes under /api/v1/voice. These are separate from the Playground microphone. Your own client must handle audio capture, audio playback, and the endpoint's event protocol. The handlers authenticate the Orbis Studio user and require access to an OpenAI API key; an ElevenLabs API key is optional for supported output voices.

  • /api/v1/voice/ws/flow_as_tool/$FLOW_ID uses an OpenAI Realtime session with the selected flow registered as a callable tool.
  • /api/v1/voice/ws/flow_tts/$FLOW_ID transcribes incoming audio, invokes the selected flow for each completed transcript, and streams audio generated from the flow's text response. Its flow execution can add latency.

Both routes accept an optional /$SESSION_ID suffix. Without one, the server creates a new session ID for each connection. Reusing an ID does not by itself guarantee persistent chat history; that depends on the flow's memory configuration.

For a flow-as-tool session, follow the OpenAI Realtime event protocol as implemented by the handler; the transcription-and-TTS route additionally sends flow progress and response.audio.delta events. Do not assume that the Playground uses either WebSocket route.

See also​