The registry lists every model served this way with
"type": "realtime", dated versions included.
Gates work on every path. Put a gate’s id where the model goes (the URL’s model, Live’s session.start, Gemini’s setup, the transcription model, or the model a client secret is minted for); the gate’s model serves, and the session is logged under the gate. A model that speaks a different protocol from the path is refused with realtime_provider_not_served, and the message names the path to use.
OpenAI voice
A speech-to-speech conversation on OpenAI’s Realtime protocol. Use theopenai package, version 7.53 or later, with its ws peer dependency:
turn_detection: null you decide when a turn ends, as above. Leave it out and OpenAI’s voice detection answers on its own when the speaker pauses.
The WebSocket handshake answers with x-verlon-served-model (the model that serves, after any gate) and x-verlon-session-id (the session’s id in Verlon, for support).
From a browser
A browser can’t set anAuthorization header on a WebSocket, and your Verlon key must never reach it. Instead, your server mints a client secret, and the browser opens the session with it.
1
Mint a client secret on your server
POST https://api.verlon.ai/v1/realtime/client_secrets, the same shape as OpenAI’s endpoint. Minting needs a Verlon API key.2
Open the session in the browser
With the OpenAI SDK:Or with a plain WebSocket, offering the secret as a subprotocol:Then send audio and read the answer, as on the server.
- The session is already set up. It opens with the session the secret was minted with (
session.updatedarrives first), so the browser sends only audio. - It works once. A second connection with the same secret gets
401 invalid_client_secret. - It expires. The default is 10 minutes. Set
expires_after: { anchor: 'created_at', seconds }when minting to choose between 10 seconds and 2 hours. - It’s bound to its model or gate. Mint with a model id or a gate’s id, and name the same one, or none, when connecting. A secret minted for a gate opens the session through that gate.
- It’s checked like your key. Your key’s checks run again when the secret is used: if the key has been revoked or your account suspended since minting, the session is refused.
- It opens
/v1/realtimeonly: OpenAI voice, xAI voice and OpenAI transcription. Open OpenAI Live, Gemini Live and Mistral sessions from your server.
xAI voice
xAI’s voice agent,grok-voice-3.0, speaks OpenAI’s Realtime protocol at the same endpoint, so use the same client as OpenAI voice and name the xAI model:
- Wait for
session.updated. Send the audio after xAI confirms the session, as above. Audio sent before the session is applied may not be heard as you meant. - Voices are xAI’s, such as
Ara, not OpenAI’s. - It bills by time, not tokens. xAI reports no token usage, so an xAI voice session bills its per-minute rate for every second it’s open, silence included. Close the socket when the conversation ends.
- From a browser, mint a client secret with
model: 'grok-voice-3.0'and the session above, and open it as for OpenAI voice. The minted session is applied before the browser’s first message, so the browser can send audio as soon assession.updatedarrives.
OpenAI Live
gpt-live-1 holds a full-duplex conversation: it listens and speaks at the same time, so there are no turns to commit. It speaks its own protocol, which the openai package doesn’t yet wrap for WebSockets, so use any WebSocket client:
session.startcomes first. If it doesn’t arrive within 10 seconds, the session gets anerrorevent and closes. Wait forsession.startedbefore sending audio.- Send audio in real time. Audio faster than 1.2× real time, or a burst over 5 seconds, is dropped with an
input_audio_rate_limit_exceedederror. - Output audio never stops.
session.output_audio.deltastreams continuously, silence included, so play it as a stream rather than waiting for it to end. - End with
session.close. You getsession.closedwith the final usage, then the socket closes.
Gemini Live
Use Google’s@google/genai SDK with its base URL pointed at Verlon and your Verlon key as the API key:
- Set
apiVersion: 'v1beta'. From version 3.0,@google/genaiconnects onv1by default, where Live isn’t served, by Google or by Verlon. gemini-3.8-live-extended-thinkingneeds a thinking level. AddthinkingConfig: { thinkingLevel: ThinkingLevel.LOW }(orHIGH) to the config, withThinkingLevelimported from@google/genai. Without it, the session closes as soon as it opens, with1007and Google’s reason: “Thinking level must be specified for this model.”- Thoughts can arrive as text.
gemini-2.5-flash-native-audio-preview-12-2025sends its thinking as a text part markedthought: truebefore the audio. Skip those parts if you only want the answer. - A turn can end empty. If the speaker pauses mid-question, the model may start to answer, be interrupted, and send
turnCompletewith no audio. Wait for the turn that carries the answer before closing. gemini-2.5-flash-native-audio-preview-12-2025is deprecated. It shuts down on March 31, 2027; move togemini-3.8-live.- Server only. Google’s ephemeral tokens aren’t served, so open Gemini Live sessions from your server.
- Refusals arrive as the close reason. Once the socket is open, a refused session closes with a code and the reason as its text, the way Google’s own server does; there’s no
errorevent. When Google refuses, you get Google’s own code and words:1007for a setup it can’t use,1008for a request it refuses. When Verlon refuses (a model it doesn’t serve on this path, a spend limit), the code is1008.
Live translation
gemini-3.5-live-translate-preview listens and speaks the translation as it hears it, with no turns. Name the language to translate into in translationConfig:
- Stream in real time, and close when you’re done. The translation follows the speech as it comes in. There’s no
turnCompleteto wait for. - Speech already in the target language is silent unless you set
echoTargetLanguage: trueintranslationConfig. The default target is English (en). - It needs
@google/genai2.8 or later. Earlier versions droptranslationConfigwithout a word, and the session then translates into English.
OpenAI transcription
Stream speech up and the transcript streams back down as it’s recognized. Open a transcription session with the same client as OpenAI voice:- The first event names the model. It must be
session.updatewithsession.type: "transcription"andsession.audio.input.transcription.model. If none arrives within 10 seconds, the session gets anerrorevent and closes. - Commit the audio yourself.
gpt-live-transcribedoesn’t take server voice detection, so setturn_detection: nulland sendinput_audio_buffer.commitwhen a stretch of speech ends. - From a browser, mint the client secret with the transcription session above (
session.type: 'transcription'), and open it with{ intent: 'transcription' }or the plain WebSocket from From a browser.
Mistral transcription
Use Mistral’s SDK with its server URL pointed at Verlon:Billing and logs
- What a session costs. Each session bills by what the provider reports:
- OpenAI voice and Gemini Live: the text and audio tokens in and out, at the model’s rates;
- OpenAI Live and xAI voice: every second the session is open, including silence;
- transcription: the seconds of audio transcribed.
- Whose key. A session runs on your own provider key when you’ve added one, otherwise on Verlon’s.
- Spend limits. Spend is reserved a minute ahead and renewed as the session runs. If your spend limit or credit can’t cover the next minute, the session closes (code
1008), so it never runs past what you can pay for. - Logs. Each session is one entry in Logs: the model, the gate, the cost, and the session’s length as its latency. Audio and transcripts aren’t stored.
Limits
Errors
A session that can’t open is refused before the WebSocket opens. The refusal is an HTTP status with the provider’s error body: OpenAI’s{ "error": { "message", "type", "code", "param" } } on /v1/realtime and /v1/live/sessions, Google’s on the Gemini path, and Mistral’s on Mistral’s path.
OpenAI Live, Gemini Live and OpenAI transcription read the model from the first message, so their model refusals come after the socket opens: an
error event with the same code (OpenAI), then the close.
Once a session is open, it ends with a WebSocket close code:
Related
- OpenAI SDK: the same client for chat, responses and audio
- xAI, Google and Mistral: their other models through Verlon
- Gates: route a session through a gate

