Call an AI on the Phone

A small Python server that lets you call a phone number and talk to Gemini.

Gemini 2.0 shipped with the Multimodal Live API: two-way streaming with audio in and audio out. Twilio can stream a phone call's audio over a WebSocket. Connect the two and you have the base of an interactive voice agent for support lines, appointment booking, or surveys.

This is sample code for learning. Write your own for production.

One Bridge, Two Sockets

A Quart app sits in the middle. Quart is an async Flask, which suits holding two streams open at once.

What You Need

• A Google Cloud project with Vertex AI enabled.
• A Twilio account and a phone number.
• Python 3.12 or earlier. The code uses audioop, which was removed in Python 3.13. On newer versions, install audioop-lts.
• ngrok, so Twilio can reach your local server.

pip install google-genai quart

gemini_twilio.py

Three jobs: read Twilio's stream, convert audio both ways, and relay Gemini's replies.

from google import genai
from quart import websocket
import json, base64, audioop

class GeminiTwilio:
    def __init__(self):
        self.client = genai.Client(
            vertexai=True, project='YOUR_GCP_PROJECT', location='us-central1')
        self.model_id = "gemini-2.0-flash-exp"
        self.config = {"response_modalities": ["AUDIO"]}
        self.stream_sid = None
        self.rate_state = None

    async def twilio_audio_stream(self):
        """Yield caller audio as PCM. Twilio sends JSON events."""
        while True:
            data = json.loads(await websocket.receive())
            if data['event'] == 'start':
                self.stream_sid = data['start']['streamSid']
            elif data['event'] == 'media':
                mulaw = base64.b64decode(data['media']['payload'])
                yield audioop.ulaw2lin(mulaw, 2)
            elif data['event'] == 'stop':
                break

    def to_mulaw(self, pcm: bytes) -> str:
        """24 kHz PCM from Gemini to 8 kHz base64 mu-law for Twilio."""
        pcm, self.rate_state = audioop.ratecv(pcm, 2, 1, 24000, 8000, self.rate_state)
        return base64.b64encode(audioop.lin2ulaw(pcm, 2)).decode('utf-8')

    async def gemini_websocket(self):
        async with self.client.aio.live.connect(
                model=self.model_id, config=self.config) as session:
            try:
                async for response in session.start_stream(
                        stream=self.twilio_audio_stream(), mime_type='audio/pcm'):
                    if data := response.data:
                        await websocket.send(json.dumps({
                            "event": "media",
                            "streamSid": self.stream_sid,
                            "media": {"payload": self.to_mulaw(data)},
                        }))
            finally:
                await websocket.close(1000)

To use a Gemini API key instead of Vertex AI, create the client with an API key. Add system instructions or tools to config to give the agent a job.

app.py

One WebSocket route. Each call gets its own bridge.

from quart import Quart
from gemini_twilio import GeminiTwilio

app = Quart(__name__)

@app.websocket('/gemini')
async def talk_to_gemini():
    await GeminiTwilio().gemini_websocket()

if __name__ == "__main__":
    app.run(host='localhost', port=8080)

Wire Up Twilio

1. Start the server: python app.py
2. In another terminal: ngrok http 8080. Copy the forwarding host.
3. In Twilio, create a TwiML Bin with the XML below. Use wss://, not https://.
4. Buy a number. Under Voice, set "A call comes in" to your TwiML Bin.
5. Call the number and talk.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://YOUR_NGROK_HOST/gemini" />
  </Connect>
</Response>

What I'd Do Differently

• Port to the current SDK. session.start_stream was an early helper in google-genai and may not exist in your version. The model name has changed too.
• Declare the sample rate. Twilio audio is 8 kHz. Tell Gemini that, or resample to 16 kHz before sending.
• Handle interruptions. When the caller talks over the agent, clear Twilio's buffered audio so it stops speaking.
• Deploy it. ngrok is for testing. Cloud Run supports WebSockets and is a natural home for this.

Three fixes from the original version are already in the code above: the stream loop exits on stop, the resampler keeps its state between chunks, and the socket closes with a valid code.