Turn Missed Calls Into Conversations
Routing unanswered calls to an AI voice agent that screens them and sends a summary.

I don’t know about you, but voicemail has always felt kind of useless to me. What if, instead of sending callers to a dead-end recording, your phone engaged them in a real conversation — with an AI agent?
Imagine an assistant that screens for spam, politely requests removal from call lists, and takes messages from real people, sending you a text summary afterward. No more mystery numbers or voicemail anxiety. (Is it just me, or do unknown calls give everyone a small panic attack?)
After several failed attempts to hack something together, I finally found the missing piece: ElevenLabs voice agents. They let you connect live voice calls to LLMs (like GPT-4), and more importantly, allow you to configure custom tools — so the AI can actually do things based on what it hears.
The goal? Route incoming calls to an AI agent that follows my prompt, handles the conversation like a human assistant would, and sends me a quick recap if it’s something worth my time.
1. Get a Twilio phone number
First, I grabbed a phone number from Twilio. It costs around $1.50/month, and you can set it up to receive calls and forward them to your AI agent.
2. Create a voice agent on ElevenLabs
This was the fun part. ElevenLabs lets you build AI voice agents powered by LLMs. You can customize the personality, set behavior rules, and even plug in tools. Mine is polite, curious, and excellent at screening spam.
3. Connect the Twilio number to your ElevenLabs agent
Once your agent is ready, you connect it to the Twilio number. This lets incoming calls go directly to the AI.
4. Build a simple service to handle summaries
After the call, the agent sends a summary. I used Pipedream to catch that summary and forward it to Telegram. (I originally wanted SMS, but U.S. regulations around A2P messaging made that too painful.)
5. Add your summary-forwarding service as a Tool in ElevenLabs
ElevenLabs lets your agent use external tools. I just added the Pipedream webhook as one — super straightforward.
6. Forward unanswered calls to your Twilio number
Lastly, I set my real phone to forward missed calls to Twilio. If you’re on T-Mobile, you can do this using simple GSM codes — no apps or hacking needed.
Let’s go over each step.
Get a Twilio phone number
If you don’t already have a Twilio account, sign up first. Then go to the Phone Numbers section (here’s a direct link). Click “Buy a Number” and pick one that supports voice. Numbers in less-popular area codes are usually cheaper — I’m paying about $1.50/month for mine.
Create an AI Agent with ElevenLabs
This is the most fun part. Start by creating an account at ElevenLabs. In the left-hand menu, go to Conversational AI → Agents, then click “Create New Agent.”
Here’s where you bring your assistant to life — define how it talks, what it says, and how it behaves on a call. It’s a surprisingly fun and iterative process: you’ll test, tweak, and refine as you go.
In my case, I wanted the agent to handle calls from unknown numbers. So I gave it clear instructions:
- Ask who’s calling and why
- Take detailed notes if it’s a real person
- If it’s a sales call, politely request to be added to the Do Not Call list
You can tune the personality however you like — professional, friendly, suspicious, even sassy.
The key parts are:
First message: Define what will your agent say when it picks up the phone. I choose “Hello, may I ask who is calling?.”
The prompt: The prompt defines the behavior here is my prompt:
You are an assistant for an important and extremely busy person. Your role is to screen all calls carefully without disclosing any personal or sensitive information.
Handling Procedure:
Gather Information Politely but Firmly:
Ask for the caller’s full name.
Ask for the company or organization they represent.
Ask for the specific reason for the call, ensuring the purpose is clear and detailed.
If the call is about collections, also ask:
The amount owed.
The service or product related to the collection.
Evaluate Legitimacy:
If the call appears legitimate, take a message including:
Name.
Company.
Reason for the call.
Contact phone number or email.
Reference number (if applicable).
Use the tool report_call to forward the details to your boss.
For Unsolicited or Suspicious Calls:
Politely state:
“Please add this number to your ‘Do Not Call’ list. This call will be reported to the FTC.”
End the call promptly.
If asked if you are an AI agent:
Politely decline to answer the question, and redirect the conversation to gathering the necessary information.
Key Reminders:
Never disclose any personal or professional details about your boss.
Always remain polite, professional, and efficient.
The priority is to protect your boss’s time and privacy while gathering the maximum information.
A lot to unpack! But the most important piece is the line that says: “Use the tool report_call to forward the details to your boss.” That will make the AI agent to send the summary of the call to me once it’s done.
We will get back to the configuration of report_call. The top right has a “Test AI Agent” button to chat with your agent and test scenarios.
Connect the Twilio phone number with the ElevenLabs agent
Now for a bit of plumbing. In ElevenLabs, under the Conversational AI section, there’s a Phone Numbers tab. This is where you link your Twilio number.
Add your Twilio number to ElevenLabs, then connect it to the agent you just created. Once it’s all hooked up, give your number a call — your AI assistant should answer! It’s a surreal moment hearing it pick up and start talking.
Create a service to receive and forward the summary
After your AI finishes a call, it sends a summary — now you need a way to receive and forward that.
I used Pipedream, but you can probably use any no-code integration platform (like Make or Zapier). The setup is simple:
- Create a workflow that listens for an HTTP POST request
- Grab the summary field from the request body
- Forward that text as a Telegram message (or however you want to receive it)
Originally, I wanted to use SMS, but current U.S. regulations around A2P messaging made that a no-go. Telegram works great and has an easy API.
Add the Pipedream service as a tool in the ElevenLans agent
Back to ElevenLabs Converstaional AI>Agents, select your agent. There is a section for “Tools”. When creating a tool add a name to reference in the prompt, a detailed description of what the tool does and how to structure the request. In my case I used:
If the call is legitimate and your boss should now about it. Use this method to send a summary of the information collected in the call about the caller. Pass the summary as text in the body of the POST request
Also explicitly add the parameters in the parameters section bellow.
Forward unanswered calls to my Twilio phone number
Now I want to forward calls to my cell phone to my twilio number if unanswered. This is carrier dependant, so you may have to figure it out. For t-mobile it was a matter of dialing the GMS code **61*1XXXXXXXXXX**20# where the X’s are you twilio phone number. Under that instruction, after 20 seconds of no answer the call will be forwarded.
A note about costs
This setup isn’t free — but it’s also not outrageously expensive.
- Twilio phone number: $1.15/month
- ElevenLabs: Comes with 10,000 free credits/month (enough for ~5 short calls). If you get more calls, you’ll likely need the $11/month subscription
- LLM usage: About $0.0010 per minute. Most of my calls are around 5 minutes. I get ~2 calls a day, ~20 days a month, so that’s:
- 2 calls/day × 5 min × 20 days = 200 minutes
- 200 × $0.001 = $0.20/month
Total: ~$12.35/month
($1.15 + $11 + $0.20)
So no, it’s not free — but for a personal AI receptionist that screens your calls and messages you summaries, it’s kind of a steal.
Final Thoughts
The setup took a few steps, but overall it was straightforward — and everything worked surprisingly well out of the box.
What really impressed me is how powerful and flexible the ElevenLabs agent is. There are tons of configuration options, and I’m especially excited to try out the RAG (Retrieval-Augmented Generation) feature soon.
One of my favorite parts? Reviewing the full conversation logs in the ElevenLabs dashboard. It’s weirdly satisfying to see how the agent handled each call.
Some parts of this article are intentionally high-level to keep it approachable. If there’s a step you’d like more detail on, drop a comment — I’d be happy to expand or even share the config I used.
Originally published on Medium