Hermes ElevenLabs Voice Agent: Call Your AI Agent by Phone

Key Takeaways

• Three components to the phone agent: ElevenLabs (voice synthesis and speech recognition), Twilio (phone number), Hermes (brain — memory, tools, files, past sessions)
• The voice agent can write files, schedule calendar events, recall past sessions, and perform tasks while you're away from your desk — all via a regular phone call
• Jarvis in-app voice mode works without Twilio — just click a button and talk within the Hermes dashboard; all conversation sessions are saved for later review
• Model speed matters: Claude 3.5 Haiku gave the fastest replies; heavier models add noticeable lag on calls — pick faster, lighter models for voice interactions
• Best for simple tasks (quick notes, reminders, calendar slots, memory recall); complex builds over the phone take too long — match the job to the moment