OpenAI Gives Its Agents a Voice

OpenAI is expanding its controversial stable of AI voices to include agentic models. Agentic models are the hot trend in generative AI, enabling two-step processes such as asking an AI to buy plane tickets or change a customer’s order. Specifically, the new models include:

• Gpt-4o-transcribe and gpt-4o-mini-transcribe, both of which are speech-to-text models.
• Gpt-4o-mini-tts, a text-to-speech model.

Developers can access them on the OpenAI API and integrate them with the Agents SDK. Adding text-to-speech and speech-to-text to the API allows them to be used in a variety of AI applications, including agentic tools.

Advanced synthetic voices can make scams more convincing

The company wants to enable “deeper, more intuitive interactions with agents beyond just text,” but adding flexibility and greater autonomy in voice models raises the possibility of more convincing scam bots.

“We’re continuing⁠ to engage in conversations with policymakers, researchers, developers, and creatives around the challenges and opportunities synthetic voices can present,” according to a news release.

SEE: Have Some Spare Cash? You’ll Need it for OpenAI’s New API

Models have been tuned for accuracy, reliability, and realism

On March 21, OpenAI released new speech-to-text and text-to-speech audio tools in the API. The models have been tuned for accuracy and reliability, particularly in conversations including “accents, noisy environments, and varying speech speeds.” The models are intended for customer call centers or transcribing meetings.

They can also be instructed to speak in specific ways, from intentionally specific to dramatic or cheerful. OpenAI envisions some of these AI models being used for “expressive narration for creative storytelling experiences.” I can imagine this being used at theme parks or theatrical events – use cases that raise the specter of AI replacing creative professions. Example voices OpenAI suggests include “bedtime story,” “surfer,” “true crime buff,” and “medieval knight.”

Gpt-4o-transcribe and gpt-4o-mini-transcribe are designed to transcribe speech more accurately, particularly in conversations with accents, background noise, or varying speech speeds.

Gpt-4o-mini-tts can follow instructions to match tone or take on personas. OpenAI is careful to point out that all of the text-to-speech voices on the API are “artificial, preset voices” – definitely not Scarlett Johansson, who has accused the company of mimicking her voice without consent.

Agentic video AI may be on its way

Next, OpenAI said developers will be able to bring “custom voices” for “personalized experiences in ways that align with our safety standards.” The company is also pursuing ways to use video in agentic AI experiences.