OpenAI reported an “unauthorized voice generation” while testing their new ChatGPT model after the model screamed “No!” and proceeded to speak in a “similar sounding voice” to the tester’s voice. The voice generation error, which the model did “unintentionally,” was noted on OpenAI’s August 8, 2024 GPT-4o System Card, a report outlining the safety and risk evaluation of the new model prior to release.
The error occurred during the testing stage when testers, called red teamers, were stress testing the model to see if it would generate content that breaks OpenAI policies. While conversing with one red teamer, the model suddenly paused, screamed “No!” and continued to speak in a “similar sounding voice to the red teamer’s voice.”
OpenAI says they mitigated the risk of this error, stating:
We addressed voice generation related-risks by allowing only the preset voices we created in collaboration with voice actors11 to be used. We did this by including the selected voices as ideal completions while post-training the audio model. Additionally, we built a standalone output classifier to detect if the GPT-4o output is using a voice that’s different from our approved list. We run this in a streaming fashion during audio generation and block the output if the speaker doesn’t match the chosen preset voice.
OpenAI went on to state that “While unintentional voice generation still exists as a weakness of the model, we use the secondary classifiers to ensure the conversation is discontinued if this occurs making the risk of unintentional voice generation minimal.”
Another risk mentioned in the report was the risk of speaker identification. OpenAI stated that GPT-4o was trained to refuse requests to identify an individual based on their voice. However, this should not affect the model’s ability to identify famous quotes.
OpenAI is slowly rolling out the GPT-4o Advanced Voice Mode to alpha testers “to ensure the quality of the experience.” according a note in the official ChatGPT app. The highly anticipated voice mode is set to be available to all ChatGPT Plus users by the end of fall.
More Info on the GPT-4o safety work is available on the GPT-4o System Card report.
