A few days ago, the startup ElevenLabs launched a beta version of a platform that allows users to synthesize entirely new voices by converting text to speech or by replicating someone’s voice. However, internet users quickly began using the neural network for dubious purposes. The company announced on Twitter that it has observed a “growing number of cases of misuse involving voice cloning” and is now exploring solutions to address this issue by “implementing additional safeguards.”
While ElevenLabs did not specify what constitutes “misuse,” posts on the site 4chan have been found featuring audio files that sound as if generated voices are making celebrities say questionable things. Users also posted audio clips with homophobic, transphobic, violent, and racist content. It’s unclear if ElevenLabs’ technology was used in all these files, but a post on 4chan containing a large collection of these clips did link back to the startup’s platform.
ElevenLabs is now working to compile information to prevent future misuse of its technology. Currently, they are considering measures such as account verification, requiring users to provide payment information or ID before using the voice cloning feature, and possibly verifying copyright ownership of voices. For example, they might request users to submit a sample test to confirm ownership of the voice they wish to clone. Ultimately, the company is considering discontinuing its Voice Lab tool or implementing manual verification for voice cloning.