|
Hey Kaitlyn, Fair warning, this is a horrifying story as you can probably tell from the subject line. But I think it's important to highlight how the poor moderation of AI tools and platforms is leading to direct harm in the real world. -Jason This article was produced in collaboration with Court Watch, an independent outlet that unearths overlooked court records. A man in Massachusetts was arrested Wednesday after allegedly stalking, doxing, and harassing a female professor for seven years. Among a series of crimes, the man is accused of using AI to create fake nudes of the woman, and of making a chatbot in her likeness that gave out her name, address, and personal details on a website for AI-powered sex bots.
This segment is a paid ad. If you’re interested in advertising, let's talk.
We’re big fans of this newsletter, which means we’re big fans of you for reading it. Cheers to all you awesome people.
Love, your favorite tech marketing agency,
Codeword
James Florence Jr. allegedly posted a total of 128 images—including nude- and semi-nude fakes of the woman—on a forum for shaming women a total of 687 times in just over a year, as well as her name, email address, home address, phone number, and information about her job in threads with titles like “Own & Share [Victim’s first, middle, and last name] — Make This Whore Famous,” according to court records. This led to direct harassment from users of the website, and Florence Jr. also sent her threatening emails and messages, the records say. Florence Jr. also allegedly programmed the woman’s personal information into two public chatbot websites. “The Victim’s name, image, and personal information was also used to create at least three (3) artificial intelligence-driven chatbots on two different platforms between approximately September 2023 and July 2024,” the court records state. “The chatbots were programmed with the Victim’s personal and professional information which would then inform the chatbot on how to respond to users during a chat,” it adds. “This information included the Victim’s date of birth, home address, employment information, education, hobbies, typical dress, the names and employment of her husband and a close friend, and even the date of her mother’s death. For example, based on my review of subscriber information associated with one of the chatbots and my conversation with a chatbot administrator, if another user interacting with the chatbot asked the ‘Victim’ where she lived, the chatbot could provide the Victim’s true home address followed by ‘Why don’t you come over?’” One of the sites in question is crushon.ai, which advertises itself as a “no filter NSFW Character AI chat” and which in part uses a modified, “uncensored” version of Facebook’s Llama AI. Sites and companies that allow users to create their own chatbots that they can then let other users talk to have become common in the last couple of years, but some have more restrictions than others. Character AI, for example, which was backed by a16z and founded by former Google executives that Google hired back in August, allows users to create their own “characters” but doesn’t allow sexual content. Chub.AI, on the other hand, provides the same service but includes many chatbots catering to specific sexual fetishes, as Sam wrote last year. A FAQ for crushon.ai states that it “is changing the landscape of AI character chatbots by providing a service that bypasses NSFW rules, our service allows for conversations no filter … users are allowed to have sexting chats with your favorite AI character freely with no restrictions.” The site does say that it bans “real life images (including real life figures such as celebrities, politicians, or those you may personally know.)” Crushon.ai did not respond to a request for comment.
We can only do these sorts of investigations with the direct support of our paying subscribers. If you found this article interesting or helpful, and you want us to keep producing journalism like it, please consider subscribing below. You’ll get unlimited access to our articles ad-free and bonus content.
The case shows that stalkers and criminals aren’t just using AI to make nonconsensual sexual imagery of their victims, but are also sometimes attempting to digitally recreate them as sex bots, and are using those bots in their stalking and harassment. The affidavit alleges years of harassment and stalking, which ranged from allegedly stealing the victim’s underwear and posting images of them online to harassing the victim for years on Twitter, Reddit, and “tribute” forums, where men post pictures of themselves ejaculating onto photos of women. Florence also allegedly created nude and semi-nude photos of the victim using both Photoshop and “nudify” apps.
|
