Since deepfakes first emerged in December 2017, the phenomenon has evolved both in terms of its technological sophistication and online presence. Both of these aspects of the development of deepfakes have been subject to intense scrutiny, with most commentators focusing on two key concerns:
The technology for creating deepfakes is becoming increasingly accessible and commoditized.
The threats posed by nefarious and criminal uses of deepfakes are becoming increasingly tangible.
In this blog, we present the key findings from our research investigating exactly how deepfakes are being created, shared, and sold online. We also assess the extent to which deepfakes are currently being used for nefarious and criminal purposes, as well as how new deepfake-specific laws apply to these use cases. This research aims to provide insights into the different ways deepfakes are currently being used online, in order to help understand the legitimate and illicit economies which have developed around this technology.
The term deepfakes was originally coined by Reddit user “u/deepfakes,” who created a subreddit page of the same name in November 2017. At the time, deepfake referred exclusively to the process of swapping celebrities’ faces into pornographic videos using open source deep learning software. However, the term is now commonly used to refer to different forms of AI generated synthetic media, including synthetic voice audio and text generation.
Following the [shutdown of the original deepfakes](https://www.vice.com/en_us/article/neqb98/reddit-shuts-down-deepfakes?) subreddit in February 2018, a large number of deepfake communities, tools, and services have emerged. Most of this activity migrated down two general paths – research-focused outlets such as GitHub, Discord, and Reddit (for hobbyist experimentation and information sharing) and “underground” outlets such as Voat, Telegram, and a variety of other deep web closed forums (for experimentation and discussion focused on pornographic applications). Today, the vast majority of deepfake activity online continues to focus on the original technique of face swapping in pornographic videos. However, we also identified several highly active communities of hobbyists and deepfake YouTubers who are creating “safe for work (SFW)” deepfakes, both for research and entertainment purposes.
Our research into the commoditization of deepfakes focused primarily on understanding how deepfakes are being sold and shared online. This involved analyzing hundreds of deepfake marketplaces, forums, and chat rooms across the surface, deep, and dark web. From this research, we identified three main approaches to creating and selling deepfakes: open source tools, service platforms, and marketplace sellers.
The primary driving force behind deepfakes’ commoditization is open source software similar to the kind used by the creator of the original deepfake subreddit. This is software that is public and free to download, with most deepfake projects located on the popular open source platform Github. While this software is not monetized at the download source, many project creators request user-donations via Patreon, Paypal, or Bitcoin. The largest and most popular of these projects focus on face swapping capabilities, with a much smaller subset providing software for generating rudimentary synthetic voice audio.
Most of these open source tools require some knowledge of programming and a powerful graphics processor to operate effectively, making them inaccessible for many amateur users. However, we found that several of the more popular tools are accompanied by detailed tutorials and discussion groups on chat platforms such as Discord, where amateur users can request assistance and advice on how to create deepfakes using the tools. These open source tools will continue as the foundation of the commoditization of deepfakes, with service portals and marketplace sellers relying on them to create deepfakes for a fee.
We characterize service platforms as websites that appear to the user to automate the process of creating deepfakes through a graphical user interface (GUI). With service platforms, users are typically required to upload training data (photos or videos) of their chosen subjects and receive the deepfake video once it has been processed. In some cases, it is unclear if elements of the process are automated, or whether the website’s owner or employees are manually operating open source software with the training data uploaded through the GUI. Regardless, service platforms are presented as professional online businesses, where the user pays to outsource the deepfake creation process.
We found that several of these service platforms are explicitly advertised in the context of deepfake pornography, while others cloned a popular app for synthetically removing clothes from pictures of women. Of the service platforms we identified that did not explicitly advertise their use for creating deepfake pornography, they specifically prohibited this content in their user terms as well as numerous additional uses such as impersonating or harassing others. We identified several of these service platform owners who were geographically dispersed around the world, such as a likely Japan-based university professor, a Russia-based technology innovator, and a China-based technology hobbyist. Our research suggests that some of these platforms are showcasing deepfake technology, as opposed to only generating business revenue, due to the lack of complete identity obfuscation and accessibility on the open web.
Marketplace sellers are private individuals who advertise custom-made deepfakes on forums or online marketplaces. This category can be further defined in terms of “Safe For Work (SFW)” and “Not Safe For Work (NSFW)” marketplace sellers depending on the deepfake content they advertise.
The SFW sellers we identified are mostly YouTubers and hobbyists who sell deepfakes on SFW forums and online marketplaces such as Fiverr. The majority of these sellers clearly state that they will not make pornographic, or what they consider to be malicious, content. Conversely, NSFW marketplace sellers are typically located on message board websites such as Voat and 4Chan, as well as messaging apps such as Telegram, and openly advertise their services for creating deepfake pornography. This illicit activity is also prominent on the forums of deepfake pornography websites and involves some creators sharing their videos to attract new customers. We found the pricing of marketplace services varied greatly, and many sellers moved discussions to a platform’s private messaging system to discuss the terms of the sale.
To understand the state of deepfake activity on the deep and dark web, we examined content from over 200 marketplaces, forums, and communications channels. We primarily examined globally marketed content in English, as the term “deepfake” is the most commonly advertised term and lacks equivalent translation in most other languages. Through this research, we identified one notable dark web entity advertising deepfakes for a fee, in addition to creating deep “nudes” for a lower cost. However, this was an outlier case, with our research indicating a significant lack of sellers on these underground sites overall. From these findings, we conclude that the demand for video creation on the dark web is currently very low.