Deepfake Harassment on X: A Case Study
A high-profile Chinese activist in the UK, Apple Peiqing Ni, was subjected to a barrage of deepfake posts on X (formerly Twitter) that falsely depicted her as a sexually promiscuous drug addict. Despite reporting the abuse, X's automated systems initially determined that the content did not breach the platform's rules on harassment or violent speech. The incident raises serious questions about the effectiveness of X's content moderation, particularly concerning targeted deepfake abuse.
The Attack
Ni, the 27-year-old founder of the UK-based China Dissent Network, was targeted by an account she believes is a pro-regime bot. The abuse began after she posted about joining a commemoration of the Tiananmen Square massacre on June 4 in Sutton, south London. The account posted 12 items tagging Ni, containing fake photographs and videos. Captions described her as having "chronically chaotic sexual relationships" and being a heavy drug user. One post incorrectly claimed Ni had been "beaten badly on the red streets of London while protesting with other anti-China groups," celebrating it as "perfect retribution for her extreme behaviour." This appeared to reference an attack by masked men on a male activist at a sit-in organized by Ni in Trafalgar Square in March.
X's Response
Ni reported the posts to X, but the platform's automated systems replied that the content did not violate its rules on harassment or violent speech. A follow-up complaint to X's support service was also rejected. This is despite X's own policies prohibiting "the malicious, unreciprocated targeting (such as mentioning or tagging) of individual(s), particularly when shared to humiliate or degrade someone." The account was only suspended hours after The Guardian raised the decisions with X's press office. Ni was later informed that X had acted in response to "different reports" regarding the content.
Ni's Experience
Ni, who moved to the UK in 2019 to study, expressed bewilderment at X's initial inaction. "I posted a poster for the Tiananmen Square massacre commemoration in Sutton and right after that, that account made deepfake images about me. They've been tagging my username since they started it," she said. The UK police had advised Ni to complain to X, highlighting the platform's role in addressing online abuse. The delay in action underscores potential gaps in X's moderation, especially for non-English content or politically sensitive topics.
Broader Implications
This incident is part of a worrying trend of deepfake abuse targeting activists and journalists. Deepfakes, which use AI to create realistic but false images and videos, are increasingly used to harass and discredit individuals. X's handling of Ni's case suggests that its automated systems may fail to detect nuanced harassment, particularly when it involves deepfakes. The platform's reliance on automated moderation, combined with reduced trust and safety staff under Elon Musk's ownership, has been criticized by experts. Ni's case highlights the need for more robust human oversight and clearer policies on deepfake abuse.
Conclusion
The targeting of Apple Peiqing Ni on X illustrates the challenges platforms face in combating AI-generated abuse. While X eventually suspended the offending account, the initial failure to act raises concerns about the platform's commitment to protecting users from coordinated harassment. As deepfake technology becomes more accessible, platforms must adapt their policies and enforcement mechanisms to prevent such abuse. For activists like Ni, the stakes are high: online harassment can have real-world consequences, including psychological distress and reputational damage. X's response in this case will be closely watched by civil society groups and policymakers advocating for stronger online protections.
This article is based on reporting by The Guardian. Read the original article.
Originally published on theguardian.com
