This document is so absolutely scary that I don’t even know where to begin, so here are some links and some quotes, and I’ll try to do more over the weekend if I can stomach it. It would be hilarious if it wasn’t being pushed hard in the House of Lords, having the ear of both Jess Phillips and Lord Nash, and doubtless many credulous peers.

---

Disclaimer

Before I begin, some may ask so I might as well explain: I am an unchurched atheist, but not an antitheist; I grew up in a non-observant CoE/Methodist family where God largely happened to other people, but attended a Catholic Boys’ School, had any number of Muslim and Jewish favourite uncles, and eventually married into a Hindu family. I believe in the power of stories rather than faith, and I find it astonishing that people will even kill each other to validate their personal belief whether Star Trek or Star Wars is the better series, or even whether Kirk, Picard or Janeway was the best Captain.

I don’t hate religion, but I don’t rate it as a moral argument in itself, rather than seeing a declaration of faith being an individual’s hazy claim to a certain moral approach.

Reading the attached, I feel quite certain that for the authors of the white paper, those claims do not include “liberalism” nor “respect of individual privacy”; but faith does seem to provide a slightly cliquish common thread to their approaches towards matters of liberalism and privacy.

Context & Background

• Safe To Net: provider of CSAM-scanning technology
• Two founders of Safe To Net, talking about God
• International Justice Mission (IJM) Wikipedia Page
• IJM UK Project: Building Smart Devices that Can’t Create or Display Child Sexual Abuse Images and Videos
• IJM UK White Paper: “Designing for Camera and Screen Rendering Incompatibility with Unambiguous Child Sexual Abuse Images”

…and the white paper is where it gets really scary.

White Paper Quotes

Page 5 & 6 – apparently the best way to address child abuse being livestreamed from the Philippines is to embed machine learning classifiers into all the world’s computers and have those running 24×7

This paper therefore proposes a proactive, preventative intervention that can protect children at scale, both as users and non-users of online services. Specifically, the paper examines the technical feasibility and key considerations of deploying visual machine learning/artificial intelligence (ML/AI) classification technology via the operating system (OS) of camera-enabled smart devices that can connect to the internet (smartphones, tablets, and laptops) to detect and block suspected CSAM—an approach referred to as “on-device CSAM blocking.” By embedding an AI model into operating systems such as Android, iOS, and Windows, device manufacturers and OS developers can disrupt the image capture, video recording, livestreaming, and screen rendering of child sexual abuse

page 6

For International Justice Mission (IJM), this concept of on-device CSAM blocking emerged from the Scale of Harm project, a prevalence study in the Philippines conducted by IJM and the University of Nottingham Rights Lab in consultation with survivors whose sexual abuse was livestreamed and captured as CSAM. The Scale of Harm study estimated that nearly half a million children in the Phil- ippines were sexually abused to create new livestreams, images and videos for sale to offenders online, in a single year. The study also estimated that nearly a quarter of a million adults in the Philippines trafficked children to produce new CSAM. In light of that scale and how easy it is for people to livestream child abuse without meaningful tech detection or disruption, consulted survivor leaders emphasised the need for a safety-by-design approach to prevent and disrupt harm; in other words, the use of technologies that prevent CSAM from being created and consumed in the first place, particularly through live video-enabled platforms.6 The logic is simple: because the primary motive of sellers of livestreamed abuse is receiving payment from demand-side offenders (“buyers”), disrupting the ability to create, transmit and view CSAM live disrupts the transaction itself and protects children from new or recurring hands-on abuse. In IJM and law enforcement experience, traffickers or ‘facilitators’ are opportunistic offenders, largely abusing children because it is easy. Meaningful technological disruption has the potential to deter many of them, thus protecting children from abuse.

page 7 – the white paper repeats the (there is no better word) deception that detection can/would not lead to automatic reporting; when Apple proposed NeuralHash a few years ago with a similar promise, at subsequent industry discussion NCMEC were demanding (I heard this) that each and every detection be reported to them, immediately, in adherence with the law:

Due to factors such as ongoing political inaction and debates over user privacy, the paper proposes a prevention-first approach to on-device CSAM blocking. Under this framework, attempts to create, livestream, or watch suspected CSAM would be disrupted and blocked directly on the device without transmitting information to third parties or triggering moderator review and reporting mech- anisms. This is data protection by design and default. As such, content shared within encrypted private communications would not become public or accessed by anyone other than the sender and recipient. On-device detection and disruption seeks to overcome existing unwillingness to detect CSAM or live child abuse within E2EE platforms by instead disrupting this abuse purely on device pre-encryption and post decryption without interfering with encryption architecture. This paper examines key considerations for a prevention-first approach, including mitigating the risks of false positives and the implications for law enforcement and victim identification efforts

page 9 – they have not bothered to model performance impact yet

Model performance is a critical consideration for on-device CSAM blocking. It cannot be captured by a single metric as effectiveness varies depending on the type and quality of content being assessed and the context in which the system operates. Performance should therefore be evaluated using a range of measures, including precision, recall, false positive rates, and confidence levels, while accounting for the trade-offs between detecting abusive content and minimising erroneous classifications. While false positives may occur, their impact can be mitigated through proportionate interventions and ongoing model refinement. Additionally, models should be deployed responsibly through representative data, robust testing, privacy and security safeguards, and continuous improvement under real-world conditions.

and they handwave-away “slippery slope” arguments

Slippery slope concerns are frequently raised in discussions about CSAM detection and prevention technologies, with critics arguing that tools designed to protect children could be repurposed to monitor lawful content. While these concerns are legitimate, the paper argues that such concerns can be mitigated through clear legal limits, independent audits, trans- parency requirements, privacy-by-design safeguards, and accountability mechanisms that

and they equally handwave-away creative abuse by motivated actors, arguing that “friction” (making computers more troublesome to use) will benefit humanity

Some bad actors will inevitably attempt to evade, reverse-engineer, or misuse security systems, including on-device CSAM blocking technology. However, risks can be significantly reduced through measures such as privacy-preserving design, robust cybersecurity safe- guards, tamper-resistant models, adversarial testing, limited outputs, and regular independent security evaluations. While determined bad actors may seek older or unsupported devices, widespread deployment across mainstream devices and operating systems could create substantial friction, reducing opportunities to create, distribute, and consume CSAM over time.

page 10 – also: user identity is going to have to be drilled into the platforms to enable it to determine whether you are on the whitelist of “good guys” who are allowed to deal in child abuse imagery; and of course nobody is going to be able to patch the code which enforces that determination in order to circumvent the whole house of cards. (see below)

Exemptions should be available for a limited number of authorised entities, such as law enforce- ment agencies, child protection hotlines, and digital forensic specialists who have a legitimate legal need to access and analyse CSAM for their work. These exemptions can be implemented through secure, time-limited authorisation mechanisms that allow approved personnel to carry out investigations and victim identification activities on approved, monitored devices without weakening protections for the broader public.

page 11 – all the bad things that could happen are only bad if you don’t redefine them as acceptable collateral damage to a bigger goal, like for instance “let’s redefine end-to-end security as transport encryption and enforce on-device surveillance”:

At the same time, the authors recognise that there are legitimate user privacy concerns that deserve serious technical consideration. Mass surveillance architectures, overbroad content scanning systems that lack transparency, or implementations lacking safeguards to prevent scope creep represent potential risks to civil liberties. These concerns are not in opposition to child protection but rather are part of the safety-by-design, security-by-design, and privacy-by-design considerations this paper seeks to address.

Taking a Break

I need to step away from this for a while, but let me leave you with quotes from pages 116/117

A2. THE REALISTIC OBJECTIVE

Any technical control on consumer devices operates against a population of users that varies by many orders of magnitude in technical capability and motivation. It is important that this paper is honest about what the exemption mechanism can and cannot achieve.

The mechanism is a chain-link fence, not a vault wall. Its purpose is to make circumvention imprac- tical for the great majority of users most of the time. A determined, technically literate offender— particularly one willing to invest time in jailbreaking, custom firmware, or hardware modification—may be able to defeat controls implemented at the operating system level. This is not a failure of the design; it is a property of the platform. The same is true of every other software-based control on a general-purpose computing device, from Digital Rights Management (DRM) to parental controls to corporate device management.

Several categories of device will sit outside the scope of any such regime altogether:

• Some legacy devices manufactured before the regulation comes into force may not carry the mechanism. Depending on the method of implementation, some legacy devices may carry the mechanism by means of operating system update. However, there will be devices that cannot be brought into scope of regulation even through OS update. The installed base of such devices will diminish over years, not months, and any honest policy assessment must account for a tail.

• Out-of-scope devices that are nonetheless capable of accessing relevant content— single-board computers such as Raspberry Pi with attached displays, self-built systems, and devices manufactured outside the regulatory perimeter—will remain available, unless and until regulation catches up. The regulation can sensibly target the dominant consumer device cate- gories (mobile phones, tablets, mainstream laptops and desktop computers) without claiming to cover every general-purpose computing surface.

• Devices imported or operated outside the regulating jurisdiction present the usual cross- border enforcement difficulties.

The proportionality argument is therefore that the regulation, and its exemption mechanism, exists to reduce harm at population scale—to make the casual or developing offender’s path materially harder—and not to construct an un-bypassable system.

They are talking about making all software, everywhere, worse and even slower and more flaky than it already is, rather than try to help Filipino kids directly.

If you read section “A3. THE THREAT LANDSCAPE” (p117) – they clearly have no concept of patching a live system, nor how easily this stuff will all be circumvented by motivated individuals. All that will happen is 6 billion people have a slippery slope into censorship and software crashing, whilst criminals circumvent it all.

Also: “single-board computers such as Raspberry Pi with attached displays, self-built systems, and devices manufactured outside the regulatory perimeter—will remain available, unless and until regulation catches up” – what the hell do they plan to have happen to open source operating systems?