A few months ago Meta followed Apple’s lead and announced in-app nudity detection for Messenger to attempt to dissuade teenagers from getting themselves into trouble by acting unwisely and without regard for their own privacy. Various people approached me, asking questions like:
THEM: Look at this! Isn’t this client-side scanning (CSS)? Doesn’t this break end-to-end encryption and end-to-end security? Isn’t this a huge privacy problem?
ME: The answer is: no, it’s fine.
THEM: But what kind of tech is this if not CSS?
ME: The test is whether it leaks even one bit of data to third or fourth parties. A similar test should be applied to Grammarly, for instance. People should be free to manipulate their own data and messages as they see fit, including the freedom to opt-in to features to detect incoming or outgoing messages that contain rude words or too much fleshy colour. The question is: whom do such features serve? If they are obligated by the state or law enforcement or if they leak content or analysis or other information to third parties, they will fall outside of the definition of end-to-end encryption.
The important point that such people were missing is “Cui Laboras?” or “For whom is the software working?”; this matter is discussed at length in my primer on end-to-end encrypted messenger software, but in terms of risks to privacy the “traffic light” system that I describe in the primer, is useful:
This leads us to a question which I call “cui laboras?” – for any given software feature we should ask: “who does this software serve?” and “how much agency does the first party have over its operation?”; such agency being one of:
- “can they avoid it?” or
- “can they choose an alternative that is not afflicted by it?”
I consider the results as a kind of traffic-light protocol:
GREEN Features: The software feature directly serves Alice, the first party; for instance: spell-checking and grammar-checking text (even via some helper-app installed by Alice) or similarly…
AMBER Features: The software feature exists to provide non-compliance-related value to the Platform, viz: the third party which provides the “field” in which E2E takes place. AMBER features might include: user identity verification via phone number or email confirmation (or) active monitoring of metadata to enable the platform’s “fraud prevention” and “abuse reporting” mechanisms, thereby to maintain the platform’s “safety” value proposition to Alice and Bob. AMBER features must not compromise [end to end privacy], but if Alice and Bob do not want them then they are free to choose some other platform that approaches (e.g.) abuse detection differently, or which ignores it entirely.
RED Features: The software features will compromise [end to end privacy], or exist to offer value to a fourth party such as “the state” or “law enforcement,” or to serve an abstract notion such as “protection of children,” “prevention of terrorism,” or “national security.” Often these features are intended to be pervasive and Alice is intended to have no means to avoid them without inviting suspicion. RED features may include otherwise-AMBER features that have suddenly become obligatory for legal compliance; this is a reflection of the dual-use problem that (in this case) data collected for a GREEN or AMBER reason might equally be used for RED. Each distinct “intentional use” to which the data is put, ought be considered a separate mens rea to the overall data collection’s actus reus in the traffic-light protocol.
The reason that it’s important for Civil Society to understand this aspect is that confusion about what is/is-not “Client Side Scanning” serves the interest of the people who want to break end-to-end security; this is evidenced by journalist Gareth Corfield having to publicly rebut a child-safety advocate who argued that WhatsApp’s use of simple on-device regular-expression pattern-matching to detect and warn about phishing URLs being sent to chat participants.
The argument from the pro-Client-Side-Scanning folk is “…look, scanning is already being done, that means security is broken, so we might as well send everything to law enforcement in order to save the children!”
This is why it’s important to distinguish “detecting nudity to nudge a teenager into safer behaviour” versus “logging a hash of every message and image you send with a central authority to prevent criminal activity” – again, Cui Laboras, for whom is the software working?
Why is this relevant now?
Google just announced on-device AI conversation monitoring during phone calls:
Personally I don’t have any problem with this — as described it is “GREEN” software — if and only if there are no logs or other records or cues being sent off-device. Of course the Governments of the world could demand those to be added later, but THAT is what we need to be alert for and for us to resist.
Assuming that the Gemini Nano software is working as-described, it’s not a problem, and presenting it as defacto being Client-Side-Scanning / a “problem” will actually bolster the positions of those who want entirely to erase end-to-end online privacy.
Leave a Reply