LWN: Comments on "Scanning "private" content"

Scanning "private" content

Jandar — Wed, 18 Aug 2021 11:32:45 +0000

> Android already has a feature that completely blocks out non-Google Play apps.

Can you elaborate? On my non rooted Android phone I often install apps from Free-Droid.

Scanning "private" content

NYKevin — Tue, 17 Aug 2021 16:52:17 +0000

> because my 4 years old goes to the beach naked.

There has been some confusion on this point, so to be clear: The phrase "visually similar," in this context, refers to things like cropping, adding/removing a watermark, greyscale/color, hue/saturation, etc., not the https://xkcd.com/1425/ problem. If you take a brand-new photo, regardless of the subject matter, this technology is not intended* to find that photo.

* False positives exist.

Cloud = big brother

NYKevin — Tue, 17 Aug 2021 16:45:30 +0000

> I have three countries within two hours travel which I can move to if I wanted to.

Ha. Americans have that at the state level, in some parts of the country, but if you dislike what the federal government is doing, your nearest options are Mexico or Canada. Because the US is simply huge, at least one of those options is guaranteed to be unreasonably far away.

Also, we don't have a Schengen-like-arrangement with either of them, so you'd have to go through the whole immigration process, which can take months or years.

Cloud = big brother

marcH — Tue, 17 Aug 2021 16:21:43 +0000

> I think you mean simple JOURNALISTS.

I don't and I respectfully disagree. I think journalism is one of the markets where there is pretty decent competition and where crap wins because that's what most people "consume" preferably, not because of a lack of hard working (but underpaid) journalists trying to do the Right Thing. Because crap is emotional, "infotaining" or free or all of the above. Even when we understand that "if it's free, we're the product" it's still hard to resist and easy to fall for it.

Scanning "private" content

rodgerd — Tue, 17 Aug 2021 00:45:22 +0000

To follow up on my own comment, this is what Google Drive looks for and enforces: https://support.google.com/docs/answer/148505#zippy=%2Cmi...

Scanning "private" content

rodgerd — Mon, 16 Aug 2021 23:04:12 +0000

> they were very likely already doing some sort of CSAM scanning on the server side anyway. That's basically standard practice in the industry, barring E2EE products that are incapable of it.

Pretty much. OneDrive, Google Drive, Dropbox will almost certainly all be doing this, and I'd be surprised if Slack, Teams, and so on likewise don't.

WhatsApp have made a big deal about *not* doing this, but they provide an even (IMO) creepier feature for law enforcement, which is using metadata analysis to de-anonymise the social graph of their users.

(Even creepier overreach is that the T&Cs for some smart TVs - which is any TV you can buy now - specify that you agree to the TV screen capping and sending the screen caps back to base)

> The only reason this should be controversial is because of the possibility that it later expands to include more stuff.)

There's a few more things to it than that, in my opinion:

1. Not everyone wants US government orgs setting legal policy for their devices. CSAM is pretty much that.

2. How are false positives handled? We've seen woo like polygraphs misrepresented (i.e. lied about) in courts, along with other types of forensic pseudo-science like ballistics work. The possibility of very serious legal trouble from a misrepresented or misunderstood application of hash collisions is not a comfortable thought.

3. Once it is well-understood that this sort of scanning is available, pressure to expand is inevitable, with the same leverage: you provide this facility, or you're out of our market.

Scanning "private" content

scientes — Mon, 16 Aug 2021 21:00:54 +0000

The EU is nothing more than a giant cabal like the defence pacts that lead to WW1. A bunch of incompetent beurocrats telling people what to do, and Coronavirus demonstrated that the EU doesn't exist.

Scanning "private" content

NYKevin — Mon, 16 Aug 2021 20:03:14 +0000

> Huh, interesting. So what about people who already own iPhones? Can they get a refund because Apple has fundamentally changed the terms of the agreement? (I doubt it, sadly.)

Doubtful, but they could disable iCloud photo uploading, at which point (as far as I can tell) they would no longer be subject to this scanning. At least until Apple changes the policy again.

(If you think that Apple *won't* change their policy again, then this whole controversy is a complete nothingburger, because they were very likely already doing some sort of CSAM scanning on the server side anyway. That's basically standard practice in the industry, barring E2EE products that are incapable of it. The only reason this should be controversial is because of the possibility that it later expands to include more stuff.)

Scanning "private" content

k8to — Mon, 16 Aug 2021 15:49:49 +0000

Technology that opens the door to this type of abuse is a problem. The laws that criminalize totally reasonable behavior are a problem. And yes, the law enforcement full of predators is multiple problems.

Scanning "private" content

scientes — Mon, 16 Aug 2021 13:11:34 +0000

In these cases the problem is not bad law but bad people (and I am not talking about the criminals).

Scanning "private" content

scientes — Mon, 16 Aug 2021 13:08:47 +0000

Android already has a feature that completely blocks out non-Google Play apps.

that seem to be of a sexual nature,

scientes — Mon, 16 Aug 2021 13:05:12 +0000

> that seem to be of a sexual nature,

The "I know it when I see it" horseshit.

Democracy

smurf — Mon, 16 Aug 2021 12:33:45 +0000

Possible, yes. Highly unlikely, also yes.

Democracy

immibis — Mon, 16 Aug 2021 11:09:25 +0000

> There's a long history of expanding the franchise with people incorrectly predicting drastic consequences if this is attempted and then nothing interesting happening.

Indeed, there is a long history of *people doing anything at all* with people incorrectly predicting drastic consequences if this is attempted and then nothing interesting happening.

It is entirely *possible* that Apple stops at child porn, and only ever detects child porn, and the hashes are good enough that false positives are rare and random.

Scanning "private" content

immibis — Mon, 16 Aug 2021 11:07:00 +0000

> If prosecutors asked me to break in (or even strongly suggested or encouraged it), that would be different.

Many commenters are speculating that Apple made this decision because the alternative was for the government to demand an end to encryption altogether.

Scanning "private" content

immibis — Mon, 16 Aug 2021 11:02:19 +0000

One could more charitably guess that they don't want to send the set to the device because the people who enjoy this kind of content could then simply Google the hashes to find more of it.

Cloud = big brother

Wol — Mon, 16 Aug 2021 09:46:04 +0000

> "Carefully balanced", numbers, science and complexity are unfortunately all dead; simple people watching cable news and social media demand yes/no answers.

I think you mean simple JOURNALISTS. Many people probably would like more detail, unfortunately the gutter press has learnt that a lot of people enjoy watching journalists and politicians fighting, and as always, the bad drives out the good - decent investigative journalism has died ...

Cheers,
Wol

Cloud = big brother

marcH — Mon, 16 Aug 2021 09:16:33 +0000

What many people miss is the possibility of these two evils cancelling each other out, more specifically a very carefully balanced regulation trimming the worst excesses of the private sector. This includes of course anti-trust regulations to ensure some competition but not just. Also known for a very long time as "checks and balances".

"Carefully balanced", numbers, science and complexity are unfortunately all dead; simple people watching cable news and social media demand yes/no answers.

Is government evil? Yes / No.
Is Big Tech/Pharma/Oil/... evil? Yes / No.

Afraid even the words "less" and "more" are gone.

Etc.

Scanning "private" content

giraffedata — Sun, 15 Aug 2021 16:46:56 +0000

In addition, the Fruit of the Poisonous Tree rule in the US applies only to searches by the government.

If I suspect my neighbor, so I break into his house and find a bloody shirt that ties the neighbor to a murder and I take that shirt to the prosecutors, they can use that in court.

Here's why: My neighbor already has protection against me doing this -- I can go to jail for it. He really doesn't have any protection against the government doing it; letting people in his situation get away with a crime seems to be the only effective incentive to the government not to do it.

If prosecutors asked me to break in (or even strongly suggested or encouraged it), that would be different.

Cloud = big brother

k8to — Sun, 15 Aug 2021 13:22:27 +0000

It's a false narrative for sure, though I think accurately described. Government can be influenced by your votes, organizing, and lobbying. Corporations there's no guarantee you have any influence at all.

Scanning "private" content

k8to — Sun, 15 Aug 2021 13:16:00 +0000

Err I meant the photos app,not camera.there are too many similar sounding apps.

Scanning "private" content

k8to — Sun, 15 Aug 2021 13:13:28 +0000

Google does this. At least the camera app pushes images to the cloud by default (backup features or some shit) and they definitely scan that data for CP.

I uninstall all the Google image related apps for this reason, but who knows when the mechanisms will change, so for they most part I don't keep any images on my phone. Who knows when the overreach will change or when something will be falsely flagged.

Scanning "private" content

k8to — Sun, 15 Aug 2021 11:04:16 +0000

These problems aren't theoretical either. "Protecting minors" had been used systematically to attack, harass, and incarcerate gay men by law enforcement for totally innocuous shit all over the USA.

I know people who were threatened with jail for you know, horrible things like crossing state lines to see a minor. You know, a 18 year old driving 30 minutes to hang out with a17 year old who considered themselves boyfriends.

There will always be an out group who can be attacked via stuff like this.

Democracy

mpr22 — Sat, 14 Aug 2021 16:58:07 +0000

> In the current USA, citizens are no longer guaranteed trial by jury, or even their right to life.

This phrasing suggests the existence of some prior state of the union in which they were.

And, well.

Neither the Federal government nor any state in the union has ever seriously considered forbidding the use of lethal weapons by law enforcement officers.

Cloud = big brother

BirAdam — Sat, 14 Aug 2021 15:18:53 +0000

A lesser evil is still evil.

Democracy

BirAdam — Sat, 14 Aug 2021 15:14:15 +0000

I could easily be convinced that 18 is too young for a person to be trusted to vote. I could equally easily be convinced that 14 is old enough. This depends entirely upon the person in question and his/her relative life experience, intelligence, and sense of caution. When rules are made for a large society, any type of individual consideration is removed due to expediency.

Democracy is not good in itself. Many dictators and horrible people have been elected to office. In the current USA, citizens are no longer guaranteed trial by jury, or even their right to life. Selling lose cigarettes? Death penalty. Your father was a US citizen but also Muslim? Death penalty (Abdul Rahman and Nawar al-Awlaki).

Part of my dislike for Apple’s move here is based upon these exact considerations. Even in supposedly free countries, freedoms are frequently done away with in the interest of stopping something horrible. Today, this is child exploitation which is far from well defined. Tomorrow, it will be the even more weakly defined “terrorism,” where terrorist is truly just whomever the state deems bad.

Scanning "private" content

jkingweb — Fri, 13 Aug 2021 22:26:36 +0000

Moreover, it's why we need open hardware, which is becoming less common with the passage of time, not more.

Democracy

jkingweb — Fri, 13 Aug 2021 22:25:03 +0000

> Among obvious groups that still ought to be enfranchised: Convicted criminals from prison -- at least all those convicted of crimes unrelated to the functioning of democracy, and frankly probably those too unless you're bad at prisons and can't keep them from tampering with the vote from inside a prison

Here in Canada this is happily already the case after the Supreme Court deemed the restrictions of the time unjustified (though not unjustifiable: the government simply failed to make their case).

> Children -- certainly all teenagers and there's no particular reason not to enfranchise any child that seems to actually have a preference, their preferences certainly can't be less _informed_ than those of adult voters so why not?

Ensuring that a toddler can vote secretly, in safety, and free from coercion by their guardians all at the same time would probably be a challenge, but I agree 18 years is definitely not a magically appropriate age.

Scanning "private" content

dskoll — Fri, 13 Aug 2021 21:15:22 +0000

Huh, interesting. So what about people who already own iPhones? Can they get a refund because Apple has fundamentally changed the terms of the agreement? (I doubt it, sadly.)

What if Google also did this? Realistically, if you're going to own a cell phone, it's either going to be IOS or Android. Remaining devices have a minuscule market share.

Scanning "private" content

NYKevin — Fri, 13 Aug 2021 16:59:38 +0000

Nope, they're going to rely on the old standby, the third-party doctrine.

Historically, the rule was that you didn't have any legally-defensible privacy interest whatsoever in any information which you voluntarily disclosed to someone else. In Carpenter v. United States, the Supreme Court said hey, wait a minute, that's going to allow all sorts of mass surveillance of people's location data, so maybe we should make an exception for pervasive monitoring that can't be reasonably avoided. But iCloud is definitely not going to fall into that exception, because you have the option of not using it (perhaps surprisingly, the Supreme Court was prepared to recognize that most people don't realistically have the option of *not* owning a cell phone, but they are not going to extend that to iCloud specifically).

Scanning "private" content

mathstuf — Fri, 13 Aug 2021 14:01:24 +0000

> The reason that the fact that they're using a (effectively) government provided list doesn't cause a problem is because they have a human manually review the content (which is why Apple includes that step-- without it the 'search' wouldn't happen until the NCMEC opened the files, which would require a warrant).

According to this post[1], 18 U.S.C. § 2258A is clear that the *only* legal way to transmit suspected CSAM is to NCMEC. Not the FBI, not the local police, and certainly not Apple. Apple sending themselves content which has a "1 in a trillion" (not that I believe they have the numbers to back up such a claim, but let's go with their PR here) chance of *not* being CSAM is blatantly illegal here.

[1] https://www.hackerfactor.com/blog/index.php?/archives/929...

Scanning "private" content

flussence — Fri, 13 Aug 2021 13:40:49 +0000

While I agree with the premise that horrible people use iPhones (the ultra-wealthy have plenty of hush money), I question why they went to these lengths to work with a government that practically gloats about how it abducts children by the thousands in broad daylight and disappears them into battery farm conditions. Maybe Apple's long game is hoping to catch a few of the officials responsible for that? That's my optimistic interpretation anyway.

Scanning "private" content

dskoll — Fri, 13 Aug 2021 12:55:23 +0000

OK, but if someone is charged based on what Apple finds, wouldn't Apple's evidence be inadmissable since it was obtained without a warrant? The United States has the Fruit of the Poisonous Tree doctrine, so I can certainly envision a situation in which Apple makes it easier for someone to dodge possession of child pornography charges simply because of the tainted evidence.

I'm not a lawyer, but would be very interested to hear what lawyers have to say about this.

Cloud = big brother

kleptog — Fri, 13 Aug 2021 11:39:00 +0000

> Changing government is more difficult.

Is it though? I have three countries within two hours travel which I can move to if I wanted to. Getting away from local or provincial government is even easier. Getting away from Google, Amazon, Apple, etc OTOH...

For both government and business trust needs to be earned. And frankly I trust my government way more than Google, though I can understand this doesn't apply to everyone. And the US political system is particularly... opaque.

Scanning "private" content

james — Fri, 13 Aug 2021 09:12:40 +0000

Most criminals, like the rest of us, are not particularly technically proficient. Catching the incompetent ones is worthwhile -- ideally, because it frees up resources to concentrate on the competent ones.
EU governments are quite capable of coming up with "secure" end-to-end communications for criminals to use where the police can evade the encryption: see Encrochat. It's actually quite difficult to protect against that.
One of the problems with secret communications between criminals is that it is only as strong as its weakest link, which is likely to be the least technical member: the ones who will copy "interesting" images onto their phones.

Cloud = big brother

alfille — Fri, 13 Aug 2021 02:52:34 +0000

There is certainly some truth to the thesis that government is more distrusted in US culture than in Europe. The counter to that is that being subjected to a corporate choice is usually not compulsory, you can vote with your wallet and move to another platfom. Changing government is more difficult.

And the motives of a corporation are clearer -- profit. What is the motive for a government? Staying in power, control, and a complex mix of competing lobbying groups.

At least that is the view of distrustful Americans.

Scanning "private" content

NYKevin — Fri, 13 Aug 2021 01:20:33 +0000

> Nobody _even at Apple_ has an insight into what the actual images are,

To be fair, under US law, it is illegal for Apple to possess copies of the actual images. I share your concerns about transparency here, but there are legal limits to what Apple can do.

> They feel they ought to do something, and this is something, so they feel they ought to do this.

I think you're short one level of abstraction. It's not necessarily the case that Apple themselves believe this. They may know perfectly well that this is a Bad Idea, but nevertheless promote it to take the wind out of the sails of some of the anti-encryption arguments. They may believe (rightly or wrongly) that the perfect ("nobody ever looks at the user's data") is the enemy of the good ("the user's data is only looked at under very narrow circumstances, and is otherwise E2E encrypted and beyond reach"). But of course, we won't know if that is actually true unless and until Apple introduces E2E encryption for iCloud (which as I understand it, is currently not a thing). Nevertheless, the system does seem to be very conveniently designed in such a way that it won't break if E2E encryption is introduced later (because it happens on the client, and not on Apple's servers).

Bear in mind, if Apple doesn't do *something*, then sooner or later one of the big governments is going to simply outlaw E2E encrypted user products altogether. They may see this as the least-worst alternative to that. Of course, that doesn't mean they're right, just that they probably do have a fairly comprehensive understanding of what they are actually doing and how it plays in the broader political context.

Scanning "private" content

tialaramex — Thu, 12 Aug 2021 22:55:26 +0000

In fact Apple trust "a database of known CSAM image hashes provided by NCMEC and other child safety organizations".

While NCMEC gets a name check, "other child safety organizations" is entirely for Apple to define. Nobody _even at Apple_ has an insight into what the actual images are, and if they allow some unnamed "child safety organization" to submit one or more hashes that organization gets indirect access to Apple's devices and users.

The "verification" step also is concerning. These are perceptual hashes, meaning they recognise images that are similar to a target in a way determined by an algorithm, likely a proprietary algorithm. So, unavoidably there can be false positives and you must actually verify matches. Apple of course doesn't have the original images, and certainly doesn't want to hire people to look at probably illegal images, so instead this is done by sending the matching image off to the same organizations which generated the hashes... The effect is that it may be possible to provide Apple with hashes that match most photographs of a dissident's face, and then you'll be sent any new photographs of that dissident for "verification". You can confidently inform Apple that these were false positives, not CSAM, and needn't be blocked, and of course Apple have no way to to determine whether you kept a copy of the images or acted on what you saw...

It's probably more reasonable to think of Apple and other proponents of these systems as useful idiots rather than part of a conspiracy. They feel they ought to do something, and this is something, so they feel they ought to do this. They believe that because they have good intentions, the arguments against this system are null - they know what they're doing is good. The problem is that they in fact cannot know whether what they're doing is good, the system intentionally blinds them and that ought to be a red flag.

Even the wider campaigns on this topic are most likely from useful idiots rather than deliberate enablers. Their idea is that if we can stop copies of an image from existing the thing depicted in the image is erased too, this is an obviously false belief about the universe but it's actually really commonly wished for. In many countries some such CSAM databases include images of events that never took place, or of children who never existed, they're of no value for prosecuting a hypothetical perpetrator of sexual assault because the offence never actually occurred - it'd be like showing video of a popular "death match" first person multi-player video game as evidence for a murder charge - but whereas you won't anywhere with that, prosecuting possession of a copy of these image really means somebody goes to jail to "protect" the imaginary children from imaginary abuse. Like prosecuting vagrancy this seems like an obviously bad idea but is really popular.

e2e encryption

gmaxwell — Thu, 12 Aug 2021 21:54:55 +0000

That's a little like arguing that e2e encryption wouldn't be broken by requiring everyone only use the encryption key "password". :) It's technically true, but it undermines the purpose of the encryption.

Scanning "private" content

gmaxwell — Thu, 12 Aug 2021 21:53:05 +0000

They put a tremendous amount of engineering into it to cryptographically shield themselves and their list providers from accountability, too. I find this extremely concerning.

The obvious construction for such a system would simply deliver to the DRM-locked-down devices a list of hashes to check against and self-report. But that construction would make it possible for people to perform a web crawl and check it against the list, which would have a fighting chance of exposing instances of inappropriately included popular images. It would make it much harder to secretly use the system to target particular religions, ethnicity, or political ideologies...

But, instead, the use a complex and computationally expensive private set intersection to make sure that the users can't learn *anything* about the list they're being checked against except an upper bound on its size.