Image "Cloaking" for Personal Privacy

[Posted July 22, 2020 by jake]

SAND Lab at the University of Chicago has announced Fawkes, which is a BSD-licensed privacy-protection tool available on GitHub. "At a high level, Fawkes takes your personal images, and makes tiny, pixel-level changes to them that are invisible to the human eye, in a process we call image cloaking. You can then use these "cloaked" photos as you normally would, sharing them on social media, sending them to friends, printing them or displaying them on digital devices, the same way you would any other photo. The difference, however, is that if and when someone tries to use these photos to build a facial recognition model, "cloaked" images will teach the model an highly distorted version of what makes you look like you. The cloak effect is not easily detectable, and will not cause errors in model training. However, when someone tries to identify you using an unaltered image of you (e.g. a photo taken in public), and tries to identify you, they will fail."

Image "Cloaking" for Personal Privacy

Posted Jul 22, 2020 22:53 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (5 responses)

It's BSD-licensed, yes, but the "copyright" section of their README says this:

> This code is intended only for personal privacy protection or academic research.
>
> We are currently exploring the filing of a provisional patent on the Fawkes algorithm.

I'm going to assume the first line is just a standard CYA "there's no warranty" disclaimer, and not an actual condition on use (because it would flatly contradict the LICENSE file). However, the patent is a great deal more alarming, and in fact, I'm not sure I can recommend using this thing as long as that sentence remains there. It basically amounts to "You can do what you like with our software, but we could turn around and sue you at any time, once the USPTO rubber stamps our patent."

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 5:37 UTC (Thu) by epa (subscriber, #39769) [Link] (3 responses)

If they have already published the software but not yet filed the patent, surely that means it’s too late? The patent office will probably grant it anyway but it would be invalidated in court. Or is that not how it works?

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 5:56 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

You can file for a patent for up to a year after a publication or any other form of unveiling of the invention.

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 8:46 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

Only in the US ...

Cheers,
Wol

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 19:27 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

For better or for worse, most of the major tech companies are incorporated in the US, so the industry as a whole has no choice but to care about this problem.

Image "Cloaking" for Personal Privacy

Posted Jul 24, 2020 13:57 UTC (Fri) by t-v (subscriber, #112111) [Link]

And they don't comply with "reproduce copyright notice" for the stuff they use from third parties... Sigh. It's really hard to copy MIT licensed code and not screw up apparently, even though what the license requires might have been common courtesy even if it didn't.

Regarding patent, I must admit I have a hard time telling what the real innovation compared to the now classic adversarial example work (of the Goodfellow... and Carlini... variants implemented in CleverHans) is except uploading the image somewhere. That may or may not mean that one would find prior art.

Image "Cloaking" for Personal Privacy

Posted Jul 22, 2020 23:46 UTC (Wed) by FLHerne (guest, #105373) [Link] (9 responses)

> The cloak effect is not easily detectable, and will not cause errors in model training. However, when someone tries to identify you using an unaltered image of you (e.g. a photo taken in public), and tries to identify you, they will fail.

This assumes not only that all existing facial-recognition systems are vulnerable to their specific tweaking approach, but that future ones will be too. The modified photos will be out there indefinitely.

A lot of research is being done on solving this class of weakness -- it's a serious problem for self-driving vehicles too, there've been demonstrations of minor changes to street signs that lead to completely different recognition outcomes.

While their list of current systems fooled is impressive, I think the absolute assurance of privacy given here is unwarranted.

street signs vs. faces

Posted Jul 23, 2020 1:53 UTC (Thu) by gus3 (guest, #61103) [Link] (8 responses)

Street signs tend to have hard image borders. Muck with any of those borders, and any algorithm using them will get knocked about.

Images of human faces are a mix of hard borders and soft shading. A proper facial-recognition system doesn't depend on these; it uses the points of the face (eyes, nostrils, lips, visible teeth, ears, visible hair-line, jaw, cheekbones, musculature) to build a face it can recognize.

The hackers and crackers already have tools against the Fawkes system. The images aren't cloaked, no matter how much you want them to be.

Remember: the enemy always has the better hand. It's your job to close the gap between the enemy's hand and yours.

street signs vs. faces

Posted Jul 23, 2020 2:17 UTC (Thu) by felixfix (subscriber, #242) [Link]

It may be more realistic to say the the enemy has the better hand against bulk data. Individuals who invest some effort in differentiating themselves from the bulk data may hide from the bulk processor. But the very fact of having differentiated themselves may also make them stand out.

street signs vs. faces

Posted Jul 23, 2020 7:08 UTC (Thu) by smurf (subscriber, #17840) [Link] (3 responses)

There are lots of mucking-with-street-signs examples out there that don't touch the borders.

Their point isn't to make the image in the manipulated photos unrecognizeable, but to make it not-your-own. The problem I see with their idea is that as soon as there are two sets of images of you out there, any adversary worth their salt will not simply replace the old parameter cloud with the new, as the Fawkes authors assume, but split them off into two sets of clouds which are both recognized as "you".

So this probably works WRT shop surveillance systems that try to find who that repeat customer is, might conceivably defend against run-of-the-mill police surveillance cameras if you can get your passport photos replaced with a Fawkes pic (more difficult as authorities start to insist on taking the pics themselves instead of you walking in with one from the photo booth), but not at all when the opponent is the NSA and their ilk.

street signs vs. faces

Posted Jul 23, 2020 8:44 UTC (Thu) by Sesse (subscriber, #53779) [Link]

All the street sign examples I've seen involve pretty heavy tampering with the signs (e.g. several measures of thick tape). Obvious enough that you could really just as well write “80” where it says “30” instead.

street signs vs. faces

Posted Jul 23, 2020 12:15 UTC (Thu) by ibukanov (subscriber, #3942) [Link]

In Norway police has been taking passport photos themselves for at least 5 years in big cities. One cannot bring own photos at all. But Norwegian embassys still ask to bring own photos both when they issues passports and visas.

street signs vs. faces

Posted Jul 23, 2020 19:31 UTC (Thu) by nilsmeyer (guest, #122604) [Link]

This can also throw a wrench in systems that scrape data from social media to build/train their facial recognition software.

street signs vs. faces

Posted Jul 23, 2020 14:13 UTC (Thu) by clump (subscriber, #27801) [Link] (2 responses)

Street signs were designed for humans to understand. This will continue to be the case for a long time. As autonomous vehicles become popular it would be a good idea to augment human-readable signs with machine-friendly identifiers. Why not add a small RF box, or some kind of id that can be scanned by machines?

By all means, continue to learn to read symbols designed for humans. However it should be relatively inexpensive (and safer) to tell a machine "this is a stop sign".

street signs vs. faces

Posted Jul 23, 2020 15:52 UTC (Thu) by magfr (subscriber, #16052) [Link]

We all know that whenever the same information is encoded twice there will inevitably be a discrepancy.

street signs vs. faces

Posted Jul 24, 2020 15:04 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link]

If you add machine-friendly identifiers, you'd better make sure they have the same kinds of legal rules surrounding them that human-readable signs do. Otherwise malicious actors will be able to mess with the system with legal impunity. It could be very bad if people could create new traffic signs only autonomous vehicles knew about.

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 19:18 UTC (Thu) by jem (subscriber, #24231) [Link] (6 responses)

This bites yourself in the arse if you have a cloaked image of yourself in your passport, and the unaltered image of you is one taken by a camera at an automated border check booth.

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 19:28 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (5 responses)

Yeah you might get into a lot of trouble with that too. In that case you would just go to the booth with a person behind it (and spend a lot of time in a queue) - if available.

Image "Cloaking" for Personal Privacy

Posted Jul 23, 2020 19:56 UTC (Thu) by sjj (guest, #2020) [Link] (4 responses)

Why and how would you get in trouble?

Image "Cloaking" for Personal Privacy

Posted Jul 24, 2020 3:44 UTC (Fri) by gdt (subscriber, #6284) [Link] (1 responses)

Typically when you supply a digital photo for use on your passport you state that the image is a "true likeness". Whereas the image has in fact been altered. Obviously the detail varies by jurisdiction but a "false declaration on the passport application" would be the approach.

Image "Cloaking" for Personal Privacy

Posted Jul 24, 2020 9:56 UTC (Fri) by nilsmeyer (guest, #122604) [Link]

I remember people using microwaves to fry the RFID in their passports or ID cards. I don't necessarily mean by "trouble" that you'll be convicted of a criminal offence, but you may be severely inconvenienced, perhaps even arrested or detained.

We already know this happens to people based on other criteria. It also happens a lot where facial recognition is used, especially if you have a darker skin tone where facial recognition causes a lot more false positives.

Image "Cloaking" for Personal Privacy

Posted Jul 24, 2020 9:44 UTC (Fri) by nilsmeyer (guest, #122604) [Link] (1 responses)

The situation I had in my head was trying to enter another country through an airport for example. If the facial recognition doesn't work someone may suspect that the passport is fake. "Trouble" doesn't necessarily mean that you end up in prison, but at the very least you could be detained and harassed at the airport.

Image "Cloaking" for Personal Privacy

Posted Jul 24, 2020 19:08 UTC (Fri) by NYKevin (subscriber, #129325) [Link]

It is important to remember that, when you are entering a country other than your own, you (usually) do not have the legal right of entry, even if you are fully compliant with all of the legal requirements (and depending on the country, possibly even if you already have a visa, although that is admittedly a rare outcome). If they decide they don't trust you, most countries will simply put you back on the plane and turn it around. If you're inadmissible in the departure country too (e.g. you entered on a single-entry visa, the border officials don't like you, etc.), then things get messy (but you will *probably* make it back to your home country, eventually, and the airline will certainly bill you for their trouble). If you have multiple citizenships, this can get really fun, since you might get sent to the wrong "home country" and have to arrange further transportation yourself. Even when you are entering your home country, you may have difficulty asserting your right to entry. If they don't believe you're a citizen, proving it is generally your problem.

So, in my opinion, doing anything that makes a passport look less valid in the eyes of border officials is a really bad idea, regardless of whether doing so is technically legal or not.