Another fun DMCA case [LWN.net]

Attorney William Patry's weblog looks at a new DMCA case, which, at its core, is claiming that a failure to heed a web site's robots.txt file is a circumvention of a technical copyright protection measure. "Those who decry the DMCA as an (attempted) tool of oppression will find more than ample support in this effort."

to post comments

Another fun DMCA case

Posted Jul 12, 2005 16:46 UTC (Tue) by ksmathers (guest, #2353) [Link] (32 responses)

I think the law actually reads "effective copy-prevention technology" (which you should read as technology which has the effect of preventing copying, whether originally intended for that purpose or not), which probably covers robots.txt within the intent of the DMCA.

The robots.txt file lists spider applications and the path names which they may not access, so it is clearly an effective copy-prevention technology. The case will probably turn on whether the technology was really circumvented or was merely accidentally ignored due to some programming error.

Another fun DMCA case

Posted Jul 12, 2005 17:38 UTC (Tue) by kokopelli (guest, #11341) [Link]

The purpose of the robots.txt file is NOT to list directories robots shouldn't access, much less "may" not access. It is to identify pages that shouldn't be indexed because they're dynamically generated or otherwise unsearchable. That means indexes would be useless since the pages they index would be different on the next visit.

In practice the file has taken on the secondary meaning of listing directories that should not be scanned at all. Every so often there's a story about some moron who put his sensitive information on the web but "hid" it in the robots.txt file, but that's as misguided as driving around town for an hour so you can fry some eggs sitting on the engine.

In either case it's purely advisory.

Another fun DMCA case

Posted Jul 12, 2005 17:40 UTC (Tue) by JoeF (guest, #4486) [Link] (10 responses)

Since the robots.txt is a) just a recommendation and b) reading it is not a prerequisite to access a site, there is in my non-lawyer view no case. Not reading robots.txt is not a circumvention. Not reading is is not a "programming error," either. It is completely voluntary to read it or not, as well as it is completely voluntary to follow the recommendation in it or not.

Another fun DMCA case

Posted Jul 12, 2005 18:56 UTC (Tue) by MathFox (guest, #6104) [Link] (7 responses)

I agree with you that the robot.txt file is not a "technical protection measure" against users that visit the website with a browser. On the other hand it serves as a "spiders not allowed here" sign for attempts to copy entire sites. There have been several convictions for "computer tresspassing" where companies spidered websites with disregard of the robots.txt file.

Another fun DMCA case

Posted Jul 12, 2005 19:27 UTC (Tue) by jhardin@impsec.org (guest, #15045) [Link] (6 responses)

> There have been several convictions for "computer tresspassing" where
> companies spidered websites with disregard of the robots.txt file.

And that is the proper way to treat it. "robots.txt" is *not* a copyright-enforcement technology and no amount of contortion can make it one.

Another fun DMCA case

Posted Jul 12, 2005 19:49 UTC (Tue) by job (guest, #670) [Link] (5 responses)

If robots.txt isn't a copyright enforcement tool, then what is? How do
you argue that DVD CSS or other kinds of DRM is one? (Which is basically
what the law was designed to protect.)

Another fun DMCA case

Posted Jul 12, 2005 20:04 UTC (Tue) by rfunk (subscriber, #4054) [Link]

robots.txt is like "keep out" written in a brochure inside the house.
CSS is like a cheap lock on the door.

Another fun DMCA case

Posted Jul 12, 2005 20:30 UTC (Tue) by jhardin@impsec.org (guest, #15045) [Link]

> If robots.txt isn't a copyright enforcement tool, then what is? How do
> you argue that DVD CSS or other kinds of DRM is one?

They are *very* different.

DVD CSS and other types of DRM employ encryption and (supposedly) strong access controls on the keying material used to decrypt the protected content.

robots.txt is not intended to be used by interactve browsers, only webcrawlers, and only to indicate content that should not be indexed.

In the context of a website, the closest you could get to a "copyright enforcement" tool would be to password-protect access to the website. *then* you might be able to argue that attempting to crack the password is a DMCA violation.

Security through obscurity is inherently broken. Trying to enforce it through legislation does not fix it, it only makes you look like an idiot.

Another fun DMCA case

Posted Jul 13, 2005 11:37 UTC (Wed) by rev (guest, #15082) [Link] (2 responses)

Use your little brain please..

The situatuions are only slightly asymmetrical.

You have to decrypt an encrypted DVD to gain access to the copyrighted material on the DVD. I.e. you do have to make an effort to gain access to the copyrighted material.

You don't have to read robots.txt to gain access to the copyrighted material on a website.

How can robots.txt qualify as a copyright enforcement tool if you have to do nothing to gain access to the copyrighted material? Please explain..

Another fun DMCA case

Posted Jul 19, 2005 9:22 UTC (Tue) by job (guest, #670) [Link] (1 responses)

Please refrain from ad hominem attacks in this forum, which is otherwise so spared from it. I had a point to make and I would only be grateful to explain the reasoning further:

I understand there is a difference of intent. You can much easier claim that you had no idea the web site content was forbidden (it usually isn't) than that the DVD video was (it usually is, codified by CSS). While this will make a big difference in court it does not affect the actual wrongdoing.

By the logic of your own reasoning, you may well argue that you bought a DVD to actually watch it or an e-book to read it. We already know that is a wrongdoing in the broad sense. In the line of the content providers, you are not allowed to "read" to book or "watch" the video -- you only have a license to access the information with a legitimate tool.

What these web content providers are doing is nothing different. They also only allow you to access the material with a legitimate tool of their choice, a "regular" web browser. Whether this is codified in law or code does not matter to them. (And it should not, in the general sense, much as a break-in is a break-in even if the door is unlocked.)

That is why this is similar to DVD CSS. It may be an unjust law (I don't live there) but it is the same reasoning covering both cases, DRM codified in law.

Another fun DMCA case

Posted Jul 24, 2005 12:43 UTC (Sun) by rev (guest, #15082) [Link]

You are confused on what an ad hominem argument is.

I did mix in some ridicule, yes. But I did make a solid argument in observing that the two situations are quite different. The ridicule follows as a conclusion to the observation that someone is able to argue that both situations are equivalent.

An ad hominem argument is when one attacks the person without arguing the position the person defends.

Another fun DMCA case

Posted Jul 13, 2005 5:42 UTC (Wed) by ksmathers (guest, #2353) [Link] (1 responses)

Not reading the robots.txt would be hard to show to be a violation, since you can't circumvent something you don't recognize in the first place. If on the other hand my web log indicates that you pulled down robots.txt and then proceeded to access locations that were restricted for your client then that be like accessing any computer or site without authorization. Copies obtained under those circumstances would probably be considered unlicensed and if in discovery I could show that you had source code for a hacked version of 'wget' to ignore the results of the robots.txt file, then I could probably get you for the DMCA violation as you have taken affirmative steps to bypass an effective technological copy prevention mechanism in order to copy my site.

Reading through the complaint, I'm guessing that the company in question somehow thought that the change to their robots.txt file would be instantaneous and were surprised when the content was still available from archive.org the next day. There is a way to push a request for deletion to Archive, but it is by e-mail request, not by use of the robots.txt file.

Another fun DMCA case

Posted Jul 13, 2005 17:04 UTC (Wed) by TwoTimeGrime (guest, #11688) [Link]

> If on the other hand my web log indicates that you pulled down robots.txt
> and then proceeded to access locations that were restricted for your client
> then that be like accessing any computer or site without authorization.

Nonsense. robots.txt doesn't imply authorization. It has nothing to do with access controls. If you don't want pages on your site read then put access controls in place or remove the content.

Another fun DMCA case

Posted Jul 13, 2005 5:00 UTC (Wed) by jwb (guest, #15467) [Link] (6 responses)

Riddle me this, ksmathers.

Suppose I say to example.net "GET / HTTP/1.0" and example.net says "HTTP/1.0 200 OK" and proceeds to send me a string of octets. In this scenario, who has made the copy?

Another fun DMCA case

Posted Jul 13, 2005 5:55 UTC (Wed) by ksmathers (guest, #2353) [Link] (5 responses)

As a rule, the person who conducts themselves in bad faith is likely to be held liable for the copyright infringement. If you walk into Kinko's and pretend that you are the author of some document and have them copy it for you, it is you who are liable for the infringement. If Kinko's knew or should have known that you were not the author (for example your name doesn't match the name on the cover) then they are liable.

I've had friends who are professional photographers go around to the local print shops on a regular basis to show the employees their copyrighted work with the dual goal of keeping them from copying the images, and ensuring that they will be held liable if they copy them anyway.

Another fun DMCA case

Posted Jul 13, 2005 7:06 UTC (Wed) by jwb (guest, #15467) [Link] (1 responses)

You misundestand my point. In this scenario, it is actually impossible that I made the copy, because I was never in possession of the original. The maximum number of instances of the work which I simultaneously possessed was 1. The server, on the other hand, is the party which made the copy. It has the original, and at one point in time it was in possession of 2 instances.

It's not like asking Kinkos to copy copyrighted photographs. It is like walking into photo bureau, and asking politely for a copy. Then as you leave they have you arrested.

Another fun DMCA case

Posted Jul 13, 2005 18:38 UTC (Wed) by ksmathers (guest, #2353) [Link]

I don't know why you think it is important to have been in posession of the original -- that part is completely immaterial. That said, if your use is a fair use, and you haven't been infringing the copyright on the work, then obviously you don't have anything to worry about as regards copyright law. Of course the DMCA can sometimes be invoked even when copyright law can't.

In the case in question though, Archive had copied and distributed the pages they had collected from the site alleging harm, so there is basis for both copyright and DMCA action. I don't think the DMCA part will hold up though unless Archive took some affirmative action to bypass the robots.txt file.

(The statutory maximums for the copyright violation by itself probably wouldn't even make it worthwhile to go to court.)

Another fun DMCA case

Posted Jul 13, 2005 9:36 UTC (Wed) by man_ls (guest, #15091) [Link] (2 responses)

I've had friends who are professional photographers go around to the local print shops on a regular basis to show the employees their copyrighted work with the dual goal of keeping them from copying the images, and ensuring that they will be held liable if they copy them anyway.

You are kidding, right? This gives a new dimension to the exercise of covering your ears and singing "Lalalala I'm not listening lalalala."

Another fun DMCA case

Posted Jul 13, 2005 18:19 UTC (Wed) by ksmathers (guest, #2353) [Link] (1 responses)

That would be hilarious... I can imagine it now.

Plaintiff: Your honor, plaintiff aleges defendant knew or should have known that the works in question were protected by copyright, as established by the documented fact that on at least three occasions plaintiff visited defendant's place of business and showed examples of the works.

Defense: Defendant objects your honor, there is insufficient basis for the fact that defendant saw the works in question. Defendant held his hands to his ears, closed his eyes and said: "La, la, la, la, I'm not listening on each of those three visits until plaintiff had departed...", so you see...

I would see an idiot.

-kls

Wilfully or will-lessly

Posted Jul 13, 2005 19:48 UTC (Wed) by man_ls (guest, #15091) [Link]

Well, the example of covering your ears and singing was not to be taken literally. But I can imagine the lines of the defense: defendant was not in the shop / did not look / did not even want to look / did not pay attention / does not remember.

The whole thing looks so ridiculous... Maybe it's the fact that under Spanish law, showing your works around would be completely pointless. While intention is obviously important in penal law (it is what tells apart homicide from murder), it is not for civil liability; this "bad faith" stuff must be something left over from the Reform and that pesky "free will" doctrine. Here you are responsible for your actions in the same degree and fashion regardless of your intention.

IANAL, although I have consulted this specific point with one.

Another fun DMCA case

Posted Jul 13, 2005 14:02 UTC (Wed) by pyellman (guest, #4997) [Link] (12 responses)

ksmathers,

robots.txt was not designed as a copy-protection mechanism, has never been used as such, and therefore it cannot be argued that it is a copy-protection mechanism, effective or otherwise. Read this http://www.robotstxt.org/wc/norobots.html.

robots.txt is simply a suggestion. It is no more an effective copy protection "technology" than putting something at the bottom of your home page that say "Oh, by the way, we'd appreciate it if you wouldn't copy or archive the following pages on this web site: etc., etc."

Peter Yellman

Another fun DMCA case

Posted Jul 13, 2005 18:07 UTC (Wed) by ksmathers (guest, #2353) [Link] (11 responses)

Look, I'm just trying to help you make the paradigm shift from the technical arena to the legal arena. I really don't give a damn whether you think the law is absurd or not, but you are splitting technical hairs in a area where the law is pretty plain.

The DMCA doesn't reference copy protection measures as such, instead it says effective copy protection measures. This includes technical measures which although not intended to protect copyright, never the less have that effect.

The classic example is altering the vertical synch signal on a television broadcast so that an unmodified receiver is unable to lock synch. Vertical synch was never intended as a copy prevention mechanism, but altering that synch has the effect of preventing the copy from being viewed, and is thus an effective copy prevention measure.

Another fun DMCA case

Posted Jul 14, 2005 1:00 UTC (Thu) by pyellman (guest, #4997) [Link]

Fair enough. I guess I'm trying to help you understand that robots.txt is not a copy protection mechanism, effective or otherwise. I've often used the butt handle of a screwdriver to pound on stuff, but I still know the difference between a screwdriver and a hammer.

Peter Yellman

Another fun DMCA case

Posted Jul 14, 2005 11:42 UTC (Thu) by rev (guest, #15082) [Link] (9 responses)

Look ksmathers, the posters are trying to get it into your skull that robots.txt does not constitute a copyright protections scheme. The point is, it is a far cry from being mandatory. The point is in any proper copyright protection scheme a wannabee infringer would find himself to need to put effort into gaining access to the material. If a pretended copyright protection scheme does not require any effort at all to gain access to the 'protected' material, but instead requires one to put effort into *NOT* gaining acces to the material, as the robots.txt convention does, it is not a copyright protection scheme at all, much less an effective one.

That is the point ksmathers. Not too difficult to get into the synapses for anyone with minimal analytical skills, now is it?

Another fun DMCA case

Posted Jul 14, 2005 14:42 UTC (Thu) by ksmathers (guest, #2353) [Link] (4 responses)

I don't know why you think that the robots.txt usage has to be mandatory. Whether the control is mandatory, optional, accidental, a matter of convenience, or used for any other reason doesn't play into the law.

Regarding making an affirmative effort, I've already written the same thing myself elsewhere in this discussion. You seem to think that just because the use of robots.txt is optional that this means that no one could show that you had made a special effort to avoid the use of robots.txt. That simply isn't true.

There are several things that would tend to imply to a court that you had made an effort to bypass the control. For example, altered source code on your computer that disables the robots check in some tool which normally uses it, statements from business partners, neighbors, or aquaintances characterizing your activities, suddenly changing a previous business practice, and logs indicating that the robots file was sent but was not respected.

In each of these cases you'd have to explain why you had taken the steps you did, or why the evidence was in error, and either a judge or a jury would decide which was more believable. Coming into court and claiming that 'it wasn't a copy protection mechanism, so bypassing it was okay', would only be a quick way to get yourself a negative judgement.

Another fun DMCA case

Posted Jul 14, 2005 16:19 UTC (Thu) by rev (guest, #15082) [Link] (3 responses)

"I don't know why you think that the robots.txt usage has to be mandatory. Whether the control is mandatory, optional, accidental, a matter of convenience, or used for any other reason doesn't play into the law."

Well, I am curious to see evidence supporting this claim, as your interpretation of "copyright protection scheme" defies common sense. Moreover, the consequences of your interpretation are absurd: I place a file called blahblah.txt somewhere on my web site containing a list of filenames though shalt not HHTP GET from my web site. You neverthless do. You, you dirty copyright infringer, I sue the hell out of you.

I would be interested you quoting the law and/or case law supporting your claim.

Another fun DMCA case

Posted Jul 14, 2005 18:22 UTC (Thu) by ksmathers (guest, #2353) [Link] (2 responses)

Moreover, the consequences of your interpretation are absurd: I place a file called blahblah.txt somewhere on my web site containing a list of filenames though shalt not HHTP GET from my web site. You neverthless do. You, you dirty copyright infringer, I sue the hell out of you.

Well, the situation you describe wouldn't qualify as a DMCA violation unless there were a client/tool that normally reads that file and uses it to decide which files are accessible (the technological measure). Secondly you would have had to use that client/tool, and have bypassed the control in some way.

It seems rather simple to me, namely unless you have taken some affirmative step to bypass the control then you can't very well be liable for bypassing the control. If you did affirmatively bypass the control, then arguing that it isn't a control isn't very useful unless the control is truly absurd; that is unless you can guarantee that a judge or jury wouldn't be able to construe it that way, even with the help of a litigator clearly describing all of the reasons that it should be.

Another fun DMCA case

Posted Jul 14, 2005 19:16 UTC (Thu) by niner (guest, #26151) [Link] (1 responses)

But nobody said, that anyone put any effort in circumventing the "robots.txt copy control measure". There are only a handful of tools out there that even support robots.txt, as it's a file that has a very special purpose and this only in connection with internet search engines.

E.g. wget as one of the tools that support robots.txt even lets you switch off the support via a command line parameter: -e robots=off

The easiest and probably most used way to copy a webpage is to just use the "save page" feature of your webbrowser. An I know of no webbrowser that even recognizes a robots.txt file.

If I wanted to write a service like archive.org which crawls and saves webpages I'd do it in perl and would have to knowingly put effort in supporting something like a robots.txt. So how can you call something a "copy protection measure", if I have to know that you have it and what way you tell me how to behave and actually write code to behave like you want. My other option is to just do nothing and how can I have done something bad if I have done nothing?

Head over to Groklaw

Posted Jul 14, 2005 19:40 UTC (Thu) by ksmathers (guest, #2353) [Link]

Yeah, if the plaintiff can't show that there was any effort to defeat the copy control measure then they are probably hosed.

For what it's worth, Groklaw seems to have picked up the story, so if you'd like to read a lot of different points of view on the subject, I'd suggest heading over to Groklaw. By the way, PJ's view is that robots couldn't be considered a technical copy prevention mechanism. So there's at least one dissenting opinion for you.

Another fun DMCA case

Posted Jul 14, 2005 14:59 UTC (Thu) by pyellman (guest, #4997) [Link] (3 responses)

By repeatedly drawing attention to the term "effective" without addressing the common knowledge that compliance with robots.txt is purely voluntary and actually requires proactive effort, it almost seems that ksmathers is arguing for an interpretion of that word to mean "Operative; in effect", as in "your honor we were, in effect, using robots.txt as a copy protection mechanism" rather than what I think most people assume that term to mean, "producing a desired effect" or "efficacious". "We believe it's a copy protection mechanism, ergo it is"!

If such an interpretation were upheld, it would indeed be a significant "paradigm shift", as winning a DMCA case would be as easy as bringing one. Others have pointed out here on LWN that such an interpretation would lead to such things as "psychic fingerprinting" as a method of copy protection. I can see it: "Now, your honor, through the magic of the DMCA, we will turn this screwdriver into a hammer before your very eyes simply by calling it a hammer".

I doubt that ksmathers is alone in his interpretation, and I almost wish it were true, as the resulting total chaos would virtually ensure that the DMCA would be revisited by legislators.

Peter Yellman

Another fun DMCA case

Posted Jul 14, 2005 18:51 UTC (Thu) by ksmathers (guest, #2353) [Link] (2 responses)

Sorry, I really don't get your argument. Yes, I'm drawing attention to the term 'effective'. I disagree that compliance with robots.txt requires proactive effort. Let me rephrase that: I agree that not complying with robots.txt need not require proactive effort, but there are many uses of robots.txt which are compliant, and which are already encoded in common technologies for accessing web content (the 'wget' tool for example), and therefor require no effort on the part of the user in order to comply with robots.txt.

Whether it is voluntary to use robots.txt or not really doesn't make any difference. Almost all copy prevention technologies are voluntary. There is no law requiring VCRs to interpret brightness signals differently than television sets, but this difference is still the basis for Macrovision's copy prevention technology.

Another fun DMCA case

Posted Jul 14, 2005 20:21 UTC (Thu) by pyellman (guest, #4997) [Link] (1 responses)

You wrote:

> Regarding making an affirmative effort, I've already written the same thing myself elsewhere in this discussion. You seem to think that just because the use of robots.txt is optional that this means that no one could show that you had made a special effort to avoid the use of robots.txt. That simply isn't true.

and

> I disagree that compliance with robots.txt requires proactive effort.

and

>Whether it is voluntary to use robots.txt or not really doesn't make any difference. Almost all copy prevention technologies are voluntary. There is no law requiring VCRs to interpret brightness signals differently than television sets, but this difference is still the basis for Macrovision's copy prevention technology.

First of all, in regard to that last statement: Huh? In the first place, Macrovision apparently enjoys the unique pleasure of having a clause inserted into the DMCA just for its own benefit (http://www.unmediated.org/archives/2005/06/macrovision_in...). Second, are you really comparing the difficulty level of not implementing robots.txt with designing and building a new VCR?

You repeatedly suggest that the difficulty of avoiding a copy protection "scheme" (I'm not even going to use mechanism anymore) is irrelevant, but the rest of us are saying the opposite -- that that is the only rational interpretation of the meaning "effective". Get it? No "psychic fingerprinting".

Finally, if if what you say were true, various senators and the MPAA would not have found it necessary to introduce the Broadcast Flag amendment (which was struck down as an FCC regulation), as they (the MPAA) would have had all the authority they needed under the DMCA.

Peter Yellman

Another fun DMCA case

Posted Jul 14, 2005 22:02 UTC (Thu) by ksmathers (guest, #2353) [Link]

Well, I wish you good luck should you attempt a defense in the future with that approach.