LWN: Comments on "The WordPress mess"

Criminal?

Wol — Sat, 02 Nov 2024 15:16:19 +0000

> Nah, it's two companies fighting each other. I don't see why the government should get involved - they only do and should when the public gets harmed. And here, public doesn't really mean 'customers' either, after all - if you have a contract with WP Engine and they don't deliver, they are to blame. If they decided to depend on a 3rd party without a contract because they're trying to make their VC investors as much money as possible by contributing as little as possible to the ecosystem, well - their fault, you can sue them.

Except this is (allegedly) Microsoft versus Lotus / WordPerfect / NetScape etc all over again.

Lying or misleading your co-opetition is a serious market offense - called "monopolisation", and from what I can tell, this fits it to a T. The resulting damage to the public is serious, and even 30 - 40 years after the event I would say computing is still not recovered from the damage MS did. Do you really want to see the same long-lasting damage in the Wordpress arena?

The WP Engine claim is basically that Automattica told the markets one thing, and then failed to deliver, enticing their competition to rely on promises that meant nothing. In other words, blatant AntiTrust.

Whether a prosecutor wants to take those claims up, we'll have to see, Whether those claims will hold water, I don't know. But they are clear claims of AntiTrust, and Market Manipulation. Which are clear threats to the public.

Cheers,
Wol

Criminal?

jospoortvliet — Sat, 02 Nov 2024 10:22:56 +0000

Nah, it's two companies fighting each other. I don't see why the government should get involved - they only do and should when the public gets harmed. And here, public doesn't really mean 'customers' either, after all - if you have a contract with WP Engine and they don't deliver, they are to blame. If they decided to depend on a 3rd party without a contract because they're trying to make their VC investors as much money as possible by contributing as little as possible to the ecosystem, well - their fault, you can sue them.

Sadly this is the 'business model' of a lot of companies, leaching off of open source projects. Not sure what to do about it, we as community have to find some solution that doesn't mean going closed source or doing stupid shit like Automattic - tricky.

No mirrors?

TRS-80 — Thu, 10 Oct 2024 05:56:37 +0000

TensorFlow is responsible for 17% of data transferred from PyPI:

https://kristoff.it/blog/python-training-wheels/

The whole post is about the cost of PyPI and worth reading.

Risk management anyone?

james — Mon, 07 Oct 2024 10:56:54 +0000

Actually, Squid can "bump" or non-transparently MITM HTTPS traffic (and therefore cache it). In a CI environment, this might be a reasonable thing to do, if you consider Squid to be part of the same security domain as the CI environment.

Risk management anyone?

dskoll — Sat, 05 Oct 2024 14:16:22 +0000

I don't think confidentiality is an issue for CI/CD pipelines, especially for open-source products where anyone can just look at what exactly the CI/CD pipeline is doing.

Risk management anyone?

LtWorf — Sat, 05 Oct 2024 07:20:52 +0000

Unless your attacker can infer that the debian.org hostname is a mirror and uses that information to understand what you're downloading from the sizes of the files that get downloaded.

Risk management anyone?

intelfx — Sat, 05 Oct 2024 02:11:28 +0000

> If your packages are signed and you verify the signature, https doesn't buy you anything.

That's not strictly true. At the very least, transport-layer encryption buys you confidentiality.

Risk management anyone?

dskoll — Sat, 05 Oct 2024 01:08:40 +0000

Apt doesn't require https. In fact, all of my sources.list entries are http.

If your packages are signed and you verify the signature, https doesn't buy you anything. If a package is validly-signed, then it doesn't really matter where you downloaded it from.

Risk management anyone?

kleptog — Fri, 04 Oct 2024 21:40:53 +0000

Squid proxies only work for HTTP traffic. For HTTPS it doesn't help at all. Since all the package managers (rightfully) depend on HTTPS for security any caching cannot be transparent.

So even of GitLab of Azure DevOps wanted to provide caching for npm or PyPi, they couldn't do it in a way that's transparent. And if they provide a non-transparent mechanism, it makes it a potential MITM.

Ideally there'd be an extension to HTTPS to allow clients to opt into caching, while still preserving the authentication properties of HTTPS, but I think the ship has sailed on that one.

Risk management anyone?

LtWorf — Fri, 04 Oct 2024 08:22:53 +0000

In my own experience, it's most usually few organizations spamming, rather than many organizations. Easy to see because overnight the daily downloads might halve or double. And I doubt it's because thousands of organizations all went to use a different library all in the same day.

Criminal?

yeltsin — Fri, 04 Oct 2024 01:00:48 +0000

Maybe: https://wpengine.com/wp-content/uploads/2024/10/Complaint...

I know very little about the US legal process, but this seems important enough to maybe update the article, or even post a separate news entry?

Risk management anyone?

yeltsin — Fri, 04 Oct 2024 00:53:10 +0000

Depends on where you live. I'm thousands of kilometers away from major datacenters, and every developer worth anything sets up aggressive dependency caching for every CI job because it's unbearable to use otherwise.

Risk management anyone?

pizza — Thu, 03 Oct 2024 21:55:18 +0000

> Is it vital to check that the original file is still online 5 thousands times per day?

For a single organization? Probably not. But if it's 50000+ different orgs each checking once a day?

> And what if it isn't?

Then you have to determine why, and adjust your system's data source accordingly.

(Note "original file" can easily point at a private/internal mirror or some sort of SW BoM artifact storage. Granted, some ecosystems make this sort of thing ...challenging to set up and transparently utilize)

Risk management anyone?

SLi — Thu, 03 Oct 2024 21:50:49 +0000

It certainly is vital to check it often if your build does fetch it from somewhere external. Now fetching it from somewhere external may not be the right thing to do, but absolutely, a CI should detect if a build breaks, and a cache would only mask this failure. A proper mirror with no unpredictable expiry rules is another matter.

Risk management anyone?

LtWorf — Thu, 03 Oct 2024 21:39:02 +0000

At work at least for local builds I made our system use a cache that gets mounted into the container. It saves considerable amount of time (and allows working on mobile connections) to not download several hundreds of MB of things every time.

Risk management anyone?

LtWorf — Thu, 03 Oct 2024 21:36:47 +0000

Is it vital to check that the original file is still online 5 thousands times per day?

And what if it isn't?

Debian is full of packages whose original websites are gone. Every once in a while someone uses the last .tar.gz from debian to make a fork.

Risk management anyone?

dskoll — Thu, 03 Oct 2024 19:46:49 +0000

It's pretty easy. In my case, the build code was downloading a tarball over HTTPS, so it could easily have used the If-Modified-Since: header.

However, I suspect the build was done one a virtual machine that was spun up from scratch anew each time, so there was no existing tarball for it to check the timestamp against.

Risk management anyone?

raven667 — Thu, 03 Oct 2024 18:19:36 +0000

Yeah, mirroring and caching web content is far less emphasized in general than it used to be when connection speeds were slower. I don't hear about people setting up Squid proxies or creating internal mirrors, both of which used to be pretty routine IT infrastructure. Now directly updating/installing from public mirrors is fast enough in most cases that people don't keep working to optimize their process and move on, whereas before when you might only have a 1Mbit connection to the Internet shared by your whole office, you couldn't afford the time spent downloading the same thing more than once

Risk management anyone?

pizza — Thu, 03 Oct 2024 16:59:17 +0000

> How easy is it for a remote download to tell nothing has changed?

There's another aspect to that -- caching is important, but another thing the CI needs to test for is that the original resource is still available.

...I've had plenty of CI runs that _falsely_ succeeded because they used a cached copy of a no-longer-available resource, leading to unexpected failures when (eg) doing a production build or spinning up a new developer environment.

Risk management anyone?

Wol — Thu, 03 Oct 2024 16:52:34 +0000

How easy is it for a remote download to tell nothing has changed?

Running gentoo, I'm conscious that seems to download everything every time, but it also doesn't download unless something has changed (be it ebuild, use flags, whatever).

I would think it *should* do a shallow git clone, and keep that lying around unless the user explicitly clears it (it leaves enough stuff lying around, why not that), so even if use flags and stuff has changed, it would have no need to get the source afresh unless there really is an upgrade.

But not knowing python, or the guts of portage, I have no way of knowing if that's actually the case ...

Cheers,
Wol

Risk management anyone?

dskoll — Thu, 03 Oct 2024 16:11:11 +0000

Yes. I've blocked downloads of some of my software from various places that re-download the same thing every single time they do a build. There's no excuse for that sort of abuse.

Risk management anyone?

aragilar — Thu, 03 Oct 2024 12:32:44 +0000

While I do run both PyPI and npm caches (both of which were fairly easy to set up), I would argue the majority blame should be shifted to the hosted CI providers (especially GitHub via its Actions ecosystem as it has the worst setup for caching I've used), who could all provide a caching server and make it easy to use (either by setting the required environment variables by default, or via whatever configuration mechanism makes sense for their system), rather than the developers themselves.

Currently, for hosted CI, you would need to spin up your own cache servers, which given the value of hosted CI is to not need to run servers, would seem to be a much larger ask of individual developers.

No mirrors?

aragilar — Thu, 03 Oct 2024 12:24:14 +0000

I can't comment on how you would cache wordpress.org, but there are numerous tools which provide caching/mirroring of PyPI (for various use-cases, requirements and scales). If Azure (or more likely one of their customers) became abusive of the service (as someone did for the XMLRPC service), I don't see PyPI wouldn't and shouldn't as a last resort block Azure (as happened with the XMLRPC service). I would expect Azure to be reasonable and provide a cache/mirror and/or deal with abusive customer, but it would appear in the wordpress case reasonableness has gone out the window.

Risk management anyone?

LtWorf — Thu, 03 Oct 2024 09:07:40 +0000

Check the download counters on pypi or npm. Basically the idea of "mirroring" is completely unknown to current developers.

You of course redownload all your dependencies every single time that you run your CI!

No mirrors?

edgewood — Wed, 02 Oct 2024 18:31:21 +0000

I agree that it would be legal (unless there's already a contract) but unreasonable for PSF to cut off Azure in this hypothetical situation. If there was an excessive bandwidth usage, throttling or a warning that they could be cut off in the near future would be reasonable.

However, unlike in the hypothetical, WP Engine sent a cease and desist/preserve documents letter the day before the cutoff. I think that makes the cutoff more reasonable: if you're freeloading, maybe you should take some steps to stop relying on those free services before you go making legal threats.

No mirrors?

Kalenx — Wed, 02 Oct 2024 17:12:30 +0000

Legal? As I already said, absolutely.
Reasonable? No sure I agree (IOW: I strongly disagree)

No one is obliged to keep up a Python package index. The Python Software Foundation does it because, presumably, it helps them fulfilling their own stated mission: "We are devoted to creating the conditions for Python and the Python community to grow and thrive."

If they start cutting off random people, including end users who did nothing wrong (other than choosing the "wrong" cloud provider), they are not, IMHO, "creating the conditions for the Python community to grow and thrive".

> If it were found that Azure users actually overload PyPI service and Microsoft does nothing to compensate that and this affects non-Azure users… then it would have been the right thing to do.

That would be the nuclear thing to do; not sure it makes it "right". Just as an example, throttling could also be an option. But anyway, we are going off topic, since this is clearly _not_ what happened in the Wordpress/WP engine case. This "resource usage" was not mention until after the fact...

No mirrors?

khim — Wed, 02 Oct 2024 09:40:43 +0000

> Well, personally, I consider this action equivalent to the Python Software Foundation abruptly locking out Azure users from Pypi, stating a vague "Microsoft does not contribute enough to Python" to claim the moral high ground.

IOW: something that's perfectly reasonable and legal thing to do. Free software does come with source but it doesn't come with a free support license, one have to always remember that.

> Sure, I guess there is no legal obligation for the PSF to provide services to Microsoft clients (or anyone, for that matter) but it would still be a highly dubious move, highly detrimental for the Python community.

This would depend entirely on the situation around Azure, PyPI, etc. If it were found that Azure users actually overload PyPI service and Microsoft does nothing to compensate that and this affects non-Azure users… then it would have been the right thing to do.

Criminal?

raven667 — Tue, 01 Oct 2024 20:31:40 +0000

It'd be really funny if any of the threats that Automattic made against WP Engine were legally actionable crimes, since they seem to be well documented, although prosecutors tend not to go after those with deep pockets if they can avoid it as those cases can consume a lot of resources and they are unlikely to win.

Risk management anyone?

hkario — Tue, 01 Oct 2024 17:02:08 +0000

not when they're just burning VC money, at least not by all definitions

Risk management anyone?

notriddle — Tue, 01 Oct 2024 15:05:26 +0000

That is an insult to the cloud computing people. At least they pay money for what they use!

Risk management anyone?

pizza — Tue, 01 Oct 2024 11:53:30 +0000

> Besides the social media drama for which I couldn't care less, I cannot grasp why WP Engine would run into that trap. If you and your users depend on something, than you have to make sure you have it availabl

But that costs actual money to provide, and doing so is apparently antithetical to their business model of externalizing costs to maximize profits.

No mirrors?

aragilar — Tue, 01 Oct 2024 10:39:11 +0000

Not entirely the same thing, but PyPI does block outlook emails (https://blog.pypi.org/posts/2024-06-16-prohibiting-msn-em...), which is due to spam account issues.

It's not clear to what extent WP Engine's use of Wordpress.org infra could be called excessive (if at all), but presumably they could have had a cache in front of the services (which would seem to be a wise thing to do anyway) or contribute to running the services if using a cache is not possible due to how wordpress is designed?

Risk management anyone?

mb — Tue, 01 Oct 2024 05:46:20 +0000

Running your own stuff is soooo 2000s.
We have 2024! We now only put stuff on somebody else's magic machines and call it "cloud".
It's much betterer, because somebody else does the work!

Risk management anyone?

JanSoundhouse — Tue, 01 Oct 2024 05:37:32 +0000

Besides the social media drama for which I couldn't care less, I cannot grasp why WP Engine would run into that trap. If you and your users depend on something, than you have to make sure you have it available. That would mean you have to run your own mirrors. Always. Did they not take any notes from the 2016 npm incident where one minor dependency vanished and it broke the (www) world?

No mirrors?

Kalenx — Tue, 01 Oct 2024 01:35:18 +0000

Well, personally, I consider this action equivalent to the Python Software Foundation abruptly locking out Azure users from Pypi, stating a vague "Microsoft does not contribute enough to Python" to claim the moral high ground.

Sure, I guess there is no legal obligation for the PSF to provide services to Microsoft clients (or anyone, for that matter) but it would still be a highly dubious move, highly detrimental for the Python community.

No mirrors?

hailfinger — Mon, 30 Sep 2024 21:54:02 +0000

So... some companies were using a single centralized foreign resource directly and didn't even think of mirroring that resource or paying the entity hosting/maintaining the resource? And then those companies sold the stuff they got for free? And now the entity maintaining the resource is expected to serve and shut up?

That's really stupid from a supply chain perspective and really questionable from an ethical perspective.

If I have a business, but no contracts with my supply chain, my supply chain can disappear or turn hostile any second. The complaints by various hosting providers/resellers read like "Mommy, Annie is not letting me play with her toys anymore!". Note how absent the "but we contribute equally" argument is.

Very nice recap

pwfxq — Mon, 30 Sep 2024 19:33:26 +0000

It's much clearer than other analyses of the situation than I've seen on other news sites. The background of the trademark ownership is also much appreciated.

Thank you.

Very nice recap

post-factum — Mon, 30 Sep 2024 18:40:59 +0000

Exactly, thanks a lot for this article.

Very nice recap

mdolan — Mon, 30 Sep 2024 18:02:54 +0000

This is a very thorough recap of the situation - nicely done!

Mullenweg is a problem

fishface60 — Mon, 30 Sep 2024 17:40:10 +0000

Mullenweg should've been sidelined after he took to Twitter to bully and dox a former Tumblr user for questioning the moderation policy.
If I were forced to pick a side this would make it easy.