|
|
Subscribe / Log in / New account

The WordPress mess

By Joe Brockmeier
September 30, 2024

WordPress is the world's most popular open‑source blogging and content‑management platform. In its 20‑plus years of existence, WordPress has been something of a poster child for open source, similar to Linux and Firefox. It introduced the concept of open source to millions of bloggers, small‑business owners, and others who have deployed WordPress to support their web‑publishing needs. Unfortunately, it is now in the spotlight due to an increasingly ugly dispute between two companies, Automattic and WP Engine, that has spilled over into the WordPress community.

Background

WordPress is a PHP‑based, GPLv2-licensed, content‑management system (CMS). It was forked from b2 by Matt Mullenweg and Mike Little in 2003, after b2's development stalled. As blogging became mainstream, WordPress was the tool of choice for many aspiring bloggers due to its ease of use and the fact that it was free as in beer and speech.

In 2005, Mullenweg founded Automattic, which started out offering WordPress hosting via WordPress.com and comment-spam protection via the Akismet service. Its portfolio of services and holdings (including the Tumblr micro‑blogging platform) has grown substantially since then; it includes the WooCommerce online-store platform, Gravatar web-profile service, Newspack news-publishing platform, and others. WordPress hosting, however, remains at the core of its business.

Foundation and trademark

Automattic's press page lists 11 investors in the company and describes it as a "later‑stage growth phase" company. In 2010, Mullenweg started the non‑profit WordPress Foundation to manage WordPress development and hold the WordPress trademarks. The open‑source project is often referred to as WordPress.org to distinguish it from Automattic's WordPress.com hosting business. When the marks were transferred, Mullenweg wrote that "the most central piece of WordPress's identity, its name, is now fully independent from any company".

The reality is that the name is still under the control of Automattic, and thus still under Mullenweg's control. The WordPress Foundation trademark policy says that any commercial use of the WordPress trademark is controlled by Automattic, which has an exclusive license. The foundation only controls non-commercial uses, and the only active member of the foundation's board seems to be Mullenweg. According to the most recent tax filing by the foundation, there were only three board members in 2022: Mullenweg, Mark Ghosh, and Chele Chiavacci Farley.

One might assume that the foundation is involved in running WordPress.org, but it is not. Donations to the foundation primarily support events and education about WordPress. Its financials for 2023 detail expenditures for events and publishing videos on WordPress.tv. Individuals and organizations that want to support the WordPress project itself are encouraged to contribute resources to "Five for the Future". This program was introduced in 2014 by Mullenweg for others in the WordPress ecosystem to help "grow the WordPress pie".

The dispute

The other major player in this drama is WP Engine. That company was founded in 2010 by Jason Cohen and Cullen Wilson, who described themselves as "power users" who wanted to provide hosting by WordPress experts. The company now bills its offerings as the "most trusted WordPress platform". Like Automattic, WP Engine's business is based on hosting and supporting WordPress. In 2018, WP Engine took funding from private-equity firm Silver Lake. The upshot is that both companies compete for the same customers, have substantially similar offerings, and have investors to please.

On September 20, at WordCamp US 2024, Mullenweg used his talk to air some grievances against WP Engine and Silver Lake. He followed up the next day with a blog post titled "WP Engine is not WordPress" that criticized WP Engine's lack of contributions to WordPress, and its disabling of WordPress's revisions feature that saves each saved draft or published version of a post or page. A standard WordPress installation will have no limit on the number of revisions stored, or how long the revisions will be stored. Users can compare revisions to see what has changed, and restore an older revision if desired. WP Engine turns this feature off by default, though users can contact support to turn on as many as five revisions that will be stored up to 60 days.

Mullenweg described the feature as being a core piece of the promise to protect user data, and claimed that WP Engine wanted to turn off revisions to avoid paying for storage. "They are strip-mining the WordPress ecosystem, giving our users a crappier experience so they can make more money."

Mullenweg claimed that the companies enjoyed similar revenue, but complained that WP Engine's contributions to the WordPress project averaged 40 hours per week compared to Automattic's nearly 4,000 hours per week. He had choice words for WP Engine, calling it a "cancer to WordPress" and encouraged customers to "consider literally any other WordPress host". He argued that because of WP Engine's modifications, customers were not getting WordPress. "And they're profiting off of the confusion. WP Engine needs a trademark license to continue their business."

The foundation's policy page used to say that the WP abbreviation "is not covered by the WordPress trademarks and you are free to use it in any way you see fit". It has been recently updated, sometime after September 19, to say that the abbreviation is not covered, "but please don't use it in a way that confuses people":

For example, many people think WP Engine is "WordPress Engine" and officially associated with WordPress, which it's not. They have never once even donated to the WordPress Foundation, despite making billions of revenue on top of WordPress.

Mullenweg's blog post focuses heavily on making the point that WP Engine's offering is "not WordPress" because it turns off revisions—but the trademark policy is silent on what modifications are or are not permitted to describe something as WordPress. It should be noted at this point that WordPress.com does not offer unadulterated WordPress to all of its customers either. Another core feature of WordPress, the ability to install third‑party plugins and themes, is restricted to customers who pay for the Business tier of hosting or above (about $300 a year).

Stop it

On September 23, WP Engine sent a cease-and-desist letter to Automattic, which it also made public. In the letter, the company claimed that Mullenweg had "suddenly began demanding that WP Engine pay Automattic large sums of money", or he would "go 'nuclear' on WP Engine" during the keynote at WordCamp on September 20. The letter includes what are alleged to be text messages from Mullenweg that describe the way he will attack WP Engine and Silver Lake during the keynote, unless the company was willing to negotiate a payment of a percentage of its revenues for use of the trademark.

Automattic also sent (and published) its own cease-and-desist letter the same day, that demanded WP Engine stop using an assortment of trademarks, including WordPress. It demanded an accounting of profits from use of service offerings that were marketed under the trademarks, and compensation for unauthorized use. As examples of infringement it specifically called out "many references to WP Engine being 'WordPress Engine'". The letter was also accompanied by pages of examples highlighting instances of the phrase "WordPress Engine" as well as social-media posts where apparently confused users ask questions about WordPress Engine.

Automattic's cease-and-desist put a tentative figure on what the company was seeking from WP Engine: a "mere 8% royalty" on WP Engine's estimated $400 million in annual revenue, or more than $32 million per year in "lost revenue". It left open the potential for amicably resolving the matter "including through a licensing relationship for use of its trademarks".

In a post about the dispute on his personal blog, Mullenweg said that Automattic had been trying to make a licensing deal with WP Engine for a long time, but had been strung along. It was not a money grab, he said, but an attempt to make WP Engine pay its fair share, and that the company had been offered the options of "either pay a direct licensing fee, or make in‑kind contributions to the open source project".

Escalation

Because Mullenweg posted the "WP Engine is not WordPress" post to WordPress.org's news page, it also showed up in the WordPress administrative dashboard in a "news and events" widget. That meant the message was immediately visible to WP Engine customers if they happened to have that widget enabled and were using the dashboard. WP Engine apparently removed the widget from its hosted customers' WordPress installs on September 24. On September 25, Mullenweg posted on WordPress.org announcing that WP Engine was "banned from WordPress.org", meaning that users on WP Engine could no longer use built‑in WordPress update features and more:

WP Engine wants to control your WordPress experience, they need to run their own user login system, update servers, plugin directory, theme directory, pattern directory, block directory, translations, photo directory, job board, meetups, conferences, bug tracker, forums, Slack, Ping‑o‑matic, and showcase. Their servers can no longer access our servers for free.

Why now?

The obvious question on many people's minds is "why now?", which was not really addressed in any of the public statements. Mullenweg discussed the timing in a live-streamed interview by Michael B. Paulson on September 26. Paulson asked "at what point did this become something that must be addressed, and in this way?" Mullenweg responded that other companies in the WordPress ecosystem contribute or pay for trademark agreements "even though they make way less revenue" than WP Engine:

They kept saying they were going to do something for years, and there's a lot of friendly people there [...] I guess I didn't realize how, like, capricious and evil private equity could be, so that is totally my fault and I apologize for that.

Paulson followed up and said that, "as an outsider" it was unclear why this happened now and not a year ago or another point in time. Mullenweg said that he had been "trying to connect" with WP Engine for years but the company had been unwilling to negotiate: "they were just like, nope, not anything, you know? Let's delay, delay, delay". Mullenweg said that WP Engine had "tried to make it look like I was extorting them last minute" but he had wanted to avoid the fight. "In fact, they could sign a trademark license today and this would all go away".

Another bone of contention that Mullenweg said he had failed to communicate to the community was over transactions made via Stripe with the WooCommerce plugin. Mullenweg said that WP Engine had "hacked" the plugin to change the attribution code so that "tens of millions of dollars" in fees go to WP Engine rather than Automattic. Note that the plugin is licensed under the GPLv3, and the information page makes no mention of any referral codes or expectations that the referral codes should remain unchanged. The privacy policy that is linked from the WooCommerce information page mentions several reasons that information is collected, but none of them address affiliate fees or attribution codes that generate revenue for Automattic.

Reactions

The ban of WP Engine from using WordPress.org services has sent WordPress users and developers scrambling. Cullen Whitmore, a WordPress contributor and owner of an agency that provides WordPress services, wrote an open letter that made an appeal to Mullenweg to remove the ban on WP Engine and provide a timeline to reach a deal. The only brand confusion, he said, is for new users confused about where to get started with WordPress, WordPress.com or WordPress.org. He also defended WP Engine's contributions, saying that it contributes back "through other methods", such as supporting users and developers in adopting and implementing WordPress features.

He found the ban troubling, but he said "my biggest concern was raised by your response to a user in the Make WordPress Slack". (The conversation is here but requires creating or having a WordPress.org account and then signing up for the Slack channel.) A user asked what they should tell their customers who need updates. Mullenweg told the user to contact WP Engine and "be mad at the person you're paying". He added that WP Engine "can fork WordPress, too, and publish their own version. I'd love that". Mullenweg, Whitmore said, should "rethink your position on speaking about those caught in the middle. You didn't sound like a leader here".

User "FeralRobots" on mastodon.social said that, without an easy way to specify alternative repositories for WordPress, "Matt Mullenweg is basically a living supply chain vulnerability in the WordPress ecosystem".

Part of Automattic's urgency in trying to force WP Engine to the table may be the increasing commoditization of WordPress hosting as a business. Eric Mann, a "power user" of WordPress who used to work with it professionally, wrote that all of the companies exhibiting at WordCamp US were "commodity plays". Every one, he said, claims to be the fastest or most innovative, but "I can't tell you what makes them different other than the color of the socks they were giving away as swag". He later wrote that Mullenweg had raised "valid concerns" around "WP Engine's failure to contribute back to the open source community". Mann also blamed WP Engine for failing to have protections in place "for an unexpected outage of WordPress.org API services".

The value of infrastructure

Brian Coords, who is a technology director at a WordPress development agency, wrote on September 27 that "Matt's intentions in waging a battle of FOSS vs PE [private equity] are good, but to be clear: I don't agree with his tactics." (Emphasis his.) He said he never expected Mullenweg to "compromise individual WordPress installations and act in such a user-hostile way" but that what made WordPress successful was more than the code, it was "the seamless distribution of core, theme, and plugin updates that are hosted and deployed by WordPress.org". If users still had to manually upload PHP files and SFTP into servers, as they had to do in the early days, the project wouldn't be nearly as successful. That distribution network, he said, is subsidized by Automattic and is "Matt's to do what he pleases with it":

If you've been relying on one-click core updates and the plugin directory to manage the distribution of open source plugins to/from websites (which of course we all have), then you have been playing in Matt's sandbox. This sandbox is not "open source WordPress" or the WordPress Foundation. It is WordPress.org, which is a wonderful thing that Matt Mullenweg owns and has stewarded for the last two decades. But it is not "ours".

Paolo Belcastro, an Automattic employee who has been with the company since 2011, wrote that the costs of maintaining WordPress.org and its infrastructure are "tens of millions of dollars every year", which is sponsored by Mullenweg personally and Automattic. He notes that WP Engine actually charges customers for the automatic updates ($3 per month) that are provided free to others via WordPress.org.

A brief reprieve

On September 27, Mullenweg announced a reprieve for WP Engine until October 1 to allow the company to "spin up their mirrors of all of WordPress.org's resources that they were using for free while not paying". He blamed WP Engine and Silver Lake for the impact on customers:

I've heard from WP Engine customers that they are frustrated that WP Engine hasn't been able to make updates, plugin directory, theme directory, and Openverse work on their sites. It saddens me that they've been negatively impacted by Silver Lake's commercial decisions.

WP Engine was well aware that we could remove access when they chose to ignore our efforts to resolve our differences and enter into a commercial licensing agreement. [...] You could assume that WPE has a workaround ready, or they were simply reckless in supporting their customers. Silver Lake and WP Engine put their customers at risk, not me.

Calls for change

Joost de Valk wrote that he would like to see "big WordPress companies" contribute, but there was a need for transparency and community-based governance. He had founded Yoast, a company that provides search-engine optimization (SEO) tools and plugins for WordPress (as well as other platforms). He wrote that if there are requirements to contribute, "we can call that taxation", and that should come with representation:

In my opinion, we all should get a say in how we spend those contributions. I understand that core contributors are very important, but so are the organizers of our (flagship) events, the leadership of hosting companies, etc. We need to find a way to have a group of people who represent the community and the contributing corporations.

Just like in a democracy. Because, after all, isn't WordPress all about democratizing?

Josh Collinsworth, a front-end developer who has worked at WP Engine in the past, called for Mullenweg to be removed. He said that Mullenweg "has, for far too long, enjoyed unchecked powers at the top of WordPress" and that those powers were "all too often a direct and flagrant conflict of interest". Collinsworth said that he was not defending his former employer, and he would happily discuss "how much WP Engine sucks" but that "the ability to block an entire hosting provider from accessing the plugins repository is a power that nobody should have":

I don't care about Automattic giving 5% to WordPress anymore. I want it to give up Matt's unchecked, unilateral power. Because it's clearer than ever he can't be trusted with it.

Uncertainty continues

The tit‑for‑tat between Automattic and WP Engine may continue for some time, but the impact of the conflict has already sent tremors throughout the WordPress community. Users on Mastodon, Twitter, and other social-media sites are voicing complaints with the way that Mullenweg has handled the situation and the uncertainty it has caused. It seems likely that "are we next?" is a popular topic of conversation in the hallways (or Slack channels...) at other companies providing WordPress hosting and services.

No matter how long the two companies have been negotiating behind the scenes, this incident came as a surprise to far too many WordPress users caught in the middle. And, if one takes a moment to think about Mullenweg's argument that "WP Engine isn't WordPress" for lack of the revisions feature, it's unclear how paying Automattic a licensing fee for use of the name is going to make anything better for anyone but Automattic. Presumably, if WP Engine forks over a big enough pile of cash, Automattic is willing to let the company continue to skimp on revisions, and contributions.

It is apparent that the foundation is simply a fig leaf to allow Automattic continued control of the WordPress marks, while giving the appearance of independence. There is no benefit to the WordPress community to have a non-profit holding the marks if the entity simply boomerangs the exclusive rights back to the for‑profit that turned them over in the first place.

As Coords wrote, the infrastructure around WordPress.org is wholly controlled by Mullenweg and funded by Automattic. That is a burden that shouldn't be carried by one person or company, and it gives too much power over the larger community. A week ago that argument was a hypothetical, but Mullenweg has demonstrated willingness to wield that power unilaterally, without warning, and with no checks or balances aside from public outcry. The arrangement that places Automattic as the sole benefactor providing hosting and other services for WordPress.org, and Mullenweg as the sole decider about its policies, is a weak point for the larger community and ecosystem around WordPress.

The WordPress mission is to "democratize publishing". The project has enabled millions of users and organizations to easily publish content to the web. I include myself as one of them, since I've been using WordPress (off and on) for almost its entire history. The project itself, unfortunately, seems to be leaning toward the autocratic. One can find WP Engine's lack of contributions to be problematic while still being concerned that Mullenweg has too much power over a project—and its infrastructure—that powers a significant portion of the world's web sites.



to post comments

Mullenweg is a problem

Posted Sep 30, 2024 17:40 UTC (Mon) by fishface60 (subscriber, #88700) [Link]

Mullenweg should've been sidelined after he took to Twitter to bully and dox a former Tumblr user for questioning the moderation policy.
If I were forced to pick a side this would make it easy.

Very nice recap

Posted Sep 30, 2024 18:02 UTC (Mon) by mdolan (subscriber, #104340) [Link] (2 responses)

This is a very thorough recap of the situation - nicely done!

Very nice recap

Posted Sep 30, 2024 18:40 UTC (Mon) by post-factum (subscriber, #53836) [Link]

Exactly, thanks a lot for this article.

Very nice recap

Posted Sep 30, 2024 19:33 UTC (Mon) by pwfxq (subscriber, #84695) [Link]

It's much clearer than other analyses of the situation than I've seen on other news sites. The background of the trademark ownership is also much appreciated.

Thank you.

No mirrors?

Posted Sep 30, 2024 21:54 UTC (Mon) by hailfinger (subscriber, #76962) [Link] (7 responses)

So... some companies were using a single centralized foreign resource directly and didn't even think of mirroring that resource or paying the entity hosting/maintaining the resource? And then those companies sold the stuff they got for free? And now the entity maintaining the resource is expected to serve and shut up?

That's really stupid from a supply chain perspective and really questionable from an ethical perspective.

If I have a business, but no contracts with my supply chain, my supply chain can disappear or turn hostile any second. The complaints by various hosting providers/resellers read like "Mommy, Annie is not letting me play with her toys anymore!". Note how absent the "but we contribute equally" argument is.

No mirrors?

Posted Oct 1, 2024 1:35 UTC (Tue) by Kalenx (subscriber, #120295) [Link] (6 responses)

Well, personally, I consider this action equivalent to the Python Software Foundation abruptly locking out Azure users from Pypi, stating a vague "Microsoft does not contribute enough to Python" to claim the moral high ground.

Sure, I guess there is no legal obligation for the PSF to provide services to Microsoft clients (or anyone, for that matter) but it would still be a highly dubious move, highly detrimental for the Python community.

No mirrors?

Posted Oct 1, 2024 10:39 UTC (Tue) by aragilar (subscriber, #122569) [Link]

Not entirely the same thing, but PyPI does block outlook emails (https://blog.pypi.org/posts/2024-06-16-prohibiting-msn-em...), which is due to spam account issues.

It's not clear to what extent WP Engine's use of Wordpress.org infra could be called excessive (if at all), but presumably they could have had a cache in front of the services (which would seem to be a wise thing to do anyway) or contribute to running the services if using a cache is not possible due to how wordpress is designed?

No mirrors?

Posted Oct 2, 2024 9:40 UTC (Wed) by khim (subscriber, #9252) [Link] (4 responses)

> Well, personally, I consider this action equivalent to the Python Software Foundation abruptly locking out Azure users from Pypi, stating a vague "Microsoft does not contribute enough to Python" to claim the moral high ground.

IOW: something that's perfectly reasonable and legal thing to do. Free software does come with source but it doesn't come with a free support license, one have to always remember that.

> Sure, I guess there is no legal obligation for the PSF to provide services to Microsoft clients (or anyone, for that matter) but it would still be a highly dubious move, highly detrimental for the Python community.

This would depend entirely on the situation around Azure, PyPI, etc. If it were found that Azure users actually overload PyPI service and Microsoft does nothing to compensate that and this affects non-Azure users… then it would have been the right thing to do.

No mirrors?

Posted Oct 2, 2024 17:12 UTC (Wed) by Kalenx (subscriber, #120295) [Link] (2 responses)

Legal? As I already said, absolutely.
Reasonable? No sure I agree (IOW: I strongly disagree)

No one is obliged to keep up a Python package index. The Python Software Foundation does it because, presumably, it helps them fulfilling their own stated mission: "We are devoted to creating the conditions for Python and the Python community to grow and thrive."

If they start cutting off random people, including end users who did nothing wrong (other than choosing the "wrong" cloud provider), they are not, IMHO, "creating the conditions for the Python community to grow and thrive".

> If it were found that Azure users actually overload PyPI service and Microsoft does nothing to compensate that and this affects non-Azure users… then it would have been the right thing to do.

That would be the nuclear thing to do; not sure it makes it "right". Just as an example, throttling could also be an option. But anyway, we are going off topic, since this is clearly _not_ what happened in the Wordpress/WP engine case. This "resource usage" was not mention until after the fact...

No mirrors?

Posted Oct 2, 2024 18:31 UTC (Wed) by edgewood (subscriber, #1123) [Link]

I agree that it would be legal (unless there's already a contract) but unreasonable for PSF to cut off Azure in this hypothetical situation. If there was an excessive bandwidth usage, throttling or a warning that they could be cut off in the near future would be reasonable.

However, unlike in the hypothetical, WP Engine sent a cease and desist/preserve documents letter the day before the cutoff. I think that makes the cutoff more reasonable: if you're freeloading, maybe you should take some steps to stop relying on those free services before you go making legal threats.

No mirrors?

Posted Oct 3, 2024 12:24 UTC (Thu) by aragilar (subscriber, #122569) [Link]

I can't comment on how you would cache wordpress.org, but there are numerous tools which provide caching/mirroring of PyPI (for various use-cases, requirements and scales). If Azure (or more likely one of their customers) became abusive of the service (as someone did for the XMLRPC service), I don't see PyPI wouldn't and shouldn't as a last resort block Azure (as happened with the XMLRPC service). I would expect Azure to be reasonable and provide a cache/mirror and/or deal with abusive customer, but it would appear in the wordpress case reasonableness has gone out the window.

No mirrors?

Posted Oct 10, 2024 5:56 UTC (Thu) by TRS-80 (guest, #1804) [Link]

TensorFlow is responsible for 17% of data transferred from PyPI:

https://kristoff.it/blog/python-training-wheels/

The whole post is about the cost of PyPI and worth reading.

Risk management anyone?

Posted Oct 1, 2024 5:37 UTC (Tue) by JanSoundhouse (subscriber, #112627) [Link] (23 responses)

Besides the social media drama for which I couldn't care less, I cannot grasp why WP Engine would run into that trap. If you and your users depend on something, than you have to make sure you have it available. That would mean you have to run your own mirrors. Always. Did they not take any notes from the 2016 npm incident where one minor dependency vanished and it broke the (www) world?

Risk management anyone?

Posted Oct 1, 2024 5:46 UTC (Tue) by mb (subscriber, #50428) [Link] (2 responses)

Running your own stuff is soooo 2000s.
We have 2024! We now only put stuff on somebody else's magic machines and call it "cloud".
It's much betterer, because somebody else does the work!

Risk management anyone?

Posted Oct 1, 2024 15:05 UTC (Tue) by notriddle (subscriber, #130608) [Link] (1 responses)

That is an insult to the cloud computing people. At least they pay money for what they use!

Risk management anyone?

Posted Oct 1, 2024 17:02 UTC (Tue) by hkario (subscriber, #94864) [Link]

not when they're just burning VC money, at least not by all definitions

Risk management anyone?

Posted Oct 1, 2024 11:53 UTC (Tue) by pizza (subscriber, #46) [Link]

> Besides the social media drama for which I couldn't care less, I cannot grasp why WP Engine would run into that trap. If you and your users depend on something, than you have to make sure you have it availabl

But that costs actual money to provide, and doing so is apparently antithetical to their business model of externalizing costs to maximize profits.

Risk management anyone?

Posted Oct 3, 2024 9:07 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (18 responses)

Check the download counters on pypi or npm. Basically the idea of "mirroring" is completely unknown to current developers.

You of course redownload all your dependencies every single time that you run your CI!

Risk management anyone?

Posted Oct 3, 2024 12:32 UTC (Thu) by aragilar (subscriber, #122569) [Link]

While I do run both PyPI and npm caches (both of which were fairly easy to set up), I would argue the majority blame should be shifted to the hosted CI providers (especially GitHub via its Actions ecosystem as it has the worst setup for caching I've used), who could all provide a caching server and make it easy to use (either by setting the required environment variables by default, or via whatever configuration mechanism makes sense for their system), rather than the developers themselves.

Currently, for hosted CI, you would need to spin up your own cache servers, which given the value of hosted CI is to not need to run servers, would seem to be a much larger ask of individual developers.

Risk management anyone?

Posted Oct 3, 2024 16:11 UTC (Thu) by dskoll (subscriber, #1630) [Link] (8 responses)

Yes. I've blocked downloads of some of my software from various places that re-download the same thing every single time they do a build. There's no excuse for that sort of abuse.

Risk management anyone?

Posted Oct 3, 2024 16:52 UTC (Thu) by Wol (subscriber, #4433) [Link] (7 responses)

How easy is it for a remote download to tell nothing has changed?

Running gentoo, I'm conscious that seems to download everything every time, but it also doesn't download unless something has changed (be it ebuild, use flags, whatever).

I would think it *should* do a shallow git clone, and keep that lying around unless the user explicitly clears it (it leaves enough stuff lying around, why not that), so even if use flags and stuff has changed, it would have no need to get the source afresh unless there really is an upgrade.

But not knowing python, or the guts of portage, I have no way of knowing if that's actually the case ...

Cheers,
Wol

Risk management anyone?

Posted Oct 3, 2024 16:59 UTC (Thu) by pizza (subscriber, #46) [Link] (4 responses)

> How easy is it for a remote download to tell nothing has changed?

There's another aspect to that -- caching is important, but another thing the CI needs to test for is that the original resource is still available.

...I've had plenty of CI runs that _falsely_ succeeded because they used a cached copy of a no-longer-available resource, leading to unexpected failures when (eg) doing a production build or spinning up a new developer environment.

Risk management anyone?

Posted Oct 3, 2024 21:36 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (3 responses)

Is it vital to check that the original file is still online 5 thousands times per day?

And what if it isn't?

Debian is full of packages whose original websites are gone. Every once in a while someone uses the last .tar.gz from debian to make a fork.

Risk management anyone?

Posted Oct 3, 2024 21:50 UTC (Thu) by SLi (subscriber, #53131) [Link]

It certainly is vital to check it often if your build does fetch it from somewhere external. Now fetching it from somewhere external may not be the right thing to do, but absolutely, a CI should detect if a build breaks, and a cache would only mask this failure. A proper mirror with no unpredictable expiry rules is another matter.

Risk management anyone?

Posted Oct 3, 2024 21:55 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> Is it vital to check that the original file is still online 5 thousands times per day?

For a single organization? Probably not. But if it's 50000+ different orgs each checking once a day?

> And what if it isn't?

Then you have to determine why, and adjust your system's data source accordingly.

(Note "original file" can easily point at a private/internal mirror or some sort of SW BoM artifact storage. Granted, some ecosystems make this sort of thing ...challenging to set up and transparently utilize)

Risk management anyone?

Posted Oct 4, 2024 8:22 UTC (Fri) by LtWorf (subscriber, #124958) [Link]

In my own experience, it's most usually few organizations spamming, rather than many organizations. Easy to see because overnight the daily downloads might halve or double. And I doubt it's because thousands of organizations all went to use a different library all in the same day.

Risk management anyone?

Posted Oct 3, 2024 19:46 UTC (Thu) by dskoll (subscriber, #1630) [Link] (1 responses)

It's pretty easy. In my case, the build code was downloading a tarball over HTTPS, so it could easily have used the If-Modified-Since: header.

However, I suspect the build was done one a virtual machine that was spun up from scratch anew each time, so there was no existing tarball for it to check the timestamp against.

Risk management anyone?

Posted Oct 3, 2024 21:39 UTC (Thu) by LtWorf (subscriber, #124958) [Link]

At work at least for local builds I made our system use a cache that gets mounted into the container. It saves considerable amount of time (and allows working on mobile connections) to not download several hundreds of MB of things every time.

Risk management anyone?

Posted Oct 3, 2024 18:19 UTC (Thu) by raven667 (subscriber, #5198) [Link] (7 responses)

Yeah, mirroring and caching web content is far less emphasized in general than it used to be when connection speeds were slower. I don't hear about people setting up Squid proxies or creating internal mirrors, both of which used to be pretty routine IT infrastructure. Now directly updating/installing from public mirrors is fast enough in most cases that people don't keep working to optimize their process and move on, whereas before when you might only have a 1Mbit connection to the Internet shared by your whole office, you couldn't afford the time spent downloading the same thing more than once

Risk management anyone?

Posted Oct 4, 2024 0:53 UTC (Fri) by yeltsin (guest, #171611) [Link]

Depends on where you live. I'm thousands of kilometers away from major datacenters, and every developer worth anything sets up aggressive dependency caching for every CI job because it's unbearable to use otherwise.

Risk management anyone?

Posted Oct 4, 2024 21:40 UTC (Fri) by kleptog (subscriber, #1183) [Link] (5 responses)

Squid proxies only work for HTTP traffic. For HTTPS it doesn't help at all. Since all the package managers (rightfully) depend on HTTPS for security any caching cannot be transparent.

So even of GitLab of Azure DevOps wanted to provide caching for npm or PyPi, they couldn't do it in a way that's transparent. And if they provide a non-transparent mechanism, it makes it a potential MITM.

Ideally there'd be an extension to HTTPS to allow clients to opt into caching, while still preserving the authentication properties of HTTPS, but I think the ship has sailed on that one.

Risk management anyone?

Posted Oct 5, 2024 1:08 UTC (Sat) by dskoll (subscriber, #1630) [Link] (3 responses)

Apt doesn't require https. In fact, all of my sources.list entries are http.

If your packages are signed and you verify the signature, https doesn't buy you anything. If a package is validly-signed, then it doesn't really matter where you downloaded it from.

Risk management anyone?

Posted Oct 5, 2024 2:11 UTC (Sat) by intelfx (subscriber, #130118) [Link] (2 responses)

> If your packages are signed and you verify the signature, https doesn't buy you anything.

That's not strictly true. At the very least, transport-layer encryption buys you confidentiality.

Risk management anyone?

Posted Oct 5, 2024 7:20 UTC (Sat) by LtWorf (subscriber, #124958) [Link]

Unless your attacker can infer that the debian.org hostname is a mirror and uses that information to understand what you're downloading from the sizes of the files that get downloaded.

Risk management anyone?

Posted Oct 5, 2024 14:16 UTC (Sat) by dskoll (subscriber, #1630) [Link]

I don't think confidentiality is an issue for CI/CD pipelines, especially for open-source products where anyone can just look at what exactly the CI/CD pipeline is doing.

Risk management anyone?

Posted Oct 7, 2024 10:56 UTC (Mon) by james (subscriber, #1325) [Link]

Actually, Squid can "bump" or non-transparently MITM HTTPS traffic (and therefore cache it). In a CI environment, this might be a reasonable thing to do, if you consider Squid to be part of the same security domain as the CI environment.

Criminal?

Posted Oct 1, 2024 20:31 UTC (Tue) by raven667 (subscriber, #5198) [Link] (3 responses)

It'd be really funny if any of the threats that Automattic made against WP Engine were legally actionable crimes, since they seem to be well documented, although prosecutors tend not to go after those with deep pockets if they can avoid it as those cases can consume a lot of resources and they are unlikely to win.

Criminal?

Posted Oct 4, 2024 1:00 UTC (Fri) by yeltsin (guest, #171611) [Link]

Maybe: https://wpengine.com/wp-content/uploads/2024/10/Complaint...

I know very little about the US legal process, but this seems important enough to maybe update the article, or even post a separate news entry?

Criminal?

Posted Nov 2, 2024 10:22 UTC (Sat) by jospoortvliet (guest, #33164) [Link] (1 responses)

Nah, it's two companies fighting each other. I don't see why the government should get involved - they only do and should when the public gets harmed. And here, public doesn't really mean 'customers' either, after all - if you have a contract with WP Engine and they don't deliver, they are to blame. If they decided to depend on a 3rd party without a contract because they're trying to make their VC investors as much money as possible by contributing as little as possible to the ecosystem, well - their fault, you can sue them.

Sadly this is the 'business model' of a lot of companies, leaching off of open source projects. Not sure what to do about it, we as community have to find some solution that doesn't mean going closed source or doing stupid shit like Automattic - tricky.

Criminal?

Posted Nov 2, 2024 15:16 UTC (Sat) by Wol (subscriber, #4433) [Link]

> Nah, it's two companies fighting each other. I don't see why the government should get involved - they only do and should when the public gets harmed. And here, public doesn't really mean 'customers' either, after all - if you have a contract with WP Engine and they don't deliver, they are to blame. If they decided to depend on a 3rd party without a contract because they're trying to make their VC investors as much money as possible by contributing as little as possible to the ecosystem, well - their fault, you can sue them.

Except this is (allegedly) Microsoft versus Lotus / WordPerfect / NetScape etc all over again.

Lying or misleading your co-opetition is a serious market offense - called "monopolisation", and from what I can tell, this fits it to a T. The resulting damage to the public is serious, and even 30 - 40 years after the event I would say computing is still not recovered from the damage MS did. Do you really want to see the same long-lasting damage in the Wordpress arena?

The WP Engine claim is basically that Automattica told the markets one thing, and then failed to deliver, enticing their competition to rely on promises that meant nothing. In other words, blatant AntiTrust.

Whether a prosecutor wants to take those claims up, we'll have to see, Whether those claims will hold water, I don't know. But they are clear claims of AntiTrust, and Market Manipulation. Which are clear threats to the public.

Cheers,
Wol


Copyright © 2024, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds