PostgreSQL 19's "scary patch contest"

PostgreSQL 19 was expected to be released in September, in keeping with the database project's longstanding tradition of a major release every year. However, some late-breaking concerns about several of the features slated for inclusion has some developers worried about the quality of the release. On August 25, PostgreSQL contributor Robert Haas sent an email with the subject "scary patch contest" about several patches that have required an unusually large number of bug fixes leading up to the release, which has raised questions about their readiness for a stable release. One of the patches has been reverted, but several are still under heavy revision, and an extra beta release has been slotted in to allow for additional testing.

Scary patches

Development on a major PostgreSQL version begins more than a year before its expected release date; the first CommitFest to review patches for PostgreSQL 19 was held during July 2025, and the first for version 20, which is expected in 2027, was held in July 2026. A release enters feature freeze several months before the final release is scheduled to allow time for the project to publish pre-releases for testing and to fix any bugs found. Typically there are three beta releases released by August, and then a release candidate in late August or early September, with the stable ("GA") release in late September. PostgreSQL 19 entered feature freeze on April 8—the project has been working since to stabilize the final release.

That stabilization phase seems to have involved more bug fixing than usual. In his scary patch email Haas said that he "asked Claude to evaluate which v19 patches were the scariest based on the number and type of bugs fixed post-freeze", and then shared the results "with a few particularly cutting remarks from the LLM edited out". There were several patches that had seen heavy revision after the feature freeze that Haas felt might be in bad enough shape to merit reverting before PostgreSQL 19 shipped, though he did not have a firm position on what to do.

"I think it's pretty clear that none of these were as robust at commit time as we would like, but that doesn't mean that they're still broken." He believed that some of the patches identified by his LLM inspection were low risk, "because they are only going to come up in fairly niche situations", but he worried that some of them could have "a long tail of bugs that we haven't found yet, in pretty critical areas".

One of his examples was a pair of patches from Amit Langote and Junwang Zhao designed to speed up PostgreSQL's foreign-key checks. The first patch provides a way to speed up foreign-key queries by bypassing the usual method of using PostgreSQL's Server Programming Interface and, instead, doing a direct index scan for foreign keys. The second patch builds on that to add batching. According to Haas's email, the two patches had required about 16 fixes after feature freeze, including "five distinct classes of incorrect [foreign key] enforcement (cross-type, domains, column order, nullable keys, non-btree indexes)" and a redesign of the feature's transactional model after the freeze.

Langote said that he had reviewed the fixes made since feature freeze; he was persuaded that the concerns about the batching code were justified and "considerably more concerning" than the per-row fast-path fixes. He acknowledged that both patches had required a number of fixes after the freeze, but the fast-path changes did not have "the same pattern of complexity as the fixes needed to manage the lifetime of batched checks". He sent patches to remove the fast-path batching from PostgreSQL 19, though leaving it in the PostgreSQL 20 development branch, while keeping the per-row fast path intact. On September 8, he promised an updated patch to deal with some comments from Amit Kapila about problems in the first patch to revert the batching feature.

Haas said that he generally agreed with Langote that "the batching stuff feels much riskier than the per-row fast path stuff". He was not entirely sure, though, that the batching patch was in good enough shape to keep for PostgreSQL 20, or that the per-row fast-path patch was good enough to keep in 19. "I think we should do at least as much as what you're proposing here, but possibly more. However, I'm not very sure what the right answer is at this point." For now, the per-row fast-path patch looks as if it will be included in the upcoming beta release.

Two patches contributed by Antonín Houska, and committed by Álvaro Herrera, were also identified as worrisome. The first patch adds a REPACK command to PostgreSQL, with a second patch that implements an option (CONCURRENTLY) for the command. Haas said that the two patches had a cumulative total of 28 bug fixes since the feature freeze, "including data loss [...] and two security-flavored ACL [access-control list] fixes". There is at least one outstanding patch for the CONCURRENTLY option that is still being discussed, which would disallow the use of REPACK CONCURRENTLY on user catalog tables.

Haas said he was most concerned with the two sets of patches described above as well as one from Daniel Gustafsson and Magnus Hagander that allows data checksums to be enabled or disabled in a running PostgreSQL cluster without restricting access to the cluster during processing. Before this patch, enabling or disabling data checksums was only possible as an offline operation. According to the LLM analysis, the patch had required about 25 "substantive fixes" since feature freeze, including false checksum failures. "A corruption-detection feature producing false positives is exactly the wrong failure mode." Gustafsson replied that he would prepare a revert for the feature.

The intent, Haas said, was not to jump straight to reverting the patch: "My sincere intent was to start a conversation, not jump straight to a conclusion." If Gustafsson was already considering reverting the patch then it might make sense to do so, but otherwise Haas was just looking to discuss the patches.

Since then, Gustafsson has continued to work on the feature. On September 4 he addressed the case for reverting the feature and noted that he had posted a patch to revert the feature "to give anyone interested a chance to see what it would look like, and what I propose leaving behind in v19". For now, it seems that the data checksums feature will be in the next beta. Whether it survives to the GA release remains to be seen.

LLM effect

PostgreSQL 19 release management team member Melanie Plageman said that she was not sure if there was consensus that Gustafsson's patch needed to be reverted. She also wondered "if the ease with which LLMs allow people to pressure test features means we are finding more bugs sooner than we have in the past." Nathan Bossart, also a member of the release-management team said that he had thought use of LLMs would make new features more stable by feature freeze, "but it seems that that hasn't turned out to be the case".

However, Haas replied to Plageman that he was not convinced that the problem with Gustafsson's patch was as simple as finding more bugs with LLMs. He was unsure what the right course of action was; it might be fine if the remaining bugs "have few architectural implications and can likely be cleaned up in a minor release without causing much trouble". But he expressed concern that there could be remaining bugs that would not be easily fixed in a point release without bumping the catalog version ("catversion" in PostgreSQL parlance): in other words, making a change that would cause a database created with PostgreSQL 19.1 (for example) to be incompatible with 19.0.

Gustafsson noted that most of the large features in PostgreSQL 19, "were written, reviewed and tested, before AI tools were either available or even remotely as good as they are now". He thought it would be a different story when the feature freeze rolled around for PostgreSQL 20. Kapila agreed, and suggested that LLMs would be good for long-term stability of the project, even though the project was seeing more post-freeze fixes this time around.

LLMs may be better at spotting bugs, but having more time helps as well. Haas observed that all of the features that have had a large number of bug fixes post-freeze were committed close to feature freeze in April. He suggested that "there's a good chance that they would have had most of the bugs flushed out by now" if the features had been committed to PostgreSQL's master branch in August 2025.

PGQs pulled

PostgreSQL core team member Tom Lane said he was "quite afraid" of another patch from Haas's list, an implementation of SQL property graph queries (PGQs) by Peter Eisentraut and Ashutosh Bapat. That feature had seen 17 bug fixes that Haas had characterized as lower severity. Lane worried that the project was still discussing fixes for the feature that might cause a catversion bump. "At this point I'd be willing to bet dinner that if we ship it in v19 there will be post-release bug discoveries that are unfixable until v20."

Haas said, without taking a stance on that particular patch, that no feature that requires a catalog version bump belongs in PostgreSQL 19 at this point. "The time for working out what the catalogs should look like was sometime well in advance of feature freeze, not four months after it".

On September 2, Plageman said that the release-management team had been looking at the current state of the PGQ patch and had "serious concerns about the number of design issues with either recently posted patches or without consensus on behavior". She said that the team felt it would be better to revert PGQ for 19, and fix things properly in time for 20, "without having to be burdened by backwards compatibility and backpatching". Eisentraut agreed and PGQ was ultimately backed out of both PostgreSQL 19 and the master branch. Whether it will resurface in time for PostgreSQL 20 is unclear.

Andres Freund said he was worried about the patch from Paul A. Jungwirth, committed by Eisentraut, that extends PostgreSQL's UPDATE and DELETE commands to add a "FOR PORTION OF" clause to work with temporal tables. (Hubert Lubaczewski has a good blog post that explains the new features.)

Haas said that Jungwirth's patch had 17 fixes, including three security fixes, that had to be addressed after freeze. Freund said that several of the bugs seemed fairly obvious. "Not recognizing & testing some complicated interaction, race conditions, etc. are one thing, but the oversights here seem to be pretty ... broad." He felt that this patch was on the high-risk side because the feature was available to users with relatively low privileges, which meant an attacker did not need to have superuser privileges to do "something intentionally adverse".

Jungwirth reported that Noah Misch had found an additional set of bugs in his FOR PORTION OF feature, "with some LLM investigation". He later said that he had also received a report of a memory leak from Tomas Vondra off list. Between Misch's and Vondra's additional reports, the bug count for Jungwirth's patch is now north of 30 since the feature freeze, and there may be others. So far, there has been no indication it will be removed.

Unprecedented situation

Core team member Bruce Momjian expressed concern about reverting patches like Gustafsson's: "We are in a situation where we might be reverting multiple big features, and we don't even know if their removal will affect other changes that were made assuming those feature existed". He said another beta release would be needed and that it felt like the project was in "an unprecedented situation". Jonathan S. Katz announced the extra beta to be released on September 24, and the release candidate and stable release dates are still unknown.

On September 7, Joshua Drake said that with the number of reverts under discussion the confidence in the release would be lower, whether the features were actually reverted or not. He proposed shifting the release to spring 2027. That would allow more time for testing and remove what he considered a "long-standing issue with the development cycle": the fact that PostgreSQL's testing window takes place during summer (and presumably vacation season) in the Northern hemisphere.

He argued that none of his customers had told him they needed PostgreSQL 19: "most people don't even run 18 [yet] so why are we in such a hurry?" He would rather have 19 ship late than for "all these great features and hard work [to be] gutted". So far, no one has responded to the idea of shifting the release to next year.

It seems most likely that the developers will try to get the release out as close to the original schedule as possible. The project hasn't put out a fourth beta release in recent memory, so it's unclear how many weeks it will allocate for beta testing—at least two weeks seems likely. The stable release is usually two to three weeks after the release candidate, so it seems there's a good chance that PostgreSQL 19 will slide into November even if all goes well from here.

Note: you can avoid this step in the future by logging into your LWN account.