Public-inbox 2.0.0 released
Version 2.0.0 of public-inbox, the mail archiving system behind
lore.kernel.org and LWN's email archive, has been released. "
This release includes several new features and fixes; mostly around improved integration between inboxes and coderepos for solver. Portability and reliability is also improved, especially in the internal process management of lei."
| From: | Eric Wong <e-AT-80x24.org> | |
| To: | meta-AT-public-inbox.org | |
| Subject: | [ANNOUNCE] public-inbox 2.0.0 | |
| Date: | Fri, 07 Nov 2025 23:59:45 +0000 | |
| Message-ID: | <20251107235945.M850058@sed> | |
| Archive-link: | Article |
This release includes several new features and fixes; mostly
around improved integration between inboxes and coderepos for
solver. Portability and reliability is also improved, especially
in the internal process management of lei.
Search now supports `thread:{SUBQUERY}' for installations
with Xapian development files (and necessary C++ toolchain).
More Xapian functionality will be available in the future
for users with development files installed.
public-inbox-cindex is a new command to index coderepos for
WWW search and perform automatic associations between
coderepos and inboxes. This makes blob reconstruction via
solver vastly more useful for the WWW UI as admins no longer
have to manually associate coderepos with inboxes.
public-inbox-clone gains the ability to mirror entire (or partial)
grokmirror-compatible manifests.
Internal process and object management data structures are vastly
simplified throughout and error handling made more robust.
Pathologically slow Xapian queries no longer block non-Xapian
requests when using the new --xapian-helpers switch.
git SHA-256 support remains a work-in-progress for inboxes and
extindex due to the need to interoperate with SHA-1 epochs.
Upgrading:
lei users must "lei daemon-kill" after installation to load
new code. Normal daemons (read-only, and public-inbox-watch)
will also need restarts, of course, but there's no
backwards-incompatible data format changes so rolling back to
older versions is harmless.
Compatibility:
Uppercase newsgroup names were always broken with IMAP, POP3, and
-extindex. Uppercase names will now be lowercased by default and
warnings will be emitted. Conflicting newsgroup names (and `inboxdir'
entries if `newsgroup' isn't specified) will also generate warnings
since they break -extindex and the new -cindex (coderepo index).
New users + hackers:
The install/ directory includes tools to automate installation and
removal of dependencies for stripped-down or full setups. See
install/README for more details.
devel/try-lei now exists for testing out an empty lei instance
without writing to an existing one.
treewide
* support raw UTF-8 headers from SMTPUTF8 hosts
* standardize on `#' prefix for stderr diagnostics (previously `I:')
* SHA-256 coderepos are fully supported (but not inboxes, yet)
* for daemons serving public traffic, MALLOC_MMAP_THRESHOLD_=131072 is
recommended to reduce fragmentation in glibc malloc, while jemalloc
(tested as an LD_PRELOAD) is another option (at least for 64-bit).
* embrace ->DESTROY in more places for control flow and reliability
* enforced error handling for ENOSPC, FS corruption, etc. for
parallel indexing
PublicInbox::WWW
* support `+' in inbox names
* support coderepo displays for systems without cgit
(example at <https://80x24.org/public-inbox.git/>)
* improve display of git tags, commits and trees in $INBOX/$OID/s/ endpoint
* numerous memory usage reductions by avoiding Perl scratchpads
* add #related anchor and search form to find related patches
based on blob OIDs (IOW, exposing `lei p2q' to the web)
* fix footer in listing of >200 inboxes
* support dumb HTTP clones of SHA-256 git repos
* add /$INBOX/$MSGID/d/ endpoint to show diffs in reused Message-IDs
(`lei mail-diff' for the web)
* support POST /$INBOX/$MSGID/?x=m&q= to limit mbox results to a thread
* add topics_(new|active).(html|atom) endpoints
* linkify peer public-inbox addresses in To/Cc headers
* publicinbox.nameIsUrl allows admins to avoid specifying
a publicinbox.*.url entry for every single inbox
public-inbox-watch:
* watching MH folders is now supported
* publicinbox.*.watch=false can prevent attempted writes due to
publicinboxwatch.watchspam
lei
* use http.proxy / http.<remote>.proxy from system-wide git-config if
unconfigured for lei
* improve IMAP and NNTP error reporting
* reduce default IMAP connections to avoid overloading servers
* compatibility with SQLite <3.8.3 on CentOS 7.x
* fix `lei q -tt' on locally indexed messages (still broken for remotes:
https://public-inbox.org/meta/20230226170931.M947721@dcvr/ )
* `lei q --thread-id=$MSGID' is now supported to search for messages
within certain thread
* `lei import' now sets labels+keywords consistently on all
already imported messages
* `lei import' is less noisy by default, with a new --noisy switch
to restore previous behavior.
* fix `lei up' on saved local queries which previously used -t/--threads
* `lei convert' output to v2 public-inboxes is now idempotent
* improved bash completion for labels (see contrib/completion)
* support for reading (but not writing) MH folders
* `lei index' accepts `+L:$LABEL' like `lei import' does
* fix several reliability problems with `lei import'
solver (used by lei (rediff|blob), and PublicInbox::WWW)
* handle copies in patches properly
* no longer redundantly parallelized within each WWW process
* improve ambiguous OID debug messages
* can use coderepo indices generated by public-inbox-cindex
portability
* SIGWINCH is handled properly on less common architectures and OSes
* fix EINTR handling for kqueue users
* various fixes for CentOS 7.x
* fix `git cat-file --batch' on systems with small getdelim(3)
buffers (mainly affects musl)
* support Alpine Linux, Dragonfly, NetBSD and OpenBSD. This resulted
not only in bugfixes to our code, but also to Dragonfly and OpenBSD.
* Inline::C||Socket::MsgHdr no longer required for SCM_RIGHTS
with sendmsg/recvmsg on supported *BSDs.
* inotify support no longer requires Linux::Inotify2 XS package
for most architectures
public-inbox-learn:
* support --keep-going/-k switch to continue after unwritable inboxes
public-inbox-mda / public-inbox-learn / public-inbox-watch:
* support publicinboxImport.dropUniqueUnsubscribe config to
remove unique identifiers in List-Unsubscribe headers
public-inbox-pop3d
* support `limit=NUM' and `initial_limit=NUM' query parameters
in mailbox names to limit results
public-inbox-nntpd
* fix LISTGROUP with range (affects neomutt)
public-inbox-clone / public-inbox-fetch / `lei add-external --mirror'
* mtime of downloaded manifest preserved
public-inbox-clone:
* parallel mirroring of multiple inboxes/coderepos via manifest,
public-inbox-fetch is not used in this mode
* new flags to support manifest mirroring include:
--dry-run, --inbox-config=, --project-list=, --prune, --purge,
--keep-going, --jobs, --include=, --exclude=, --objstore=,
--manifest=, --remote-manifest=
See public-inbox-clone(1) man page for more details.
PublicInbox::SaPlugin::ListMirror
* List-ID handling special-cased according to RFC 2919 rules
Search improvements (lei and PublicInbox::WWW)
* quoted text inside base-85 binary patches is no longer indexed
* `public-inbox-cindex --join' prefers using Xapian's C++ API
directly to avoid Perl method dispatch overhead to get usable
performance associating ~300 inboxes with over 1K coderepos
(and vice versa). Users requiring such performance will need
a C++ compiler, pkg-config, and the Xapian development files
(see INSTALL).
* -X/--xapian-helpers may be specified for public-facing daemons
to use aforementioned C++ helper (or SWIG/XS fallback) to avoid
expensive searches blocking non-search HTTP and IMAP requests.
* the aforementioned C++ helper supports `thread:{SUBQUERY}' and
queries `thread:MSGID' like notmuch(1)
* --reindex and long-running (ext)index operations will auto
checkpoint every 5s to avoid SQLite timeouts in public-facing
frontends where SQLite WAL isn't possible, use
`--commit=SECONDS'
* --commit=SECONDS adjusts commit interval to increase or decrease
the rate at which SQLite and Xapian data is made available to
read-only processes
* --wal switch added to public-inbox-init(1), public-inbox-index(1),
public-inbox-extindex(1), public-inbox-convert(1) to enable WAL
for SQLite, which allows higher commit intervals to be used without
causing readers to timeout.
* publicinbox.$NAME.altid is now supported with public-inbox-extindex
* publicinbox.$NAME.indexheader added to support custom header indexing,
see public-inbox-config(5) for details.
* --defrag=MSGCOUNT defragments SQLite and Xapian DBs on btrfs(5)
* --split-shards speeds up initial Xapian indexing and reduces
fragmentation
* --no-multi-pack-index disables writing git-multi-pack-index(1) to
avoid OOM errors on memory constrained systems
* --cow / --no-cow may be used with public-inbox-init(1),
public-inbox-compact(1), public-inbox-convert(1),
public-inbox-cindex(1), public-inbox-xcpdb(1),
public-inbox-index(1), public-inbox-extindex(1)
to enable or disable (the default) No_COW attribute on
Linux CoW filesystems (e.g. btrfs).
public-inbox-daemon (-netd, -imapd, -httpd, -nntpd, -pop3d)
* per-listener backlog=, servername=, serverport=, and multi-accept=
parameters, --multi-accept= is also supported globally for
all listeners.
* listen(2) backlog no longer overridden when inheriting listeners
at startup, respecting `Backlog=' in systemd.socket(5)
* premature HTTP client disconnects are detected on expensive
requests (e.g. solver) and can be logged as 499 (like nginx)
using Plack::Middleware::AccessLog(::Timed).
Thanks to all the bug reporters and users who made this release
possible, and thanks for bearing with my anxiety over making releases.
Please report bugs via plain-text mail to: meta@public-inbox.org
See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.