|
|
Subscribe / Log in / New account

coreutils-9.0 released

The GNU Core Utilities (coreutils) has announced the release of version 9.0 of "the basic file, shell and text manipulation utilities" used by the GNU operating system and various Linux distributions. In the year and a half or so since the last major release (8.32), various new features were added, including:
cp has changed how it handles data
  • enables CoW [copy on write] by default (through FICLONE ioctl),
  • uses copy offload where available (through copy_file_range),
  • detects holes differently (though SEEK_HOLE)
  • This also applies to mv and install.


From:  Pádraig Brady <P-AT-draigBrady.com>
To:  info-gnu-AT-gnu.org
Subject:  coreutils-9.0 released [stable]
Date:  Fri, 24 Sep 2021 15:01:38 +0100
Message-ID:  <b1ebefdb-ea16-a341-67ad-7ff07fec2808@draigBrady.com>
Cc:  coordinator-AT-translationproject.org, coreutils-AT-gnu.org, coreutils-announce-AT-gnu.org

This is to announce coreutils-9.0, a stable release.

This is a new major release, with these significant changes:
   - cp has changed how it handles data
     - enables CoW by default (through FICLONE ioctl),
     - uses copy offload where available (through copy_file_range),
     - detects holes differently (though SEEK_HOLE)
     - This also applies to mv and install.

   - utilities are more tuned to the hardware available
     - wc uses avx2 instructions to count lines
     - cksum uses pclmul instructions for --algorithm=crc
       - Other digest tunings remain delegated to libcrypto

   - More amalgamation of utilities
     - cksum now supports the -a option to select any digest.
     - This is the preferred interface, rather than sha*sum etc.
     - This is similar to the amalgamation of encoding utilities
       introduced in the basenc command in v8.31.
See the NEWS below for more details.

Thanks to everyone who has contributed!
There have been 257 commits by 25 people in the 81 weeks since 8.32
   Andreas Schwab (1)              KOBAYASHI Takashi (2)
   Arman Absalan (1)               Kamil Dudka (4)
   Assaf Gordon (1)                Kristoffer Brånemyr (3)
   Ben Pfaff (1)                   Nikolay Nechaev (1)
   Benno Schulenberg (1)           Nishant Nayan (1)
   Bernhard Voelker (17)           Paul Eggert (97)
   Carl Edquist (2)                Pádraig Brady (110)
   Emanuele Giacomelli (1)         Tianjia Zhang (1)
   Erik Auerswald (1)              Tim Gates (1)
   Grigorii Sokolik (2)            Tobias Stoeckmann (1)
   Jason Kim (1)                   Zorro Lang (1)
   Jim Meyering (7)                nl6720 (1)
   Justin Tracey (1)

Pádraig [on behalf of the coreutils maintainers]

==================================================================

Here is the GNU coreutils home page:
    https://gnu.org/software/coreutils/

For a summary of changes and contributors, see:
    https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;...
or run this command from a git-cloned coreutils directory:
    git shortlog v8.32..v9.0

To summarize the 1615 gnulib-related changes, run these commands
from a git-cloned coreutils directory:
    git checkout v9.0
    git submodule summary v8.32

==================================================================


Here are the compressed sources:
   https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.gz   (14MB)
   https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.xz   (5.4MB)

Here are the GPG detached signatures[*]:
   https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.gz.sig
   https://ftp.gnu.org/gnu/coreutils/coreutils-9.0.tar.xz.sig

Use a mirror for higher download bandwidth:
   https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

027a318930f295cb5bbc0dd06fb47a3b8552fc80  coreutils-9.0.tar.gz
b9TriKUVAEl3/HLX9HtAYgQJzEHfrwBBn90b4XZjxDQ  coreutils-9.0.tar.gz
e2623469f37259d4a89ced5f91af5eaf0ab8792d  coreutils-9.0.tar.xz
zjCs30pBvFuzDdlV6eqnX6IWtOPesIiJ7TJDPHs7l84  coreutils-9.0.tar.xz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

   gpg --verify coreutils-9.0.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

   gpg --keyserver keys.gnupg.net --recv-keys DF6FD971306037D9

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
   Autoconf 2.71
   Automake 1.16.4
   Gnulib v0.1-4937-g9aca7b673
   Bison 3.7.4

==================================================================

NEWS

* Noteworthy changes in release 9.0 (2021-09-24) [stable]

** Bug fixes

   chmod -v no longer misreports modes of dangling symlinks.
   [bug introduced in coreutils-5.3.0]

   cp -a --attributes-only now never removes destination files,
   even if the destination files are hardlinked, or the source
   is a non regular file.
   [bug introduced in coreutils-8.6]

   csplit --suppress-matched now elides the last matched line
   when a specific number of pattern matches are performed.
   [bug introduced with the --suppress-matched feature in coreutils-8.22]

   df no longer outputs duplicate remote mounts in the presence of bind mounts.
   [bug introduced in coreutils-8.26]

   df no longer mishandles command-line args that it pre-mounts
   [bug introduced in coreutils-8.29]

   du no longer crashes on XFS file systems when the directory hierarchy is
   heavily changed during the run.
   [bug introduced in coreutils-8.25]

   env -S no longer crashes when given unusual whitespace characters
   [bug introduced in coreutils-8.30]

   expr no longer mishandles unmatched \(...\) in regular expressions.
   [bug introduced in coreutils-6.0]

   ls no longer crashes when printing the SELinux context for unstatable files.
   [bug introduced in coreutils-6.9.91]

   mkdir -m no longer mishandles modes more generous than the umask.
   [bug introduced in coreutils-8.22]

   nl now handles single character --section-delimiter arguments,
   by assuming a second ':' character has been specified, as specified by POSIX.
   [This bug was present in "the beginning".]

   pr again adjusts tabs in input, to maintain alignment in multi column output.
   [bug introduced in coreutils-6.9]

   rm no longer skips an extra file when the removal of an empty directory fails.
   [bug introduced by the rewrite to use fts in coreutils-8.0]

   split --number=K/N will again correctly split chunk K of N to stdout.
   Previously a chunk starting after 128KiB, output the wrong part of the file.
   [bug introduced in coreutils-8.26]

   tail -f no longer overruns a stack buffer when given too many files
   to follow and ulimit -n exceeds 1024.
   [bug introduced in coreutils-7.5]

   tr no longer crashes when using --complement with certain
   invalid combinations of case character classes.
   [bug introduced in coreutils-8.6]

   basenc --base64 --decode no longer silently discards decoded characters
   on (1024*5) buffer boundaries
   [bug introduced in coreutils-8.31]

** Changes in behavior

   cp and install now default to copy-on-write (COW) if available.

   cp, install and mv now use the copy_file_range syscall if available.
   Also, they use lseek+SEEK_HOLE rather than ioctl+FS_IOC_FIEMAP on sparse
   files, as lseek is simpler and more portable.

   On GNU/Linux systems, ls no longer issues an error message on a
   directory merely because it was removed.  This reverts a change
   that was made in release 8.32.

   ptx -T no longer attempts to substitute old-fashioned TeX escapes
   for 8-bit non-ASCII alphabetic characters.  TeX indexes should
   instead use '\usepackage[latin1]{inputenc}' or equivalent.

   stat will use decomposed (major,minor) device numbers in its default format.
   This is less ambiguous, and more consistent with ls.

   sum [-r] will output a file name, even if only a single name is passed.
   This is consistent with sum -s, cksum, and other sum(1) implementations.

** New Features

   cksum now supports the -a (--algorithm) option to select any
   of the existing sum, md5sum, b2sum, sha*sum implementations etc.
   cksum now subsumes all of these programs, and coreutils
   will introduce no future standalone checksum utility.

   cksum -a now supports the 'sm3' argument, to use the SM3 digest algorithm.

   cksum --check now supports auto detecting the digest type to use,
   when verifying tagged format checksums.

   expr and factor now support bignums on all platforms.

   ls --classify now supports the "always", "auto", or "never" flags,
   to support only outputting classifier characters if connected to a tty.

   ls now accepts the --sort=width option, to sort by file name width.
   This is useful to more compactly organize the default vertical column output.

   ls now accepts the --zero option, to terminate each output line with
   NUL instead of newline.

   nl --line-increment can now take a negative number to decrement the count.

   stat supports more formats for representing decomposed device numbers.
   %Hd,%Ld and %Hr,%Lr will output major,minor device numbers and device types
   respectively.  %d corresponds to st_dev and %r to std_rdev.

** Improvements

   cat --show-ends will now show \r\n as ^M$.  Previously the \r was taken
   literally, thus overwriting the first character in the line with '$'.

   cksum [-a crc] is now up to 4 times faster by using a slice by 8 algorithm,
   and at least 8 times faster where pclmul instructions are supported.
   A new --debug option will indicate if pclmul is being used.

   md5sum --check now supports checksum files with CRLF line endings.
   This also applies to cksum, sha*sum, and b2sum.

   df now recognizes these file systems as remote:
   acfs, coda, fhgfs, gpfs, ibrix, ocfs2, and vxfs.

   rmdir now clarifies the error if a symlink_to_dir/ has not been traversed.
   This is the case on GNU/Linux systems, where the trailing slash is ignored.

   stat and tail now know about the "devmem", "exfat", "secretmem", "vboxsf",
   and "zonefs" file system types.  stat -f -c%T now reports the file system
   type, and tail -f uses polling for "vboxsf" and inotify for the others.

   timeout now supports sub-second timeouts on macOS.

   wc is up to 5 times faster when counting only new line characters,
   where avx2 instructions are supported.
   A new --debug option will indicate if avx2 is being used.

-
Also posted at https://savannah.gnu.org/forum/forum.php?forum_id=10052

-- 
If you have a working or partly working program that you'd like
to offer to the GNU project as a GNU package,
see https://www.gnu.org/help/evaluation.html.


to post comments

coreutils-9.0 released

Posted Sep 24, 2021 18:56 UTC (Fri) by james (subscriber, #1325) [Link] (15 responses)

ls --zero | xargs -0 ... looks like a useful addition to the toolkit.

coreutils-9.0 released

Posted Sep 24, 2021 19:37 UTC (Fri) by jond (subscriber, #37669) [Link] (6 responses)

It'll take me a long while to overcome the ingrained mantra "never parse the output of ls"

coreutils-9.0 released

Posted Sep 24, 2021 19:57 UTC (Fri) by dskoll (subscriber, #1630) [Link] (5 responses)

Yes, but ls -1 --zero is probably safe and looks kind of pleasing, as long as you don't try to read it out over the phone to someone. "ls minus one minus minus zero..."

coreutils-9.0 released

Posted Sep 24, 2021 20:05 UTC (Fri) by jkingweb (subscriber, #113039) [Link]

Oh, dear. I had start writing a message saying "Well, you could use 'dash' instead of 'minus'!", only to realize belatedly that zero is spelt out while one is not.

coreutils-9.0 released

Posted Sep 25, 2021 12:10 UTC (Sat) by tux3 (subscriber, #101245) [Link]

It has a very http://slashdot.org ring to it

ls -1 --zero and the phonetic alphabet

Posted Sep 26, 2021 22:24 UTC (Sun) by stephen.pollei (subscriber, #125364) [Link] (2 responses)

lemma sierra space dash one space dash dash zulu echo romeo oscar

ls -1 --zero and the phonetic alphabet

Posted Sep 27, 2021 14:32 UTC (Mon) by mradziej (subscriber, #124815) [Link] (1 responses)

Lima, not lemma! Learn your alphabet!

ls -1 --zero and the phonetic alphabet

Posted Sep 27, 2021 18:14 UTC (Mon) by Wol (subscriber, #4433) [Link]

Well if you heard it, lemma and lima are close - WHICH IS THE ENTIRE POINT.

You shouldn't be able to confuse the letters across dialects, accents, or whatever.

Cheers,
Wol

coreutils-9.0 released

Posted Sep 25, 2021 5:03 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (6 responses)

Can't you already do that with some arcane find invocation? Or is this a "don't make me dig out the man page info doc" quality-of-life feature?

coreutils-9.0 released

Posted Sep 25, 2021 7:56 UTC (Sat) by pabs (subscriber, #43278) [Link]

find -mindepth 1 -maxdepth 1 -print0 | xargs -0 ...

coreutils-9.0 released

Posted Sep 25, 2021 17:41 UTC (Sat) by Hello71 (subscriber, #103412) [Link] (3 responses)

in the vast majority of cases, a single directory won't contain enough files to overflow ARG_MAX, so one would simply write the logical "handle *" command. this is potentially vulnerable to file names starting with hyphens, but so is the ls version.

if you really want to use xargs though, either because you have a truly enormous directory or because you want to use -P, the sane invocation is the admittedly even-more-arcane "printf '%s\0' ./* | xargs -0 -Pn handle". printf is a built-in and exempt from ARG_MAX in all modern Bourne-like shells, so this will work regardless of the length of file names.

coreutils-9.0 released

Posted Sep 25, 2021 18:10 UTC (Sat) by rbtree (guest, #129790) [Link]

> this is potentially vulnerable to file names starting with hyphens

Prefix it with a path and it should be fine: `cmd ./*`

Or the usual `cmd -- *`, if your command supports it.

coreutils-9.0 released

Posted Sep 28, 2021 8:26 UTC (Tue) by niner (subscriber, #26151) [Link] (1 responses)

It's not just single, large directories. It can also be something like ls */*.foo. Doesn't take that many directories and files per directory to exceed ARG_MAX.

coreutils-9.0 released

Posted Oct 1, 2021 10:56 UTC (Fri) by xav (guest, #18536) [Link]

... or ls -R

coreutils-9.0 released

Posted Oct 5, 2021 8:08 UTC (Tue) by murukesh (subscriber, #97031) [Link]

The various sort options that ls supports are quite cumbersome to get done with find.

coreutils-9.0 released

Posted Sep 25, 2021 10:59 UTC (Sat) by swilmet (subscriber, #98424) [Link]

Instead of xargs, I often use GNU parallel. For simple cases its syntax is more handy.

coreutils-9.0 released

Posted Sep 25, 2021 16:00 UTC (Sat) by NightMonkey (subscriber, #23051) [Link] (1 responses)

Thanks for continuing this awesome volunteer work! So much value has come from the software ecosystem that these programs help enable.

Its important for us as users to remember that, yes, there are people who maintain programs like 'ls' - programs that seem to exist in a vaccum, that seem to just 'be', that feel "authorless". To me, they don't seem to be doing it for the riches and fame... just for love of programming and of operating systems. :) Hats off to them.

coreutils-9.0 released

Posted Sep 26, 2021 7:36 UTC (Sun) by oldtomas (guest, #72579) [Link]

Yes, absolutely. Thank you, folks!

coreutils-9.0 released

Posted Sep 26, 2021 10:03 UTC (Sun) by bluss (guest, #47454) [Link]

XFS got support for reflink: https://lwn.net/Articles/684826/ but ext4 never seemed to develop in that direction. I guess it's unlikely it will come up again?

coreutils-9.0 released

Posted Sep 28, 2021 8:11 UTC (Tue) by Homer512 (subscriber, #85295) [Link]

Re wc with AVX2: Kind of a shame that GCC's vectorizer and the vector extensions themselves are a bit lacking. The pattern that wc uses worked that way since MMX. But you have to replicate a lot of code with minor tweaks to make it work for all styles of vector instruction sets plus dealing with dynamic dispatch depending on the actual CPU.

coreutils-9.0 released

Posted Oct 10, 2021 17:15 UTC (Sun) by motiejus (subscriber, #92837) [Link] (1 responses)

Nice!

Looking forward to untraining `cp --reflink` muscle memory; from the way I read this, it will do a CoW copy on btrfs without `--reflink`?

coreutils-9.0 released

Posted Oct 10, 2021 17:32 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

Yes, that seems clear, from the announcement:

"cp and install now default to copy-on-write (COW) if available"


Copyright © 2021, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds