fix uniq -c [LWN.net]

fix uniq -c

Posted Feb 12, 2025 15:37 UTC (Wed) by stijn (subscriber, #570)
Parent article: Rewriting essential Linux packages in Rust

Sounds like a good project; itches scratched and flowers blooming. uniq -c is one of the most irritating commands I know, as the first (count) field is right-justified without any option available to avoid this white-space padding. It is quite puzzling that Richard Stallman let this program loose on the world as it violates usual Unix well-behavedness of textual interfaces. I will make my way over to uutils to see if it has righted this wrong by providing such an option or is open to such. Checking the manual page I see A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars. which again makes me shudder as it seems to indicate failure to respect empty fields in tab-separated data.

to post comments

fix uniq -c

Posted Feb 12, 2025 19:05 UTC (Wed) by jmalcolm (subscriber, #8876) [Link] (2 responses)

Not the author but I imagine his stance is the same as for ripgrep. Writing "better" utilities that fix old wrongs is a different undertaking. These utilities are meant to be drop-in replacements and that means compatibility which means replicating old behaviour regardless of design quality. He is open to extensions like --progress but those are window dressing and not changes in behaviour.

fix uniq -c

Posted Feb 13, 2025 13:30 UTC (Thu) by stijn (subscriber, #570) [Link] (1 responses)

It would equally be possible to add new options that add new behaviour, such as for example outputting well-formatted output, that is, usable for further consumption by adhering to a standard table format such as tab-separated data. I'll enquire.

fix uniq -c

Posted Mar 1, 2025 17:00 UTC (Sat) by jkingweb (subscriber, #113039) [Link]

The trouble with writing a drop-in replacement for X while trying to also improve on X is that as X changes, you may cease to be a drop-in replacement because your enhancements become incompatible with new functionality. MariaDB ran into this problem.

fix uniq -c

Posted Feb 12, 2025 19:06 UTC (Wed) by NYKevin (subscriber, #129325) [Link] (2 responses)

You may dislike that interpretation of "field," but it is specified in the POSIX standard[1]:

> A field is the maximal string matched by the basic regular expression:
> [[:blank:]]*[^[:blank:]]*

But note that this is describing how the -f flag works, not the -c flag (which is also the case for the man page you quote).

As far as I can tell, POSIX makes no allowance whatsoever for right-justifying the -c output, and in fact specifies the opposite:

> If the -c option is specified, the output file shall be empty or each line shall be of the form:
> "%d %s", <number of duplicates>, <line>

(%d means "a number in decimal," not "a number in decimal, but possibly with some whitespace in front of it.")

If a uniq implementation right-justifies its -c output, that is either a bug or a deliberate non-conformance to the standard. I would suggest reporting it upstream if you have not already done so.

[1]: https://pubs.opengroup.org/onlinepubs/9799919799/utilitie...

fix uniq -c

Posted Feb 13, 2025 13:55 UTC (Thu) by stijn (subscriber, #570) [Link]

> But note that this is describing how the -f flag works, not the -c flag (which is also the case for the man page you quote).

Yep I know. It is a horror I noted in passing.

> You may dislike that interpretation of "field," but it is specified in the POSIX standard[1]:

I very much dislike that POSIX standard then. POSIX probably codified existing behaviour, but anyway it does not matter. Clearly existing options have to continue to carry the meaning that they have, I'm not tilting at that windmill.

Coreutils combined with utilities such as datamash can be quite powerful in composing filters, maps or computations in a streaming paradigm, but it does require having a meaningful definition of field. One workhorse in scientific computing is the dataframe encoded as tab-separated data with column headers. This format (bar column headers) is already half-embraced by a lot of unix utilities, I hope this progresses further. A good check is always 'does it work for the empty array / empty string', and this is required to make the tab-separated format usable.

> As far as I can tell, POSIX makes no allowance whatsoever for right-justifying the -c output, and in fact specifies the opposite:

In the savannah gnu git repo the code for uniq has 'if (count_occurrences) printf ("%7jd ", linecount + 1);'

http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;...

I'm not interested in left-justification either - datamash -g 1 count 1 is a more usable alternative.

fix uniq -c

Posted Feb 15, 2025 22:59 UTC (Sat) by pdewacht (subscriber, #47633) [Link]

The "-c" behavior would be a bug in POSIX I assume, since historic Unix uses the equivalent of "%4d %s" and I can't find anybody who has adopted the strict POSIX reading.

fix uniq -c

Posted Feb 16, 2025 16:33 UTC (Sun) by ck (subscriber, #158559) [Link] (3 responses)

You might be setting yourself up for disappointment. uutils is aims to be bug for bug compatible with coreutils. Any deviation is considered a bug.
While I generally like the push to abolish the old coreutils, this compatibility makes it a huge waste of time on my opinion.

fix uniq -c

Posted Feb 17, 2025 13:52 UTC (Mon) by taladar (subscriber, #68407) [Link] (2 responses)

The compatibility is a necessary first step to make anyone consider replacing the old coreutils with these new ones. Once that replacement has happened everywhere I am sure some adjustments could be made (in 10-20 years).

fix uniq -c

Posted Feb 17, 2025 13:55 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

...Ah yes, the classic embrace-extend-extinguish approach.

I thought that was a bad thing?

Embrace-extend-extinguish

Posted Feb 17, 2025 14:15 UTC (Mon) by farnz (subscriber, #17727) [Link]

That depends on the licence of the new project; when Linux embraced, extended and then extinguished proprietary UNIXes, that was a good thing because it moved from a world of gatekeepers to an open source world. When Microsoft attempted to embrace-extend-extinguish the open web with ActiveX, that was a bad thing because it would move us from an open world, to one in which you paid Microsoft as a gatekeeper for technology access.

fix uniq -c

Posted Feb 16, 2025 16:57 UTC (Sun) by dskoll (subscriber, #1630) [Link] (6 responses)

... | uniq -c | sed -e 's/^ *//'

You could wrap that in a shell function or script, I guess.

fix uniq -c

Posted Feb 17, 2025 9:30 UTC (Mon) by stijn (subscriber, #570) [Link] (5 responses)

> ... | uniq -c | sed -e 's/^ *//'

The thought had occurred to me. Implicit in my point here is that this is a fudge, easily fixed by adding a new option to uniq that does the right thing. Shell programming with unix pipes can be an elegant and very concise way to mutate data in a functional way. Having to include fudges like the above (at the expense of a process) grates and creates an impression of crummy (shell) programming that should be completely unnecessary.

fix uniq -c

Posted Feb 17, 2025 11:18 UTC (Mon) by mbunkus (subscriber, #87248) [Link] (4 responses)

I fail to see how this is a problem, let alone a bug. The most important thing here is that the output of such programs can be meant for widely different audiences, each with their own peculiarities:

• humans: we need data formatted so that it visually very clear where columns start & end. We also prefer to be able to determine at a glance when a number is bigger than another number. This means that columns must be aligned in the first place in order to satisfy the first requirement, and for numbers right-aligning satisfies the second. We (humans) might even profit from table borders.
• other programs: here it depends on what the other program is & what it expects as input. For example, if you want to process it further via pipes then then awk & bash don't care at all about the right-aligned numbers[1], whereas other programs might. If your goal isn't pipe-processing but e.g. copy-pasting into spreadsheets, then CSV-formatted data might be much better (though that would make processing in awk/bash much harder)

You cannot satisfy all those requirements with a single format. Therefore I consider your argument to be completely wrong. The default output for uniq is to be easily readable by humans. That's a design choice. It's not a bug.

[1] Examples with bash:

[mosu@velvet ~]$ printf "moo\nmoo\ncow\n" | uniq -c | awk '{ sum += $1 } END { print sum }'
3
[mosu@velvet ~]$ printf "moo\nmoo\ncow\n" | uniq -c | ( while read line ; do set - $line ; echo $1 ; done )
2
1
[mosu@velvet ~]$

fix uniq -c

Posted Feb 17, 2025 18:46 UTC (Mon) by stijn (subscriber, #570) [Link] (2 responses)

I chose a very poor title ('fix unic -c') for what I wrote, implying the presence of a bug. What I meant was

- current default behaviour of uniq -c is poor for composing.
- let's add an option so that we can have the behaviour that I like.

With this, we can have both a 'visually clear' format and a suitable-for-composing format. For compatibility the current format is of course the default in that scenario.

> For example, if you want to process it further via pipes then then awk & bash don't care at all about the right-aligned numbers[1], whereas other programs might.

I work a lot with dataframes, which are essentially mysql tables in tab-separated format with column headers, or equivalently a single table in a spreadsheet, or the things you might want to read with Python pandas or in R. Tab separated is preferred, as I've never encountered a need to escape embedded tab characters. In this wider ecosystem there is no automatic white-space scrubbing of data and a there is a requirement that tables are well-formatted. Programs such comm, join, datamash, shuf and a fair few more can be very handy in summarising, QC'ing or (even) manipulating this data. Hence I clamour for the ability (not necessarily as default) to have all tuple/table type data formatted as tab-separated tables, with or without column names. This should go well with unix composability of processes.

fix uniq -c

Posted Feb 17, 2025 19:18 UTC (Mon) by mbunkus (subscriber, #87248) [Link] (1 responses)

Alright, I think I understand where you're coming from a lot better now. It wasn't just your title, though; in your first comment you wrote:

> It is quite puzzling that Richard Stallman let this program loose on the world as it violates usual Unix well-behavedness of textual interfaces.

And to that my argument was that first and foremost `uniq -c` was most likely designed to be easy to read by humans. By that metric it is very much well-behaved & doesn't violate anything. Furthermore, even with it being designed to be human-readable its output is actually useable as-is by a lot of other traditional Unix programs, making it arguably even less of a "wrong" that has to be "righted" (your choice of words, again from your first post). Apart from awk & bash which I mentioned earlier, "sort -n" works fine as well.

fix uniq -c

Posted Feb 17, 2025 20:15 UTC (Mon) by stijn (subscriber, #570) [Link]

I let a pointless/misdirected rhetorical flourish get the better of me again. I understand the history - and view it as beneficial to strive towards (recasting it from wrong/right) to make outputs (tuples/tables) preferably by default composable or at least add options to do that without assumptions about white-space padding. The table format (mysql dump, spreadsheet, dataframe) is pervasive and super useful. In the right setup shell pipelines can do amazing things with them. Working in those type of table environments has made the various stripes of unix white-space padding and stripping jar with me.

fix uniq -c

Posted Feb 18, 2025 8:11 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

I still think the current behavior is bad.

> • humans: we need data formatted so that it visually very clear where columns start & end. We also prefer to be able to determine at a glance when a number is bigger than another number. This means that columns must be aligned in the first place in order to satisfy the first requirement, and for numbers right-aligning satisfies the second. We (humans) might even profit from table borders.

Then why is it a static number of columns wide? I know to make it actually the right width, the entire output needs to be known so that you can't output anything until the whole thing is read, but if human viewing is most important, why not buffer and Do It Right™? Either the output is small and quick enough to not really matter or it is so large that…what human is really going to be looking at it directly anyways?

> • other programs: here it depends on what the other program is & what it expects as input. For example, if you want to process it further via pipes then then awk & bash don't care at all about the right-aligned numbers[1], whereas other programs might. If your goal isn't pipe-processing but e.g. copy-pasting into spreadsheets, then CSV-formatted data might be much better (though that would make processing in awk/bash much harder)

Sure, awk and bash do separator coalescing. But `cut` doesn't, so one needs to `sed` before `cut`, but not before `bash` and `awk`. Great. Yet another paper cut to remember in shell scripts. Given that `bash` and `awk` do support the mechanism that `cut` would understand and the general human-ness usefulness is of questionable quality…it really seems like an unnecessary quirk of a tool's output.