Python gets a "Developer-in-Residence" [LWN.net]

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 1:14 UTC (Thu) by jerojasro (guest, #98169) [Link] (4 responses)

While getting your code reformatted on version-bump is quite annoying (hasn't happened to me, though, but I take your word for it), I don't really see the fuss about it, or the need for all-caps, at least in *this* article; the code-formatting library is one of the few that I'm quite comfortable pinning to a specific version for a project's lifetime to avoid the issue you mention.

Your comment feels very OT, indeed.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 5:21 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (3 responses)

Well it's not a library that is part of the project. When developers are having different distributions they will get different versions, so their reformat will keep changing the same stuff back and forth.

Just thought of saving people some waste of time, as i wish I knew about this.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 14:24 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

This is why, IMO, code formatting should be enforced at the integration time rather than development time (such as as a commit hook). Instead, CI should verify it and rewrite the branch (or just make a "reformat new code" commit if you're ok with that…) to apply the formatting as desired. Then everyone agrees on what version of the formatter is being used and you don't have your Debian developers "fighting" with your Arch developers via their formatting tool availability.

Since the projects I work on only have a single release branch maintained at a time, right before we branch is a great time to update the formatter (clang-format has the same issue as black described at the top here) and reformat the entire repository using its updated opinions. With more than one release branch, there's no good time to reformat wholesale however.

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 7:56 UTC (Mon) by massimiliano (subscriber, #3048) [Link] (1 responses)

This is why, IMO, code formatting should be enforced at the integration time rather than development time (such as as a commit hook).

I have a slightly different take on this.

IMHO, CI should never rewrite (reformat) your code, it should just fail the build if it is malformed (just like if a test failed).

What commit hooks should do, instead, is run tools that are in sync with whatever CI is doing on that branch, so that they will not introduce failed builds. I usually achieve this making so that every dev has the "official" development environment available using docker, the Dockerfile is in the repo, and the scripts that the developer (or commit hook) uses run the same tools that CI runs (because CI ultimately uses the same Dockerfile: it's in the repo).

Keeping the Dockerfile in the repo is what keeps CI and the dev env in sync. It is a technicality, an implementation detail, but the general principle is to specify the development tools unambiguously and make so that both developers and CI use exactly the same tools at every commit by keeping that specification in the same repo as the code. Often using docker is a practical way to do it, but of course, others are possible.

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 11:42 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

CI doesn't *automatically* do it. You need a `Do: reformat` command (so you can plan for it at least).

> it should just fail the build if it is malformed (just like if a test failed).

Given how unsure many developers are with `git rebase -i` and resolving intra-branch conflicts due to reformatting, this will just end up littering the history with useless "fix clang-format" commits instead of people integrating the diffs back into the appropriate commit.

> I usually achieve this making so that every dev has the "official" development environment available using docker

That assumes you can even *have* an "official" development environment. It also doesn't consider non-Linux platforms at all.

> run the same tools that CI runs (because CI ultimately uses the same Dockerfile: it's in the repo).

One of the design decisions we had when making our workflow tool (ghostflow) is that *no* code in the repo is trusted, so none of it is executed (be it in a Dockerfile or no). It can indicate *what* to run (e.g., "file X uses clang-format-9, but file Y is third party code, so ignore it"), but how it is run is in the workflow deployment, not the repository.

> Keeping the Dockerfile in the repo is what keeps CI and the dev env in sync. It is a technicality, an implementation detail, but the general principle is to specify the development tools unambiguously and make so that both developers and CI use exactly the same tools at every commit by keeping that specification in the same repo as the code. Often using docker is a practical way to do it, but of course, others are possible.

We're not making a *product*, so there's not "one build" that is blessed. We get bugs about it not working on Windows, macOS, Arch, Fedora, Ubuntu, HPC deployments, weird MPI deployments, vcpkg, etc. A diverse development base helps flush things out that we just can't test for.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 7:53 UTC (Thu) by hynek (subscriber, #55397) [Link] (13 responses)

As someone who has used Black since the earliest alphas, I have to say this is a misrepresentation.

Łukasz and his team are doing their best to keep changes meaningful and at a minimum. It has been _months_ since it happened to any of my many code bases, and it was always worth it.

> Interestingly formatting a formatted code with a newer version and then reformatting with the older version will not reset the format completely.

Since Black goes for a canonical code form, the only reason why this could be true is a bug. Software sometimes has bugs, it's sad but true. On the other hand, YELLING that nobody should should use Black is rude FUD. It certainly isn't an outlier bug-wise.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 10:05 UTC (Thu) by rsidd (subscriber, #2582) [Link] (9 responses)

Thanks. I hadn't encountered Black before, but on reading the docs, it certainly looks like any version differences are minor and shouldn't restyle everything (if already restyled by a previous version of Black). But it does sound like you should use only Black and not mix it with any other styler. So the shouty comment does sound like FUD. The author probably had a genuine bad experience, but there are more productive ways to share it.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 12:28 UTC (Thu) by hynek (subscriber, #55397) [Link] (5 responses)

Yes, since Black formats everything to a canonical shape, it makes no sense to have other formatters.

With one exception: isort, because Black doesn’t format imports.

Python gets a "Developer-in-Residence"

Posted Jul 29, 2021 19:14 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

> With one exception: isort, because Black doesn’t format imports.

AIUI, this is a deliberate choice on the part of Black, because imports can have side effects.

(Imports *should not* have side effects. You should be able to reorder them without breaking anything. But bad code exists, so...)

Python gets a "Developer-in-Residence"

Posted Jul 30, 2021 7:37 UTC (Fri) by hynek (subscriber, #55397) [Link] (1 responses)

It's even more complicated than that.

Since PEP 8 asks you to sort your import into blocks of standard library, 3rd party packages, and app packages, a non-trivial insight into the packaging situation of the current app/package is required. Thus it makes sense to punt on it, since it would open a completely new can of worms and the code is already present in isort.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 21:57 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

Right. But Black's documentation also says that it is not supposed to change the AST at all, except in a small number of special cases which should not affect runtime behavior.[1] Since import-reordering actually could change the runtime behavior, I think they consider it permanently out-of-scope, rather than just "too hard."

(As mentioned before, there is no good reason for import-reordering to change runtime behavior. It's just that Python doesn't have any hard limits which prevent you from shooting yourself in the foot like that.)

[1]: https://black.readthedocs.io/en/stable/the_black_code_sty...

Python gets a "Developer-in-Residence"

Posted Jul 30, 2021 8:22 UTC (Fri) by rsidd (subscriber, #2582) [Link] (1 responses)

If I understand the Black docs right, Black formats imports (I just checked: it does), but doesn't sort them?

Python gets a "Developer-in-Residence"

Posted Jul 30, 2021 14:01 UTC (Fri) by kpfleming (subscriber, #23250) [Link]

Correct. (plus some characters)

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 7:43 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (2 responses)

> So the shouty comment does sound like FUD. The author probably had a genuine bad experience, but there are more productive ways to share it.

It's not FUD if it's real.

It's a tool that in theory wants to be like go fmt, but in practice isn't, and creates the need to reformat the whole codebase (and pollute commit history) periodically.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 10:53 UTC (Sat) by rsidd (subscriber, #2582) [Link] (1 responses)

It aims to minimize diffs. The version-to-version changes suggest that some small specific bits, at most, would be reformatted, so the diff from formatting a previously Black-formatted file with a new Black version should be tiny.

If you really have an example where the entire file was changed by a new version of Black, why not report that as a bug? I don't see how that can happen with any format tool, actually.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 13:42 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

> I don't see how that can happen with any format tool, actually.

The usual culprits in my experience include (in a rough order):

- line length breaking heuristics
- updates to the language (e.g., C++03 → C++11 made lots of "> >" → ">>" diff noise)
- more insight into comments (reflowing paragraphs or such) or macros

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 7:42 UTC (Sat) by LtWorf (subscriber, #124958) [Link] (2 responses)

> It has been _months_ since it happened

It should just not happen, because commits that change hundreds of files at once, with "style" as description make going back in history incredibly hard.

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 15:21 UTC (Sat) by rsidd (subscriber, #2582) [Link] (1 responses)

You have a python project with hundreds of files? That was bitten by this issue where every file was totally changed by a new version of black? Is it then that hard to report a bug, saying, file example x formatted by black version y, then reformatted by black version z, has an enormous diff?

Python gets a "Developer-in-Residence"

Posted Jul 31, 2021 17:51 UTC (Sat) by LtWorf (subscriber, #124958) [Link]

Since I only use stable distributions, how is that going to help? The authors can't change the past, and any "fix" will only generate more diff.

My issue is more on the general approach of the project, that seems to be ok with periodically having to reformat stuff.

Python gets a "Developer-in-Residence"

Posted Aug 1, 2021 15:03 UTC (Sun) by sjj (guest, #2020) [Link]

Apparently some code formatters can create deep emotional traumas that burst forth in unrelated blog comment sections. Dangerous stuff.

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 13:10 UTC (Mon) by obspsr (subscriber, #56917) [Link] (3 responses)

What is the reason of calling a formatter "Black" ? ! Here in Europe we do not understand US jokes, hardly KerryMen's jokes so ... :-((

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 13:24 UTC (Mon) by farnz (subscriber, #17727) [Link] (2 responses)

The author of the "Black" code formatter is Polish, and IIUC chose the name because it's a formatter that "gives you any colour you want, as long as it's black", in reference to the apocryphal quote from Henry Ford about the Model T.

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 13:43 UTC (Mon) by obspsr (subscriber, #56917) [Link]

For sure ! ... I remember now that Henri Ford told that but my grand-father was the first reseller of General Motors in 1925 and
we where more GM and Sloan oriented ! Thanks.

Python gets a "Developer-in-Residence"

Posted Aug 2, 2021 13:50 UTC (Mon) by excors (subscriber, #95769) [Link]

> The author of the "Black" code formatter is Polish, and IIUC chose the name because it's a formatter that "gives you any colour you want, as long as it's black", in reference to the apocryphal quote from Henry Ford about the Model T.

The documentation for Black prominently says "Any color you like", so that does sound like the source of the name.

Regarding the authenticity of the quote, Henry Ford's autobiography says: (https://en.wikisource.org/wiki/My_Life_and_Work/4)

> No business can improve unless it pays the closest possible attention to complaints and suggestions. If there is any defect in service then that must be instantly and rigorously investigated, but when the suggestion is only as to style, one has to make sure whether it is not merely a personal whim that is being voiced. Salesmen always want to cater to whims instead of acquiring sufficient knowledge of their product to be able to explain to the customer with the whim that what they have will satisfy his every requirement--that is, of course, provided what they have does satisfy these requirements.
>
> Therefore in 1909 I announced one morning, without any previous warning, that in the future we were going to build only one model, that the model was going to be "Model T," and that the chassis would be exactly the same for all cars, and I remarked:
>
> "Any customer can have a car painted any colour that he wants so long as it is black."

It seems colour wasn't really the point - it was to illustrate his broader idea that it was better to just produce a single model of car and tell customers that that's what they ought to want, in order to lower production costs and widen the market for cars, which would more than make up for any losses in the luxury end where customers expect companies to cater to their whims.

(It's possible that Ford or his ghostwriter made up the quote with the benefit of hindsight, but at least it's a less apocryphal source than many famous quotes.)