Linux distributions and Python 2

Linux distributions and Python 2

For high quality codebases with decent test coverage the migration is straightforward

Good thing that most code out there is a high-quality code base with decent test coverage then, right?

I just ported a 750 line script from Python 2 to Python 3; it's only fairly recently that I've found there are enough reasons to do so (in this case, availability of Python 3 and not Python 2 in the base install of our distro, plus the typing module and mypy to add static types), and enough barriers have been removed.

Here are the steps I went through to make sure the port was working:

1. Run 2to3 over the script
2. Run autopep8 to fix a few issues introduced by 2to3
3. Run pylint and flake8, fix up issues found
4. Add type annotations, check types with mypy, fix up type issues that had crept in
5. Write unit tests (this had originally been a quick script, so not covered by unit tests, but had grown enough that they'd be useful for this port), fix issues discovered by unit tests that still remained even after mypy and pylint fixups
6. Realize that I'd ported to Python 3.6, and target system was running 3.5, so set up tox and fixed up more things that differed between 3.5 and 3.6 (tests that worked in 3.6 due to deterministic dict ordering started failing non-deterministically in 3.6; thanks to tox for printing the random number seed on each test run for making that easier to reproduce and fix)
7. Actually try building my package and tests on the target system, have to fix more test issues due to an older version of pytest on the target system.
8. Finally actually run the result on the target system, and even after all of those steps above (pylint, flake8, mypy, unit tests with pretty good coverage) still have issues to fix

And this is for a fairly simple little script, dealing with some files which just contain ASCII text and ASCII filenames.

Our main codebase uses Twisted, using PB and Jelly to pass objects over the network. There were no automated tests when I started at the company, and while we've been trying to add tests for new features or code that we're touching, we still have pretty poor coverage. Additionally, it's is a networked application that also has to interact with networked filesystems on Windows, macOS, and GNU/Linux, which makes the encoding issues even more fun to deal with. Oh, and most of what is jellied and shared between systems consists of dicts, with different value types for different keys.

At some point, we are going to have to port this to Python 3, but I imagine it's going to take a long time, cause a lot of grief for the developers in the process, and still probably cause regressions for customers even if we're as thorough as we can be by adding unit tests, integration tests, and type annotations.