A rift in the NTP world

Posted Feb 9, 2017 17:19 UTC (Thu) by fallenpegasus (guest, #58173)
Parent article: A rift in the NTP world

Hi!

I'm the PM for the NTPsec Project.

My apologies to LWN and to Bruce Byfield, I must have somehow missed your attempts to reach me for an interview for this article, but for the record, my email address working on this project is <mark.atwood@ntpsec.org>, and I'm happy to answer anyone's questions about the history, state, and goals of the NTPsec Project.

We were not happy about having to fork from NTP Classic, and did so with regrets.

The main point of contention that caused the fork was BitKeeper vs Git. Harlan insisted on staying on BitKeeper. At that time, BitKeeper was still closed source, proprietary, and was a huge barrier to recruiting contributions, large and small. Even now still, the official Git repos for NTP Classic are out of date with the official BK repos, and are lacking tags. And the official public BK repos are out of date from Harlan's internal working repos.

We work from our public repos at https://gitlab.com/ntpsec and we welcome gitlab pullrequests and git-patch emails from everyone. Our contribution workflow is at https://www.ntpsec.org/contributor.html and our "hacking guide" is at https://gitlab.com/NTPsec/ntpsec/blob/master/devel/hackin...

The only exception to our working from our public trees is when we fix embargoed CVEs that get reported to us, under the principle of Responsible Disclosure. Our security issue and reporting policy is at https://www.ntpsec.org/security.html

As other commenters here at LWN have pointed out, we are very public on our website at https://ntpsec.org/ and on our blog at https://blog.ntpsec.org/

We are very clear about how we have decided to remove code and features. We removed autoconf and replaced it with WAF. We removed clock drivers for time source hardware that are no longer available even on the used market, and for time source hardware that is worse than a cheap GPS PPS receiver. We removed code for uncompleted features in NTP Classic that were never surfaced to the users. We removed compatibility shims and ifdefs for operating systems that are no longer running in the wild or no longer supported by their vendors. It turns out that all the world is now POSIX/C99, and when we encounter cases where that is not the case, it's easier to start with a clean POSIX/C99 state, and then carefully add what little is necessary.

We also gifted to the larger time and NTP community a working step-by-step howto on how to build a Stratum 1 NTP time server on a Raspberry Pi, at https://www.ntpsec.org/white-papers/stratum-1-microserver...

Again, I'm happy to answer any questions that LWN may have about the NTPsec project, and I welcome everyone to clone and build from our repos, and give NTPsec a try.

Thank you!

A rift in the NTP world

Posted Feb 13, 2017 18:37 UTC (Mon) by clemensg (guest, #94377) [Link] (10 responses)

I was just considering a switch to NTPsec but then I saw you moved away from the well established autotools to one of the new shiny build systems available nowadays and no one can say for sure if it is still maintained in a few years. Autoconf/-make will be, that's for sure. It's the de-facto standard build system, not only in the GNU world. It is especially nice if you are doing cross-compilation.

Now you introduced more barriers for users and for developers. To build, you have to install Python. To understand the build process they need to know Python + waf: https://github.com/ntpsec/ntpsec/blob/master/wscript
autotools is complicated but waf is not a big improvement, imho. Only a different kind of complexity.

Many new build systems come and go. Remember Scons? Google wrote ninja/gyp/gn to "replace" it. waf is also a Scons fork/rewrite.

Why not just keep using Autotools? Developers need to know it anyway if they work on other free software projects and it does its job.

A rift in the NTP world

Posted Feb 13, 2017 21:10 UTC (Mon) by fallenpegasus (guest, #58173) [Link] (9 responses)

I'm sorry you feel that way.

The decision to move off autotools was not done to move to the "new shiny", but as a deliberate decision to help drive the cleanup. The existing autotools was ancient, was itself out to date with the latest autotools generators and so was very difficult to update and revise, it will full of errors, it ran very slowly, and it was tightly entangled with all the out of date portability shims and ifdefs.

We made the technical decision that having a Python dependency to build would not be a difficult barrier. Every modern distro of every still shipping and still supported UNIX has a sufficiently up to date Python package, and most UNIX distros in fact now either actually require a Python install, or have Python in their core package set.

Regular users or even builders of NTPsec are not required to know or code in Python. Just do a "./waf build" and it should work. If it doesnt work on your system, please email us at contact@ntpsec.org and let us fix it for you.

If the WAF project ever dies, we will port to another build system. Maybe even to a modernerized autotools, if that is appropriate at the time.

A rift in the NTP world

Posted Feb 14, 2017 1:44 UTC (Tue) by pizza (subscriber, #46) [Link] (1 responses)

> The decision to move off autotools was not done to move to the "new shiny", but as a deliberate decision to help drive the cleanup.

Yeah, the last time I experienced a "help drive cleanup" migration away from autotools, it was GPSd to scons. It turned out to be easier to just backport fixes rather than fix the build system to work in a cross-compiled environment.

To paraphrase, Autotools is the worst build system, except for all the others.

A rift in the NTP world

Posted Feb 14, 2017 3:22 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

WAF does support cross-compilation for real-life large projects: https://wiki.samba.org/index.php/Waf#cross-compiling

I've had my share of issues with build toolchains and debugging even esoteric issues with Unicode capitalization on MacOS X in Scons was easier than finding out why autotools is not rebuilding stuff properly.

A rift in the NTP world

Posted Feb 14, 2017 11:52 UTC (Tue) by chrisV (guest, #43417) [Link]

Thankfully the only time I have come across WAF is when trying to build pycairo-1.10.0 (the latest version available) with python-3. The WAF build system for pycairo is broken for versions of python after python-3.2. Anything as brittle as this which breaks when a new minor version of python comes out is surely unusable?

A rift in the NTP world

Posted Feb 15, 2017 15:47 UTC (Wed) by rkeene (guest, #88031) [Link] (5 responses)

The information about building NTPSec provided by the parent commenter is wrong. Running "./waf build" returns the following error:

--- building host ---
The project was not configured: run "waf configure" first!

Further, running "./waf configure" fails predictably when cross-compiling (as generally you need to tell the system what platform is being targeted, though it could figure it out):
+ ./waf configure
Setting top to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-pcXttoUpzzwX
Setting out to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-pcXttoUpzzwX/build
--- Configuring host ---
Checking for 'gcc' (C compiler) : /home/rkeene/devel/aurae/common/compiler/online/x86_64-coreadaptive-linux/bin/gcc
Checking for program 'bison' : /home/rkeene/devel/aurae/common/detected-tools/bison
Checking compiler : no
The configuration failed
(complete log in /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-pcXttoUpzzwX/build/config.log)

The "build.log" referenced is full of escape sequences that hinder readabilty (but I'm sure if I were using "cat" to view it would make it very colorful on my terminal) and a huge, hundred or so line Python stack trace that, from what I can gather from the stack trace, means it tried to run the code it just compiled... which of course won't work, since the code being generated isn't for the platform I'm compiling it on.

Helpfully, an "INSTALL" file is provided which tells us how to cross-compile NTPSec:

== Cross-compiling ==

Set up a cross-compile environment for the target architecture. At minimum
it will need its own binaries for the OpenSSL library.

Configure NTPSec with:

waf configure --enable-cross=/path/to/your/cross/cc

But when we actually try to use that:
+ ./waf configure --enable-cross=gcc
waf [commands] [options]
<...hundreds of lines of usage omitted...>
waf: error: no such option: --enable-cross

The usage information, however, indicates a different option of:
--cross-compiler=CROSS_COMPILER

So we try with that...
+ ./waf configure --cross-compiler=gcc
Setting top to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-oyqK4b6LB8gE
Setting out to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-oyqK4b6LB8gE/build
--- Configuring host ---
Checking for 'gcc' (C compiler) : /home/rkeene/devel/aurae/common/compiler/online/x86_64-coreadaptive-linux/bin/gcc
Checking for program 'bison' : /home/rkeene/devel/aurae/common/detected-tools/bison
Checking compiler : no
The configuration failed
(complete log in /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-oyqK4b6LB8gE/build/config.log)

And the build.log looks like the same stack trace as before.

Let's assume "waf" is really dumb and tries to look at the filename of my compiler to determine the platform and give it a different name -- no change.

Let's assume it's ultra-extra dumb and is running "$CC" expecting it to be a native-compiler instead of using HOST_CC or CC_FOR_BUILD -- that seems to have gotten us further !

+ CC="${CC_FOR_BUILD}" ./waf configure --cross-compiler=gcc
Setting top to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-JoHiSwi4ewtB
Setting out to : /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-JoHiSwi4ewtB/build
--- Configuring host ---
Checking for 'gcc' (C compiler) : /home/rkeene/devel/aurae/common/detected-tools/gcc
Checking for program 'bison' : /home/rkeene/devel/aurae/common/detected-tools/bison
Checking compiler : yes
Compiler found : GCC
...
Asking python-config for pyembed '--cflags --libs --ldflags' flags : yes
Testing pyembed configuration : yes
Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
Testing pyext configuration : Could not build python extensions
The configuration failed
(complete log in /home/rkeene/devel/aurae/node/root/packages/ntpsec/workdir-JoHiSwi4ewtB/build/config.log)

At this point, things are failing because it's asking our build system's "python-config" how to compile a Python extension, which differs from the host system Python... but there doesn't seem to be a way to tell it to use the correct "python-config", so we play some PATH tricks and get going further...

At this point, after reading the wrong comment, wrong documentation, and finally following the usage information and making a few guesses, then some workarounds ... the configuration is complete. Time to build it !

Of course there's no Makefile but the INSTALL documentation says we should NOW do "./waf build". It seems to work -- it's a bummer that it's not using a Makefile since it would otherwise integrate into the parallel build system.

Now we have to figure out how to install it -- with no conventional "make install DESTDIR=..." and the INSTALL file lacking (hey, atleast it's not just wrong this time) information on how to use DESTDIR. There seems to be a --destdir option to "waf", we try it out and it works on the first try... but at this point we notice we forgot to specify "--prefix" and "--sbindir" options to the configure step...

But there's no option for "--sbindir" and the "--bindir" option doesn't stop it from installing "ntpd" in /sbin, so we have to move it to the correct directory after the installation completes... and finally, we have a working package.

A rift in the NTP world

Posted Feb 20, 2017 19:55 UTC (Mon) by fallenpegasus (guest, #58173) [Link] (2 responses)

Thank you for discovering those issues with our INSTALL file instructions for cross compiling. We're now working on fixing them.

A rift in the NTP world

Posted Feb 20, 2017 23:08 UTC (Mon) by nix (subscriber, #2304) [Link] (1 responses)

Er, if you think the problem there was just with the *instructions*, you're missing something. Not supporting DESTDIR in this day and age will make packagers want to strangle you. Not supporting parallel make is even worse.

Was this *really* an improvement? It's not looking anything like one from where I'm sitting. Any autobuilder out there understands configure/make/make install based systems supporting DESTDIR, like upstream ntp, and will at most need telling "use configure, here are the configure script options". Handling your special snowflake build system will be a lot more work, a lot more cursing, and a lot more packagers asking "why does this project even exist?".

(I hate scons too. Building Samba 4 wasn't terribly pleasant, either, particularly not for platforms with no Python installation but with a working compiler. I actually had to have the autobuilder *compile Python first*, install it in a virtualenv, then build Samba with that. Samba was worth it -- just. ntpsec... wouldn't be.)

A rift in the NTP world

Posted Feb 20, 2017 23:19 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

WAF supports DESTDIR out of box, but it's a command-line option and not a variable. It also is parallel, but since it doesn't integrate with make's job server it can't be parallel across multiple builds.

A rift in the NTP world

Posted May 21, 2020 14:08 UTC (Thu) by hridhya09 (guest, #139051) [Link]

Can you share the steps for ntpsec implementation working package

A rift in the NTP world

Posted May 27, 2020 9:26 UTC (Wed) by hridhya09 (guest, #139051) [Link]

I need a recipe for implementing ntpsec in an arm system

A rift in the NTP world

Posted Feb 15, 2017 15:22 UTC (Wed) by HedgeMage (guest, #114072) [Link] (2 responses)

Jumping in, while this article is not paywalled (it has been at other times I've tried to visit).

I'm Susan Sons, the "ISO Emeritus" who presented on NTPSec at O'Reilly Security Conference. While I've stepped down from my official role with NTPSec, I do what I can to raise awareness of NTPSec, both for the good of its community, and as a case study in what much of our infrastructure software is going through. Mark has already done a more than adequate job of addressing points related to the NTPSec/NTP classic split, so I shall limit myself to some personal notes:

I was not, to my knowledge, contacted by Bruce Byfield or anyone else researching this article. I find this surprising as I am incredibly easy to find. Feel free to drop "Susan Sons NTP" or "Susan Sons ICEI" or any similarly logical set into a search engine and see what happens. I've chosen to make my email addresses ( susan@icei.org for open source organization things, hedgemage@binaryredneck.net for personal, and sesons@iu.edu at work) quite public, and finding at least one of my phone numbers as well as my office address and home address should be trivial as well. While I do travel extensively for work, I'm good at following up, and there are people in my office who will bridge the gap if needed.

Being a hacker, I find this is a great defense against getting doxxed: it's already out there, and I don't care.

The full video of my interview with Mac Slocum is here for anyone who would like to see my remarks in context and un-edited:
https://www.oreilly.com/ideas/the-internet-is-going-to-fa...

I speak at length about the education I received at the feet of previous generations of software engineers, and how I built my career on their mentorship. I also talk about how one day I looked around, and there were not enough people like me in my generation waiting to take the hand-off. Succession planning matters, and it has not been happening. I created New Guard to attract and mentor new infrastructure software maintainers, and specifically to help them find opportunities to work under the Old Masters and learn as I did. We, the community of hackers, need to plan ahead, or our software infrastructure will be taken over by centralized powers who are happy to Balkanize it and curb the many freedoms many now take for granted.

My slides from the security conference presentation are here: http://slides.com/hedgemage/savingtime
The video of the presentation is available on O'Reilly Safari

I can't promise to follow up in this comment thread, as it has not been accessible at all times I've tried to visit. However, anyone who has questions for me is welcome to email me.

A rift in the NTP world

Posted Feb 15, 2017 17:35 UTC (Wed) by jubal (guest, #67202) [Link]

LWN has this useful feature called “subscriber link”, where a subscriber may provide a non-subscriber with a direct link to the paywalled article; and the article becomes visible after two weeks anyway.

(If neither of the options appeals to you, the actual subscription is dirt cheap and supporting high quality and independent tech journalism is very important.)

A rift in the NTP world

Posted Feb 15, 2017 17:40 UTC (Wed) by Nelson (subscriber, #21712) [Link]

So there was a single machine in the world that could build ntp and the root password was lost? (You said that around 2:20 in that video.)

Can you explain any of that in more detail or is it simply hyperbole or is it worse? I mean, I'm pretty sure I've built in myself before and I was unaware that I was dependent upon that machine.

I get the message, I understand that there are these old project and they need transition plans and fresh air and such and that it's important. I'm even open to the idea that long-term supporting an opensource project might not be emotionally healthy, maybe it's better to hand things off for many reasons. I also get that things change and most problems have a different looking solution after the fact. I'd suggest that respect should be a key component of hacking. We should assume hacker solved problems the best they could with the information and knowledge and tools that they had at the time when we re-examine their solutions. It feels really disrespectful to hear stories that might not be entirely true being told as part of the process. In your presentation you also state the the tooling choices and build process were made to "maintain control," is that something you could also elaborate on? NTP was fundamentally a forkable opensource project, you can take it and patch it however you want, fork it, do whatever, what was it that constituted a control issue? Versus maybe a comfort issue. I only ask because, again, it sounds a little disrespectful, if he was a real control freak, it wouldn't be opensource at all and he'd have never asked for help, right?

I can email if I don't hear back here.