Weekly Edition
Return to the Development page
| |||||||||||||
The SCons build tool reaches the 1.0 milestoneAfter a two month release candidate stabilization period, version 1.0 of the SCons build tool has been released. The SCons description states:
SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.
SCons is being distributed under the MIT license. Steven Knight is the main developer, the rest of the SCons Development team consists of Chad Austin, Charles Crain, Steve Leblanc, Greg Noel, Gary Oberbrunner, Anthony Roach, Greg Spencer and Christoph Wiedemann. The SCons project history is described:
SCons began life as the ScCons build tool design which won the Software Carpentry SC Build competition in August 2000. That design was in turn based on the Cons software construction utility. This project has been renamed SCons to reflect that it is no longer directly connected with Software Carpentry (well, that, and to make it slightly easier to type...).
An SCons document entitled TheBigPicture and the Wikipedia entry explain some of the unique SCons features. These include:
To get an idea where SCons stands in the variety of build tools that are available, the documentation includes a comparison between SCons and other tools. The project's documentation is quite voluminous. The nearly 10,000 line man page is somewhat daunting, it even dwarfs the 8000 line long mplayer man page. Fortunately, the document is available in an indexed html version for easier reading. ![[SCons]](/images/ns/SConslogo.png)
A test installation of SCons 1.0 was tried on an Ubuntu i386 Hardy Heron machine. The code was downloaded, uncompressed and untared, then the following command was executed as root from the source directory: python setup.py install. A test of SCons was performed on a relatively simple C program that prints out the data from a stepped sine wave (sine2hex.c). After plowing through some of the man page and doing a bit of digging through the SCons User Guide, your author succeeded in compiling and linking the program. An SConstruct file was created to describe the project, it consisted of the following line: Program('sine2hex.c', LIBS = 'm')
Typing scons caused SCons to compile and link the program.
That is, of course, only the tip of the iceberg, but it shows that
the software is not too difficult to get started with.
SCons is being used by a variety of closed and open-source code software projects, the References section lists these and includes user comments about the advantages of switching from other build tools. If you need a next-generation tool for maintaining a large cross-platform project, SCons should be able to do the job. (Log in to post comments)
Hope it's better than when we were using it... Posted Aug 28, 2008 1:42 UTC (Thu) by tshow (subscriber, #6411) [Link]
We tried scons on a project a couple of years back. It's good for very simple projects, but unless things have changed it doesn't scale very well.
The problems we had:
1) SLOW! Orders of magnitude slower than makedepend/make, which starts to show up on large projects.
2) As with most build tools that try to be clever, you'd better want to organize your project the way the tool designer intended you to. Doing something as simple as changing the way scons printed progress reports meant delving into the guts.
3) Again, as with most "clever" build tools, if you want to extend the functionality (say, in our case, by adding architecture-specific directories to a project tree, and having arch-specific compilers) you'd better be prepared to rewrite large parts of the tool.
In the end, it was easier and faster to write our build system with make, some hand-rolled tools and a Ruby makefile generator. It did what we needed better than scons could, ran orders of magnitude faster, and took much less effort.
Which, really, says more about scons than anything.
Perhaps it's gotten better. I hope so. Our experience with it was terrible, however. The whole scons/bugzilla "yeah, we have a configuration editor, it's called emacs and the source code" thing is just not acceptable.
Hope it's better than when we were using it... Posted Aug 28, 2008 11:36 UTC (Thu) by vputz (subscriber, #5639) [Link]
I'd say it's more a mismatch for your needs, if anything. I've never had a problem with SCons and I've used it across multiple projects, though admittedly my own projects were pretty small.
1) Slow? To do what exactly? I'm assuming you're meaning complete tree analysis of targets. Perhaps that is a problem on very very large projects; I've never had it be a problem, though. Admittedly it's a widgy bit slow on my current project but that's primarily because my home directory (and thus all my nonstandard Python libs etc) are on a networked drive, and that makes startup painful, but I've always found SCons itself to be pretty spry.
2) I think you have a different definition of "extend the functionality" than most people consider for a make tool. Changing the way progress reports are printed isn't the usual way I find myself having to extend a build tool. I find myself having to create new types of targets, which is TRIVIAL in SCons compared to other tools, thanks to the fact that it's in Python. I'm currently doing a lot of scientific development, and being able to have different types of graphs, etc which depend on output files from programs, which depend on... means that I'm constantly making new builders, new target types, etc. I could NEVER imagine doing this with makedepend/make.
On a previous project I had SCons take Eiffel source code files, use them to generate another Eiffel build file, drive Eiffel to generate C, run an external Python process to do analysis on and mangle the C code, generate C dependencies, run the PalmOS compiler to generate programs, then use prc-tools to create a PalmOS pdb--and SCons managed dependencies correctly throughout. I couldn't IMAGINE getting the same effect through any other build tool. SCons is incredibly extensible.
3) Not sure I follow the problem here either. Making a somewhat more involved SConscript file for a subdirectory hardly qualifies as "rewriting large parts of the tool". I'm not saying you didn't have troubles, just that saying you had to "rewrite SCons" is probably overstating your case.
Short form: I've never had trouble with SCons and highly recommend it over make. Your mileage, of course, may vary.
Slow indeed. Posted Aug 28, 2008 14:21 UTC (Thu) by dank (guest, #1865) [Link]
On one project I built recently, scons
took 40 seconds to even start building any C++ files, and kept the cpu 100% busy doing pythony stuff.
To get any work done, I run it in verbose mode,
This was with a project on local disk on a very fast 64 bit cpu on Hardy.
Other people building the same project swore up and down it only took ten seconds for them, so it seems YMMV.
Slow indeed. Posted Aug 28, 2008 16:19 UTC (Thu) by vputz (subscriber, #5639) [Link]
> On one project I built recently, scons
> took 40 seconds to even start building any C++ files, and > kept the cpu 100% busy doing pythony stuff.
Hmm. Pretty large project with many files? With 100% cpu usage, that could have been just doing the MD5 checksums; try the "Decider('MD5-timestamp')" mentioned below and at http://www.scons.org/doc/1.0.0/HTML/scons-user/c792.html
Slow indeed. Posted Aug 28, 2008 16:23 UTC (Thu) by vputz (subscriber, #5639) [Link]
...or the other suggestions at http://www.scons.org/wiki/GoFastButton
I'm guessing that many folks who have "SCONS IS SLOW" problems can fix them using some of the above... SCons defaults to "correct build over any other priority" mode, but the above tweaks are really pretty safe and can speed things up quite a bit (using timestamp-based deciders probably works fine for most people--it's what make does--but the default behaviour is MD5 because it's more careful).
Hope it's better than when we were using it... Posted Aug 28, 2008 14:37 UTC (Thu) by Nelson (subscriber, #21712) [Link] In the end, it was easier and faster to write our build system with make, some hand-rolled tools and a Ruby makefile generator. It did what we needed better than scons could, ran orders of magnitude faster, and took much less effort.You can always do "better" if you're doing it custom. The custom tool that dynamically builds makefiles approach always seems to be one of the solutions to the problem, it just strikes me as an odd design pattern. It seems like the goals of the tool builders vs. users must be different. I worked on a giant project at a huge nameless company once that had a big Korn shell script to configure the environment (maybe 50 variables, you needed 5 to 10 but there were a lot that could impact things) and then a perl script to dynamically generate the make files based upon the environment and some configuration text files and then run make, it worked but it was kind of scary and the problem it seemed to solve was different developers had different ideas about how to slice the problem when adding new features or creating a new project. In retrospect, the real problem was that we would build libraries that did like 10 things when some of the consumers really were only interested in like 2 things and so rather than restructuring the code in to a set of 10 (or really like 12 libraries with the common parts) we kept it as one and conditionally compiled things in and out based upon which product was being built. We had a broken build award that you'd get when you screwed up the nightly build and I remember the dread when I had to add stuff to the build. Essentially, the guy who "architected" it had created himself a permanent job and we always went to him, even if it was a fairly mundane task. It was originally supposed to make things better but I have a really hard time seeing how that was accomplished, people were kind of scared of the build and didn't fully understand it. Scons and a-a-p and rake are all kind of cool, make does a fairly simple task and it's usually not quite enough to do what you need in a professional environment. At my current employer, we label and have a unique build number for each full build of the product. Try doing that in make, you end up doing a lot of wonky shell script escaping which isn't the most maintainable and it's specific to a platform. (every team usually has some guy who needs to run a different shell just to be different.) Just some of the little things like that make these tools with a standard programming language seem a little nicer. The steps from building source to packaging up a product, we script those, you only build and deploy code to a test environment like 95% of the builds but the production build packages up the binaries and builds a CD and deposits it somewhere and updates the build database, sends an email to some people; make simply cannot handle that stuff and you're back to platform specific custom scripts or maybe having some sort of possible re-use by including some of those kinds of functions in a higher level tool like scons. Another thing, we build some windows code, some mac code, and some linux code. Is it too much to expect a common build on each platform? Ant can get you close but it suffers the same problems, you sometimes want some logic to be performed at build time, ant can't do that, it doesn't have conditionals. Now scons isn't a silver bullet, I was under the impression that its goal was to remove the need for autoconf and automake and I don't really see it doing that... not yet. Actualy, if it was used more I'd predict autoconf and then autoscons. I'd feel better building a fairly large build in it over make and custom scripts, at least to give it a shot. I've really only used it for fairly small projects of just a couple hundred files in a single module. I've just never seen a build I thought was "good" with all the custom stuff, it was "Better" than what it replaced and it worked but it just doesn't feel "good" to write programs to generate programs that generate a program.
Hope it's better than when we were using it... Posted Aug 28, 2008 23:09 UTC (Thu) by broonie (subscriber, #7078) [Link]
As well as the suggestions that others have posted it's worth pointing out that a lot of work went into optimising the performance in recent years so it should be doing better than it was when you tried it even with the default setup.
IME the extensions thing is a lot more straightforward if you layer on top of SCons rather than trying to modify the core - I added things like architecture specifics that way when I was using it for a major project fairly smoothly. I ended up using it mostly as an engine for managing the build dependency graphs. using the standard builders but programatically working out what I wanted them to build
The SCons build tool reaches the 1.0 milestone Posted Aug 28, 2008 10:34 UTC (Thu) by rvfh (subscriber, #31018) [Link]
Can Scons track mtime rather than checksum? I do mean track. not just check for 'greater than the dependent objects', but for 'different from last time'. That would be awesome.
The SCons build tool reaches the 1.0 milestone Posted Aug 28, 2008 11:41 UTC (Thu) by vputz (subscriber, #5639) [Link]
...not sure what you're requiring here. Can you elaborate? I'm sure there's some way to do what you mean, although I've found the checksum-based approach to work pretty darn well (example: I have C code which creates a simulation program which generates an output file which generates an .eps graph. If I change the C code, the simulation gets rebuilt and run, but if the new output file is exactly the same as the old output file, the graph is NOT generated because SCons knows the actual input has not changed even though the mtime of the file has changed).
Just curious--why do you see mtime as better than stored checksum?
The SCons build tool reaches the 1.0 milestone Posted Aug 28, 2008 13:27 UTC (Thu) by rvfh (subscriber, #31018) [Link]
Checksum works fine, but if you have thousands of files over a slow network connection, it takes forever.
Checking the mtime takes much less time as you do not need to read the file contents, just its meta-data.
But I do think it a good idea to detect _any_ change in mtime (unlike make), which means storing the mtime of the file used (just like the checksum is currently stored by SCons) and use it to decide whether to re-build the file dependents or not.
To simplify: s/checksum/mtime/ and s/computeChecksum/stat/
The SCons build tool reaches the 1.0 milestone Posted Aug 28, 2008 14:13 UTC (Thu) by vputz (subscriber, #5639) [Link]
Well, I poked around and found this page (http://www.scons.org/doc/1.0.0/HTML/scons-user/c792.html) which answers some of your questions.
Short form: you can set a "decider" to choose whether the dependency tracker uses MD5, timestamps, or both.
Decider("MD5") will look at the MD5 of each file (default) and rebuild if the stored MD5 does not match.
Decider("timestamp-newer") will rebuild if files timestamp is newer than the stored one (like make)
Decider("timestamp-match") will rebuild if files timestamp is different from the stored one (like what you describe)
Decider("MD5-timestamp") will check to see if the timestamp is newer; if it is, THEN it will recalc the MD5 and rebuild if stored MD5 does not match. This is evidently quite an improvement, but can have issues if files are materially changed within 1 second of its previous timestamp, etc.
And, being SCons, you can mix and match the above for different files, and you can write your own decider functions. So it can almost certainly do what you need.
SCons: timestamps vs checksums Posted Aug 29, 2008 18:04 UTC (Fri) by giraffedata (subscriber, #1954) [Link] Decider("timestamp-newer") will rebuild if files timestamp is newer than the stored one (like make) That's not what make does. Make doesn't store the timestamp of the input file. Instead, it considers the output file invalid if the input file's timestamp is newer than the output file's timestamp. I don't know whether SCons does the comparison you describe or does what make does, but I can't imagine either one being useful, given that Decider("timestamp-match") exists.
The SCons build tool reaches the 1.0 milestone Posted Aug 28, 2008 23:50 UTC (Thu) by alextingle (guest, #20593) [Link]
SCons does NOT do what autoconf does. Autoconf enables me to package my software so that it can be built on a very wide set of Unix boxes. SCons only works if...
a) you've managed to get Python to build on your Unix box. (Python is a huge package which can be quite tricky to compile and install.)
b) somebody's bothered to teach SCons about your platform.
When I tried to build an app that uses SCons, it was a real pain because my "obscure" platform (Debian stable) wasn't supported! God help anyone who wants to build on a 10-years old AIX machine or whatever.
The SCons build tool reaches the 1.0 milestone Posted Aug 29, 2008 0:16 UTC (Fri) by nix (subscriber, #2304) [Link]
Those who have not experienced imake are doomed to reinvent it?
autoconf Posted Aug 29, 2008 13:47 UTC (Fri) by shane (subscriber, #3335) [Link] Autoconf enables me to package my software so that it can be built on a very wide set of Unix boxes.In theory at least. In practice one must tweak the Autoconf build for almost every new environment; the closer you are to the system the more this is true. (I work mostly with networking code, and even seemingly simple operations are implemented vastly differently on each Unix-like OS.) One exception is that all of the *BSD environments seem to have very similar code-bases.
The SCons build tool reaches the 1.0 milestone Posted Sep 2, 2008 13:22 UTC (Tue) by daniel_svensson (guest, #28746) [Link]
I've used SCons in a pretty complex setup at work and it kept impressing me all the time. The persons complaining about the performance probably have a broken setup. In the beginning my SCons-fu was weak and it took about 34 seconds before anything started to build. With some nice tweaks here and there this was reduced to 2-3 seconds, and we're talking about a HUGE build here with lot of files, and lots of tasks to be performed. The GoFastButton page on the wiki helps a bit, but the most gain I got was to structure my build process (not source code) better, taking better desicions behind the scenes. I build a highly configurable modular operating system for in about 50 diffrent configurations, for 7 different embedded devices, plus Windows/Linux and OS X and SCons lets me abstract this very nice so that the developers don't have to think about what platform, compiler or anything.. they just write their very minimal and almost plain-text configuration files and it's all tied together behind the scenes. SCons is a very awesome framework to program a build process upon. Extremly flexible! And as a fast-to-use build system, it rocks!
|
|||||||||||||