Introducing the "Debian's Automated Code Analysis" (DACA) project

[Posted December 16, 2010 by jake]

From:		Raphael Geissert <geissert-AT-debian.org>
To:		debian-devel-announce-AT-lists.debian.org
Subject:		Introducing the "Debian's Automated Code Analysis" (DACA) project
Date:		Thu, 16 Dec 2010 12:00:21 -0600
Message-ID:		<201012161200.31008.geissert@debian.org>

Hi,

It's been a while since I started working on this project and even longer 
since I had the idea. It's therefore a pleasure to finally announce the DACA 
project.

= What is DACA? =

Automated Code Analysis helps detect and fix bugs and other issues in source 
code. The project aims to give users easy access to a wide variety of tools to 
improve quality of software distributed by Debian, while giving the tool's 
developers a test bed, more visibility, and more feedback.
This is achieved by running those tools on the complete Debian archive.

http://qa.debian.org/daca/

= What is there for everyone? =

At the moment there are only partial reports from two tools, but the list of 
tools to be evaluated and possibly included goes over twenty.

Current tools: cppcheck, and checkbashisms (at the source package level.)

= Limitations =

Most of the tools are CPU-bound, limiting considerably the number of tools and 
time it takes to check the whole Debian archive. For example, with the typical 
sid repository update (i.e. not during the freeze and with a working ftp-
master) it is impossible for the server running cppcheck to keep up with all 
the changes.

= How can you help? =

* First of all you can go and squash bugs! 
Please keep in mind what's in the notices at the bottom of the pages. They are 
rather static now, but they may change later.

* Second, report false positives, fix bugs, improve the tools
Every report page (at the footer) should mention the version of the tool used 
to generate it. They are usually the latest.

* Third, join the DACA project
More hands are needed to evaluate other tools, setup an infrastructure for 
running them, and finally generating the web reports.
Discussing tools already available at DACA is also welcome.

There's a project request at Alioth pending its approval, but once accepted 
access to the repository and mailing lists will be found at:

http://alioth.debian.org/projects/daca

(the contact email address at the DACA website will be updated accordingly)

* Fourth, donate hardware

If you have equipment with very powerful CPUs or other hardware you (or your 
company) can donate, please take a look at the following page:

http://www.debian.org/donations#equipment_donations

Note that this may be handled by people unrelated to the DACA project, and as 
such you should mention what you would prefer your hardware to be used for.

* And finally, figure out what I missed, provide feedback, and go back to step 
one.

Thanks!

Cheers,
-- 
Raphael Geissert

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 16, 2010 23:19 UTC (Thu) by MisterIO (guest, #36192) [Link] (20 responses)

cppcheck is 2 releases old in debian(months old) and they're gonna use it to check the packages? It's probably better to start maintaining it better then.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 16, 2010 23:20 UTC (Thu) by MisterIO (guest, #36192) [Link]

And I'm talking about sid|experimental obviously.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 0:24 UTC (Fri) by naoliv (guest, #21066) [Link] (3 responses)

Well... why not help us then? ;-)

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 1:37 UTC (Fri) by MisterIO (guest, #36192) [Link] (2 responses)

There's still version 1.44 probably because of the freeze for v6.0 release(I've seen that answer to many bug reports(whishlist) asking to upgrade a package), so probably lack of help isn't the problem here. Why exactly does the freeze include sid+experimental, I don't know. Or maybe, simply because there's the freeze,the maintainer isn't interested in updating the package, even if in sid|experimental it's permitted, I'm not sure. Valgrind too is an svn snapshot, when the official release is already out. Clang is months old too. The gcc suite is well maintained instead.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 4:40 UTC (Fri) by tao (subscriber, #17563) [Link]

The freeze doesn't include experimental (so packages in experimental can still be freely upgraded), but it does include sid. This is because unstable is the staging ground for packages that enters testing.

Using experimental is not very common though (neither among developers nor among users), since it's not a complete distribution (which unstable is), only a partial repository (if it was a complete distribution it wouldn't be possible to cherry pick from experimental, because of dependencies).

The fastest way to ensure that new versions of cppcheck gets integrated in unstable at a timely manner is to help get Debian 6.0 out of the door, so if you feel like you can help fix any of the open RC-bugs, please do.

Of course, playing the Devil's Advocate it's likely that extensive testing of the packages in the archive with cppcheck might actually increase the number of RC bugs, in case it uncovers a lot of severe bugs. But that's flawed reasoning...

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 20, 2010 19:37 UTC (Mon) by k8to (guest, #15413) [Link]

The freeze of sid leading up to a release is pretty annoying. I think as a project management issue it's pretty important though, if you believe in the value of a stable release. You have to have a stick that makes it unavoidably important to stabilize your continuous development line if you want to release a relatively unchanging version that you keep working for years.

Personally, I don't give a rats ass about the stable releases, and would rather use a continuously updated (but tested) set of packages than an unchanging one. But I have actually met people who use stable on their servers and appreciate what it is.

Project management is full of compromises.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 0:30 UTC (Fri) by tao (subscriber, #17563) [Link] (7 responses)

So you're saying that version 1.44 of cppcheck was a totally useless tool then?

FWIW, cppcheck 1.46 was release 2 days ago.

Also, Debian is in freeze pending the upcoming 6.0 release.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 1:07 UTC (Fri) by xxiao (guest, #9631) [Link] (1 responses)

why use cppcheck instead of splint? my first time to hear about cppcheck, did a quick run, and it reports a few errors in my code, looks like it's pretty helpful.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 18, 2010 17:25 UTC (Sat) by ballombe (subscriber, #9523) [Link]

I found that cppcheck sound/noise ratio was much better than splint on C code that was not prepared to be run through splint. I did not try C++.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 1:27 UTC (Fri) by MisterIO (guest, #36192) [Link] (4 responses)

When did I say that 1.44 was useless?

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 4:31 UTC (Fri) by tao (subscriber, #17563) [Link] (3 responses)

This is the message I replied to:

"cppcheck is 2 releases old in debian(months old) and they're gonna use it to check the packages? It's probably better to start maintaining it better then."

While my interpretation of your statement was hyperbole, you have to admit that your comment was borderline trollish.

Instead of questioning the use of anything less than the latest release of cppcheck (if you look at the release date, I think that you'll find that even cppcheck 1.45 was released after Debian was frozen), why not simply state that "There's now an even newer and better version of cppcheck available from upstream" or similar.

And, as another poster already said, I'm sure any effort to help packaging it for experimental or for that matter to fix the cppcheck related bugs in BTS would be appreciated.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 20, 2010 19:41 UTC (Mon) by k8to (guest, #15413) [Link] (2 responses)

I think the general expectation from some users is that the distribution is a means to conveniently track upstream releases. When this expectation is not fulfilled, it's perceived as a problem. I don't think this expectation is really fundamentally unreasonable, but it does conflict with some other goals. From this conflict and the gap in behavior, gaps between perception and reality arise, and thus criticism.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 22, 2010 10:40 UTC (Wed) by liljencrantz (guest, #28458) [Link] (1 responses)

I must confess to disagree with pretty much every opinion in your entire post. :-(

I don't think users expect distributions to track upstream releases, I think users expect the distributions to put together a set of software that works well together. Sometimes that implies keeping an unstable release back (KDE4.[012]), on rare occasions it means distributing unreleased betas that actually work better than the latest stable release (Emacs).
I think that to the extent users have this expectation, it is fundamentally completely and utterly unreasonable. A distribution needs to consist of packages that work well together. When the latest version of package X switches to a new API, and the critical system package Y uses the API of package X but has not yet been updated to the new API, I think there is a strong expectation that the distribution doesn't jump in and release a new X package and making Y useless.

The above is in complete conflict with the idea of tightly tracking the latest release of upstream, and if a user expects this, informing the users of why his/her expectations are unreasonable is the right course of action, not criticizing the distribution.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 22, 2010 17:05 UTC (Wed) by k8to (guest, #15413) [Link]

Your view of users is too simplistic. Some expect blissful functionality. Some expect currentness. Some expect both. A significant number don't think too much about the conflicts here.

I was just talking about the trend or set who expect currentness, almost implicitly.

So you aren't actually disagreeing with me at all.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 3:46 UTC (Fri) by geissert (guest, #61258) [Link] (6 responses)

I don't want to kill the discussion, but picking a (not-so) random results page[1]:

"This report was generated on Thu, 16 Dec 2010 04:03:51 +0000, based on results by cppcheck 1.46"

[1] http://qa.debian.org/daca/cppcheck/sid/apache2_2.2.16-4.html

Cheers

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 7:42 UTC (Fri) by MisterIO (guest, #36192) [Link]

That's ironic!

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 12:44 UTC (Fri) by mgedmin (subscriber, #34497) [Link] (3 responses)

That's a nice page!

Any chances of having the filenames + line numbers be actual hyperlinks to the source code, for a convenient closer look?

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 18, 2010 20:30 UTC (Sat) by geissert (guest, #61258) [Link] (2 responses)

At the moment Debian doesn't have such service and the web reports are generated directly from the tool's report on a machine other than the one that actually ran the tool, making it impractical.
I'm not even sure if those working on the source.d.o project intended to provide HTML-ised source code (with lxr or the like.)

I do agree that it would be a great feature to have and is even in my long term To-Do list, though.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 20, 2010 23:17 UTC (Mon) by NightMonkey (subscriber, #23051) [Link] (1 responses)

I'm sure your team knows about this, but the included (at least in the source package) htmlreport will generate in-line source code. I use it to make a full report on LibreOffice's git code every 4-6 hours, for their dev's benefit:

http://libreoffice.boldandbusted.com/

DO NOTE: This is a huge report. Your browser may creak under the strain.

Big thanks to cppcheck! :)

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 21, 2010 2:01 UTC (Tue) by geissert (guest, #61258) [Link]

I'm aware of it (and have even used it locally for some projects.) However, like I replied to people who suggested it via email, DACA doesn't consist of only one tool, and all of them would benefit from a service that provides the html-ised source code with anchors and all.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 17:47 UTC (Fri) by jthill (subscriber, #56558) [Link]

Ouch. I sure hope they didn't do signoffs back in 2004 (specifically r109623), those are blatant.

I notice it didn't flag the unchecked malloc returns.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 17, 2010 15:11 UTC (Fri) by robert_s (subscriber, #42402) [Link] (1 responses)

As long as people don't go and use the output to blindly "fix" code.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 18, 2010 12:40 UTC (Sat) by renox (guest, #23785) [Link]

I think that this was mainly the fault of the code: using unitialised variable without meaningful variable names and pragma to avoid valgrind complaining?
A desaster waiting to happen..
Not fair to blame only the maintainers even though Debian should really use a 'fix upstream before' policy also.

checkbashisms

Posted Dec 17, 2010 21:42 UTC (Fri) by alecs1 (guest, #46699) [Link] (10 responses)

Ah, I always thought Debian's efforts on this could have been better concentrated on something else. Now they are at it again.

I didn't find discussions about the effort on Google, so I have no idea how much work went into this, but if I were to decide where to invest say 200 hours of work, I would have made Bash standard and then try to improve on whatever performance problems it might have. But they rather had this as a release goal, had discussion threads on it, opened bugs and who knows how many scripts where modified, for an improvement a majority of users won't feel (I sure didn't notice the difference, whatever that might be; but I did waste time with dpkg-reconfigure to test this). I'm pretty sure Bash is not unfixable, it is a damn console, not the entire kernel.

checkbashisms

Posted Dec 17, 2010 22:10 UTC (Fri) by cmccabe (guest, #60281) [Link] (9 responses)

I think it's good that they're moving away from using bash for things that it doesn't belong in-- like init scripts. /bin/sh should just be a POSIX shell, not bash. If you want bash, ask for it by making your package depend on bash and invoke it explicitly.

> I sure didn't notice the difference, whatever that might be; but I did
> waste time with dpkg-reconfigure to test this.

You probably didn't notice the difference in thousands of other small optimizations people made, either. But added together, they make a difference.

> I'm pretty sure Bash is not unfixable, it is a damn console, not the
> entire kernel.

I'm pretty sure you've never seen the source code to bash.

checkbashisms

Posted Dec 18, 2010 21:07 UTC (Sat) by epa (subscriber, #39769) [Link] (8 responses)

Why 'should' /bin/sh be a plain POSIX shell and not bash?
(Hoping for some answer other than 'because it has always been that way')

checkbashisms

Posted Dec 19, 2010 0:18 UTC (Sun) by foom (subscriber, #14868) [Link] (6 responses)

Because if you meant bash, you can just say "/bin/bash"?

checkbashisms

Posted Dec 19, 2010 0:48 UTC (Sun) by dlang (guest, #313) [Link] (2 responses)

so why not just go through every init script and sed #/bin/sh#/bin/bash# and not have to worry about it any longer?

checkbashisms

Posted Dec 19, 2010 1:08 UTC (Sun) by foom (subscriber, #14868) [Link]

Yup. That sounds like a good plan to me :) (that's what I did with all my shell scripts when Ubuntu first started making a fuss about this.)

checkbashisms

Posted Dec 19, 2010 13:22 UTC (Sun) by pkern (subscriber, #32883) [Link]

That's a valid option of course. However, if it's only a small bit of work you can save a bit of time on every invocation of the script, by using a faster shell than bash as `/bin/sh'. As long as you intentionally expect bash with its feature set for a script, it's fine to just write `/bin/bash' as your interpreter. This work is supposed to check for the accidental expectation that the POSIX `/bin/sh' has the bash feature set.

checkbashisms

Posted Dec 22, 2010 12:31 UTC (Wed) by epa (subscriber, #39769) [Link] (2 responses)

Or, just as well, declare /bin/sh to be bash and have
a separate /bin/minimal_bare_bones_sh for the tiny
number of cases where it is truly needed.

Fifteen years ago bash might have been seen as bloated.
That really isn't an issue any more.

checkbashisms

Posted Dec 25, 2010 5:07 UTC (Sat) by foom (subscriber, #14868) [Link] (1 responses)

POSIX sh is standardized. Regardless of what /bin/sh actually points to, or how bloated bash is or is not, if you write a script that depends on nonstandard extensions in bash, you should put #!/bin/bash at the top.

checkbashisms

Posted Dec 31, 2010 11:52 UTC (Fri) by epa (subscriber, #39769) [Link]

A little ironic that you use POSIX standardization as a reason for a particular shebang line, since the #! syntax is not part of POSIX at all.

I think you are probably right though - to keep everybody happy, if you want bash it's best to say what you mean.

checkbashisms

Posted Dec 19, 2010 2:58 UTC (Sun) by MisterIO (guest, #36192) [Link]

Because if you're invoking the shell as /bin/sh, it should be a POSIX shell. If you want bash, invoke it as /bin/bash. Anyway, I guess the main point of this was that dash has a smaller memory footprint than bash.

Introducing the "Debian's Automated Code Analysis" (DACA) project

Posted Dec 19, 2010 8:26 UTC (Sun) by danmar (guest, #71921) [Link]

I am the maintainer for Cppcheck.

This is interesting news. I hope DACA will be a success.

I assume there are some false positives in those reports now. We (the Cppcheck developers) will really try to fix all false positives that are reported to us as quickly as possible.

I feel that it will be quite a milestone to reach for Cppcheck when the whole Debian distribution can be checked and there are no false positives!

And I hope that some day Debian will be Cppcheck-clean. :-)

Security analysis tools?

Posted Dec 23, 2010 15:49 UTC (Thu) by dwheeler (guest, #1216) [Link]

I'd like to see tools that work to find security vulnerabilities (aka "vulnerability scanners"). Certainly some generic "bugs" also create security vulnerabilities, but some patterns are especially likely to create vulnerabilities, so searching for them would make sense. Good luck!