GNOME Platform Stormclouds [LWN.net]

GNOME Platform Stormclouds

Posted Mar 26, 2004 6:56 UTC (Fri) by bronson (subscriber, #4806) [Link]

"A solution to this is to either use proper unit tests for each
class, as prescribed by extreme programming, or to just make
a stub for the communication..."

or, just use a statically typed language. Languages are tools: use the right one for the job. mly, don't you think you're laying on the Python advocacy awfully thick?

GNOME Platform Stormclouds

Posted Mar 26, 2004 13:36 UTC (Fri) by mly (guest, #2171) [Link]

> or, just use a statically typed language. Languages are tools:
> use the right one for the job. mly, don't you think you're
> laying on the Python advocacy awfully thick?

I agree that we seem to have left the subject of GNOME rather far behind. :)

I'll just summarize and then I'll try to stay away from this thred. I hope you will see my point.

* Static typing is not a full substitue for tests. The compiler will only find particular types of errors. Never logical errors.

* Whether we use a language with static or dynamic typing, we *should* write automated unit tests when we develop systems of significant size. Automated regression tests are important for the long term maintenance of any software project.

* Proper unit tests will find all errors that the compiler would. If your unit tests fail to find typing errors in your code, you either have dead code that you should remove, or incomplete tests. This is not because you write extra tests to find spelling errors, but because spelling errors you make will have an effect on the actual function of the code, which you should test.

* With an appropriate language (such as Python) and good working habits, the code/test cycle is typically faster than the code/compile cycle in e.g. Java or C.

While this might sound good in theory, I won't deny that there are problems involved. It might be difficult to maintain a large set of tests if requirements change, and some things, such as GUIs (which are certainly relevant in the case of GNOME :) aren't always easy to unit test in an automated way with current free tools.

I think the solution to this is to make the tools and methods for testing even better, not to go back to old fashioned languages where you have to write several times more code than you need with a modern language.

GNOME Platform Stormclouds

Posted Mar 28, 2004 18:18 UTC (Sun) by bronson (subscriber, #4806) [Link]

Writing a comprehensive set of unit tests is realistic only for easily-definable portions of your program. Think of the amount of code required to fully test, say, a web browser. It would take years to write and you could never hope to make it complete. The problem space is just too complex. Unit tests are excellent, but they can't be applied universally.

"...not to go back to old fashioned languages where you have to write several times more code than you need with a modern language."

And yet you advocate writing comprehensive volumes of unit tests? Even if your surprising claim of "several times more code" were true, given the length of each of your replies to this story, I'm surprised that you should be so afraid of a few more keystrokes. :-)

mly, have you ever released a project that demonstrates your approach to unit testing? I'd like very much to see an example of what you advocate. For comparison, I wrote unit tests into Trestlemail 4 years ago. It took a very long time to write and debug those 150 tests, yet they were hopelessly insufficient. And Trestlemail has a simple problem space.

GNOME Platform Stormclouds

Posted Mar 28, 2004 22:57 UTC (Sun) by mly (guest, #2171) [Link]

Writing a comprehensive set of unit tests is realistic only for easily-definable portions of your program.

I've used the term Unit Test the way it's been used in Extreme Programming. These tests are not complete functional tests for the program in question. They test small units (typically classes) in isolation. Their aim is to assert that a piece of code does what the programmer inteded. They are rarely functionally complete, but often exercise every line of the unit under test, and then they will find everything that the compiler would, and many other things. See http://www.c2.com/cgi/wiki?ProgrammerTest

Functional testing on a system level is an interesting subject, but it has nothing to do with static typing or the lack thereof.

Even if your surprising claim of "several times more code" were true...

Is that controversial? Just a few examples:

Glyph Lefkowitz rewrote Twisted in Python and went from 20000 lines to 3000 lines.
Greg Stein reported that rewriting eShop in Python led to 90% fewer lines than C++.
Bruce Eckel, author of Thinking in C++ and Thinking in Java reports a tenfold productivity gain with Python.

I'm sure there are many situations where the improvement is much smaller, the study by Prechelt I referred to before had smaller differences, but it showed more than 50% reduction in lines of code and developer time, as well as fewer bugs.

...given the length of each of your replies to this story...

I wish I was able to explain things more briefly. It's not a skill I have I'm afraid. :( It impresses me when people can get the same message through in 50 words that I need 250 for, but at least I can do that stunt in code! :)

...I'm surprised that you should be so afraid of a few more keystrokes.

Fewer lines of code means fewer bugs. Many studies verify this.
Fewer lines of code means that it's easier to understand the program.
Fewer lines of code means shorter development time. This has also been verified many times. Less time spent coding means more time to test, rewrite bad parts and relax. I.e. better program, happier programmer.

All this naturally assumes that the fewer lines of code comes from using a language which lets you use a higher level of abstraction (or from finding a simpler solution to your problem). Every seasoned C or C++ programmer knows that squeezing too much code into one line will make debugging much harder, and that it's better to use a few lines more. That might make it hard to think that another language will allow you to reduce development time and bug count while making you type much less, but it works. Just like C was a big step forward from assembly language, Python was a big step forward from C.

Among the actively used and developed languages, Python stands out as a language where readability and ease of use and learning has a high priority in its design. It's not without warts, but it's a lot better than most, even if several others reach the same level of abstraction.

have you ever released a project that demonstrates your approach to unit testing?

SystemSpecifyer is publically available at sourceforge (see also www.systematik.se). I haven't been involved for a year, but there are few checkins since then. It's a rather odd piece of software though, and it has dependencies to particular versions ZODB and wxPython, but if you want have a look at it, go ahead. It's far from perfect, but the unit test were very helpful to me.

I wrote unit tests into Trestlemail 4 years ago.

Did you write the tests before you wrote the code to test? For me that makes a big difference. I've never managed to catch up with unit tests if I rushed ahead and wrote working code before the tests. This will then come back and haunt me when I need to change the code and don't have tests that show me at once when I break things. I'm not sure extreme programming works well in all situations, but I think they are right about writing the tests first. Among other things, it makes you write programs that are testable! Having the focus on testing from the start makes a difference.

There is much more to do on the subject of testing, and I think this is a very interesting subject, but we've strayed far off the issue of GNOME now.

Suffice to say that many project show that it's possible to write dependable software in Python. That Python is one of three permitted languages on the floor of the New York Stock Exchange and that it's used to plan space shuttle launches for NASA also suggest that you can write fairly reliable code with it.

GNOME Platform Stormclouds

Posted Mar 29, 2004 21:18 UTC (Mon) by bronson (subscriber, #4806) [Link]

If unit tests only test classes in isolation, then there's no way that they could be a good substitute for static type checking (as you claimed earlier). What about inter-unit errors?

FYI, the examples you posted are contrived. They either involve a significant amount of refactoring and redesign (eShop) or a heavy reliance on standard libraries (Twisted). Python is definitely less verbose than C++, but 90% is laughable. Remember that we're comparing langauge features here, not programming environments.

GNOME Platform Stormclouds

Posted Mar 29, 2004 23:31 UTC (Mon) by mly (guest, #2171) [Link]

If unit tests only test classes in isolation, then there's no way that they could be a good substitute for static type checking (as you claimed earlier). What about inter-unit errors?

I'm sorry if I was unclear. The purpose of the unit test is only to test the unit under test. That doesn't mean that you typically replace the calls to external units with stubs. You only use stubs when there is a good reason for it, e.g. when networking or database access is too slow, and then you can probably still have some simple test included that uses the interface.

In SystemSpecifyer for instance, I use ZODB's (non-persistent) dummy storage when I test my business logic. I access ZODB through the same module (kbase) to store things, but it's never saved to disk in these tests. On the other hand, the unit tests for the kbase module use filestorage, like my real application.

Another extreme programming practice that helps solve this problem is continous integration, but this is turning into a full XP tutorial... :)

Python is definitely less verbose than C++,

Great! We agree on something! :)

but 90% is laughable.

I never claimed than a ten-fold productivity gain was typical. I quoted a number of sources claiming from 2-3 and up to 10 times improvement. Your gain will depend on your use case. I think that several times more code in e.g. C than in Python is typical though, and that's what I wrote. That's my personal experience after seven years with Python and fifteen with C/C++.

Remember that we're comparing langauge features here, not programming environments.

I've just tried to explain why programming in Python will typically lead to fewer bugs, not more bugs, than e.g. C, even though Python lacks static typing. (But I guess that's easier to experience from programming than to convince others about in a comment thread on LWN...)

I'm not sure what you mean by "programming environment", but Python's standard library is certainly a part of Python, and a significant reason for Python's success.

GNOME Platform Stormclouds

Posted Mar 30, 2004 20:24 UTC (Tue) by bronson (subscriber, #4806) [Link]

mly, I really wish you would edit your replies to a more manageable size. I now have no idea what you're trying to say.

This thread started when you were advocating using unit tests instead of static type checking. I have been saying that in most real-world situtations, unit tests are a poor substitute for static type checking, and I maintain that position.

I don't understand why you're trying to sell Python so hard. It's a language, not a used car! And I'm not even in the market...

GNOME Platform Stormclouds

Posted Mar 29, 2004 23:56 UTC (Mon) by mly (guest, #2171) [Link]

I just stumbled over a blog by Bruce Eckel that might
explain what I've tried to say better than I can.
http://mindview.net/WebLog/log-0052

GNOME Platform Stormclouds

Posted Apr 1, 2004 13:27 UTC (Thu) by joib (guest, #8541) [Link]

That was a nice argument from Bruce. He has also argued similarly before, in a previous blog entry. He even spells it out in bold font: "If it's not tested, it's broken."

GNOME Platform Stormclouds

Posted Apr 2, 2004 4:10 UTC (Fri) by dvdeug (subscriber, #10998) [Link]

Proper unit tests will find all errors that the compiler would.

That's not at all true. Compilers may find errors that no amount of testing on your system would ever find, but the instant they run on a big-endian system or a system with a different libc or different size of integer they would have failed. They can find undefined variables that would have defaulted to a value that would usually work. One problem with unit tests is that programmers frequently have blind spots; while compilers have blind spots, they're much more consistent and usually different from the programmers.

If you create a variable for seconds in a minute, and handle every case from 0 to 59, you will be wrong once every leap-second. And if you use the standard time library in Ada, it will emit a warning on compilation, whereas a language that was interfacing two integers would have no warning, and no bug unless you specifically tested that case, which many programmers might not even know exists.

What about the cases where you don't what answers to expect? A book was published about the results from an expirement done in Fortran, and afterwards it was discovered that there was two different variables (one implicitly defined) where there should have been one. If he could have written a unit test to find this, he would have noticed it before the book was published.