EuroPython 2003 Conference Report, day 2

                       Europython conference report, day 2
                       ===================================


:Date: 2003/06/26
:Location: Charleroi, Belgium
:Author: Stéfane Fermigier, Nuxeo (http://www.nuxeo.com/)

Note: 

- Text is a transcription of what the speaker said, or what I read on the
  slides. Out of band comment are marked like this [SF: this is a comment]. 

- Most errors (including grammar and spelling) are mine, not the speakers.

- Since there were actually 3 tracks running at the same time, I only cover 
the
  talks I did attend.

- Day 1 report still available at http://lwn.net/Articles/37835/


Stefan Holek: Unit Testing Zope
===============================

"There is always one more bug": we tend to think that it is the reality, but
that sucks !

"Unit testing is a practice adopted by programmer so they put in less bugs".
The key idea is early feedback. With unit testing, we become better
programmers, with more confidence in our code.

"Never touch a running system": this sucks, because 90% of the software's
lifetime is spent on maintenance. Automated tests allow us to make
changes and add features, without the risk of breaking functionality.
Unit tests can also serve as maintenance documentation.

-> Adopting automated test will reduce maintenance costs.

*Unit tests* test individual classes, they should test everything that could
possibly break, and they are written by programmers. See Test Driven
Development (TDD) by Kent Beck. Unit test are kept with the code and ship with
the code.

*Integration tests* are written to make sure the individual units work
well together. They are also written by programmer. 

In our case, we will use the same framework for unit tests and integration
tests.

*Regression tests* make sure an application still does what it is supposed to
do. Today's unit tests are tomorrow's regression tests.

*Functional tests* test the software from the user point of view. They are 
also
known as "user tests" or "acceptance tests" in Extreme Programming, and should
be written by the customer.

In Zope, we use PyUnit, the Python offspring of the xUnit family, because
it is the de facto standard for writing unit tests in Python. 

The basic concepts are:

- Test case (a single scenario)

- Test fixture (preparation to run a test case)

- Test suite (aggregation of multiple tests)

A TestCase class has a setUp() and tearDown() hooks to create/destroy the
fixtures, if necessary, and test methods (testXXX()).

Unit tests should run in a minimal environment, without dependencies, but for
Zope, "minimal" means we still have to start a Zope instance.  We need: the
ZODB, the Application object, and Zope products.

We need to configure the Zope instance, with the SOFTWARE_HOME, INSTANCE_HOME,
PYTHONPATH and PRODUCTS_PATH environment variables.

We also need to simulate requests: open the ZODB, handle the Application
object, the transaction, the REQUEST object, and security credential.

A simple way of starting Zope is::

  import Zope
  Zope.startup()
  app = Zope.app()
  ...
  app._p_jar.close() # Close the ZODB connection

Last line is important because we only have a limited number of ZODB
connections.

The Testing package, included in Zope, configures SOFTWARE_HOME and
INSTANCE_HOME, provides a test ZODB by mean of the custom_zodb.py file, ans
provides a utility to set up the REQUEST object. This will probably change in
Zope 2.7, because we won't be using environment variables like today.

Example::

  import Testing, Zope
  from Testing.makerequest import makerequest

  app = makerequest(Zope.app())
  get_transaction().begin()
  ...
  get_transaction().abort() # Ensure we don't modify the ZODB
  app._p_jar.close()

But the Testing has some shortcomings: no INSTANCE_HOME support, 
Zope.startup()
takes *ages*, no support for security testing.

There are a lot of tests written like this anyway, like the standard
Zope and CMF tests.

So I designed the TestCase package, with the following goals:

- Make it possible to distribute tests with the Zope products

- Make it possible to write tests for junior Zopistas (-> make it as close as
  "real" Zope as possible) so you have to know about the internals of the
  security policy and so on.

So I based ZTC on Testing, and provide a "battery included" Zope sandbox
to the tests, and support Zope security.

There is a framework.py file that helps run tests without having to set
environment variables by hand: it uses SOFTWARE_HOME to locate the Testing
ZopeTestCase packages, detects and handles INSTANCE_HOME installation of Zope,
supports setting up a ZODB from a custom_zodb.py (including connecting to a 
ZEO
server).

The ZopeLite.py module works like the Zope module, but is much
faster to load. It does not install products or help files. Ex:

  import ZopeLite as Zope
  Zope.installProduct("someproduct")
  ...

The ZopeTestCase.py module provides a ZopeTestCase base class that sets up a
default fixture for us: open the Application (self.app), set up folder + role
(self.folder), user folder + user with user logged in, handles the ZODB
connection, transactions, security...

With ZopeTestCase, you're not allowed to use setUp() and tearDown(), but you
have new hooks: afterSetUp(), beforeTearDown(), afterClear(), beforeSetUp(),
beforeClose().

It also provides a security interface: setRoles(roles),
setPermission(persmissions), login(), logout().

More information: read the howto on zope.org.


Niels Mache: icoya XML management
=================================

Struktur AG is a software development and consulting company based in
Stuttgart, Germany.

We're using XML for content management because of single source
publishing ("write one, publish everywhere"), durability of content,
ease of data exchange.

Icoya is a system that does XML processing. It let users create XML without 
the
technical knowledge of XML. We use Tamino (from Software AG) as an XML
database. We use ZSLT and ZPT for page rendering, with multi-stage caching.
(Note: writing XSLT is long and hard, and usually not for your average web
designer.)

Development on Icoya starter in march 2000 in Java (using IBM Visual Age,
Apache / Tomcat / Cocoon / Lucene, Tamino, Delpi and VC++ for Windows GUI
development).

June 2001: restarted development with Python, after 2 weeks of tests. Decision
was made to port Icoya from Java to Python & Zope (first contact with Zope
corp).

December 2001: first alpha. 

February 2002: icoya 2.0 XML CMS release.

Why did we port icoya in Python & Zope ? We are much faster working with 
Python
and Zope than with Java. Python = "the Operating System for Internet
Services"). Python is very flexible wrt integration of C/C++ code, as well as
with the Windows platform (COM).

The port was done in 6 months, and development speedup was about 300% faster
than Java, including QA. ZODB saved months of development time. But some Java
programmers quit because they didn't want to change.


Phil Thompson: Introduction to PyQT
===================================

There are a lot of things to say, so I'm going to go very quickly over the
slides. [SF: too bad, that means I won't probably be able to properly take
notes.]

Facts & figures: v0.1 in nov 1998, v3.6 in apr 2003; 9 Python modules, 300
classes, 2750 methods and functions. It supports Windows, MacOS X (only X11 
for
now), Unix/Linux and the Zaurus.

Licensing: similar to Trolltech model (GPL for Linux/Unix, commercial for
Windows / Unix / Linux / MacOS, educational for Windows).

SIP is the bindings generator (similar to SWIG) used by PyQt (which is just a
collection of .sip files). The aim is to integrate Python and C++ as cleanly 
as
possible, with no imposed conventions or constraints. It comprises a code
generator, a Python extension module and a couple of .h files. A SIP file 
looks
a lot like a C++ file. SIP includes support for Qt's signal/slots mechanism,
but it can be used with regular C++ class libraries. It supports version
management so that on set of .sip file will support all the Qt versions.  SIP
v4 will support new-style classes, and will also support PyQt on MacOS X with
native GUI.

Tools and resources: pyuic to be able to leverage Qt Designer, pylupdate for 
Qt
Linguist. You have a set of IDEs, like eric, BlackAdder, Wing. And you have a
book by Boudewijn Rempt (written for Qt 2, but it's still a good 
introduction).

PyQT supports several development models:

- Prototype in Python, re-use the GUI in the final C++ version.

- Use SIP to convert existing business functions implemented in C++ to Python
  modules.

- Prototype in Python, implement selected functionality in C++.

- Do everything in Python.

PyQt modules:

- qt: the majority of Qt classes (GUI widgets and threads, Unicode, file
  management, process management, drag-n-drop, clipboard...)

- qtnetwork: socket and DNS classes

- qtcanvas (QCanvas), qtgl (OpenGL), qttable (QTable), qtsql (SQL support +
  data aware widgets that can be connected to tables and views)

- qtxml (XML parser / SAX and DOM l2), qtui, qtext (contains the Scintilla
  editor class).

You can use Qt Designer with Qt: it creates XML files that describe the GUI,
then you can use uic to generate C++ code to implement the GUI, or puic to
generate Python code. You can also load the XML directly with QWidgetFactory.

I18N: You can use pylupdate to extract messages from Python code that need
translations, then Linguist to manage translations process, the lrelease to
generate binary files containing translated messages for a particular 
language.

Eric is an IDE written mainly in PyQt (38000 LOC). It can do project
management; it has a syntax-aware editor with support for code folding, call
tips, auto-completion; it has a debugger, including remote and embedded
debugging; interfaces to CVS and Subversion; it has links to Designer, 
Linguist
and Assistant; it has syntax checking, code coverage, code metrics, profiling
and wizards for automatic code writing. [SF: wow !!!]

[Demo: we can see class browsing, code folding, interactive shell, debugging,
wizards... in action. Impressive.]

Signals and slots are Qt's alternative to callbacks for communicating
between objects. [Phil had to skip the end because he was short of time.]


Stéfane Fermigier [me]: Nuxeo Collaborative Portal Server
=========================================================

[This is actually the notes I made before the presentation, with some
editing afterwards. My slides are available at
http://www.nuxeo.com/docs/europython2003/ (HTML) and
http://www.nuxeo.com/docs/europython2003/cps_europython_2003.pdf (PDF)]

I'm the head of Nuxeo, a French Zope company based in Paris. We provide Zope /
content management / collaborative work services: training / consulting /
integration / development / hosting, for global customers We're mostly 
focusing
on CPS now since it provides exactly what our customers want.

Nuxeo CPS is a free / open source (GPL) software based on Zope / CMF for
content management and collaborative work, used for intranets, extranets and
public web sites. It's both a framework and an application that works out of
the box. It's currently used by central / local administrations (mostly in
France and french-speaking countries), big and small companies, non-profit
organizations as well as regular people like you and me (both technical and
non-technical).

Since it is based on Zope/CMF, it provides all you would expect from Zope/CMF,
plus...: flexible editorial control (workflow), powerful user and access 
rights
management (user groups) versioning of documents...

Since the architecture is based on CMF, it is easily extensible with new
modules and tools (there are already more that 30 third-party modules and
add-ons available on cps-project.org). It's also possible to take off a tool
from CPS and put it into another CMF-based CMS (but standard CMF products
should also work with little or no modification).

CPS history goes back to 2001, with Intercom', the Extranet for the French
Government communication services.  This is the first time we used the concept
of "virtual publication" (documents located in one part of the site are
published in "virtual hierarchies", using metadata and the catalog to maintain
relevant access to the objects).

In 2002, we did SIT du Bas-Rhin, that started the collaborative work part of
CPS, as well as the personalization engine ("portlets", also known as
"boxes"). This project led eventually to the release of CPS 2.0 in October
2002.

At the end of 2002, we did the GroupZope that included the following groupware
components: group calendar, metadirectory manager and webmail. It's now
distributed under the name "Nuxeo Groupware Suite for CPS".

From 2002 to now, we did many projects based on CPS 2, and released CPS 2.1
(end of 2002) and CPS 2.2 (final release soon, after many beta versions). CPS
now is the basis for all our projects, including web projects for the French
Government Communication Services and for the Brittany and Oise local
administration, as well as several Intranets in various administrations and
various sized companies. CPS is also used now by other companies, like Cap
Gemini Ernst & Young.

Now our main development focus is now on CPS 3. This is a major refactoring 
and
rearchitecturing of CPS, which is needed to deal with upcoming bigger projects
(French Ministry of Defense, French Ministry of Culture, French Ministry of
Interior, French Atomic Energy Commission (CEA)). We expect to make an alpha
release soon, and a final release at the end of the summer.

Now, let's see CPS' fundamental functionalities: a customizable
web portal with portlets, publication spaces, work spaces and services.

The web portal provides a central access point to services and information. It
can be personalized globally (per site), locally (per section) and per user,
using "boxes" or "portlets". It also provides a flexible "skinning" system
(site manager chooses among a predefined set of template, without having to
type any code. 

Publication is done in hierarchical "publication spaces", were documents are
submitted by the portal members. Workflow and access right management
provide fine-grained control on who can publish what and where. A distinctive 
feature of CPS is that a same document can be published in several
publication spaces.

Workspaces are the place where collaborative production of content takes 
place.
You have personal workspaces, for your own documents, and collaborative
workspaces, where access right are granted by the workspace managers.  A
workspace usually maps to a user group, though there are many interesting
combination (one workspace for several user groups, for collaboration between
team on a specific project, etc.). Workspace services include: document
versioning, comments (as in the CMF), notification (when a new document is
created or modified), forums, wikis, request trackers...  You can add new
services using standard CMF products.

CPS can manage (including indexing and previewing) content produced by the
usual "productivity" suites (support for MS-Office, OpenOffice.org, PDF, RTF).
It can also manage images (resized automatically to fit the text)
and provides a WYSIWYG editor for IE 5+ and Mozilla 1.3+. A recent addition
is the "Flexible Documents". For the end user, they provide TTW structured
document creation and edition, and for the developer, a simple way to create
new document types using just Python script in skins.

An important feature of CPS user management is user group support (through the
NuxUserGroups products, that is independent of CMF/CPS): the main benefit is
that groups can be used everywhere single users would be expected. We now also
provide cascading user authentication sources (work in progress) with LDAP /
PKI / SSO support. There is also a service called the Metadirectory manager,
that provides flexible authentication / user data management as well as other
directories management service. Its internal model is close to the LDAP data
model, and LDAP can be used as a backend (but is not mandatory).

Forms management is an example of an easy to use service for CPS: the
NuxCPSCollector provide an easy way to create HTML forms using a web interface
(by selecting fields among a set of predefined types, that include(title,
separator, comment, string, date, email, password, etc.) and deal with user
input (user input validation and feedback and data collection).

Shared calendars are another "killer feature" of CPS:
they provide personal and shared calendars (for resources likes rooms,
computing devices...), including advanced functionality like:
event creation / notification, meeting scheduling, delegation,
iCalendar export, advanced visualization options...

Other included or third party modules include: forums, a newsletter generator,
a webmail, a workflow editor (tailored for simple administrative workflows), 
or
CPSSkins (awesome TTW configurable skins made by J-M Orliaguet at Chalmers
University).

CPS2 is now a stable and very useful platform, but we did a major refactoring
during the last months to provider a cleaner and more flexible foundation for
our future developments. This new architecture, called CPS3, will still 
provide
all the CPS2 concept, but will also be more modular, closer to the standard 
CMF
architecture, and include new tools and technical core concepts, some of them
coming from our own involvement with Zope3.

By coming back to CMF basics, we expect to make product development for CPS
easier for third party developers. Any CMF object will be usable in CPS3,
without the need to use special mixin classes or custom skins. We use standard
CMF tools as much as possible, and provide new services as new tools. The new
CPS3 workflow is, like the standard CPS2 workflow, an instance of DCWorkflow,
but it is closer to the standard way of doing things in CMF.

An important new concept and feature is internationalization of content.  With
CPS3, it's possible to translate arbitrary content (including documents and
folder-like objects) in arbitrary language (with full Unicode support), and
submit the translations to different workflows. That means that translators 
can
have their own workflow, different to the standard publication workflow, and
that different translations can be in different states in the workflow, so
that translators can work at their own pace.

A new concept, closely related to internationalization, is revisions.  In 
CPS3,
a document has a number of available revisions, with independent publication
state. So a document can be published in v.7 and pending in v.8.

CPS3 provides an event service, inspired by Zope 3, that provides loose
coupling between the tools. Change, add, removal of objects as well as 
workflow
events can be forwarded to the appropriate tools.  The event service is used 
by
the notification tool, that provides user notification to portal events, by
email, jabber, SMS...

A new Flexible Documentation version will be available with CPS3, mostly based
on Zope3 concepts: content is made of fields, with schemas grouping them;
adapters to map schemas to objects; views, made of widgets with layouts. A
major benefit is reusability of schemas and layouts, including for instance
common schemas like the Dublin Core or LOM. Everything is configurable TTW.

The most important core refactoring from CPS2 to CPS3 is that there is now a
global repository for all object revisions, with one unique docid for the
revisions of a single document. Users see proxy objects that store a language
to revision mapping. Skins have to be modified to take them into account, but
standard (non-proxied) objects still work.

There is of course still more work to do on CPS3.  Cascading / pluggable user
folders will be available this summer, as well as hierarchical groups ("groups
of groups"), WebDAV support, XML import / export, mass storage, staging (it
works already, but is specific to one project).

We also need to work on more documentations (users and developers), provider a
better cps-project.org (based on CPS3) and installers for popular platforms
like Windows and MacOS X.

Other improvement planned for the long term (as part of a 3 years contract 
with
the French Ministry of Interior) are: an improved workflow editor, a snail 
mail
management application, multisite publication (aka Publication Server), a
centralized / advanced metadata management tool, GIS integration, thesaurus,
Jabber messaging...

In conclusion, I will say that CPS is a powerful free / open source framework
for Web Content Management and Collaborative Work, already in use in several
high profile projects, with many users (1500+ downloads / month on nuxeo.org,
500+ mailing lists subscribers). We already have 5 translations and very good
long term perspective, since CPS is already the standard for web publication 
in
two french ministries and we've got commitments for up to 3 years from some
customers and partners.

More information on: 
- http://www.nuxeo.com/en
- http://www.nuxeo.org/cps
- http://www.cps-project.org
- http://cps.demo.nuxeo.com
- http://lists.nuxeo.com


Guido Van Rossum: State of Python Union (keynote)
=================================================

[Q: when will Python have interfaces ?] It will have interfaces someday but I
want to have a perfect design. I'm glad the people in Zope and Twisted are not
waiting for me to come up with it.

[Q: were are generators head ?] Generators are basically done, there is still
one minor issue.

Python is still growing: 9.5 M requests from 288 k hosts in feb 2003 (up from
5.5 M hits / 164 k hosts in feb 2001). But we need a PR offensive (although
that's bullshit, since it started many years ago). Python is more visible than
ever, but still perceived as a small player. Most normal people know the name
"Linux" (though they don't know how to pronounce it), but not Python (they
probably know about Java). It's hard to move into enterprises, and hard to
conquer education: schools have very low budgets, but even though Python is
free, they still have to pay for the books. The trick is to reach the
professors that are teaching Java as a first language, and tell them that 
there
is no risk in switching to Python.

Python 2.3 is schedules for release soon, after a beta cycle of 1 release /
month (except that it didn't turn out as expected, since I have to commit most
of my time to the Zope 3 beta release). Community help would be appreciated.

Conditional expression, after a pretty inconclusive vote for "if C: x else: 
y",
are now in Filibuster mode ("when in doubt, don't make a change"). So we are
going to bury the issue because don't want feature bloat (it's very hard to
get back from feature bloat).

At the same time I have been thinking about Python 3.0 (I can't be only
focusing on Python 2.3 and Zope 3 and other depressing subjects all the time).
One major goal is to make the language smaller [cheers from the crowd].  Take
the shit out !: string modules / strings methods, xrange() / range(), classic 
/
new-style classes, int / long, 8-bit / unicode, map+filter / lists
comprehensions, lambda / def, string exceptions (who still uses string
exceptions ? [crowd: Zope!]). Optimizations: change rules ever so slightly to
benefit optimizing built-ins. Anyway, Python 3.0 is not going to be
available soon.

Sometimes, I dream I would get a grant for the EC or whatever, and I dream
about some of the cool problems I would like to tackle.

PyPy: it's a very fun and at the same time ambitious project. PyPy separates
interpreter from runtime. This could lead to many possible payoffs: C code
generation (JIT), better PyChecker, program info database for IDE. We can also
think off Python

Python 3000: it goes beyond dropping classic classes. It's about building
whole new things: up (design your own language), down: efficient machine code
(instead of using C; we already have something similar in the form of PyRex),
sideways: more application libraries.

Interfaces: what is the Pythonic notation to talk about types ? You might want
to mix specification and implementation.

My basic philosophy about Python is that I want to keep the compiler really
dumb.

Education (CP4E): Python is a great teaching language, but we need more
libraries (like PyGames), tools (a UI constructor), books & tutorial & class
projects (ex: Guido Van Robot, similar to Karel the Robot) for all ages.

Usability: how many kind of string literals does a language need? (Isn't 16
kinds excessive). 

UI: all those toolkits (Tkinter, Gtk, Qt, wx, Win32all, Mac Carbon, Mac OSX)
are just clones. What can we do t reduce UI clutter ? Can a "Pythonic" UI 
exist? We should ask Jeff Raskin to do a keynote.

Last word: Python is an excellent *Agile Programming Language* (cf.  Extreme
Programming / Test Driven Design). *Let's have more sprints !*

Announcement: Help save Jython! Jython needs to be brought up to 2.3. There
are plenty of users, but there is only one Jython developer (Samuele can't do
it alone).


-- 
Stéfane Fermigier, Tel: +33 (0)6 63 04 12 77 (mobile).
Nuxeo Collaborative Portal Server: http://www.nuxeo.com/cps
Gestion de contenu web / portail collaboratif / groupware / open source !