I'm interested to see this process start, because I have been attending Linux Conference Australia (http://lca.linux.org.au) for many years now. Once you've got an established event, demographic and location in time and date, you'll find it gradually grows from there.
I think it's interesting that you mentioned two FOSS conference management packages, as there's also Zookeepr (http://zookeepr.org/) that LCA uses. It would be great to see these packages merge or share features to try and produce a small set of conference management packages that can be deployed not just by FOSS conference organisers but by anyone organising a conference. I also think there's a market for a SaaS site for these installations, featuring a step-through process that creates a new subdomain and installs the software and branding, which could fund new development.