API ? Data format is not the API... [LWN.net]

API ? Data format is not the API...

Posted Feb 9, 2008 4:09 UTC (Sat) by malefic (subscriber, #37306)
In reply to: API ? Data format is not the API... by khim
Parent article: PostgreSQL 8.4 development plan

> Nope. API/ABI are within the PostgreSQL/kernel. I don't care about that. If you want the
analogy with kernel then sysfs will be a better one

Well sysfs is a bad analogy too. Sysfs is an explicit *API*. PostgreSQL binary data was never
meant to be used other than by the core server itself.

I was not saying that dump/restore cycle is a good thing. I was just saying that keeping
backwards compatibility with old binary format places a huge burden on the developers and
threatens the overall quality of the product. I don't know what is the official position of
PostgreSQL developers, but I suspect that it's in line with stable_api_nonsense.txt.

I agree, however, that there could be some better way to migrate, than dump/restore. Maybe
something like running the old server in backup mode until the new one finishes the restore,
and then replaying the logs on the new server. And, again, you can use replication to migrate
your data with near zero downtime.

(Log in to post comments)

API ? Data format is not the API...

Posted Feb 9, 2008 10:27 UTC (Sat) by jd (guest, #26381) [Link]

> I agree, however, that there could be some better
> way to migrate, than dump/restore. Maybe
> something like running the old server in backup mode
> until the new one finishes the restore,
> and then replaying the logs on the new server. And,
> again, you can use replication to migrate
> your data with near zero downtime.

Let's see. Here is a short-list of options I can think of:

1. Provide PostgreSQL with a platform-independent read and a playform-independent write, such
that you could essentially perform the backup and restore as a chain of pipes.

2. Decouple implementation-specific details of the database file format, so that the file
structure is outside of PostgreSQL itself. The engine would merely have stubs that run on
seperate threads. You could then plug in as many formats as you like, up to one a thread,
provided the plugin supported the stub's API. The transfer would not "just happen", it would
presumably be up to third parties as to what file routines go with what APIs, but it's a
realistic way to do things.

3. Write a PostreSQL "kernel" on which all threads are run, akin to an OS kernel. When
upgrading PostgreSQL, you add new upgraded threads and older threads do not get replenished
when finised. Upgrades can now be in-situ. However, this still requires the first or second
ideas to allow the migration to take place.

4. Build a  working TARDIS, obtain the database file format as used in the year 3,000 AD and
bring it back here. No further upgrades will then be needed for close to a millenim.

As an added incentive, I'm offering my entire Slasdot karma to the first person who
demonstrates bi-directional time travel as an open source development model.