The birth of the open source enterprise stack [LWN.net]

The birth of the open source enterprise stack

Posted Jun 26, 2006 23:08 UTC (Mon) by khim (subscriber, #9252)
In reply to: The birth of the open source enterprise stack by davidw
Parent article: The birth of the open source enterprise stack

Don't know about PHP but for MySQL... they are mostly right. I've seen huge systems based on MySQL which are handling billions of $$ every year - and they do work. IMO it's mostly matter of programming culture and less matter of pure SQL-capability of the system used. While MYSQL is not perfect (and neither is PostgreSQL) it does scale, you can make it work - may be not as easily as PostgreSQL, but it can be done: I've seen it done.

PHP... I'm yet to see big, complex, reliable system with heavy use of PHP. Sometimes PHP is used "on the side", but when system becomes complex and PHP is in the middle of it... it's inevitable disaster...

P.S. I've seen huge systems written in Python, Java (of course) and even Ruby (hmm, it works) or Perl (horrors), but PHP... nope. Don't really know why... But may be it's the attitude ? PHP is the only language where "security update" can break half of the scripts on your site and when the question will be raised in the mailing list the answer will be "oh, these scripts where never written to the specifications, so it's Ok to break them - you just need to fix them".

(Log in to post comments)

The birth of the open source enterprise stack

Posted Jun 27, 2006 7:44 UTC (Tue) by nix (subscriber, #2304) [Link]

In any case, it's amazing how often huge systems can get away without using transactions, as long as their load is comparatively low with respect to system speed and the things they connect to are reliable enough that they don't need to roll back often.

One of my least finest hours was a day in 1999 when I broke the transaction management in a (large!) financial application by typoing in a hook and making all rollbacks throughout that application into commits instead. Nobody noticed for *six months*, until a major stockmarket feed went down. (Then all our customers tried to roll back at once...)

(And as for isolability, we all know how good certain major databases are at *that*. The PostgreSQL manual makes the point that perfect isolation isn't possible without giving the database what amounts to a theorem prover *and* complete knowledge of your app's control flow: but most databases, including major expensive ones with clowns with E in their surnames as CEOs of their controlling company, don't even try: you have to *ask* for half-decent isolation, and when you do your transaction goes read-only! PostgreSQL never had *that* problem, but it doesn't seem to stop people using said major database...)

The birth of the open source enterprise stack

Posted Jun 30, 2006 6:43 UTC (Fri) by oak (subscriber, #2786) [Link]

> One of my least finest hours was a day in 1999 when I broke the
> transaction management in a (large!) financial application by typoing
> in a hook and making all rollbacks throughout that application into
> commits instead. Nobody noticed for *six months*, until a major
> stockmarket feed went down.

This seems an important functionality which should have had a test case
(preferably written before the functionality was fully implemented)
which tests both conditions where rollbacks should happen and where not.

How easy it's to write test-cases for LAMP (and particularly PHP)
software? Is there some test-suite that could be used? I would think
the end user UI tests (needing to use "HTTP API") to be particularly
onerous, but at least LAMP would be good for storing and visualizing
the test results. ;-)

The birth of the open source enterprise stack

Posted Jun 30, 2006 17:06 UTC (Fri) by nix (subscriber, #2304) [Link]

Yes, it should have had a testcase. Alas the environment in question had such an insane build system that I hadn't at the time written a testsuite framework for it. (I wrote one immediately after this, ahem, incident: I only need my face rubbing in the bleeding obvious *once* ;) )

There are several test frameworks for webbish stuff: Cactus (part of Apache Jakarta) is half of what's needed. I don't have much experience with any of them (due to violent allergies to HTML and webbish stuff in general).

The birth of the open source enterprise stack

Posted Jul 5, 2006 19:48 UTC (Wed) by pimlott (guest, #1535) [Link]

it's amazing how often huge systems can get away without using transactions

It is amazing, but it doesn't make it right! When something finally does go wrong, it's extremely hard to debug and fix. Yes, I know, sometimes the cost of cleaning up the occasional mess is lower than the cost of doing it right. But given the ubiquity of DBMSs with transactions, it's hard to see why you'd risk it.

The PostgreSQL manual makes the point that perfect isolation isn't possible without giving the database what amounts to a theorem prover *and* complete knowledge of your app's control flow

I'm curious what you're talking about, because I can't find anything like that in the PostgreSQL manual. I think perfect isolation is straightforward using read and write logs, and that PostgreSQL does exactly that. What am I missing?

that perfect isolation isn't possible without giving the database what amounts to a theorem prover *and* complete knowledge of your app's control flow: but most databases, including major expensive ones with clowns with E in their surnames as CEOs of their controlling company, don't even try: you have to *ask* for half-decent isolation, and when you do your transaction goes read-only!

I agree that they don't even try. It's a travesty that the isolation level required for reliability in most applications--that the SQL standard says must be the default--is not the default in any DBMS I know. And when you do turn it on, many systems perform badly; PostgreSQL, with multi-version concurrency control, does quite well. But what do you mean about transactions going read-only?

The birth of the open source enterprise stack

Posted Jul 7, 2006 0:38 UTC (Fri) by nix (subscriber, #2304) [Link]

The PostgreSQL manual makes the point that perfect isolation isn't possible without giving the database what amounts to a theorem prover *and* complete knowledge of your app's control flow
I'm curious what you're talking about, because I can't find anything like that in the PostgreSQL manual. I think perfect isolation is straightforward using read and write logs, and that PostgreSQL does exactly that. What am I missing?

See section 12.2.2.1 of the PostgreSQL manual, and the discussion of predicate locking. (Note, when it says `the details of every query', it means the details. In the presence of random-access cursors I suspect this reduces to solving the halting problem.)

But what do you mean about transactions going read-only?

If you turn on serializable isolation in some transaction in said expensive proprietary RDBMS, you can no longer carry out INSERTs or UPDATEs in that transaction. It makes it really very useful :/

The birth of the open source enterprise stack

Posted Jul 7, 2006 1:24 UTC (Fri) by pimlott (guest, #1535) [Link]

See section 12.2.2.1 of the PostgreSQL manual, and the discussion of predicate locking.

Thank you, I am enlightened. I guess this could be solved by logging at the table level: when the where clause is non-trivial, mark the entire table read, and mark the table written on insert. Then, when you commit, you would see (in one of the transactions) that the table has been modified since you queried it.

If you turn on serializable isolation in some transaction in said expensive proprietary RDBMS, you can no longer carry out INSERTs or UPDATEs in that transaction.

Hmm... I've used serializable in (if memory serves) Informix, Oracle, and Sybase, reading and writing. Have I not gone expensive enough?

The birth of the open source enterprise stack

Posted Jul 8, 2006 22:58 UTC (Sat) by nix (subscriber, #2304) [Link]

My experience of Oracle is that at least in some releases, it had (and was documented as having!) the nasty read-only semantics I described :(

The birth of the open source enterprise stack

Posted Jun 29, 2006 6:01 UTC (Thu) by jwb (guest, #15467) [Link]

Well, I'd be happy to introduce you to some of these applications. For an example, I have written a system which caches the information about the last trade of hundreds of thousands of stocks. During active trading, this information comes at the rate of thousands of updates per second. The amount of information to be stored is basically fixed, as there are a fixed number of stocks to be traded. In MySQL, the database size stays nice and constant, I get atomic updates, and I also get a transaction rate up to 10k/sec on hardware costing less than $2000.

If I were to use PostgreSQL for this load, I would end up with a huge, ever-growing file, because updates are handled like inserts. I would have an always-running VACUUM process, but the file would probably grow slightly at peak times anyway. I would likely have unbounded index growth, as VACUUM does not reclaim index space (only REINDEX does this, and it requires exclusive access in some versions). In all likelihood, I would need a nightly maintenance period, which I can't afford because there are stock markets all over the planet.

So there's an example of a load where you'd better use MySQL.

The birth of the open source enterprise stack

Posted Jun 29, 2006 16:08 UTC (Thu) by dskoll (subscriber, #1630) [Link]

If I were to use PostgreSQL for this load, I would end up with a huge, ever-growing file, because updates are handled like inserts.

Correct, sort-of. VACUUM would reclaim the space.

I would likely have unbounded index growth, as VACUUM does not reclaim index space

This is no longer true. Since 8.0 (and possibly since 7.4), VACUUM reclaims index space.

The birth of the open source enterprise stack

Posted Jun 30, 2006 17:07 UTC (Fri) by nix (subscriber, #2304) [Link]

... and therefore so does the autovacuum daemon in 8.1+.

The birth of the open source enterprise stack

Posted Jul 4, 2006 7:57 UTC (Tue) by nlucas (subscriber, #33793) [Link]

You might want to check SQLite for this kind of applications.
No server setup as it's only a library using a single file as a database, but *very* fast (but ACID by default, so you need to know how to make it fast).

It's not good for multiple simultaneous writers, because it locks the entire database on a write, but most applications only write once and read many.