User: Password:
|
|
Subscribe / Log in / New account

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

The MariaDB blog reports that Oracle has stopped bundling new test cases with the MySQL source. Evidently the revision history is also no longer public. "MySQL test cases were always an important part of the MySQL source tree. They were particularly useful for storage engine developers and for other people extending MySQL, for example, at Facebook, Twitter, and Taobao. But also for Linux distributions which add their patches to the base MySQL, and even to users, who don’t modify the sources — they still want to confirm that a particular bug was fixed or that their custom-built binary has no obvious flaws."
(Log in to post comments)

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 18, 2012 9:59 UTC (Sat) by Imroy (guest, #62286) [Link]

Oracle continues on their mission to piss off as many people as possible.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 18, 2012 13:25 UTC (Sat) by mcon147 (subscriber, #56569) [Link]

I presume someone has a copy of the tests before the removal?

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 18, 2012 14:01 UTC (Sat) by cesarb (subscriber, #6266) [Link]

From what I understood reading these articles, it is just new tests that are no longer being added, instead of removing old tests, and the source history is still available.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 21, 2012 9:45 UTC (Tue) by mariuz (guest, #24892) [Link]

It looks much worse

1.The bugs database is censored, and does not provide information to users about why they should upgrade
2.The public trees under Revision Control System are mutilated. In fact, it looks like Oracle has just stopped updating them.
3.contributions to MySQL, which weren't easy before, are now made extremely harder;
4.trust in Oracle good faith as MySQL steward is declining.

And this is the moderate comment

http://datacharmer.blogspot.co.uk/2012/08/is-oracle-reall...

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 18, 2012 15:24 UTC (Sat) by sigg3net (guest, #86309) [Link]

Judging by the number of people still using MySQL, it seems they must try harder.

No worries! They probably will:)

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 18, 2012 21:47 UTC (Sat) by copsewood (subscriber, #199) [Link]

Another LibreOffice type fork then due to lack of community confidence in Oracle's stewardship, or does a MySQL fork exist which could carry community contibutions towards testsuite development ? I can't imagine wanting to do development work on this engine without as good a testsuite as possible.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 1:30 UTC (Sun) by cmccabe (guest, #60281) [Link]

There's already two major forks, Percona Server and MariaDB. Both of them seems to be "downstream" from Oracle's branch.

MariaDB has Monty Widenius, the original main author of MySQL, and is trying to position itself as the true community distribution. Percona's blog has a bunch of stuff about performance, and that seems to be what they are going for.

Oracle requires copyright assignment when contributing to their upstream releases. They offer the upstream releases under the GPL, or, for a fee, under a proprietary license. The other two are just GPL. I haven't seen a lot of discussion about this issue online. However, as far as I can see, this seems to be a stumbling block for back-porting contributions from Percona or MariaDB. So if you want your patches to make it into every version of MySQL, you have to assign your copyright to our good friend Larry E. (Or in confusing legalese, "grant joint copyright interests."). [http://www.oracle.com/technetwork/community/oca-486395.html]

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 1:50 UTC (Sun) by ewan (subscriber, #5533) [Link]

What there doesn't seem to be is a single preferred fork that everyone just moves to as there (more or less) was with LibreOffice, and for a non-Oracle example, X.org. Without that a lot of users stay with MySQL out of sheer momentum.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 2:56 UTC (Sun) by drag (subscriber, #31333) [Link]

Most people will probably just stick with the version that is easiest to install.

Distros and repos

Posted Aug 19, 2012 9:56 UTC (Sun) by man_ls (guest, #15091) [Link]

Which most of the time means: most people stick with whatever their distro has. For instance, neither MariaDB or Percona are packaged in Debian so they are out for me.

That is, if I ever wanted to leave the marvelous MongoDB and come back into the land of fixed schemas, which I don't. Ironically, for MongoDB I am ready to pay the price of adding a repo just to use a more recent version, 2.0 in this case. Having vendor-maintained and up-to-date repos is good. Note that MariaDB has no repo for Debian testing (wheezy), for example -- I use MongoDB on my dev machine with wheezy and it runs very well.

First I would not use a database not in my distro, then I actually use a database on a separate repo. Why the contradiction? Actually there is no contradiction: I started using MongoDB from my distro, and only once I was sure it was worth it I upgraded to the distro-supplied version. But having it available inside Debian was a big plus.

Coming back to your comment, both were really, really easy to install. A separate repo makes it harder to replicate the configuration on a new machine, but only by a small amount: add a file to /etc/apt/sources.list.d/.

Distros and repos

Posted Aug 20, 2012 7:24 UTC (Mon) by drag (subscriber, #31333) [Link]

Personally, I wouldn't hesitate to use a database if it's a supported configuration by the vendor (aka 'upstream') with binaries they provide. Adding a repo (or equivelent) isn't a big deal because I am going to have to have some way to manage configurations anyways.

It's more important to me for upstream support then distro support... but whether or not I choose to do that depends on the circumstances.

Usually I am not concerned with the database so much because when I, and most people use databases, it is generally because we have a app or framework we want to run that depends on them. The configuration and everything is going to be managed by the application install scripts and deviating from their documentation just means irritation and time spent on something that isn't really going to matter.

If the MariaDB folks want to have people use their software then the biggest first step is going to get the distros to swap in their software in replace of Oracle's. This sort of thing happens fairly frequently with software in distribution repo's.

:)

Distros and repos

Posted Aug 21, 2012 8:53 UTC (Tue) by mariuz (guest, #24892) [Link]

you can install quite easily Firebird in debian sid/testing/stable and ubuntu

https://help.ubuntu.com/community/Firebird2.5

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 10:10 UTC (Sun) by amacater (subscriber, #790) [Link]

There is also the Drizzle fork - available in Debian Wheezy / Testing.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 17:31 UTC (Sun) by armijn (subscriber, #3653) [Link]

Oracle just continued the existing practice of copyright assignment that was already in place when Monty ran MySQL, which seems to be forgotten at times.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 2:03 UTC (Sun) by akeane (guest, #85436) [Link]

One of my company's current projects under development is
a minimal sql client implementing a small subset of the mysql protocol (for use in very resource limited embedded systems), on the mysql website there is some documentation which purports to describe "internals" in this case the mysql client-server protocol.

The documentation is incomplete at best (packet formats are not described correctly, the state machine description one would expect for such a protocol is barely sufficient, the method for authentication is described vaguely) and wrong at worst: packet formats, client server interactions etc.

Now, I had initially thought that was just a case of over worked/stressed developers just not having enough time to do the documentation job properly as well as having to write code, fix bugs, maintenance etc.

Now I am not so sure.

Use the source Luke? Nope, we are writing the code from scratch to avoid copyright assignment issues, which is why we hoped the documentation would be usable.

Lucky for us our code is modular, we can always switch to postgresql if required.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 5:42 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

Try https://launchpad.net/drizzle-jdbc/ - it's a BSD-licensed MySQL client. It's in Java but can be easily translated.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 11:16 UTC (Sun) by gebi (subscriber, #59940) [Link]

There are also various native mysql driver implementations in erlang. IMHO that's the best documentation for binary protocols. But you should know at least a bit of erlang.

eg. https://github.com/Eonblast/Emysql/blob/master/src/emysql...

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 14:33 UTC (Sun) by przemoc (subscriber, #67594) [Link]

Was doing a similar thing a few months ago, because we needed an extremely simple non-blocking client. And I do remember that docs [1] had errors (not much though), but I no longer remember what they were. I also remember that WireShark was helpful a bit in this case, but on the other hand MySQL protocol dissector was also buggy on its own. :)

[1] http://dev.mysql.com/doc/internals/en/client-server-proto...

(Strange, I think there was an one-page version of this document, but I don't see it anywhere now.)

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 19:36 UTC (Sun) by akeane (guest, #85436) [Link]

Thanks everyone for the recommendations!

We actually got the mysql specific code finished a couple of weeks back, I was just questioning whether the documentation was such poor quality due to "ask the dev to write the documentation as well as all his other jobs" syndrome or more a case of malign neglect.

Yep przemoc, wireshark was definitely a help ;-) I am not surprised the protocol dissector had problems given the way the mysql people differentiate between different versions, it's like it was just made up as they went along!

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 19, 2012 20:42 UTC (Sun) by justincormack (subscriber, #70439) [Link]

I always understood that the reason for the obscurity was otherwise you could write a non-GPL lib to talk to mysql (the shipped one is GPL) over the network and therefore isolate the GPL-ness to just the DB, and therefore not have to buy a non-GPL license to Mysql for your proprietary software...

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 20, 2012 2:51 UTC (Mon) by ewan (subscriber, #5533) [Link]

"Use the source Luke? Nope, we are writing the code from scratch to avoid copyright assignment issues"

There are no copyright assignment issues with using the MySQL source, unless you want Oracle to integrate your changes into their releases. You're just as much at liberty to fork it as you are any other GPL project.

MariaDB: Disappearing test cases or did another part of MySQL just become closed source?

Posted Aug 20, 2012 3:51 UTC (Mon) by dlang (subscriber, #313) [Link]

they don't want the result to be limited by the GPL

Another "is anyone really still using MySQL?" post

Posted Aug 20, 2012 11:13 UTC (Mon) by robert_s (subscriber, #42402) [Link]

I'm certainly not the only person who thinks Oracle are doing the world a favour in killing MySQL.

Another "is anyone really still using MySQL?" post

Posted Aug 20, 2012 12:56 UTC (Mon) by man_ls (guest, #15091) [Link]

Not that I like MySQL much. But it would just lead more people to Oracle express edition, SQL Server or some other monstrosity without even source code.

Now, killing Oracle's product might be a good idea if a better fork were to prevail... but that is also unlikely given the brand recognition MySQL has. I wonder how a package under the GPL can have such a high degree of fragmentation, I don't recall other prominent examples.

Another "is anyone really still using MySQL?" post

Posted Aug 20, 2012 14:44 UTC (Mon) by mpr22 (guest, #60784) [Link]

I wonder how a package under the GPL can have such a high degree of fragmentation

Very easily: The upstream is a corporation that is widely distrusted by the user community and which only accepts contributions accompanied by an agreement allowing them to issue closed-license versions for profit.

Another "is anyone really still using MySQL?" post

Posted Aug 20, 2012 19:08 UTC (Mon) by robert_s (subscriber, #42402) [Link]

"But it would just lead more people to Oracle express edition, SQL Server or some other monstrosity without even source code."

Yes, because postgresql doesn't exist.

People being forced onto postgresql would probably be a boon for FOSS databases, as users realize that FOSS databases are far more capable and robust that they thought and perfectly capable of taking over the jobs of the majority of installed proprietary DBMSs.

PostgreSQL, I wish

Posted Aug 20, 2012 20:19 UTC (Mon) by man_ls (guest, #15091) [Link]

Let's face it, PostreSQL (which I have used and like very much) doesn't have the brand recognition that those other proprietary databases (or MySQL) have. People only know it as an alternative for MySQL.

Sad but true: PostgreSQL is the Sybase of the Free software world, and I would love you to prove me wrong. Perhaps the shadow of MySQL is what is keeping PostgreSQL from greater acceptance, who knows; I agree that a wider known PostgreSQL would be a great outcome.

PostgreSQL, I wish

Posted Aug 20, 2012 23:15 UTC (Mon) by khim (subscriber, #9252) [Link]

Perhaps the shadow of MySQL is what is keeping PostgreSQL from greater acceptance

Nope. It's the same thing which keeps PHP alive: availability on cheap hosting. Now, just why MySQL is available on cheap hosting and PostgreSQL is premium? First of all is of course, speed: on tiny databases and trivial requests MySQL is still faster. Another important capability: upgradeability. MySQL can be upgraded and downgraded in seconds. The latter capability is important for cheap hosters: they can plan for upgrades, but they can not test these upgrades. They really need the ability to migrate clients to new version and back in seconds. You can easily install two versions from MySQL and PostgreSQL side-by-side, but with MySQL you can easily go back and forth (depending on the reaction of clients) while PostgreSQL requires dump/restore cycle. Not acceptable.

Now, you may say that for "serious users" this is not an important capability. And I agree. But "serious users" are not born that way. They start from some position. And usually they start from MySQL. And then few years down the road PostgreSQL is fighting with users who knows MySQL, who lives with MySQL and for whom PostgreSQL is technology from Mars. Why will they switch without really serious reasons?

It's much easier to attract newbie rather then attract someone who've invested time and money in the competitor technology - and PostgreSQL developers for years refuse to admit that.

PostgreSQL, I wish

Posted Aug 21, 2012 1:27 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

>PostgreSQL requires dump/restore cycle. Not acceptable.
Not true. PostgreSQL now has live migration feature - you can migrate between _major_ _releases_ live, without interrupting service at all.

PostgreSQL, I wish

Posted Aug 21, 2012 9:46 UTC (Tue) by andresfreund (subscriber, #69562) [Link]

Which features are you referring to? Neither Hot-Standby+Streaming Rep, not pg_upgrade really give you that.
You can stitch something together if you add something like londiste or slony to the mix, but thats rather complicated to get right.

Getting really seamless migration is still very, very hard.

PostgreSQL, I wish

Posted Aug 21, 2012 14:46 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

You need to set up streaming replication (yes, it works across Postgres versions) slave, upgrade it (in-place upgrades are fast), let it catch up with the master and then switch over to it. The switchover procedure is the most complicated step here.

PostgreSQL, I wish

Posted Aug 21, 2012 15:27 UTC (Tue) by andresfreund (subscriber, #69562) [Link]

> You need to set up streaming replication (yes, it works across Postgres versions) slave
Unfortunately no. Streaming Rep doesn't work across versions. Streaming rep requires that the wal version number/magic, the catalog version number and several other identifying factors are the same between both ends.

SR/HS work by replaying the wal from the primary on the standby. Unfortunately its format changed in at least each of the last 5 releases. Even 9.3 - with 9.2 being in beta atm - already has significant changes to the format.

> upgrade it (in-place upgrades are fast), let it catch up with the master
And unfortunately, while pg_upgrade upgrades are way much faster than the traditional pg_dump/pg_restore, they aren't that fast because they require the planner statistics to be rebuilt. A whole database analyze can take a bit on a larger database.

Don't get me wrong: I *really* like postgres. I really like SR/HS. I even like that pg_upgrade exists although the way it works isn't that elegant from my POV.
Unfortunately that doesn't make those features appear :(.

PostgreSQL, I wish

Posted Aug 22, 2012 11:12 UTC (Wed) by robert_s (subscriber, #42402) [Link]

"on tiny databases and trivial requests MySQL is still faster"

Yes, that the conventional wisdom. But you never see any real proof of this against a modern and properly tuned postgres.

PostgreSQL, I wish

Posted Aug 22, 2012 18:29 UTC (Wed) by dlang (subscriber, #313) [Link]

The problem is that it is non-trivial to properly tune postgres.

What Postgres really needs is a "benchmark this box and set the parameters accordingly" utility.

The Postgres defaults are suitable for a many-years-old system that is going to be running postgres as well as many other things and try to make sure that postgres isn't going to interfere with the other things running on the system.

If Postgres is the reason for the system, the default values are grossly off from anything sane, and there really aren't many good resources to help a newby who's not an experienced DBA figure out what to set them to (even within an order of magnatude)

PostgreSQL, I wish

Posted Aug 22, 2012 18:56 UTC (Wed) by raven667 (subscriber, #5198) [Link]

What's your opinion on pgtune fitting this bill? There seems to be only a handful of truly critical postgresql performance tuning knobs with quickly diminishing returns that will be highly workload-specific and not hardware-specific. I wonder if there is any idea in the postgresql community to just have it detect cpu/memory at startup time and automatically adjust the internal buffers unless overridden by the config file, making pgtune unnecessary.

PostgreSQL, I wish

Posted Aug 22, 2012 19:31 UTC (Wed) by andresfreund (subscriber, #69562) [Link]

Some of that is happening (wal_buffers is autotuned since 9.1, max_fsm_* doesn't exist anymore since 8.4, default_statistics_target has a sane default since 8.4, 9.3 won't need adjustment of sysv /proc settings for most things), in most of the other situations its either hard to do or it very much depends on the environment. Many pg instances don't run on a dedicated machine. Many database servers serve multiple pg clusters at the same time. Many database servers run in virtualized environments where you get better average performance if you use less memory.

There are other variables where nobody fought enough to get them changed. Other changes are beneficial in a wide range of systems but hurt others badly...

PostgreSQL, I wish

Posted Aug 22, 2012 19:40 UTC (Wed) by andresfreund (subscriber, #69562) [Link]

I agree that its way too hard to tune it and that lots of folkloristic knowledge is needed to do so.

Unfortunately the "benchmark & set" idea doesn't really work in real life. Many of the parameters really, really depend on the workload you want to run:
* a high shared_buffers hurts in write intensive workloads if the dataset is much bigger than the available ram
* a high shared_buffers greatly improves read intensive workloads with a large hot set if it fits into s_b entirely
* a high shared_buffers hurts predictive answer times in write intensive workloads pretty badly on certain linux kernel versions
* a high shared_buffers setting hurts on high connection counts because of the large page table (can be alleviated with hugepages, probably coming in 9.3)
* a high max_connections hurts performance in high throughput oltp'ish workloads but is needed in beginner setups
* a high default_statistics_target hurts high throughput oltp workloads noticeably but greatly improves olap-ish workloads
* a high checkpoint_segments *greatly* improves write performance
* a high checkpoint_segments setting considerably increases recovery time after a crash/immediate restart
* a low checkpoint_timeout setting + small checkpoint_completion_target decreases response time jitter
* a low checkpoint_timeout setting + small checkpoint_completion_target considerably increases the amount of overall writes (due to checkpoints + full page writes), especially if the workload is update heavy

I could go on without a problem for quite some time.

For some of those idea exists to make a setting more generally acceptable, for others not.

PostgreSQL, I wish

Posted Aug 22, 2012 19:56 UTC (Wed) by dlang (subscriber, #313) [Link]

then possibly the answer isn't a 'benchmark ahead of time" approach, but is instead a "run it for a little while and then ask it for recommendations of things to try"

I am the type of person who tends to appriciate manual knobs to tune things, however when the normal answer to any performance question is "well, you must have things tuned wrong" without there being any way for a newby to know how to tune it, there is a problem.

The fact that MySQL has historically done better for simple queries with the default configuration leads people to be scared of Postgres.

In many ways, this is the same way the Microsoft made it's way into the corporate datacenter. They pitched it as "no expertise needed to run it, just install and go", and for small, simple setups they were close enough to right for people to believe it. The fact that running a large setup on Microsoft products frequently requires more resources, and more experise than running the same userbase on a *nix solution is missed by many people because they got started cheaply and so they assume that the ramp-up is going to be equally hard for competing products.

PostgreSQL, I wish

Posted Aug 21, 2012 9:49 UTC (Tue) by andresfreund (subscriber, #69562) [Link]

> Sad but true: PostgreSQL is the Sybase of the Free software world, and I would love you to prove me wrong. Perhaps the shadow of MySQL is what is keeping PostgreSQL from greater acceptance, who knows; I agree that a wider known PostgreSQL would be a great outcome.

While still not too nice results, I think its more meaningful to compare the actual server packages installed because the -common packages are installed as dependencies on *loads* of things.

That comparison only is valid if people install the version independent meta package and not a specific version, but I have no idea how to do a more meaningful comparison ;)

http://qa.debian.org/popcon-graph.php?packages=postgresql...


Copyright © 2012, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds