Weekly Edition
Return to the Front page
| |||||||||||||
Engine Yard transitions to PostgreSQLPostgreSQL version 9.2 was released on September 10th, with many enhancements that web application developers—and companies who host web applications—are excited about. One of the most excited is Ruby on Rails and PHP application host Engine Yard, who recently switched its default database option from MySQL to PostgreSQL. Combined with recent data on database migrations, Engine Yard's switch of databases signifies a sea change in the long rivalry between PostgreSQL and MySQL, as well as in Ruby on Rails development. For coverage of the PostgreSQL 9.2 features, see the LWN article on PostgreSQL 9.2 Beta. For information about the switch, I interviewed Engine Yard Lead Data Engineer Ines Sombra. But first, some background. Readers who are already familiar with Rails, MySQL, and PostgreSQL can skip down to the interview. Ruby on Rails and MySQLRuby on Rails is a full-stack web framework, based on automated code generation. Rails is implemented in Ruby, a programming language which existed for ten years before Rails was introduced in 2004. It's "full stack" because it handles all parts of a web application above the database and operating system level, including querying the database, generating HTML pages, and responding to HTTP requests. Rails has become one of the top five web development platforms because of its rapid and beginner-friendly web application building. The centerpiece of this is a code-generation engine, which creates the initial "scaffolding" for the web application developer, using the Model-View-Controller (MVC) design pattern, saving the developer a lot of time by handling many of the repetitive tasks that are normally required. The simplest Rails applications perform what is known as CRUD, for Create, Read, Update, and Destroy records in a database. Early Rails supported only the MySQL database system, since it was regarded as the simplest, most developer-friendly SQL database available. That allowed developers to focus on keeping all of the application logic inside Rails. While there were early attempts to introduce PostgreSQL support, none of them really caught on until Rails 3.0 was released, so the vast majority of Rails developers used only MySQL through 2010. Rails hosting, Engine Yard, and HerokuWhile Rails made creating and developing a web application much easier than before, it did nothing to reduce the difficulty of hosting a web application. If anything, the highly dynamic nature of Rails makes it one of the most resource-consumptive web frameworks, requiring skill and experience in deployment, scaling, and uptime maintenance. Recognizing this dichotomy in 2006, Tom Mornini, Lance Walley, and Ezra Zygmuntowicz founded Engine Yard, a "fully managed web host" or "cloud host" for Rails projects. Engine Yard allowed the developer to just write code, and leave installation, scaling, and uptime to others — for a monthly hosting fee. Initially, Engine Yard supported only MySQL for data storage, and hired a large, expert MySQL team to manage thousands of MySQL databases. Engine Yard built a sophisticated "database as a service" infrastructure on MySQL to support its customers. As the number one Rails hosting option, this meant that the majority of hosted Rails applications were not just on MySQL, but on Engine Yard's MySQL. However, a year after Engine Yard launched, another team of entrepreneurs founded a competing Rails hosting service: Heroku.com. Heroku introduced a Git-centric model of application deployment that appealed to startups practicing agile development, and by 2010 it had grown into a strong competitor to Engine Yard for Rails users. Unlike Engine Yard, Heroku used PostgreSQL as its default—originally its only—database for customers. Heroku began promoting PostgreSQL among Rails developers and its user base, culminating with the introduction of a PostgreSQL cloud hosting service this year. Today, both Engine Yard and Heroku support additional platforms, such as PHP, Python, and Java, in addition to Rails. Migrations from MySQL to PostgreSQLIn April of this year, Engine Yard introduced PostgreSQL 9.1 as an option for its users. In August, Engine Yard announced that, with the release of PostgreSQL 9.2, it would become the default database option for new applications. Engine Yard's users are not alone in migrating from MySQL to PostgreSQL. 451 Research recently released a subscription-only study, called "MYSQL VS. NOSQL AND NEWSQL: 2011-2015", that looked at tech sector MySQL users and their plans two years after the Oracle acquisition. For the one out of six MySQL users planning to migrate away from MySQL, the most popular option is PostgreSQL. In the Rails community, Planet Argon found that 60% of Rails developers now preferred PostgreSQL, up from 14% just three years ago. Interview with Ines SombraTo fill us in on the details of Engine Yard's move from MySQL to PostgreSQL, I interviewed Ines Sombra. Ines, who was born in Argentina, became a Rails developer and enthusiast while working at Engine Yard. Josh: What's your role at Engine Yard?
As the Lead Data Engineer, I'm fortunate to work with a
fantastic cross-functional team of engineers and DBAs [database administrators] to formulate Engine Yard's data storage strategy. We help customers meet their data needs and develop products to bring emerging data storage technologies into our platform. Our work also includes maintenance of all supported databases.
Josh: What was the primary motivation for Engine Yard to offer PostgreSQL support?
Our primary motivation is to provide an extensive and robust stack and give our customers a wide array of excellently supported choices. MySQL had traditionally been our only supported database on Engine Yard Cloud but we decided to expand our stack with PostgreSQL to provide:
PostgreSQL also was internally loved, since the majority of our internal applications are already running on Postgres. We truly believe that PostgreSQL is the future of relational open-source databases and we are happy to provide our customers with great support for it. Josh: Do you expect Engine Yard customers currently using MySQL to migrate to the new PostgreSQL databases?
We do, a few of our largest customers (with established MySQL applications) have already inquired about the feasibility of migrating to PostgreSQL. We are working with them to address their needs and find ways to better perform these types of migrations.
While we will always support customers using MySQL, we expect the number of new PostgreSQL applications to grow as the new default makes it easier than ever to get started. Josh: What was the timeline of this change? How long did it take?
It took us a little over a year to roll out our new PostgreSQL infrastructure. We started in Q2 2011 and our release schedule included both 9.0 and 9.1 versions ...
Our early PostgreSQL 9.0 release was amongst the most popular we've ever had. Customers started using it in production applications immediately, so we accelerated our engineering processes to better serve their requirements. Within 4 months after 9.1 became available in our platform we were able to make it our new default database. Josh: What were the biggest technical obstacles you encountered in deploying a PostgreSQL infrastructure, and how did you solve them?
We didn't encounter major technical obstacles per se, but rolling out PostgreSQL support helped us restructure and enhance some of our existing architecture and processes. Here are a few of the notable changes:
Redefined Assumptions: We have traditionally been a MySQL shop and our product made assumptions based on the existence of a MySQL database. Our tests and continuous integration suite assumed that every new environment would have a MySQL database associated with it. Defaulting to PostgreSQL in our codebase allowed us to introduce the concept of a new default. We were able to break away from dependencies by refactoring tests and redefining what we expect from customer environments. Allowed Extensions: PostgreSQL has a rich extension ecosystem and we want to encourage our customers to explore it. Engine Yard Cloud customers have dedicated instances that can be further customized by applying custom Chef recipes. We provide a way to enable over 30 available extensions on PostgreSQL environments. We curate this repository for all supported versions of PostgreSQL and continually add new extensions based on customer requests. PostGIS and hstore have been the most popular extensions installed. Standardized Architecture: Engine Yard Cloud sits on top of AWS [Amazon Web Services] and we rely on EBS [Elastic Block Storage] volumes for database persistence. Unfortunately, snapshots taken on 32-bit [PostgreSQL] instances cannot be used on volumes mounted in 64-bit architectures. Customers with PostgreSQL databases had to dump and restore their data in order to vertically scale to bigger instance sizes. We solved this problem by rolling out 64-bit instance types for small and medium sizes and defaulting all databases to use a 64-bit architecture. Ease of Upgrades and DR [disaster recovery]: At the moment we are working to make the process of transitioning between database versions easier and more automated. We are looking at tools (like repmgr) that would allow us to replicate across versions and environments. One of our high priority items is to roll out PostgreSQL 9.2 support based on best practices we've learned along the way and allow customers to upgrade with minimal impact. Josh: Which 9.2 features are Engine Yard users most interested in, and why?
We are very excited about the features that are included in the PostgreSQL 9.2 release. We think our customers will be particularly interested in native JSON support, covering indexes, replication and performance enhancements.
Over the last year, we have seen an increase in the use of document-oriented databases like MongoDB. With native JSON support, developers have access to the schema flexibility of NoSQL databases while continuing to enjoy the ACID guarantees, operational simplicity, and transactional efficiency of PostgreSQL. JSON validation in the database helps simplify application logic and ensures that any client that connects to the database will have a consistent way to manipulate and save this type of data. We think this feature alone will be a great upgrade motivator and are looking forward to seeing it live. [Josh Berkus is a member of the PostgreSQL Core Team.] (Log in to post comments)
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 8:18 UTC (Fri) by bronson (subscriber, #4806) [Link]
Hm, this reads like an Engine Yard ad.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 8:29 UTC (Fri) by paulj (subscriber, #341) [Link]
Definitely a puff piece, and yes, not far from something that should have "this is an advertisement" in small letters at the top.
I actually wouldn't have a major problem with LWN raising revenue that way. Such articles can sometimes be interesting (certainly more interesting than normal ads). It seems to be something that other reputable magazines sometimes do (e.g. New Scientist).
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 11:47 UTC (Fri) by zuki (subscriber, #41808) [Link]
Hmm, the first part of the article apparently describes how Engine Yard chose presumably inferior technology, since they are switching, and is now catching up. The second part is an _interview_ with a guy from the company, so it's pretty obvious that he's providing their viewpoint. The article talks too much about the competition and reversed technological choices to be a puff piece.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 13:11 UTC (Fri) by corbet (editor, #1) [Link] There were a lot of internal discussions about this piece, and some changes were made in the process. We believed there was enough useful information here to make it worth publishing. Given the number of subscriber links that have been sent, at least a few others seem to agree.Let me clarify one thing, though: we did not "raise revenue" by running this article. Indeed, we paid for it. There's no shortage of companies wanting to "contribute" articles they themselves have written; we do not have any interest in running those.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 13:13 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]
If they have useful content and you mark it differently, it might be a good source of additional revenue without affecting the quality of LWN.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 13:22 UTC (Fri) by paulj (subscriber, #341) [Link]
To be clear: There definitely is useful information in this piece, and it was worth publishing. At the same time, I imagine Engine Yard are quite happy with it too.
Also, I didn't mean to imply this might have been a paid article. It clearly contains balancing information.
That said, there's nothing wrong with paid articles, IFF they go through the usual editorial process (with an exception made perhaps for needs for balance) and are marked as such. If LWN felt this was a way to raise revenue, there would be no shame at all in it. This subscriber, at least, would be quite happy with them, as long as the main text was a click-through to a separate page and LWN feed and weekly pages were restricted to an easily-scrolled-past abstract.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 13:27 UTC (Fri) by paulj (subscriber, #341) [Link]
Oh, placed articles that are clearly marked, and go through something resembling the usual editorial process, are clearly preferable to the kind of recooking-press-releases journalism which lesser outlets than LWN often swamp their sites with. ;)
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 14:41 UTC (Fri) by njwhite (subscriber, #51848) [Link]
> There's no shortage of companies wanting to "contribute" articles they themselves have written; we do not have any interest in running those.
Good. And that is an ideal which ensures the sort of quality you consistently achieve, to a unique level.
I agree that the tone of the article isn't great. I think the articles you run that are based largely on interviews, but aren't just "question" "pr speak answer" are much more interesting. Granted I am not a database person, but I found this far less interesting an article than the average here. And it was annoying to read, thanks to the tone (others who read a lot from less reputable places probably don't worry about that so much; for me, it isn't something I expect from lwn.)
Engine Yard transitions to PostgreSQL Posted Sep 15, 2012 8:57 UTC (Sat) by jospoortvliet (subscriber, #33164) [Link]
I read the introductory part explaining rails and PostgreSQL and that was real good. Yes, it's written by a PostgreSQL developer - which is both a blessing and a curse. It means it is technically correct and thought out and complete; it is also written from a particular point of view. But I doubt you have a problem with that, do you?
Then the interview part - yes, the answers are VERY marketingy. The dude who answered the questions either is a marketeer, had marketing experience or was helped by a marketing person. And obviously didn't realize that is absolutely the wrong tone for LWN.
But you can edit what someone answers to your questions only so much - and I wouldn't blame LWN for publishing it like this. All in all - for me, knowing not too much about Rails and database stuff, it was a good read. Next time, tell the interviewee that this is for LWN - not a marketing site and it'll be perfect.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 19:43 UTC (Fri) by jberkus (subscriber, #55561) [Link]
In general, I wrote this article in order to synthesize some news from the last month which had come in about migrations from MySQL to PostgreSQL in the web developer world. EngineYard was featured because (a) it's recent news and (b) EngineYard had, historically, been a very adamant MySQL-only shop. I expected it to be generally interesting, in the same way that it would be if Yahoo migrated from FreeBSD to Linux.
To the extent that I failed to convey that, my apologies. This is my first time writing this kind of a piece, and it's a bit of an experiment for LWN as well. We'll take the criticism into account in adjusting the next article I do which contains an interview. This article went through 5 drafts, BTW.
To be completely clear: I wrote this article as a paid writer for LWN, and did so because I believed that it exemplified changes in the open source database world which would be interesting to readers, and the LWN editors thought that it would be as well.
Oh, and Ines is a woman.
Engine Yard transitions to PostgreSQL Posted Sep 15, 2012 8:59 UTC (Sat) by jospoortvliet (subscriber, #33164) [Link]
Let me, instead of repeating myself, just link to my reply:
https://lwn.net/Articles/516453/
In short, I think it was fine, just tell the interviewee next time that too much marketing speak is not so much appreciated on this site. Otherwise, it was interesting and complete. Keep writing ;-)
Engine Yard transitions to PostgreSQL Posted Oct 5, 2012 9:55 UTC (Fri) by oak (guest, #2786) [Link]
Maybe LWN could offer interviewed people some www-page where they can paste their reply text and it will give back a "marketing smell" index, with a scale from "geekspeak" to "vomit inducing PR speak"?
That rating page could use e.g. Bayesian spam filtering to categorize the answers. There could be another page where LWN subscribers could feed these filters with content that triggers their bullshit alarms.
Engine Yard transitions to PostgreSQL Posted Sep 14, 2012 20:35 UTC (Fri) by cmccabe (guest, #60281) [Link]
And the Oracle "touch of death" continues... too bad for MySQL.
Engine Yard transitions to PostgreSQL Posted Sep 15, 2012 9:00 UTC (Sat) by jospoortvliet (subscriber, #33164) [Link]
Yeah. Then again, the open core and copyright assignment models aren't that great - depending on how you deal with them of course. Lots of companies doing it follow the MySQL way: keeping development (mostly) closed in practice. Bad, imho. There are good examples - ownCloud is doing things far better imho, and so does Qt for example.
|
|||||||||||||