Facebook, Google, LinkedIn, and Twitter have decided to make sure that a relational databases is “web-scale”, so they have put their efforts behind WebScaleSQL, a branch of MySQL 5.6 Community Edition.
MySQL is already used by the above mentioned companies at a web-scale level, but they are facing “similar challenges with the technology”, and they came up with the idea to “create and maintain the common set of changes required to use MySQL and further customize it for our needs, rather than each of us maintaining separate branches,” said Steaphan Greene, Software Engineer at Facebook, in an interview with InfoQ. The project is “open, so everyone in the MySQL community can benefit and contribute to the code,” added Greene.
When asked why MySQL and not MariaDB, Greene said that they “reached a consensus that MySQL-5.6 was the right choice for this, as it has the production-ready features we need to operate at scale, and the features planned for MySQL-5.7 seem like a fitting path forward for us,” but they will “continue to revisit this decision as the ecosystem evolves.”
It is important to note that this is a branch of MySQL 5.6.17 and not a fork, and the changes introduced in WebScaleSQL are going to be pushed back into the trunk as long as “MySQL community releases continue,” according to the project’s FAQ.
The 4 web-scale companies have collaborated on MySQL for the past few months, setting up an automated framework and a suite of stress tests for evaluating proposed changes to the codebase. When a code change is submitted it gets included in the codebase if it useful to at least another company from those participating to the project. Patches specific only to one company are not included.
So far LinkedIn has contributed “some changes to the test framework, suggested improvements and reviewed changes submitted by other members,” told us Davi Arnaut, a LinkedIn engineer.
Twitter has contributed with a back port of a buffer pool optimization from MySQL 5.7, support for NUMA interleave policy, and a number of fixes.
All project commits can be tracked on GitHub’s project page.
For the near future, Facebook intends to contribute an asynchronous client, including “production-tested versions of table, user, and compression statistics,” compression and logical read-ahead.
For the long term future, WebScaleSQL will attempt to address the scalability issues that very large MySQL deployments face as they arrive.
WebScaleSQL will not provide builds but only source code. This allows each of the four companies to download the code, add company-specific patches and optimizations and create the builds as desired.