WebScaleSQL RPMs for CentOS 6

Looks like this post was rather unclear. See the bottom for how to build the rpms quickly.

WebScaleSQL was announced last week. This looks like a good thing for MySQL as it provides a buildable version of MySQL which includes multiple patches from Facebook, Google, LinkedIn, and Twitter needed by large users of MySQL, patches which have not been incorporated into the upstream source tree.  Making this more visible will possibly encourage more of these patches to be brought into the code sooner.

The source is provided as a git repo at https://github.com/webscalesql/webscalesql-5.6 and as detailed at http://webscalesql.org/faq.html the documentation says there is currently no intention to provide binaries.

Instructions on building the binaries and the build requirements for WebScaleSql can be found at http://webscalesql.org/faq.html and do not look too hard. However, I prefer to install my software as rpms as this makes upgrading or removing it later much easier.

With that in mind I thought I’d try and build some webscalesql rpms.

As I’m currently using MySQL-5.6 rpms, downloaded from http://dev.mysql.com/downloads/mysql/, I wanted to build WebScaleSQL rpms which were compatible with these.  I’m aware of the Oracle-built “community” rpms which are downloadable directory from their yum repo (https://dev.mysql.com/downloads/repo/) and will probably use these when upgrading to MySQL 5.7 but moving over to that now requires changing internal infrastructure and is currently not worth the effort.

In order to build the WebScaleSQL rpms I did the following:

• download the latest source rpm, MySQL-5.6-17-1.el6.src.rpm from http://dev.mysql.com/downloads/file.php?id=451516
• extract the spec file: rpm -ivh MySQL-5.6.17-1.el6.src.rpm and look in the directory specified by rpm –eval ‘%{_specdir}’ for the spec file (mysql.spec)
• install the devtools package in order to use GCC 4.7:

http://people.centos.org/tru/devtools-1.1/readme says:

sudo wget http://people.centos.org/tru/devtools-1.1/devtools-1.1.repo -O /etc/yum.repos.d/devtools-1.1.repo
sudo yum install devtoolset-1.1

•  clone the webscalesql.git repo to any directory, let’s call it  $WSS_HOME
• create a webscale-5.6.tar.gz tar ball and put it in the RPM SRC_DIR:

   cd $WSS_HOME/.. && tar cz --exclude-vcs -f $_sourcedir/$tarball $WSS_HOME

• create a minimally changed webscalesql.spec file, based on mysql.spec, to include the new path for the compiler toolchain
• build the rpm by cd’ing into the directory with the webscalesql.spec file and running:

echo "Building WebScaleSQL..."
MYSQL_BUILD_PATH=/opt/centos/devtoolset-1.1/root/usr/bin:$PATH \
MYSQL_BUILD_CC=/opt/centos/devtoolset-1.1/root/usr/bin/gcc \
MYSQL_BUILD_CXX=/opt/centos/devtoolset-1.1/root/usr/bin/c++ \
MYSQL_BUILD_CFLAGS= \
MYSQL_BUILD_CXXFLAGS= \
MYSQL_BUILD_LDFLAGS= \
MYSQL_BUILD_CMAKE= \
MYSQL_BUILD_MAKE_JFLAG= \
rpmbuild -ba --define "distro_specific 1" webscalesql.spec

Warning: no explict check is made for the devtools chain to be installed and compilation will break if you try to use the native gcc compiler.

•  this built the following rpms:

[sjmudd@myhost PKG]$ ls -l webscalesql-*
-rw-rw-r-- 1 sjmudd sjmudd 30925511 Mar 30 13:21 webscalesql-5.6-0.20140317.155729.rhel6.src.rpm
-rw-rw-r-- 1 sjmudd sjmudd 19333116 Mar 30 13:21 webscalesql-client-5.6-0.20140317.155729.rhel6.x86_64.rpm
-rw-rw-r-- 1 sjmudd sjmudd  1962332 Mar 30 13:23 webscalesql-devel-5.6-0.20140317.155729.rhel6.x86_64.rpm
-rw-rw-r-- 1 sjmudd sjmudd 81725224 Mar 30 13:24 webscalesql-embedded-5.6-0.20140317.155729.rhel6.x86_64.rpm
-rw-rw-r-- 1 sjmudd sjmudd 54210064 Mar 30 13:21 webscalesql-server-5.6-0.20140317.155729.rhel6.x86_64.rpm
-rw-rw-r-- 1 sjmudd sjmudd  2062312 Mar 30 13:23 webscalesql-shared-5.6-0.20140317.155729.rhel6.x86_64.rpm
-rw-rw-r-- 1 sjmudd sjmudd 53369596 Mar 30 13:23 webscalesql-test-5.6-0.20140317.155729.rhel6.x86_64.rpm
[sjmudd@myhost PKG]$

These rpms should work for CentOS 6, RHEL 6 and other equivalent distributions.  I have not actually tried to use any of the packages except the webscalesql-server.

I have had very little time so far to play with this, but did replace the MySQL-server package with webscalesql-server on a development server and let it run for a few hours.

One thing I did notice is that the performance_schema* settings I had in /etc/my.cnf were not recognised by webscalesql-server and had to be commented out. That said performance_schema still seemed to be there.

I need to check further but guess that this may be due to differences between MySQL and webscalesql or potentially something I have not done correctly when building.

Other than that the server replicated fine and I saw no issues.

This has given me some basical rpms for testing.  I have not tested the package on anything other than CentOS 6 and it is likely that other changes are needed. I probably need to do a few other things like:

• Clean up the package further maybe adjusting copyrights or other messages about the packages.
• Obsolete the installed MySQL-server, … rpms so I can just do rpm -Uvh webscalesql-server …. rather than remove the MySQL-server package first.
• Add a bit of scripting to incorporate the date of the latest webscalesql commit into the version/release settings in the spec file. This avoids having to manually change the different values and as updates happen a rerun of the build script should just build a new package transparently.

I have not yet had time to look at the patches that have been applied to WebScaleSQL. It would certainly be nice to have some sort of list of functional changes (such as the performance_schema difference I noted earlier, assuming this is not a build error) of WebScaleSQL compared to the upstream source and any new configuration settings. Perhaps that will happen later?

At least for those of you who want to run a quick test of the binaries, or look at my spec file, you can find them on my website: http://ftp.wl0.org/webscalesql/.  No guarantees of any kind as you can imagine but feedback and improvements to the current spec file or build procedure would be most welcome.

2014-04-03 Update

See: https://github.com/sjmudd/webscalesql-rpm/ which I have created as a quick helper script to do the build. It still probably needs quite a bit of work but avoids copying instructions and doing stuff by hand.

2014-04-22 Update 2

Basically all you need to do is:

Install the devtoolset-1.1 or devtoolset-2 rpms (as indicated above)
Ensure that all required rpm build directories are created.
$ git clone https://github.com/webscalesql/webscalesql-5.6.git # clone WebScaleSQL sources
$ git clone https://github.com/sjmudd/webscalesql-rpm.git      # clone my builder script repo
$ cd webscalesql-rpm
$ ./build ../webscalesql-5.6                                   # build the rpms

If you later need to build an updated version of the rpms:

$ cd ../webscalesql-5.6
$ git pull # update sources
$ cd ../webscalesql-rpm
$ git pull # update my builder script (if necessary)
$ ./build  # no need to pass the webscalesql-5.6 directory location as it is remembered.