On operating system upgrades and a packager’s nightmare

A fairy tale

Once upon a time I did an operating system upgrade, a minor one that should do no harm, but just get me up to date by fixing any bugs in the version I had been using. It seemed like a good idea.

All seemed to be fine. I use a package provided by an external vendor and not the one produced by the operating system provider as this vendor provides a newer version of the package and I need that. The vendor has to make his package fit in the os environment his package is built for and normally does a pretty good job.

I use automation to build my systems and when I built a new one some issues appeared. Related to the new version of the OS the provider had enhanced one of his packages and the installation pulled in new dependencies. The install of the external package I use then broke as it conflicted with the new dependency provided by the OS.  While a workaround is possible: uninstall this, install that, … the nice world the package manager provides you to make these problems a non-issue was suddenly broken. Horrible.

To make matters worse the external vendor can not easily fix this. If he builds a package which obsoletes this new dependency then in earlier versions of the OS he changes stuff that is not needed. If he does not do this the breakage remains in this latest version. Of course in the real world many people use a mixture of OS versions and expect a single package to do the right thing at least for the same major version of an operating system. An external vendor’s packagers life is not a happy one. It’s hard to provide a package that integrates seemlessly without it breaking something.

Back to real life

The story above is a real story relating to RHEL 5.9 versus the previous RHEL 5.8, or CentOS in my case and MySQL 5.5 and 5.6 rpms provided by Oracle. RedHat have enhanced the version of postfix they provide (a core component of the servers we use) and added MySQL library dependencies which were not there in RHEL 5.8. This pulls in the ‘mysql’ package as a dependency, and so installing the MySQL-server, MySQL-client and MySQL-shared-compat libraries I had been using breaks as these do not currently have an rpm “Obsoletes: mysql” entry needed to make the package install go nice and smoothly.

Real life means I do not run the latest version of MySQL 5.5 but a version which works on my systems and has been tested with my software, so if I use that on CentOS 5.9 now it will break. Oracle may not mind having to update their latest MySQL 5.5 rpm (for CentOS 5) but are unlikely to be interested in updating all their previous MySQL 5.5 or 5.6 rpms for this version of CentOS just in case I might be using one of their older versions. I also am not interested in upgrading all these MySQL 5.5 servers just to get a newer version which may have issues, so probably I will end up doing a work around “remove this package”, “add that one” in a adhoc way to avoid this problem. It should not be necessary.

It has always been stated when building software: do not make functional changes in a minor version upgrade as you will break something somewhere and upset someone. I understand that providing MySQL support to postfix (and perhaps other packages) seems like a good idea but this breaks this “rule”. I also think that having to provide a package within an existing OS framework, and try to avoid breaking any native OS packages is hard, but still think as I have commented in previous posts that coming up with a way of allowing external vendors a way to do this which does not interfere with “native OS” packages would be good. Recommend that these packages get named a certain way, do not get located in “native OS” locations, and do not conflict with “native OS” libraries.
If people want to use these packages show them how to link against them and add dependencies in a way which may favour these vendors’ packages if that’s a deliberate intention. Provide these rules and life will be easier for all of us. One world where this is done is the *BSD ports systems, which are intentionally independent of the “native OS” and located separately and people can use them if needed, or not at their convenience. In my opinion something like that in the rpm world would be nice indeed and external rpm repos would not have to worry so much about adding new functionality and breaking the OS.

I am not fully sure how I am going to work around this issue, but it has been causing problems for a while now and will require some poking. I will try and get in touch with the parties concerned to alert them to this change which has broken something for me.

Am I the only one who has noticed this?

Also likely to be an issue I guess for MariaDB and Percona rpms for RHEL 5/ CentOS 5 as they’ll likely implement the same dependency behaviour.

References:

  • http://blog.wl0.org/2009/08/a-packagers-thoughts-on-rpm8/
  • https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/pdf/5.9_Technical_Notes/Red_Hat_Enterprise_Linux-5-5.9_Technical_Notes-en-US.pdf

Update: 2013-06-19

This is what I saw:

On a server that runs MySQL 5.5.23 (CentOS 5.8) I see:

which actually looks wrong. CentOS 5 does not have a mysql-libs package, but a package named mysql.  CentOS 6 does have such a named package and on early releases of the MySQL 5.5 rpms some packaging issues similar to the one described here did have to be ironed out.

Checking a later package I see things have changed but there still seems to be an issue.

The first two entries need fixing (first first needs no version and the second should not be there) as they cause other issues, but I had not noticed until now the package name mistake in the last line (which mentions a package in CentOS 6, not CentOS 5 and should be just ‘mysql’).

Maybe, given I am not using the latest version of MySQL, I will need to repackage this version myself. A quick change to the rpm spec file and a rebuild should give me the package I need. Given the src rpm is available it is good I can do that.   …. but no. Further inspection shows that the MySQL-shared-compat package is not built from the MySQL-X.Y.Z-r.src.rpm but is built independently of it and I think old libraries files are “bundled” in manually to make this package.  I vaguely remember that maybe the source tar ball has the spec file for this shared-compat package. Let me check as maybe I have to do some deeper magic to roll my own package.

Oracle: please do not do this. Provide enough of the source in the normal tar ball to be able to build the old version libraries, and build _everything_ from that single tar ball.  This would avoid all the magic you are doing now to build this package and others who need to adjust these packages can do the same.

Tags: , , , , , , , , , , , , , , ,

3 Responses to “On operating system upgrades and a packager’s nightmare”

  1. Jeremy Cole says:

    Simon,

    It seems the issue is that the postfix package does likely not really “Requires: mysql” to work. It should instead be depending on a specific version of libmysqlclient.so, which would be satisfied by any package that can provide it (including compat packages that provide *only* that file for older libraries). Unfortunately “Obsoletes: mysql” makes things even worse, as it makes it impossible to install ‘mysql’ anymore as long as any repository with a package that obsoletes it is configured.

    Sadly this is a common mistake with packaging, using “Requires: mysql” — and it breaks all over the world.

    Regards,

    Jeremy

  2. We have a similar case with Centos5/SL6 and the Percona packages when our Hosting Ops provision machines with a generic template that requires Anacron. Anacron depends on Postfix and Postfix depend on the, as Jeremy stated, a specific version of libmysqlclient.so which currently gets default satisfied by Percona-Server-client-51.

    This works well until the moment we need to deploy Percona 5.5 server after the machine was provisioned and the fastest way to resolve this is to remove all of them, install Percona-Server-shared-compat and reinstall Anacron/Postfix. As this is a new machine it should be alright but I can imagine in your case for an existing host this is not the way to proceed…

    Currently we made a MySQL template that will install everything correctly from the start, so of course when they provision the machine with the MySQL template everything will be fine from the start.

  3. Simon J Mudd says:

    Art, indeed. The thing is all the systems I manage are under configuration management and usually you expect this to work (after checking updated software) on the updated OS version. So while my original configuration management worked fine up to CentOS 5.8 the upgrade to 5.9 meant that things broke. Given CentOS 5.x like any other OS should not really be expected to behave differently when it undergoes a minor version change, having to work around this currently requires special case behaviour for 5.9 and “5.8 or earlier”. When you manage a few servers doing this by hand may not be an issue but as the number of servers you manage increases this sort of unnecessary special casing should not happen. Again this also will require Oracle to come out with an updated package to resolve this issue, which comes back to the title of this post.

Leave a Reply