[Gllug] Back-out plans on Debian/Redhat?

[Bruce Richardson]

On Sat, Mar 07, 2009 at 11:15:25AM +0000, Khusro wrote:
> Hi, I was reading
> 
> "The Practice of System and Network Administration"
> http://www.amazon.co.uk/Practice-System-Network-Administration/dp/0201702711
> 
> and the chapter on Change Management states that you must have a
> back-out plan in case of problems when you are making major updates to
> a system.
> 
> What I'm not sure about is for example, if I have a critical
> Debian-based webserver which has a lot of updates due (Apache, Mysql,
> Kernel, Libc, etc) how do I implement a back-out plan for
> dist-upgrades in case there are problems?

Is your critical server tracking Debian stable?  If so, then the
dist-upgrades would only be relevant when shifting from (say) Etch to
Lenny and this is more of a special event.  Or is it thqat you track
testing and so have to do dist-upgrades every so often to cope with new
dependencies?

If you are working on a small scale (maintaining a handful of servers)
and have time to fiddle with individual systems if they fail, then you
can get away with a simple strategy making sure that you keep a copy of
every package you install, so that you can always roll back to an older
version.  If working on a larger scale or with strict uptime
requirements, then you need to look at a system that includes some or
all of the following:

  * Maintaining local mirrors of the Debian archives, so that you have
    complete control of what will happen when you do an upgrade or
    dist-upgrade.  This allows you to test the impact of upgrades in
    complete confidence that your main systems will not be affected.
  * Automated provisioning systems (e.g. fai, replicator, cobbler) so
    that you can deploy/redeploy test and production systems in minutes.
  * Configuration management systems (cfengine, puppet), preferably
    with the configuration stored in a version control system.
  * High Avaliability and load balancing implementations so that no one
    component is a single point of failur
  * Test networks which allow you to duplicate the conditions of
    production networks and then test the impact of upgrades

The more of these things you implement, the more confident you can be
about both the impact of a change and your ability to recover from it.

> 
> In the case of Apache, would a simple "apt-get install
> previous-version.deb" work?

Not without more work.  Firstly, the default behaviour of "apt-get
install" is to install the latest available version - apt-get doesn't
let you specify a version at all, although it does let you specify which
distribution (e.g. stable, testing, lenny, etch, some custom local
distribution you made) to use.  So you would need to make sure you
either had a repository available in which the previous version was the
latest available, or just make sure you kept the old package and use
dpkg to install it.

Whether you also then had to do some configuration fixing would depend
on whether there were configuration format changes between versions.

> 
> Is that really all there is to backing out or is there more?

There is often more but it all depends on the particular
package/component and the resources it manages.  A new version of a
package might have completely changed its storage backend.  It may not
be easy to roll back to the previous version unless you are prepared to
lose all new data since the upgrade happened.  This is the kind of thing
you have to plan for.

> Also,
> what about a Redhat / CentOS system, can you just install the older
> RPMS on those and things will be fine?

The same basic issues apply.

-- 
Bruce

A problem shared brings the consolation that someone else is now
feeling as miserable as you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 204 bytes
Desc: Digital signature
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20090307/4cdac5f5/attachment.pgp>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug