[Gllug] best way to update a single production server?

[Peter Corlett]

On Fri, May 01, 2009 at 11:21:02AM +0100, Khusro Jaleel wrote:
[...]
> Since unfortunately we have only this server, and it MUST be up 24/7,
> I'm not sure how to apply these updates, since a reboot is needed.

You're pretty much stuffed. What were you planning to do when - not if -
your "MUST be up 24/7" keels over?

> Some suggestions have been:
> 1. Ask the DC to transparently re-route the IPs on that server to
> another server, run the website from there. Then, apply updates to
> original server, reboot, and point things back.

That'll probably take a few moments until the ARP cache on the upstream
router notices the MAC address has changed. Moving the MAC address too will
just move the cache problem into the switches instead.

> 2. Change DNS for the website to somewhere else for a few days
> (depending on TTL), update/reboot, then change DNS back.
>      - I have been told that this will be tricky because not everyone
> respects TTLs, allegedly AOL servers will keep the old IPs for upto 2
> weeks! I'm not sure if this is true or not.

Not just AOL. It's a Windows and Internet Explorer "feature". Yes, a two
week tailoff is about right. I assume it's related to how often IE or
Windows crashes and/or gets restarted, but haven't been sufficiently
bothered to investigate further. AOL's proxies may well exacerbate this.

> 3. Setup another server that just shows a maintenance page and point the
> main website to that, then we can update and reboot and point things back.

You have the same cut-over problem as before. However, you've now allowed
yourself the possibility of a "maintenance page", which means it's not quite
as 24/7 as initially claimed.

Given that, I'd just quietly do the aptitude dist-upgrade, which will cause
about a 5-10 minute downtime if it works correctly, and perhaps blame the
ISP if somebody complains about the server being down momentarily.

[...]
> Another option might be to use VMs? So we setup each server with 2 VMs, so
> while we update 1, the other one takes over, but uses the same IPs?

That would work to allow you to perform updates, but doesn't give you any
protection if the hardware keels over. So you'd need more than one server
doing virtual machines, and before you know it, you've got a private
computing cloud. Instead, you might want to consider renting space on
somebody else's cloud instead, such as Amazon EC2.

-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug