[Nottingham] Office server rooms

[Martin]

T J,

sounds like a goodly frustrating hot comedy of high heat...

Unfortunately, also something that I guess is being repeated throughout
Europe and around the 'developed' world... Especially with our present
"global warming" spike!


To summarise:

On 25/07/18 18:45, T.J long thing via Nottingham wrote:
> Hi All,
> 
> An interesting event happened in the office I work in today. We host
> some servers in our rented office server room no guarantee of up time
> but usually we have no problems. But today the air con in the server
> room packed up. We weren't informed of this at any point. 

> ... noticed the door was open and heard the server fans in the server
> room going nuts ... check the air con settings (these
> have been turned up by the centre manager or maintenance guy to save
> energy in the past) ... set to 18 like usual 

> ... air con unit ... is off.

> ... they know and it will be
> fixed today and that they had no intention of telling us.
> checking of cpu temps we decided to shut down all none critical services

> ... check the air con myself to find the door still
> open and the air con on. At no point today have we been informed of the
> situation ...
> already started discussing moving our servers out of such a poor managed
> system with my managers. Rant over

Sorry, but all completely normal and to be expected from such a setup.

Sounds like the usual case of ignorant management +
incompetent/non-existent facilities people with no concern for
consequences or for the cost to others.

Can you most reasonably expect them to consider that room to be an
irritating cost and unwanted hassle for them?...


> On a side note could we hold the centre responsible if any of our
> servers fail in the next 6 months?

Extremely unlikely unless you sort of catch them red-handed physically
reprogramming your equipment with an axe.


And... I've got far more fun stories to tell of abuse of server rooms
and data centre equipment and of management ignorance and support staff
incompetence, all with added supposed "nobody's problem" syndrome. Makes
the BOFH look like a sweet helpful guardian angel!


So, do you have...

Any SLA?

Any live monitoring?

Any alarms with responsible people to jump to the rescue?

(Including out-of-hours?)

Any pre-planned emergency procedures? And practised?

Any automated emergency procedures?


And can we guess that the servers are critical to the business and hold
all the live work?

Is your (ignorant) management (in their ignorance) aware of the
vulnerability and whatever (likely/unlikely) consequential costs?


Also, it sounds like that room cannot be trusted... Open access, only
one aircon, and unknown people can adjust the aircon settings...

I'd guess that the Centre Manager and the aircon engineer have no clue
as to what all the fuss and frustration is all about. After all, you
wouldn't expect to be informed when a plumber dives into the basement
there to unblock the janitor's toilet...


In other worlds, I have some unbelievable but real stories to tell...
(Could be good for a few TheRegister articles! :-) )


(Jason, keep quiet! ;-) )

One of the better ones that comes to mind has got to be where a certain
technician, a second technician, and two others from the Facilities
Dept, each, individually, checked a server room where there was a loud
piercing alarm screaming away. Others in the office were complaining
about the noise! There is an emergency list pinned to the door.

Over about a one hour period, each of those personnel individually
unlocked and opened the door to peer inside. Each saw that all the
lights were on and flashing and heard that the fans and disks were
humming away as usual. The aircon temperature felt ok. And so they all
carried on to go get their morning break and engaged their "somebody
else will sort it" SEP field.

While the two technicians were still on their break in the kitchen room
opposite, one by one, the multiple UPSes exhausted their batteries. The
sirens scream died away.

To be replaced by the scream of the MD...

Facilities had inadvertently left the room mains power OFF.


All gobsmacking and all too often repeated...

There's lots more first hand stories of abuse of backup generators...
Circuit breakers... Crazy plumbing/piping with no care for electricity
or access... Bad earths... And...

More beer? ;-)


Also sounds like:

A fantastic opportunity to sort them out and gain a pay rise!? :-)


Good luck!

Cheers,
Martin


BOFH: B*****d Operator From Hell

SLA: Service Level Agreement
(Special note: This is in reality only the 'maximum that can be hoped
for' from a supplier. Be sharp about penalties and 'who pays'
regardless. Be aware of the consequences!)

SEP: Somebody Else's Problem

UPS: Uninterruptible Power Supply (until they are interrupted)

MD: Managing Director


Disclaimer: All just my own humble personal opinion! Enjoy! :-)