[Gllug] hourly/daily rate for linux systems admins?

[Steve Parker]

On 24/11/10 00:42, Karanbir Singh wrote:
> On 11/23/2010 11:26 PM, Steve Parker wrote:
>    
>> I wrote this very short script a few months ago to identify processes by
>> state and CPU, it works on RHEL5 (2.6.18):
>>      
> Technically, only on 2.6.18-144+ ( it wont work on<= EL 5.3 )
>    
Hmm, works for me on == EL5.3 (.128 IIRC)
>> I am neither. Like you, I just read the source. I wouldn't expect anyone
>> to come up with this kind of level of detail in an interview situation
>> though.
>>      
> This is the bit I am struggling to understand. The per process i/o stats
> are quite useful in planning and testing stages, and I even monitor (
> and trend ) it on production systems. I am sure you wrote this script
> since you had a need to look at those numbers :)
>
>    
Yes, I needed to find it, so I found it. In an interview situation - 
unfamiliar environment, artificial situation, five minutes to hear, 
understand, and fix the problem, that would be very difficult. In this 
case, it took a few meetings with business/app/os people each speaking 
different languages before I came up with this solution to translate 
what everybody else was saying ("I/O Waits are too high!" from the 
application people basically meant that they were allocating and writing 
to lots (like 60Gb) of physical RAM, which takes time. With 4 x 
quad-core hyperthreaded 3GHz CPUs, any CPU tasks were done in 
milliseconds, so the only thing the process could do, was to wait for 
the memory allocation.

Finding out (again, from the source code; can't find any documentation 
which says this) that RAM allocation is also tagged as I/O Wait (eg, man 
mpstat says it is Disk IO, even /proc/$$/status says Disk Sleep or 
something similar), finding what goes into /proc/$$/stat, and writing 
this 5-line script, probably covers two weeks.

Now that I know, it'll stay with me, but I would not expect a typical 
RHCE to know it, nor am I surprised to hear that Nix hadn't come across 
it. It takes a specific set of circumstances to come across this 
particular thing, and if you haven't been there, it does not make you an 
unsuitable candidate. As IT professionals, I find it to be a common 
thing that one person finds a troublesome problem, which they can't 
explain, only to find a colleague who happens to have seen it before.

I had a few interviews with a well-known search engine / advertiser a 
year or so ago, which ended when I got an interview with someone who 
wanted really specific answers to obscure hypothetical questions about 
DNS MX records, which I told him I couldn't answer; could we please move 
on to something else? I could find it out quite easily if I wasn't busy 
having a job interview, but I simply couldn't give him the answer he was 
looking for. I had aced the other interviews; the next step after that 
one would be onsite, but I got a phone call the next day saying that 
they weren't interested.
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug