[SWLUG] Server exploited

[James Edgeworth]

Dave Cridland wrote:
> On Mon Aug  3 04:36:30 2009, James Edgeworth wrote:
>> I run a small debian-based server which hosts just a personal page, and
>> a couple development versions of other sites (and svn, and motion
>> detection with webcams). Seems I made a big mistake thinking it was
>> secure (against a common attack).
>
> By the sounds of it, one of your users had a dictionary password. 
> Things like this happen, don't beat yourself up about it. Similarly, 
> it's often easy to get hit by zero-day (and more than zero-day) 
> attacks, just because your IP range was targetted quicker than you 
> patched.

This seems to be the case - the user account that was exploited had a 
very weak password. Changed to something very random until the user 
actually needs to access the system at all.
>
>> Sorry for the long email - hoping someone has a few pointers/things to
>> check for in this case?
>
> As Matthew Moore said, some of the rootkits that the kiddies can get 
> easily are very hard to detect and remove totally - although my bet 
> would be that in this case, you don't have a rootkit, it's not a bet 
> I'd put any money on.
>
> If it were my server - and hence under my desk here - then I'd be 
> willing to spend the time to carefully check it over and remove the 
> rootkit. I've done so before, when I've been bitten, and contrary to 
> popular belief, rootkits are not magical, and can be removed.
>
> However, doing so really requires console access, and takes 
> considerably longer than a simple reinstall. Just make sure that when 
> you restore the backups, you do not also restore the rootkit...
>
> If you happen to have multiple servers, do consider restricting the 
> SSH access to only a few IP addresses. Similarly, it's possible to 
> restrict the addresses to ranges you use, etc. This won't utterly 
> remove the attack vector, but it's very likely to mean that the casual 
> attacker won't bother.
>
> If you do have the time and inclination, by the way, then a 
> straightforward replacement of the server (or its harddisk), and 
> keeping the rooted machine [image] to clean up as a hobby is quite 
> interesting, and will teach you a great deal about security in general.
>
> Dave.

I am thinking that there isn't a rootkit, but I'm by no means an expert 
on them - the Aol folder was created with the exploited user as the 
owner:group. All files which that user owned, were removed by the 
attacker. The only places that user had write access to, were a couple 
of shared directories for media files, etc (backed up rather strictly), 
and the webroot, which was 755 for directories, 644 for files, except, 
strangely, for the actual web root folder itself, which stupidly was 777.

Checking the auth logs, about 5 different IP's accessed the weak user. 
Now I just hope to christ they didn't take any files. I don't worry 
about music files etc, but scripts for some of the dev sites would be 
the worry. I'll change the mysql passwords just in case anyhow.

Annoyingly, if I just restricted ssh to the external IP range, it would 
have prevented this. Amongst making sure the file permissions were what 
they should be.

James