[Sussex] Ruby DNS resolver

[Ronan Chilvers]

On Wed, Sep 06, 2006 at 12:56:54PM +0100, Nic James Ferrier wrote:
> Ronan Chilvers <ronan at thelittledot.com> writes:
> 
> > Been kicking around with ruby and DNS resolving, as you do!! Finding the
> > DNS resolution is VERY slow. Putting together a little class to check IPs 
> > in DNSBL lists. Looking up 1 ip in 4 DNSBLs takes over a minute which is 
> > way too slow. The eventual idea is putting together a little policy
> > script for postfix which scores a set of mail attributes based on
> > various stuff including DNSBL results.
> >
> > Anyone any thoughts on speeding this up? I've tried both resolv and
> > resolv-replace libraries and there's no discernable difference in
> > speed.
> 
> Any idea why it's so slow?
>
> Are normal DNS lookups ok?

Yup.  Here's an example with a single lookup.

ronan at rncubuntu:rbl $ time dig +short @ns1.njabl.org 2.50.159.24.dynablock.njabl.org TXT
"Dynamic/Residential IP range listed by NJABL dynablock - http://njabl.org/dynablock.html"

real    0m0.123s
user    0m0.004s
sys     0m0.000s
ronan at rncubuntu:rbl $ time ruby dnsbl.rb
"Checking 2.50.159.24.ns1.njabl.org"
"DNSBL_DYN_NJABL: 0"
"SCORE: 0"

real    0m20.369s
user    0m0.024s
sys     0m0.004s

> 
> Are DNS lookups in python (or perl) ok?
> 
> Try:
> 
>   python -c 'import socket ; print socket.gethostbyname("www.tfnet.co.uk")'

Had a fiddle about with this and some ruby equivalents... 

First python:
ronan at rncubuntu:rbl $ time python -c 'import socket ; print socket.gethostbyname("dynablock.njabl.org")' 
209.208.0.104

real    0m0.035s
user    0m0.020s
sys     0m0.004s

Whoosh!!

Now ruby sockets:
ronan at rncubuntu:rbl $ time ruby -e "require 'socket'; p Socket.gethostbyname('dynablock.njabl.org')"
["www.njabl.org", ["www.njabl.org"], 2, "\321\320\000h"]

real    0m0.014s
user    0m0.008s
sys     0m0.000s

Even more whoosh!!!!!

Now via the ruby resolv library:
ronan at rncubuntu:rbl $ time ruby -e "require 'resolv'; res=Resolv::DNS.new; p res.getresource('dynablock.njabl.org',Resolv::DNS::Resource::IN::A)"
#<Resolv::DNS::Resource::IN::A:0xb7cb27b0 @address=#<Resolv::IPv4 209.208.0.104>>

real    0m5.036s
user    0m0.024s
sys     0m0.004s

Not whoosh!!!!

And now via the pure ruby resolv-replace library:
ronan at rncubuntu:rbl $ time ruby -e "require 'resolv-replace'; res=Resolv::DNS.new; p res.getresource('dynablock.njabl.org',Resolv::DNS::Resource::IN::A)"
#<Resolv::DNS::Resource::IN::A:0xb7c41870 @address=#<Resolv::IPv4 209.208.0.104>>

real    0m5.094s
user    0m0.024s
sys     0m0.000s

Sorry if that's horribly wrapped... long lines...

So it seems that the resolv libraries are the bit that's slow... seems
excessive though...
 
> I use a local, caching name server to speed up regular calls... but
> that wouldn't solve an upstream problem.

Don't really want to do this... I'd rather have simpler code and cache
results within the software... after all I only need to have a yes or no
answer to the DNS lookup so maybe I'll play around with using
socket.gethostbyname and see if that gives me enough.

Cheers
-- 
Ronan
e: ronan at thelittledot.com
t: 01903 739 997

This email has been digitally signed using GNUPG to verify the identity
of the sender. Please see http://www.gnupg.org/ for further information.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
Url : http://mailman.lug.org.uk/pipermail/sussex/attachments/20060906/45cb3580/attachment.pgp