[Gllug] Web Spider

[will]

neil chambers wrote:
> Hey all,
> 
> Can anyone recommend some Web Spidering software (yeah yeah - Google is 
> pretty cool ;-)) Basically I want to catalog all valid sites on a given 
> subnet. I can't do zone transfers and don't want to put something 
> together (nmap piped output looks like a strong option) if there is 
> already a round peg for my round query.

If any of the hosts on the subnet are using HTTP 1.1 to serve up 
different websites based on the hostname requested (virtual hosts) then 
simply knowing that there is a webserver running on a given IP will not 
help you much, there is no way of querying a webserver to find out what 
virtual hosts there are.  You could potentially miss most of websites in 
the subnet.

I don't think what you are after is possible.

If you know that all hostnames are assigned their own IP in a particular 
subnet however, the nmap option would work.

> (a quick google turned up JSpider. Anyone used this?)

nope
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug