.GR Xray package
Angelos Karageorgiou Nov 2000
Updated Nov 2001
Old stuff , skip to the end for newer stuff. Thank you people and developpers
on the net for your support and code all these years.
============================================================================
Ok this is the description of this package.
First we do a transfer of the .GR domains from a trusting DNS server.
Note, do not run this ofter since it puts the DNS servers under a lot of
strain and they might decide to block transfers. Which they did in 2001.
Then we extract the .GR domains and their name servers.
We sort all domains according to name server and we
know how many domains there are and which local DNS server has them
registered. We also generate the top 10 of DNS hostings in greece
Now we can scan all these domains, and see which ones are active,
get the Server they are running and the IP of their hosting machine.
The timeout is 5 seconds , if the name does not resolve and the machine
does not answer within 11 seconds the domain is considered irresponsive.
This timeout is programmable. The source for the scanner is in the
masscrawler package unide http://www.unix.gr
Now we know all the Software used so we can have the top 10 of server
software.
WE also have the IPs os all the machines and we can find out the
number of active web hosting computers.
now we do a reverse dns and find out the company that owns the computers
and have the top 10 of hosting companies.
Also we can have 2nd order statistics of DNS vs WEB hostings for each company
In other words this is a complete Xray of the GR domain.
This process takes about 6-8 hours to run and it grinds the source machine
heavily, so I would suggest to run it only once or twice a month.
*see notes below
For Tuesday Nov 6 2000 we have
=============================================================================
There are 24430 Unique domains
Web count Specifics are in file Tue-7-11-2000//count_per_server
DNS count rankings are in Tue-7-11-2000//ranking
To 10 DNS hosting companies are
1638 forthnet.gr.
1470 otenet.gr.
1352 hol.gr.
703 domi.gr.
589 thewebpower.net.
535 internet.gr.
506 compulink.gr.
482 pegasus.net.gr.
440 combos.net.
432 nameserve.net.
Domains' results appear in Tue-7-11-2000//domains_scanned
There are 8016 bad domains and 16413 active domains
There are 5768 actual web server with different IPs
Top 10 of webserver hosting machines are
703 forthnet.gr.
618 193.92.26.84
586 hol.gr.
467 otenet.gr.
303 combos.net.
301 compulink.gr.
291 incredible.com.
242 mbn.gr.
217 spark.net.gr.
184 hellasnet.gr.
Rankings of servers are in Tue-7-11-2000//software_ranking
Top 10 software used is
7832 Microsoft-IIS
6989 Apache
711 Netscape-Enterprise
290 Rapidsite
163 WebSitePro
113 Zeus
30 WebSTAR
29 Lotus-Domino
29 AOLserver
28 Apache-AdvancedExtranetServer
Top 10 of OSes used *
171 Windows NT4
81 Unknown Irix?
66 Linux 2.0.35-37
12 Linux 2.1.122
11 Windows 2000
5 MS Windows2000
2 MacOS 7.5.5
2 FreeBSD 2.2.1
1 Solaris 2.6
1 Raptor Firewall
==========================================================================
* Note OS data is incomplete, please run xray and let it finish
Notes:
First time this program runs it will be very slow, since all the
names will have to be resolved and all the IPs also.
Subsequent runs will be much faster, since all the above information
will be cached in the DNS server.
Of course if you restart BIND all this information will be lost.
There are companies with very miserable reverse DNS tables.
For these common IPS and maps that do not resolve reversely
their IPs should go in your computer's hosts file like so:
195.119.142.126 someserver.combos.net
195.119.137.2 someserver.interagora.gr
The local DNS server caches all the requests it receives from the Xray
In other words the stats system gets better every time you run it
IT would be interesting to get second order statistics like
given the amount of registered domains per company see
how many of them are active , or How many web serving companies
are at the about 100 web sites mark. The size of the involved
companies. etc etc
to Get OS results you must be able to run queso/nmap as root
either make queso/nmap SUID , or run the scripts as root
Better OS results can be obtained with Nmap , BUT, it is extremely slow
to get. My current guestimate is that it will take more than 4 days to get
results with nmap , and even that might be an understatement.
==================================
SSL SCAN
to see how far e-commerce has made inroutees in Greece I scan
the active IPs for SSL servers. This will give us the software used as
well as the certificate authority and certificate.
Unfortunatelly the majority of SSL hosts is badly misconfigured so I had
to redo the scan including the criterion of certificate validation
i.e. that the certificate has not expired and the forward and reverse DNS
entries matched exactly. This will narrow significantly down the
number of actual SSL hosts.
==============================================================================
Historical and upgrade note written in Nov 2001.
Xray is not limited to the .GR domain. It can be used anywhere in
the world, just edit some lines in the Xray script.
The Xray package started as a Perl engine, during runs I realized
that to gain fine control over alarms (time outs) I needed to code in C.
So I opened up the trunk of old school projects, pulled out webdump and
hacked it heavily.
I left most of the other code, the one generating the statistics
from the raw data that masscrawler generates well enough alone. It is
not smart to reinvent the wheel even for artistic purposes some times.
One optimization trick in the scanner was to leave the hosts
input file unsorted. This would align the domain names by Name server.
So when I scan a host and retriece its software I already have its IP address.
If the next availlable host's IP resolves to the same as before, then
I do not need to scan it. Of course it is not entirely accurate, but for
statistical purposes it is more than enough given that most machines
now-adays use a signle server software for multiple domains.
The above optimization trick allows me to use a sigle variable
to maintain state, rather than a large hash table, not to mention that I
could not get the hash table implementation to work properly under Linux
and Gnu-C. Any ideas why not ?
The trick to fast scanning is to break your dataset in pieces
and give each piece to a different masscawler process. When they are done
you connect the pieces together and you are done.
There is a separate stats script that generates the better statistics
from the raw data. Use that to obtain the final data. The package
leaves laying around too many files. I thought at the time that they were
usefull, please visit them and see what they have.
Once upon a time I had an nmap based OS scanner, but some intrusion
detection systems , unabashed plug packetlog.pl , started carping; so I
removed that bit from executing, but it is still buried in the code somewhere.
The little proggie that displays added and removed domains from
each run is called ddiff. It is self explanatory. There is also a lot of legacy
perl code in the package. This is for your edification purposes only.
Oh, this is not a GNU package, this is totally freeware, use it
and abuse it at your own free will. But I would appreciate it if you mentioned
my name and my web site as follows.
Xray package created by: Angelos Karageorgiou http://www.unix.gr
Hack well and have fun.