So it’s been a while since I’ve posted (things have been pretty hectic). I’ve actually got another fairly lengthy post started about some recent issues with Subversion, but it’s not done yet, so it will have to wait a bit. However, I couldn’t help but take a moment to post about my recent discovery of a few moments ago.
I was working on setting up some dynamic DNS service through dyndns.org (which is free by the way), and so I was trying to figure out my external IP. After going to www.whatsmyip.com, and discovering it was some crap URL squatting site, I then decided to google “what’s my ip”. Aside from discovering the real URL was www.whatsmyip.org, in the search results I also noticed that the descriptions on some of the related sites disclosed the IP address (188.8.131.52) of Google’s search spider (http://www.google.com/search?q=whats+my+ip&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official)!! Over a period of time, if done each time Google updates its search database (which according to my familiar source who works there, it can be anywhere from daily to every few weeks), you could begin to develop a list of their spidering IPs.
Since I’m a relatively respectful person when it comes to online activity (and my online presence is pretty freakin’ small compared to the big G), this probably won’t really be of any use to me. I’m sure you could also dig around and find a published list of their spidering IPs. Google I imagine is also pretty respectful when it comes to the standard robots.txt file and other spidering deterrents that sites can employ, but that doesn’t mean everyone is. So I tried the same trick with several other search engines and the same result occurred. Yahoo!’s last search was coming from 184.108.40.206 (http://search.yahoo.com/search?p=what%27s+my+ip&fr=yfp-t-500&toggle=1&cop=&ei=UTF-8). AskGeeves was 220.127.116.11 (http://www.ask.com/web?q=what%27s+my+ip&o=0&l=dir&qsrc=0&qid=1F618F69063F9E097B55B544E303F854&page=2). MSN was 18.104.22.168 (http://search.msn.com/results.aspx?q=what%27s+my+ip&FORM=MSNH). The list would undoubtedly go on. Employing this tactic on a less respectable search engine site may indeed reveal some info on which IP’s they are using to harass the rest of the Internet.