Testing the Web Through GoogleBot's Eyes

How does Google see your site? How do we know? One common problem with dynamic sites are the need to use URL based session IDs. What do you do about people who don’t have cookies turned on. There are two main options:

1. Give them a nice message that they must have cookies turned on to view the site
2. Use URL embedded session IDs.

Fortunately PHP has a cool feature that allows it to detect if cookies are enabled and will automatically embed the session ID in links if not. The problem is though that with that feature enabled, now search engine spiders will be feed session IDs which essentially renders the site un-spiderable. So our work around is to check a browser's useragent against a list of know spiders to see if the page requestor is in fact from a search engine. After all that work the question then becomes, how do you know if it works? E.g. how do we mascarade as a search engine spider to test we are not getting session IDs? The answer is to use two handy little FireFox plug-ins developed by Chris Pederick:

1. Web Developer
2. User Agent Switcher

The first plug-in offers an incredible toolbox for analyzing your web page code. For our search engine spider spoofing solution we need to use its ability to turn off cookies. (I know you don't need this plug-in to turn off cookies, however every web developer should have this and it is quicker than using FF's settings) The second step is to use User Agent Switcher to change your useragent to a search engine spider such as GoogleBot. If you are new to using UAS you will notice only a limited number of stock useragents, so you will need to add GoogleBot and others. For a great set of useful useragents, upload the following file into User Agent Switcher. It will give you the settings for GoogleBot. <useragentswitcher> <useragent description=":: BROWSERS - Windows ::" useragent="" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="MSIE 6 (Win XP)" useragent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" appname="Microsoft Internet Explorer" appversion="4.0 (Compatible; MSIE 6.0; Windows NT 5.1)" platform="" vendor="" vendorsub=""/> <useragent description="MSIE 5.5 (Win 2000)" useragent="Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )" appname="Microsoft Internet Explorer" appversion="4.0 (compatible; MSIE 5.5; Windows NT 5.0)" platform="" vendor="" vendorsub=""/> <useragent description="MSIE 5.5 (Win ME)" useragent="Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 4.8 (Win XP)" useragent="Mozilla/4.8 [en] (Windows NT 5.1; U)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Opera 8.0 (Win 2000)" useragent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; en) Opera 8.0" appname="Microsoft Internet Explorer" appversion="4.0 (compatible; MSIE 6.0; Windows NT 5.0; en)" platform="" vendor="" vendorsub=""/> <useragent description="Opera 7.51 (Win XP)" useragent="Opera/7.51 (Windows NT 5.1; U) [en]" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Opera 7.5 (Win XP)" useragent="Opera/7.50 (Windows XP; U)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Avant Browser" useragent="Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 7.1 (Win 98)" useragent="Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Multizilla (Win xp)" useragent="Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Opera 7.5 (Win ME)" useragent="Opera/7.50 (Windows ME; U) [en]" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 3.01 gold (Win 95)" useragent="Mozilla/3.01Gold (Win95; I)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 2.02 (Win 95)" useragent="Mozilla/2.02E (Win95; U)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description=":: SPIDERS - search ::" useragent="" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Googlebot 2.1 (New version)" useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Googlebot 2.1 (Older Version)" useragent="Googlebot/2.1 (+http://www.googlebot.com/bot.html)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Msnbot 1.0 (current version)" useragent="msnbot/1.0 (+http://search.msn.com/msnbot.htm)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Msnbot 0.11 (beta version)" useragent="msnbot/0.11 (+http://search.msn.com/msnbot.htm)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Yahoo Slurp" useragent="Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Ask Jeeves/Teoma" useragent="Mozilla/2.0 (compatible; Ask Jeeves/Teoma)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description=":: BROWSERS - Mac ::" useragent="" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Safari 125.8 (Mac OSX)" useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/125.8" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Safari 85 (Mac OSX)" useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/125.2 (KHTML, like Gecko) Safari/85.8" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="MSIE 5.15 (Mac OS 9)" useragent="Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Firefox 0.9 (Mac OSX )" useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7a) Gecko/20040614 Firefox/0.9.0+" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Omniweb563 (Mac OSX)" useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US) AppleWebKit/125.4 (KHTML, like Gecko, Safari) OmniWeb/v563.15" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description=":: BROWSERS - linux/unix/beos ::" useragent="" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Mozilla 1.6 (Debian)" useragent="Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Debian/1.6-7" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Epiphany (Linux)" useragent="ozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Epiphany/1.2.5" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Epiphany (Ubuntu)" useragent="Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.7.3) Gecko/20040924 Epiphany/1.4.4 (Ubuntu)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Galeon 1.3.14 (Linux)" useragent="Mozilla/5.0 (X11; U; Linux; i686; en-US; rv:1.6) Gecko Galeon/1.3.14" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Konqueror 3 rc4 (Linux)" useragent="Konqueror/3.0-rc4; (Konqueror/3.0-rc4; i686 Linux;;datecode)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Konqueror (Gentoo)" useragent="Mozilla/5.0 (compatible; Konqueror/3.3; Linux 2.6.8-gentoo-r3; X11;" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Firefox 0.8 (Linux)" useragent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040614 Firefox/0.8" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="ELinks 0.9.3 (Kanotix)" useragent="ELinks/0.9.3 (textmode; Linux 2.6.9-kanotix-8 i686; 127x41)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Elinks (Linux)" useragent="ELinks (0.4pre5; Linux 2.6.10-ac7 i686; 80x33)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Links 2.1 (Linux)" useragent="Links (2.1pre15; Linux 2.4.26 i686; 158x61)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Links 0.9.1 (Linux)" useragent="Links/0.9.1 (Linux 2.4.24; i386;)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Opera 7.23 (Linux)" useragent="MSIE (MSIE 6.0; X11; Linux; i686) Opera 7.23" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Lynx 2.8.5rel.1 (Linux)" useragent="Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/0.8.12" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="w3m 0.5.1 (Linux)" useragent="w3m/0.5.1" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Links 2 (FreeBSD)" useragent="Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Mozilla 1.7 (FreeBSD)" useragent="Mozilla/5.0 (X11; U; FreeBSD; i386; en-US; rv:1.7) Gecko" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 4.77 (Irix)" useragent="Mozilla/4.77 [en] (X11; I; IRIX;64 6.5 IP30)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Netscape 4.8 (SunOS)" useragent="Mozilla/4.8 [en] (X11; U; SunOS; 5.7 sun4u)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Net Positive (BeOS)" useragent="Mozilla/3.0 (compatible; NetPositive/2.1.1; BeOS)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description=":: SPIDERS - misc ::" useragent="" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="gulperbot" useragent="Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="Email Wolf" useragent="EmailWolf 1.00" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="grub client" useragent="grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="download demon" useragent="Download Demon/3.5.0.11" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="omni web" useragent="OmniWeb/2.7-beta-3 OWF/1.0" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="winHTTP" useragent="SearchExpress" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description="ms url control" useragent="Microsoft URL Control - 6.00.8862" appname="" appversion="" platform="" vendor="" vendorsub=""/> <useragent description=":: source: techpatterns.com ::" useragent="" appname="" appversion="" platform="" vendor="http://techpatterns.com/forums/about304.html" vendorsub=""/> </useragentswitcher> Thanks to jeffd for posting this list on TechPatters