Have you ever thought that it would be helpful and “really cool” to be able to transform yourself into a bot? Well, maybe not. But the next best thing for us webmasters and web site owners is to use a tool to see exactly how Google sees your web page. Why? There may be something on your web page(s) that you need to fix (or want to fix) on your site. Something that you didn’t realize that Google’s search engine spider, Googlebot, sees that you weren’t aware that they were seeing.
When it comes to search engine optimization, it’s important to note that there are both “on site” and “off site” factors that can influence your search engine rankings. Nowadays a lot of us are focused on building links, getting noticed on the social media web sites, and bringing in more traffic and sales: but we fail to take a deep hard look at what we have on our own web pages that might be causing a search engine to not “like” the page for some reason. And I’ve seen cases where it’s just one simple change on a web page (something that you may have never thought of) that makes a huge difference in search engine rankings.
For many years, there have been online tools that allow you to see your web page just as if it’s a search engine viewing your web page. They’re called “search engine spider simulators”. There’s one here, and another here. Using a search engine spider simulator can be helpful and can help you see issues that you may not be aware of: like text on the page that you thought was viewable by a search engine: but may not be (because it’s in a graphic or image).
Google, through their Google Webmaster Labs, has recently introduced what they’re calling “Fetch as Googlebot“. It’s a tool that’s part of Google Webmaster Tools that allows you to “see” what Googlebot sees. Once you have verified your web site with Google Webmaster Tools, you can log into Google Webmaster Central and start learning all sorts of great info about your web site.
Here is how to see what Googlebot sees, using the Fetch as Googlebot tool:
1. Log into Google Webmaster Tools. Select “Labs” on the left side.
2. Select “Fetch as Googlebot” on the left side. You will be taken to the Fetch as Googlebot section of Google Webmaster Tools.
3. Enter the page on your web site that you wish for them to crawl. Once you enter the page, click on the Fetch button.
4. You may need to wait a few minutes while they process the request. Once it’s done, you’ll see a “Success” note.
5. Click on the “Success” link next to the check mark to view the page.
That’s it, it is pretty easy to view the page exactly how Google’s Googlebot (their search engine spider) views your web page. So, what can you do with this data? Well, there are several reasons that I can think of, which might include things like whether or not Googlebot can actually get to your web page and index it. Other issues might arise where you are not sure that Googlebot is seeing the same content that you’re serving up to visitors. If your page is dynamic in nature or if the page uses Session IDs or you think you might be cloaking, you may want to see what Googlebot sees by fetching the page as Googlebot.