Jan 3, 2012

SEO of AJAX Content – ASP.net Code to Crawl AJAX

In 2009, Google made a proposal about making a website’s AJAX content crawlable for SEO of AJAX powered websites. Before discussing the strategy proposed by Google I want you to remind that traditionally Search Engines/ Web Servers sees tayyabali.com and tayyabali.com#abc identical. Means Search engines ignore the URL after hash ( # ) called URL Fragment.

Google proposed Web Masters to replace ( # ) with ( #! ) for AJAX Requests on their website. In simple words tayyabali.com #abc should be replaced with tayyabali.com#!abc. Look quite simple so far.

When Google Bot see a URL containing #! it will request the URL from web server by replacing #! with ?_escaped_fragment_= . I would say to read the last sentence once again because this is the most important point we need to understand. So when Google Bot want to crawl content of the page tayyabali.com#!abc it will actually request the URL tayyabali.com?_escaped_fragment_=abc from the server whose content is indexable (AJAX content produced with #! is not indexable for Search Engines).

After understanding the above point, I hope you start thinking that on development end you need to do some extra effort to produce same content with query string ?_escaped_fragment_= for Google Bot while for users of the websites you are already creating URLs with #!

Interesting point to understand is that Google Bot (itself) doesn’t crawl AJAX content. Rather it’s an agreement between Google Bot and your Web Server for making AJAX content crawlable.

Google agrees to

  • Discover Pretty URLs Sitemaps and hyperlinks
  • Modify Pretty URLs to Ugly URLs
  • Index Content from Ugly URLs
  • Display Pretty URLs in Search

Web servers agree to

  • Produce Indexable Content via Ugly URLs
  • No cloaking by always giving same content to browser (via Pretty URL) and crawler (via Ugly URL) regardless of request.

To see all this in action, you can download a very simple workable solution using ASP.net and jQuery (for AJAX request).

Download Crawl AJAX Content in ASP.net