Cast for two

Monday, February 01, 2010

How to deal with search crawlers for your mobile site

So you've setup your mobile site (for example hosted at http://m.yoursite.com) derived from a desktop version (for example hosted at http://www.yoursite.com).Typically, you're using a Content Management System and by providing adapted templates for your mobile items, you can provide a mobile version of your site. The question is now how to deal with robots that crawl the web to build a search index. The danger exists that the robots detect duplicate content because the mobile version from a content item might contain the same text and pictures but wrapped on another template. I think the following steps should be taken:


  1. Only allow the mobile web crawlers with the following robot.txt in the root of the mobile site (for example http://m.yoursite.com) by allowing bots with user agent "Googlebot-Mobile" or "YahooSeeker/M1A1-R2D2" and to disallow all others:

    User-agent: Googlebot-Mobile
    Disallow:

    User-agent: YahooSeeker/M1A1-R2D2
    Disallow:

    User-agent: *
    Disallow: /

    Also, disallow mobile crawlers to your desktop version of your site by adding the following robot.txt in the root of your site (for example http://www.yoursite.com/robot.txt ):

    User-agent: Googlebot-Mobile
    Disallow: /

    User-agent: YahooSeeker/M1A1-R2D2
    Disallow: /

    User-agent: *
    Disallow:

    With the first robot.txt in the mobile root and the second one (here above) in the root of your desktop site, your mobile site items should only appear when people search with a mobile search engine (for example by using http://m.google.com ) but not when searching with the desktop version (for example http://www.google.com ).

    As far as I know, the MSNbot that crawls for the Microsoft Bing index together does not have a bot version that crawls strickly for the mobile Bing search engine at http://m.bing.com.

  2. Add your mobile site to Google : http://www.google.com/support/webmasters/bin/answer.py?answer=40348 , Bing http://www.bing.com/webmaster/SubmitSitePage.aspx, Yahoo (http://siteexplorer.search.yahoo.com/mobilesubmit) and other relevant mobile indexes

  3. Create a mobile sitesmap : http://www.google.com/support/webmasters/bin/answer.py?answer=34648&cbid=-1rt6r3us7wrvl&src=cb&lev=answer


Let me know I've you have anything to add to this strategy in the comments.