How to deal with search crawlers for your mobile site

So you've setup your mobile site (for example hosted at http://m.yoursite.com) derived from a desktop version (for example hosted at http://www.yoursite.com).Typically, you're using a Content Management System and by providing adapted templates for your mobile items, you can provide a mobile version of your site. The question is now how to deal with robots that crawl the web to build a search index. The danger exists that the robots detect duplicate content because the mobile version from a content item might contain the same text and pictures but wrapped on another template. I think the following steps should be taken:

1. Only allow the mobile web crawlers with the following robot.txt in the root of the mobile site (for example http://m.yoursite.com) by allowing bots with user agent "Googlebot-Mobile" or "YahooSeeker/M1A1-R2D2" and to disallow all others:
   
User-agent: Googlebot-Mobile
Disallow:

User-agent: YahooSeeker/M1A1-R2D2
Disallow:

User-agent: *
Disallow: /

   Also, disallow mobile crawlers to your desktop version of your site by adding the following robot.txt in the root of your site (for example http://www.yoursite.com/robot.txt ):
   
User-agent: Googlebot-Mobile
Disallow: /

User-agent: YahooSeeker/M1A1-R2D2
Disallow: /

User-agent: *
Disallow:

   With the first robot.txt in the mobile root and the second one (here above) in the root of your desktop site, your mobile site items should only appear when people search with a mobile search engine (for example by using http://m.google.com ) but not when searching with the desktop version (for example http://www.google.com ).
   
   As far as I know, the MSNbot that crawls for the Microsoft Bing index together does not have a bot version that crawls strickly for the mobile Bing search engine at http://m.bing.com.
   


2. Add your mobile site to Google : http://www.google.com/support/webmasters/bin/answer.py?answer=40348 , Bing http://www.bing.com/webmaster/SubmitSitePage.aspx, Yahoo (http://siteexplorer.search.yahoo.com/mobilesubmit) and other relevant mobile indexes


3. Create a mobile sitesmap : http://www.google.com/support/webmasters/bin/answer.py?answer=34648&cbid=-1rt6r3us7wrvl&src=cb&lev=answer

Let me know I've you have anything to add to this strategy in the comments.

Set all links target to _blank on a XHTML 1.0 Strict using jquery

As explained here, opening links in a new window is no longer possible via target=_blank in XHTML 1.0 strict. This code is a workaround using Javascript:

function externalLinks() { 
 if (!document.getElementsByTagName) return; 
 var anchors = document.getElementsByTagName("a"); 
 for (var i=0; i<anchors.length; i++) { 
   var anchor = anchors[i]; 
   if (anchor.getAttribute("href") && 
       anchor.getAttribute("rel") == "external") 
     anchor.target = "_blank"; 
 } 
} 
window.onload = externalLinks;

A link with attribute rel set to external will open in a new window:

<a rel="external" href="http://castfortwo.blogpspot.com/">cast42</a>

On the Friendfeed realtime embed code, I saw they use jquery to do the job:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3/jquery.js">
<script type="text/javascript"> 
$("body").mousedown(function(e) {
for (var target = e.target; target; target = target.parentNode) {
  if (target.tagName == "A") {
    if ((document.location.href.split("#")[0] + "#") == target.href) {
      return;
    }
    target.target = "_blank";
    return;
    }
}
});
</script> 

The advantage is that all links will open in a new window, except if the refere to an anchor in the page. Hence there's no need to add the attribute rel to each anchor.

Urbain Airship: a webhook based service for Iphone 3.0 Push Notification

This morning I discovered a new recipe on Google App Engine Cookbook to send out Iphone 3.0 Push notifications via Google App Engine. In fact, the recipe makes use of a service called Urban Airship. They provide a free service that allows to send upto 5000 Push notifications a month. For larger volumes, the price seems reasonable. Image now that all content producers become pubsubhubbub (the pseudo realtime webhook based enabled publish/subscribe protocol proposed by two Googlers) enabled and maybe we scratch the surface of the future internet architecture that will drive the realtime web. Who knows?

Pubsubhubbub: a webhook based publish/subscribe

On Friday 9 July 2009, Google engineers Brad Fitzpatrick and Brett Slatkin showed a demo of a new realtime protocol called pubsubhubbub. This new technology could lead to a more reactive, pseudo realtime web. I my humble opinion, this might be much bigger news from Google then the announcement of Chrome OS. Of course, only the latter made a big splash in the mainstream news.

The issue of messaging is a topic addressed already many times in the course of computer science. Common wisdom is that the publish-subscribe pattern is a sound solution to this. The problem is that due to all kinds of practicalities it often regresses into a polling solution. Even with pubhubsubbub, the website that want to display realtime updates has to poll the hub to check if an update has happened. So there's still a nut to crack! In the example of a subscriber, a Javascript polls the hub to check for an update.

If anybody would know an implementation of the client that does not poll, please let me know.

In the demo at Techcrunch, updates made by blogger appeared almost realtime in google reader. Currenlty, not all blogger accounts support pubsubhubbub.

To be ready when your blogger account is updated, do not forget to add your feed to Feedburner and to activate Pingshot:

A pubsubhubbub plugin for wordpress exists and the Drupal community is probably already working on it. Check the overview of publisher code for clients on the pubsubhubbub project page.

Pushed on publish post on monday 20 Juli 2009, at 10:41. Updated at 11u17

Spore Galactic Adventure Released in Europe, today 23 June 2009

This afternoon i received a SMS that my preordered expention kit, called Galactic Adventures for the EA Spore game, was available. I picked it up for 30 euros and installed it. Everything went smooth.

After a succesfull installation, we clicked on the Spore icon in the dock of the mac and prepared for the new fun....that we didn't find. Looking up the documentation and calling the EA helpdesk brought no prevail. I tried a second install. Nothing helped. Until I went looking in the installation directory Spore in the Applications directory. I saw that a new executable was there called "Spore Ruimteavonturen" (dutch for Spore Galactic Adventures). Dubbel click this new executable and of we go:

links to JPEG images in ATOM feed entries must be typed image/jpeg

After some debugging, I found a nasty bug in an ATOM feed that validated by the W3C validator service. I was trying to parse image url's out of an ATOM feed. ATOM 1.0 feeds are defined by the RFC 4287. In paragraph "4.2.7.3. The "type" Attribute" it is defined that the type must conform to the syntax of a MIME media type. The MIME media type is in turn defined by RFC 4288. Media type registrations are listed by the IANA at: http://www.iana.org/assignments/media-types/
If we look up the media subtypes of image/ we find that JPEG images have MIME type image/jpeg . The feed I was parsing had images with MIME type image/jpg instead of image/jpeg as it should have been. Apparently the W3C validator does not validate the type of links. I reported the bug/unconformence. Hopely it will be fixed soon.
A handy list with all MIME types to be used as reference is also provided by w3cschools.

First experiment with Imovie '09

This weekend I bought Ilife '09 for the mac. The software packet has four components : Iphoto, Imovie, Garageband and Iweb. So I played around with Imovie '09 and made a small project with my ski trip movies I shot with an Panasonic DMC-FS3. The movies are 640x480 and come in a .mov file container. The video tracks are encoded in Photo Jpeg and the audio tracks are in 8-bit unsigned mono 8KHz. The resulting movie project is uploaded to Youtube by Imovie's built in share function. Here's the result:

The soundtrack is from World of Goo. The track is "Brave Adventure" by Kyle Gabler.

I really enjoyed using Imovie'09 for this task. I works fast and intuitive. It proves that the choice to abandon the timeline is really working out for consumer oriented movie editing. Time will tell if this will also hold for professional editing. Imovie'09 has an export to Final Cut Pro (Apple professional editing program). Hence an escaped path is on board.