If you’ve been using Google Buzz on a mobile phone recently, you would know that you can choose between two filters to the real-time stream of content:

• Social – choose ‘Following’ to filter the stream based on your social graph, or social ‘circle’ as Google prefers to call it. You will see posts from your friends, and also some public posts from friends-of-friends if the Buzz filtration algorithm thinks you want to flex your social circle.
• Geo – choose ‘Nearby’ to filter the stream based on your location, as detected by your mobile phone. You will see public posts from nearby Buzz users, as a chronological list, or located on a map. Most of the value here comes from the stream being updated in real-time.

Now, with the release of the new Google Buzz API from Google Labs, I have added the real-time stream of geo-located Google Buzz content to GeoMeme, my pet project.

GeoMeme detects real-time geo-located trends, now based on millions of daily posts from various Google Buzz and Twitter and MySpace mobile apps.

GeoMeme can detect, for example, that Justin Bieber beats Lady Gaga in New York City.

If you’re curious about Justin Bieber, or about the amount and contents of geo-located Buzz posts, compared to geo-located Twitter and MySpace posts, check it out and let me know what you think.

The awards were judged by Mike Jones, MySpace’s new Co-President, and Ron Conway, renowned angel investor, and David Glazer, Director of Engineering at Google, and Robert Scoble, tech blogger and uber-geek.

GeoMeme uses the new MySpace Real-Time Stream API to tap into the flood of geo-located updates being posted by MySpace users all around the world.

Activity Streams from MySpace are mashed up with tweets from a number of mobile Twitter apps, and located onto a Google Map. Local trends are identified using semantic analysis services from Yahoo.

For example, GeoMeme knows that Rihanna beats Lady Gaga in New York and that Avatar beats Hurt Locker in Los Angeles.

Beyond the discovery and measurement of real-time local trends, GeoMeme also provides a unique view into local activity streams, as a way to discover new like-minded and nearby friends. You can then buy the t-shirt (really, you can!) to share your trends with your friends.

GeoMeme is a lightning fast web app, and is also available on iPhone as a mobile web app, optimized for mobile using Google Maps v3 API. GeoMeme is built on Google App Engine for massive scalability.

And congratulations to the other winners of the Challenge:

• Best New MySpace App: Paradise Paintball
• Most innovative use of the Real-Time Stream API: GeoMeme
• Most innovative use of the Open Search API: Social Mention
• Most innovative use of the Photos API: Browser Not Included
• Most innovative MySpace Integration on Mobile: iSkoot

GeoMeme uses the new Real-Time Stream API from MySpace to tap into the flood of geo-located updates being posted by MySpace users all around the world.

MySpace content is mashed up with tweets from a number of mobile Twitter apps, and located onto a Google Map. Local trends are identified using semantic analysis services from Yahoo.

A couple of example GeoMemes generated by all this real-time geo-located content: Rihanna beats Lady Gaga in New York, and Avatar beats Hurt Locker in Los Angeles.

There are now some quite interesting public datasets available, and the developer community is hard at work turning this data into useful APIs, and building innovative applications to consume the data.

Some of the notable apps to emerge from OpenAustralia include:

• It’s Buggered, Mate – from the Canberra Hackfest, a geo app to crowdsource the reporting of broken public infrastructure.
• Suburb Matchmaker – the winner of the Sydney Hackfest, a tool to help you find your ideal suburb to live in.
• FridgeMate – currently winning the MashupAustralia contest and only a couple of days away from the $10,000 prize. FridgeMate lets you assemble a map of local public amenities to stick on your fridge door. My advice to the Creative Possums behind FridgeMate would be to look at using the Zazzle API so people could buy the actual fridge magnet.

My own presentation focussed on some mobile, geo and social technologies to create location-aware mobile mashups to share OpenGov content with friends on Twitter, friends on Facebook, and *real* friends on a t-shirt. Here’s the deck:

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that ‘love’ beats ‘hate’ in Manhattan:

GeoMeme is a desktop web application and also a location-aware mobile web app for iPhone and Android phones.

Implementing the mobile version of GeoMeme as a web app has some advantages and disadvantages, compared to building native iPhone &/or Android applications.

Native apps are great because they currently offer the deepest integration to the full capability of the phone, for example using device APIs to access Contacts, the Camera Roll, an Accelerometer, or the GPS chip. For some applications, this deep device integration is essential and so a native application is beneficial.

On the other hand, emerging HTML5-based mobile browsers are aiming to standardise integration to such device APIs, starting with Geolocation APIs; meaning that location-aware mobile web apps are now becoming viable. Aligned with this development is the new version of the Google Maps API. v3 has been greatly simplified since v2, and is now optimized for use on mobile phones. Less is more.

The deciding factor for me choosing to build a mobile web app for GeoMeme rather than a native app was development speed. A mobile web app enjoys far greater code re-use from the desktop web version, and it is possible to push regular updates and improvements to users, without having to wait for appstore approval or for users to upgrade.

I believe this need for development speed is common among a good proportion of mobile apps that are still in ‘rapid iteration’ or ‘release early, release often’ mode, so this post is intended to share some of the techniques used in GeoMeme with developers wanting to build their own location-aware mobile web apps.

Let’s build an example location-aware mobile web app called ‘Here I Am!’, for the photographically challenged. The app will present some local photographs (from Panoramio) which can be shared with friends on Twitter or Facebook.

Where on earth is that mobile phone..?

The first job of a location-aware mobile app is to work out where on earth the mobile phone currently is. Unfortunately, at the time of writing, there is still no universally reliable and accurate solution for a mobile web app to detect the location of the mobile phone it is running on. However the following partial solutions can be combined to good effect:

a) Google Loader

Load the Google Maps v3 API using Google Loader:

<script type="text/javascript" src="http://www.google.com/jsapi?key=YOUR_API_KEY&autoload=%7Bmodules%3A%5B%7Bname%3A%22maps%22%2Cversion%3A3%2Cother_params%3A%22sensor%3Dtrue%22%7D%5D%7D"></script>
<script type="text/javascript">
google.setOnLoadCallback(function() {
    // initialize the mobile map...
});
</script>

Google Loader requires developers to sign up for an API key, however the advantage of this approach is that the approximate location of the user is revealed, based on network IP address.

In the case of a mobile phone user, this network IP address often refers to the mobile operator’s internet gateway, which can be shared between a large number of subscribers spread over an entire country. This technique becomes more accurate for mobile phones which are connected to the internet via wi-fi rather than GPRS, or becomes less accurate for some phones (e.g. Blackberry) which can connect to the internet via international proxy servers.

Generally, this technique can be successful in working out which country the user is in, but cannot be assumed to be any more accurate than that.

Some example javascript to locate a map based on Google Loader:

if (typeof(google.loader.ClientLocation) != 'undefined') {
	var lat = google.loader.ClientLocation.latitude;
	var lng = google.loader.ClientLocation.longitude;
	var position = new google.maps.LatLng(lat, lng);
	map.setOptions({
		center: position,
		zoom: 10
	});
}

b) W3C Geolocation API

The World Wide Web Consortium (W3C) has been working since 2008 to standardise web geolocation APIs.

Mobile Safari on iPhone (since OS3.0) already supports the emerging W3C standard, to expose location information derived from its GPS chip. The new Chrome mobile browser on Android (from 2.0 Eclair) will also support this standard. For example:

if (typeof(navigator.geolocation) != 'undefined') {
	navigator.geolocation.getCurrentPosition(function(position) {
		var lat = position.coords.latitude;
		var lng = position.coords.longitude;
		var position = new google.maps.LatLng(lat, lng);
		map.setOptions({
			center: position,
			zoom: 15
		});
	});
}

The W3C Geolocation API looks likely to feature as part of HTML5 (the next major revision of HTML), and will also be supported by the next generation of Webkit-based mobile browsers, so expect this technique to start working on an increasing proportion of new mobile phones.

c) Google Gears for Mobile

Gears adds a number of HTML5 features, including a Geolocation API, to legacy web browsers. Although it is possible for Windows IE Mobile and Mobile Opera browsers to become location-aware in this way, the bigger opportunity here is that Android phones (from version 1.0 through 1.6) come with Gears already pre-installed. This means that we can detect the location of these Android phones with the following javascript:

if (typeof(google.gears) != 'undefined') {
	var geo = google.gears.factory.create('beta.geolocation');
	geo.getCurrentPosition(function(position) {
		var lat = position.latitude;
		var lng = position.longitude;
		var position = new google.maps.LatLng(lat, lng);
		map.setOptions({
			center: position,
			zoom: 15
		});
	});
}

Google Gears for Mobile works out the location of a mobile phone from the GPS chip in the phone, or by crowdsourcing the location of the cell-tower to which the mobile phone is connected.

… and fetch me some location-based content, fast!

Once your mobile web app is location-aware, the next thing to do is to retrieve some relevant content, based on current location. GeoMeme retrieves local tweets using the Twitter API, but for ‘Here I Am!’ we’ll retrieve local photos from Panoramio (see http://www.panoramio.com/api/ for details of the API).

Google Maps v3 API includes a convenient new ‘idle’ event which is fired when the map becomes idle after panning or zooming, or after automatic geo-location using the techniques above. By listening for this event, we can fetch the most popular local photos from Panoramio whenever the map is moved. For example:

google.maps.event.addListener(map, 'idle', function() {
    loadPhotos();
});

function loadPhotos() {
    var url = 'http://www.panoramio.com/map/get_panoramas.php?order=popularity&set=public&from=0&to=20&size=mini_square&callback=addPhotos';
    var bounds = map.getBounds();
    url += '&minx=' + bounds.getSouthWest().lng().toFixed(6) + '&miny=' + bounds.getSouthWest().lat().toFixed(6);
    url += '&maxx=' + bounds.getNorthEast().lng().toFixed(6) + '&maxy=' + bounds.getNorthEast().lat().toFixed(6);
    url += '&ts=' + new Date().getTime(); // prevent caching

    // use JSONP to retrieve photo data
    // and trigger a callback to addPhotos()
    var script = document.createElement("script");
    script.setAttribute("src", url);
    script.setAttribute("type", "text/javascript");
    document.body.appendChild(script);
}

The above code sends the resulting JSON data to the addPhotos() callback function, which handles the display of the photos on the map using Icons, another new feature of Google Maps v3 API.

For speed, we also implement a performance optimization here. Existing Icons are re-used, rather than simply removing and adding new Icons each time the map is relocated:

var markers = {};

function addPhotos(data) {
	var new_markers = {};

	for (var i = 0; i < data.photos.length; i++) {
		var photo = data.photos[i];

		// for speed and to reduce flicker,
		// reuse existing markers rather than removing and re-adding
		if (photo.photo_id in markers) {
			new_markers[photo.photo_id] = markers[photo.photo_id];
		} else {
			// create new marker
			marker = new google.maps.Marker({
				position: new google.maps.LatLng(photo.latitude, photo.longitude),
				icon: photo.photo_file_url,
				map: map
			});
			new_markers[photo.photo_id] = marker;
		}
	}

	// remove old markers
	for (var photo_id in markers) {
		if (!(photo_id in new_markers)) {
			markers[photo_id].setMap(null);
			delete markers[photo_id];
		}
	}

	markers = new_markers;
}

‘Here I Am!’ – a simple location-aware mobile web application

The final part of ‘Here I Am!’ is a feature to share one of the Panoramio photos to friends on Twitter or Facebook. This is done using ordinary web links to Twitter and Facebook share pages. No complex platform integration is required for web apps to do this because, as they say, “the web is the platform.” Simply click on a photo on the map and follow the prompts.

Visit http://j.mp/lbsdemo on your mobile phone to try out ‘Here I Am!’

The full source code of ‘Here I Am!’ (weighing in at just under 8Kb) can be located here.

The app has been tested to work on iPhone, iPod Touch, Android, and Symbian Series 60 (which remains – for now at least – the dominant device family for mobile data consumption).

I hope you enjoy using some of these ideas in your own location-aware mobile web apps.

This post is one of a series that aims to share some of the technology innovation that can be found in GeoMeme. Other posts cover topics such as using Google App Engine for scalable and fast hosting of your location-based content, and fast map re-location using Google Static Maps v2.

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that ‘District 9′ beats ‘Inglorious Basterds’ in Manhattan.

As well as offering users the normal pan and zoom controls to move the map around, GeoMeme also introduces an innovative geo-autocomplete control which is powered by the geocoder service from Google Maps v3 API and the new Static Maps v2 API.

This blog post shares some details of how the geo-autocomplete control works, and offers some code so you can build your own geo-autocomplete controls.

1. Based on a partial location typed by the user, obtain a list of possible matching locations:

If the user has already typed ‘San’ into a form field, we can obtain a list of possible matching locations by passing this partial location to the geocoder service from Google Maps v3 API, as follows:

var partial_location = document.getElementById("location").value;
var geocoder = new google.maps.Geocoder();

geocoder.geocode({'address': partial_location}, function(results, status) {
    if (status == google.maps.GeocoderStatus.OK) {
        document.getElementById("results").innerHTML = '';
        for (var i = 0; i < results.length; i++) {
            showResult(results[i]);
        }
    } else {
        document.getElementById("results").innerHTML = 'Geocode was not successful for the following reason: ' + status;
    }
});

function showResult(result) {
    document.getElementById("results").innerHTML += 'It could be: ' + result.formatted_address + '<br/>';
}

2. Decorate these possible matching locations with a Static Maps thumbnail to help the user choose the right one:

The geocoder returns more than just a formatted address. The full response is a JSON object of the following form:

results[]: {
    types[]: google.maps.GeocoderLocationType,
    formatted_address: String,
    address_components[]: {
        short_name: String,
        long_name: String,
        types[]: String
    },
    geometry: {
        location: LatLng,
        location_type: String,
        viewport: LatLngBounds,
        bounds: LatLngBounds
    }
}

geometry.viewport represents the recommended viewport for each returned result, from which we can derive the required Static Maps URL, as follows:

function showResult(result) {
    var src = 'http://maps.google.com/maps/api/staticmap';

    // the visible parameter specifies one or more locations
    // to show on the map. so we need to specify the south-west
    // and north-east corners of geometry.viewport
    src += '?visible=' + result.geometry.viewport.getSouthWest().toUrlValue() + '|' + result.geometry.viewport.getNorthEast().toUrlValue();

    // other parameters as per http://code.google.com/apis/maps/documentation/staticmaps/#URL_Parameters
    src += '&size=100x100&maptype=terrain&key=YOUR_API_KEY&sensor=false';

    document.getElementById("results").innerHTML += '<img src="' + src + '" /> ' + result.formatted_address + '<br/>';
}

Working out the required Static Maps URL is now a much simpler task with v2 of the Static Maps API, because there is no longer any need to manually calculate a zoom level or latitude / longitude span values.

You can see this working on GeoMeme by clicking on the location control at the top of the screen and entering a partial location. When you choose from the list of possible matches, the main map is moved to that location, providing a refreshingly quick way for users to re-position the map.

You can also download a hello world example here, built as a jQuery plugin to work with a modified version of the excellent Autocomplete plugin:

Using this geo-autocomplete plugin is recommended because it includes a number of features which protect the geocoder service from being hit too often. Including; waiting for keystrokes to stop before sending a request to the geocoder, waiting for a minimum number of characters to by typed, and caching geocoder responses to avoid duplicate requests. Options are also available to set the size and maptype of the Static Map thumbnails.

Update: geo-autocomplete has been rebuilt upon the jQuery UI framework, as a UI Widget rather than a jQuery plugin, with more powerful options for customization.

This post is one of a series that aims to share some of the technology innovation that can be found in GeoMeme. Other posts cover topics such as location-aware mobile web apps using Google Maps v3, and scalable geo apps using Google App Engine.

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that beats in San Francisco:

A large amount of geo-data is generated by GeoMeme, and so arises a need shared by many geo apps: scalable, fast, and accurate spatial queries, used to select a subset of geo-data for display as markers on a map, or on Google Earth.

Google App Engine

Google App Engine is an obvious choice for hosting your geo app. The App Engine datastore is built on top of Google’s BigTable technology which scales very well, and is optimized for fast data retrieval. And it doesn’t cost the earth like some traditional GIS database solutions.

Inequality constraint

If you are coming from a background of relational databases, you might think the solution here would be to store the latitude and longitude of all your markers in a database table, and do a simple query to retrieve only those contained within the bounds of the map.

However, the flipside of being optimized for fast data retrieval is that BigTable only allows inequality filters on a single dimension, to avoid the burden of full table scans. For example, the following form of spatial query is not supported because it specifies inequality filters on both latitude and longitude dimensions:

SELECT latitude, longitude, title FROM myMarkers
WHERE latitude >= :south AND latitude <= :north
  AND longitude >= :west AND longitude <= :east

>>> BadFilterError: invalid filter: Only one property per query may have inequality filters

Geohash to the rescue

Fortunately, there is an answer to this: geohash, the geocoding system invented by Gustavo Niemeyer.

My suspicion is that Niemeyer has travelled back in time after working out how to collapse two-dimensional space into a single dimension, but you might want to read the Wikipedia explanation of the algorithm instead.

An example: the location of the Sydney Opera House can be specified in two dimensions as {latitude: -33.858, longitude: 151.215}, or in a single dimension as {geohash: r3gx2ur29zzg7}.

Schuyler Erle has written an open source geohash python module, which enables the following form of spatial query on Google App Engine, because the inequality filter is specified only on a single dimension:

sw_geohash = Geohash((west, south))
ne_geohash = Geohash((east, north))

SELECT latitude, longitude, title FROM myMarkers
WHERE geohash >= :sw_geohash AND geohash <= :ne_geohash

Don’t hash me, coz I’m close to the edge

However…! We’re not done yet, because an artifact of the geohash algorithm is that queries like the one above can often return rogue markers which are outside of the required bounds, when the map spans what we shall call a geohash “faultline”.

This problem is particularly evident near the equator and the Greenwich Meridian, which are the biggest faultlines, but there are actually faultlines all over the place at every zoom level.

Faultline correction

GeoMeme solves this problem using “faultline correction”, an approach that I would like to share here:

• If a spatial query is vulnerable to faultlines, it is split into multiple sub-queries that do not cross the faultline.
• Sub-query limits are approximately weighted according to their relative size.
• Sub-queries are executed in parallel, taking advantage of BigTable’s distributed goodness, and the results combined, so all this happens very fast.
• Even though the sub-queries are executed in parallel without any significant impact on user experience, App Engine CPU costs can increase to around 2x the CPU cost of a single less accurate query. Sub-query results are memcached to reduce this CPU overhead.

Here’s a demo showing the effect of geohash faultlines, and the relative accuracy of spatial queries with or without faultline correction. Rogue markers are shown in the area surrounding the map. Switch between Correction: off / on / double and compare the accuracy.

Note: “double” correction is an advanced option which splits the sub-queries into sub-sub-queries so that they do not cross any faultlines either. This can further increase accuracy, but with further CPU cost (up to 8x the CPU cost of a single less accurate query).

All the source code can be found here, including ffGeoSearch, a python module to handle faultline-friendly geo search, if you want to use this technique on your own geo app.

Enjoy!

This post is one of a series that aims to share some of the technology innovation that can be found in GeoMeme. Other posts cover topics such as fast map re-location using Google Static Maps v2, and location-aware mobile web apps using Google Maps v3.

I am pleased to announce the launch of GeoMeme, the fun way to measure and share real-time local twitter trends.

I got thinking about this when a recent Los Angeles earthquake was being measured in tweets per second rather than using the Richter Scale.

Then came the Magnitwude Calculator as a standard way to measure the magnitude of Twitter trends.

[Then came twotspot.com but that domain name was just too damn rude, so it was quickly renamed to GeoMeme.]

What does GeoMeme do?

GeoMeme measures real-time local twitter trends.

Tweeps are located on the map using public data from a number of iPhone twitter apps. When twitter launches its geolocation API, that will be used to locate even more people on the map.

GeoMeme measures and compares how many people on the map are tweeting about each of your two search terms:

The ‘magnitude’ of each search term is equal to the number of unique people tweeting per hour per square kilometer, so it increases when more people are tweeting in a smaller area.

Example: if 100 different people in an area of 10km² have tweeted about ‘love’ in the last 2 hours, the magnitude is 5.0 (100 divided by 10 divided by 2).

So you can search for ‘love’ and ‘hate’ and GeoMeme works out which one “beats” the other with the higher magnitude.

The default search terms are and smiley faces which provides a good measure of local happiness, as an example.

Can I use my iPhone?

Sure, or your iPod Touch. Here’s the screenshot:

Give me an example!

Thanks to some early coverage on The Register, Mashable, and Google Maps Mania, and winning Mashup of the Day on ProgrammableWeb, we’re off to a flying start. I’m glad GeoMeme is hosted on Google App Engine for scalability.

Here’s a selection of the most popular GeoMemes so far:

• beats in New York City
• Mega Shark beats Giant Octopus in LA
• Snow Leopard beats Windows 7 in Cupertino
• bridge beats swimming in San Francisco Bay
• wtf beats ftw in Washington

How does it all work?

I will leave the details of how it all works to another post, stay tuned for that.