Nov
10
2009

Scalable, fast, accurate geo apps using Google App Engine + geohash + faultline correction

GeoMeme is a web app (and also a mobile web app for iPhone and Android) that I recently developed as a pet project. It measures real-time local twitter trends.

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that :) beats :( in San Francisco:

A large amount of geo-data is generated by GeoMeme, and so arises a need shared by many geo apps: scalable, fast, and accurate spatial queries, used to select a subset of geo-data for display as markers on a map, or on Google Earth.

:)Google App Engine

Google App Engine is an obvious choice for hosting your geo app. The App Engine datastore is built on top of Google’s BigTable technology which scales very well, and is optimized for fast data retrieval. And it doesn’t cost the earth like some traditional GIS database solutions.

:( Inequality constraint

If you are coming from a background of relational databases, you might think the solution here would be to store the latitude and longitude of all your markers in a database table, and do a simple query to retrieve only those contained within the bounds of the map.

However, the flipside of being optimized for fast data retrieval is that BigTable only allows inequality filters on a single dimension, to avoid the burden of full table scans. For example, the following form of spatial query is not supported because it specifies inequality filters on both latitude and longitude dimensions:

(more…)

Written by bob in: everything | Tags: , , ,
Nov
10
2009

Fast map re-location using Google Static Maps v2 + geocoder

GeoMeme is a pet project of mine. It’s a web app, and also a mobile web app for iPhone and Android, that measures real-time local twitter trends.

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that ‘District 9′ beats ‘Inglorious Basterds’ in Manhattan.

As well as offering users the normal pan and zoom controls to move the map around, GeoMeme also introduces an innovative geo-autocomplete control which is powered by the geocoder service from Google Maps v3 API and the new Static Maps v2 API.

This blog post shares some details of how the geo-autocomplete control works, and offers some code so you can build your own geo-autocomplete controls.

1. Based on a partial location typed by the user, obtain a list of possible matching locations:

If the user has already typed ‘San’ into a form field, we can obtain a list of possible matching locations by passing this partial location to the geocoder service from Google Maps v3 API, as follows:

(more…)

Feb
04
2010

Fresh From Twitter

the aircon is broken in the Xumii Sydney office. 27.79 degrees celcius and 1 Engineer asleep!

mobile browser apps winning over native apps for Mobile Social http://j.mp/96h3Nf but native apps still best for Mobile Gaming /via @rww

interesting Aardvark paper on social search http://j.mp/c9nFTh @vark methinks social graph + ‘PeopleRank’ also helps the Library paradigm

i wonder if this GoogKe mobile phone spotted in China includes any spyware? or any software? http://twitpic.com/10ypfp /via @stevesaul
twitpic photo

meta iPad comedy http://bit.ly/cLDyvA (iPad vs. Stone) vs. (iPad vs. Etch-a-Sketch) via @parislemon

awesome RT @renailemay Sydney Powerhouse Museum holds retro gaming weekend: http://bit.ly/akeOVY

GeoMeme finds Twitter Local Trends for *any* location, a whole country or a street corner http://www.geome.me #biased /cc @eladgil

RT @raffi WOEIDs in Twitter’s Local Trends http://post.ly/KydM

Twitter Local Trends is promising http://j.mp/dwbPqf does the small choice of locations reflect a non-scalable manual moderation process?

Avatar is #1 in box office dollars, but only #26 measured by number of (expensive) tickets sold http://bit.ly/cpSove /via @GuyKawasaki

my last iPad tweet, iPromise: does anybody know the iPad User Agent so we can mimic it and see which sites adapt the fastest?

iPad 3.2 SDK has a ‘Take Photo’ button but the iPad has no cam… yet! http://j.mp/9THYV0 please can we get 1 cam for skype + 1 cam for AR?

Apple threw another punch at mobile operators today: the new iPad 3.2 SDK now supports VoIP calls over 3G http://j.mp/dvisNc /via @snaglepus

instead of watching the Steve Jobs keynote, try the Hitler response instead http://youtu.be/lQnT0zp8Ya4 [video]

Powered by Fresh From
Written by bob hitching in: Twitter | Tags: ,
Jan
20
2010

Fresh From Flickr

just as we are leaving China we finally catch a rare glimpse of the elusive panda
Image

Written by Bob Hitching in: Flickr | Tags: ,
Dec
10
2009

Fresh From Twitter

new Facebook privacy paves the way for geo-Facebook http://bit.ly/5MqHXa “… uses your location in order to provide … recommendations”

hmmm … share my Facebook posts with Everyone ? … add geo-location to my posts ! … share my location with everyone :) #geoFacebook

likes an API with personality. meet the new MySpace real-time stream, the ‘Flood API’ http://bit.ly/7ZooBm one up on Twitter’s Firehose?

i have nose burn from eating too many wasabi peas

new mobile Twitter site is great dogfooding http://j.mp/4Ge8cv @Leland please add a mobile oAuth page that works on an average phone

Gmail Goggles was Good http://bit.ly/61XDvd but Google Goggles is Great http://www.youtube.com/watch?v=Hhgfz0zPmH4 [video]

Written by Bob Hitching in: Twitter | Tags: ,
Nov
14
2009

Murdoch should worry less about the Googlebot and more about social media

I remember in January 2000, old media mogul Rupert Murdoch said he was not going to waste his money buying any ‘dotcom’ upstarts. The very next day, AOL bought Time Warner. Not the other way around!

Murdoch had apparently failed to grasp the significance of the interwebs.

However, ten years later Time Warner has regained its mojo and is now trying to offload a spent and jaded AOL. Did Murdoch get it wrong ten years ago, or did it simply take a whole decade for him to be proven right?

In 2009, the mob is rushing once again to the conclusion that Murdoch is losing his marbles, planning to charge for his online content and blocking the Googlebot from stealing it.

Personally I believe that Murdoch should worry less about the Googlebot, and more about how social media is turning his industry on its head.

The problem is that all of those dotcom upstarts have brought us information overload. There has been an exponential increase in the amount of information and content available to us, way beyond the capacity of the human brain to process.

The solution is social media, which empowers us to easily share the content we care about with our friends and contacts, and adds valuable metadata to that shared content, such as Likes or Retweet counts. This metadata helps us filter the signal from the noise, so that we can focus on just the best quality content from our trusted circle of friends.

This works great for movie reviews. People have always listened to the advice of friends when it comes to choosing what movie to watch. Social media simply provides an efficient and scalable way to do this.

The best example of this social filter is currently FriendFeed, although we can expect to soon see something equally impressive on Facebook. Twitter Search could do this even better if only it were possible to search the entire tweet history of just your friends, or a chosen social distance into your social graph, rather then merely search 7 days of the public timeline. I am hoping that the Google Social Search Experiment will enable this sort of social filter when Google completes its Twitter integration.

Back to Mr. Murdoch… Social media also works for the filtering of news content, however it’s more tricky than movie reviews because there is a need for trustworthy fact rather than mere opinion. This is why Eric Schmidt believes that figuring out how to rank real-time social content, perhaps based on a reliable measure of reputation and authority, is “the great challenge of the age“. It also explains why Twitter’s Retweet feature does not allow the original tweet to be modified, because this makes the Retweet count a more reliable indicator of authority.

So my advice to Rupert Murdoch and other media companies struggling with this; worry less about the Googlebot and more about social media. Focus on improving the quality of your content, so that people share it with their friends.

And if your own social media strategy is not delivering any tangible benefits, try moving it from your Marketing department to your Customer Service department. Use social media to listen more carefully to the needs of your customers, so you can improve the quality of your content to the point where a paid online content model becomes viable.

If Marketing and Customer Service argue about who owns the customer relationship, remind them both that thanks to social media it’s actually the customer who owns and controls the relationship with your business. Not the other way around!

Nov
11
2009

OpenAustralia Hackfest: ‘Mobile + Geo + Social’ slides

I popped into the OpenAustralia Hackfest at the weekend to learn and talk about some of the latest developments in the Gov2.0 revolution.

There are now some quite interesting public datasets available, and the developer community is hard at work turning this data into useful APIs, and building innovative applications to consume the data.

Some of the notable apps to emerge from OpenAustralia include:

  • It’s Buggered, Mate – from the Canberra Hackfest, a geo app to crowdsource the reporting of broken public infrastructure.
  • Suburb Matchmaker – the winner of the Sydney Hackfest, a tool to help you find your ideal suburb to live in.
  • FridgeMate – currently winning the MashupAustralia contest and only a couple of days away from the $10,000 prize. FridgeMate lets you assemble a map of local public amenities to stick on your fridge door. My advice to the Creative Possums behind FridgeMate would be to look at using the Zazzle API so people could buy the actual fridge magnet.

My own presentation focussed on some mobile, geo and social technologies to create location-aware mobile mashups to share OpenGov content with friends on Twitter, friends on Facebook, and *real* friends on a t-shirt. Here’s the deck:

Written by bob in: everything | Tags: , , , , ,
Nov
10
2009

Location-aware mobile web apps using Google Maps v3 + geolocation

When hiring Engineers, I always look for evidence of pet projects, so recently I thought it was fair to create one of my own: GeoMeme, the fun way to measure and share real-time local twitter trends.

Visitors to GeoMeme choose a location on the map, and two search terms to compare. GeoMeme then measures and compares the number of matching tweets within the bounds of the map, based on public data from a number of mobile twitter apps.

As an example, GeoMeme can work out that ‘love’ beats ‘hate’ in Manhattan:

GeoMeme is a desktop web application and also a location-aware mobile web app for iPhone and Android phones.

Implementing the mobile version of GeoMeme as a web app has some advantages and disadvantages, compared to building native iPhone &/or Android applications.

Native apps are great because they currently offer the deepest integration to the full capability of the phone, for example using device APIs to access Contacts, the Camera Roll, an Accelerometer, or the GPS chip. For some applications, this deep device integration is essential and so a native application is beneficial.

On the other hand, emerging HTML5-based mobile browsers are aiming to standardise integration to such device APIs, starting with Geolocation APIs; meaning that location-aware mobile web apps are now becoming viable. Aligned with this development is the new version of the Google Maps API. v3 has been greatly simplified since v2, and is now optimized for use on mobile phones. Less is more.

The deciding factor for me choosing to build a mobile web app for GeoMeme rather than a native app was development speed. A mobile web app enjoys far greater code re-use from the desktop web version, and it is possible to push regular updates and improvements to users, without having to wait for appstore approval or for users to upgrade.

I believe this need for development speed is common among a good proportion of mobile apps that are still in ‘rapid iteration’ or ‘release early, release often’ mode, so this post is intended to share some of the techniques used in GeoMeme with developers wanting to build their own location-aware mobile web apps.

Let’s build an example location-aware mobile web app called ‘Here I Am!’, for the photographically challenged. The app will present some local photographs (from Panoramio) which can be shared with friends on Twitter or Facebook.

Where on earth is that mobile phone..?

The first job of a location-aware mobile app is to work out where on earth the mobile phone currently is. Unfortunately, at the time of writing, there is still no universally reliable and accurate solution for a mobile web app to detect the location of the mobile phone it is running on. However the following partial solutions can be combined to good effect:

(more…)

Sep
17
2009

Xumii acquired by Myriad Group

Exciting news: Xumii has been acquired by Myriad Group (SIX:MYRN). We are now part of Europe’s largest mobile technology business with software in more than 2 billion phones.

This is great for Xumii as it means we can take our mobile social networking platform to the next level. And it’s a great win for Australian mobile technology.

Here’s the press release and some of the blogosphere coverage: GigaOM and TechCrunch and VentureBeat.

Written by bob in: everything | Tags: , ,
Sep
13
2009

GeoMeme: measure and share real-time local twitter trends

I am pleased to announce the launch of GeoMeme, the fun way to measure and share real-time local twitter trends.

I got thinking about this when a recent Los Angeles earthquake was being measured in tweets per second rather than using the Richter Scale.

Then came the Magnitwude Calculator as a standard way to measure the magnitude of Twitter trends.

[Then came twotspot.com but that domain name was just too damn rude, so it was quickly renamed to GeoMeme.]

What does GeoMeme do?

GeoMeme measures real-time local twitter trends.

Tweeps are located on the map using public data from a number of iPhone twitter apps. When twitter launches its geolocation API, that will be used to locate even more people on the map.

GeoMeme measures and compares how many people on the map are tweeting about each of your two search terms:

The ‘magnitude’ of each search term is equal to the number of unique people tweeting per hour per square kilometer, so it increases when more people are tweeting in a smaller area.

Example: if 100 different people in an area of 10km2 have tweeted about ‘love’ in the last 2 hours, the magnitude is 5.0 (100 divided by 10 divided by 2).

So you can search for ‘love’ and ‘hate’ and GeoMeme works out which one “beats” the other with the higher magnitude.

The default search terms are :) and :( smiley faces which provides a good measure of local happiness, as an example.

Can I use my iPhone?

Sure, or your iPod Touch. Here’s the screenshot:

Give me an example!

Thanks to some early coverage on The Register, Mashable, and Google Maps Mania, and winning Mashup of the Day on ProgrammableWeb, we’re off to a flying start. I’m glad GeoMeme is hosted on Google App Engine for scalability.

Here’s a selection of the most popular GeoMemes so far:

How does it all work?

I will leave the details of how it all works to another post, stay tuned for that.

Written by bob in: everything | Tags: , , , , , ,
Jul
20
2009

How to measure Twitter trending topics

2009 has already seen some big Twitter moments, including Michael Jackson’s death and memorial service, #iranelection, Oprah’s mainstreaming, and the race between @aplusk and @cnn to reach 1 million followers.

But how can we objectively measure and compare the scale of such things?

A little while ago I got thinking about this when a Los Angeles earthquake was being measured in tweets per second rather than using the Richter Scale.

And now here is my solution, the Magnitwude Calculator, which measures the current magnitude of tweets on any topic within any location.

Please have a fiddle. Type in a search term or select from the autocomplete list of currently trending topics, move the map around, and tell me what you think:

You can link directly to the Magnitwude Calculator at http://hitching.net/magnitwude

Jun
04
2009

Wave goodbye to spam

Google Wave combines the best of email, instant messaging and real-time collaborative editing into a new form of online communication.

The email paradigm of ’send and receive’ is replaced with a model of hosted conversations, in which “people can communicate and work together with richly formatted text, photos, videos, maps, and more.”

Wave is refreshingly ambitious. In years to come, I hope we will be waving nostalgically about email as “something that my parents used to do.”

This blog post describes an idea built upon Google Wave that could also turn email *spam* into the stuff of nostalgia.

Spam sent by people you don’t know is a real pain in the inbox. But simply ignoring emails from people you don’t know is not the answer. (Otherwise I would never have learnt about my recent win on the Nigerian lottery. Just kidding.)

So how might Google Wave help us to finally wave goodbye to spam?

  • assume that developers will build robots to connect my wave account with the rest of my social graph (either that &/or Google plugs in Friend Connect)
  • if someone (or a spambot) outside of my social graph invites me to a wave, my wave server responds to that invite with a reCAPTCHA challenge (try one out below)
  • (more…)

Powered by WordPress. Theme: TheBuckmaker.