Picture This: Leveraging Big Data to Bolster Social Image Search

Image: Franco Bouly/Flickr

Hidden within each photo is a wealth of information about the objects, people, settings, and environment in which the photo was taken. Decades of research in image recognition have enabled the automated labeling and classification of images with reasonable accuracy. While these research efforts have been promising, and some of them have made their way to search engines such as Google and Bing, they have generally not made a dramatic impact on visual search. Not yet. But another decade or two of research will change this.

Using extremely large training data, researchers have been able to create surprisingly simple and accurate algorithms that can determine whether a photo was taken at night or during the day, whether there is a face, or whether it has a sports theme. However, a lower hanging fruit for visual image search, one which could be immediately applied with dramatic impact, involves the manual tags that are already part of most images on social networks such as Facebook and Flickr.

Now, you might ask, social search is based on these tags already? In a simple way, yes. But hidden within the location of each tag is a significant amount of information about the relationship between people and objects.

In photos, the location of people or objects matters. Best friends tend to stand together for photos, not always, but often. Parents tend to stand close to their kids, again not always, but often enough. The positional information of tags, when viewed from the prism of hundreds or thousands of photos, can tell us a significant amount about people and relationships. And this can be done both easily and efficiently.

Why would this be useful? In one context, photos taken of a city with multiple buildings tagged per photo can result in a rough map of the city. While city maps already exist, family and friend relationship maps may not exist as easily, and at the very least may not be known to search engines.

By analyzing the tag positional information, relationship maps (which I like to call relativity graphs) emerge which tell us that a search for a mom should bring results up from her kids and husband. Not only are these results more relevant, but these related photos can often contain untagged objects that we are searching for (e.g. untagged mom standing by her kids). This will make social search better and more aware of what we are searching for. And, we can do this today.

For relational search to be most effective, however, it needs to be implemented at a network level. While you may have access to your 100 tagged photos, to truly understand all relationships you will need full access to photos of friends (and friends of friends) and their tags. A site like Flickr can build relativity graphs for all users, and use this to improve the relevancy of the search results.

However, the use of spatial tag information in images is one of the most obvious areas for social search improvement given both its low computational overhead and its relatively high impact on understanding social relationships.

I wouldn’t be surprised to see a form of this approach working behind the scenes the next time you are searching for a friend on Flickr or Facebook.