June 16, 2009

Social recommendation reaching long-promised potential

Content on the Internet has been exploding exponentially. YouTube, the largest video site on the Internet, adds ten hours of new video every single minute. By some measure, 10M new pages are added on the World Wide Web every day. The world of blogs has also grown tremendously, with Technorati, the popular blog search engine, tracking over 120M blogs worldwide (excluding millions that are not tracked in China, country with the world's largest Internet population).

This makes discovery of new content a huge challenge for both users (how to discover) and publishers (how to get discovered). I'm going to focus only on the user challenge in this post.

As the content on the Internet continues to multiply dramatically, the ability for users to find the content that may exist out there, which they may like and therefore want to consume, poses one of the most significant problems on the Web. Search engines have become a popular tool for users to find what they're looking for, but Search requires users to express their intent first. Since you don't know what you don't know, discovery of new content that you are unaware of, but may like, necessitated the development of recommendation engines, which proactively suggest what content users may like without much, if any, explicit action on their part.

Recommendation engines broadly fall under three categories:

1) Editorial recommendation: This is the most basic form of recommendation method whereby the editor/publisher picks what to highlight on its site amongst all available content - e.g., "featured video of the day", "featured story of the week," etc. This is a one-size-fits-all approach, where the recommendation focuses on publisher's goals and/or the average consumption behavior/demand amongst all visitors to the site, and is not customized for each individual visitor.

2) Customized recommendation based on individual's behavior: This method involves analyzing each user's content consumption pattern and using technology to recommend similar content to that user. This is the most difficult approach technically. Complex algorithms are needed to 1) "read" and categorize the repository of all available content on one hand, 2) track, capture and analyze each user's consumption patterns, and then 3) match those patterns against the categorized repository to make recommendations which are uniquely customized for each user.

Examples of this approach include the Pandora Internet radio that provides customized radio channels to users based on the songs users initially select to hear. Pandora's Music Genome Project utilizes 400 different characteristics to classify all available songs into categories that are leveraged by Pandora to understand a user's music taste and recommend him/her relevant songs. Cinematch, Netflix's movie rental recommendation system, is another such example.

"Reading" the content repository and categorizing them, say, in logical genres, is the most difficult aspect of this approach. Getting the recommendation correct with a high level of accuracy on a consistent basis is extremely hard. Since the recommendation engine works in the background, employing complex algorithms and other behavioral science factors, users expect the "black box" to make correct recommendation every single time. Getting it 90% correct, though impressive, may still not win user's loyalty and trust. If I hate chick flicks, make one such movie recommendation to me, and that's the last time I'd trust the technology behind the recommendation engine.

Netflix, recognizing the complexity of this recommendation approach, has an on-going contest, launched in October 2006, that will award $1M to the person that can improve the accuracy of its already-impressive Cinematch movie recommendations by 10%.

3) Social recommendation: This approach makes content recommendation to a user based on usage patterns of other users instead of his/her own consumption pattern. It's essentially a popularity contest, where popularity amongst a set of users is measured to make recommendations.

Social recommendation comes in two flavors. The set of users used to measure popularity can be either a limited set - user's friends (social graph), members belonging to a particular group or affiliation, etc - or the general set of all users consuming content on the site (e.g., most viewed, highest rated, etc).

Of the two, social recommendation based on preferences of my social graph - people I trust - holds the biggest promise amongst all approaches currently being utilized to make content recommendations. Reasons: 1) the approach requires rather simple technology, 2) users expectations are managed - not all recommendations need to be on the mark because everyone has friends whose tastes are different from their own - and 3) the approach simply works, because it follows a long and well established norm in the real world. By some measures, more than 30% of new content consumed by people is based on recommendation from someone they know and trust. Web 2.0 tools and changing user behavior where more and more people are sharing and contributing more and more stuff on the Internet is making social recommendation based on a user's social graph a mainstream reality.

The phenomenal growth of Twitter (32M users in May vs. 1.6M a year ago), and status update feature (copied from Twitter) used by Facebook's 200M+ users has immensely contributed in the social discovery of new content on the ever-growing World Wide Web. Given the relevancy due to the context provided by my social graph, I check out most of the links, photos, videos, and stories suggested to me on Facebook.

Publishers will very soon look at social recommendation engines as a major source of traffic to their sites, maybe more important than Web search engines, which on average contributes to as much as a third of all traffic to a publisher's site today. The impact on Google's business as a leading Web search engine remains to be seen. But users are not complaining. Why should they, if their trustworthy social graph is at work, 24/7/365, to open the doors to exciting, new content on the Web for them.

June 5, 2009

Tribute to Rajeev Motwani

Rajeev Motwani, a computer science professor at Stanford University, a Silicon Valley angel investor, and a close friend, unexpectedly passed away today in a seemingly unfortunate accident in his house. He was 47, and is survived by his wife Asha Jadeja and two daughters.

I'm currently in New Delhi on work, and just received a call with this shocking news from a common friend in San Francisco. This is a terrible, terrible loss.

I first met Rajeev around seven years back at the TiECon conference in Silicon Valley. We quickly became good friends given our common background - we both went to the same undergraduate engineering school in India (IIT Kanpur), he was from Delhi and I'd spent a significant part of my life, including senior schooling, in Delhi, and we shared our passion about Consumer Internet and technology.

Rajeev had been trying to pursue me to move to Silicon Valley ever since we met. He kept encouraging me to do a start-up, and felt I was not a big firm guy. He was one of the few people I always tried to meet when I visited the Valley. I'll always cherish all those moments when we would lounge around in the Stanford cafeteria and discuss and debate about industry trends, technology and interesting start-ups coming up in the Valley - Rajeev loved to mentor, advise, and even provide seed funding to aspiring students and entrepreneurs. He was an early adviser and backer of Google and PayPal. Sergey Brin, the Google co-founder, was one of the first ones to pay his tribute to Rajeev and acknowledged Rajeev's valuable contribution in his professional development as well as in the birth of Google in the halls of Stanford.

Rajeev was always helping others. He never hesitated to make my introductions to digital media & entertainment start-ups that wanted to work with NBC Universal, or entrepreneurs/students who needed my advise in this space. I've also been involved with Media X at Stanford University, a collaboration of the university and industry that brings together Stanford's technology research with companies committed to technical advancement and innovation (NBCU is a member).

Last I met Rajeev was only a couple of days before starting my current Asia trip - I was in Silicon Valley to speak at TiEcon, when we briefly met over coffee. We were supposed to have lunch the next day before my flight back to New York - he wanted to talk about a few startups he was working with. Unfortunately we couldn't coordinate the lunch due to some last minute changes in our schedules - his cell phone battery ran out, and by the time he got my message, I was already at the airport. We decided to catch-up upon my return from Asia. That will never happen now...such is life!

I'll always miss Rajeev's friendship, his quest for knowledge, his warmth, and his genuine desire & ever-willingness to help others. I would like to pay my most sincere condolences to Asha and her two daughters.