Social recommendation reaching long-promised potential

Content on the Internet has been exploding exponentially. YouTube, the largest video site on the Internet, adds ten hours of new video every single minute. By some measure, 10M new pages are added on the World Wide Web every day. The world of blogs has also grown tremendously, with Technorati, the popular blog search engine, tracking over 120M blogs worldwide (excluding millions that are not tracked in China, country with the world's largest Internet population).

This makes discovery of new content a huge challenge for both users (how to discover) and publishers (how to get discovered). I'm going to focus only on the user challenge in this post.

As the content on the Internet continues to multiply dramatically, the ability for users to find the content that may exist out there, which they may like and therefore want to consume, poses one of the most significant problems on the Web. Search engines have become a popular tool for users to find what they're looking for, but Search requires users to express their intent first. Since you don't know what you don't know, discovery of new content that you are unaware of, but may like, necessitated the development of recommendation engines, which proactively suggest what content users may like without much, if any, explicit action on their part.

Recommendation engines broadly fall under three categories:

1) Editorial recommendation: This is the most basic form of recommendation method whereby the editor/publisher picks what to highlight on its site amongst all available content - e.g., "featured video of the day", "featured story of the week," etc. This is a one-size-fits-all approach, where the recommendation focuses on publisher's goals and/or the average consumption behavior/demand amongst all visitors to the site, and is not customized for each individual visitor.

2) Customized recommendation based on individual's behavior: This method involves analyzing each user's content consumption pattern and using technology to recommend similar content to that user. This is the most difficult approach technically. Complex algorithms are needed to 1) "read" and categorize the repository of all available content on one hand, 2) track, capture and analyze each user's consumption patterns, and then 3) match those patterns against the categorized repository to make recommendations which are uniquely customized for each user.

Examples of this approach include the Pandora Internet radio that provides customized radio channels to users based on the songs users initially select to hear. Pandora's Music Genome Project utilizes 400 different characteristics to classify all available songs into categories that are leveraged by Pandora to understand a user's music taste and recommend him/her relevant songs. Cinematch, Netflix's movie rental recommendation system, is another such example.

"Reading" the content repository and categorizing them, say, in logical genres, is the most difficult aspect of this approach. Getting the recommendation correct with a high level of accuracy on a consistent basis is extremely hard. Since the recommendation engine works in the background, employing complex algorithms and other behavioral science factors, users expect the "black box" to make correct recommendation every single time. Getting it 90% correct, though impressive, may still not win user's loyalty and trust. If I hate chick flicks, make one such movie recommendation to me, and that's the last time I'd trust the technology behind the recommendation engine.

Netflix, recognizing the complexity of this recommendation approach, has an on-going contest, launched in October 2006, that will award $1M to the person that can improve the accuracy of its already-impressive Cinematch movie recommendations by 10%.

3) Social recommendation: This approach makes content recommendation to a user based on usage patterns of other users instead of his/her own consumption pattern. It's essentially a popularity contest, where popularity amongst a set of users is measured to make recommendations.

Social recommendation comes in two flavors. The set of users used to measure popularity can be either a limited set - user's friends (social graph), members belonging to a particular group or affiliation, etc - or the general set of all users consuming content on the site (e.g., most viewed, highest rated, etc).

Of the two, social recommendation based on preferences of my social graph - people I trust - holds the biggest promise amongst all approaches currently being utilized to make content recommendations. Reasons: 1) the approach requires rather simple technology, 2) users expectations are managed - not all recommendations need to be on the mark because everyone has friends whose tastes are different from their own - and 3) the approach simply works, because it follows a long and well established norm in the real world. By some measures, more than 30% of new content consumed by people is based on recommendation from someone they know and trust. Web 2.0 tools and changing user behavior where more and more people are sharing and contributing more and more stuff on the Internet is making social recommendation based on a user's social graph a mainstream reality.

The phenomenal growth of Twitter (32M users in May vs. 1.6M a year ago), and status update feature (copied from Twitter) used by Facebook's 200M+ users has immensely contributed in the social discovery of new content on the ever-growing World Wide Web. Given the relevancy due to the context provided by my social graph, I check out most of the links, photos, videos, and stories suggested to me on Facebook.

Publishers will very soon look at social recommendation engines as a major source of traffic to their sites, maybe more important than Web search engines, which on average contributes to as much as a third of all traffic to a publisher's site today. The impact on Google's business as a leading Web search engine remains to be seen. But users are not complaining. Why should they, if their trustworthy social graph is at work, 24/7/365, to open the doors to exciting, new content on the Web for them.

Celebrities on Twitter

The growing popularity of the micro-blogging service Twitter, having grown by 13x over the past year to 7M+ uniques in Feb 09, and its increasing use for promotion/marketing, has made it popular among celebrities as well. Michael Phelps, Martha Stewart, Jane Fonda, Jimmy Fallon, P. Diddy, Deepak Chopra, Ashton Kutcher, Al Gore, are among a wide range of celebrities who are active Twitter users. Fans definitely welcome it, as they prefer direct and unfiltered updates from their stars, who also get an opportunity to connect with their fans in a manner that was not possible before such technology tools came along. Some celebrities, of course, appoint a staff member to manage their Twitter account.

Below is a representative celebrity Twitter ecosystem from The New York Times. The chart shows celebrities following one another on Twitter (as of March 18). Click of the chart to enlarge it.

Ambient Awareness - Social scientists explain Facebook & Twitter

Clive Thompson at The New York Times has provided a sociological & psychological analysis of microblogging tools popularized by Facebook and Twitter in his wonderful essay Brave New World of Digital Intimacy.

I'd highly recommend everyone, whether you lead a digital life or not, to read the full article - especially older people (over thirty) who are puzzled by the phenomenal success of microblogging.

Here are my highlights from the article, though it does not capture the storytelling essence that the full article would provide:

Social scientists have a name for this sort of incessant online contact. They call it "ambient awareness." It is, they say, very much like being physically near someone and picking up on his mood through the little things he does — body language, sighs, stray comments — out of the corner of your eye. Facebook is no longer alone in offering this sort of interaction online...

For many people — particularly anyone over the age of 30 — the idea of describing your blow-by-blow activities in such detail is absurd. Why would you subject your friends to your daily minutiae? And conversely, how much of their trivia can you absorb? The growth of ambient intimacy can seem like modern narcissism taken to a new, supermetabolic extreme — the ultimate expression of a generation of celebrity-addled youths who believe their every utterance is fascinating and ought to be shared with the world...

This is the paradox of ambient awareness. Each little update — each individual bit of social information — is insignificant on its own, even supremely mundane. But taken together, over time, the little snippets coalesce into a surprisingly sophisticated portrait of your friends' and family members' lives, like thousands of dots making a pointillist painting. This was never before possible, because in the real world, no friend would bother to call you up and detail the sandwiches she was eating.

Facebook and Twitter may have pushed things into overdrive, but the idea of using communication tools as a form of "co-presence" has been around for a while. The Japanese sociologist Mizuko Ito first noticed it with mobile phones: lovers who were working in different cities would send text messages back and forth all night — tiny updates like "enjoying a glass of wine now" or "watching TV while lying on the couch." They were doing it partly because talking for hours on mobile phones isn't very comfortable (or affordable). But they also discovered that the little Ping-Ponging messages felt even more intimate than a phone call.

"It's an aggregate phenomenon," Marc Davis, a chief scientist at Yahoo! and former professor of information science at the University of California at Berkeley, told me. "No message is the single-most-important message. It's sort of like when you're sitting with someone and you look over and they smile at you. You're sitting here reading the paper, and you're doing your side-by-side thing, and you just sort of let people know you're aware of them." Yet it is also why it can be extremely hard to understand the phenomenon until you've experienced it. Merely looking at a stranger's Twitter or Facebook feed isn't interesting, because it seems like blather. Follow it for a day, though, and it begins to feel like a short story; follow it for a month, and it's a novel.You could also regard the growing popularity of online awareness as a reaction to social isolation, the modern American disconnectedness that Robert Putnam explored in his book "Bowling Alone." The mobile workforce requires people to travel more frequently for work, leaving friends and family behind, and members of the growing army of the self-employed often spend their days in solitude. Ambient intimacy becomes a way to "feel less alone," as more than one Facebook and Twitter user told me.

Online awareness inevitably leads to a curious question: What sort of relationships are these? What does it mean to have hundreds of "friends" on Facebook? What kind of friends are they, anyway?

In 1998, the anthropologist Robin Dunbar argued that each human has a hard-wired upper limit on the number of people he or she can personally know at one time...psychological studies have confirmed that human groupings naturally tail off at around 150 people: the "Dunbar number," as it is known. Are people who use Facebook and Twitter increasing their Dunbar number, because they can so easily keep track of so many more people?

Many maintained that their circle of true intimates, their very close friends and family, had not become bigger. Constant online contact had made those ties immeasurably richer, but it hadn't actually increased the number of them; deep relationships are still predicated on face time, and there are only so many hours in the day for that.

But where their sociality had truly exploded was in their "weak ties" — loose acquaintances, people they knew less well. It might be someone they met at a conference, or someone from high school who recently "friended" them on Facebook, or somebody from last year's holiday party. In their pre-Internet lives, these sorts of acquaintances would have quickly faded from their attention. But when one of these far-flung people suddenly posts a personal note to your feed, it is essentially a reminder that they exist.

This rapid growth of weak ties can be a very good thing. Sociologists have long found that "weak ties" greatly expand your ability to solve problems. For example, if you're looking for a job and ask your friends, they won't be much help; they're too similar to you, and thus probably won't have any leads that you don't already have yourself. Remote acquaintances will be much more useful, because they're farther afield, yet still socially intimate enough to want to help you out. Many avid Twitter users — the ones who fire off witty posts hourly and wind up with thousands of intrigued followers — explicitly milk this dynamic for all it's worth, using their large online followings as a way to quickly answer almost any question.

It is also possible, though, that this profusion of weak ties can become a problem. If you're reading daily updates from hundreds of people about whom they're dating and whether they're happy, it might, some critics worry, spread your emotional energy too thin, leaving less for true intimate relationships. Psychologists have long known that people can engage in "parasocial" relationships with fictional characters, like those on TV shows or in books, or with remote celebrities we read about in magazines. Parasocial relationships can use up some of the emotional space in our Dunbar number, crowding out real-life people.

Psychologists and sociologists spent years wondering how humanity would adjust to the anonymity of life in the city, the wrenching upheavals of mobile immigrant labor — a world of lonely people ripped from their social ties. We now have precisely the opposite problem. Indeed, our modern awareness tools reverse the original conceit of the Internet. When cyberspace came along in the early '90s, it was celebrated as a place where you could reinvent your identity — become someone new.

"If anything, it's identity-constraining now," Tufekci told me. "You can't play with your identity if your audience is always checking up on you.

...Leisa Reichelt, a consultant in London who writes regularly about ambient tools, put it to me: "Can you imagine a Facebook for children in kindergarten, and they never lose touch with those kids for the rest of their lives? What's that going to do to them?" Young people today are already developing an attitude toward their privacy that is simultaneously vigilant and laissez-faire. They curate their online personas as carefully as possible, knowing that everyone is watching — but they have also learned to shrug and accept the limits of what they can control.

It is easy to become unsettled by privacy-eroding aspects of awareness tools. But there is another — quite different — result of all this incessant updating: a culture of people who know much more about themselves... The act of stopping several times a day to observe what you're feeling or thinking can become, after weeks and weeks, a sort of philosophical act. It's like the Greek dictum to "know thyself," or the therapeutic concept of mindfulness.

Laura Fitton, the social-media consultant, argues that her constant status updating has made her "a happier person, a calmer person" because the process of, say, describing a horrid morning at work forces her to look at it objectively. "It drags you out of your own head," she added. In an age of awareness, perhaps the person you see most clearly is yourself.