Over the last few months, recommendation startups have sprouted up—getglue, Hunch, Foodspotting, Parse.ly, Miso, Xydo (in beta), Bubbalon, etc.—to offer suggestions about restaurants, books, web sites, or just about anything in this world.
If you add in Facebook (with its like button, and lots of 3rd-party rating apps ), Amazon, and NetFlix, there’s enough of a universe to merit a service that rates and recommends recommendation services. There’s a startup, no doubt, working this out.
All share the idea that there’s wisdom in the crowd, and to various extents use stats about the mob to algorithmically classify tastes—clustering, nearest neighbor,decision trees—and then generate suggestions. There’s a nice summary of these collaborative filtering techniques in the reference section below.
What about a more conventional, common-sense approach that derives wisdom from actual knowledge of the subject?
I examined a few of the collaborative filtering methods that are currently being exploited. Let’s just say I’m troubled by a taste algorithm that in effect says, “let’s take the square root of Terminators, divide by Ace Ventura, and add Steven Spielberg to come up with Gigli.”
Like many others, I’m fascinated but incredibly cautious about silicon wafers simulating human insight. And have felt that way ever since legendary computer scientist Joseph Weizenbaum offered up the world’s first suggestion engine:Eliza, the computerized therapist and personal coach.
A more appealing approach for me involves software that tries to analyze the actual content of the thing it is recommending.
Humans still do a better job of this, and most of the advice I take more seriously comes from reviews, postings, and tweets on conventional news sites, as well as blogs and newer social networking communities.
A compromise of sorts can be achieved through software that maps out a knowledge space, typically as a complex graph, and then let’s you navigate to find useful information.
Freebase’s movie topic is rich with links to sub-topics
We all do this when manually searching an on-line movie or book database: finding a book we want and then browsing for other titles, based on, perhaps author, subject, or genre. This style of virtual window-shopping is rewarding and plain fun.
This all leads to Freebase, an immensely ambitious public knowledge database (recently purchased by Google) that supplies more search opportunities than conventional databases.
It is both a database platform, built on a non-relational network model, and a Wikipedia-style content creation service, where editors review content generated by the community.
Anyone can search Freebase’s public database, which has millions of facts stored away in all the usual areas—film, people, movies, and music—and go on a wild ride following the deep set of relationships: “So, Gigli had a budget of $54 million. Lainie Kazan had a bit part in it. What other movies was she in?”
Of course, Freebase has a web-service API that lets you create programs, thereby enabling developers to assemble true applications.
I took a shot at crafting an API query to rove Freebase’s film topic territory—fyi: the curve for grokking the Freebase search language is steep—and would like to reveal …
An early beta version of The Technoverse Blog Movie Recommendation widget.
You enter a director whose movies you enjoy, and my knowledge app will respond with films, by other directors, who share the same cinematographers. It is obviously a very simple recommendation concept, and no doubt would gain more precision if I added dimensions such as similar genre, writers, producers,etc.
The key point is that there’s no reliance on the crowd or forcing the user to first rate zillions of other films.
Considering the effort (not much), the results are not half bad. Enjoy. (Hat tip to Pengoworks for their slick jQuery autocompleter script.)
Technoverse Movie Recommender
If you like movies by director (try Woody or Steven or Robert)
you may also enjoy:
Related articles
- The Freebase Blog (freebase.com)
- Freebase
- Freebase: Semantic Sandwich for Google (technoverseblog.com)
- Semantic Web News (bueda.com)
- A Survey of Collaborative Filtering Techniques (hindawl.com)
- Jospeh Weizenbaum’s ELIZA (nyu.edu)
- jQuery Autocomplete (pengoworks.com)
Great post! I’m one of the founders of Parse.ly, and I think you bring up a very significant point regarding recommendation services. At Parse.ly, we’ve experimented a ton with different research ideas, as well as our own proprietary tech. One thing that’s emerged as a result of our work is the fact that sometimes the simplest recommendation algorithms yield the best results.
For example, when we launched the Parse.ly Reader (consumer-facing beta), we allowed a user to give us a few topics that they were interested in and we would use simple keyword matching to find those articles and show it to the user. As we moved forward we started to implement behavioral technology combined with data mining and NLP, as well as collaborative filtering technologies. What we found was that, in the short-term, although they do improve clicks, it was nothing compared to simple keyword matching (in fact I found this article using our reader which I’m sure just saw that you mentioned us). In the long-term though, these technologies have played an important role at keeping the user engaged. What collaborative filtering, in particular, can do is bring a healthy measure of serendipity to the reader. With this technology the probability that a reader finds content that he or she likes and is on the fringe of their interest graph increases. This, in turn, expands their interest profile which gives us more areas of content to recommend. The more we know about the user’s various interest the higher the chance we have of finding content that is resonant with the users. And this, at the end of the day, is the ultimate goal. I suppose Netflix found this as well, which is why they put so much effort in to improving this aspect specifically.
Ultimately though, simple algos can get you a healthy margin of the way there — it’s the hard stuff that you need to get to the rest.
Thanks for your comment! And for helping me (and others as well, I assume) understand the role of collaborative filtering in recommendation services. Point taken about the power of pattern matching, even ofa simple nature, in finding interesting and relevant content.
Actually, this post was partly inspired by a few videos on the web by legendary AI guy, Peter Norvig, who is Google’s Director of Research. He shows how even pretty basic pattern matching algorithms can learn from the data, but you need _lots_ of data. I still prefer humans building the basic categories and properties of a knowledge space, as with Freebase. In this Norvig video (http://www.youtube.com/watch?v=nU8DcBF-qo4), he outlines a more sophisticated approach that learns patterns and relationships automatically. As you said, you can get pretty far with simple pre-determined patterns, but to get the rest of the way requires much more work. 🙂