About two years ago while researching a blog post on crowdsourcing, I discovered Amazon’s Mechanical Turk service. As a crowdsourced labor solution, the Turk completely turns around the relationship between computers and people. Instead of asking silicon to perform complex pattern matching (finding words or concepts in text documents), speech recognition (transcription), or image processing (identifying road signs in photos), why not have software call out for help? That is, we the humans become non-software subroutines that tap into our unique biological processing engines especially suited for CPU-stumping tasks.
Other companies have followed suit delivering what are essentially systems to manage piece-meal work for conceptually intensive tasks. CrowdFlower is one of them. They’ve taken the “labor in the cloud” model one step further.
I remember the Amazon Mechanical Turk service providing a basic interface—command line and APIs—really more of a platform on which businesses could then build friendlier solutions. CrowdFlower has done just that. It plays the role of a true application layer to Mechnical Turk’s middleware function.
In fact, it leverages the Mechanical Turk as an underlying jobber service, along with options to select others as well.
With CloudFlower, you drag-and-drop to create a form containing questions that can be responded to with either free-form text, check-boxes, or multiple-choice answers.
One key area in which CrowdFlower has pushed forward the state of the crowdsource art is the way that content to be examined by workers is included in the job. You merely point CrowdFlower to an RSS feed, and it then renders a web page with each RSS entry married to a question.
This makes perfect sense for so many web-centric jobs involving tagging or scanning HTML content that’s already in the cloud.
CrowdFlower’s business model is based on taking a slice of the job price, the rest, of course, going to the workers. Investors seem to like the cash-flow potential: this start-up has been successful in gaining additional financing.
To help with quality control, CrowdFlower analyzes the number of “judgements” needed for each answer—the more data points from workers, the greater assurance you have that answers are correct.
To weed out spammers, CrowdFlower also lets you seed the form with “Gold Questions” for which answers are already known, thereyby helping CrowdFlower identify more conscientious workers.
To check out CrowdFlower for myself, I set up an experiment in which my remote workers will analyze FCC documents for specific keywords.
I’ll let you the know the results in a subsequent post.
For businesses, crowdsourcing means their labor pool for clerical workers becomes global—which is both amazing and discouraging at the same time.
As a fan of Mad Men, and its immersive recreation of office-life in the 1960s, I wonder how a Peggy Olson or a Don Draper would now fare in a crowdsourced world?
As scrappy and creative go-getters, their talents would lay undetected in a digital interaction.
Now, if someone could come up with a solution to spot creativity over the Internet, I’d feel a little better.
Related articles by Zemanta
- CloudCrowd Takes On CrowdFlower To Outsource Labor To The Cloud (techcrunch.com)
- CrowdFlower Raises $5 Million to Boost Crowdsourcing (bits.blogs.nytimes.com)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=9b17486e-3573-4a84-870b-abc8abe0705e)
Great piece on CrowdFlower. For creativity, you should check out 99designs.com. They are doing some awesome, creative work — in the graphic arts, advertisng world too — just like Mad Men — one of my favorites too!
Pingback: Playing with Amazon.com and Mechanical Turk | Broadcasting Brain - different thoughts about thinking differently
Great post!
I can’t seem to find the “subsequent post” with the analysis of the results. Were there any?
Hi Joakim. There is a subsequent post, in this somewhat convoluted thread: http://technoverseblog.com/2010/05/the-real-fcc-plot-open-internet-access/
I ultimately used the Crowdflower workforce to analyze videos of FCC chairman Genachowski’s public utterances–they went searching for “network neutrality”, “competition”,etc. I didn’t formally tally the accuracy of my workers results. But I did review the videos checking for the timestamps that were reported to indicate where the keywords were spoken.
So … my gut feeling is that for very simple tasks, Crowdflower is fine. This particular project was really at the **limits** of what the workers were willing to do–there were many errors. They had to listen to videos of up to 10 minutes in length, which takes far more time than having to scan through, for example, text documents. To get accurate transcriptions, of course, you can pay for a service that specializes in this, but that costs far more money.
You get what you pay for also applies to crowdsourcing.
–Editor, Technoverse Blog
I someone who has done some of Crowdflower’s tasks, I have to say that some of Crowdflower’s customer’s are not being honest with their gold. It is pretty obvious that some of them are trying to stack the deck against the worker in order to get the task done for free. All this is doing is encouraging the worker to bail out of the task and possibly not do this type of work in the future. Crowdflower really needs to randomly screen the gold to ensure that they are honest and fair. Doing so will help insure that this type of work is workable in the future, otherwise it is more than likely to fail.
Arcosanti. There may be a way to game the system. I haven’t used CrowdFlower in a while so I forget some of the details. I think it’s about time to take another look at Crowdflower!
Thanks for you comment.
Editor, Technoverse Blog