Robots and Magicians: Yahoo! Pipes

In one of Carl Sagan’s writings he mentions that in the 1800’s the average reader could read every book in something like 60 years.  At the time of the writing, he said it would take the average reader something like 1,000 years to do it.  With all of the information available, there seems to be no excuse for not staying up-to-date on the latest information on technology, telecommunications, the evolving media landscape, and all the other factors relevant to my professional life. Except for one small detail: There are only 24 hours in the day. And even if I spent every minute of every day reading, I would still never keep up with all the e-mails, websites, blogs, and other sources of news that come across my screen. With massive amounts of information, it’s important to keep it organized and accessible. And while my rapidly expanding team of coworkers are pretty amazing, they are not speed-reading robots. Luckily, I’ve found a tool that has reduced our news-reading problems.

What’s my magic answer, you ask? In my best Steve Jobs voice: “Yahoo! Pipes… (crowd murmurs).  You heard me, Yahoo! Pipes people! (general crowd applause).  That’s right, Yahoo! Pipes! (hysterical crowd screaming and jumping up and down).”  Pipes might be the most awesome free online service out there.  As its website describes, “Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.”  People have been using it for several years already to find prime real estate, monitor eBay bids, and follow stock market trends. Doesn’t this sound like the perfect match for sorting through the news, too? Believe me, it is.

Most Pipes will end up looking like a tree. It doesn’t work like a tree.  Water usually starts at the bottom and works its way up.  With Pipes our water droplets are the individual news items found by team members (in the form of an RSS feed).  Rather than fall to the ground to get sucked up by the roots, each individual branch up top sucks up the items and funnels them down into single feed. This fits in with the broader use of RSS feeds, which are a fairly ubiquitous way to publish updating content like news headlines, blog articles, audio, video, and other web content in a standardized way.  They work here by combining hand-picked material that our staff posts to Delicious, Google Reader, Tumblr, and their blogs in a robot friendly output, while also automatically tracking sources from our organization website including publications, press we show up in, and more formal than this online articles.

Google Reader and Delicious rock for this purpose because they will both create RSS feeds for specific categories/tags/taxonomies. When you’re gathering feeds from a large number of people, you don’t necessarily want a steady stream of every RSS feed they subscribe to, so rather than input a person’s entire RSS feed from Delicious or Google Reader, have the user create a tag specific to your topic area, and take the RSS feed associated with that tag, only, to create your Yahoo! Pipe.

Gather a list of multiple RSS feeds that each participant uses to filter their web content (Delicious, Google Reader, Tumblr, Blogs, News Sources, Twitter, etc.). You can find these by searching the page for “RSS” or looking for the orange icon. Use each RSS feed as a source into Yahoo! Pipes’ Fetch Feed Source tool (IMAGE 1). With Pipes’ Regex Operator, take each feed and prefix the title of each item with the source (eg. the original title of “IRAQ WAR ENDS” is prefixed to become “Via Dan: IRAQ WAR ENDS”) (IMAGE 2). With Pipes’ Union Operator, combine each individual prefixed feed into a single feed (IMAGE 3). With Pipes’ Sort Operator, sort each individual item by the date each item was found by a user. With Pipes’ Unique Operator, remove duplicate items by filtering items pointing to the same URL and items that have identical titles. With Pipes, publish the combined, sorted, and filtered stream of items into a single RSS feed (IMAGE 4).  You can visit http://pipes.yahoo.com/danblah/otinews to see the current running version of our OTI News Pipe.

IMAGE 1 IMAGE 2 IMAGE 3 IMAGE 4

Google Reader