Jon Holato

Yahoo released a new service today dubbed “Pipes.� In what it describes as “an interactive feed aggregator and manipulator,� Yahoo Pipes seeks to make connecting data, processing it, and redirecting the output an easy process that is no longer restricted to web programmers. Tim O’Reilly says that Pipes will create a “programmable web for everyone.�

Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique, count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for() loops. It can extract dates and locations as well as solicit user input and build URL lines to submit to sites. The drag and drop editor lets you view and construct your pipeline, inspecting the data at each step in the process. And of course, you can view and copy any existing pipes, just like you could with shell scripts and later, web pages.

As most of my readers are probably asking me to repeat this in English, let me try to explain this in more layman’s terms. First we start with the concept of an RSS feed, whereby we subscribe to a data source’s updates so that we can receive updated information as soon as it is posted. For most of us this data source is typically a web site, i.e. blog. As RSS usage has become more prominent, RSS subscribers are syndicating more and more feeds through various RSS readers such as Bloglines, Google Reader, etc.

Let’s consider this from a problematic standpoint. While RSS readers provide a very useful service to millions of internet users, there is a serious limitation to their functionality. They are only trained to grab information from the places you have subscribed and to display it to you. Thus, we are continuously presented with a whole slew of information from a single source, and are unable to do anything but read it. Enter Yahoo Pipes.

The technology implemented in Yahoo Pipes is revolutionary because it allows us to overcome the inherent limitations of RSS the way we currently use it. It recognizes that the RSS protocol can’t do this and that RSS readers don’t have the ability to that, and it presents a solution whereby both can accomplish what the other previously couldn’t. I am repeatedly mentioning RSS because it is probably the most well known dynamic data source, however in theory the information available for use in this technology is limited only by how much web sites are willing to share with others.

In the spirit of trying to make this as simple to understand as possible, let us compare the data sources to colors. Currently, we’re able to subscribe to standalone data sources. For example, we can subscribe to the colors red, blue and yellow, but these are the only ones available and we can only see them as themselves. Now, enter the technology behind Yahoo Pipes, suddenly we’re able to combine red and blue to make purple, and blue and yellow to make green, and so on and so forth. Not only are we able to continue seeing the original three as they are, but now we can see and create colors that were previously unknown.

Such is the case with Yahoo Pipes. By combining data sources and providing instructions on how they should interact with each other we are, in essence, creating new information by using old information in unforeseen ways.