I remember having this conversation with Mor Naaman on the fact that crossing data from different companies was not possible or easily accessible to the public. Apparently I am now wrong as Yahoo Research Lab released a new service called Yahoo Pipes, which allow the user to merge data from different streams and manipulate the information from useful purposes.

Pipes is an interactive feed aggregator and manipulator. Using Pipes, you can create feeds that are more powerful, useful and relevant.

The service offer an interactive interface where it is possible to play with the data stream (like an RSS feed), manipulate the data (like applying a filter), passing the output to another service (like a web search engine), collect the output from the engine, end so on till the end: the output page. The screenshot below shows a pipe called: “apartments near something” and tries to answering the question: show me apartments for rent near a school or something. Here is how it works:

This Pipe uses craigslist and Yahoo! Local. First, we look at a search on craigslist for apartments. The search results page will show an RSS feed at the bottom which we can use as a starting point. Using the URLBuilder (in Url) we can put together a dynamic feed address that searches by whatever location people enter using the Location Input module. This module automatically translates locations between city/state and zip codes as needed. the same location entered can then be fed into Yahoo! Local using the Yahoo! Local module. We can use another text input to get something we want to find apartments near (e.g., laundromats). After putting the craigslist feed through a Location Extractor module to get the location of each listing, we can compare the two and filter out any listing that isn’t a minimum distance from our search term. Now you can get apartment searches for apartments near things you care about! You can make a copy of this Pipe right here to play with or try running the pipe.

This service shows an example of what the web is going to be really soon: a jungle of data streams that can be intelligently re-aggregated and manipulated for different purposes. Interfaces like that of Yahoo Pipes will help users to recycle the web for their own purposes. Also, there is a rising number of studies on how bots (software agents) can use services made for humans for the same or other purposes and what kind of difference in results we can expect.

Tags: clustering, collaboration tools, information metric, map algorithms, new technology, search engine, society