A while ago I experimented with Yahoo Pipes to put together a search tool that aggregates links everyone has saved to social bookmarking sites Digg, Pinboard and Delicious and returns the most popular recent sites based on a simple keyword search. NB: I’m not talking about only the bookmarks I’ve saved, but all bookmarks saved by the communities on these sites.
So, if you enter the phrase “technology” you might get the following results list:
The results are displayed in popularity order and the number in square brackets indicates the number of times anyone has bookmarked the site recently on Digg, Delicious or Pinboard. Each of the sites that appear in the results list also act as a clickable link to that site.
As it’s been created in Yahoo Pipes you can also get a variety of useful data formats as output, including RSS, JSON and PHP.
I decided to put it together as a way of discovering new sites, based upon sites other people had recently found useful. It’s doesn’t currently provide a comprehensive list of sites, but it does offer an alternative way of discovering sites that I might not have been returned by big name search engines.
It’s something I’d like to develop, but had forgotten about it until @AgentK23 mentioned something to me recently about collaborative bookmarking.
How I’d like to develop it…
- Include as many social bookmarking sites as possible as part of the aggregation process to improve the comprehensiveness of the search results. The 3 mentioned are ones that I could easily generate a hackable and useful search/result query url for. For example, I couldn’t do anything useful with Diigo bookmarks, as it limits the results of community RSS feeds to 20 items (Edit: See positive update at foot of blog post). I’d be happy to receive suggestions about other social bookmarking sites I could tap into in this way.
- The clickable links to the websites mentioned in the search results currently just go to the home page of those sites, but I’d like to work out a way to go directly to relevant articles on the site instead. Because different websites have different search query structures I couldn’t turn the links into ones that just focus on the search keyword that had been entered. For example, the New York Times link for the “technology” search mentioned earlier goes to www.nytimes.com , not http://query.nytimes.com/search/sitesearch/#/technology
- Yahoo Pipes is a useful tool to try out ideas like this, but I’m still not sure about its reliability. So, I should think about developing this without relying on Yahoo Pipes.
Here’s the link to it if you want to try it out. Any feedback would be appreciated… and remember, it’s just an experiment and not a commercial product.
As most search tools have a daft name I thought I’d call it “DiPiDel POP!” – An abbreviation of Digg, Pinboard, Delicious Popular. 🙂
Update: Thanks to Marjolein Hoekstra who followed up on this post and got in touch with Diigo about my issue. They have now extended the RSS feed to 100 items, which is very responsive of them and great news too, as I can now use the site as an aggregation source. As well as including Diigo in the aggregation process, I’ve also now included Blogmarks and Bibsonomy. Thanks to Marjolein for suggesting them too.