Popular Bookmarks Yahoo Pipes Search Experiment #MashLib

A while ago I experimented with Yahoo Pipes to put together a search tool that aggregates links everyone has saved to social bookmarking sites Digg, Pinboard and Delicious and returns the most popular recent sites based on a simple keyword search. NB: I’m not talking about only the bookmarks I’ve saved, but all bookmarks saved by the communities on these sites.

So, if you enter the phrase “technology” you might get the following results list:

http://www.nytimes.com [13]

http://www.theatlantic.com [13]

http://www.theverge.com [9]

http://www.youtube.com [7]

…etc

The results are displayed in popularity order and the number in square brackets indicates the number of times anyone has bookmarked the site recently on Digg, Delicious or Pinboard. Each of the sites that appear in the results list also act as a clickable link to that site.

As it’s been created in Yahoo Pipes you can also get a variety of useful data formats as output, including RSS, JSON and PHP.

I decided to put it together as a way of discovering new sites, based upon sites other people had recently found useful. It’s doesn’t currently provide a comprehensive list of sites, but it does offer an alternative way of discovering sites that I might not have been returned by big name search engines.

It’s something I’d like to develop, but had forgotten about it until @AgentK23 mentioned something to me recently about collaborative bookmarking.

How I’d like to develop it…

  • Include as many social bookmarking sites as possible as part of the aggregation process to improve the comprehensiveness of the search results. The 3 mentioned are ones that I could easily generate a hackable and useful search/result query url for. For example, I couldn’t do anything useful with Diigo bookmarks, as it limits the results of community RSS feeds to 20 items (Edit: See positive update at foot of blog post). I’d be happy to receive suggestions about other social bookmarking sites I could tap into in this way.
  • The clickable links to the websites mentioned in the search results currently just go to the home page of those sites, but I’d like to work out a way to go directly to relevant articles on the site instead. Because different websites have different search query structures I couldn’t turn the links into ones that just focus on the search keyword that had been entered. For example, the New York Times link for the “technology” search mentioned earlier goes to www.nytimes.com , not http://query.nytimes.com/search/sitesearch/#/technology
  • Yahoo Pipes is a useful tool to try out ideas like this, but I’m still not sure about its reliability. So, I should think about developing this without relying on Yahoo Pipes.

Here’s the link to it if you want to try it out. Any feedback would be appreciated… and remember, it’s just an experiment and not a commercial product.

As most search tools have a daft name I thought I’d call it “DiPiDel POP!” – An abbreviation of Digg, Pinboard, Delicious Popular. :-)

Update: Thanks to Marjolein Hoekstra who followed up on this post and got in touch with Diigo about my issue. They have now extended the RSS feed to 100 items, which is very responsive of them and great news too, as I can now use the site as an aggregation source. As well as including Diigo in the aggregation process, I’ve also now included Blogmarks and Bibsonomy. Thanks to Marjolein for suggesting them too.

About these ads
Leave a comment

7 Comments

  1. Gary, you write you couldn’t do anything useful with Diigo. I believe you are an active Diigo user, right? What was it you couldn’t do with Diigo? Usually this service stands out from the other social bookmarking in functionality and versatility. Thanks!

    Reply
    • Hi Marjolein. Thanks for your comment. You’re right. I am an active Diigo user. My main problem in this case was the limitation of 20 items in the RSS output for recent bookmarks. However, I’ve added the feed for Diigo to the pipe now. I really should make an effort to get to know and learn how to use the APIs for all of these services. I know it would help. Thanks.

      Reply
  2. Hi Gary, I just checked with one of my own Diigo feeds, and for me appending “?count=100″ to the Diigo feed URL works.

    Reply
  3. I can’t get it to work either, Gary. Great thread.

    What’s worse, the feed items don’t seem to be sorted chronologically. Curious: what’s the “tab=153″ parameter doing in that url you constructed, why do you use quotes around the tag name, and why combine ?& as the first parameter separator?

    BTW a little while ago a little bird informed the Diigo Care team that there was an opportunity for them to chime in to the conversation here and show that they actually are listening. Their responsiveness on their own Uservoice forum hasn’t been very promising so far, but maybe this time they’ll kindly pick up the nudge.

    Some other services to consider for your project:
    Keeeb http://keeeb.com
    XMarks http://xmarks.com
    PearlTrees http://pearltrees.com
    Kippt http://kippt.com
    Bitly Realtime http://rt.ly.
    I haven’t checked whether these services offer an API, feed, JSON or other mechanism to retrieve data.

    Among the various IFTTT channels, and maybe on Zapier there also might be a couple you could put to good use.

    Reply
    • Hi Marjolein

      The tab=153 is used for recent bookmarks. They also use tab=151 (if I remember correctly) for most popular bookmarks.

      Both the “quotes” and “&” were errors. Apologies. I’d been hacking the url seeing if the order of the parameters made a difference to the number of items that came through. I’d also used a tag that contained 2 words and you need to use quotes around them so it recognises it’s a single tag and then I forgot to take them out before posting the link.

      I rechecked the link this time with a properly formed url and the problem is still the same.

      Re Diigo. I’ve got to say I did look on the forum for my particular problem and someone had asked the same question on there but never had a reply.

      Thanks for the links to those other bookmarking services. I’ll see if I can feed them in. I always forget about bit.ly as a bookmarking service too. I just automatically think of it as a link shortener, and forget the other aspect of it.

      I also had a look at ProgrammableWeb site and that’s come up with a few possibilities too.

      Thanks for your input on this thread. :-)

      Gary

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: