Yahoo Pipes Retires… | OUseful.Info, the blog…

Standard

http://blog.ouseful.info/2015/06/04/yahoo-pipes-retires/

I’m also not surprised Yahoo Pipes is closing down.

Out of all my uses for Yahoo Pipes I think the key ones I have to work out how to move over somewhere else are the feeds that go into my library jobs, Surrey Mix, Hot Poppi apps and also the combined feed of the library blogs.

Most of this information is centred around combining RSS feeds, filtering them and generating an RSS output.

I might be able to do this with one of the IFTTT alternatives I posted about a while ago, or maybe I’ll actually just end up doing some programming myself to resolve it.

I’ve seen a couple of suggestions for a replacement for Yahoo Pipes, but it looks like they involve some programming themselves and fiddly setups. If that’s the case I’d rather stick to the programming tools I already know and resolve it that way.

Advertisements

Popular Bookmarks Yahoo Pipes Search Experiment #MashLib

Standard

A while ago I experimented with Yahoo Pipes to put together a search tool that aggregates links everyone has saved to social bookmarking sites Digg, Pinboard and Delicious and returns the most popular recent sites based on a simple keyword search. NB: I’m not talking about only the bookmarks I’ve saved, but all bookmarks saved by the communities on these sites.

So, if you enter the phrase “technology” you might get the following results list:

http://www.nytimes.com [13]

http://www.theatlantic.com [13]

http://www.theverge.com [9]

http://www.youtube.com [7]

…etc

The results are displayed in popularity order and the number in square brackets indicates the number of times anyone has bookmarked the site recently on Digg, Delicious or Pinboard. Each of the sites that appear in the results list also act as a clickable link to that site.

As it’s been created in Yahoo Pipes you can also get a variety of useful data formats as output, including RSS, JSON and PHP.

I decided to put it together as a way of discovering new sites, based upon sites other people had recently found useful. It’s doesn’t currently provide a comprehensive list of sites, but it does offer an alternative way of discovering sites that I might not have been returned by big name search engines.

It’s something I’d like to develop, but had forgotten about it until @AgentK23 mentioned something to me recently about collaborative bookmarking.

How I’d like to develop it…

  • Include as many social bookmarking sites as possible as part of the aggregation process to improve the comprehensiveness of the search results. The 3 mentioned are ones that I could easily generate a hackable and useful search/result query url for. For example, I couldn’t do anything useful with Diigo bookmarks, as it limits the results of community RSS feeds to 20 items (Edit: See positive update at foot of blog post). I’d be happy to receive suggestions about other social bookmarking sites I could tap into in this way.
  • The clickable links to the websites mentioned in the search results currently just go to the home page of those sites, but I’d like to work out a way to go directly to relevant articles on the site instead. Because different websites have different search query structures I couldn’t turn the links into ones that just focus on the search keyword that had been entered. For example, the New York Times link for the “technology” search mentioned earlier goes to www.nytimes.com , not http://query.nytimes.com/search/sitesearch/#/technology
  • Yahoo Pipes is a useful tool to try out ideas like this, but I’m still not sure about its reliability. So, I should think about developing this without relying on Yahoo Pipes.

Here’s the link to it if you want to try it out. Any feedback would be appreciated… and remember, it’s just an experiment and not a commercial product.

As most search tools have a daft name I thought I’d call it “DiPiDel POP!” – An abbreviation of Digg, Pinboard, Delicious Popular. 🙂

Update: Thanks to Marjolein Hoekstra who followed up on this post and got in touch with Diigo about my issue. They have now extended the RSS feed to 100 items, which is very responsive of them and great news too, as I can now use the site as an aggregation source. As well as including Diigo in the aggregation process, I’ve also now included Blogmarks and Bibsonomy. Thanks to Marjolein for suggesting them too.

Mashups At CPD25

Standard

Along with Chris Keene and Paul Stainthorp, I was recently asked by Craigie-Lee Paterson to present to around 20 people at a CPD25 training event at Goldsmith’s (University of London) on the theme of mashups and Web2.0 tools. The audience was mainly made up of academic and health library staff, so as a public librarian it was an opportunity for me to see things from another angle.

The morning was based around presentations from all three of us. Chris opened with a discussion about mashups, giving examples of what people have created with mashup tools and how mashups have developed. I followed up with a look at the tools/ resources you need to create mashups, such as RSS, Yahoo Pipes, library catalogues. Paul then went into detail about a catalogue project called Jerome at the University of Lincoln, as an example of what can be done when you get to the stage where you are able to programme and tinker with data. We finished the morning with questions and answers and a few more mashup examples.

"Simplest)l(" by Chrstphre

In the afternoon Paul and I ran a practical session to create a mashup with Yahoo Pipes. All in attendance sat at their own computer and followed an example, which took an original RSS feed from the Guardian newspaper, filtered out unwanted news articles and tweaked the information so that it was presented in a particular way. After this Yahoo Pipes tutorial we gave people the opportunity to build their own.

I really enjoyed the event. Being involved in a training session outside of my public libraries role provides me with another perspective on how mashups can be used and also what is going on in libraries in the broader arena. It’s always great for me to be involved in these events, as I also get the chance to learn from other presenters – I think Paul and Chris would agree that we all have our own specialist areas and (as with Mashed Libraries events) sessions like this help me fill in gaps in my knowledge.

I think the day worked well and the feedback from Craigie-Lee and those in attendance was positive. Hopefully it will have inspired some of those people in attendance to make use of these tools and get mashing.

Leeds Libraries use Mapped in Yahoo Pipes

Standard

Recently, Ian Clark blogged about proposed closures in Leeds Libraries on the Voices For The Library site. Following the Freedom of Information request this blog post was based on, we thought it might be useful to map some of the data, as a simple way of comparing libraries in Leeds. (NB: These figures were only a starting point for the findings.)

The FoI request included details of issues, visits and PC bookings. After tracking down unemployment figures for electoral districts, I mapped them to postcodes so they related to the correct libraries. The data was then combined in a Google spreadsheet and the spreadsheet was mapped in Yahoo Pipes.

Each library appears as a marker on the map and they contain information such as “Middleton Library. Change in issues: 20185 . Change in visits: 28409 . Change in PC Bookings: 945 / Unemployment 2010: 13.6%“.

As I say, it was just a simple way of comparing usage figures of libraries situated close to each other alongside unemployment figures. It beats scanning a list of figures on a spreadsheet. I also just wondered if I could put Yahoo Pipes to practical use. My pipe tinkerings have previously been aimed at seeing what I could do with pipes, whereas this was more to do with putting it to good use and maybe building on it in some way.

Finding Packrati Popular Library Links

Standard

I’ve been wondering about how I can pull out popular links about libraries from Twitter, for current awareness purposes. I’m talking about the sort of links that people find so interesting they get retweeted. I suppose I could just create a twitter search and look at which links have been retweeted the most, but it’s a pain in the bum to perform the same search all the time and trawl through a load of search results. Plus the fact, I thought it would be interesting to try and do something a bit more techy.

I decided to make use of packrati.us, which is a bookmarking service used to automatically save links you tweet to your delicious.com account. It can also be used for saving links in HistoriousInstapaperPinboard.in, and Diigo accounts too, but I just use it for Twitter. Loads of other people use it too, so I thought I could make use of links that everyone has saved via this method.

(c) National Media Museum/Flickr

By default if a link is saved in delicious.com using packrati.us it saves it with the tag “via:packrati.us“. This gave me a starting point to create relevant RSS feeds to pull into Yahoo Pipes. I then built on it to pull in tags such as “library”, “libraries” and “librarians”.

Delicious is a bit of a nuisance, because it does rank bookmarks, but it doesn’t do it by the number of times a link has been bookmarked. It provides links to popular bookmarks (using some kind of relevance ranking), not necessarily links that have been saved the most. Strangely enough, even though delicious.com users have been asking for ranking by the number of times an item has been saved for a while, this feature hasn’t appeared.

I then:

  • Put the RSS feeds into Yahoo pipes
  • Combined the feeds into one feed
  • Filtered them (so that each link only appeared once in the list)
  • Sorted them by number of times the link appeared in the original RSS feed & date (to get most recent at the top of the list when it’s refreshed)
  • Pulled out keywords from the original Tweet and delicious bookmarks (I just wanted it to give me an idea of the focus of the link. eg literacy; reader development, etc.)
  • Deleted any irrelevant words (‘quot’, which appears in the text if ” is used)
  • Mapped the keywords to the description field.

This is the resulting pipe.

It does what I want it to do, but it would be better if:

  • All packrati links could be pulled out. At the moment I’m relying on people tagging anything they save via packrati with a tag reference to libraries too, so I may be missing out on library links that are popular, but haven’t had an extra tag added. There’s no other way of getting an RSS feed for a search on any keywords. RSS feeds in delicious.com are limited to tag searches.
  • My regex skills aren’t great, so some odd keywords like “RT” and “amp” appear in the description field of the results. I couldn’t get rid of them.
  • The term ‘library’ or ‘libraries’ can also refer to programming code collections, so I might end up with the odd false hit in the results.

As far as I’m concerned they’re not massive issues, but I’d like to get them ironed out if I can.

Anyway, now I don’t have to perform loads of searches every day to find the most popular library links.

Generating Blog Keyword Tags 2

Standard

I had another go at automating the tagging process for my blog using Yahoo Pipes, as I wanted to improve on my original idea, which was a bit scrappy.

So, I’ve reworked the pipe to pull out all of the keywords from all of the posts (using Category RSS feeds). The original pipe listed the blog post title and keywords associated with that blog post. The new pipe lists the most frequently used keywords in all of the blog posts. When the keyword is clicked on (in the RSS feed) it runs a search on that keyword and returns any blog post mentioning the keyword.

In the pipe I’ve manually filtered out certain irrelevant words eg ‘blog’, ‘amp’ and ‘doc’. As time goes on I’ll have to manually add more words.

The only problem at the moment is that, even though the pipe returns an unlimited number of keywords, WordPress.com is limited to showing the first twenty items. I decided to compromise and call the feed ‘Top automatic Tags’. Unsurprisingly the most common phrase is ‘Yahoo pipes’.

You can see it on the right hand side of this blog (if Yahoo pipes is working, of course 😉 )

PS. I’ve not abandoned the original Tagxedo idea, but I need a bit more time to tinker with it.

Twitter follower/friend map

Standard

I seem to be getting into the swing of things with Yahoo Pipes at the moment and I seem to be creating lots of maps. Every time I use it, something else clicks in my head and puts a smile on my face. Yesterday, Aaron Tay asked me if I knew how to create a Twitter followers or friends map. I didn’t, but I thought it would be a good way to see if I could get to grips with some of Twitter’s APIs and also if they’d play more nicely with Yahoo pipes than previously. It was also nice to be asked by someone else to do something like this – my own projects seem to be a bit self-centred, so being able to do something useful for someone else made a nice change.

The Twitter API lets you pull out details of a users friends/followers. It does this via their Twitter id number, but by creating a URL with their id added to it, you can pull out full details. You can use a programming language to do this too, but if it goes into Yahoo pipes I’d rather do it there. Once you’ve got this, you can narrow the info down to the various bits you need. In my case I wanted biography details, location, photo and a link to Twitter profile.

In summary, I had to:

(1) Create user input boxes for ‘username’ and to identify if the map was for ‘followers’ or ‘friends’. This meant anyone can enter their user details, rather than just myself.

(2) I then had to build a url to point to the Twitter API and include the detail in (1).

(3) This url then fetched the details of the users followers or friends. ie their id numbers only.

(4) I then built another url using the id’s, to fetch full details of every follower or friend of the user.

(5) Each users profile contains a location field and if you put this into the ‘location builder’ module it extracts very detailed geographic location. Pretty impressive, considering some users only give the vaguest of details. It’s not perfect though, as, for example @therealwikiman is mapped to the USA, even though his location info is detailed. As he’s really based in England, I imagine the commute in the morning is a bit of a nightmare. 😉

(6) From various fields in each profile I then built a description that contained Twitter image, biography and location in text.

(7) I also added a link to each of their Twitter home pages.

(8) Finally I mapped all of the data to standard RSS/map data fields (title, link, description, y:location). When Yahoo pipes works with data it changes field names to reflect what it’s done to the data, so you need to change them to a format that is recognised.

(9) I connected it to the pipe output.

Twitter follower and friends map

When it ran, because it saw the field ‘item.y:location’ in there, it automatically displayed the information as a map, which you can see here. You can also add your own user info into the search box and create your own map. (NB: Sometimes Yahoo pipes & Twitter don’t play nicely together. If you have a problem with this pipe and have a Yahoo account, try copying the pipe and adding your own information into the search boxes.)

One thing I would like to get to grips with in Yahoo pipes is to be able to embed the output of a pipe into a web page and also allow users to add their own input on the same page, but I’ve not cracked that yet. So, if anyone else can help me with that side of things it would be appreciated. Thanks.