GNIP – “data services bus” for Consumer Web Content

GNIP is an early stage start-up that is providing an interesting solution to solve the problem of fresh content and data for content centric consumers and producers on the Internet. I happened to recently meet the CTO since they are here in my backyard (Boulder), but I’m writing about it today because of the new Twitter integration they just launched.

Today, GNIP is offering a single feature in GNIP Notifications, which turns any traditional publisher API into a “push” API. Why is this cool? Well it means that anyone looking to keep up with fresh content, events and activity streams from “foo service” now can just set that up with GNIP and not have to write code to “poll” the “foo service”. This also is a real benefit for content publishers since it removes a lot of traffic from other complimentary consumers who are polling for fresh data. Obviously the producers are not able to easily separate polling traffic from real traffic unless they do things like issue account keys and throttle certain consumers, but that also is not a real great thing for anyone. The answer is creating a more loose coupling via push notifications that can be filtered based on the producer and consumer requirements. Not hugely complicated, but GNIP is the first I have seen tackle the problem in this way.

Why is this good for Twitter? The scaling issues at Twitter have been documented a lot and GNIP could really help them better manage traffic while they are working to fix things by simply cutting down on polling traffic and pushing appropriate data out to consumers in the form on the Twitter XMPP feed that can be segmented by GNIP or by anyone. Or as TechCrunch put it:

With XMPP Twitter just sends out all of their data in a constant stream, whether you ask for it or not. The third party, in this case Gnip, takes the data and parses it for further use.

Gnip acts as an intermediary between applications that create social content and those that consume it. They take the Twitter feed, which is a list of usernames, Twitter status URLs and time stamps, and make it available to any third party that requests it. Both Plaxo and MyBlogLog are already using the new feed, and more partners will add it immediately. And every third party that takes data from Gnip doesn’t have to take it from Twitter, easing the overall load on Twitter’s servers

It will be very interesting to see the future features promised on GNIP’s website: Polling, Transformation and Identification. All of these are areas where a set of common infrastructure services could be very useful to the Internet ecosystem and also to enterprise IT as they look at leveraging external services.