Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Twitter Cuts Off Firehose Partner DataSift

Twitter Cuts Off Firehose Partner DataSift

Twitter announced on Friday night that it was cutting off firehose access to DataSift one of their few remaining firehose data resellers. The announcement has spawned discussion in the API community about Twitter's longer-term intentions and whether this is a positive or negative move for the Twitter platform.

Twitter APIs never provide full access to the half a billion tweets per day that run through their infrastructure. Organizations that wish to gain full access to the Twitter firehose have been required to go through third-party resellers – Gnip, DataSift and NTT Data - who have managed the technical logistics of ingesting and managing large streams of undifferentiated tweets. The firehose data is re-used by hunderds of downstream companies for a variety of purposes including social metrics, advertising and analytics.

Exactly one year ago Twitter acquired Gnip the biggest of these firehose resellers, signalling to many that Twitter itself was moving into this part of the ecosystem. That signal appears to have played out with the latest announcement. Ironically, Twitter made Friday’s announcement through the Gnip Blog.

DataSift appears to have been surprised by the announcement which, they write, comes after months of negotiating with Twitter. DataSift assures its supporters that their business model remains sound, but attack Twitter for this move:

“The bottom line: Twitter has seriously damaged the ecosystem this week. 80% of our customers use technology that can’t be replaced by Twitter. At the end of the day, Twitter is providing data licensing, not processing data to enable analysis.”

Also critical of the move is Steve Willmott, CEO of 3Scale and a prominent writer about the API economy. In a blog post provocatively titled “How Not to be a Platform: Twitter's Firehose Mistake.” Steve fears that in this move, Twitter has just lost a "critical innovation layer." Steve writes that while Twitter's claims of wishing to be “closer to the data consumers” may be valid in the short-term, ultimately a single provider cannot properly address the smaller and niche use-cases. According to Steve, platform providers should “grow the pie, don't try to eat all of the pie.”

Steve harks back to a similarly controversial move by Twitter in 2012 when it tightened controls on its API use by consumer clients which “mimicked” the functionality of the official Twitter client. In a parallel with the current closure, that move was also presaged by Twitter’s acquisition of TweetDeck, a third-party twitter client which depended on the Twitter API. Steve says that the resulting state of the Twitter client ecosystem, three years on remains unchanged:

“The Twitter apps are neat and slick, but there is nothing radical. Would there have been something radical if API access was open? We’ll never know.”

However not everyone agrees with this view. While Steve might be concerned with the API Economy, Tyler Singletary, Director of Platform at Klout is more concerned with the Politics of APIs. In a blog post entitled “How to Be A Platform: Making Tough Partnership Choices” (a clear challenge to Steve's thesis), Tyler writes that platforms often need to make difficult choices and that Twitter's choice not to renew DataSift illustrates Platform realities. Tyler argues that DataSift drifted from being a data distributor into the role of an analytics platform. DataSift added value in their own right and their filtering even choked down some of Twitter's upstream revenue.

Tyler asks, when building a platform, what if the products, the partners and the customers aren't all “right”? Does the platform provider have the right to change any one of these ingredients? Continuing with a hypothetical case, Tyler writes that a platform provider, when dealing with a “badly behaved” partner can choose to vary the form of partnership, or could simply “cut ties” with the partner, concluding:

“I think we’ll find that Twitter made the right choice for Twitter, and it’s very likely the right choice for the ecosystem. There will be a period of adjustment, and the transition will not go as smoothly as it could have — but their hand may have been forced by being faced with spending another year or more in a bad situation.”

Platforms have a long history of this dynamic. Chris Dixon, a serial entrepreneur, investor and General Partner at Andreessen Horowitz predicted the Twitter showdown in a 2009 blog post pointing out that “...especially in the technology sector, some of the most brutal competition has been between complements.” Chris recalls that Microsoft was for a long time “...famous for destroying companies that offer complementary products.”

“Twitter Apps...will eventually find their biggest competitor to be Twitter itself, not the substitute product they see themselves as competing against today.”

It's interesting to hearken back to Twitter's last controversial API change when they cut off consumer clients with the API v1.1 updates. Twitter's rationale identified four quadrants in their ecosystem and singled out the top-right quadrant as their target at that point. The latest firehose closure may signify that the bottom-left quadrant is now in play.

What do you think? Is this just a healthy adjustment to partnerships, or is it the latest move in platform tic-tac-toe?

Rate this Article