BT

如何利用碎片时间提升技术认知与能力? 点击获取答案

推特Answers,一天50亿次会话处理是如何做到的?

| 作者 Sergio De Simone 关注 12 他的粉丝 ,译者 楚晗 关注 0 他的粉丝 发布于 2015年3月26日. 估计阅读时间: 2 分钟 | Google、Facebook、Pinterest、阿里、腾讯 等顶尖技术团队的上百个可供参考的架构实例!

Answers是Twitter的移动APP分析业务,它现在可以一天处理50亿次的会话。Ed Solovey是推特的软件工程师,他描述了他们的系统是如何工作以提供“可靠、实时和可行动”的数据,这些数据是基于亿万个移动设备每秒发送的百万次的事件。

如Solovey所讲述,Answers涉及的主要职责包括:

  • 接收事件;
  • 收集归档这些事件;
  • 进行离线和实时的计算;
  • 合并这些计算结果并产生相关性的信息

负责从设备接收事件的服务是用Go语言写的,并且它使用了亚马逊的弹性负载均衡器( Amazon Elastic Load Balancer)。它将每个消息负荷放入到持久化的Kafka队列中。Kafka可以存储大量的事件,但这里的存储应仅仅被用作一种临时性的缓存,它可以保存几个小时有价值的数据。而接下来Storm会传输数据到Amazon S3中完成存储。

当数据存储到S3中后,Amazon Elastic MapReduce就会对数据进行批次性计算处理。计算的结果会存储到Cassandra数据库集群中,可以通过API来查询使用。

到这里,故事只讲了一半。剩下比较确切的是Answers可以实时性地处理数据。在这个目标中,Kafka存储的内容被输送到Storm,在Storm中这些数据会被像Bloom FiltersHyperLogLog这样的统计算法处理以提供及时性的结果, 代价是“可忽略的精确度损失”。计算结果会被Cassandra数据库再次保存起来。

所以处理结束后,Cassandra既保存了批处理计算结果,也保存了实时计算的结果。查询API负责合并这两种计算流,并基于查询参数给出一致性的视图。当它们都可用时,批处理计算的结果更优先一些,因为它们更准确,如果批处理计算结果不可用时,就会优先使用实时性计算结果。

按Solovey所说,描述的这种架构在故障处理的情况下也是有效的,这要归功于连接Answers组件时使用了持久化的队列。这也保证了一个组件的任何中断不会影响其它的组件。所以,这种架构可以保证从故障中恢复,并且假设系统能在给定时间内恢复正常,那么就能保证数据不被丢失。

查看英文原文How Twitter Handles Five Billion Sessions a Day

评价本文

专业度
风格

您好,朋友!

您需要 注册一个InfoQ账号 或者 才能进行评论。在您完成注册后还需要进行一些设置。

获得来自InfoQ的更多体验。

告诉我们您的想法

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

机荷高速的话费工商局的 by 阿萨德 阿萨德

你说我第一次给你打电话的时候脸红了,我觉得很可爱。我的大号QQ是你要了两次才给你的,你把我空间所有的说说和写的日志都看了一遍,我说我也喜欢从一个人的文字里了解一个人。不得不承认,你所有的细心都成为了我爱上你的致命伤。
ikandou.com/book/99589863/
ikandou.com/book/54172960/
ikandou.com/book/25975431/
ikandou.com/book/55702900/
ikandou.com/book/316759879/
ikandou.com/book/95523138/
ikandou.com/book/8323150/
ikandou.com/book/610174928/
ikandou.com/book/56315916/
ikandou.com/book/88974447/
ikandou.com/book/25519887/
ikandou.com/book/68227532/
ikandou.com/book/712824594/
ikandou.com/book/65361137/
ikandou.com/book/110482494/
ikandou.com/book/65238970/
ikandou.com/book/39528250/
ikandou.com/book/110884135/
ikandou.com/book/711919490/
ikandou.com/book/81795517/
ikandou.com/book/58594518/
ikandou.com/book/610398993/
ikandou.com/book/212251319/
ikandou.com/book/87124090/
ikandou.com/book/57410119/
ikandou.com/book/66731588/
ikandou.com/book/87090779/
ikandou.com/book/72038243/
ikandou.com/book/710555414/
ikandou.com/book/414452635/
ikandou.com/book/96010036/
ikandou.com/book/311967867/
ikandou.com/book/83771368/
ikandou.com/book/410972523/
ikandou.com/book/73862404/
ikandou.com/book/65056665/
ikandou.com/book/43843457/
ikandou.com/book/915779482/
ikandou.com/book/27787497/
ikandou.com/book/910025857/
ikandou.com/book/113218851/
ikandou.com/book/12170494/
ikandou.com/book/18421789/
ikandou.com/book/511190877/
ikandou.com/book/86641990/
ikandou.com/book/114497344/
ikandou.com/book/814975281/
ikandou.com/book/810287742/
ikandou.com/book/912136596/
ikandou.com/book/776616/
ikandou.com/book/34470624/
ikandou.com/book/910004296/
ikandou.com/book/5416736/
ikandou.com/book/413760504/
ikandou.com/book/22895872/
ikandou.com/book/110021345/
ikandou.com/book/52990033/
ikandou.com/book/2419140/
ikandou.com/book/17796433/
ikandou.com/book/17378218/
ikandou.com/book/19745971/
ikandou.com/book/76452951/
ikandou.com/book/310767719/
ikandou.com/book/19683329/
ikandou.com/book/413908133/
ikandou.com/book/86514488/
ikandou.com/book/211559048/
ikandou.com/book/85962555/
ikandou.com/book/512547125/
ikandou.com/book/815420534/
ikandou.com/book/816690661/
ikandou.com/book/64369920/
ikandou.com/book/612876616/
ikandou.com/book/52931360/
ikandou.com/book/711700771/
ikandou.com/book/616448355/
ikandou.com/book/912456656/
ikandou.com/book/1720478/
ikandou.com/book/314045415/
ikandou.com/book/44118807/
ikandou.com/book/74909222/
ikandou.com/book/19318976/
ikandou.com/book/610598098/
ikandou.com/book/66046485/
ikandou.com/book/12517025/
ikandou.com/book/99573898/
ikandou.com/book/73336822/
ikandou.com/book/415403960/
ikandou.com/book/714610234/
ikandou.com/book/31819365/
ikandou.com/book/22396017/
ikandou.com/book/84172783/
ikandou.com/book/316523922/
ikandou.com/book/45770548/
ikandou.com/book/77239740/
ikandou.com/book/710204035/
ikandou.com/book/49263316/
ikandou.com/book/85809414/
ikandou.com/book/81582157/
ikandou.com/book/614393544/
ikandou.com/book/41712642/
ikandou.com/book/415715707/
ikandou.com/book/216409890/
ikandou.com/book/311404215/
ikandou.com/book/86869134/
ikandou.com/book/710412594/
ikandou.com/book/11937775/
ikandou.com/book/314116742/
ikandou.com/book/212733730/
ikandou.com/book/510428952/
ikandou.com/book/42012828/
ikandou.com/book/58792032/
ikandou.com/book/516407275/
ikandou.com/book/92211438/
ikandou.com/book/69527327/
ikandou.com/book/511309407/
ikandou.com/book/31211965/
ikandou.com/book/113223296/
ikandou.com/book/312318457/
ikandou.com/book/713521689/
ikandou.com/book/31933104/
ikandou.com/book/46290727/
ikandou.com/book/79053639/
ikandou.com/book/810635615/
ikandou.com/book/816269039/
ikandou.com/book/94278158/
ikandou.com/book/916752525/
ikandou.com/book/15161274/
ikandou.com/book/44684528/
ikandou.com/book/25737729/
ikandou.com/book/7346775/
ikandou.com/book/412617339/
ikandou.com/book/92439655/
ikandou.com/book/51169668/
ikandou.com/book/216474369/
ikandou.com/book/515907049/
ikandou.com/book/48238769/
ikandou.com/book/911387173/
ikandou.com/book/916322494/
ikandou.com/book/48784531/
ikandou.com/book/74932906/
ikandou.com/book/35213064/
ikandou.com/book/55000813/
ikandou.com/book/610504663/
ikandou.com/book/515471020/
ikandou.com/book/710991229/
ikandou.com/book/610210211/
ikandou.com/book/115162782/
ikandou.com/book/12317284/
ikandou.com/book/79272429/
ikandou.com/book/97928166/
ikandou.com/book/714906805/
ikandou.com/book/716122951/
ikandou.com/book/515504503/
ikandou.com/book/81308100/
ikandou.com/book/36812091/
ikandou.com/book/712694454/
ikandou.com/book/413876968/
ikandou.com/book/23574223/
ikandou.com/book/913795554/
ikandou.com/book/515331524/
ikandou.com/book/211351980/
ikandou.com/book/8384810/
ikandou.com/book/111295050/
ikandou.com/book/314412016/
ikandou.com/book/75211945/
ikandou.com/book/5160363/
ikandou.com/book/87059745/
ikandou.com/book/715568750/
ikandou.com/book/112359294/
ikandou.com/book/512694940/
ikandou.com/book/42563698/
ikandou.com/book/310216241/
ikandou.com/book/52956898/
ikandou.com/book/313605212/
ikandou.com/book/22770549/
ikandou.com/book/92912814/
ikandou.com/book/58250283/
ikandou.com/book/89547322/
ikandou.com/book/91016481/
ikandou.com/book/411651135/
ikandou.com/book/71533989/
ikandou.com/book/910598394/
ikandou.com/book/36878604/
ikandou.com/book/110494778/
ikandou.com/book/910072012/
ikandou.com/book/910678487/
ikandou.com/book/38949724/
ikandou.com/book/72236720/
ikandou.com/book/614625868/
ikandou.com/book/26943552/
ikandou.com/book/911535999/
ikandou.com/book/98270999/
ikandou.com/book/210927939/
ikandou.com/book/910999202/
ikandou.com/book/713889998/
ikandou.com/book/315681743/
ikandou.com/book/95208068/
ikandou.com/book/811639818/
ikandou.com/book/77875864/
ikandou.com/book/914939704/
ikandou.com/book/62504957/
ikandou.com/book/510142107/
ikandou.com/book/14769365/
ikandou.com/book/915331026/
ikandou.com/book/5457042/
ikandou.com/book/511662316/
ikandou.com/book/53548754/
ikandou.com/book/76002123/
ikandou.com/book/411411678/
ikandou.com/book/31793927/
ikandou.com/book/68109716/
ikandou.com/book/410341315/
ikandou.com/book/62651908/以前,我很多的小脾气小任性,认识你之后我在...

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

允许的HTML标签: a,b,br,blockquote,i,li,pre,u,ul,p

当有人回复此评论时请E-mail通知我

1 讨论

登陆InfoQ,与你最关心的话题互动。


找回密码....

Follow

关注你最喜爱的话题和作者

快速浏览网站内你所感兴趣话题的精选内容。

Like

内容自由定制

选择想要阅读的主题和喜爱的作者定制自己的新闻源。

Notifications

获取更新

设置通知机制以获取内容更新对您而言是否重要

BT