BT
x Share your thoughts on trends and content!

Funnel Analysis at Twitter for Improving User Engagement

by on Feb 25, 2016 |

Funnel analysis is used to analyze a sequence of events to help with user engagement on a website or a mobile application. Data Science team at Twitter uses this concept to learn how users interact with user interfaces during user sign up or tweeting for improving user engagement with Twitter. They use it for analyzing user interaction logs to ensure they are delivering product features that are useful and engaging for the users.

Krist Wongsuphasawat, Staff Data Scientist at Twitter, recently wrote about their experimental visual analytics approach used to count the specified sequence and also aggregate and visualize information between steps within sequences to provide broader perspectives.

Log analysis can be as simple as counting a single event such as the click on Tweet button. But, this event only opens the Tweet composer and doesn't necessarily mean the user successfully Tweeted. Funnel analysis or counting funnels can provide a broader picture to check how often users abandon a Tweet after they start composing. This leads to more exploratory questions but the scale of Twitter data poses challenges with the data analysis. The data includes more than 10K types of events and hundreds of millions of users. Twitter team has built a unified logging infrastructure that captures user activities across all clients, which makes these logs one of the largest datasets in the organization.

The team designed Flying Sessions, an experimental visual analytics tool for funnel analysis, to support funnel exploration with less required effort and provide more information than simple counting. This tool helps the data scientists to extract insights from the log data. Users can specify the granularity of their analyses by selecting parts of the sessions. Then the tool provides the users with aggregated results that can be explored interactively in a visualization interface.

The data pipeline for the solution consists of three steps: sessionization, segmentation, and aggregation. The technologies used for the visual analytics tool include Hadoop, Scalding, D3 and d3Kit. Scalding is used to filter and summarize large size raw log event data in Hadoop, into smaller JSON files that can be visualized using a web user interface developed in D3 and d3Kit.

In sessionization stage, a session is created for each user to add consecutive events (sorted by timestamp).

Segmentation stage extracts subsequences of events relevant to the analyst-specified alignment points from sessions, and groups the subsequences into segments based on the alignment points they contain. This stage includes identifying session fragments and then extracting and grouping subsequences between alignment points.

Finally, Aggregation stage pipes the segments from the previous stage through a variety of aggregators in parallel to produce summaries (e.g., average count of events) that can be visualized in the front end. This design allows for new aggregators to be added when there is a demand for additional types of summaries.

Twitter team also envisions few design improvements in the future to add new types of aggregation that employ pattern mining or more sophisticated algorithms.

 

Rate this Article

Relevance
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss
General Feedback
Bugs
Advertising
Editorial
Marketing
InfoQ.com and all content copyright © 2006-2016 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT

We notice you're using an ad blocker

We understand why you use ad blockers. However to keep InfoQ free we need your support. InfoQ will not provide your data to third parties without individual opt-in consent. We only work with advertisers relevant to our readers. Please consider whitelisting us.