Transcript
Yao Yue: Today, I'm going to take a stab at line charts, which I'm sure each of you have looked at hundreds, if not thousands. I think we can do better. A little bit about me. I am a builder, operator, and occasionally researcher in the form of shooting the ideas of puppy-eyed PhD students telling me what they want to do. I tell them, no, that's never going to work in production. Over my career, I actually spend more time debugging and triaging incidents than doing programming. My job title is sometimes still software engineer. I don't admit that I'm a fraud, so builder is still the first thing on the title. Fifty percent plus of my time was operating. Over that time, I handled some large-scale systems, not the largest in the industry, but getting up there. Seven of those 15 years, I was a pager-carrying, on-call person for a very large tier-1 service.
Thankfully, not tier-0, so never DNS for me. At a company that shall not be named, aka Twitter. I have got some stories to tell. This is why there's a hyperlink. If you read that blog post, which I co-wrote with one of my teammates, it will take you well above an hour. I'm not going to talk about that. Here, you can read it. It's all the cache incidents that I handled with my teammates back at Twitter. I've learned some things. During this time, as I said, because I spent so much time trying to understand why things break, a lot of that time was spent staring at visibility tools, at charts, in particular. Built charts, looked at charts, and I wondered why I couldn't see the things I wanted to see on charts. That is the foundation of this talk, is the things I have realized after all these years of using charts.
Reliability Engineer (2025 Edition)
This is a meme from 10 years ago. This is 2025 now, we're going to do a little bit of a refresh. I don't know about you, but whenever I operate a service, I imagine myself being the launch control manager at NASA, which obviously I am not. I imagine myself commanding a fleet, although I have to do all the jobs myself. That's how I like to think about myself. Society still see reliability engineer and operations people as people who keep typing in a bit of a garbage lot. Because it's 2025, if you're not using AI, then something is wrong. Either your boss is expecting you to be five times as productive by using LLM GenAI tools, or they're just quietly waiting for you to be replaced by GenAI 5.0. We'll see which one happens first. Reality is, still to this day, I believe, a lot of us, whether you're a reliability engineer or you are just a software engineer with actual responsibilities, many spend a lot of your time staring at dashboards, which look like this.
What Is a Line Chart?
Grace Hopper is my hero. One of the things that she said is, the most dangerous phrase in language is, we've always done it this way. I think we have a little bit of, we've always done it this way, with dashboards. What I mean by that, in particular, I'm picking my target here, is we've been staring for far too long at charts like this. You can say, no, I have a dark background, and I have the Viridis color theme. My chart is much better than this. Fundamentally, you have a line chart, or most of the time, you have a line chart. What is a line chart? It is a type of chart where you have all data points. What you do is you draw a line between each consecutive data point, and then you get a line. What exactly is that line joining activity? That's a form of extrapolation.
You're actually creating pixels on the chart that is not there. This is, I think, fundamentally why I have issues with line charts in cases where I think visibility data can be the most useful. Things get much worse. Again, if you're good at your job, you probably offer something that has more than one instance. You will have something like this. This is madness. You cannot make any sense of the data, even though it has a lot of color, a lot of pixels. It surely has a lot of lines, even though I cannot even tell where the lines are. Occasionally, it's beautiful. It's beautiful on the boundary between order and chaos, but only this. This would not be beautiful if you only have the second half to the right of the chart. There's a fundamental problem here, which is line charts make sense when the data is very clean. They don't make sense when the data is very messy. When the data is very messy is when there are problems, and this is when you want to make sense. This is why I think line charts is not the best one.
What Is Good Visualization?
First, let's establish some standards. We're not saying, we're not accepting the status quo, as in, we're not trying to be complacent here. We want data and data visualization to help us. What is good visualization? Good visualization makes the answer pop out to you. It shouldn't be so hard that you have to be extremely good at looking at data to get the answer you want. It should just speak to you. If that is the very high bar we're establishing here, then we can see where line charts may fall short. Same chart, now I'm asking a question. If this is a throughput chart, do you think the throughput is changing over time? You don't have to answer that. I obviously will have an answer later. That's question number one. Question number two, here are two charts showing a few different latency quantiles. You have the flat one at the bottom is P50, and then you have P99 and P99.9 at the top, which is very noisy.
These are from two different sources. They look roughly of the same shape, but not identical. How different are they? Is there any meaningful difference? Do I need to care about one behaving differently from the other, or it's just all in the noise? Can you answer that question? Finally, back to our trusted and true single line chart. In the middle, there is a red line where let's assume there's a deploy happening in the middle. Let's say this is something like a latency chart. Let's say it's a P99 latency chart. It goes up and down. As you can see, it's noisy. Is there anything meaningfully different between before and after the deploy? Can you answer that question? These are the motivating questions, and this is why I'm putting a spotlight on the cases where line charts, I think, are performing poorly. Let's see what we can do.
We have a wonderful meme of Brian my coworker squinting at something. When someone's squinting, something is up. In fact, I like seeing the looks on some of the most experienced reliability engineers. You can see in your head that they're stroking their neck beard and they're squinting, and then something insightful is going to pop out of their mouth at any moment. Then the person who's just a new grad coming out is like, "What are you looking at? I don't see anything." This is what we're dealing with. The better you are at your job, the better you are at squinting at some really not that informative charts, and then the more it feels like magic. It's great for job security, but I don't think it's very good engineering. What is my problem? This is what I'm trying to establish in my talk is that I think they can be distracting exactly because they create pixels that are not necessarily there.
Sometimes the pixels are real, sometimes the pixels are not. It's extrapolation, so there's some kind of assumption that's baked into the visual style. The other thing is, I didn't even bother telling you what x-axis is. Everybody knows. Everybody assumes the x-axis is always time, because this is how things have always been done. I'm also challenging this concept that do we really need time? So far, that's always there.
Why Are Line Charts Synonymous with Telemetry?
No good narrative is complete without a little bit of history. I think the fact that we landed on an enormous collection of line charts as the industry standards for observability is not a coincidence. It is, in fact, baked into the entire stack that handles the telemetry data that we deal with every day. This is the crux. If you think about observability, especially metrics, they come fast, and they come in large quantities, and they never stop. This is an extremely big challenge for any system builder who wants to build a system to host and potentially serve a company. Every minute, every 10 seconds, every 5 seconds, all your machines report back to you, and then you're better taking the data or you're in trouble. The design of anything that can handle that type of data has to evolve very tightly with the assumption that data is coming fast and furious, but on the other hand, you may not have new time series all the time because you only have so many sources, or so many instances directly to be run, and so many metrics that developers can't remember to bake into their software.
It's a very asymmetrical system where writes really dominate every aspect, from design to query language to presentation. We have about 30 years of dedicated time-series database and/or other databases that are suitable for this write-heavy append-heavy workflows. At this point, everybody has converged on the one true data format that will handle all the telemetry data, which looks like this. As you can see, there's basically two tiers. There is the tier that identifies what data, what type of metric you're collecting. There's a name. There's also a bunch of labels, you may call them attributes, whatever it is. There's the identifying properties about the telemetry you're collecting, but not the values. There's the second portion, which is the time series value themselves. Because it's always coming in only to like clockwork, there's a timestamp and there's the value associated with that particular timestamp for this particular metric. This is what I call this baked in.
This is an industry-wide assumption. Almost every database that you use to store metrics will have this as the core of their storage format. What that means is that it's very easy to do any kind of time-centric query. It is also not too difficult to look up metrics based on their properties and their name, but there's very little crossover between the two. You should see that that eventually end up mattering.
If that is the data format, and everything else flows from there. You will have a bunch of data and the data will have a timestamp and a value. The most natural way of representing that in 2D space is to have one dimension being the time and the other dimension being the value. Because when you have a lot of dots, it is disorienting, and time actually has a flow, it makes a lot of sense to connecting all the dots with the time flow, so your eyes can be led from left to right. You see this as the passage of time. This is helpful. You end up with a time-guided flow of values. Because software engineers are lazy, like where you can draw just one without the dots, why don't you do that? Also, not to mention, it is difficult. It's actually harder than you think to draw dots compared to lines, because if you have too many dots, you need to make them small.
If you have too few dots, you have to make them big, otherwise they're not visible. It's actually much easier for frontend engineers to just draw the line, which is how we end up with this one, what we call marker, which is a way of drawing data. This line marker become dominant because it ticks all the primary boxes and it fits the underlying data reasonably well.
Telemetry - Data Shape
What is better? What does better look like? I would argue, like most things in software, in particular where we have so much freedom, we should consider the underlying function when we decide a form. For telemetry, I think there are three things that are the most important. One is data shape. Do you have a lot of data, a little bit of data? Are they all fairly concentrated at a single value or are they spread way out over time? That's one set of things. The telemetry type, and there are several well-known types that we can go through one by one. Finally, what do you do with the data? I think that's probably the most important because that's where the money is. Line charts are not always bad. They're just bad when the data is very busy, such as this example. This example, it doesn't look obvious because this throughput data nominally, throughput data, it goes up and down quite a bit relative to the actual absolute value of it.
What can you do? You make it less busy. Here, it's a very simple, like since we're asking a question, does it change over time? Then let's just zoom out a little bit and then do a windowing operation. I'm sure you've done this in some capacity. Then you take a lot of the noise out. Then, by looking at this windowed average line, you can see, yes, the throughput is indeed going up a little bit. What we've done here is we've turned something that is a bad behaving data, very noisy data, into something that is less noisy and therefore it's more suitable for the underlying data presentation format versus the line chart. With this, we need a little bit more treatment because there are too many lines. You've all learned statistics in high school, hopefully. If not, there's some basic operations everybody can pick up in about five minutes.
Like you can take the min, you can take the max, you can take the average. It tells you a lot about maybe not every detail of the data but data shape in a way that you can act upon. For example, if you know the max, you know how to provision. If you know the average, you know how to look at the trend. When data is very busy, generally speaking, you need to do something to make it less busy, especially if you're using line charts.
Let's look at our second example. The problem here is that, again, I think everybody can more or less squint at P50 and be ok with it. It's like, yes, P50, it looks fine. It is the P99.9 that is the most noisy and therefore we're the least certain about whether something has meaningfully changed. There's like those two little spikes early on in the bottom chart. Is that meaningfully different? What about everything else? Again, you can apply the same thing. It's like if the data is not well-behaving and therefore the line chart is not working for me, can I make the data less busy? Then here we apply this local, this is an algorithm that can find the local mass. By combining the data of nearby timestamps, you can see, there actually is a shift of gravity on the bottom at P99.9, but the same thing is not true for the top one.
Yes, it seems like at P99 there is some difference which is not obvious in a very busy chart. Again, transformation of the underlying data to make the conclusion more obvious. Now, what about this one? Let's apply the same technique. Let's compute the average. Yes, it looks about the same. Is that it? What other techniques do we have? Here, I want to remind one thing, my line chart is at extrapolation. How do we look at data without extrapolation? We have a lot of data here. That is your good old dot plot. Just drawing every data point as a circle, no lines. It's bifurcated. Like with this chart, it's very obvious that something happened. Whether it's better or worse compared to before, it's up for debate, but it's not the same. This is masked by the busyness, by the existence of those imaginary pixels between data points.
Statisticians have this phrase, they say you should always look at the raw data. A lot of time when we look at line charts, we think we're looking at the raw data, but in fact, we're not just looking at the raw data. I have another example. This is a well-known thing amongst statisticians. It's really cute. You should look up Datasaurus, and there's a wiki page. You can read all about it. There are 13 charts, all have the same statistical summaries, like mean, standard deviation, but they look wildly different when you plot them. This is what people mean when they say look at the raw data, there's no extrapolation, just the data values themselves. Imagine if you were using my chart to plot this lovely T-Rex. You would be figuratively butchering the T-Rex. Think about looking at raw data as important, and when you do that, you probably should not be using my chart.
These three examples, what I'm trying to say is there's not one-size-fits-all, what is the best visualization for the data, even if we're nominally the same type of data. It depends on the property of the data. There's no golden rules, you just have to try. You try until you find something that makes the conclusion more obvious. The most important thing is trying and making it easy for more people to try different visual styles.
Telemetry Types
Then I want to say a bit more about telemetry type. This is easy. This is the opposite of data shape because data shape is always different, you cannot predict. The telemetry types, completely predictable because it's dictated by the underlying storage. We know exactly what those are. We really only have three types of data: counters, gauges, and histograms. There might be some other, some people allow just discrete values, but those are relatively in the margins. Most of the volume comes in these three. They have their distinct properties that we can take advantage of. Counters, I think, is actually probably the perfect candidate to be drawn with a line chart because it keeps going up and up. Between them, you can draw a line and get an idea of the slope, which is the rate at which the value changes. This all makes perfect sense, but what chart do you have that actually plot raw counters?
Maybe your total storage of bytes. Most of the time, when we're talking about throughput, when we're talking about error rate, we're talking about the delta between the nearby two values. We're not really looking at counters as counters themselves, we're looking at counters as how they change. When we take that differential, we're transforming counters into something else, which is called delta counter. If we do that, and this is the chart we think is ok to show for the raw counters, what is the corresponding chart that is correct if we're showing delta counter? The answer is this. If you have a straight line between the other points on the raw counter, you take the derivative, you will have a flat line on the first order derivative. If you really want a line chart, this is the correct line chart, not the chart between two values if you can take the difference between two counters.
Then the area under this, like the bar-seeming segments actually have meanings too, because the area is a cumulative sum, the quantity that in the previous chart will be the y difference between two data points, so that everything makes sense. With this, you can also apply that to multiple lines. We can slightly transform what we were doing with min and mass and average, and they will all have well-defined meaning. As a bonus, I also show standard deviation, which are these little bars. Now they look visually correct, because everything is a little bar. You can also talk about how variable the quantities are.
Now, let's move on to gauges. Gauges is a spot measurement, which means it can be really any value. There is no rule that it has to go up or down. There's another thing that's implied, but I think people don't think about it as much, which is, because it's a spot measurement and you read it once and then you read it again in a minute, there really is no saying what happens between those two readings. It could be way higher than what you read in both times, or it could be way lower. It's almost dishonest when we draw a line between the two readings and say this is where the value should be between readings, because you don't know. This is a case where I think strongly calls for plotting the data exactly as is, with as little extrapolation as possible. Here I did a little bit of a line, very faded, just to help guide the eyes and then making sure you know how the dots are ordered.
The fakeness of the line means you should not trust the data. I use the Bézier curves because I think it looks nice. Is the slope believable? I don't know, because that knowledge cannot be derived from the data itself. Where maybe you know something about the system, and you can say, this curve can never be this steep, and therefore I should change the shape of the curve. That is extra knowledge, that is not coming with the data itself. The busier, the more data you have, the scale tends to tilt towards removing as many extrapolated pixels as possible. In fact, you should tilt in the other direction, maybe combine some pixels. Even though this is still extremely busy data, it tells you more about what kinds of readings we're actually getting compared to the line charts. That has a lot to do with both the data type and also the data shape.
Finally, histograms. This is one of my favorite subjects. I actually worked on a derived version of HDR Histogram. I think histograms are severely underused and misunderstood by most systems that ostensibly care about latencies. Here's my side grunt. As you've seen in the histogram I've shown earlier, often histogram has extremely wide dynamic range. The P99.9 can be 100 times, 1,000 times higher than your average. Not capturing that detail at the tail is the bane of many reliability engineering problems. You should always be very skeptical of anybody who claims they have a latency chart that has only one line, because latency is a distribution. It is not a number. End of grunt. Let's say we do the right thing. We are measuring something that should be a distribution with a histogram, which means a lot of numbers to describe one thing. You usually have a bunch of buckets, and those buckets cover different value ranges.
Then you count how many values fall into each bucket, so you can later recall what is my P50, what is my P90, what is my P99.999, if you have enough data center. If you have data like that, one thing you can do is what we did before. Like, I care about the SLO I set for myself, so this is what I have. I want to plot the key quantiles as a line chart, subject to the noisiness of the line chart. Wait a second, if I extract a quantile from the histogram, isn't that a spot measurement? Doesn't that tell me what latency distribution I'm seeing at that moment? In some sense, we're staring at gauges again. The same rules that we applied to gauges, saying we should have raw data for what it is, should apply here too. Then you can see, this actually, it's still busy, but it gives you a sense of where are the masses of the data, much better than the line chart, because again, we don't play the extrapolated pixel.
There's another way of looking at it. If you look at this one, we very intentionally picked the colors for different quantiles. They go from orange to red, so there is a progression of the higher the quantile, the deeper the color. The values are on the y-axis. You can actually flip this around. You can say, what if I want to know what my P1 is, P25, P50, P75? Do I really want to look at 20 lines? I could just plot them all. You can flip the axis and say, ok, the y is going to be the quantiles. Now I need another mechanism to encode the value, and that can be done by color. We used a green, yellow, red color scale in this one. I think it fits people's intuitive understanding of what is good and bad, at least in this culture. Then we have the entire range of quantiles on display.
I call this the optimistic view of latencies, because the previous view, you're just seeing all red. Everything's bad, because you're staring very hard at the tail, which is something bad. If you're a reliability engineer, you spend a lot of time at the tail. Sometimes it's also helpful to look at the big picture. This is the big picture view, like most of the time things are ok, but things are bad at the tail. If you know the metric type, which you all do, I think there are some very strong hints of what is likely to be a good default spot to realize these. To this day, I think most of the projects out there don't take advantage of our understanding of what the nature of the data is. I think we can do better defaults.
Operational Questions
Finally, I want to talk about operational questions. If you're hired to do operations, chances are you're not explicitly hired to say, look at the chart and make sure you notice every dip or spike. That may or may not be relevant. What is relevant, and what is in the job description are things like, make sure you right-size the fleet. Make sure you have picked the right kind of hardware and configuration to run the software with. Make sure you don't push out a known bad deploy. Do some testing. You got to know what's happening before you push it into production. This is the job. If you look at these very common questions, I'm sure every one of you have dealt with at least one of these three questions at some point in your career. There's no time mentioned. The thing you're measuring against is not time, but it is load, it is software versions, it is all the choices you have to make in setting up your fleet.
This is where we are getting the least help from the underlying time series data that we get. Again, the fundamental limitation of the time-series database is that you have essentially two planes. You have your metadata plane where all the attributes and metrics live. You have your data plane where all the values and timestamps live. Then they do not really cross, which means if you want to do things about value over time, that's easy. Lots of operators do that. If you want to group values by labels across different metrics, that is still possible. You can group different metrics by label. There are some limitations here and there, but by and large, you can do it. It is when you want to group one set of values by the values of another set of metrics that things become basically impossible within the kingdom of telemetry storage. This is where we roll up our sleeves as engineers, like we can do things, or no, we can do things with the help of LLM.
Let's look at how you would get answers to these money-making questions directly. The first one, what happens to my SLO? Some kind of latency quantile that you decide is important to yourself. What happens to that if the load goes up? We need essentially two pieces of data. One is something about the load. Maybe this is, if your success rate is near 100%, it's equivalent to throughput. You get your throughput data. Then you have your SLO-related data, which is some kind of latency, histogram, a particular quantile, you name it. You take these two pieces of data, and then, remember, you've got to do the work yourself. Because load is often considered as continuous, we need to do some basic bucketing. We say, let's have some 5k QPS buckets or whatever, 500 QPS buckets. You bucketize them, and at this point, you can merge the latency histogram.
You've got the table on the right. This is a direct relationship between request or load and latency. Notice, time is not completely useless here. We are aligning the data by making sure their times didn't match. After that, we can forget about time, because what we really care about is the load and latency. This is exactly what we're getting. Then you can plot it using something that is not a line chart at all. You can say, these are my 5k QPS buckets, and these are the quantiles that I care about, because those define my SLOs. You get the direct answer to your question. If you want things to look green all the way through, you should keep that at fairly low load, maybe 20k or less. You know when you are at 50k or above, everything goes to hell. It's a very direct answer to the question. You don't have to think, when is my peak load? Let me go look at the latency during peak. Maybe today is a bad day, maybe let me look at yesterday. None of that, because you have translated the data directly to answer the question.
The second question, with a bit of practice with the first one, I think this should become a bit more obvious. If you want to compare between two versions, what you do is you get metrics reflecting those two versions. Again, you take the data from a time-series database. You group them based on the thing you're studying. You're studying against versions, so you group them based on version. Here, time almost doesn't matter, because all I care about is version. I'm just going to merge all the timestamps together, as long as they have the same version. Then, again, you can plot without time, because your question does not involve time. The question is, is there regression? You can look at all the quantiles and then see, do I have regression at any point on this chart? The answer is yes. The new one is strictly worse at every quantile before and after the change.
Here it's really obvious. You look at this like, ok, blue is better, because blue is to the left and higher. If you have read a CDF chart before, you know that it's better. If you have not, spend some time on it, CDF is useful. The answer is yes. Finally, if you use EC2, you have 100 options? Realistically, you have a dozen options. It's one of those questions, like, am I using the best thing possible? Everything else is more or less the same. You know what to get, you know you want a throughput metric, but you also want some data that doesn't even exist in a time-series database. Because we are wanting to do things ourself, it doesn't matter. We'll pull the pricing data from a dashboard, or maybe just by picking out some CLIs that does this for you. In the end, you get this table.
Then you say, let me find the data that directly relates to my question. I want to know which skew it is. I want to know what price it is running at. I want to know what kind of throughput it's giving me. I'm going to join these two pieces of data together to get, again, a table on the right that directly answers my question. Then I can plot it however I want. In this case, we just did a table.
This is an embarrassingly old LLM experiment with some of the open-source models. If your question was, how can I find out the best skew to run some open weights LLM model from 2023? Then you can get the answer right from this data. Again, it's not a line chart. There's no time here because the question is timeless. If the question's timeless, I would argue, the schema might actually be timeless too. You should be able to eventually get to a version of the data where time just fades. This is not new. This is extremely mundane. If you think about the data we get as time series, what if instead of looking at them as a bunch of time series, like a bundle of time series, what if we just think of them as columns? The one unifying thing about them is actually timestamp. If you collect all the data points at the same time, you can lay them out as each time series becomes a column.
Now you just have this very nicely looking thing that you probably have learned and forgotten in your whatever, college SQL database query class. That's the foundation of a lot of data analyses, the traditional way. I was wondering, what does pandas stand for? It turns out, pandas stands for panel data, which is exactly this, a DataFrame. Economics, statisticians have been looking at this thing for like 50 years. Nothing new here. If you think of data that way, then time becomes optional. You probably still need to do a lot of operations by lining things up, but it does not have to be always data. There is a version of it that you can hide all the time data and you can focus on something else. You can use tools that people have been developing in other fields that has not suffered observability for literally more than 50 years. A lot of the answer will become straightforward.
Call For Action
Here is my call for action. I think for a lot of the observability providers, just the thing to be said. Like, if you know the data types, there's a better default design you can apply based on the type of the data. Things like calculating the windowed average, the sum, the max and min should be very easily accessible. It shouldn't require me to rewrite my PromQL query to get a new time series. Because all the data is there, if you think about it. I've already downloaded all time series. The raw data is in my browser. It should be as easy as clicking a button or clicking a chat box, and I should get all of the different ways of visualizing things as lines, as fonts, with summary, without summary, and then just let people try so they can find telemetry. Finally, it's not the entire universe.
There is a lot of things people want to do that is not within the capabilities of time-series database, and it should be easy. There should be an interface that you can seamlessly export your data into something like pandas or whatever your favorite plotting tool, and then you can take it from there. To practitioners, my call for action is this. I think the biggest thing is know that your observability data does not have to die inside the time-series database that you pay a lot of money for. This is the reality is you have aggressive time, obviously, because it's too expensive. At the meantime, we're not getting nearly enough value out of that, because the data just sits there for a month, and then it gets garbage collected. It never resides in a format that is actually the most natural and useful for doing meaningful queries on this.
Knowing this fact, I think, is pretty much the most important thing, because in this day and age, you don't even need to be good at writing queries, visualizing data. I wrote half of this presentation with the help of GenAI. If you know there's something else that can be done, you can probably figure out a way of getting it done, and actually very quickly. Just know that you need to try a few things. How do you know what to try? Just go look at the last century, or at least the last 50 years' worth of literature, and then see what other people are doing. Economics, statisticians, people make maybe half as much money as you do, but know twice as much about numbers. Those are good people to look for. Finally, the goal is always to transform the data in a way that is suitable to directly answer the question, by not squinting. If you need to summarize it, summarize it. If you need to change the axes of what the data is indexed by, do all of those things. Once you arrive at a format that matches the problem, I think the answer will jump out to you.
These are all the charts that we did. I have a little startup called IOP Systems. These are all the charts we did using time series data, but look very different from what you generally look at on dashboards, because we dare to say we don't absolutely have to have line charts.
See more presentations with transcripts
