Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations Using Data Effectively: beyond Art and Science

Using Data Effectively: beyond Art and Science



Hilary Parker talks about approaches and techniques to collect the most useful data, analyze it in a scientific way, and use it most effectively to drive actions and decisions. This talk provides actionable insights about how to apply the lens of design thinking to help use data more effectively in our everyday job.


Hilary Parker is a Data Scientist at Stitch Fix where she focuses on what sorts of data to collect from clients in order to optimize clothing recommendations, as well as building out prototypes of algorithms or entirely new products based on new data sources. She is a co-founder of the Not So Standard Deviations podcast, a bi-weekly data science podcast that has over half a million downloads.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


I am so excited to be here. I've been really looking forward to this talk, and one of the biggest reasons is because I have learned so much from the group here. From transitioning from academics to Etsy, into the startup world, I feel like it really helped me develop as an art programmer, as a data scientist. So, my hope is that I can give back a little bit.

So, like Randy mentioned, I wanted to give a little context for my path and how I became a data scientist. Randy mentioned I went straight to grad school out of undergrad; I'd been kind of a math and science person my whole life. So, I got a PhD in biostatistics there. At the end I, you know, wanted to flee the ivory tower, as many people do. So, I went to Etsy in Brooklyn and I worked there for about three years. That was back starting in 2013 so that was under John Allspaw who I think is kind of a well-known person in this circle. So, I learned a ton there and then after that, I moved to the West Coast, came to Stitch Fix to a really data science-oriented company.

Just to give you a little context, I wanted to just explain Stitch Fix. How many people here are familiar with Stitch Fix, the product? Cool. Okay, so some people. So I wanted to give a little detail here because Stitch Fix is a really cool company to work at as a data scientist, because data science is part of the core product offering. So, the idea: it's a personal styling service, so you sign up for Stitch Fix and we assign you a stylist, and you get to fill out a profile and we take a bunch of information from you. And then you receive five handpicked items of clothing and we mail it to you, you can decide to keep what you like, what looks good on you, what feels comfy and send back the rest. So if you don't like shopping, which maybe there's people in this room who feel that way, it's a great way to keep your clothing current without having to go to the mall.

And so what's happening behind the scenes there, we do have human stylists, we have over 3,000 stylists who do that styling service. The way the data science is integrated is that we actually presort the inventory into what we think will be the best match with the person based on what they filled out in their profile and their behavior on the site, and what we've learned about them over time. And so, this cartoon here is that we have inventory, we do a lot of machine learning in order to rank these items so that the stylist is given kind of a limited slice of the inventory for that styling process.

So, I just wanted to give that background because in Grady's [Booch] talk yesterday, he talked so much about how machine learning AI is just one component of a system, and that's very much the space I'm working in, where machine learning is there. But it's really to augment this, this very complex system, looking at something that's deeply personal, which is the clothing you wear, which is really how you express your identity. My one-liner on this is that getting dressed every day is like the one form of artistic expression that we all have to do and probably feel a little befuddled from time to time. So it's really fun to work on those problems.

So, I want to transition to the content of this talk; so Randy mentioned that I have a podcast with a colleague of mine from Hopkins, Roger Peng. And he started off the podcast three years ago with this interesting question, which was that he was describing a class he'd had maybe 10 years ago or so, where he taught this intro to statistics class to people at the School of Public Health. So, it's a lot of people in the medical field or in the public policy, public health field. And he had a student come up to him at the end and say, "Okay, I feel like you've told me everything that I should not do, kind of what happens when I make bad assumptions, how statistical tests can fail, but how will I actually know what to do? I get a new problem, a new data set, and what do I do with it?" And so he describes going back to his desk and sitting down at the desk and getting a pen and paper out and trying to answer this question and he couldn't, I don't think it's an easy question to answer.

And so we started talking about this topic intermittently over the last three years, and so this talk is really an evolution of that thinking, and where I feel like we've gotten to for this fairly complex problem of how do you solve a problem with data?

Data Science is an Art

So, generally, when you ask someone about this, the thing that you start to hear is people saying, "Data science is an art," which is a very good way of waving your hand over the problem and saying, "You just figure it out." One of the things that's been really inspirational about being at this conference is hearing how much people in this audience are thinking about their field as a science. And I just kind of had to chuckle because it's like, oh well, meanwhile, the people who have science in their title are saying, "It's an art." So, I don't think any of us are actually that happy with defining our fields so it's just kind of comical. But, the idea here is that, when it comes to these more foundational questions about how to do the work, we don't really have a good way of articulating it.

Most data scientists don't know that much about art, so they just say, you'll have to figure it out on your own. And I wanted to give a few examples of this popping up in different places. So, this is a book that's by my co-host, Roger Peng, and a colleague, Elizabeth Matsui. And in this, the first line of this book is saying, "Data analysis is hard, and part of the problem is that few people can explain how to do it. It's not that there aren't people who are doing it on a regular basis, but it's that the people who are really good have yet to enlighten us about the thought process that's going on in their head”. So, people are doing this, but they can't really describe to us how they're doing it.

Another - this was from a talk called "Zen and the Art of Data Science Maintenance," which I thought was pretty funny. I don't know how many people here know "Zen and the Art of Motorcycle Maintenance" but it's like a play on that. So, in this slide, this presentation was given this year and the author was talking about how, data science is an art. There's so much about intuition and qualitative insights on the problem you're working on and I really like this line about you explore a problem through the solutions. So, you start to work on the problem and you end up discovering more about the nature of the problem before you get to a well-defined problem. So, you explore it through solutions.

There was also this article, "The Art of Data Science," where the author was talking about how the demand for right brain thinking in data science is increasing, especially in the era of increased automation, which we talked so much about at this conference. So in this era, "the need for 'art' of data science will be the increasing cry of business." So, yeah, this idea of art comes up over and over again. One of my favorite visualizations of this problem of what to do comes from Hadley Wickham who's a really well-known art developer; he does the "Tidyverse" for those who are familiar.

And so, the idea here, he has this graphic in his great book, "R For Data Science," where he talks about this process you go through, where you import the data, you tidy it, which is essentially getting the data into a database-like format. And then you enter this cycle where you're transforming the data, you're modeling it, and then you're visualizing it. And by doing that, you start to explore this problem and you'll uncover something about the data, then you'll visualize it again, and it's just sort of this playful space where you're playing with the data, you're discovering things, you're not necessarily thinking about the end result. Eventually, you come to a conclusion about the nature of the data and the problem you're working on and then you transition out of this playful space to the communication aspect. So, you usually want to deliver this one way or another, whether it's a report, whether it's a dashboard, whether it's an email to the CEO, like there's some end result and end communication in mind.

If Data Science is an Art, Why Don’t We Teach Like an Art?

So, one of the natural questions that comes up here, it's kind of obvious when you think about it, is that, "If data science is an art, then why don't we teach it like an art?" If we're going to keep saying that, we might as well put our money where our mouth is and start to look at how art is taught. So I mentioned I went to Hopkins, and Hopkins has a really world class music institute called the Peabody Institute. So, kind of like an n equals one study, I went to their website and I checked out what their areas of study are, like how is art taught in these, world class art institutes. And so what comes up from that is that obviously, the primary focus at a music school is going to be on the instruments and really mastering the instruments, you know, guitar, harp, piano. And the idea here is that, with music, your instrument is the language you're speaking, it's the thing that you're using to express emotion. And so having a really high degree of fluency and mastery of that instrument is critical for being in the musical field and mastering all the aspects of it.

But, you know, in addition to mastering the instrument, there are other aspects of music education. So, one of the big ones is the idea of music theory. So, there's a lot of properties of music, there's sound waves and so there's physical properties to them, there's cognitive, like you want to hear chords resolve, you want to hear dissonance resolve. So, there's a lot of theory around how music is made and what composes good music. So they teach that explicitly. And then finally, there's a big emphasis on composition. So, how do you take the instrument? How do you take the theory? And how do you compose music from those parts, from those tools in your toolkit as Grady was describing yesterday? I want to walk through an analogy that pops out to me, which is that when you have, you know, they talk about music theory and in data science and the data science world, that's really statistical theory, that's the theory that we're building upon.

And for the instrument, our equivalent of a musical instrument is really the programming language that we're using. And I could even expand that to the programming that we're doing to build the architecture necessary to solve data science problems. And then that last part, when we're talking about composition, the equivalent in the data science world is the narrative of an analysis or a data science system. So, what story are you going to tell with the data, how are you going to weave it into some sort of dashboard that illuminates you about some problem or analysis you're going to deliver to someone, how do you go about composing that narrative? So, in terms of my experience in the field, I wanted to start with the theory aspect. So, I just wanted to give a big check mark here. I don't think it's any surprise that when you're being trained in statistics, in any of the kind of online course on data science, the idea that statistical theory is a part of data science is not going to be a shocker for anyone in the field and probably anyone in this room.

The idea that machine learning, statistics is what composes data science is not a big shocker, and I was certainly trained extensively in this area. So I feel like we get a big check mark, we're good at that, and we can move on. In terms of an instrument, so in terms of the programming languages needed, I want to give us an X. I know the field's evolved a lot since I was trained and since I even entered tech; you know, it's such a fast moving field, especially when you're talking about statistical programming languages. I mean, there's so much evidence by all the tracks and the talks here that this is a rapidly evolving field. But still, I think when we're talking about data science, we're still stumbling in this area a little. And so the reason, as I mentioned earlier, why I'm so excited to be here is that really mostly, for this part, I want to talk about what I've learned from you all coming into tech and coming into the startup world from this very academic background.

So, I mentioned earlier, so the reason I wanted to give so much of my background is because I want to follow my career and what I've learned over time. So, I mentioned that I was at Hopkins. So, Hopkins was kind of an R shop, so I learned a lot of programming in R. And this was primarily for creating analyses, for papers, for consulting work with people in the public health field. And so I can only describe this part of my career … I came from a math background, a biology background, I did not have much training … I had one Perl [SP] class in college, but otherwise I did not have any training in programming. And so I can only describe this part of my career as walking through mud, where I felt like every time I started analysis, it was just like a complete deer-in-headlights moment of, "Oh, I have this data set, I need to make something happen with it," and fumbling around, writing code that I thought would eventually create the result I wanted. Even managing the data, knowing how to process it and pre-process it, it was a very frustrating feeling.

I imagine that's how many people in this room felt when they were first starting to program but it was even more so 10 years ago, working in R for the first time and being with people who also were all trained in academic statistics. So, I just want you to leave this slide knowing that it was extremely frustrating.

Oh, and I just wanted to mention that the reason … there was almost a resistance in that field at the time to teaching much about the coding process, because there was a real attitude that this was going to limit your creativity, and that you needed to do what the data needed. And so you would have any sort of conventions around how to code would limit you from being able to explore the problem. There's also a real aversion to- we call it "cook-bookery"- like using cookbook methods. I think that's a well-earned aversion where there's a lot, in statistics, we kind of have this love-hate relationship with things like hypothesis testing.

And you hear a lot, I don't know how many of you have heard in the general media about controversies in psychology around how many false positives or how many papers are really the truth. So I think statistics has a lot of aversion to cook-bookery so that was another reason why programming was not emphasized.


So, I felt like I was walking through mud, and then toward the end of my PhD, I came across a project called ProjectTemplate that I can only describe it as kind of Ruby on Rails but for analysis code. I'm sure that analogy has many shortcomings. But the idea was that it was an opinionated approach to structuring the code within an analysis. And I love this line, you probably won't be able to read it but the first line describing the architecture is that, "ProjectTemplate is based on the idea that you should structure your data analysis projects in the same way so that you can exploit convention instead of writing boilerplate code."

And so I found this to be awesome; it started to address that feeling of walking through mud and it was based on, again, things from this room and things from the software engineering world. And so I started to get a taste of what it felt like to be able to express myself a little more fluidly in the programming sphere. So, after that, I moved to Etsy and from what I understand Etsy is fairly famous in this community for the DevOps culture there. Again, John Allspaw was a really inspirational leader at the company. And so the idea of blameless postmortems became super, it was super, it was ingrained in the culture in a really fundamental way where people outside of the engineering org were also participating in postmortems, and I felt like I learned a lot there about postmortems. And I thought these were totally awesome as I'm sure lots of people here think, too, because it really was the first time that I started to understand how we could actually communicate about code in a way that didn't make everyone defensive and mad. The statistical community, I think, is fairly famous for some pretty vicious language wars between R and Python, for example. And that doesn't even get to the big data stuff and all the consternation there.

Good Analyst Vs. Bad Analyst

But one of the things that had been persistent was this idea of a value judgment of you, as an analyst or as someone writing analytical code, that you were either good and you didn't have errors in your code, and you created good results, and you had papers that weren't retracted because of some embarrassing error, or you were not good at it. And you didn't write reproducible code, you didn't care about the programming. So there was this toxic dynamic going on and, of course, it was on Twitter a lot, and I like to think of Twitter as a platform designed for miscommunication. So, it made any sort of language debate so much worse and, you know, I was alone, in grad school, and then at Etsy... Most of my discussion on this was on Twitter.

It felt like this blameless postmortem thing was a pretty good idea and just thinking about doing analysis, this slide, you do not need to read all this, but when I think about doing an analysis, there were so many things that could go wrong. The biggest one is you rerun your analysis, and you get different results and that is horrifying. That happened to me a few weeks before my dissertation and it was awful, and my eye was twitching all the time. I figured it out and it was my fault but... not my fault, it was the system's fault. But the idea is there's so much that can go wrong and it seemed like some people have this figured out and usually when they talk about it, if they talk about it, it's like, "You should use R," "You should use Python," and, "You should use this project management system," or, "You should use this testing, you should test your code." So, it started to feel like, hey, maybe we can, as a community, start to talk about this in a more non-vicious way.

And so, you know, this great quote, this great paradigm shift in the DevOps field, talking about how engineers who think they're going to be reprimanded or, you can even say "shamed" there, they're going to be disincentivized to talk about their solutions to these problems as well as the problem themselves. They're not going to want to give the details necessary to understand the mechanism, pathology and operation of a failure if they feel like they're going to be embarrassed, or they're going to be fired or whatever. And so this was really exciting and blameless postmortems, the whole paradigm, again, I think so many people here have probably gone through it; this allowed me to talk about these concepts in a way that didn't make people defensive, and allowed me to talk about these general principles, rather than digging into just the language war situations.

Building a System

And actually at this conference, and especially hearing Grady's talk yesterday, I felt like I had a little bit of a revelation that I think the reason this works so well is because any analysis, any data science system, any recommender system, any AI system, you are building a system when you do that analysis. So, this diagram here is a system. So even something as simple as getting a number for your product manager about how many people are using an application, that itself, you have to build a little mini system to do it. And the way you architect that will mean that it's either very prone to error, very prone to letting you down, or you're going to be able to get a result that you feel really confident about. So, I really like Grady [Booch], saying over and over that everything is the system and then those are macro systems; I felt like that was really illuminating.

So, I liked it so much that I wrote a paper for the statistics community on it, talking about... I called it "Opinionated Analysis Development," because I love the idea of opinionated code, I'd felt like it was really helpful. I also love the idea of talking about general principles, the idea of blameless postmortems but in this I made it really clear that I wanted to talk about the development of an artifact rather than the narrative, so I hedged my bets and left it out. So, that brings this last part of the data science world and that music analogy, which is the narrative of a data science problem, or of an analysis, really, an analysis, analytical narrative. And so, again, hopefully, not a surprise that I'm going to give an X here and say that this is something we definitely don't teach. And that gets us back to the "data science is an art" thing.

I'm referencing Grady [Booch] so much - I was really inspired by his talk, because he had this great line at the end, talking about, "Software is the invisible writing that whispers the stories of possibility to our hardware," and, "You are the storyteller." So, I was like, "Oh, you're facing the same problem as me where you've been told that you're a storyteller, that you need to compose narratives and I'm guessing that you haven't had a ton of training on that either." So, we both kind of arrived at the same place where we need to tell stories, we need to create systems that you have purpose and have meaning and work well, but we've not been told how to do that at all. So, I want to help, if possible.

Product Development

So, for this part of the talk, again, tracking my career and what I've learned from it, is transitioning over to Stitch Fix and being part of a company where the core product was a data science system. And what that meant for me personally was that I was much more involved in the product development process and specifically, we subscribe to a sprint-based product development cycle. How many people here know design sprints? So, I will describe them. Design sprints are something that was developed at Google Ventures by some folks who work there. And the idea is that it's a way of getting rapidly to a prototype of a product. And so it's a very cool system in my opinion, because they've gone through and for the startups that they funded, they would actually go to those places and try to get a scrappy prototype out the door in a week. So, they've gone to hundreds of places and they've really honed this process based on observing what was working and what wasn't from those experiences. And so the idea here is that you take the week and you go through, every day, you do a different thing. You map out the problem, you sketch it, you decide what to do, you prototype, and then you user-test at the end.

And all these materials are available online, I highly recommend checking it out. But to just draw on a few of the themes that come out here, what the design sprint is really focused on is this idea of interviewing stakeholders. So, really, everything is built off of bringing in people, asking them what they want to see from this product, what their current problems are, maybe some expert witnesses talking about the technology that's available, but it's really very much focused on these interviews. And then you craft solutions to their problems, you work alone together a lot. So, you're able to do your own idea, sketch out your own solutions, and then you come together. And the design sprint has a lot of structure around how you combine ideas together. You'd spend a lot of time validating each other, thinking about taking the best parts of everyone's sketch, and eventually building one approach to the prototype.

Then you create the prototype, and then you go back, you interview people, cycle continues. So I was able to work on one for a internal application and I left super excited because, first of all, I felt like I was very creative in it and I'm not someone who had traditionally thought of myself as a very creative person. So I was like, "I came up with a great app, that's awesome." And then I also just felt really energized about the fact that I'd been in an environment that had created a very special productive space for coming up with a new system. And I thought, you know, "This would be great for dealing with data science." There's no reason when we're crafting data science systems, even just a very complex analysis or an experimental design, why we couldn't do the exact same thing and bring in stakeholders and talk to them and intelligently build a system and a collaborative system.

So, this is a totally not staged photo from one of the design sprints that I put on for data scientists at Stitch Fix, where I've had the opportunity to do a few, I’m not going to act like they're perfectly honed at all. But by doing a design sprint for data science, we're able to start to think about how would we build a recommender system that really integrates into all of the use cases of the people within the company, the customers, how can we build a system that will actually solve problems?

And so I want to illustrate with just one very briefly; this will not be at all like a comprehensive dive into the system, but we were looking at one for size. So, as you can imagine, getting people's size right is extremely important to our business and it's also a hilariously complex problem. One of the things I learned, this is just a total aside, but one of the things I learned working here is that the way that they do size at companies is that they literally have one person, and then they bring their clothes and put it on the person. And they're like, "Oh, let's bring in the waist a little," … and then that's how they determine what a small is. And so that blew my mind because I thought they just had a bunch of measurements and put everything to those fit specs but they totally don't, which as a scientist, I was just like, "This is insane."

Anyway, the point is that there's a lot of complexity in this paradigm. And so we did these interviews; these are actual pictures from the data science design sprint we did, there's rules about, how sizes should fit, like where the seam should hit. There's a lot around the merch intent like, "Is it supposed to be an oversized shirt," that's kind of like the bane of existence, if you ever look at reviews for oversized clothes, everyone's like, "This was way too big," and it's like, "It was supposed to be big."

But, you know, this is a very complex problem, people give all sorts of feedback. There's brand loyalty, there's within the size, if you've ever put on two pairs of the same pants, they'll probably fit a little different, so there's variance within the size. So, these are just a few of the things that came up and one of the systems, this is really from Patrick Foley who's on the styling recommendations team, the team that I'm also on. He built this model that really elegantly solves some of these problems, where you have the size of the person, you have the size of the items, and then you have their interaction. So, the person receives the item and says whether or not it fit right, and we've continued to build out more and more ways to ask that in more and more detail, but then it goes into this big, black box algorithm for this talk. And then out of it, you can get predicting for a new item, whether or not the person will think it fits as well as some nice little side products of learn size for the client, learn size for the item.

So, you know, this, again, is just to illustrate- I actually have a link at the bottom here if you want to read more about it. But this is just, again, to illustrate the idea that we can craft data science systems that meet the needs of the users and meet the needs of the business in a way that it's structured. We don't just have to try to come up with this out of thin air.

Design is Not a “Mythical” or “Mysterious” Talent

And so I got really into design at that point, because I was like, "Wow, this seemed to solve a problem pretty elegantly," that is this big hand-waving in the data science world right now, which is that, "Everything is an art, it's an art." And so it's like, no, this isn't some mythical or mysterious talent, this is actually a discipline. This book is by someone named Nigel Cross, who I super highly recommend reading his work. He's one of the founders of design thinking, and the idea of design as a discipline. And so I've learned a lot and … Roger, my co-host, I started texting him lots of ideas, like, oh, my God, I took, I sent photos of Kindles, which he thought was really hilarious, with quotes up there like, "This is the answer."

And so, yeah, design isn't some mythical thing, it's actually a discipline. And again, to his point, people haven't been able to enlighten us, it just seemed like this mythical talent. But it's like, no, actually, we can approach this in a structured way and understand it better.

The Design Process

There were also a lot of other themes that came up from reading design thinking literature, that seem, again, very applicable. One of them was that design process is very solution focused instead of problem focused. So, there is an example of a study they did with architects, kind of in the design field, versus scientists where they said, they asked them to arrange blocks in a specific way. And they had some rules that were stated, and some that were unstated. And they found that the scientists were … they focused immediately on trying to define the rules that had been unstated, whereas the architects would just sort of rapidly prototype through and just try to get to a solution and they're much less focused on the problem, but focused on getting to a solution.

And so this, again, statistical training is all about understanding the problem, and all about diving into that deeply. And data science, I think, as a product-focused field is much more about the solution. And to bring that back, that gets exactly to that first foundational question of the talk of, "You told me what not to do but what if I actually want to know what to do?" Another thing that comes up in the literature a lot is that it's this form of constructive thinking. And Grady actually mentioned yesterday this oppositional logic or oppositional thinking, and the idea here is that you start building a solution in order to understand the problem. And again, that calls back right to that slide from earlier about you having to explore the problem by building solutions; you won't understand it just by sitting and trying to understand it.

The design process uses a lot of left brain and right brain. So, they actually do studies of the people with the split-brain syndrome or situation where that impairs their ability to do design work. And so again, going back to before, this is very related to what's coming up in data science discussion about how this is a right brain thing in addition to the left brain thing. This is my favorite point, which is that Nigel Cross has actually spent a lot of time defining design as an independent version of... an independent form of rhetoric, and specifically nonverbal rhetoric, where the language is really sketchy, so it's a very nonverbal form of thinking.

And there's this great quote that describes the importance of sketching that he has. So, "One thing that is clear is that sketches enable designers to handle different level of abstractions simultaneously. Clearly, this is something that's important to the design process. We see that designers think about the overall concepts and at the same time, very detailed aspects of the implementation at the same time.” And so what I love about that is that that describes this playful space here where we're doing a lot of transformation and we're doing a lot of visualization and modeling, and you'll be thinking about aspects of the data as well as the overall concept. I also think this is really applicable to system design, where you're talking about how many people are sketching out boxes, and how they connect. And I think that that itself is a form of this sort of design thinking, nonverbal rhetoric. And so to wrap up that section, what I love about Nigel Cross's framing of the field is that he says, "Design ability is, in fact, one of three fundamental dimensions of human intelligence. Design, science and art form an 'and,' not an “or” relationship to create this incredible human cognitive ability." So, to me, that's great, because it's like, "Okay, data science doesn't have to be a science, it doesn't have to be an art, it can be all of these things." And that missing component really seems to be the design aspect of data science problems.

Two Sides of the Same Coin

So, I want to wrap up this talk talking about these two constructs that I found really pivotal in my work and my development as a data scientist, and that's these postmortems and these design sprints and it's like, "Okay, well, what do these have in common?" And I actually think that these are really two sides of the same coin, where one is about building a system, designing a system and then the second is about iterating on it. So, I actually think they could learn a lot from each other; I think postmortems could learn a lot from the way that sprints are structured or design thinking in general.

And the other thing that's really in common between the two of them is that there's a lot of rules for playing nice with each other. So, there's a lot- blameless literally is like telling you, "You cannot blame other people in the room for this." And from what I understand, if you're in a postmortem, there will be like, oh, yeah, you have to, someone will stop you if you start to blame someone. And so the idea here is that with all of these constructs, there's really an idea of having empathy.

And I've defined it here by Wikipedia. So, empathy is this, you know, capacity to understand or feel what another person is experiencing from within their frame of reference. And so these design sprints and these postmortems are forcing you to be empathetic and saying, "Here are the rules, this is what it would look like if you were an empathetic person, and so please do these things so that you can at least mimic empathy even if you don't feel it inside."

And so I want to wrap up this by saying, I think there's a lot of... Oh, right, this is from Nicole and Jez's talk, and I really love this, I thought it tied in really well, where this idea of psychological safety, which it comes from people empathizing with you, and feeling like you're not alone, and that people aren't judging you all the time. That's the number one thing for successful teams. And so this idea of empathy isn't we're just all trying to be like mushy 21st century people; it's really critical for success.

And so what I want to say in this last part is that, I'm just going to guess in this room that a lot of people are probably like, "Well, I'm just kind of not an empathetic person," and certainly in the data science world, I've seen that a lot so I'm just going to generalize here. But, you know, I think I've just seen a lot of data science colleagues where this causes a lot of distress, because it's like, "Oh, I'm not an empathetic person, I'll never be great at my job." And so what I want to say in this last part is that there is hope and there's ways to actually- this is not a static character trait and it is something that you can develop.

My Experience with Meditation

And so this is where the San Francisco aspect with this talk comes in, and I had Randy talk about where I live in order to introduce this. And so what I want to talk here is about my experience with increasing my capacity for empathy, and my capacity to understand other people. So, I live in this beautiful building, the Zen Center. The dashing tall man here is my partner, Michael. So he's a Zen priest and so I've had this intensive immersion into the world of meditation recently with Zen Buddhism in San Francisco. And so I want to describe the theory there; it's not a dogma-based religion at all so I don't want that to come across. But the idea with meditation in Zen Buddhism is that you approach things with a lot of curiosity, and a lot of non-judgmental observation. So, you literally sit and stare at a wall for hours a day, and you notice what arises.

And so one of the things, talking earlier with Randy, is that one of the things that a lot of people in this room probably experienced is imposter syndrome. So, the idea with meditation is that you sit there and you are staring at a wall, and maybe this horrible feeling of you're not good at your job, or you're not good enough to be giving a talk arises, and the practice of meditation is that you would sit there and say, "Okay, that's interesting that that's coming up," and you're not scolding yourself for having it. You're not saying that this is bad, but you're just saying, "That's interesting," and, "Can I be curious about this?" and, "Can I try to understand why this is happening?" And you don't necessarily follow that analytical thing in the moment, but as you meditate more, you'll archaeological, you'll have an archaeological dig, where you start to understand a little bit more about why these feelings arise in you. And the idea here is that you kind of stare at the monster under the bed, and the things that are making you feel bad and then you're actually able to see that it's just the kitten, or it's not as scary as it seemed.

And so really it's an acceptance practice and the central tenet is that you start to learn to accept yourself more, and the feelings that arise, the feelings of inadequacy, the feelings of rage toward your coworker for using Python, whatever is coming up, you start to accept that with yourself and then because you've built that reservoir, you actually can start to accept others more. And so I'm not trying to sit here and say that people should do Zen Buddhism but what I want to emphasize is that I've personally experienced and I think it's possible to build this capacity in yourself. And the reason that's important for this job is because we've seen over and over that that's a central part of all of these systems that we seem to buy into and think make us productive. So, I don't want you to sit here and think, "Oh, great, I have to be a storyteller and I have no clue how to do that and I don't know what to do". I think that there's a lot that you can do for that problem. And again, this is empathy with your users, empathies with co-creator and empathy with yourself. So, I'll end on that note, so thank you so much for your attention.

See more presentations with transcripts


Recorded at:

Nov 28, 2018

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Using Data Effectively

    by Manish Sharma,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Thanks, Hilary such a nice article post. I read your article it is very helpful for post finally I got it this type of post. Again I will say thank you

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p