Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Presentations How Big Tech Lost Its Way

How Big Tech Lost Its Way



Andy Walker discusses his experiences of seeing things change from the inside, how big tech makes decisions and what can be learned from how we got here.


Andy Walker has spent over 30 years working in tech companies from the early days of the World Wide Web with Netscape through to Google. During that time he’s built teams and products all over the world. These days he confines himself to writing about his experiences for the next generation techies and doing the odd bit of coaching.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.


Walker: My name is Andy Walker. I'm here to talk to you about how big tech lost its way. I'm doing this particularly in light of the recent layoffs, which have been announced across the tech industry. For those of you who haven't been reading the newspapers, at the start of this year, companies like Google, Meta, Facebook, and Microsoft laid off large numbers of people without warning. They did it mostly by just removing the people's access and telling them not to sign up to work anymore. People found out they weren't employed anymore, simply because they'd lost access to their work accounts. Some people lost access while they were in customer presentations, other people lost access while they were on maternity leave. This is a pretty big departure for an industry which really sees itself as making the world a better place. Today's talk is about how that has come to be. Also, to try and understand how a company can start off with such high aspirations and then suddenly find itself over a period of time coming to one which has really lost its ethical compass. It's also worth understanding this both in terms of how we can prevent it in our own existence, but also the wider implications it has for society as a whole.

A Missed Opportunity or Business as Usual?

I spent 10 years working at Google. A long time before that, I spent some time working at Netscape in the early days of the internet. Some of my examples are over quite a long period of time. If you think about it, you can look at this very much as a missed opportunity, because here are companies which are making absolutely crazy sums of money. If you look at it between the 4 companies I mentioned, they made $150 billion profit over the last year alone. If you want to put that into context with my favorite metrics of how much money you will have earned a day since the birth of Jesus Christ, then that's $200,000 a day since the birth of Jesus in a single year. That's just profit, let alone revenue. That's including Amazon, which doesn't even seek to make profit, because it's all about growing its business. If you think about some of the founding charters of these companies, at the beginning, Google said it wanted to make a very different kind of company. A company which was really invested in its people. A company which took its profits and invested them back to make society a better place. Suddenly, we found ourselves in a situation where not only are these companies not investing their profits back and making the world a better place. In fact, if you look at Google this year, they're looking to invest the $70 billion of profit they intend to make, back into stock buybacks, which is not making the world a better place at all. The question is, how did they lose their way? Is this inevitable for any company when it reaches a certain size? What's the journey taken from the early days where the company was genuinely looking to make huge improvements, to where it went about building its data centers, the way it went about looking after its people, really did set the tone for the industry, and a lot of people followed. Now, suddenly, it feels a lot like any other large corporation, like an oil company, or big tobacco. We need to understand how this thing can happen so that wherever possible, we can't be part of it.

Ford vs. Dodge

This isn't a new thing. In fact, if I go back to the 1920s, there was a very famous lawsuit raised by the Dodge brothers against Henry Ford. Henry Ford had perfected the production line, which allowed him to deliver cars at about 10% the cost of his rivals. In doing so he found himself with a near monopoly of the car market, because nobody else could compete with the prices he was able to sell his cars for. Two of his shareholders were the Dodge brothers who were looking to start Chrysler. They really wanted to get dividends from Ford, so they could invest it in their car company, but Ford wanted to invest it back in his company. He wanted to invest it back in his people. He wanted to invest it back in his customers. Actually, what happened is the Dodge brothers took him to court and won, and he wasn't able to do so in the way that he wanted to. This highlights one of the biggest problems that all publicly traded corporations face and that is one of fiduciary duty, which is the company needs to act in the best interest primarily of its shareholders at all times. Frequently, this is interpreted to mean that the company's duty is to maximize profits. In fact, many activist shareholders do see it in exactly those terms. As long as you can prove that you're acting in your shareholders' interests, there isn't a lot your shareholders can do against you. If you go back to friendlier companies like Google, and then even Meta, they went to great lengths to ensure that the founders would retain control of the company, even as the equity in the company became diluted over a period of time.

The Innovator's Dilemma

As a company grows, there's also one kind of thing called the innovator's dilemma. This is the companies which are in the best position to actually make use of new innovations are the ones least able to do so. There are a number of reasons for this, but primarily is the fact that when you're trying to adopt a new technology, if you already have a large established business, that technology probably isn't a perfect fit for that business. If you're operating as a large established business, you look at the return you're getting back for that particular technology, and it doesn't seem to be worth it compared to doing business as usual. In fact, when you build a very large, profitable business, you even start running into some fairly perverse incentives around not only not innovating, but trying to prevent other people from innovating.

To give you a couple of examples of this, let's go back to the very early days of the internet, and the emergence and disappearance of Netscape. When Netscape was founded, Bill Gates famously sent out a memo saying a new competitor has been born on the internet. Microsoft's response to this was quite telling. First of all, they realized that they needed a foot in the door, an entrant in the browser wars. To do this, they invested heavily in Internet Explorer. By about the time Internet Explorer 4 came out, it was probably at least on a par with Netscape Navigator. In fact, Netscape took a step backwards between Navigator 3 and Communicator 4, as they completely rewrote the code base. At the same time, as this happened, Microsoft also decided not only did they have a competing product here, which was comparable, but they were going to give it away for free. They started bundling Internet Explorer into Windows and effectively destroyed Netscape's business overnight. What followed next was Netscape was bought by AOL. AOL stopped caring about the browser. In fact, what happened was, the browser was open sourced, which then eventually led to the creation of Firefox and Internet Explorer's downfall.

Now having established this position of market dominance with Internet Explorer, Microsoft stopped innovating. They actually started building things into their product, to make it deliberately non-standards compliant, so that other people would not be able to get into their marketplace. Therefore, they would maintain their lead. Eventually, this burned them, because what happened was, as they built more non-standard features, they had to backwards compatible support all of these. They stopped trying to build a better browser, because they thought the battle was won. When Firefox and then Chrome came out, and were significantly better offerings, Microsoft was in a position where they had a lot of tech debt to work around, and were playing catch-up. It's only recently that their browser with the rewrite for Edge has started to become comparable again to the best browsers on the market. In actually innovating to get rid of a competitor, Microsoft then started preventing innovation, and prevented themselves from making things better, setting the tone for companies like Google, also Apple with Safari to come in and build a much better product and actually start taking the mantle away from them and disrupting them. This caused a lot of problems for Microsoft in the early 2000s.

If you take another example of the innovator's dilemma, the iPhone. It turns out that Steve Jobs really didn't want to make a phone. He was very happy with the iPod. In fact, iPod, in 2007, represented about 40% of Apple's total revenue, and he didn't want to get rid of a product he already had. He really didn't want to work with the telcos, who had a disproportionate say over what handset manufacturers actually did and how they configured their devices. The iPhone actually started as a skunkworks project, which was hidden from Steve Jobs until it was ready to actually show him what they could do. Even then, it was only reluctantly that he disagreed and committed with his team to release the product and actually go live with the phone, which completely changed the handset business. One of the interesting things about the innovator's dilemma is that as companies grow, not only are they stopping innovation, but innovation can only happen in an environment where people can actively disagree with their leadership. If you've just fired 50,000 people as Google, Meta, Microsoft, and Amazon have done, you are putting a fairly large barrier in place to people in future disagreeing with their leadership team. Which means innovation is not going to come from the people with ideas, it's going to come from primarily people who are able to disagree with people, which is the leaders of the company. This is a really big problem.

Implications of the Internet

While we've been here before with companies, never before have we had a technology which has had the reach of the internet. As of January 2023, it was estimated that 5.1 billion people have access to the internet in some form or another. That is insane. Not only do they have access to the internet, by having access to the internet, they have the ability to potentially talk to 5.1 billion other people. While in many cases, this has been incredibly powerful as the networking effect has taken hold, we haven't really been able to keep up with the implications this has had on society. We really need tech companies to be innovative right now, because the sheer scale of the internet has led to a number of things becoming true. The first of which is that regulation cannot keep up with innovation. We're talking here about the problem that there are no effective laws for moderating the internet. Technology is moving at such a pace that government regulation is so foreign an instance, that it has become something of a Wild West. You find yourself in a situation where it is impossible to hold people to account on the internet, except for a few very rare cases, for bad or malicious behavior. Because there are no transnational treaties on what to happen when somebody in your country is operating a troll farm, for example, which is causing harm to somebody else's. There are no recourses set in place for people manipulating each other's elections by creating hundreds of thousands of fake accounts to spam spurious information at each other. There are very few mechanisms for actually holding users to account. In some cases, someone may be prosecuted if they're in the originating country. Even then, mostly, what might happen is you lose your account. Creating a new account on the internet doesn't take very long indeed. It's very hard to prevent people from impersonating anybody, or reestablishing a presence at scale, no matter what you do.

We find, because of this disconnect between law and the technology, nobody wants to be responsible for it. If you talk to government, they will tell you that the tech companies need to effectively regulate and police themselves. If you talk to the tech companies, they'll say that they're just following the law, and they will comply with all local law enforcement requirements. The net result of this is the users who are stuck in the middle of this, this lack of responsibility means that it's very easy to get away with a lot of bad things on the internet. This is further not helped by the fact that we've got this idea that freedom of speech is a basic human right. While in some cases this is true, we fail to tack on the thing at the end of that, which is, while freedom of speech is important, people should also be able to be held accountable for the harm that they cause by their speech. You can see people on the internet now having incredibly profitable careers and business models, through spreading disinformation and effectively hate speech. Nobody wants to deal with this.

If you go back to the nobody wants to deal with it government versus tech companies thing. On the government perspective, this is too much of a hot potato for any democratically elected government to handle, because they don't want necessarily to be getting into a game of what people can and can't say on the internet. They are very nervous about passing any laws which would require people to be held to account. On the other hand, if you're a tech company, you absolutely do not want to be in the business of being the moderators for what constitutes acceptable speech on the internet. It's not just necessarily acceptable speech on the internet, as well there's disinformation. Throw in some of the developments with generative AI which have come through, like for example, ChatGPT, and you find yourself in a situation where it is possible to generate convincing sounding fake information at industrial levels now. As those things start becoming more widely adopted by malicious actors, we'll find ourselves in a situation where the internet becomes less and less a source of trustworthy information, and more an echo chamber where our own views are reflected back at us.

We also have the misconception that the people we're building for on the internet are good people. One of the conversations that I've had a number of times is around, we should just assume our users have good intent, and that will be fine. If you look at it by number of users, then over 99% of users are trustworthy users. The 1% of bad actors are so harmful and are able to harm at such a scale due to the scaling nature of the internet, that the harm they do is disproportionately large to the number of them. When Google Maps was looking to increase the number of people being able to edit maps information, because keeping that worldview up to date without an authoritative data source turns out to be hard to do at scale, this was exactly the argument which was used. What happened was, they found that while 99% of people making edits were doing so in good faith, they accounted for less than 1% of the edits being made. Because the people who were making malicious edits were doing so at such scale, because they had incredibly profitable business models from doing so, that it completely drowned out the value of the useful single unit they were getting, to the point they had to switch it off until they had some proper safeguards put in place.

As somebody working in big tech, you're going to have a conversation along the lines of how much abuse is acceptable, because it turns out, you can't stop all bad behavior on the internet. All you can do is try and skim off the worst of it, and figure out where you draw that line. Again, if you have companies which are earning hundreds of billions of dollars a year in profit, and they would rather choose to invest that profit in to their shareholders, rather than protecting the people who are using their platforms and making them profitable, then you have to consider the short-term thinking that's going on here to enable this thing to happen. Because long term, this is not a good business strategy. If you even look back five years, the internet is a much darker place than it used to be. It's getting worse every single day.

One of the reasons that also big tech companies don't want to handle their bad users, is because they measure success the wrong way. If you ask a company for its success metrics, it's going to start to talk to you about daily active users and growth. Because if you want a quiet time on the stock market, the simplest way to do it is to continue growing year after year. They chase after these metrics, which we can refer to as vanity metrics. If user growth and engagement growth are the way you measure how your product is successful, all that winds up being is you have a very wrong set of incentives for handling abuse, because abuse is engagement, and it is usage. Not only are big tech companies failing to reinvest their huge profits in how they handle abuse. They have an incentive not to do so, because if they do so their vanity metrics are going to look worse.

Cultural Entropy

How do we get from a company which wants you to make the world a better place to one which is largely happy to turn a blind eye to all kinds of bad behavior. I'm going to introduce you to the concept of cultural entropy. This is how a company's culture degrades over a period of time. The default state of the universe is entropy unless you're pouring energy and organization into it. Company culture is the same thing. Because if you're not actively promoting or reinforcing your culture over a period of time, then what will happen in a publicly traded company is that company culture will start suffering from entropy to the point where fiduciary duty to the shareholders becomes the primary concern, rather than also feeling duty of care to the customers and users who are also relying on that company.

It Begins and Ends with Leaders

To go to another organization, which I have a massive amount of respect for, NASA. Gene Kranz was head of mission control when the Apollo 13 disaster happened. Back at the turn of the century, he was asked about where NASA stood today. His quote around NASA's ability to tackle big goals really resonated with me when I read it recently, because if you look at NASA when they did the moon landings, the Moon lander computer was not very powerful. In fact, your mobile phone is 25,000 times more powerful than the computers running the Moon lander. Technology has progressed so far since the 1960s and 1970s. Yet, we haven't really made any progress at landing people on the Moon. We haven't really made a huge amount of progress in space. Some people may say launching an electric car into space playing music into the cosmos is progress. I might disagree with you there. In fact, if you look at some of the ways some of the people who founded companies are reinvesting their profits into going into space, you might also question their ego in doing so. It just completely stands out. You look at the resources that a company like Google, or Meta, or Microsoft, or Amazon have, and they have incredible talent. In fact, they've gone to massive lengths to hoover up the best people they can find to the extent where, in some cases, they have destabilized the job market by offering so much money to people, and making it so hard for other people to hire. They've got the technology. They have incredible setups. The ability to compute at scale has grown at such a rate, it's astonishing. Yet with all of this, they are unable to make appreciable progress on the problems which really matter.

I think it's down to the leadership of these companies. Because if leaders are unable to have somebody come to them and disagree, and really keep pushing the iPhone, unless they're turned down, even when they're turned down over again, then nothing really happens. When you have hired so many people, and hired so many people into leadership positions, then it's very hard for people to actually disagree with their leadership. Leaders remember their time working through the ranks, where they were solving problems, and they miss that. They haven't realized that their job has changed from being the ones proposing ideas to being the ones who are the arbiters of the problems and the outcomes, which are looking to be achieved by their people. In doing so, they act as a dampener on the ability of their organizations to innovate, and so that you find companies degrading in very predictable ways.

Personal Ambition Replaces Mission

The first of which is that personal ambition replaces mission. I'll go back to the founding missions of some companies. Google, organize all the world's information and make it universally accessible. Facebook is connecting the world. If you actually reexamine what the companies are doing today versus those missions, and you'll see that there's a pretty big drift from what the company set out to do, and what it's doing now. You look at what galvanized NASA in the '60s, being the first people to land on the moon, and where they are now, which is they're not really quite clear what they're trying to achieve. They have some big goals, but they're not as unifying and as impossible as the goal they set out on in the 1960s. Once you lose that sense of mission, then ambition takes over. What tends to happen is you hire a bunch of really ambitious people. There are a limited number of really senior roles for these ambitious people. What will they start doing? Getting promoted based upon impact, as you become more senior, takes a very long time. If you're looking to demonstrate even going to staff engineer at one of these companies, you need about 18 months of demonstration that you're operating at that level and the problems you're working on are stable enough to be able to solve over that time period. If you're looking to go from that to director level, it's order of 3-plus years. If you're looking to get to VP level, it's even longer.

Ambitious people have one thing in common, which is they tend to be quite impatient. Rather than trying to solve these problems, they start building organizations. They start hiring lots of people in from the outside. As they hire people in, they need more management, and they don't necessarily have people ready to go into management. They start packing up layers between them and their people with middle managers, so that their organization looks the right way so that they can get promoted to the next level. This happens a lot. By doing so, what they do is they create a management chain, which doesn't know how to work with each other. I was talking to a friend of mine back at Google, and his reporting chain looks like him, Director, Director, VP, VP, VP, Senior VP, and then the CEO of the company. Every single person between him and the CEO of the company is trying to add value wherever they can to show that they can move to the next level. Which means that decision making takes forever, because you have to have too many people in the room who are used to being the decision maker, who are all competing to be the ones making that decision, which in turn absolutely kills innovation stone dead because it's like trying to swim through concrete.

Novelty Is Confused for Innovation

Novelty gets confused for innovation. It's very hard when you are a very successful company to determine whether your new feature is any good or not. I'll give you an example. If Google made a system for designing cardboard flamingos, and linked it off of the search page, they would have tens if not hundreds of millions of users of people overnight. For a startup, that is extraordinary success. For a company like Google, all it proves is that people click on stuff. It doesn't prove their features are any good. It doesn't prove they're making their product special. It doesn't prove they're actually solving a problem. What happens is things get released, they get a lot of press attention, and over a very short period of time it looks, "Look at all these users?" That success isn't sustained, because they don't know how to measure whether their product is any good or not. Because the way they measure it is through the vanity metrics of number of people using it. Because of this, things which are novel get lots of press attention. An example which springs to mind, which is quite topical is the current so-called AI wars inside search. I have yet to see an AI search integration which makes the search experience more useful for me. I have lots of ones which are a lot like watching an ongoing car crash, ones which are slightly creepy, ones which, as somebody described it, are mansplaining as a service. It's not really making the process better. In fact, this is the innovator's dilemma encapsulated, because there probably is a way to make an incredible AI search experience but tacking it onto something you already have when the AI part of it isn't mature enough, only makes something which was very good to be something which is a bit there, and isn't making the thing better. Because so many people are already using those systems, this can easily be perceived as being a wildly successful feature. If you actually look at what makes a good search experience, you would have to argue that this isn't the case.

Executive Rewards Go Up, Accountability Goes Down

Executive rewards go up, but their accountability goes down. Let's take an example. Sundar Pichai, at the end of last year, the CEO of Google, got granted $200 million of equity over the next few years, $120 million of which is directly tied to increasing shareholder value. If you're a company like Google, which doesn't pay dividends, increasing shareholder value basically means one thing, which is improving the stock price. If you actually look at it, now, when Google fired 12,000 people, their stock price over the next few weeks went up by around about 15%, maybe 20%. If you're Sundar Pichai, you've just become $30 million to $40 million richer. You have a financial incentive for doing things that the market is going to react positively to, even if it's not necessarily a good long-term bet for your company. Accountability goes down because it's very hard to keep people in executive positions to account when you see cases of people engaged in harassment of their employees, or inappropriate relationships, or bullying, and you see those people getting rewarded over again. You wonder what's going on. The problem is, is that the more senior somebody becomes, the harder it is for their companies to hold them to account. Because very few people are going to speak out against a VP. To performance manage somebody, you have to have the very direct support of that person's manager to actually run through the performance management process. When we get to director and VP level, these are all very busy people who would rather be spending their time on making more money and doing better than actually holding people to account. Therefore, it tends to get brushed under the carpet in a way which simply doesn't happen to people at entry levels within the company.

Accountability Becomes a Blame Game

While executive accountability is going down, everybody else's accountability is going up. You start finding performance reviews becoming increasingly punitive. Leadership start saying things like, we need to hold our people to account for delivering these things. Rather than, we need to create an environment where they can be their best and succeed and reward them for it. Then you start finding horrible things where people sat in rooms where you have to manage to a performance distribution, or a certain number of people need to be fired every year. This doesn't result in an innovation culture, this results in a culture where people will do what they need to do to stay in a job, and not a whole lot more. At the end of it, you wind up in a situation where people become expendable. Where the CEO of a company, or, in fact, many companies, because if you look at the CEO emails of all of the tech companies who have laid people off over the last six months, they could all have been written by the same ChatGPT conversation. They all read along the lines of, "We hired too many people during the lockdown, and now business has changed, so we need to get rid of those people. We're very sad that we have to do so. We take full responsibility, even though our stock price has gone up, and we've just become millions of dollars personally richer, to the extent we're probably looking to buy solid gold furniture. We feel really bad about this. This is the hardest decision we've ever had to make." As a result, people working for these companies, rather than feeling this is the place they're going to do their best work. They start wondering when the next hammer is going to fall. They realize that their company is perfectly happy to move on without them. In fact, their company won't even do them the simple human decency of a conversation to explain why it's happened, because they'd rather not take the risk that somebody might do something bad. If that's not telling your people that you don't trust them, I do not know what is.

Corporate Takeover

Then, corporate BS takes over. People start coming up with meaningless phrases like, we're an AI-first company. People are unable to answer questions directly. Townhalls stop becoming about honest and vulnerable engagement from executives, and start becoming exercises in box tickings, that we've given people an opportunity to be upset with us. Now we're just going to go on with what we're trying to say. A friend of mine was telling me about an initiative to change people's desk spaces to be hot desking in one team, and it was phrased as workplace evolution. Which, if it's not trying to gaslight people, I do not know what is. Then, because people don't like living in an environment where they have little certainty, people fall back on process. Leaders impose process so that they can centralize dashboards of everything and manage it. Because if you can't measure it, you can't manage it. People fall back on processes because they're living in a world where they have very little decision-making power. Therefore, they try and use the process to preempt decisions being made against them, all of which further slows things down and destroys innovation.

What We Can Do

What can we do about it? Not a whole lot, it turns out, but there are some things which we can all do. First is, start with why? Understand what problem you're trying to solve. Understand what success actually looks like. Success is not chasing after vanity metrics. We should stop seeing jobs as a family. Your employer does not care about you. They will go on without you quite happily. Your team is not your family. We need to be understanding that we're building systems for people who aren't very nice, and we need to protect other people from them. We need to continuously invest in our culture, because if we stop investing in who we are, we forget it because we hire and we dilute it. We need to flatten our hierarchies so that decisions can be made by people doing the work and it's clear how to get things done.

What Companies Can Do

What can companies do? Continue investing in your culture. Promote leaders from within wherever possible. Look at business models which work for everyone, not just your shareholders. Work with government on regulation because it needs to be a partnership. One of the best sets of tech laws over recent years is GDPR. That was an active participation by government and technology companies. In fact, to the extent where government was slightly concerned of how involved technology companies were. Reaffirm your mission. If it's not relevant, you may need to change it. Understand that your leaders' responsibilities grow as they become more senior, and you need to invest in them. Please stop cargo culting. You don't have to continuously grow your workforce. Learn to do more with less.


See more presentations with transcripts


Recorded at:

Sep 13, 2023