Transcript
Mallika Rao: I'm very excited to be talking about one of my very close to heart topics, adaptive recommenders. When people hear the phrase, recommendation systems, typically they're thinking about models, maybe the stages that go into it, like ranking, retrieval systems, embeddings, your stores, offline, metrics, and all of that good stuff. Maybe recently, perhaps large language models. After spending many years in the area of search and discovery, personalization, recommendation systems, candidate generation systems, what I have realized and come to be convinced of is slightly different. The hardest part about building recommendation systems and engines isn't building just a model. It's so much about building the system and everything that goes end to end in building a system like that that can continuously learn, adapt, evolve, and actually deliver that value, while operating under real-world constraints like latency, cost, observability, experimentation, customer trust, compliance, and the hard constraints that really make great products. That's what I want to talk to you all about today.
Motivation
I want to start with a simple observation. Like many of the conversations that we are having around AI today and most of the conversations that have been happening over the last year, a couple of years, actually are the conversations that our recommendation systems have been having for quite some time now. What do we talk about most recently? Feedback loops, evaluation, online learning, agentic behavior, and how do we really operate under uncertainty? I feel like recommendation systems have been dealing with that for quite a few years now, long before generative AI actually became mainstream. That's very interesting is as AI is becoming more adaptive, the lessons from recommendation systems are becoming broadly applicable. That's why I feel I'm picking recommendation systems, but what we will actually be talking about today is really adaptive systems. Just systems that can learn, adapt, evolve, and in production constraints. Doing that successfully and consistently requires a certain kind of rigor. Hopefully, we'll be able to touch upon some of those things.
Context
If we go back several years, like many of the recommendation systems looked something like handcrafted heuristics. We had our business rules, and then we had carefully fine-tuned business logic, ranking functions, and surprisingly, most of those systems worked really well. They also had their own limitations. They struggled with sparse signals, evolving user intent was really challenging. They struggled with scale. They struggled with adapting quickly. Today, many systems have actually evolved towards what we call as AI native architectures. Embeddings help us understand those semantic challenges and similarities much more contextually. Also, contextual signals themselves help us adapt to changing intent. We also have multistage retrievals going on these days. Ranking pipelines definitely help us scale there. Increasingly, these systems are becoming capable of adapting more and more quickly to user behaviors. That evolution is very powerful, if you think about it. It also introduces a new class of operational challenges. Things that we didn't have before, especially with quickly evolving user behavior as you're interacting with your product, that is something very new.
Before we go any deeper, I want us to establish some sort of a common mental model. What happens during a recommendation request? A user arrives, we try to retrieve as many candidates as possible. We try to rank those candidates. Then we perform some sort of inference on it. Then we evaluate outcomes. Then you have your generated list. Then we feed that outcome back to the system. That's the feedback there. Throughout the talk, in some form or fashion, we'll keep returning back to this model, because one of the biggest mistakes I have observed is organizations treat recommendation systems as siloed problems, as isolated models. They're not. They are a big feedback system. They're distributed systems. A lot of experimentation going on under the hood. Increasingly, they are becoming more and more of these adaptive ecosystems. The interesting problems are actually happening at the boundaries of these components, and not inside each of these components necessarily. The single component problems are definitely challenging, but the coordination between the components is where all the complexity lies.
Roadmap
We are going to walk through this lifecycle, per se. First, we look at what makes recommendation systems adaptive. Then we'll dive into a little bit of the architecture, the ranking, the retrievals. How does the orchestration happen? Then we'll try and discuss a little bit about real-time inference, any production realities. I want to talk about a couple of case studies and what really shapes design decisions in this problem space. Then we'll spend a meaningful amount of time talking about evaluations. Because in my experience, I think evaluation is so much harder than just the modeling aspects of it. If you don't think about that as a first-class citizen, there is a gap. It's a critical gap. Then we'll also talk a little bit about migration strategies, and operationally, what are the lessons that we can call out? Then, finally, we'll talk about where are adaptive systems really heading over the next few years? My goal today is not to convince you that one architecture is correct versus the other. It's really to just share a practical lesson and tradeoffs that emerge when these systems meet production reality.
What Makes Systems Adaptive?
Let's start by making adaptive systems a little bit more concrete. When I say adaptive systems, I'm not simply talking about a machine learning model. It's actually so much beyond that. It's a system that is continuously changing to user behavior based on observed interactions. Many classical recommendation systems operated in a relatively static way. We trained models periodically. We refreshed features periodically. We updated our ranking logic at some cadence. The system learned, definitely, but it learned rather slowly. I think adaptive systems are very different in that format, that they're learning continuously and very quickly. They are just ingesting these signals continuously. They're incorporating context continuously, and increasingly they evolve their behavior while the user is still interacting with the product. That is a different problem all together. That distinction is very subtle because it influences how we architect our systems. Because now, latency matters, freshness matters. It already mattered, but now it mattered at a very different kind of depth and scale.
Evaluation becomes significantly harder because there's just that feedback continuously from how the users are behaving, interacting, engaging with your systems. We'll take a few examples throughout the talk to see what kinds of systems we are talking about. One way I really like to think about it is this. A static system is making decisions based on yesterday's understanding of your world, whereas this adaptive system is trying to make decisions right now, whatever is happening, however the user is interacting. It's almost like continuously living in the present, and if not, a little bit into the future. That's a really hard problem to solve. If adaptation is the goal, where does it happen?
Let's start looking at all the stages where this happens, and let's start with retrieval. When people think about recommendation systems, they usually jump to ranking, and totally understandable. Ranking models are often the most visible part of the system as to what is really influencing the best recommendations for search and discovery, and how do you really get that personalized experience? Ranking is only operating on a set of candidates that have already been retrieved. If an item never enters that candidate set, no ranking can actually happen. That can be a critical gap into what really goes down that pipeline. I often describe retrieval as a breadth problem. The goal isn't that perfect precision that's going to happen during the ranking and maybe there's a sorting layer in between. The goal is definitely to find a sufficiently good set of candidates so that when you are sending them to your ranking layer, you have a very large universe.
That's why modern systems also typically use multiple retrieval stages simultaneously. For example, you can have an embedding-based retrieval, a behavioral retrieval, popularity retrieval based on the product. Some can be trend-aware, some can be context-aware retrievals. Each retrieval strategy actually captures different kinds of signals. Together they create that search space for you, which can become a larger universal set, per se, before you start sending them down to your ranking layers. A lot of teams actually spend time fine-tuning and working on making that ranking layer very efficient, when the bigger opportunity may actually be improving your retrieval stages. How big is your candidate set, or even retrieval freshness, how recent and how fresh is that signal into your retrieval pipeline? How many people here have worked on systems where improving downstream models didn't really give you a lot of value in improving the upstream data quality? Recommendation systems are exactly that. Because retrieval stages, it's actually that stage which determines what the system is allowed to consider. If your system didn't even see it, how will you rank it? That's a loss signal. That really affects the richness of the personalization. That was retrieval, one of the first stages.
Once we have generated those candidates, the next challenge is deciding what deserves that attention. That's where ranking comes into picture. This is where most of the people will expect all of the intelligence to live. To some extent, that's true. Ranking is that intelligence layer, especially because we are estimating that value of what gets shown. What makes modern ranking systems really valuable and interesting is not really more and more model complexity. It's actually that signal complexity, how rich are those signals. We are also combining a lot of different kinds of signals. It can be behavioral, user engagement, contextual, session, which is a very important thing, temporal signals, content signals, evidence-based signals, impressions. All these which become key signals and coming in from different sources. Increasingly, one of the key differentiating signals are intent signals. What was the user's intent when they hovered on your product, when they hovered on your homepage?
Was there an intent? Was there a passive intent? That's really key. Their goals keep changing. Intent changes from what they wanted to watch or listen to at 9 a.m. yesterday versus 6 p.m. today, based on the time of the day, location, changing life circumstances. Some of those changes are happening within microseconds, seconds, minutes, days. Ranking systems increasingly need to answer a more nuanced question. Not necessarily just, what does this user generally like, but what does this user like right now as well? There is this general universe and then also the local maxim of what do they like right now, which is a fundamentally very different question. It's becoming that personalization is not just predicting who the user is, but it's also what matters to them in this moment. It's really right person, right content, and at the right time problem, which leads to the next emerging layer.
This is an emerging layer about ranking the way I think of it. That's really orchestration. That's actually one of the big shifts I see that is happening in recommendation systems which are moving beyond ranking, beyond just critical pipelines, they're becoming more and more of these orchestration systems. Historically, this job was also relatively straightforward. Search's been around for a while, so retrieve items, rank items, display items. Modern experiences are so much more complex. They are balancing multiple surfaces, multiple objectives, multiple intent types, and also multiple user goals. Increasingly, these systems need to coordinate these decisions and in near real time. That's a key constraint as well. For example, maybe a recommendation is actually relevant to this individual, but showing it right now creates a worse overall experience. Because there could be multiple viewers. Maybe the user needs exploration. Exploration is so much of a different problem when compared to, let's say, going deep.
Maybe they need more diversity in content. It's based so much more on the user's mood, and predictability of that becomes key. Maybe sometimes the user just needs something that's totally unexpected. They're being very open minded at the moment. Those are all just orchestration decisions, not just ranking decisions. You're trying to coordinate these experiences, these decisions. You're going from just generating outputs to coordination, and recommendation systems are evolving more and more from just ranking systems to these orchestration systems.
Once that system starts adapting continuously, another challenge starts emerging, and that is feedback. I really feel like this might be the most important slide in the entire talk, because every recommendation system is ultimately a feedback system. It's about how strong is your system to accommodate different kinds of feedback and to incorporate it at the right time when the decision making is happening. Every system makes a recommendation, the user reacts, the reaction becomes data, they are engaging with the product, simple things like impressions, thumbs up, what did you click? How much time did you dwell there? Did you come back to that content again? All these kinds of things is going back into the system as feedback and the cycle keeps repeating. From the top, this actually looks pretty straightforward, but it creates surprising levels of complexity, because sometimes positive outcomes compound, and sometimes undesirable outcomes start compounding.
Systems accidentally optimize themselves into narrow paths, which actually reduce that exploration aspect. They become too conservative, based on how skewed the feedback can take you in the system. They can become too repetitive. You might have all experienced that across retail, content, media. Sometimes it's just repetitive, and you're not in the mood to click on that, or you're just trapped in local optima. Something that happened from today, yesterday, but whereas you wanted to generalize a little bit more, you want to see more content, and these are not model problems. These are all like system behavior problems. If you have ever deployed a system where users started behaving very differently because of the system itself, or some changes that you might have made into the systems, either code-wise, product decisions-wise, observability-wise, and then your users start behaving very differently with your product, then you've experienced feedback. The key is, how long did it take you to reach you and the right teams? That is one of the key reasons why I feel recommendation systems just make those dynamics very visible. Every interaction becomes a very strong training signal. You can do something about it. What will you do? What is the action you will take? That defines the depth of the richness of the system overall.
Case Study: Adaptive Homepage Systems
This section, I actually wanted to bring these concepts together and talk about some case studies. I don't want to go too much into the where the rubber meets the road details, but I did feel like some aspects of this is very important for us to touch upon. We have talked about the core building blocks. Retrieval, ranking, orchestration, how are we thinking about feedback? Let's start looking at some of the things so that the stages don't start to feel like abstract. Let's talk about a homepage recommendation example that most of us would have played, because that's where most of the tradeoffs become visible and surface to the top. Imagine you're responsible for the homepage of a large consumer application. Maybe it's media, it's retail, content discovery. The details aren't actually important, but I'm guessing we all have played with a product like that, where things were personalized to you, they were recommended to you.
You were looking for something, searching for something, you were exploring for something, or maybe you exactly knew what you wanted. What matters is that this surface is responsible for helping the users discover something valuable, connecting with what you're looking for at the right time. That's the problem it's trying to solve. At first glance, this actually sounds very simple. We've been doing this for a while now, because you have the signals coming in, how difficult can it be? Homepage systems are actually deceptively challenging, because they're simultaneously optimizing for a lot of conflicting goals. They need relevance. They need freshness. They need diversity. They need to make space for your exploration. They need engagement. They also need long-term satisfaction. You don't want to incentivize only the new users, you also want to incentivize the tenured users. How do you balance all of that? They often need to do all of this within a few hundred milliseconds, that's where the latency aspect comes in.
If you've ever worked on a homepage or a landing page, you've experienced this. The homepage becomes the place where all of those organizational objectives start coming in, a product has its goals, it wants a real estate on your homepage. Your business has goals, discovery goals, personalization goals, and they all meet here. It's that real estate of that small homepage space where all of these decisions need to happen, and that's really challenging. That's the reason this becomes difficult, because user intent is surprisingly fast-changing, and you need to make space for all of that on the homepage effectively.
This is one of the key pieces of what really leads to that good user experience, sessions. It's one of those very difficult things, because the user's preferences are changing. They might like comedies today. They might be looking for a thriller tomorrow. Maybe they're watching alone in the evening, but tomorrow they're watching it with their kids, and next week maybe they want something completely new. The challenge is that these adaptive systems need to balance multiple time horizons simultaneously. They need to understand who is this user generally versus what are they trying to accomplish right now. That's a difficult balance to just juggle, because long-term preferences provide that sense of identity and engagement, and short-term signals provide you that intent and how to use that as a feedback. Think about the last time you opened Spotify, YouTube, Amazon, Netflix, any personalized application. Were you looking for exactly the same thing you looked for yesterday?
Maybe sometimes that's true, and sometimes that's not. That's really the challenge. How do you keep the system evolving, and how can your session actually take care of incorporating this balance? So much of this is actually very predictive. This is where adaptive systems start becoming very operational. Maybe in traditional systems we would have refreshed these signals periodically, maybe every few hours, every few minutes, maybe daily, maybe weekly, depending upon how fresh you wanted your data to be. As user expectations increased, freshness definitely became so much more important, and many systems have evolved toward that near real-time architectures, especially streaming architectures. User actions generate these events. Events update your features, your feature stores. Features then influence your retrieval. That influences what goes into your ranking layer. Recommendations evolve continuously. The first step of that whole process was actually getting that user's intent, the events into your system.
Either through an event processing queue or batch, or maybe you have a mix of both, however we do it. You need to get those events, whatever are those signals first, so that it can actually get into the pipeline. What has surprised me over the years is freshness is so much more important than going after that sophistication. Definitely better ranking, better search results is important, but freshness just beats every single time. It's so much more important to the user and how they feel about the engagement with the product than actually like that perfection and the sophistication that can be built into the architecture. A simpler model with fresher information frequently outperforms a more sophisticated model operating on stale information. That lesson just appears time and again on most of the systems that I've worked on. Because user experiences freshness. They are not experiencing how good your model is or what are all the backend decisions that are happening. Actually, how many people here have seen stale data becoming a larger problem than model quality, if you have been working on similar systems at work? That's really the point. I think these systems are adapting so much in the real time.
As that happens, another challenge starts emerging. That's really around, how much should these systems explore? We talked a little bit about exploration. This is also a classic search problem. Do you want to go deep or do you want to go breadth first? That's really exploration versus exploitation, and one of the oldest problems in recommendation systems. It's also a very persistent one, in my opinion. If we only optimize for what the users have historically engaged with, we create extremely predictable experiences. We reinforce existing behavior. We are showing them the same things. We are also reducing discovery. We are reducing learning. We are reducing that seeking aspect of search, which is very important for the human experience. We also reduce adaptation in the process. Whereas on the other hand, if you go through aggressively on the exploration side, then we risk irrelevance, we reduce engagement sometimes, we reduce confidence of the users in the product.
Trust gets affected. We frustrate users, especially the longer living users on your product. You might have experienced this when you are reading on Reddit. What is it optimizing for? Is it more of those historical patterns or for the new users who sign up? That really can lead to frustrated users. The challenge is really finding that balance, not just globally, not just for one single user, but for every session, for every context, for every experience that the user will have, and that balance is what causes the challenge as well. This optimization is your center of gravity. It creates that behavioral gravity in your system because you can go back to it and check, what are we optimizing for? Can we tune the parameters there?
Let's take a pause. Let's see some of the lessons that at least I have seen repeatedly show up in production that we can talk about at this point in time. The most recent one that we talked about, freshness beats sophistication. It's just worth repeating it because a simpler model with current information might give you those results than you going after a much more complicated, more powerful model. Maybe freshness as an optimizer. Evaluation is usually harder than modeling. Can we make evaluations as a first-class citizen? As you're architecting that end-to-end system, can our ICs sit together with the rest of the engineering, rest of the product to actually decide what are the criteria which defines product success? Observability, again, a very product requirement. It's not just an Eng requirement. If you can't explain your system behavior, if our product managers cannot explain why something was shown to our users, that explainability, if that is lacking, then that loses trust.
Executives lose trust, our users lose trust, and customers eventually can see something is off in the product. Adaptive systems usually incentivize, and the incentives compound. They intensify whatever we are making as positive signals, so we need to watch out for that. Whatever you ask them to optimize, they're going to optimize it for you. Keeping a stock of, is that changing? Do our users care for something else? Can we pivot? Can we tune our optimizing criteria? That's also very important because they'll just optimize for whatever we want them to optimize, and something that we optimized a week ago might not be relevant anymore. Most of the recommendation failures are still system failures. They are data quality. They're not so much model failures, they're actually more latency problems, evaluations, how are we doing rollouts, feedback loops? Those are really the real causes. How strong are our systems to detect these failure states? I think I've become convinced at this point in time that recommendation systems and their failures are still a distributed systems problem wearing a very machine learning costume. Can we keep an eye out for them sooner than later and actually bake that into our architectures?
Real-Time Inference and Operational Reality
We have talked about adaptation mostly at the recommendation layer. Let's go a little bit more deeper, because none of this matters if we can't serve these recommendations reliably with trust under production constraints. That's where I feel the inference architecture actually plays a very important design decision role. Let's talk about inference. When people hear the sub-10 millisecond inference targets, then immediately somehow, they think about model optimization. Actually, in practice, it's not so much just the model. It's probably just one part of the overall latency budget. Before you even invoke a model, there are so many things happening. You have likely retrieved some candidates. You have hydrated feature lists. You have performed your joins and already made some service calls downstream. By the time your model runs, you have already spent a lot of your latency budget credits already. That's the practical lesson here, which is pretty simple.
When you are trying to optimize for latency, try to optimize the entire end-to-end path. Let's not start with just optimizing our models. That cost, the latency, the performance is definitely part model, but it's actually end-to-end. It's like, where do we buy it back? That's the question. It comes from multiple places in your system. Don't start too late on that end-to-end latency budgeting. It needs to be part of the design decision. You don't save money on a service call that you don't make. It's just that end-to-end pipeline that really matters.
Another key thing that I feel is, not every recommendation needs to be computed in real time. Like if we talk about, how do we think about caching, fan-out, our async pipelines? How much event-driven do we need to be? Can we do this in batch? How much do we need to cache? When do you need to purge your cache and rebuild it again? There are certain signals which are very important in real time, whereas so many others can wait. Taking a very realistic stock of what needs to be fresh versus what can wait, definitely worth it. Fan-out is also one of the fastest ways to destroy latency. A homepage request that fans out to 10 downstream services isn't just 10 times harder to operate, but it's also spending so much time there making all those calls. We also rely very heavily on asynchronous pipelines, caching, graceful degradation.
Just don't make that service call if you don't have to, and that's the way we keep saving some of those micro milliseconds of the latency. Once these requests reach inference, costs become as important as latency. If you think about constraints, if latency is one of them, the next one is definitely cost. There are ways to definitely look at latency. Like, how much fan-out do we need? Do we need batch? Do we need async here? Do we need caching? Really taking a deep look into how that affects the overall user behavior.
Then, on this slide, I think the key message is really not every request deserves to be running on the most expensive model. One pattern we are seeing more broadly across AI systems is just that model tiering, having many models, deciding how do you make those calls, and that's where the ML serving infrastructure really becomes important because the downstream services, your upstream are not really talking to the models directly, but you have created a platform for it so that you have a clean tiering system, layered architecture where you know what models to go to for what kind of job. You have a portfolio of sorts, of model, and not just one model to go to. For example, you can have something like a very lightweight model on CPU, and you can have one of your richer, more complex models running on your GPUs. Or simple things like your fallback models are using overload, or, I've also seen progressive quality tiers.
That is another methodology. The goal isn't really for just maximizing model quality. It's so much about also maximizing that system quality. Just being careful about how are we actually spending on these models. Then I think this one is so much about offline and online. I have seen that the metrics are all green, offline systems are working really well. Your dashboards are looking really well, but you don't get the same level of engagement or quality when you release it and your online tells you a completely different story. The signals, the events, the behaviors, the intent that we talked about is all sitting in the production path. How resilient is our system to actually detect that mismatch? Can we do it before we release to the production pipelines? Just catching that skew sooner than later before it hits our users. Briefly touching upon observability. With the best of the pipelines around retrieval, ranking, evals, feedback, if you don't have a rich infrastructure to actually have these metrics logged, the right level of insights into what happened when, and every trace available, it really takes away that explainability from our products, which is so key to establishing the user trust. Making sure that we actually think about observability as a first-class citizen.
Evaluation Systems that Don't Lie
Evaluation, one of my favorite topics again, that's where I feel like most of the work still needs to happen. Every metric eventually gets gamed. How resilient are we towards it? If there's one thing recommendation systems actually teach us, it's that metrics are pretty much like proxies. They're not goals. Most teams might start with engagement. It might be clicks, watches, dwell times, or maybe like, did you come back to something, interactions, opens, closes? Over time you actually discover something. Users can engage more while having a worse experience. You can increase clicks. You can increase that watch time. You can increase all those things you are measuring for, but you can still reduce trust. You can reduce the quality of engagement for the user. The easiest thing to optimize is often not the thing you actually care about. That's a key insight, because if you don't have good evaluation criteria, if you're not sitting together to actually define them together, then there is a mismatch between the kinds of technical systems we build and the product outcomes that they promise to our users.
Evals happen to be one of the key things there. Multi-objective, this is exactly where our systems are balancing multiple objectives simultaneously: diversity, retention, engagement, freshness, ecosystem health, business outcomes. There are a lot of things, lot of incentives for various cohorts. The challenge is most of the times these disagree with each other. The goal isn't really to find that perfect objective, but it's mostly about how strong is our Eng strategy with product strategy so that there is some level of tradeoffs that is shared and visible to the rest of the company as we make these decisions. What is ok from a product's perspective, whereas what is not? We talked about this offline versus online metrics, and how can we blend them much earlier into the process and not after something is released. This is another case study where I talk about some details about when the dashboards actually look very good, everything's green, but your SLAs are not getting met, your users are not engaging with your product. How do we actually detect when does engagement improve versus getting worse? How does the ecosystem evolve through all of this? How do we make that distinction between what are the metrics that are working versus not?
Migration and Organizational Scaling
This is something that's important. This is one of the most important non-technical lessons I've learned. As systems become more and more adaptive, success depends less on any one individual model or one individual component in the system, and it's so much more about that organizational alignment. Who decides the tradeoffs between each of these decisions? Simple things like engagement and discovery. What are the tradeoffs? Do we agree on them? Who defines success metrics? Who owns experimentation, across product and Eng? These questions sound organizational. The more aligned organizationally we are, we are able to make those decisions better, but they also ultimately shape the system behavior, which is very critical. Adaptive systems amplify that organizational structure. If teams are aligned, then the systems learn faster. If the incentives are misaligned, then the system tends to amplify that too.
Future Evolution, and Closing
That really brings us to where I feel like all this is going. Three things that I really notice are recommendation systems are truly feedback systems, very tight feedback loops. They optimize exactly what you want them to optimize. If you're looking at something that is not happening according to what you want, then you'll have to go back to your optimizing criteria and change that. The hardest part is definitely not building the model. It's about building this entire system end-to-end and taking care of each one of them as a serious enough citizen. The future is definitely not better ranking, not just better interaction, but it's all of this coming together where users are able to have some level of connection and engagement with the product. I feel like if I see the industry headed in the space of recommendations and personalization, I don't think the biggest shift is model quality.
We'll definitely see better models coming in as we are currently, but I think the shift is going to be something else. Our users are wanting to participate in that recommendation process more and more, and they want to feel that level of connection that they did something and that's why they're seeing something. They want to steer. They want to refine. They want to express intent. That was not happening in traditional systems. You're also seeing that in copilot search systems, discovery experiences. Mostly the winners won't necessarily be the systems that predict perfectly. We don't want those perfect sophisticated predictions, but I think the winners will be those systems which can learn very quickly, adaptively, and evolve very quickly to that user feedback. That's what is the most trust establishing. Because in production, if you think about all of this as intelligence in the system, then intelligence is only useful if the system can use it in a very sustainable fashion.
I feel like the future of AI in the space of recommendations, search, and discovery, which is going to happen more and more as an engagement with the user, is really about how can these systems be set up to learn, adapt, and evolve effectively. Instead of just focusing on model intelligence, can we focus holistically on orchestrating and coordinating these components in a way that can lead to the success state?
See more presentations with transcripts
