Transcript
Bhavuk Jain: My topic is, "Engineering AI for Creativity and Curiosity on Mobile," in which I'll talk about two products that I personally worked on, along with our brilliant team, one where we used AI for generating beautiful images, and other for making visual search deeply intuitive and better. I'll first talk about how modern AI systems become products, starting from foundational models, and then dive deeper into the specifics of the two features, discussing the product principles, their architecture, and how we build these features. I want to ask you all a quick question, while searching for a product on Google Search, while scrolling through Instagram, you come across a cool pair of sneakers or a handbag, so earlier, who all have come across this problem that you had to take a screenshot of the product, switch apps, upload the image, and then search for it, just to figure out what that product is? Second, who here has spent more than 10 minutes scrolling through different images, trying to figure out that one perfect wallpaper for your device? I'm sure you have already experienced these points of friction that inspired the two features that we'll be covering today.
Roadmap
We are going to start with how we moved from a general foundational model to a user-facing feature involving post-training, fine-tuning, inference, guardrails, what are the tradeoffs to make, and what works well. Once we have that architectural context, we'll move to the two concrete applications where we applied those lessons. The first case study would be about AI wallpapers, where we taught AI to be a creator, to basically take an idea and produce something entirely new and beautiful. The second will be Circle to Search, which will be about how AI looks at the pixels of your mobile screen and provide most contextual answers.
How Modern AI Systems Become Products
Most modern AI products, they don't start from scratch. We usually begin with a massive foundational model. However, since the raw model is too unpredictable and unconstrained, it must be refined into a product through four non-negotiable architectural steps. The first is post-training, and this is where we align the massive base model with human preferences, style, and essential safety rules. The next is fine-tuning. This is the phase where we specialize that model, basically move from a generalist model into an expert, tailored exactly for your use case. The third is retrieval and grounding, which has become more important of late because of hallucinations. This is the phase which makes sure that the system is accurate, factual, and up-to-date. The next is inference and guardrails and deployment, which means wrapping the whole thing in a robust layer of deployment, infrastructure, and most importantly, we implement those essential guardrails that allow it to run reliably, safely, and at scale. The final result is a viable product, whether it can be an app, an API, or a feature. In the next few slides, I'll break down why these architectural steps are important before we zoom in to the two case studies.
The first is post-training, and this is a good analogy that I often use. If the foundational model is the raw, brilliant student, post-training is where we teach it manners, personality, and safety. The goal is to align that powerful base model with human preferences and style. To do this, we generally rely on two major families of techniques. On the left, we have the RLHF family, reinforcement learning from human feedback, which is flexible, however complex. It requires managing a base model, a separate reward model, and a complex RL loop. On the right, we have the DPO family, Direct Preference Optimization. These methods are generally simpler and easier to scale as they skip the separate reward model and work directly on preferred versus rejected outputs using a supervised learning objective. However, rigorous evaluation is very crucial, since good behavior is subjective. For example, what makes a good summary, we must engineer the metric.
This is a point which I'll also talk about during the case studies, which involves defining clear, measurable evaluation criteria for human raters, focusing on, for example, measurable aspects like format, quality, completeness, and accuracy. Post-training forces an early architectural decision, more flexibility versus simpler, more scalable path. Whatever you choose, this entire alignment step has to be locked on before we move on to the next stage.
So far, we have aligned our foundational model to human preferences in the post-training phase, but that model is still a generalist. The next crucial question is, how do we make it actually good for our specific use case? Our goal here is to take that aligned base model and truly specialize it for specific tasks, domains, or even individual users. We have a few options here. The first is, we have task and instruction tuning. This is the standard supervised fine-tuning. We collect lots of input-output pairs, like question-answers or document-to-summary, and we train the model to map those inputs to a desired target output. This is how we teach the model to be specific and to learn the new behavior we need in our product. Next, we have Parameter-Efficient Fine-Tuning, PEFT, and this is the modern sweet spot. Techniques like LoRA, QLoRA, and adapters allow us to train tiny specialized modules that sit adjacently to the large model, keeping most of the original weights frozen.
This makes training dramatically cheaper and also much faster. The last note is super useful. Sometimes, you don't even need to fine-tune at all. If the core problem is that the model lacks fresh, factual, or private knowledge, then the right solution is not more training. It's actually grounding, which we'll talk about in the next stage. The way I look at this is, ultimately, you are choosing a point on the spectrum. More fine-tuning gives you greater specialization and new behavior, but it costs you more in terms of complexity, maintenance, and dollars. Now that we have aligned and specialized the model, we add the step that I talked about earlier, which is absolutely essential now, retrieval and grounding. This is the layer that keeps the system accurate, factual, and up-to-date. Instead of relying purely on the model's parametric memory, which is frozen at training time, we connect it to external sources of truth.
They can be either structured knowledge, fresh web results, or, for example, on-device context. This reduces hallucinations dramatically and enables the model to answer with real knowledge and not guesses. This is also where modern AI systems plug in search, embeddings, RAG, or tool uses. Once the model is grounded in real facts, we can safely move on to inference, scaling, and guardrails.
We have, so far, aligned the model, specialized it, and grounded it, but you cannot ship an AI product without a dedicated multilayered safety architecture called guardrails. Guardrails are mechanisms added around the model to constrain what it can see, say, and do, as even the fine-tuned models can be unpredictable. How do we do this? There are multiple ways. The first is data guardrails, and this focuses at the source, deciding what information is allowed to influence the model at all. This means strictly filtering and redacting sensitive or toxic content from our training and evaluation data. The second is model guardrails. Over here, the focus is the model behavior. We use safety-tuned base models, strong system prompts, and sometimes separate safety heads or classifiers that can reshape the model's raw output. The third is runtime guardrails, and this is the real-time inspection system. At inference time, we inspect both inputs and outputs to detect things like PII, toxicity, or jailbreak attempts.
We also strictly control which external tools or APIs the model is allowed to call. The last, governance and UX. This is the human safety net. This includes logging and audits, human review flows for high-risk actions, and clear user-facing controls like reporting and feedback. Put together, these layers are what allowed us to move from an impressive lab experiment to a feature that is safe and predictable. Finally, deployment. We have spent all this time so far perfectly fine-tuning, grounding our model, adding guardrails, but that was all a rehearsal. This slide is about the moment of truth, running the model reliably at scale across many requests, modalities, and products. Once you hit production, the challenge shifts from model quality to infrastructure stability. There are three things to keep in mind here. First, unified multimodal serving. Today's models are polyglots. They don't just ingest text, but images, audio, and sometimes video as well.
We can't afford to build a separate architectural stack for every modality. The ideal goal should be to have one unified API or pipeline that handles these diverse input types. The second, latency, cost, and scaling. This is the engineering battleground where we must use sophisticated techniques like batching, caching, and autoscaling to keep costs low and responses fast. Every millisecond of latency is a potential dollar spent, so we are constantly optimizing to keep that interactive feel while controlling our expensive GPU and TPU spend. The third is observability and control. You can't fix what you can't see. Once live, rich logging and metrics are non-negotiable, allowing us to track latency, errors, and usage. In short, this entire infra stack is the bridge between having a trained model and having millions of users use it every day without incident.
Case Study 1: AI Wallpapers - The Generative Challenge
Now let's look at the case studies, how we apply these principles. What is AI wallpapers, a feature that we built? This feature lets users create unique, high-quality wallpapers instantly on their phones by describing their idea. It's an example of the generative phase of AI engineering, where we were optimizing primarily for creativity. Before we dive into the how, it's important to understand the why. Why did we start with wallpapers in the first place? For us, it was the perfect canvas, and this was about two years back when models were still coming up and we were still in the early stages of the AI models. The first reason being, it offered a clear and immediate user benefit, and that is personalization. It aligned perfectly with our Material You design philosophy. Material You is the system in Android where your phone's entire interface, your icons, your menus, and everything adapts the color palette to match your wallpaper, making the whole experience feel uniquely yours.
An AI that creates personal wallpapers wasn't just a cool feature, it was a natural extension of the Android experience that we were building. Second, there was a massive clear user need. We know from data that changing wallpapers is by far the most popular way people customize their phones. They were already spending significant amount of time scrolling through different photos to find that ideal, perfect wallpaper for their devices. That leads to our third point. We finally had the technology to solve this problem at a whole new level. Generative AI can deliver a truly infinite range of high-quality content. Suddenly, users were not limited to a static gallery of other people's photos. They could become artists and create something that was truly one of a kind.
While building AI wallpapers, we were looking at solving four major core challenges. First, the UX challenge around prompt engineering. Prompt engineering is powerful, but it's also a complex skill. Asking a user to type cinematic lighting, or 8K, or ethereal watercolor, it's a recipe for failure. At least except the power users, the normal users will not do that. The challenge was translating that immense complexity into a simple, fun, and fast mobile UI that felt intuitive to everyone. Next is the quality challenge. A generative model does not inherently have a good taste. It can create an image that's technically correct, but artistically messy or just plain weird. Our challenge here wasn't just generating pixels, it was achieving consistent artistic control that a user would like to have as their wallpaper. Then, there was the safety challenge. For a feature shipping on millions of consumer devices, safety is not just a feature, it's the foundation.
Preventing the generation of inappropriate, harmful, or copyrighted content was a non-negotiable day one problem. We could not launch first and fix it later. It had to be solved from the very beginning. Finally, we faced the emerging viability challenge, which I call the price of success. Our initial server-based models were incredibly powerful, but also very expensive to run. As more and more users started creating wallpapers, the operational cost of running the feature became very high, which also became a significant concern. A feature that's too expensive to run, no matter how popular, is not viable in the long term.
With these four challenges in mind, let's look at how we solve them. The first is the product and the UX challenge. Right at the beginning, we faced a fundamental product question, which impacted engineering too. How much control should we give to the users? On one end of the spectrum, you see the restricted approach. Think of this as the no-fail option. We give users beautiful, pre-made concepts to play with. The huge advantage is simplicity and almost guaranteed high quality, but very limited freedom. This approach essentially engineers the metric that I talked about during post-training by limiting the output space, making quality control very straightforward. However, on the complete opposite end is the open approach. This is like the classic blank text box, and for power users, maximum power and maximum freedom. You can simply ask for anything. The challenge here is the blank canvas problem. It puts a huge burden on the user to be a great prompt engineer, and also the challenge is, how do we educate the users?
Right in the middle, there is the hybrid model, which tries to find that sweet spot. This is like guided prompt building. It guides the user, teaching them how to build a good prompt, but still leaves room for their own unique ideas, and also allows us to use guardrails to guide the model's behavior. The prevailing wisdom back then was more freedom is always better. You should let users decide what they want to build. Just give users a blank text box and let them create anything they want to. We looked at the public products out there and what we saw was a lot of user frustration. The blank canvas problem is real. It puts the burden of being an expert prompt engineer onto the consumers, and more critically for engineers, the blank text approach creates a non-deterministic product. It maximizes the difficulty of setting reliable guardrails. We made a non-obvious bet, almost counterintuitive, instead of just giving our users a powerful but complex tool, we decided our role was to be their creative partner.
This is the core tradeoff you see in the diagram. On one side, you have the allure of total complexity and freedom. However, we deliberately chose the other path to push for simplicity and restriction because that was the only reliable path to have consistent quality. This allowed us to actually engineer the metric that we talked about before to define a narrow, measurable quality criteria that is format, conciseness, and accuracy. We validated this idea with the help of a user experience research team, which was absolutely critical. They built and tested these concepts with real users, which gave us the confidence to move ahead.
We made this big, non-obvious bet on restriction, but with restriction comes a huge responsibility. If you are going to guide the user down a specific path, that path needs to be absolutely beautiful and rewarding to walk. That's why, given the restriction, we decided to invest heavily in delight. Let me give you this example. On the left side of the screen, you see three prompts, and now two, that at first glance, they look almost identical. The first two chips, they load almost identically. When the UX folks gave this to us, we were like, they look exactly the same. However, if you look at the slowed-down version, they are actually different. I've tried to write down what's the difference. In the first chip, the words go around in a wavy manner. In the second, there's a horizontal scroll of the words. In the third, there's a fade-up effect.
It's hard to notice, but it's there. This is the level of detail we obsessed over. These tiny moments of motion make the interface feel alive. A huge shout-out to our visual design and motion teams for this. Also, at the same time, delight is not just about a happy path. It's also about how you handle failures, for example. AI is not perfect. Sometimes it fails, and normally, you'll get harsh error messages that feels like you did something wrong. We wanted to turn those potential moments of frustration into something playful. For example, you see on the right, we designed these uniquely fun visuals for our error states. Instead of having error 404, you might get a T-Rex that's trying to paint.
Why did we go through all that effort of building a guided experience and adding delight? This slide is why. Because these principles ultimately helped us build a great product. I have a question here at the bottom. Can anybody figure out what's wrong with this? I'll tell you guys. This character, Waluigi, it's a character in the Mario franchise, very popular with kids. Some might argue that a well-designed system should probably not be generating images of weapons in a fun, creative context like stickers. This is where the principles became a non-negotiable line of defense for us. Our decision to favor restriction and simplicity directly reduced the likelihood and the dollar cost of these catastrophic safety failures. It also makes the engineering challenge of risk assessment tractable, allowing us to ensure our output is always safe. Let's look at the high-level architecture of the feature. This diagram shows the journey of a user's request.
It all starts on the frontend where the user experience is intentionally simple. We talked about this in previous slides. The user's simple choices are sent to our prompt engineering service. Think of this as a crucial translator. It converts the user's selections into a highly-structured, detailed prompt, often injecting specific artistic modifiers and negative keywords to reliably guard the model towards a high-quality result. Then, before actually making the image, the prompt goes through our backend service. As you can see, this is where we have placed the first layer of guardrails. These are fast server-side checks, often using classifiers and block lists to ensure that the request is safe and appropriate. Once the prompt is cleared, it's sent to our inference platform. The heart of this is the media generation service, which is based on a fine-tuned model, and this platform also contains our second safety layer, a set of visual classifiers that analyze the output image to ensure it's visually safe before a final image upscaling step enhances the resolution.
Finally, the finished high-resolution wallpapers are sent back to the frontend for the user to select. It's like a complete loop with a specialized model for quality and layered guardrails for safety built in at two different steps, the backend and the inference platform. In fact, this strategy allowed us to launch the version one of the product, that was safe, easy to use, and produced consistently high-quality results. We were even able to open this feature to ages 13-plus because of our safety guardrails and practices.
Let's do a little bit of a deep dive. How did we tackle the biggest challenge that is quality? The key insight and the theme of this slide is that models aren't all made equal. An AI model is less like a computer and more like an artist. It excels at certain styles and struggles with others. Our goal wasn't to make it good at everything. It was to identify what it was naturally brilliant at and build a product around those strengths. The map you see on the right over here, it represents different idea clusters, each one a style or category the model understands, from art to animals to people and clothing. Just because a model can generate something does not mean it does it well. Our first task was to pinpoint the themes where it consistently produced outstanding results. By isolating those sweet spots, we could constrain user choices to the domains where the model shines.
This upfront work was crucial. It let us guarantee high-quality outputs, simplify quality control, and reduces the risk of low-quality generations right from the start. Once we identified the model strength, we had to validate them. That's where large-scale testing came in. We built a pipeline to mass test prompt variations, always ending with a manual human review. The visuals on this slide are artifacts from that process. We generated thousands of images, taking a single concept like a tree on a mountain and running it through dozens of stylistic modifiers, or taking a prompt like poppy flower and comparing outputs across multiple models. Importantly, the final judge was always a human. Our designers and researchers reviewed images, not just for technical correctness, but for aesthetic quality and the wow factor. To make the subjective process measurable, we created explicit evaluation criteria covering format, quality, completeness, and overall appeal. This human-in-the-loop testing was essential to moving from what the model can do to what it should do to deliver delightful wallpapers.
This leads to the next key step, turning our learnings into the actual prompt keywords. This is where prompt engineering comes in. Our goal was to design suggestion chips and a structured prompt format that would reliably produce beautiful images, essentially giving users creative building blocks that just work. The grid you see over here is just a small snapshot of that process. For every potential chip, we had to make a tough call. Should we keep it or should we drop it? Does this term consistently yield great images or does it introduce unpredictability? Across all combinations, we generated and reviewed over a million images just to decide which few words should appear in the final UI. This is why the system is more than just a text box. Users don't need to be prompt experts to get high-quality results. This structure, it had an engineering payoff too. By using consistent, repeated prompt templates, we unlocked inference optimizations like prefix caching, allowing the model to reuse shared prompt segments for faster, more efficient generation.
Finally, our prompt engineering work was supercharged by leveraging our internal research teams, and this was a game changer for quality. First, we established a powerful feedback loop. Our product data helped the research teams make their core models better for everybody. Second, it allowed us to be smarter, and we investigated fine-tuning for specific categories where we needed that extra polish. Finally, this collaboration means that our product is built on an ever-improving foundation. As the underlying AI model gets better, the user experience automatically gets better too.
We have gone deep on the principles and the process. Now, let's pull back and look at the entire journey to scale from start to finish. Our philosophy was always to build with one thing in mind, scaling, since we had to build for a global user base from day one. It all started with relentless UX exploration where we did not just design a simple flow, we understood the user's intent on mobile and used that to define our evaluation criteria and metrics. That UX was powered by a relentless focus on quality, which brings me to the second image in this slide. This meant establishing a rigorous evaluation pipeline with human raters to move beyond subjective feelings to define consistently high-quality results. All of that foundational work led to the final and most critical step for scaling, the architecture. Instead of building a one-off service, we engineered our feature to use a scalable endpoint that is reused internally across multiple Google products, which helped us to reduce overall cost and also improve reliability.
Finally, the moment of truth arrived and it landed. For us, a successful launch was measured in two key ways, by the praise from our users and also to deliver it at a true Google scale. First, the praise. The response from users was incredible. They called the concept freaking insane and a childhood dream, which is exactly the kind of delight we were hoping for. This was the perfect validation of our non-obvious bet on partially restricted input. Second, we delivered this at scale. That is, we did not just launch this as a niche feature on one device. It was thoughtfully integrated across different surfaces and form factors, and also adopted by major OEMs like Samsung.
Case Study 2: Circle to Search - The Understanding Challenge
That was our journey into AI as a creator with AI wallpapers. Now let's switch gears and explore the other phase of AI engineering, and that's understanding. This is a feature I'm really excited about called Circle to Search. The concept is very simple but powerful, that if you can see it on your screen, you can search for it instantly without leaving the app that you are in. From an engineering perspective, this is a complex, real-time, multimodal retrieval task. We are taking a visual input, a gesture on a complex screen, and translating it into a query that delivers high fidelity and relevant results. In the example you see on the right, you might be scrolling through a social feed and you see a handbag you love, and without breaking your flow, you just long press, circle it, and just get the shopping results. Let's talk about the motivation first.
For years, the standard workflow for visual search was a multi-step process. You had to take a screenshot, then switch to a different app, then find and upload that screenshot, and then you can finally search. Every single one of those extra steps, it's a point of friction, a moment where a user can just abandon the search and walk away. To erase that friction, we wanted to create a search experience that was fast and intuitive. Our North Star for the project was that ideal state, zero context switching, zero interruption, and just a seamless flow from curiosity to answer. The final point here is the key. We knew this vision was achievable because we were not starting from zero. Google Lens already possessed incredibly powerful world-class visual understanding. We realized we can build on top of this by adding two things, to more deeply integrate AI into this, and also a much deeper system-level OS integration.
To understand Circle to Search, we need to start with its foundation, Google Lens. Google Lens has long been a powerful way to search the world through your camera. It came with friction. You had to stop, open a separate app, and then point the camera. Circle to Search was the next natural step in that evolution. Its core innovation is simple. Lens is world-facing. Circle to Search is screen-facing. It turns whatever is on your screen into the query. By replacing multiple steps with a single gesture, it removes nearly all friction, letting you act on curiosity instantly without leaving your current app. This seemingly small shift has huge implications for the tech behind it, though. It transforms the task into a real-time retrieval problem, one that demands extremely low latency and highly relevant results at every interaction.
Let's look at the high-level architecture to see how all of this comes together. I have broken it down into two parts, the overall end-to-end flow, and then a deeper dive into a subpart that's the AI pipeline of the architecture. First, the overall architecture, it all starts on-device. A user gesture, like a circle or a tap, it triggers the capture of on-screen content. This is not just a flat screenshot. It's what we call a multimodal prompt, containing the pixels as well as the structural layout of your device. This rich prompt is then passed on to the AI pipeline, which you see here as a single component. This is where the core AI reasoning and heavy lifting happens. Once the pipeline has done its work, the final answer is sent back to the device, and it appears as the results overlay seamlessly integrated into where the user currently is.
Specifically looking at the AI pipeline, the multimodal input from the device, it first hits the core multimodal model. This is not just a generic model, but rather fine-tuned to understand the relationship between an image, the user's question, and the context of the screen. Once this model understands the user's intent, it kicks off reasoning and retrieval across multiple sources in parallel. It queries the knowledge graph for structured facts. At the same time, it queries web and image search to find visually similar items, products, and articles. For example, if you circle a pair of sneakers, the system queries the knowledge graph to identify the brand and model, while also simultaneously querying web and image search to find shopping links, reviews, and photos of them in different colors. The final stage is synthesis and delivery. All the potential answers from our retrieval systems are fed into an AI overview synthesis model.
This is the generative part of the system that crafts the helpful natural language answer. Before that answer is ever sent, it must pass through a final non-negotiable checkpoint, or safety and responsibility filters. This is the runtime guardrail ensuring every answer is helpful and harmless. We rely on two distinct layers from the pipeline you saw earlier that make this instant, seamless experience a reality. On the left, we have the MUM, the Multitask Unified Model. This moves our system beyond simple keyword matching to intent understanding across modalities. For example, if a user circles a pair of shoes and types in blue, which is a text, and the shoe is an image, MUM understands both the visual attributes and the semantic refinement to provide a precise answer. On the right, we see the Gemini integration, which powers the richer AI overviews and contextual help. This is the layer that handles reasoning. Gemini integration, it elevates the tool from a simple identification to complex reasoning.
We have talked about the AI model so far, but another thing that makes this feature a game changer was the deep system-level integration. This integration is what made the feature effortless to invoke and not have the same issue that we had with Lens. Instead of forcing you to hunt for an app, we built it directly into the Android OS. This means a simple universal gesture that works everywhere. It's a seamless entry point that removes all the friction of the old screenshot and upload workflow. Because we are integrated at the system level, when you circle an object on your screen, we are not just guessing what you are pointing at in a flat image. The system gives the AI a deeper level of context. This gives us a huge hybrid advantage, means we can focus on the exact item that you selected along with a semantic layout.
Ultimately, this means that when you circle a handbag in a busy photo, we know exactly which specific pixels belong to the handbag in the entire screen, which allowed us to deliver search results that are very accurate. Now, coming to the guardrails. First, we ground the visual AI in factual data. Any model can make mistakes. It might see a plant that looks like a rare species, but it is not. To prevent these kinds of errors, we don't just trust the pixels alone. We cross-reference any recognized entity with a structured, reliable knowledge base that is the knowledge graph. This grounding in facts is our first line of defense against misinformation. Second, we implement a dedicated safety layer, which is a core part of the architecture. Every single synthesized result, especially those that use generative models, must pass through a final, non-negotiable responsible AI filter before it is ever shown to the user.
This layer is designed to catch anything that might be harmful, hateful, or inappropriate. Finally, we design for ambiguity. Sometimes, the AI simply will not have a confident answer. Maybe the image is blurry, or the object is too obscure. In these low-confidence scenarios, the worst thing you can do is guess and be wrong. A bad answer erodes trust faster than no answer at all. We design the user experience to handle these situations gracefully, offering broad suggestions instead of returning a wrong answer. Our core hypothesis was that removing friction through deep system-level integration would unlock a new level of engagement. It did indeed turn out very well. For example, we achieved massive scale, launching on over 300 million devices. This easier access is directly driving a 70% year-over-year increase in visual searches. It's also becoming a fast-growing query type, especially with younger users, with shopping and translation working out very well.
Key Learnings
After walking through these two very different AI engineering challenges, one of generation, the other of understanding, what are some of our key learnings? Of course, many of these principles apply to both projects. Every product needs to scale and be grounded in quality. I wanted to highlight some distinct lessons that each case study taught us. From AI wallpapers, our generative product, the unique challenge was managing the blank canvas problem. The most important lesson was the power of having clear principles and our tight product definitions. Our non-obvious bet on guided creation and our obsession over the evaluation pipeline to ensure artistic quality were absolutely critical for a generative experience. We also learned the importance of over-investing in moments of delight to make a restrictive experience more magical. From Circle to Search, we learned a different set of lessons. Improving the model certainly helped. What turned out to be amazing was the deep OS integration that made the features seamless to invoke.
That frictionless experience is what users really liked, along with the search results. This integration also unlocked a deeper level of intelligence because we were now able to get that semantic layout of the screen. Finally, to build trust, we learned the importance of grounding our visual AI in factual data. Cross-referencing what the model thinks and sees with a structured knowledge base was a crucial step to ensure our answers were not just fast, but also reliable.
Actionable Takeaways
Some takeaways for AI engineers. You should think about productizing your AI and not just deploying it. By that, I mean converting it into a proper product. A powerful model is just the starting point. As we saw with AI wallpapers, the real work was in building the entire system around it, the evaluation pipelines, the cost and latency optimizations, the safety guardrails. A productized AI is a reliable and a trustworthy experience. For tech leaders, champion principle-led development. Our non-obvious bet on guided creation for AI wallpapers was a hard decision. It was guided by our principles of quality and safety. Those principles were our North Star, and they paid off. As leaders, it's our job to define those principles and empower our teams to stick to them, especially when it is hard. For product managers and designers, obsess over the user's workflow. We saw this with Circle to Search. The AI was indeed very important. However, identifying and eliminating friction in the user's journey really helped.
Questions and Answers
Participant 1: With all of the checks and processes in place to make sure that you had all the safety, how did you go fast? How did you keep your velocity up with all the things you had to do to make sure you could support that many users in production?
Bhavuk Jain: Do you mean while in development?
Participant 1: While in development and delivery. How long did the process take? Maybe it was six months?
Bhavuk Jain: Yes, it was a long process. We had to make sure that the quality is high. It wasn't like a three-month or a four-month project. It took us like a year to build, and especially because this was done about two-and-a-half years back when the models were still coming up. They were not so advanced. They used to make mistakes. We had to make sure that the quality is right, and we invested a lot of time in that.
Participant 2: How do you test and what's the principle for you to say that testing is good enough for both of the case studies? Like considering all the inputs are arbitrary and the underneath infra could rapidly iterate.
Bhavuk Jain: That's where our product managers, our UX design team, and the human in the loop was very crucial. Whatever came out of the testing pipeline, we saw the final output, and we did not just accept or reject it. We evolved the pipeline and the model, which was being given to us by the research teams to do better. For instance, if it gives us an image and if we feel that image is not up to the mark, we'll, for example, add more prompt modifiers like, give us this image in 8K, or, for instance, give us this image in some specific artistic style. This was a continuous evolving loop where we got some feedback from humans, like actual people reviewing the output and then improving the tests that were running at scale.
Participant 3: You mentioned cost as one of the challenges, the price of the feature. How did you address this?
Bhavuk Jain: As I said, this was about two years back. The model was very expensive back then. One way to manage that was to first migrate to a more efficient model. Instead of running every prompt and getting an image, we started batching them. One of the techniques I mentioned was prefix caching. That's one thing that we did. The next thing was using the same endpoint internally across multiple products. That helped save cost a lot, like the engineering cost as well to build that. If there's an existing endpoint that's shared across multiple products, it brings down the cost of development of that feature. I feel like as models get more efficient, the cost will eventually come down.
See more presentations with transcripts
