Transcript
Balakrishnan Bala Ramdoss: Let's actually start the talk with this video.
"Hey J, this project might need a bit of zhuhzing, but you're the big enchilada. Holler back, Warren."
"The most capable individual to undertake this task. Please let me know your thoughts. Best regards, Warren."
Full video credits to Apple.
A lot of people had mixed feelings when this ad came out. Personally, I thought it was a pretty creative, funny ad for a feature that feels very obvious. This looks like something ChatGPT could have done years ago. What took Apple so long? Maybe Apple is collaborating with OpenAI. After all, this is a simple call to ChatGPT in the mail client. Or maybe Apple is trying to do AI their own way, like security, privacy, and stuff. They must have done something that took them this long. I had so many questions when I saw that ad for the first time. We use so many apps these days that has this chat-style customer experience. How do they build this dynamic CX? How does the model know what CX to choose and how to build this at scale?
Background
I'm Bala. I'm a senior software engineer at Amazon. I've been building mobile apps for the past 10 years. The last five of those years, I spent building Amazon Lens. Amazon Lens helps customers find products using your camera or image. Lens is available on iOS and Android. If you have the Amazon app installed, I highly encourage you to try it. It was not until early 2024 where AI got interesting to me. I understood what AI is trying to solve, that the search system is trying to solve for quite some time, like answering users' questions. My most recent work, Amazon Lens Live, has been released to public. I learned a few things while building this, and I'd like to share that with you. Lens Live supports live scanning, obviously, and it allows you to ask follow-up questions on what you scanned.
AI in Apps
Who is this talk for? You're an app builder, building something with AI. How many app builders or frontend engineers here? You're curious how this conversational experience works at scale. Or you're like me, you're concerned about privacy. You're trying to build privacy-first features, trying to understand what options you have. Most of us know this already, but just for the kicks, I used Gemini to do research on top 100 apps across Apple and Google Play Store to see how many apps have AI in it. Can you try to guess which among these doesn't have any AI in it? McDonald's. I'm not surprised, but it's also surprising that many apps these days have some AI feature in it. I found that 89%, almost 90% of the top 100 apps have AI in it, and 36 of them have explicit chatbot in it.
How to get a chatbot. When I was trying to find the available options, I realized that this market has been fairly matured. There are hundreds of these chatbot-as-a-service companies out there. They provide you with this plug-and-play service that you can add to your website. If you're looking to get a quick text-based chatbot, this would be the fastest way, chatbot.com, botsonic, chatbase, these are some of the top players. For mobile apps, however, I noticed these plug-and-play services don't play well. It's understandable. The apps require deep integration for you to perform certain actions, like navigation into different parts of the app or reading something from the app itself. This is how a generic integration with one of these chatbot-as-a-service looks like. The LLMs are not really driving the CX here. You're likely to give control of a tiny portion of your window to these chatbot-as-a-service companies. Because of how it's set up, it is not really possible to get real-time information in the chatbot's response. Maybe this will evolve over time. We'll see.
Frontend (Conversational Experience)
Let's look at a design where conversational experience is deeply integrated into one of the production systems. I'm sure you'll have a full-blown design far more complex than this documented somewhere. I think, personally, the overall AI conversational experience evolved from search systems. We started using ChatGPT by asking questions. That's all it did when it first came out. Any large-scale AI system will have some similarity with search systems. Obviously, I'm not going to go into detail of every little thing in this. I want to highlight the AI parts and show you how frontend plays a role in all of these. We'll start with the frontend itself. Frontend is where the journey begins. Unlike the traditional way of making an app AI-ready, this setup gets a little more involved. Four years ago, before ChatGPT, if someone had asked me, Bala, we want to build an app feature that might be slow to render, you will not get all the response at once, but you have to make the UI capable of doing that.
I would have said, "No way. I don't even understand what you're trying to say." Yet, this last year, I have built a feature slower, more than 5 to 10 seconds to respond, and rendered things in chunks. I actively worked to hide that latency from the customer. Somehow, I feel good about it. I'll share my experience building that system. Traditionally, an app makes an API call, waits for the response, update the local storage if needed, and then show something on the UI. We spent years crafting this flow. We optimized this so much that we wanted it to be fast, tracked our metrics, just so the experience is slick. When AI powers the CX, backend is now streaming the response back. You're not going to get everything at once. The clients now have to make sense out of these chunks. The UI updates are now more complex, and they have to be event-driven.
I noticed a trend among the users. We are somehow trained to wait for anything when we see the word thinking. ChatGPT trained us. It has evolved for a bit over the last two years, but still, I think, your users are still in that forgiving phase. They're ok with waiting for a bit because ChatGPT trained us. Because of this reason, the traditional time to interactive, or the round-trip time, the metric we care about the most, don't make sense. The experience should be interactive even when you're waiting for a response. Here's a good example of how not to do AI features. I'm not trying to trash this app. This is LinkedIn. If you're building something with AI, do not try to do this. This is a premium feature. You try to update something and then use, write with AI. You click the button and it waits forever.
You can't even cancel. It says cancel. Do you remember the old-school ChatGPT where you enter something and then it starts immediately spitting back chunks and chunks of text? The metric, time to first token, was first introduced at that time where it measured the time after you submitted your prompt and then the first word it rendered on the browser. Since apps became more modern, it's not really spitting out words and words at a time, the experience has become more multi-modal. The app renders one meaningful CX unit at a time. The metric, time to first chunk, become more reasonable. The time to first token is still measured but more in the backend. The system does more things before it gets to the customer. The key takeaway here is that models are slow. You can't expect everything at once, and you will need different ways to measure your latency. The loading screens. We are very used to seeing these loading screens in all our apps. On the left, we have the classic loading spinner, and on the right, we have the skeleton screen. These are great. They show the user that something is about to happen.
There's one key problem when it comes to AI features. What happens if we show this for more than 3 seconds, 5 seconds? The user thinks something is slow, the network is slow, or something is about to go wrong, or something might have gone wrong already. Of course, if you're building a conversational style CX, you can show thinking and then get away with it. What if the feature is powered by AI but it's not really part of conversation? These two apps do it slightly differently. I don't have a straight answer to this. After reviewing hundreds of designs for AI features, I can share my opinion. Apps are trying to figure out the thinking CX without the word thinking. These two apps here, they're trying to show existing CX, like the material loading and the skeleton screen but with a different style gradient, shiny colors and the AI icon at the top.
It's just trying to show a portion of this non-AI app. Look at that, it's just the weather app. You don't need AI for that, for a non-AI app to show some AI feature in it. There's also a new pattern where I noticed the product actually evolves to handle this latency. I personally like this because it's useful to the customer. Take a look at the Amazon Lens where the model takes a while to prepare the response. In that time, the user would be actually seeing the object being tracked. It's engaging the user, and created this experience where the user would be able to provide their input. If they select something else, then the model would know what the user wants to focus on. I don't know if you have tried Gemini Live or Copilot. You ask Gemini some question by opening your camera. You will not know what the models are until you get the response back. There's no information shown to the customer.
Conversational Design
We touched on general housekeeping when it comes to building AI features. I want to focus on the conversational design a little bit. These screenshots, one from ChatGPT and one from a popular news app, brings out what's really common between them. They both have dynamic layout, cards-like structure, and they both have inference scroll. You probably got the idea. The conversational design follows a very simple pattern of building a news feed. It's trying to get the most relevant information, the information retrieval part. It also chooses the right CX template for each bit, the dynamic CX part. I'll try to break it into two that I felt important for frontend engineers, reusing bits and pieces of existing CX and dynamic rendering. Focusing on reusing bits. Throughout this presentation, I'm going to use this hypothetical flight booking AI app. While you present the search results for a flight booking, you'd show something like this.
You have the from and to. It has nice formatted information for your user. If you have to build an AI feature, it's really important to reuse that same element. The level of information is still the same. You maintain your brand identity. Your users know how to read this. Remember the whole system I showed you earlier? If you already have ways to power these tiny bits in CX for your search page or detail page or whatever, it's a great opportunity to fine-tune them for your AI use case. Completing that example, if you were to expect a response from your hypothetical flight booking AI app, you'd expect something like this. You ask some question, it's going to show you a nice multi-modal CX with the answer, and then some follow-up questions. Instead of just using the model output, it would be this dense text. You'd spend more time looking at it to understand what's going on.
Moving on to the next part. There's one pattern I think would be most useful for building this fluid and dynamic CX, that's server-driven UI. If I take you back to this screen, I think I'll be able to help you understand a bit better. Server-driven UI is useful for not just AI features, it's any dynamic CX in general. Backend engineers love to build APIs. That's all they do. The APIs are very detailed, but what they don't like about it is that when you ask for a client-specific logic, they're like, no. They're not going to add any client-specific logic. In general, the APIs that they provide are often too long. For example, this flight card. The API response would be 10,000 lines, because why not? They have all the information about the flight. The clients don't need such big chunk of JSON. We need a view model for the frontend, just enough so you can render this information to the customer.
Server-driven UI really helps with that. Backend for Frontend, this is one of my favorite topics. BFF is just part of server-driven UI. It's not a pattern of its own. At this point, the frontend outside your client has enough information to know what to render. This BFF shows you how to render that. All the platform-specific logic on Android, you have to show some different CX. On iOS, you need to show a different loading spinner. All of these go into this bit. This is my favorite part about BFF. It prepares a very detailed action payload. Let me show you an example. It prepares the payload to help clients handle user actions. What to do when someone clicks on the flight card. What to do when someone long presses a question. These tiny little bit details, it takes the heavy lifting and moves all this decision to the backend, which essentially makes the client to know nothing about what's happening.
Coming back to the frontend, I mentioned a bit about server-driven UI, but there's a great talk by Christopher Luu from an earlier QCon edition. I highly encourage you to watch that video. What's my take on server-driven UI? It's not a black or white thing. It's a spectrum of how much you want to control what your client renders. You have to find the right balance between that control and the effort it takes for you to build it.
I've covered on all the frontend topics except for one thing. The frontend doesn't really get credits for handling these evolving AI use cases. All these AI apps, they work better when you provide more information about you, like critical information, like providing access to your camera or your private chat. Some highly critical information, so AI can make better decisions to show the right information about you. Frontend is the one that's in front of the user. Frontend now has more responsibility. I want you to hold on to this thought before I move on to the next. We'll come back to this.
Query Content Creation
Zooming out. I want to focus on query understanding. Query understanding is an important part of any search system, and it is important for an AI system as well. Why is it relevant? Query understanding is generally the first service call outside your frontend, and it generally adds clarity to your customer questions. Query understanding helps categorize your customer's query in a way that it can put it into a certain set list of things, so customers trying to search for something, customers trying to order something. It also adds clarity about the customer. What is the customer's preferred currency? In our case of flight booking AI app, the query understanding gathers information. This customer always prefers non-stop flights. The more information you know about the query, the better the answer will be. Query understanding is key to getting that information. When it comes to AI features, there's one more responsibility that's added to query understanding.
Now it has to also know about the app's capabilities. I want you to note that this is the first service outside frontend to actually know about frontend. I'll show you an example, the same flight booking app. When the client makes an API call to the hypothetical AI feature, the backend needs to know what the client is looking at. In this case, the URL of the flight page. It shows the from, to, and whatever input that is needed. The backend can make better decisions. Query understanding often collects this information at the very beginning so the other downstream services can benefit from it. This is not to be mistaken with building a RAG application. We are just trying to merely add clarity to the customer's query.
Orchestrator
We'll move to the next bit, orchestrator. Orchestrator has less to do with frontend, so we'll not go into the detail. For a given user query, orchestrator has three things to do, collecting the information from the information retrieval system. Choosing the right prompt. Then wait for the model to respond and make sense out of whatever comes back. Remember the model doesn't know anything on its own. It's like another Jon Snow. Someone has to feed all the information to it to get the right answer. I mentioned choosing the right prompt. This is also not so much about frontend, but stick with me for a bit. I'm sure most of you heard about the term prompt engineering. How many of you think prompt engineering is a real skill? Hope I don't get canceled for this. I asked ChatGPT what a prompt is, and then it responded to me with this, "A prompt is a set of instructions provided to a model to elicit desired output." I think that's really well put.
Let's just talk about what prompt engineering has to do with the frontend. To have the LLM decide what the right CX is to show for the query, the prompt needs to surface the app's capabilities to the model. In any large-scale system, there are bits and pieces that evolve over time. Especially when you use server-driven UI, new CX elements get added all the time, and sometimes an element gets removed. The LLM needs to know what its available options are to choose the right one. Models can summarize emails. Models can understand and write code. It can also choose the right CX when you allow it to choose. I'm not a prompt engineer. I learned that it is very easy to get started, and you can go far, but then it's very hard to get it perfect. In this presentation, I have some examples of prompts. I had one of my applied scientists work with me to validate those prompts.
Let's continue with the flight booking app as an example. The customer had asked, I can't make it to my morning flight. Are there options for me to fly in the afternoon? It's a simple information retrieval question. You want the system to respond with this nicely formatted multi-modal response. The text response would suffice. It has the same information, but you would prefer the left than the right. The prompt should include instructions on how to construct this UI. Let's take a look at one of those simple prompts. In this template, there are details about the prompt, instructions for the model, some input data. Then more importantly, the output schema, like how you want the model to respond back. In the system prompt, the system is being set up in a way that the model should decide what UI component is best for the query, and it should always respond back the expected format.
Here's how the context gets framed. I'll go bit by bit. The user context helps the model to learn more about the user. You want the model to greet the user for any question. The model is provided with data for it to answer any question. The customer had asked, what are my alternative flights? You present that information in this context piece. This is what I wanted to talk about most. We're trying to make the model look at what the available CX options are. Here's an example. It's still simplified, but I want to focus on just these two bits. You're providing these two options. When you show the available flights, I want you to choose one of these two. There's a flight list. Obviously, if there are hundreds of flight options, you want to choose the flight list. If there are four or five options for the customer, I want you to choose the carousel. It also instructs the model how it can construct that CX. When you choose this component, I want you to also frame the data spec for that CX model.
We have talked enough about prompts. At this point, we have done everything to make sure the model understands the query, has all the facts to answer that query, and also the available CX options to choose. Once it gets the prompt, the model will respond back, often token by token and in streaming. The orchestrator will then accumulate it until it sees a meaningful chunk and then goes back to the BFF. For example, if it sends you back a flight card with a flight number, your BFF then takes the responsibility of digesting it, making it ready for the view. It happens for every chunk. Unless you work at one of these companies, you're going to call an external model provider. Whatever I've shared so far is still relevant, no matter what model provider you choose. There may be slight nuances to the prompt structure, but if you choose to go this route, play with it, you'll get the hang of the right way to prompt the system.
It's been three years since ChatGPT, and all the features that came out so far has been mostly about conversational design, building a chat application. They're really trying to show the strengths of a large language model. From an app developer's perspective, we're not going to see a major shift or major AI feature outside this conversational setting. However, I noticed that these AI features started to slowly show up in bits and pieces in places where you don't expect. This is Google Journal. It has a lot of AI features in it, and it doesn't feel like you're chatting with Gemini, which I really appreciate.
AI on the Frontend
I'd like to bring you back to the frontend, this time with a slightly different perspective. Think about building an AI app that relies on very critical information, like I mentioned using the camera, accessing the private chat. Sometimes you don't have the option to take the critical information outside of your frontend client. Think about adding an AI chatbot to your WhatsApp, for example. The end-to-end encryption is enabled. Now the frontend has to evolve to handle this responsibility. It's not just rendering things to the customer. It's going to do some heavy lifting. This journal app I showed previously, the model that powers this AI feature runs entirely on the client. The data of the journal entry never leaves the device. We'll talk about AI on the frontend. It's not really a new thing. It existed long before ChatGPT. A good example is a credit card scanner. The scanner runs entirely on device.
It never uploads it to the backend. It reads your numbers, expiry, name, and it just submits that information to another firm. It is made in a way that it's compliant with payment standards. The need for having a local large language model has just begun. After all these new AI use cases, there's a need to run truly private inference.
On-Device AI
Devices that were capable of running smaller models like the credit card scanners, they were relying on CPUs and GPUs. Now they are powered by neural engines or NPUs. These on-device models, of course, you can't expect it to perform the same way as Gemini or ChatGPT, but they are surprisingly good at many smaller tasks. Of course, we can't talk about on-device AI without talking about Apple and Google. These are major platform providers in the U.S. and worldwide. They naturally became the only foundation model providers for mobile devices. They both have invested so much in on-device AI space. Remember the video I showed at first? Apple, despite being so late into the AI space, they have so much power. In my experience, running models on Apple hardware led to much higher performance than Android devices. Google, on the other hand, had done so much already. They have open models available, and they provide APIs for Android apps to run on-device inference.
They are now actually evangelizing it. They are talking about the Nano Banana. If you don't like Apple or Google's foundation models, you have an option to bring your own. I'll talk about it. Just a quick word of caution. There is so much going on in this on-device AI space, but Apple and Google are trying to do it the right way. I personally consider these two as alpha stage. If you have a use case that is something super light, I'd encourage you to try it out. Depending on your use case, you may be able to benefit saving cost or latency using these on-device models. A good example is Amazon Lens that I showed. Some of the inference happens locally, and the backend uses that information to provide the results and other features. It helps with saving cost and latency.
Some quick pros about these on-device models. There's no cost to run inference. It sounds very enticing, but under certain bad natures, it takes up storage and memory. Of course, if you use the device, the model providers from Apple and Google, there's no problem of storage, but still, when you run inference, it consumes power. Because it runs locally, there's minimum latency, and because it doesn't upload anything to the backend, you get the highest privacy. Should you choose to get your own model, then it's your responsibility to manage the logistics to get that to customer devices. I've done that once, and I can talk about it if anyone is interested.
Key Insights
Summing up our talk here. If you have to work on building an AI feature, I would encourage you to focus on your prompts. There's a lot you can do by just adjusting your prompts. Remember, LLMs can work with different tools to make API calls, write code, can write emails. It can also power your CX. When it comes to building dynamic CX, server-driven UI and BFF are your best options to scale your frontend. User hardware is evolving, and consider on-device AI if you have a lighter use case. You might end up using them sooner than you think.
See more presentations with transcripts
