Agentic AI Architecture: Current Challenges and Future Opportunities

Most software engineers and architects are innately familiar with microservices and cloud-native architectures and have been involved in building systems based on these architectures for many years. Microservice-based systems tend to consist of tens or hundreds, and sometimes even thousands of services, responsible for different types of data processing, be it API endpoints, data pipelines, back-office workflows, and anything in between, but they do have one thing in common - they merely process data using a logic determined either by code or based on data retrieved from data stores. This doesn’t mean that such systems are simplistic - far from it. Still, despite their complexity, they can be reasoned about deterministically, which helps define and scope functionality, implement and test changes, and operate such systems, even in the face of infrastructure- or application-level failures.

All of this is about to change. Indeed, a good share of future software systems will still be deterministic and resemble what we are used to today. Still, quite soon, swathes of IT systems being designed and delivered will be leveraging AI-based solutions to power user-facing functionality. For most, if not all, architects and engineers, this will require significant adaptation to the new reality of the IT industry.

The adoption of AI in software architectures is driven primarily by the new capabilities that it offers, which are particularly useful for products used by knowledge workers. Historically, IT systems focused on storing, processing, and making data available via APIs and user interfaces. At the same time, humans were responsible for a good portion of data administration, utilizing business rules, procedures, office "tribal" knowledge, and their own judgment. It’s fair to point out that some, or even much, of this intelligence was built directly into IT systems to improve productivity, particularly when scaling business operations was critical. With AI-enabled capabilities, software systems can shift from operating on raw data to knowledge, thereby elevating IT's role from merely a data management and processing function to knowledge-level support for business operations and end-user functionality.

It’s difficult to gauge to what extent AI-based solutions will dominate next-generation software systems. Still, by all signs, the impact will be so substantial that we can speak about a new type of software architecture, one where AI powers large areas of functionality, making humans using products built on top of next-generation systems much more productive. Agentic AI Architecture is a software architecture in which agentic AI systems are leveraged on a substantial scale, geared towards making optimal use of generative AI (and other) models. This type of architecture builds on microservices and cloud-native architecture and adds agentic AI capabilities.

The Emergence of Agentic AI Systems

Agentic AI isn’t exactly new, but equally, it’s not yet well established either. Over the last couple of years, agentic AI systems have evolved from simple, single-agent solutions to chains, to graphs and multi-agent systems (MAS), where multiple agents collaborate, organized in various topologies. This area is still undergoing active development, and more elaborate use cases will inevitably emerge. It’s fair to say, though, that different collaboration strategies or topologies work best for specific use cases, so choosing a simpler solution can be advantageous in many cases.

Agents make the most of the large language models (LLMs) they interact with by providing the right context. Here again, after starting with predefined prompts, the solutions gradually shifted to a dynamic context based on output from other agents, context propagation, or data fetched from external sources via retrieval-augmented generation (RAG).

As the amount of context accumulated from different sources began to cause problems with diminished reasoning, engineers developed new ways of managing context, and context engineering emerged as a separate practice. The progressive context disclosure technique can further enhance the context preparation step by allowing the agent to selectively load detailed information into the context window based on the user’s prompt.

Since LLMs have effective limits on context size, the role of context engineering is to ensure that the context used maximizes the model's likelihood of producing the desired outcome, or at least minimizes the chance of hallucinations due to vague or inadequate context. Effective context engineering requires context management to store, optimize, and reuse context elements for subsequent agent invocations.

Lately, memory systems for AI agents have also emerged, enabling further enhancements to context editing. These systems organize different types of long-term memory (episodic, semantic, procedural) for storage and querying, thus allowing agents to supplement the context with past experiences/decisions, relevant facts/definitions, rules, or learned behaviors.

Context engineering has become a foundational architectural concern for agentic AI systems. It will undoubtedly continue to evolve as the needs and expectations for building and operating AI-based products grow.

Agentic AI has developed its own specialized set of communications protocols. This process was largely driven by challenges related to the complexity of agent development, the need for standardization, and the need for improved interoperability:

• Agent2Agent Protocol (A2A) - designed for multi-agent systems, allows interoperability between AI agents
• Model Context Protocol (MCP) - focuses on communication between AI agents and external services (databases, tools, etc.).

From Agentic AI Systems To Agentic AI Architectures

For agentic AI systems to be valuable, they need to be integrated into a bigger enterprise architectures. In the enterprise setting, incorporating an agentic AI system can be done by:

• Embedding an agent into a microserive exposing an API (HTTP/GRPC)
• Using an agent as an MCP tool or with A2A protocol
• Having an agent microservice consumer messages from an asynchronous channel (queue/topic)

A large-scale enterprise-level architecture may comprise many specialized agentic AI systems. Since AI agents can use MCP to use database, services/APIs or any arbitrary data sources directly, it’s important to ensure that their interactions are constrainted within a bounded context, as defined by the domain-driven design (DDD). Limiting the reach of agentic AI systems within larger architectures reduces coupling between distant or unrelated components and data sources, allowing more cohesive areas to be maintained by engineering teams without excessive dependencies or large blast radius if changes are introduced.

Figure 1: Multiple Agentic AI Systems in Separate Bounded Contexts (Source: created by the author)

Figure 1 (above) depicts somewhat simplified architecture with two bounded context, each one embedding an agentic AI system with multiple agents (A), as well as other microservices (S). Agents only use local data sources and services for context expansion or as tools, hence limiting the coupling with other parts of the boarder software system.

Performance and Scalability of Agentic AI Systems

The scalability of cloud-based software architectures is well understood, as architects and engineers have been building and operating them for some time. The same can’t be said for agentic AI systems, where LLM call latencies are higher, and more complex multi-agent systems can further increase processing times. The performance of agentic AI systems can’t be measured solely by standard metrics such as throughput, latency, or error rates.

Researchers at Google and MIT published a paper introducing a framework for scaling multi-agent systems. The team behind the framework tested various multi-agent coordination strategies, each with several configuration parameters, including the number of LLM calls, agents, and interactions per agent. Scientists concluded that the best coordination strategy is task-dependent: financial reasoning does better with centralized orchestration, while web navigation benefits from a decentralized strategy.

The results shared by Google and MIT demonstrate that there is no one-size-fits-all approach when it comes to agentic AI, and teams building complex solutions have to experiment and empirically establish the agent coordination strategy that is best suited to the task.

Security Challenges of Agentic AI

Incorporating agentic AI into the distributed system architecture significantly alters the security model. Agentic AI contributes additional security threats, particularly around the agent runtime, tool servers (MCP or others), and the model provider. AI agents, for instance, can be susceptible to prompt injection, instruction confusion, or tool output poisoning, leading to unintended actions or leaking data they have access to. Additionally, using external LLM models introduces risks around data residency, retention, or usage.

Furthermore, present access-control models, honed over many decades, are ill-suited to cater to agents, which can’t really be treated as human users or backend services. Cameron Pavey, senior developer at Auth0, recently proposed a new permission model specifically targeting AI agents. He argues that regular (non-AI) backend services are generally more trusted because they are deterministic and auditable, while authentication sessions and user interfaces restrict users. Agents, on the other hand, can arbitrarily trigger multi-step workflows that involve other agents and tool calls without the user’s explicit knowledge or permission.

Pavey proposes using capability-scoped permissions and task-scoped credentials as key steps toward solving the problem of agent permissions and adopting fine-grained authorization systems for layered enforcement.

It’s worth pointing out that this is not the first time that the security landscape has experienced a marked expansion. When microservices and cloud-native architectures entered the scene, we observed a shift in this area as well, leading to a revision of the access control mechanism and protocol and the emergence of now-common technologies such as JWT and OAuth.

Observability

Considering the non-deterministic nature of agentic AI systems, observability is key to being able to effectively build, enhance, troubleshoot and operarate them.

Teams should aim at complete coverage across all pillars of modern observability platforms:

• Logging - use structured logs recording decisions with reasons and evidence; include correlation IDs; scrub personally identifiable information
• Metrics - measure quality, latency, spend, create SLO dashboards for performance and budgets
• Tracing - trace tasks/steps (plan, tool, validate); include correlation IDs
• Alerting - alert on performance degradation, latency spikes, anomalies
• Runbooks - maintain a suite of runbooks with regular reviews

Blending of Microservices, Cloud-Native, and Agentic AI Architectures

Despite the expected rise of agentic AI as the driving force behind future IT products and services, we cannot expect an instant switch. For many years, the distributed systems architectures we know today will coexist with the next-generation ones (which build on top of them), similar to how service-oriented architecture (SOA) was phased out in favor of microservices over a few years.

Additionally, given the relative infancy of agentic AI, this space will take time to mature and gain widespread adoption, allowing for a gradual transition and the emergence of new patterns, best practices, and technologies that will further drive adoption.

Finally, not all products or business processes need to leverage agentic AI, or the use of AI may not be immediately financially viable. It may be that certain industries or classes of products or systems will not benefit from AI at the same pace as others. In that sense, microservices and cloud-native architectures are here to stay in one way or another.

From Microservices and Cloud-Native to Agentic AI Platforms

With Agentic AI taking center stage and maturity and adoption within the industry growing, I expect many components and solutions, currently often custom-built, to be offered as features of more generic platforms that will support multi-agent system delivery and operations, as well as the integration of those into broader enterprise-level architectures.

In particular, foundational agentic AI building blocks, such as agent memory, asset registry, guardrails, and orchestration, as well as support for observability and operations, should all become built-in capabilities of agentic AI platforms, whether homegrown or offered by vendors or cloud providers.

There are several agentic AI platforms/frameworks that developers can use today, including LangChain (with LangGraph), CrewAI, Azure AI Foundry, Google Agent SDK (ADK), Amazon Bedrock AgentCore Gateway.

This process will follow the same path as it did for microservice architectures, where growing needs from organizations adopting these new architectures have necessitated the creation of internal, self-operated solutions that externally hosted and managed products will gradually replace.

The Future of Agentic AI Architecture

The industry is on the verge of a major shift, and Agentic AI Architecture has a strong chance of becoming a dominant architecture for distributed systems over the coming years and decades. This doesn’t mean that microservices and cloud-native architectures will become obsolete; rather, they will form the basis of the next generation of systems, in which AI powers much of the user experience. What architects and engineers are familiar with today will form a foundation for delivering robust AI-powered products and services that combine data, knowledge, and compute-based reasoning.

The shift described here will increase productivity and improve ease of use for many knowledge workers across many sectors. It will greatly reshape not only the IT industry but our entire modern society.

While we are still in the early days, with a lot of innovation, moving goalposts, experimentation, and learning, we can certainly expect an exciting road ahead.