Knowledge graphs describe knowledge domains based on expert input, data, and machine learning algorithms. eBay is using an application/infrastructure knowledge graph to manage its vast service architecture and provide a better experience for the roughly 200M buyers visiting the site.
eBay’s vision – known as INAR, Intelligent Architecture – is to build sustainable service architecture by providing automated visibility, assessment, and governance Intelligence. In this pursuit, we developed a new approach to model and process the application ecosystem using a knowledge graph.
In eBay's vision of an "intelligent architecture" (INAR), there are three major challenges that must be overcome, stemming from blindness, ignorance, and primitiveness. Blindness is the difficulty of observing how an architecture evolves, leading for example to inadequate dependencies, monoliths, redundant services, and duplicated functions. Ignorance is related to the lack of proper metrics, which are essential to improve operational efficiency. Finally, primitiveness is the absence of a correct approach to diagnostics, engineering and automation, which all should pave the way to the application of AI methods.
Based on this vision, eBay engineers wanted to reduce blindness, detect patterns and anomalies, and automate IT operations. The major hurdle was to identify all data sources and connect isolated management domains, an ideal task to be solved using a knowledge graph.
One of the first steps in developing a knowledge graph is to calculate the best application metrics and applied machine learning algorithms to automatically cluster the applications. We developed metrics that measured the popularity of applications based on real-time traffic flows and run-time dependencies.
Using this approach, eBay engineers could organize their service ecosystem in clusters (galaxies) based on how active they were. About 77% of the more than 3,000 eBay services turned out to belong to low-activity clusters, while 10% of the remaining, high-activity applications ended up located in an incorrect availability zone. This provided the first chance to improve customer experience by way of this novel approach. Other dimensions explored by eBay architecture knowledge graph are dependencies between services and criticality. A graph provides an immediate understanding of multi-dimensional relationships. For example, you can associate meaning to the thickness of arcs connecting nodes, to their color, or to node size.
eBay knowledge graph is built automatically from the different data sources identified and can be searched using a custom graph search tool that dynamically generates a query for the graph. This is the base for providing customizable visualization, application metrics, and intelligent layering.
eBay engineers are already at work to add support for anomaly detection, whereby anomalies are represented as graph nodes, and to include service API metadata to further enhance the AI-related aspects of their knowledge graph.