New Relic recently open sourced the El Dorado UI project, a Sinatra app to query and visualize relationship data stored in a Neo4J graph database. El Dorado is a structure warehouse that stores the relationships between all the development teams, projects, and microservices at New Relic. Ward Cunningham, the inventor of Wiki, described the origin of the project and demonstrated its capabilities at the Explore DDD conference in Denver, Colorado.
El Dorado came about after Project Upscale, when New Relic dramatically changed their entire software development process. After more than 100 engineers defined new teams, then self-selected where they wanted to work, the net relationships were not easily depicted on a simple, hierarchical org chart. Furthermore, it was important to keep track of the microservices that were freely created by these teams. Cunningham's colleague Andrew Ettinger initiated the study of New Relic's engineering records, and dubbed the project El Dorado, the place of treasure.
A graph database was a natural fit for storing relationships between teams, and El Dorado uses Neo4j. A daily ETL process keeps the central database up to date with the latest changes from several upstream data sources. Cunningham refers to this as a "'structure warehouse,' that is, a warehouse that stores structures instead of just data."
El Dorado System Diagram - Source: https://github.com/newrelic/el-dorado-ui
The power of El Dorado UI is being able to easily visualize complex relationships within the organization. Users can use canned queries, or write ad hoc queries in Neo4j's Cypher query language, and see a graphical representation of the results of a query. The Dot notation of GraphViz is used to render an SVG image that could range from simple a few simple boxes and lines to a complex diagram with dozens of entities.
As Cunningham used El Dorado, the value of having an intuitive interface to visualize complex relationships became very apparent. With five sources it was useful; with 15 it was fascinating; with double that, it will be indispensable.
More information about the El Dorado project is available on Cunningham's Federated Wiki.