In one of the last few issues of Dr. Dobbs Journal, its editor Andrew Binstock wrote an article1 that addressed one particular issue with the Agile movement: its current lack of a footprint within the field of architecture and design. Referring to several texts on the subject, he noted that you’ll rarely come across any phrases like “Agile design” and “Agile architecture”, and with that being the case, he concludes that “architecture is a significant omission of the Agile approach.” I am inclined to agree with him. But why has this been the case thus far? There are probably a few potential reasons for the current situation, but there is one in particular that stands out to me.
Despite coming across as common sense, the simple practice of iterative development is one of its most powerful practices; one could even say that iteration is one of the core tenets of the Agile method. Whether a team of developers employs simple Scrum or extreme programming, they are able to conduct iterations on a body of code and refactor it repeatedly. In the case of architecture and design, though, what is the body of content over which one would even iterate and refactor? Design meetings can result in a few models and general documentation, but typically, they will not produce a verbose, organized representation of the design. Until the managers and architects of a project communicate with their developers at length, those precious vapors of nuance will remain in the air, and those details of the architecture will only condense and take shape once some actual code is written by the team. At that point, though, the architecture has now taken the form of an actual implementation, having never existed in its raw form. The team never had the chance to practice its Agile methods on just the design since the design never had a chance to simply exist on its own.
Usually, when metadata is put into use, it exists as simply a configurable set of data that can determine some aspect of behavior in an application. C# attributes can designate the preferred mode of a defined class, and a set of .properties files can contain the values needed by a Java application in order to establish a database connection. To some extent, one can scrutinize this metadata alone and deduce its intended effects upon the program’s execution. In some cases, it even has the power to tell a narrative, and if formed meticulously, it can tell the story of an entire architecture. Truthfully, there’s no reason that the very same metadata can’t also drive such an architecture, but we’ll get to that part later.
In a recent article2 on InfoQ, Han Xu asked whether Agile projects should be innovative. There are a few questions about Agile and innovation during development, but if metadata were used as a medium during the design phase, Agile and innovation would work together in harmony. In order to illustrate how metadata can serve as the clay for an architecture, let’s imagine a simple scenario where we’re attempting to design the server portion (processes, systems, etc.) of a distributed architecture that will be responsible for various product data; one of its goals will be to listen for clients’ requests and persist any attached data to a database:
(Click on the image to enlarge it)
Of course, there are many aspects that would need our attention, but for the moment, we’re chiefly concerned with the data being persisted, and domain-driven design is employed in order to wrap the solution around our stakeholders’ needs. More than simply being an exercise in schema design, it requires us to ask the essential questions about our architecture. How are we attempting to organize this data? What do we intend to do with it? What are any special needs associated with this data? At a meeting to discuss such things and to tell long tales (of which I am usually the most guilty and candid), the stakeholders in our hypothetical scenario talk about a few data points of interest, and there is much talk about the importance of securing access to them in a granular fashion. At this point, these data points (i.e., attributes) take on two dimensions, one organized by business value and the other organized by access control. The stakeholders of this meeting then document the discussion by creating spreadsheets that describe our product data and that infer our desired functionality:
This spreadsheet qualifies as a first iteration of an increment to domain-driven design: metadata-driven design. Of particular importance, the architecture now exists in a form other than simply jotted notes and diagrams. In this particular case, the metadata exists as an analog to database tables, but for other teams, it can exist as a format that is something more familiar (like XML). In any case, we have created a tabular, malleable model of the desired functionality that can be understood and adjusted by all participants. Now, let’s showcase the ability to iterate over this metadata using the Agile method. In this case, our supposed stakeholders arrange a trio by selecting a manager, a designer, and a developer; they then assign this trio the task of refactoring and improving the original design. Since we now have an actual medium that represents our design, we also have the added bonus of being able to measure this team’s performance with actionable metrics versus vanity metrics3. Through a discussion and a comparison of notes, our fictional trio realizes that some of these attributes (like those that belong to a price) should be accessed as collections, and in accordance with the business needs, these collections should be granted access as distinct subsets (i.e., fields). Consequently, they iterate over this metadata in order to refactor the design:
And voilà, the Agile method has been used within a design session by iterating over our metadata with a variation of pair programming (plus one). Of course, we could continue to refine the design even further as we tailor it for our stakeholders’ needs, but it’s safe to say that we have illustrated the potential of metadata-driven design.
Even though it’s often unstated, this example also highlights one of Agile’s most profound benefits: its role as a vehicle for communication, whether between fellow developers or between managers and their subordinates. The actual implementation of software needs adequate collaboration for success, but the biggest risks for a project arrive when the initial design (and any subsequent altered specifications) aren’t adequately conveyed to the entire team. Since the design phase of a project usually doesn’t subscribe to a structured format, it’s difficult to apply Agile during this time; however, if a designated format is chosen to represent the design’s metadata, it would be fairly easy to apply it. More importantly, though, its usage and participation in any applied Agile sessions could more quickly convey the purpose and intentions of an architecture, more so than simply a loose pile of models and generated classes. If the metadata is a portrait of the architecture and if a picture is worth a thousand words, you can communicate paragraphs of design alterations to any seasoned developer by making only a few changes to this organized structure and its domain of values. In this way especially, the metadata can be viewed as the fuel for any software implementation.
Such a metaphor like fuel may be an appropriate way to describe metadata-driven design, as a way to transcribe the ideas for an architecture or one of its subsystems...but why can’t this sentiment also be taken more literally? I would postulate that this metadata can serve as more than a sounding board for designers; it can also be used to help create the actual software. Like cases where UML can be used to generate classes and even entire Spring framework applications4, there might be an open source solution that could be helpful here; if one is not available, a small team could create a tool that generated such classes from the tables. Returning to our hypothetical example from earlier, imagine if we created such a tool for that purpose and then generated a class based on these attribute rows:
In this scenario, our fictional stakeholders would generate a Product class that contained our attributes, along with empty stub methods that would need to be implemented by the developers. It seems somewhat satisfactory as a solution, but there’s always room for improvement. In that case, we’ll resume the story from our previous example, and we’ll assume that the trio from before has the same sentiment about enhancement. They decide to iterate both the design and the implementation by treating them as a whole, in an attempt to integrate the metadata (i.e., the design’s transcription) with the code (i.e., the implementation’s transcription). After making a few adjustments to the architecture metadata (including organizing attributes into separate groups), they generate the UML for a new set of classes:
With this iteration of the design and the implementation, it is no longer necessary to use the metadata for class generation since we have now intertwined it with the code. It’s true that the design now has an additional degree of complexity, but there are also benefits as well. Now, whenever any alterations are made to the data design of the architecture, the code will now immediately detect those changes and absorb them. When a new attribute is introduced to the data design and inserted into the metadata tables, the code will very likely not require any alteration since it treats all data points in a streamlined manner. In addition, the operations of the Product class (like readData() and persistData(), for example) will rarely ever need to be altered since this iteration’s integration has created a systematic approach that makes generic processing possible. Now, Agile can now be applied to both design and implementation simultaneously. Truly, we could iterate over this design even further, using any number of Agile tools to help us.
This particular example of deeper investment into metadata-driven design also has auxiliary benefits. Namely, it has the psychological benefit of improving morale and trust between managers and developers. Especially in those times near a milestone or an important deadline, the relationship between a manager and the developers can be tense as specifications seem to be reconsidered and reprioritized every day. However, if sufficient investment is made with metadata-driven design, every modification can have significantly less overhead attached to the relevant task, resulting in less work and stress for the team. By altering metadata records to satisfy new or different requirements, a manager can then immediately observe the desired changes in the software’s behavior. By empowering the manager with this ability, this mode of operation provides a consistent affirmation to the team about its ability to deliver expectations. By being bound to the metadata, the manager and the development team have a common bedrock upon which they can build their work environment.
Of course, the hypothetical scenario from before pertains to enterprise programming and/or software tailored for that field, and it involves a more proprietary implementation that lacks the usual amalgamation of trustworthy open-source solutions. These factors might dissuade some from the consideration of metadata-driven design since it might seem to have no relevance to their projects. However, if one observes the abstract message from this particular exercise, the lesson learned is of how this particular method can be applied to any given problem. Much like the Lean method isn’t only used by software, metadata-driven design can be used to help with solving software problems other than simply data design. For the scientist who works with niche systems that lack available solutions (or the funding to purchase them, more than likely), it can be used to outline a script framework that automates the processing of recorded data, and like the previous example, the design session’s metadata can even be integrated with it. In other cases, it can be simply used to create a contract for the interprocess communication between two different companies. If they’re feeling especially bold, they could even use the established metadata as a keystone to generate code, much like stub generation for remote procedure calls (if you’re old enough to remember that). By regarding the metadata as a potential partner to your implementation, this method has the potential to guide a domain-driven design to its optimal conclusion. In any case, metadata-driven design can probably be of some assistance to finding your solution.
About the Author
Aaron Kendall is a software engineer in New York City, with nearly 20 years of experience in the design and implementation of enterprise data systems. After beginning as a developer of device drivers and professional software, he became passionate about software design and architecture. He has created innovative business solutions using a variety of platforms and languages, as well as numerous freelance software projects that range from open source packages to game design and mobile apps. If you would like to read more about his work, you are encouraged to visit LinkedIn and his blog.
References
- Binstock, Andrew (4 November 2014). In Search of Agile Architecture. Dr. Dobb’s Journal
- Xu, Han (6 January 2015) Should Agile Projects Be Innovative? InfoQ
- Ries, Eric(September 2011) The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses
- Avram, Abel (14 November 2013) GenMyModel: An Online Real-time Collaborative UML Designer InfoQ