BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles The Elusive Dependency

The Elusive Dependency

Bookmarks

I’m guessing that you know the feeling. You are about to present at a project review meeting, and you know that you are in trouble. Looking at the 5-page slide deck, you get the sinking feeling that this isn’t going to go well. Having had three late nights in a row while hacking the original 16-page presentation into the 4-page template hasn’t helped. The admittedly precarious explanation for the unanticipated technical dependencies collapsed like a house of cards, and you were left with a set of career limiting statements of the obvious. The next 30 minutes with the program director and her management team risks furthering the perception that architects don’t do any real work.

What went wrong?

Architects are supposed to understand how the system and its elements hang together – an important role. Yet the true complexity of architectural dependencies can elude any team until it is too late.

A colleague once noted that dependencies arise whenever code is shared. APIs, service specifications, and programming by ‘copy and paste’ are examples of ‘code sharing’. But these are not the only source of dependencies.

We create architectural dependencies whenever we ‘share’ architectural information and, more generally, architectural knowledge. And we inevitably share architectural information when we build integrated systems. These dependencies are often of greater complexity and harder to find.

The tricky part is that architecture, by definition, is about the relationships between architectural elements (e.g., components, processes, or entities). One could therefore argue that the architecture specification is, in itself, the complete set of architectural dependencies. However, danger arises whenever the architecture sharing is done implicitly; that is, we make assumptions. And within large-scale enterprise environments, it is almost impossible not to assume something.

The elusive dependency

A few years back, I interviewed a project team working on replacing an old enterprise application. One of the application’s core functions was invoice generation, a capability the business had extended over the years. The extensions had wisely been built external to the core product application, for a range of reasons, including vendor support and maintainability.

What materialised during the replacement effort, were design assumptions within the extensions about the old system’s currency rounding strategy (e.g., number of decimals per calculation step). The extension designers had taken great care with the interface specifications, and the new application was both functionally and technically compatible with the extensions . . . except customer invoices now had what looked like elementary rounding errors.

The obvious lesson here is to avoid making assumptions about how another system works – a form of defensive design where you aim to minimise external dependencies as much as possible. But it is not always possible to completely eliminate:

  • Semantics and life-cycle: All architectural elements assume some form of semantics and an implied life-cycle. These can be tricky to express as part of interface specifications and architectural models, but they are important.
  • Multiple implementations of the same logical element: The minute your company starts to buy packaged applications, this becomes an unavoidable reality.
  • Manual integration (i.e., human activity is required to move data from one system to another): These activities often become invisible in architecture models. They come into existence when projects don’t want to fund the development of a complex interface to transfer relative small amounts of data. But humans don’t remove dependencies.

It is not that the concept of encapsulation doesn’t work; in fact it works very well. But problems arise when we don’t realise the dependency’s effect across several architectural concerns, and therefore how we need to mitigate and manage them.

In the above example, the currency rounding strategy represents architectural knowledge shared between two applications. While the architectural elements within the old application and the extensions performed different roles and functions, the ‘extension’ architect made assumptions about the old application’s rounding strategy.

In other words, architectural dependencies are the architectural knowledge shared between two or more elements of a system (of systems), to allow at least one of the elements to be designed, reassembled, replaced, or operated.

We can model this shared (sometimes assumed) knowledge as an abstract element with two separate implementations; and refer to it as ‘the shared element’. Note how this shared element differs from an interface while also not requiring an accurate model of the other application.

Why are ‘shared elements’ hard to find?

The complexity of our technology environments creates at least three types of problems when using traditional dependency analysis methods.

The knowledge problem

Good architects define their architecture to the best of their knowledge. They submit it for review through workshops with specialists, business representatives and other architects, but they’ll face three sets of challenges in finding dependencies:

  1. The architect’s limited knowledge about the other systems is why the reviewers were needed, but the reviewers themselves have a limited knowledge of the new system. It’s an uphill battle to gain enough mutual understanding to find potential dependencies.
  2. Large project scopes prevent architects from keeping a single view, adding the risk of solution fragmentation – see point 1.
  3. Architects cannot describe entire system of systems, forcing them to focus on the “in-scope” solution (naturally). But project scope definitions are notoriously poor dependency indicators. Dependencies don’t stop at the project-defined scope – see point 1 again.

All of this compromises the review process. The architects end up not asking the right questions, and it is somewhat left to chance whether or not the relevant dependencies and associated impacts are identified.

The data problem

Dependency analysis tools typically rely on either source code or detailed models (UML, ArchiMate or similar). Sufficiently complete models are nearly impossible to achieve in a heterogeneous technology environment – hands up those of you who have one integrated, accurate model? No, I didn’t think so. This is not to say that these tools don’t add value, it is just they don’t help with analysing dependencies between the mainframe, ERP, and the new website. Additionally, they tend to focus purely on the source code structure, leaving out all of the elusive dependencies.

The modelling problem

As long as the dependency is visible within the interface specifications, then we stand a chance. If they don’t however, then what? Modelling languages like UML and ArchiMate allow us to capture dependencies (that aren’t just normal element to element relationships) through the abstract ‘relationship’ type or ‘dependency’ links between two model elements, respectively. This is useful and allows us to document dependencies explicitly . . . for what is not written down will be forgotten.

However, while these constructs can model a dependency between two architectural elements, they don’t assist us in detailing the dependency-causing part of the element. Simply modelling Element ‘X’ is dependent on Element ‘Y’ does not express what it is about ‘Y’ that ‘X’ is dependent on – we can only assume all of ‘Y’. This leads to ‘too much’ of a dependency, as ‘X’ may only be dependent on another element within ‘Y’ (say, ‘Y.A’), but this element may not be visible through the interfaces. Then there is the question of the relevant attributes of ‘Y.A’ – in other words, what is it about ‘Y.A’ that is relevant for modelling the dependency?

Hint: ‘Y.A’ is “the shared element”.

What can we do about it?

The biggest challenge is to avoid the need to “boil the ocean” – that is, a complete technology environment search for dependencies. We can do that by flipping the original question, “Which external elements do my new system depend upon?” around and ask instead, “Which elements of my new system are not mastered by my system?”

Did you see what just happened?

The ocean became a puddle, and more importantly, you have access to that puddle – a dramatic reduction in search scope – and the knowledge and data problems became much more manageable.

But why are we looking for master elements?

Because, if an element within your new system is using data or implementing an algorithm for which your system cannot be considered the enterprise reference (or ‘master copy’), then that element is dependent on another element in some other system – i.e. it is an implementation of the ‘shared element’. We don’t need to know the location of this element to know that we have a dependency to manage. This also solves the data problem – we just need to know about our own system.

We still need to model the dependency to be able to communicate it with the relevant stakeholders. Ideally, we want to model the shared element separate to our own architecture, as the shared element can be an abstract element, rather than the actual designed elements within our system.

We also don’t want to be forced to model the architecture of the other systems, as it adds to our workload and is not part of our delivery scope. But we do need to express our approach for managing the impacts from sharing an architectural element. So, in summary, we need a dependency model to express three parts of an architectural dependency:

  1. The dependency relationship,
  2. the shared element, and,
  3. the required dependency management strategy, i.e., how can we coordinate the information represented as the shared element.

The third part is important in order to avoid dependencies becoming problematic. In our currency-rounding example, the dependency resulted in a need to synchronise design decisions between the enterprise application and its extensions. Alternatively, we could have chosen to mitigate the impact by adding functionality to reconcile totals and conceal certain intermediate calculations. Both options are valid, and we may need to both manage, and also mitigate, dependencies.

The next section describes a modelling notation to support this approach.

Dependency Driven Modelling

The first task is to turn the abstract ‘shared element’ relationship into an entity that we can model. To illustrate, consider the following example: A ‘Person’ entity is separate to the ‘Company’ entity, but if the person is employed by the Company then we have created a relationship with its own set of attributes. We can model it using the entity, ‘Staff’.

‘Staff’ can now express two things: 1) A relationship between ‘Person’ and ‘Company’, and does so independently of how Person and Company is modelled, and 2) A set of attributes specific to the relationship such as employment id, salary, office location, role etc. The attributes are important in understanding the required dependency management. For example, a person is unlikely to agree to remain employed unless the person knows the salary, location and role.

‘Staff’ illustrates the role of the shared element of dependencies.

Dependency Relationships

According to Callos et al., there are two fundamental architectural dependency types: structural and behavioural. Structural dependencies include linking of software packages, services, or other forms of APIs. Behavioural dependencies include interface validation rules or a currency rounding strategy from the above example. We can then combine these two basic dependency types with the shared element as illustrated in the below figure.

A shared element is typically an aggregate of data and processing elements, but in its most abstract form also covers references to architecture patterns, strategies, or other dependencies. The latter is what enables the modelling of transitive dependencies.

The element boundary represents the chosen scope per architectural view or style. The element views help us determine the required management (i.e., the required shared element coordination). For example, a functional view requires the coordination of functional design information, while a component-connector view requires management of the communication configuration (e.g., web service resources). For further information on architectural views and styles, Clements et al.’s book “Documenting Software Architectures” is a good starting point.

A shared element may or may not be visible. Element ‘D1’ can either ‘see’ D2’s definition of the shared element (‘external’), or not (‘internal’). We are assuming that D1 is within the controllable design scope, whereas D2 is outside. For example, a data query API is an external (structural) dependency; while the associated data validation rules represent internal (behavioural) dependencies. The combination of structural/behavioural dependency types, with external/internal shared elements, forms the dependency taxonomy as illustrated in the below figure.

And to complete the dependency model, we’ll use the abstract dependency type, ‘dependsOn’, as the abstract super class for the four dependency types.

The Dependency Taxonomy

In summary, there are the following five dependency relationship types (SE = Shared Element):

  1. ‘dependsOn’: D1 is dependent on D2 either due to an unknown dependency or multiple types of dependencies; D1 dependsOn(SE) D2.
  2. ‘communicatesWith’: D1 is structurally dependent on D2 due to the visible, external element; D1 communicatesWith(SE) D2.
  3. ‘refersTo’: D1 is structurally dependent on D2 due to the hidden, internal element; D1 refersTo(SE) D2.
  4. ‘confluenceWith’: D1 is behaviourally dependent on D2 due to the visible, external element; D1 confluenceWith(SE) D2.
  5. ‘influencedBy’: D1 is behaviourally dependent on D2 due to the hidden, internal element; D1 influencedBy(SE) D2.

Modelling Dependencies

To illustrate the dependency model in action, we assume a simple architecture consisting of a Web App Server (D1) communicating with the Inventory system via an ESB (as illustrated below). We assume that the web app implements product catalogue and shopping cart functionality, which require the entity “Product”.

The company’s Enterprise Architecture nominated the Inventory (D2) system as the master for “Product” entities. The web app is therefore dependent on the Inventory system’s definition of “Product” – that is, “Product” becomes a shared element with three implementations; one per system.

Using the previous taxonomy, we define the following dependency:

Web App Server (WAS) dependsOn(1, 2, 3) Inventory, where:

  1. WAS communicatesWith(Product) ESB
  2. ESB communicatesWith(Product) Inventory
  3. Web App Server refersTo(Product) Inventory

The WAS architect would, as a minimum, model the architecture component-connector view, in order to express the system interfaces and connectors. The architect also knows the WAS business requirements. Based on those, she identifies the following Product attributes:

  • Product identifier and category
  • Product description
  • Product stock level

The architect interviews the inventory system team and confirms that the Inventory system does manage data covering all the identified attributes. She also discovers that the product description and category data is inappropriate for the new web app, but she cannot ignore the data, as it will be required as part of introducing new products. This will form a key part of the web app’s content management process.

The product identifier and stock level are the only two parameters to be exposed via the ESB. The web app will need keep the product identifiers synchronised with the Inventory system. Given the content management requirements, the architect chooses to rely on the content management to facilitate the product identification synchronisation rather than implementing search functionality.

Note how we have, thus far, said very little about the three systems, yet we have already derived significant information about the inter-dependencies in a concise, system independent model.

The architect can now refine the dependency as follows:

Web App Server (WAS) dependsOn(1, 2, 3) Inventory, where:

  1. WAS communicatesWith(Product.ID, Product.StockLevel) ESB
  2. ESB communicatesWith(Product.ID, Product.StockLevel) Inventory
  3. Web App Server refersTo(Product.ID, Product.Category, Product.Description, Product.StockLevel) Inventory

The key trick here is that we have turned the dependency model into something that we can continuously refine using normal architectural design techniques. And importantly, cease the refinement process per dependency when we have sufficient detail.

Dependency Evaluation

Our WAS architect has identified the first two parts of our architectural dependencies; the dependency relationships and shared elements. Now she needs to make decisions about the required dependency management. The below table summarises the architectural impacts from the identified dependencies.

Interestingly, it also provides a valuable input to project managers in their efforts to identify project tasks, required resources, and their allocation per system; each dependency requires team collaboration to implement the proposed coordination.

Dependency

Shared Element

Coordination

Impact

WAS communicatesWith ESB

Product.ID, Product.StockLevel

· Interface specification agreement to mediate the interactions between WAS and Inventory.

· Service configuration management

· ESB to mediate the location and technology specific attributes of the interface

There is a direct impact if the interface specification or configuration changes.

However, the ESB should hide the actual service URI and implementation technology (encapsulation)

ESB communicatesWith Inventory

Product.ID, Product.StockLevel

As above

As above

WAS refersTo Inventory

Product.ID

WAS must synchronise product ids with Inventory. This will be managed via the content management process.

The web app will display an error to the customer, if the identifier is incorrect.

 

Product.StockLevel

Stock levels are returned in response to a web service query performed by WAS. I.e., no explicit coordination required.

Not applicable, assuming the communicatesWith dependencies are managed.

 

Product.Category, Product.Description

The product category and descriptions are synchronised via the content management process.

Failure of the content management process will cause incorrect products and descriptions within the web app.

The below table summarises the suggested set of evaluation parameters for a dependency. The set should be varied depending on the architecture, the project and the general context. But the aim of the evaluation remains the same:

  1. What are the relevant shared-element attributes?
  2. For each shared element attribute, how do we propose to mitigate and manage the dependency (shared element coordination)?
  3. What is the impact if the coordination fails?

Scenario

Identifier

Short description of the architectural evaluation scenario

Dependency

Identifier

A unique dependency identifier

 

Taxonomy Type

As per taxonomy

 

Boundary

The (D1) element scope definition, where the D1 element is within the design scope while D2 is outside except only as a dependency

Coordination

Shared Element

Description of the shared element and model if available

 

Properties

The list of relevant shared element properties

 

Management

The management per shared element attribute (share element coordination)

Impact

Potential impact if failed coordination

The perceived impact to the dependent element, if the shared element properties are not coordinated correctly

Final notes

I hope that I have given you some ideas to improve dependency identification, evaluation, and general change impact assessments. Too often, projects run into trouble because of missed and misunderstood technology dependencies, but until now, we have relied almost exclusively on experienced people’s opinions when performing dependency analysis within highly heterogeneous technology environments. While these opinions are valuable, they are difficult to reproduce, communicate, and increasingly unreliable as the size of the system grows.

Dependency Driven Modelling offers an opportunity to make dependency analysis systematic, repeatable, and therefore more reliable. The critical part isn’t whether we choose one or another architecture option; it is the explicit capture of how the architecture manages its dependency! This is critical both in terms of general quality of the architecture, as well as the quality of the overall project planning.

And the best part about DDM – it only builds on what you already know. You might just survive that review meeting.

References & Further Reading

  1. Callo Arias, T. B., van der Spek, P., & Avgeriou, P. (2011). A practice-driven systematic review of dependency analysis solutions. Empirical Software Engineering, 16(5), 544-586.
  2. Clement, P.; Bachman, F.; Bass, L.; Garlan, D.; Ivers, J.;Little, R.; Nord, R. & Stafford, J (2010). Documenting Software Architectures: Views and Beyond. ISDN: 0321552687, Addison-Wesley Professional
  3. OMG (2009). Unified Modelling Language v2.
  4. Parnas, D. L. (1972). On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15(12), 1053-1058.
  5. The Open Group (TOG) (2009). ArchiMate 1.0 Specification.
  6. Tyree, J. & Akerman, A. (2005). Architecture decisions: Demystifying architecture. IEEE Software, 22(2), 19-27.

About the Author

Dr John Brøndum is an Enterprise Architect with over 15 years of project experience. In 2010, John founded his consulting company, Brøndum Consulting Pty Ltd, to help clients design and deliver technology solutions that drive real value and impact for their businesses. John recently completed a Ph.D. in Architectural Dependency Modelling at University of NSW with support from NICTA. For more information, please visit this website

Rate this Article

Adoption
Style

BT