Evolution in Data Integration From EII to Big Data
Approaches to integrating data are changing with emergence of cloud computing.
The content has been bookmarked!
There was an error bookmarking this content! Please retry.
![]()
Posted by Lee Ackerman on Nov 21, 2011
We’ve all seen the statistics[1], and likely experienced first-hand, the reality of project failure. The majority of software projects continue to be classified as failures. In thinking about this situation we can see that there are a few different ways in which a project can fail (clearly this is not an exhaustive list!):
In today’s hyper-competitive world, later may be too late to adopt Agile development and this Roadmap for Success will help you get started. Download "Agile Development: A Manager's Roadmap for Success" now!
Technical:
Functional
If we try to distill this to a high level view, we can see that we have difficulty in understanding the problem and moving to a solution.

However, such a view is too simplistic. We shouldn’t think that the movement from problem to solution is a one-way flow of information - and we definitely shouldn’t see it happening as a one-time, waterfall progression. We’ve seen that before, and it doesn’t go well.
So let’s update our view a bit recognize that there is a flow of information between problem and solution domains and that this is an iterative effort.

Our view of the situation is getting better, but its still too simplistic. Let’s go another layer deeper. We also need to recognize that development is a team sport. We’re going to have multiple stakeholders and multiple people responsible for design, development, deployment and operations.

We’ve taken another step forward in getting a more realistic view. However, there’s one more step that’s worth taking. Although we strive to have our team co-located, the reality is that we work in a distributed, networked world.

So to summarize this progression, project success is limited by our ability to:
1. Understand
2. Communicate
And adding to the difficultly is the recognition that we need to deal with proximity and temporal challenges (i.e. locations and time zones).
There are many ways in which we can try to address these challenges - and improve our odds for project success. But where to start? Some basic Agile ideals (manifesto, principles, and some common sense) give us a good starting point.

Rather than throwing money at a bunch of tools (which I’m sure are very impressive!) and trying to adopt a prescriptive and all-encompassing process, let’s take a different approach. Let’s put more value on people, communication, interactions, responding to change and delivering software. How can we best support our people in this approach?
A key technique that is often overlooked, undervalued or misunderstood is the use of modeling - especially as we move to an agile approach. In fighting back against heavy-weight processes and tool-centered development - modeling was guilty by association. So let’s take a few minutes to set the record straight...
A good place to start is to agree on a definition for modeling. Simply put, modeling is the simplification of reality. That’s it, that’s all. It does not imply the use of a specific notation, tool or process. We’re just trying to look at something that is complex, and make some portion of it easier to understand. As they say, sometimes you can’t see the forest for the trees. Unnecessary details make it more difficult to understand the situation. Better to hide those unnecessary details and focus on the important aspects of the situation.
If we can agree on this definition of modeling, we can take things a step further and consider the idea of Agile Modeling. With Agile Modeling we follow an agile approach to using models to help us to understand and communicate. Apologies for the circular definition - but this gives us a good starting point to pose the question “How do we take an agile approach when using models?”
Much like Agile development in general we use a set of values, principles and practices to guide us in being as agile as possible. Some of the highlights of the Agile Modeling approach include:

Modeling supports us in communicating and understanding when we create software solutions. As communication and understanding are two of the most critical aspects of delivering software solutions - modeling is a valuable tool that should not be overlooked.
Break free from outdated misperceptions and incorporate modeling into your agile efforts. Agile Modeling adheres to and aligns with Agile values and principles and should be one of the practices within your Agile toolkit. And by adding Agile Modeling to your toolkit - you’ll improve the odds that your project will be a success!
Join us for Part 2 in this series where we take a closer look at the values, principles and practices of Agile Modeling.
AgileModeling.com: The Agile Modeling home page, an excellent and detailed resource created and maintained by Scott Ambler.
Essentials of Agile Modeling - Workshop focused on providing foundational skills for Agile Modeling.
Disciplined Agile Delivery - community website related to supporting a disciplined approach to agile delivery.
Lee Ackerman is the VP Products & CTO at The Emphasys Group. Over the years, Lee has investigated and evaluated new ideas, designed and developed solutions and helped others to do the same. Today he focuses on helping organizations improve how they deliver software through automation, reuse and agile best practices.
[1] This news - provides an example of a recent look at IT project risk and some startling figures on failures.
This article is slowly getting down from the mainpage, without feedback, despite being well-written and well-illustrated.
In my opinion, it does because it's about modeling.
Agile takes modeling as bullshit.
Modeling is no code (or, in case modeling is code, it's called programming)
Also, modeling was always taken by the (now prevalent) XP community as "it does work only on paper anyway, why to do it".
There's no trust in the development community that any model other than code would work, and hence, there's no interest in doing it. It produces nothing to be respected, therefore, it's regarded with a "garbage-in-garbage-out" respect, and is one of the first things to go down when we're speaking about waste management.
Yet modeling could be more than that: modeling could be the way to reach from problems to solutions.
It takes care however to only write down what surely works, it takes care to look at problems from complete viewpoints (that is, from the given viewpoint, the model is complete if there cannot be any side effect (not represented property) which would change the model of the system)
But it takes care. It takes wit and wisdom.
A few people do dare to model non-code (like, we expect text input fields to work), but for most people, code is the holy gray, the one which specifies explicitly every single facet of the system, becoming the system itself.
I guess our profession isn't enough for modeling yet.
I think modeling is a good thing. Modeling different aspects in a project and development environment creates an explicit shared view on what is and what should be. I think the fact that we do not use models as software developers breaks down our communication as all our models are thus implicit, and this creates assumptions being used as requirements while coding. I have actual experience where this approach could have assisted a team in many aspects ranging from training a new employee to reducing the amount of defects created by using assumptions as requirements (Sad but true).
But you must always take into consideration the audience, level of abstraction and the fact that anything can be a model not just diagrams.
Approaches to integrating data are changing with emergence of cloud computing.
Michele Ide-Smith presents the lessons learned in the process of introducing UX principles and techniques into a large organization through a series of small steps.
Dave Farley and Martin Thompson discuss solutions for doing low-latency high throughput transactions based on the Disruptor concurrency pattern.
Rajneesh Namta shares his thoughts, experiences, and some of the critical lessons learned while implementing software test automation on a recent Agile project.
Dale Schumacher presents several patterns of actor interaction that can be used in collaborative programs written in any language.
Rúnar Bjarnason discusses Scalaz, a Scala library of pure data structures, type classes, highly generalized functions, and concurrency abstractions to perform functional programming in Scala.
One of the main challenges when designing software architecture is considering quality attributes. Not only their design turns out to be difficult, but also the specification of these attributes.
Michael Feathers analyzes real code bases concluding that code is not nearly as beautiful as designers aspire to, discussing the everyday decisions that alter the code bit by bit.
2 comments
Watch Thread Reply