You’re Doing it Wrong: it’s Not about Data and Applications – It’s about Processes

For many organizations embracing digital transformation, business process automation represents low hanging fruit, with the potential to quickly deliver substantial cost and efficiency benefits.  After all, a process represents the way work gets done.  All too often, though, solution builders try to tackle process problems in the worst possible way.

A while back, colleagues of mine expressed some interest in using my company’s digital business process platform to build an application to manage sponsors for a fundraiser they were thinking of doing. As they showed me a preliminary effort over a remote web chat, I was struck by two things:

• A really large number of fields.
• A really small number of steps in their workflow.

I pointed this out, asking why they had extra fields to record who approved what and when, many extra multi-line text fields, and a lot of fields that were marked “Status.” They replied that they needed to know whether steps had been completed, who did them, what happened at each step, and so on.

I was both bemused and amused, responding “You realize that each process instance has a built-in audit trail, right? That the workflow’s current state already gives you the status you want to track? That the built-in comments maintain a threaded, date-stamped record of everyone’s free-for comments?”

That wasn’t all. There were extra columns for data that seemed out of scope. When I asked “do you really need that field? It doesn’t seem to have anything to do with sponsoring,” they replied that “I suppose it doesn’t, but I usually collect it for an organization in other applications, and sponsors are organizations.”

Three things had become obvious:

1. They were working too hard, making the application responsible for things the platform already provided.
2. They were asking their users to work harder than they needed to as a result.
3. They probably had data requirements they hadn’t yet thought of, but since they modeled their data before thinking of the process that would be using it, we wouldn’t find that out for a while.

My colleagues weren’t – and aren’t – dumb. Quite the opposite. But they’ve been building applications a certain way for so long that other ways felt unnatural.

It’s the Law of the Instrument in action, akin to a child with a hammer assuming everything is a nail.

How Business Process Solutions are Often Built

The following approach is too often the norm:

1. Create a data model.
2. Build some forms to edit that data.
3. Create some triggered automation scripts/flows/zaps/recipes to react to data changes.

In other words, a business process solution is treated like any other application. And if we look across the past several decades of business applications, most of them revolved around managing data. What was to be done with the data was external to the system, usually in the collective heads of management.

Why Starting with Data is a Problem

The most glaring issue is that it puts the cart before the horse. Think about it…

1. How do you know what data you need before you model the process itself?
2. People don’t edit data, they perform tasks. The form should fit what the user is being asked to do.
3. How do you know the activity will indeed be triggered and in the way you expect? For that matter, how do you know it’s the right activity?

This bias comes honestly, for what it’s worth; it’s how many if not most of us were taught. And many business problems are, indeed, data management problems at their core. But many aren’t; they’re process problems, and a data-centric approach does not address them well.

How it Ought to Happen

We’ve framed applications into multiple tiers (usually at least three) for a long time now. There’s a presentation tier responsible for the user interface, a data tier or accessing and storing data, and a logic tier that sits between them, determining what to show users and which data should be read/stored. It helps with reuse. It helps with supporting multiple UI options that leverage the same logic and data. It helps by allowing projects to be taken on by Ui specialists, database administrators, etc., instead of needing full-stack developers for every project.

The problem is that a lot of people tend to start with the data tier, and build out from there. What if, instead, we started with the logic tier? In fact, what if we started with the big picture that governs all three tiers before getting to the logic tier? It would look like this:

1. Determine what our desired outcome is.
2. Figure out the steps needed to achieve that outcome.
3. Assess who needs to be involved at each step and how.
4. See what data will be used at each step and by whom. Sometimes they’ll need to examine it. Other times they’ll need to provide it.
5. Create forms and reports that present tasks and results to the people who need them during the steps that they’re involved.

It’s almost backwards from the norm, but it’s how successful process solutions are built. To be fair, a person who starts with a data model might have done the first three steps in their head without having bothered to write anything down but table schemas and form layouts. It’s understandable, but it’s dangerous.

Why “Data First” Thinking is a Bad Fit

For one thing, a data-first approach doesn’t examine (let alone question) the process itself; it moves on immediately to automating the activity. What if the process isn’t ideal, or even effective?

We often model processes to document them, to validate them with stakeholders, to teach them to others – and most of all, to improve them. In far too many companies, what they do and why they do it is implicit, not communicated well, and invites plenty of competing points of view as to what it really is.

You need to tackle the process first before you attempt to automate any of its tasks. Not doing so would be like digging holes with a crane instead of a shovel, but without thinking about whether the holes are being dug in the right places (or should be dug at all).

It’s not enough to think about saving time and money. Automating a process (not just its activity) documents it, makes it teachable and scalable, and goes a long way to reducing or eliminating mistakes (high profile errors can be a major catalyst for process automation). It also makes a process easily audited and monitored And it’s a lot easier to figure out how to improve a process you can see..

And improvement is a must; if there’s one thing to expect when it comes to process automation, it’s change. Perfection from the start is unlikely. Exceptions will have been ignored. Assumptions will have been omitted. Steps that could be automated will still be manual. This isn’t bad, by the way, continuous improvement allows for early imperfection. Even when everything is perfect today, requirements will change over time. The motto for process automation should always be “release, review, revise, repeat.”

Power Users Often Make Similar Mistakes

Power users, or citizen developers if you prefer, suffer from the I-have-a-hammer-so-everything-is-just-another-nail problem, too. In their case, the first thing they think about is usually the form they want to see and the data they might need to collect.

To many if not most power users, the form is the data (as opposed to a window into data living somewhere else). Despite this, though, they rarely take a moment at the beginning to think about why the form exists in the first place and what we’ll be doing with it. What should happen to that form is rarely considered until much later in the solution development cycle.

While We’re At It, It’s Process Automation First, Activity Automation Second

Process logic considers process decisions like where requests are routed, what information should be fetched to make a decision, what happens if a request is approved/rejected, etc. For example, a process to handle requests for vacation time would read something like “send a request to the employee’s manager to approve or reject. Make sure the manager knows whether the employee has saved up enough vacation days to cover the time being requested. If the manager approves it, deduct the time from the user’s saved vacation days and let everyone know they’ll be gone during that time.

That’s all about what should be done. There’s also the matter of how to do it. Activity automation, as opposed to process automation. Things like:

• Checking leave balances by calling a specific API using a specific service account to reach the human resources line of business application where balances are kept.
• Calculating the number of hours in the request, omitting weekends and holidays.
• Assessing available balances against hours requested.
• Accessing user and team calendars in Microsoft 365 using the Microsoft Graph API, again using specific details.
• Formatting requests to add or update calendar entries and making the correct REST calls with the right credentials to perform the updates.
• Sending SMS requests to managers using a Trello SMS gateway.

Both process logic and activity logic are important. But data-first thinking, the bias many of us bring to every business problem, all but begs us to tackle the activity logic first. And that would be a mistake.

If you start with the activities, you can’t test the application until it’s nearly done. Users must wait until everything is finished. With luck, conditions haven’t changed much during this time, but if they did, you might have just wasted work on activities that are no longer the right ones.

If you start with the process, though, you have something to have users test very quickly. The process can be automated even if the steps remain manual at first. Tasks get assigned and monitored. Notifications take place. Steps are no longer forgotten, and mistakes are fewer in number.

And now, while users are trying out the overall process logic, you can be automating activities, folding them into the overall solution as they become ready. Users and stakeholders see a few immediate results and a steady stream of improvements instead of waiting a long time for something they’re not sure will be a good fit.      

Instead, start with a skeleton of the process. An outline. A checklist, even. Even if the steps remain manual, the process can become managed and automated often much more quickly. Long before you can automate every step, you’ll be able to track the status of work in progress, ensure that steps aren’t missed, reduce mistakes, and audit just about everything after the fact.

Aircraft maintenance, in fact, the majority of the airline industry, works this way; processes are automated even when many steps remain manual. Operating rooms started having fewer “unintended outcomes” after pre-surgery checklists were introduced.

Of course, as time permits, individual steps can and often should be automated as well. But timing is everything, and if you start by thinking about how to connect to data and how to automate manual activity, you’re missing the point.

Process-First Approaches are More Change-Friendly

If you design the process first, and start testing it early with manual steps, you get information you can use to better automate the activity. Information you wouldn’t necessarily have if you started by modeling data. Early course corrections aren’t as difficult.

Especially when the work starts with data modeling and application integration, and a lot of time is invested in those steps, those components are likely to be viewed later on as constraints. Changes will be seen as expensive. And since they were done first, the whole application is seen as hard to change.

Having the process drive the solution keeps the notion in the forefront that conditions will change and the application will need to adapt to them. We should actually want it to work this way, because we get much better information from users and business stakeholders when they have something to review, to criticize, to edit. It beats abstract envisioning every time.

A Process-Driven UI Makes More Sense, Too

Users don’t think about maintaining data – they think about performing tasks. The applications we give them ought to be purposeful. To fit real user stories.

When you open a form, you have a job to do. The form should present the information you need to know and collect from you the information you need to provide. Then it should go away.

While you might just want to browse a record, there’s usually a reason in mind. If you say you just want to someone’s phone number,I get that. But I also know that if you’re doing so, we might want to log the call you’re about to make and ask you to jot down what happened so we have a record of it for the next call. In fact, this is why companies invest in Customer Relationship Management technology.

What’s more, what information is displayed, which fields are read-only, which are required, etc.? That varies by task. And by user. Forms, or any user experience, ought to be adapting to what the user needs at the moment, not on the data they’re about to touch.

Forms might fetch and post data, but forms are for tasks. Activities. They represent verbs, not nouns. Methods, not objects.

Processes serve as a good way to think about this, and a good way to construct it, too.

Data Management Solutions Still Make Sense – Sometimes

This doesn’t mean every application is fundamentally a process application. A lot of business requirements are absolutely best served by a data-first approach. Anything involving business intelligence, for example. A lot of ad-hoc customer relationship management can work this way. Some case management scenarios where no two cases resemble each other are best served by bringing the data to the participants and getting out of the way.

In situations like that, there’s no process to automate. It’s too idiosyncratic and situation-specific to even see patterns, let alone attempt to model and repeat them.

In such situations, the best we can do when building solutions is figure out how we can best present data the users might need (this is exactly what my colleagues building the fundraiser application were doing).

And if we’re good at this sort of thing, we keep in touch with them and look for any emerging repeatable processes that become good candidates for automation.

Conclusion

Form should follow function. Data and UI should follow processes. And the right tools, techniques, and expertise should be deployed at the right times to build solutions in this vein.

Organizations and teams would do well to remember that when it comes to process automation, it’s all about the process.  Get the workflow logic right, and the data activities and presentation will naturally follow.

About the Author

Mike Fitzmaurice, WEBCON's Vice President – North America, has more than 25 years of product, consulting, marketing, engineering, and IT management expertise in workflow/business process automation, citizen development, and low-code/no-code solution platforms & strategies. His decade at Microsoft included birthing technical product management and developer relations for SharePoint products and technologies.