Large-scale automation of business processes requires a new development approach.

Across sectors, business processes are undergoing the most profound transformation since companies replaced paper files with electronic records. A new suite of technologies, including robotic process automation (RPA), smart workflows, and artificial-intelligence techniques such as machine learning, natural language tools, and cognitive agents, promises to radically improve efficiency while eliminating errors and reducing operational risk. Research by our colleagues at the McKinsey Global Institute suggests that, across industries, there is already the potential to automate more than 30 percent of the tasks that make up 60 percent of today’s jobs. In finance and insurance, for example, workers spend more than half their time collecting and processing data, tasks that are eminently suitable for automation using techniques that are already available today.

Many companies have identified significant opportunities to apply automation, and the results of pilot projects and technology demonstrators have been encouraging. So far, however, most have struggled to capture the full potential these new approaches by applying them at scale across their operations.

There are multiple reasons why implementing automation is challenging. Some of the technologies involved are still relatively immature, for example. Applying them outside a carefully controlled test environment can reveal unforeseen weaknesses and limitations. And with thousands of processes involving tens of thousands of employees, organizations find it difficult to build workable road maps for large-scale automation.

The devil in the development detail

Then there’s the challenge of software development and implementation. Companies need to tailor and customize their chosen technologies so they work in the context of the wider organization. And because automation involves significant changes to existing roles and tasks, they need to coordinate technology development within a wider change-management process.

As many organizations have already discovered, established software-development methodologies do not work well in this complex environment. The first to fail has been the traditional “waterfall” approach, in which analysis, specification, design, coding, and testing are conducted sequentially. Automation projects organized this way have been plagued by delays and cost overruns, as companies discover unexpected issues or limitations late in the project-development lifecycle. That can be a particular problem when efforts are centralized at the enterprise level. After a successful proof-of-concept project, for example, one mining company used the waterfall approach to automate an important back-office process. The company was ten weeks into implementation when it discovered that its infrastructure design couldn’t be scaled up to handle the work. By the time it identified and fixed the problem, the project was already delayed by more than four months, causing costs to spiral.

Experiences like this are encouraging more companies to pursue agile development approaches in their automation projects. With its emphasis on tight-knit cross-functional teams, focused development efforts, and continual testing, agile has proved highly successful in addressing similar challenges in other areas of software development.

Yet applying agile to automation projects has brought its own challenges. That’s because process automation differs from the development of a conventional software product in a number of significant ways.

Scrum, an agile methodology of that leverages quick iterations to develop features, works by breaking a complex problem or feature down into discrete chunks or “stories.” Teams work in these chunks one at time, focusing on quality and releasing software frequently as opposed to at the end of the project. In a conventional software product, that usually means that products start by offering a limited range of features, with new ones added over time. In process automation, however, it can be difficult to break a feature down in this way. The individual components within a process are often tightly coupled: it either works end-to-end or it doesn’t work at all.

In addition, the disruptive nature of process automation, which may involve significant changes to the roles and responsibilities of hundreds of employees, can make frequent release cycles unfeasible. Sometimes the incremental value captured by a single component is not enough to justify a release.

Then there is the issue of ownership. In scrum, there is a dedicated “product owner” who acts as the representative of the end customer, working closely with development teams to answer questions, prioritize work, and give feedback on prototypes. Process automation may span multiple business functions, units, and geographies, making it difficult to find an individual with the requisite knowledge and connections. And because automation is new, the most appropriate “process owner” within the organization may have little or no experience of working on software-development projects, let alone the fast-moving, intensely iterative agile environment.