4 Warning Signs an Integration Wall is Approaching

The Integration and API Management markets are growing,  expanding in both popularity and use.  Enterprise App integration will surpass $33b by 2020, and other markets like iPaaS and Data Integration are growing at double-digit CAGRs.  Enablers, such as  containers and serverless technologies are only accelerating the move toward increased disaggregation of applications.


All seems rosy. And it mostly is.

But with the explosive growth of APIs and endpoints, traditional centralized tools like ESBs will become unsuitable, and simple low-code snap-together tools won’t scale to address the broader scope.  We’re potentially about to hit an “integration wall” at high speed.

Consider the following four warning signs – some technical, some process – that I find are beginning to plague the integration market:

1. Waterfall Development for integration is hitting a wall. 
Although most code development has shifted to an Agile Development model, the same can’t be said for Integration tools. As the quantity and diversity of endpoints increases, and as Integration projects become more diverse and complex, use of the waterfall model is beginning to slow down integration projects.  And with a future where there will be billions of Integratable endpoints, it’s obvious that an Agile Development model for integration will need to become the norm.

2. Existing tools and programming languages aren’t optimized for Integration-at-scale.
Enterprises that currently use low-code, snap-together, centralized integration technologies (including iPaaS) will not be optimized for orchestrating, integrating, observing and governing the expansion of constantly-changing endpoints. Nor are traditional centralized approaches (think: EDI and older ESBs) prepared to handle increasing endpoint scale or diversity.  Many of these existing tools are well-adapted for Line-of-Business or Citizen Integrators of relatively small-scale implementations but are far from well adapted for more complex integration-at-scale projects.

3. Current programming languages are not optimized for Integration. 
With languages like Java/Spring or JavaScript/Node, developers can engineer flow, but must take responsibility for solving the hard problems of integration. With these languages, developers have to write their own integration logic or use bolt-on frameworks.  Clearly a new programming paradigm will be needed long term.

4. The Exploding Endpoint Problem is very real.
As I referenced above, IT is ill-prepared to address the oncoming wave of service disaggregation, the diverse types of APIs, differing sources of service endpoints, challenges from Big Data, and multiple approaches to serverless IT. The industry is about to hit a scale and diversity wall. To wit,
 • 917 apps in use per enterprise (Netscope, 2016) 
 • 893-1206 average cloud services used per employee (Kleiner Perkins, April 2017)
 • 19,000 APIs as-of January 2018 (Programmable Web, 2018) 

And if you don’t believe those numbers, Matt Eastwood of IDC recently pointed out that the number of containerized services has expanding well beyond where VMs ever were.  Yep, billions of programmable endpoints aren’t kid’s stuff.

Where does this leave us?

A new approach to addressing the future of integrating thousands – or millions – of endpoints could lie in a new programming language, Ballerina.

Ballerina is a simple programming language whose syntax and runtime have been optimized for the hard problems of integration.  Its focus is integration – bringing concepts, ideas and tools of distributed system integration into the language.  Based on the concepts of interactions within sequence diagrams, Ballerina has built-in support for common integration patterns and connectors, including distributed transactions, compensation and circuit breakers. And it supports JSON and XML, making it simple and effective to build robust integration across distributed network endpoints.

So, watch this space for future developments. And in the meantime, beware of the approaching wall.