While at the Integrate 2014 summit Microsoft introduced the new Microservices architecture vision and I felt that based on the great stuff we had seen it was worth throwing some ideas around in my Hybrid Connectivity options session around what an example integration architecture might look like if it included a BizTalk Server solution combined with Azure BizTalk Microservices.
Obviously we dont know enough about the lower level detail to make too clear a prediction of what it may look like but I felt sharing some ideas would be interesting to the audience and it brought up a subject which hadnt been discussed too much.
If we imagine an existing scenario implemented in BizTalk. I took one from my past where BizTalk Server was implemented on premise and would get files containing batches of insurance claims from a partners SFTP server. BizTalk would then debatch the file and for each claim would implement some logic and then load the claim into the line of business application making a few calls to the applications API. You can imagine the scenario like the below diagram.
While this solution was very successful there were some challenges or limitations to it. These were:
- The solution contains some logic to manage/process a claim in BizTalk which did not exist in the LOB application and one of the keys to success for the organisation was around how successfully they were able to process claims. Every claim that ended up needing manual processing would have a higher cost to process it and significant improvements in processing claims could have big cost savings. The problem here was that the business would occasionally like to change the processing logic for the claim to do new things. In BizTalk we are able to do this but the overhead of deployment for BizTalk and regression testing involved meant this was not a trivial task.
- The business also had new partners and applications which wanted to submit claims but they wanted to use an API based channel rather than using a B2B batch based channel.
I used this example scenario to think about how I might tackle this in a cloud based Microservices world. One of the key things here was that I didnt think just doing a like for like migration to Azure was a good approach. I know a number of people at Integrate 2014 brought up the question of migration but my personal belief is that trying to do a straight migration is a short sighted approach and means you will be unlikely to leverage the best of what the Azure platform has to offer. Moving to a new integration product architecture should allow you to rethink the way you would solve the problem to get the best of what the expanded toolkit has to offer. If you arent going to do this then you will not get the best return on your investment in Azure.
With the above said I felt like an interesting way to solve this problem in the new world would be to separate the solution out so that some of the claim processing logic which had slowly grown from inside of the BizTalk application so that it was in its own self contained Microservice. I would still use BizTalk Server but allow it to focus on the things that it does best.
If we walk through the solution, step 1 would look like the below diagram.
In this step BizTalk Server is polling the SFTP server and collecting the batch file. BizTalk will then debatch the batch file and for each claim it will make a call out to the Claims Processing Microservice. The Claims Processing Microservice would then implement logic to validate and approve a claim. Perhaps it would include some custom code and use the BizTalk Rules Service. At the end of processing a claim it would pop the claim onto a queue on the Azure Service Bus.
The key to this part of the solution is that I have encapsulated what changes inside the Microservice. We know from the early things we learn in programming that this is a good thing to do and in the Microservice we have this logic that the business likes to change a lot. The Microservice should be a much easier unit for us to change and deploy new versions of. It should also allow us to implement versioned approaches of the Microservice and perhaps separate out processing for different claim types. There would be lots of options.
BizTalk is great at the reliable SFTP retrieval and its transaction support and ability to implement debatching and flat file processing is something that is very mature in BizTalk Server but isnt something that seems to be at the top of the BizTalk Microservices priority list so we can probably expect this to be outside of the first wave of services (my guess).
So far we have used each product for what it does best.
In the next step we now want to expose the claims processing to new channels. To illustrate this I am using the example of a mobile application or a partner submitting claims via an API. The below example illustrates this.
In this example the app or partner submits the claim to the Claims Processing Microservice via its exposed API. The API is probably hidden and managed behind Azure API Management too. The claim is then processed and validated and approved and put on the Azure Service Bus Queue.
In this case we would be able to use the auto-scale features of Azure so if we needed to scale the Microservice to handle busy periods we can but we still need to worry about the back end on premise applications so we would use the Azure Service Bus queue to level out the load from these partners.
At this point we have some potential great re-use of the Claims Processing Microservice.
In the final step we are now using BizTalk to process the messages from Azure Service Bus and to implement the appropriate calls to load data into the LOB application. This is shown in the below diagram.
In this case we could have loaded one or more different queues from various versions of the Microservice and BizTalk would easily be able to poll all of them. BizTalk would then do what it does best by integrating with a LOB application and coordinating the right API calls to load the claim data. BizTalk has throttling features so we can manage the load effectively so that the LOB application isnt overloaded. BizTalk would also be able to make updates to other applications using orchestration if required too when it was processing the message and compensate any issues.
In this case BizTalk Server also offers a great management interface to help you deal with any issues in processing any messages.
I guess in this final step you could argue that there would also be the option of managing the workflow to update the claim into the LOB applications in the workflow services feature proposed on Azure. Obviously we are now speculating quite a bit about what the various capabilities will offer but my gut feeling is that the workflow in the cloud for a year or so is going to be a lot less mature in terms of these transnational type scenarios. I would expect implementing a complex workflow would then need to bridge down on premise multiple times and to implement compensation for any problems with limited ability to throttle might be a less reliable way to implement this part of the solution. For this reason I would expect in the short to medium term the approach described above would be the better one but longer term you might do more on Azure.
This really brings us to the key point in the Microservices architecture which we need to think about. To get the best ROI on our solutions developed in the integration space, if we think in a Microservices style and use the principles it and SOA are based upon then we should be building solutions which are more loosly coupled than our previous integration solutions. We generally try to keep applications loosly coupled within our architecture but sometimes the net result is that within the integration engine solutions sometimes become tightly coupled internally. If we can resist this then we can build solutions that are easy to change and evolve over time and will be lower risk to change. When we achieve this then it should be possible to have an architecture where we may choose to build one part of the solution in BizTalk Server today and at a later date move that part of the solution to Azure when we feel that the Azure platform is a place where it makes sense to refactor it to the cloud.
Summary
As I said at Integrate 2014, this part of my presentation is purely speculation on what I think things might be like in 18 months to 2 years time for integration developers. Remember that there is a committed road map for BizTalk Server for a long time to come so you arent really in any rush to migrate to Azure but you can do it at your pace and with the bits that are right for you.
Im keen to compare this view with what others are thinking and to see how we align to what actually happens as the new world integration platform is revealed
