Why im excited about BizTalk Microservices

Recently I was at the Integrate 2014 summit which was excellently organised by BizTalk 360. In this summit we learned about the vision and for the next wave of integration technologies and also got some demos of the bits that have been built so far. In the 2013 BizTalk Summit (the predecessor to Integrate 2014) we were told about the plans for BizTalk Services v2.0 from a vision perspective where Microsoft introduced the Workflow, Adapters and Rules offerings they would have. At the 2014 summit we were waiting to see when these things would be available but Microsoft talked about how they had taken a bit of a step back to make some architecture changes which would set us up for a much better offering longer term.

What they showed us really excited me and I would like to talk about the reasons I like what I saw in this post. Before we get into the new stuff I would initially like to discuss some things about the current situation.

Integration Now

Over the last few years integration has changed quite a lot. A few years ago it was all about ESB, EAI and B2B. Messaging was a core part of many integration patterns and we had a range of solutions which covered things like web services but still had older patterns with batch based integration being quite popular to.

While these patterns are still relevant and commonly used the last couple of years have seen an explosion of newer types of integration. API based integration has exploded and cloud and workflow automation solutions have become very common too. In addition to this we also have emerging solutions around IoT.

The technology and architecture space has changed a lot but rather than people moving from one thing to another the space has instead got wider with traditional integration being complimented with lots of newer ways of integrating, whether that be with companies like Mirantis or through in-house systems.

We now have a mix of companies we work with which fall into the following patterns:

• Ones that only do the traditional integration patterns
• Ones that only do the newer integration pattens
• Ones that do both newer and traditional integration patterns

Just to qualify for argument sake in this section I would regard “newer” integration patterns as ones that have become really popular in the last 3 years.

In addition to the newer integration patterns we also have seen a change in the last few years where companies are now strongly interested in hosting their integration components in the cloud and that is also a trend that is expected to increase for both public and private cloud scenarios and there is expected to be a reduction in traditional server hosting on premise.

Microsoft Current Product Stack

If we look at the Microsoft technology stack, it has changed a bit over the last few years too. The key changes have been the introduction of Service Bus, API Management and BizTalk Services. These newer technologies are aimed at companies doing those newer integration patterns. It was important that Microsoft began introducing technologies within their eco-system which would give customers choices in this space to begin delivering newer style integration solutions which are lighter weight and hosted in the cloud.

BizTalk Server

BizTalk Server has always been a core part of the Microsoft technology stack and has been the core server to deliver those hard on premise integration solutions. The server was well trusted and proven for delivering those difficult and complex integration solutions.

The only real criticisms of BizTalk Server were:

• The trade off of being a powerful integration server with lots of capability meant that it could be complex to use well and it was easy for unskilled people to misuse it
• People who did not know BizTalk sometimes didnt understand what it does well meaning potential customers missed opportunities to use BizTalk

BizTalk Server is a very mature product and it does most of what you would want an on premise integration server product to do. As integration has evolved though there are some challenges with BizTalk Server’s architecture which means that it will have limits in its ability to deliver the things companies will require in the future. For some reason people seem to feel that changing the architecture of BizTalk through BizTalk Services means that “BizTalk is dead” which was previously a recurring theme. If you liken this space to data and data warehousing then this area has changed phenomenally over recent years. 5 years ago hardly anyone was talking about Big Data and it was all about SQL Server Data Warehouse, OLAP cubes and SQL Reports. The reality was that when the cloud changed the IT possibilities then Big Data became a reality and companies had new requirements which allowed you to develop newer simpler solutions but also at massive scale. The tools we had are still relevant but we also needed new tools to deal with new problems and from there comes things like Stream Analytics and Hadoop, etc.

Perhaps in the integration world we needed a new name like BIG INTEGRATION to emphasis this shift. We may not have a new name but the reality is the same, we need to build simpler solutions in some cases (light weight integration) but also solutions which can deliver cloud scale. BizTalk Server still has a place but we need to compliment it with new tools to deal with new problems and evolve the eco-system.

I guess just thinking about it, the reason the industry wouldnt like the idea of “Big Integration” is that most organisations want integration to be cheap and light weight or simple and the name doesnt reflect that but we will worry about that another time and for now just focus on the paradigm shift.

BizTalk Services

From the above BizTalk Services was the first cloud hosted attempt at solving the new complex integration problems. We already had Service Bus for basic messaging but BizTalk Services was the attempt to build rich integration services on top of that. BizTalk Services currently has 4 key features:

• EDI Bridges
• EAI Bridges
• Hybrid Connections
• BizTalk Adapter Service

Being really blunt about BizTalk Services, I really like the EDI Bridges and the Hybrid Connections and felt they offered good solutions to compliment the technology stack. I spent a lot of time playing with BizTalk Services and I could just never really convince myself about EAI bridges. They just didnt have the range of features where I felt they would have wide scale adoption and would only fit niche scenarios until they had more work done on them. I also felt BizTalk Adapter Service was quite niche too.

In 2013 we were introduced to the plans for V2.0 of BizTalk Services and the plans were to include Workflow/BPM capabilities and Business Rules. While in principle this was a good thing there were always some problems that I felt existed with BizTalk Services which many of the MVP’s and Advisors fedback that we felt needed to be addressed during 2014. The main ones were:

• Architecturally the individual services in MABS were too tired together meaning I couldnt just use what I wanted, ie just use EAI without everything else
• From an execution perspective everything seemed too grouped together and we wanted isolation of execution
• We also wanted isolation of deployment and packaging. Not to have one namespace with everything in it which makes management and deployment too complicated
• The ALM piece was too immature
• We put forward the idea of rules as a service
• The tight coupling between features could limit or slow innovation in future releases

Fortunately with what was presented at Integrate 2014 we see a lot of things which have addressed and I feel like a lot of that feedback was taken on board.

What do I like about the Product Vision

The new product vision has pitched the architecture where the core Azure Application Platform will create a hosting & container platform based on the next version of Azure Websites. On top of that they will build what Bill Staples described as the “Microservices Eco-system” and within that eco-system there will be some integration microservices called BizTalk Microservices.

My interpretation of what was described was:

1. I will be able to download BizTalk Microservice templates for specific integration problems which can be deployed to the Microservices platform. These would be things like Workflow, Rules Services, Mapping Services
2. I will be able to create my own microservices which I can deploy to the platform
3. I will be able to compose microservices into an overall solution

What was described is a big paradigm shift for the typical BizTalk developer but this new architecture takes big advantage of the things Azure can offer and can allow us to build solutions made up of small discreet components which have specific jobs which are then composed together. The big benefit is that small and discreet means it should be easier to build, deploy and reliable test these components. This will improve quality. Next the scaling platform Azure has should mean we inherit the ability to scale to BIG INTEGRATION scale

The components we build in theory will be much simpler and lighter weight so some of those simpler integration patterns such as an RPC process to connect two SaaS applications to support querying data will be simpler to implement and also perform better as they wont have a message box to work about (unless they need one).

If we need to solve more complex solutions then we can compose more microservices and combine with Azure Service Bus to include pub/sub and durable messaging. Finally if we have a problem that the Services offering doesnt address or is better solved with BizTalk Server then we still have that available like we always have.

From a technology and architecture perspective there is so much to like about the vision and in my mind my integration tool kit is about to get some really cool new things and its much more about using the right integration tool for the job rather than bending BizTalk Server to solve the problem because its the only tool I have available.

Being bold and calling it BizTalk Microservices

My interpretation of what was described at the summit was that on Azure there will be the application platform which will include something which is effectively the Microservices platform. On top of this there will be some microservices that can be added which provide integration capabilities. These will be services developed by the product team and services developed by ISV partners. These services can then be extended or configured then deployed as your microservices. There will also be the opportunity to build your own microservices.

There was some discussion about whether Microsoft should call it Microservices or not.

I am quite bullish about this. Microservices is an architectural style and what Microsoft is proposing is a platform to build Microservices. With Microservices being the new trend this offers some serious marketing opportunities and can help Microsoft make a lot of positive noise about this platform.

My personal preference here is that the platform is called something like Azure Microservices and the out of the box integration services which are created are called BizTalk Microservices. I think it is important to keep the BizTalk brand for consistency and customer comfort but associating BizTalk with Microservices has almost overnight made BizTalk cool.

Lets face it the most active competitor to Microsoft’s integration offering has a product called XX ESB so they have called their product after an architectural style and their product doesnt even do durable messaging or publish/subscribe which are 2 of the most fundamental capabilities of an ESB (and is a bit like selling a car without an engine), yet they are able to market this very successfully so I would argue that Microsoft should be easily able to market a Microservices platform like they have described very successfully.

Competitors

I think Integrate 2014 is the first time in years where Microsoft’s competitors will have really sat up and took notice of what Microsoft is doing. Over the last few years Microsoft have made some interesting product advances in the integration space with things like Azure API Management & Azure Service Bus. While these products on their own have been pretty cool the problem has been that Microsoft have lacked an Integration Architecture focused voice who has been saying “This is our product stack for integration and this is how you combine the technologies to create a solution”.

Integrate 2014 was the first time in a while where this cross product voice has been heard and the vision pitched seems to be quite well aligned and is about bringing the products together to build solutions.

While the Microsoft community has done a good job of communicating how to use the Microsoft technologies, I think many of Microsoft’s competitors have benefited from the lack of a loud voice from Microsoft which has allowed competitors to question Microsofts commitment and capability around integration. With this realignment of resources to the new platform I expect to hear a serious amount of noise in the future and I am sure competitors will be very wary of this.

Cross Sell

In my opinion cross selling is Microsoft’s biggest advantage over their competitors and especially on the cloud. Unfortunately cross selling is something Microsoft has never seemed to be that good at. With the new platform for integration the big differentiator that Microsoft has over a lot of other competitors is that it is part of the Azure cloud platform which is a very rich eco-system of capability at all levels. Some competitors offer a cloud hosted iPaaS platform which has some integration features but thats it, or perhaps it offers a few other non integration capabilities.

In the Microsoft cloud I will be able to have my full iPaaS offering but I can also add services such as Databases, Web hosting similar to that found at https://www.ddev.com/ddev-live/, Servers, Security, Office 365/CRM, Networking, Hybrid Connectivity, Business Intelligence. If you are developing integration solutions then it is common that you will need more than just your basic integration capabilities to deliver your solution and the Microsoft cloud will give you a very rich tool box of almost everything you might need. I discussed a real world example of this in my Azure based Integration Platform article.

Cross selling is not just around integration though. Most organisations are also going to be building application solutions and Azure offers you a single platform to build your application, and then when you need integration you can add the integration services you need. Your adding more services to your existing agreement and just paying for what you use. There is no going out and finding another vendor who will hopefully be able to integrate with your solution but need to be managed in a different platform and potentially be hosted elsewhere. I think in the future organisations will begin to realise the real cost of this vendor management over head and how it slows down the delivery of your solutions when instead I can just add the new resource I need in a few clicks.

If Microsoft manage to master the cross sell opportunities that Azure provides then I feel that the richness of Azure will be the key to success for Microsoft and this will make a big difference to the Integration space around Microsoft technologies.

New to BizTalk

A few years ago when BizTalk Services first came out I hoped that Azure plus BizTalk Services would offer new opportunities for customers who have never considered BizTalk solutions to begin BizTalk based integration solutions. The problem was that the cost model of BizTalk Services was still focused around Enterprise customers and the vision was around a package of integration capability all bulked together.

BizTalk Microservices appears to be a realization of the feedback that BizTalk need to be broken out into individual services and in addition to the technical benefits one of the big wins is the break down of costs to move towards a pay as you use model. No more paying for features I am not using.

If this happens then suddenly BizTalk based integration becomes something which it is feasible for SME organisations to consider using. This also offers the opportunity that a project developing an application might need a basic business rules based capability but rather than trying to build their own or using open source a BizTalk Microservices Business Rule option should become an obvious choice. The project shouldnt be paying thousands for a big product but instead paying for what they need. Going back to my earlier point about cross selling, once an organisation begins using bits of BizTalk and sees value from it, it should be very easy for them to consider using other parts of BizTalk.

Microsoft has had in the region of 5000-10,000 BizTalk customers for years but if they get BizTalk Microservices they should be looking to get 1,000,000 customers using BizTalk Microservices. Breaking the product down into the capabilities you need at a cost based on what you use could create a completely different economy of scale for BizTalk usage.

Cautious Customers

One of the things that was slightly unexpected at Integrate 2014 was that although there was lots of positive noise about the new stuff there was also some concern from some customers about what this all means for existing BizTalk customers. The product team attempted to address some of these concerns but obviously they have to be quite sensitive about how they answer questions. In my opinion though the problem here isnt really a product problem its more of a guidance problem. The reality is:

1. If your a BizTalk Server user then there is a committed road map until 2023 which is nearly as long as the product has already existed and probably as good as you will get from any vendor
2. If your a BizTalk Server user then you need to recognise that although your integration requirements may not have changed for many other organisations integration has completely changed in recent years and in the future and there is a technology stack evolving to answer those new challenges
3. If you are the above BizTalk Server user then the industry analysis and trends say that in the future its very likely that you will need to play in this new integration game at some point and when you need to go there, there should be a great product set in place already

For most BizTalk Server customers the reality should be very positive but I can understand that some may be cautious by saying “but what innovation will happen in the product”. Again here the reality is that “what innovation do you want to happen”. We all have our list of nice to have features but in terms of key core requirements for BizTalk Server, as a product it has been pretty much feature complete for a number of years and everything people ask for are small changes to make like a little easier. In most cases the community has solved these problems anyway.

Most of the things that could be done with BizTalk Server which would add a lot of value would be better if they were done in a way where BizTalk Server and BizTalk Microservices could both leverage them together. As an example it is often brought up that BizTalk Server would benefit from investment in the BAM Portal. A better portal would be great but what would be far better would be a new BAM module that both BizTalk Server, BizTalk Services and anything else could plug into and then it could use the new wave of business intelligence technologies to build a really great solution which would add so much more value across the entire Microsoft stack than just making improvements in BizTalk Server alone.

Concerns

At this stage we only know a certain amount about the Microservices product plan. I guess the one concern I have is about how Microservices will talk to each other. At Integrate 2014 there was a little bit of discussion about how Microservices could be automatically using API Management out of the box but they didnt really go too much further into it.

In his article Martin Fowler talks about communication with Microservices via HTTP and Queue based technologies. This makes sense and I hope that Microsoft allow Microservices to be connected to via Azure Service Bus out of the box in the same way they described them as being communicated with via HTTP. The concern I have in this area though is about the virtualization of the microservice. As an example I would prefer that my communication between microservices or from an application to a microservice can happen via a virtual address (either by queueing or http) so that I am able to re-route messages and implement versioning approaches on the back of this decoupling. I think having tightly coupled addressing would result in some pain for the management of solutions.

In the vision there wasnt really any discussion of how this problem would be solved or if it is a capability that the platform may need.

Summary

In summary I think this is the most exciting time for Microsoft based integration people for years. I really look forward to more information on the new platform and the opportunity to play with the bits when the preview is available