In this stage of the evolution of the integration platform we had some new requirements to update Dynamics CRM with near real-time changes in one of the line of business applications on premise. In some cases there could be quite a lot of changes and in others there might be hardly any.
We were thinking that a good way to handle this kind of requirement would be to poll the database of the source application using the WCF-SQL adapter from BizTalk on a regular basis and pull back a recordset of these changes. With the details of the change we would debatch this dataset and then push each message out to the Azure Service Bus queues. This would allow others to subscribe to these messages.
We had a couple of choices with the 2nd part of the process where the messages were taken from the queue and processed into CRM. If the process had been quite complex with multiple calls to CRM or orchestration across a number of applications then we may have chosen to use BizTalk to process the messages from the queue and to put them into CRM, but in this case the message processing was very simple. Instead we would use the .net messaging framework for Azure Service Bus which is hosted as a worker role in Azure which would pull the messages and make a call to the Proxy Web Service we have in front of Dynamics CRM. This would load the message into CRM.
This would give us a light weight way to process the messages with just enough functionality to do what we need to. There wasnt really a need to put additional load through BizTalk when it wasnt really adding any value. What we did do with BizTalk though was to use BizTalk to poll the dead letter queue so that we can use the management interface for reviewing errors and to create automatic processes to recover from some error scenarios.
The biggest challenge with this architecture is how we would reach down to the SQL database on premise from BizTalk in the cloud. We had been watching the Azure BizTalk Services Hybrid Connections feature to see if this would become available from an Azure VM which would give us the option to use this to bridge a connection from the BizTalk WCF SQL Adapter to the on premise database but unfortunately this wasnt available in our timescales. the Azure BizTalk Adapter Service also provides support for SQL connectivity bridged through the cloud but we decided that we hadnt really heard that much about customer success stories using this feature and that it seemed something people dont talk about much so we were concerned about if it would be something Microsoft might not be that committed to.
Point to Note:
We did not seek any official statement from Microsoft on this around the BizTalk Adapter Service we simply looked at the general lack of Microsoft or community information and interest in the BizTalk Adapter Service and decided there were other options which were safer bets to invest our time in
The other choice was to look at the Azure networking options to be able to achieve this. This introduced a good point to invest in the Azure Site-to-Site VPN technology which we could setup to join our cloud integration platforms network with the on premise network. After a lot of shopping around to find the best service for us, such as reading reviews like the ProtonVPN Review, we came to the joint decision. Once this was in place it opens up more opportunities around how we connect the cloud and on premise. This means that BizTalk is now able to connect to the SQL Server as if it was on premise.
The main reason we hadn’t invested in the VPN technology, similar to what is offered by VPNCompass, initially was that we would have limited use for it and the infrastructure team are new to Azure and we could introduce this just in time when we need it and at this later point everyone is going to be much more comfortable with Azure. Remember what is right for one customer might not be for another. Another company might have chosen to use VPN at the start of their journey. If you’re interested in trying a VPN out, check out vpn free trial. A free trial is always a good way to test out whether it is right for you or not, you might want to try out different VPN’s in different countries to test their connections, if you fancy checking out Canada’s VPN then you can take a look at the best canadian vpn – Canada has great TV programmes after all!
If we take a look a the integration platform now, I have highlighted in red the items which are used within this solution:
In the picture you can see that we have made the following technology changes:
- We have introduced the Site-To-Site VPN to connect the cloud and on premise networks
- We have reused BizTalk Server for another project and integration scenario and its pulling the data and debatching it
- Azure Service Bus is being reused again to help us have queued messaging to control load and to provide a pub/sub pattern
- We are reusing the custom .net message processing framework in the Azure worker role to process messages into CRM via the proxy service
- We are reusing the CRM WCF Proxy Service again to simplify our integration with CRM
In the solution for this integration problem you can see lots of reuse again which is validating our investment in the integration platform. We added 1 new feature and we decided to add it just in time when we needed it.
