In part 4 of the evolution of the Azure-based integration platform we have some new requirements to handle event publishing from Dynamics CRM Online and to process these events. Without going too much into the detail of the CRM event publishing piece the simple overview of this part is that when changes are made in CRM jobs are configured which will publish a message to an Azure Service Bus queue. From here we have a lot of possibilities of things we can do with this.
In our architecture BizTalk Server hosted in Azure is our best piece to process these event messages. In most integration scenarios like this, the common requirement is to get the event then go back to the source system to enrich the message with additional data and then to send the information to other systems. In our case, the event that is published will be enriched and then we will have a workflow that will make multiple updates across multiple systems. It would look something like the below:
- Receive message from service bus
- Go back to CRM to get some additional data
- Make a call down to on-premise application A to add the record
- Go to on-premise application B to add the record and look up some additional info
- Go back and make an update to the record in application A
- Go back and make an update to CRM
The fact that we have this complex workflow with a large number of possible places where we will need error handling, potential compensation and the ability to manage the process makes BizTalk the best choice to implement the complex bit of this process.
In the architecture, we choose to use the Service Facade or Service Proxy approach to creating some web services which would encapsulate some of the complexity of integrating with our applications. On-premise we chose to implement a WCF Service in front of application A and application B because they were fairly old apps that were not easy to integrate with. This simple WCF pattern would significantly simplify the implementation in BizTalk and also support a common approach for any bits which did not go through BizTalk. We would host the on-premise WCF services in IIS.
We also chose to implement a service proxy in front of Dynamics CRM Online too. The challenge with CRM is that the SDK is very powerful but also very generic to support the dynamic data model within CRM. This means that calling these services can be complicated. We chose to implement a WCF service to encapsulate this complexity in one place so the majority of calls going to CRM would go through our service proxy which would make it easier to integrate with. For this service, we chose to host it in the cloud as an Azure Website which would give us a very cheap but scalable hosting capability that was easy to use but something which we could also extend to use for other services that would follow a similar pattern.
At this point, we know how we are going to get the events from CRM and how we are going to integrate with each app but we have not addressed the hybrid integration challenge. This is about how we will get from BizTalk hosted in Azure IaaS to those WCF Services on-premise. I have spoken at a number of events recently about the number of options that are available to do this kind of thing. In the case of this platform, we chose to use the approach of Azure Service Bus Relay with the WCF Routing Service on-premise which we can easily configure to support forwarding messages to any of the WCF Services we need. This would allow us to get up and running very quickly with low maintenance but at the same time we would use development approaches which would ensure we weren’t tightly coupled to this approach so we could change it later to VPN or something similar if we wanted to simply by using our ASUS router login to add the VPN details.
If we now take a look at what the integration platform looks like in the diagram below you can see we have added a few new capabilities and extended our use of others. I’ve marked with a red box the key things we have used to implement this solution.
To summarise the technology changes in this phase of the platform evolution we have done the following things:
- Introduced the use of Service Bus queued messaging
- Extended our use of Azure Websites to host a WCF based Service Proxy component for CRM
- Extended our use of BizTalk to implement some complex process automation
- Introduced the use of some WCF Service Proxy components on-premise to make it easier to work with our on-premise applications
- Introduce the use of Azure Service Bus Relay to reach down on-premise to access those on-premise WCF services
The key thing about our platform in implementing this solution on top of what we already have is that we have not needed to make any major capital investment. We have simply added some new resources to our Azure subscription and added a few lightweight components on-premise. There is nothing here that is going to take a long time to get up and running and the platform will let us quickly get to a stage where we are focusing on functional requirements and not spending ages on the lower level plumbing just to join the dots between applications.
This is really starting to look like an agile integration platform!
