BizTalk Time Travel Considerations

Sometimes in a project you will have time sensitive requirements where at some point in the year key events will happen which makes your IT systems need to behave differently. I have come across this a number of times with BizTalk projects and wanted to discuss some of the things you need to consider if you want to test time sensitive scenarios with BizTalk.

Travelling forward in Time

Travelling forwards in time is probably the easier scenario. In this case today would be 26^th January 2016 but you know that some kind of time critical event happens in the future (let’s say on the 1^st April 2016). We want to test how our system performs on the 1^st April. In this case we will roll forwards the date of our systems to the 1^st of April and see how the system behaves.

From a BizTalk perspective this means that your BizTalk Server will belong to a domain and when the time of the domain is changed then this change can be cascaded across the domain (we will discuss this more later). Once the time travel is complete you may need to complete some steps to get your BizTalk environment into a running place.

Recommended Steps

I have tried to come up with a list of recommended steps which can be reliable followed to get your BizTalk test environment into a running state after you have moved the date forwards. I’m sure that some environments will have additional stuff in them which will make their required steps a little different, but as a general rule this should cover most scenarios.

Simple Approach

1. Move the domain time
2. Re-synchronise the time for your BizTalk and SQL Machines
3. Restart the servers

More Detailed Approach

1. Stop the SQL Agent
2. Stop your BizTalk Host Instances
3. Stop Enterprise Single Sign on
4. Stop Rules Engine Update Service
5. Stop MSDTC on all servers
6. Move the domain time
7. Re-synchronise the time for your BizTalk and SQL Machines
8. Start MSDTC on all servers
9. Start Enterprise Single Sign on
10. Start Rules Engine Update Service
11. Start BizTalk Host Instances
12. Restart IIS

Note 1:

You may consider restarting the SQL Server service but in practice we have found that this didn’t seem to make any difference for our scenario

Note 2:

We were using BizTalk Standard Edition so in a clustered environment you may require additional steps

Some things to consider

A few things to consider include:

Running/suspended instances

I think if you have any in flight instances in BizTalk they should just continue as they were after the date move. If you have used the more detailed steps then you can be pretty confident that this will be the case as you have brought down your environment in a safe manner before changing the time.

I think this will still work ok in the simple approach too but there is a little more risk because the shutdown was less controlled.

Log Files

When troubleshooting issues after your date roll you may find there are gaps in the log files and event logs and similar monitoring sources. I would recommend to record the actual time and environment time when you take the date roll actions so that you can compare them against lots when troubleshooting any issues.

Dependant Systems

One of the important aspects of your BizTalk setup in relation to time shifting is how it will interact with dependant systems. In some cases there is likely to be no affect at all. This particularly applies to systems within the domain which have also been time shifted. For example if your BizTalk port sends messages to a SQL Server which uses integrated windows security then everything should be fine as its all within the domain and has been moved time accordingly.

Where you might get some problems is if you have services which are outside of the domain. One example might be if you connect to a Dynamics CRM instance which is hosted at your partners company. The connection from BizTalk to CRM would use soap services and WS-Security. There are tokens involved in this interaction and time to live values associated with the tokens. When the client and server in this scenario have a significant clock skew then you may get some WS-Security related errors from the web service call. In this case you can make some adjustments by manipulating WCF but in general your preferred approach should be for the partner to be “in sync” with your time. If they cannot do this then you can code/configure around it in some cases but it could depend on the protocol being used.

In addition to partner services, use of cloud services can be a challenge in time travel scenarios. If you are using IaaS services then you control a lot more of the cloud service and can do date moves in some cases. If you are using PaaS or SaaS services/applications then you will be very unlikely to be able to control the date they are working on. This may be a limitation you need to just accept or perhaps design your solution with this in mind. With one customer scenario we had a very interesting approach. We were integrating with Dynamics CRM Online. When it came to the case of needing to do some time travel testing of year end processes CRM Online is a SaaS application so we are unable to move the core date/time for the application. What we did instead though was to implement a IaaS instance of Dynamics CRM in Azure and we were able to move the date of the domain accordingly. While is was not a 100% like for like solution they are close enough that it allowed us to be able to test these special time sensitive scenarios in IaaS and do the bulk of testing in SaaS. With many SaaS applications you would be unable to do this.

Travelling Back in Time

The most likely scenario where you will move back in time is when you have moved an environment forward in time to a specific point and then completed the testing but you now want to re-execute the tests on that specific window in time so the environment is moved in time back to the start of the test period.

If we imagine a scenario where the time sensitive test window is 1^st June 2017 until the 30^th June 2017.

The steps may look something like this:

When you are in this scenario then you may come across some unexpected behaviours which I have discussed below.

Event Log

When you move the date backwards in time then you look at the event log you will find events are displayed by default in the order they are written. They may not appear to be in chronological order. This can actually be helpful if you need to see stuff from before the time change but if you start ordering, searching or filtering the events then you may find this just gets confusing.

BizTalk SSO

I have found that you often get problems when taking actions in BizTalk when an environment has been moved back in time. The error you normally get is:

The problem here is that the data in SSO is date sensitive and this includes configuration information used for BizTalk ports and other such things. When you want to take an administrative action after the environment has been moved back in time you may get the error above if the version of data in SSO is newer than what would be saved after your configuration change.

This is probably the most blocking issue during time travel.

Time Travel Recommendation

As a general rule I try to minimise the number of administrative actions we will need to do during the testing exercise. If I need to do something like a new deployment or stop an application I will normally wait until a suitable point when the environment is not being used and then move the time on the BizTalk/SQL forward in time past when the last deployment was done. I will then redeploy my application and then resynchronise the time with the domain controller.

This is probably the biggest actual constraint which will affect you during the time travel scenarios and you just need to understand the constraint and manage your activities with this in mind. I am not sure there is really a way you can manipulate BizTalk to get around this because you would then be hacking SSO. I think if this is a blocker for you then you probably need to consider undeploying everything and then deploying all of your BizTalk applications again once the environment has been moved back in time. If you have, good automated deployment and time to invest setting this up then you can definitely use this approach but I suspect for many projects it will be overkill to use just for time travel scenarios.

 

BizTalk Tracking Data

When going back in time, the tracking data could become problematic to use. You will struggle to differentiate between data from the first time you ran tests versus the current time as it will all be recorded with the same dates.

Time Travel Recommendation

Fortunately, this one is relatively easy to handle. You need to purge the tracking data when you go back in time so that you have a clean system when the tests start which will make issues easy to troubleshoot. This can be done with the BizTalk Health Monitor Tool.

 

BizTalk Inflight Messages

When you move back in time you really need to remove any in flight messages. They are likely to be in a dehydrated or suspended state anyway but when you move the date any actions they take if they were to become active again may produce unexpected results and at the very least may seem to have executed in a funny way when looking at some of their tracking history.

Some of this will depend on what exactly your orchestrations do but as a general rule just clean the system to a clean state before starting.

Time Travel Recommendation

It is recommended to clean the BizTalk message box using the BizTalk Health Monitor tool.

 

Ordered Delivery Ports

In the scenario where you go back in time you can have problems with ordered delivery ports.  The behaviour I have seen indicates that if you run a set of messages through BizTalk and the instance of the ordered delivery port is created at 5pm on the 11-03-2016 and then you move the date of your environment back to 10-03-2016 and submit more messages then they will sit in a dehydrated state in BizTalk.  It looks like there is a subscription waiting to process them but the behaviour is that the messages will sit in a dehydrated state until your environment gets back to 5pm on the 11-03-2016 and the messages will just process.

It is almost as if the port is created but not due to start processing until the future.

This is a tough one to troubleshoot as it just looks as if the messages are stuck then once the time catches up they just process.

Time Travel Recommendation

If you have ordered delivery ports then it is recommended when moving the environment time to ensure you terminate any suspended/running instances so message processing isnt delayed.

 

Scheduled Task Adapter

With the scheduled task adapter I saw some similar behaviour to what we got with ordered delivery ports.  Imagine I deploy my BizTalk application on the 11-03-2016 at 5pm and it includes bindings to setup the scheduled task adapter on a port.  I have configured the port to run at 10pm each day.  Everything works fine.

When I move the date of my environment back to the 09-03-2016 I find that the port is not triggering my process.  Once I get to the 11-03-2016 at 10pm the port starts triggering the process each night at 10pm.  It is almost as if the application of the binding file at 5pm on the 11-03-2016 was the point when the scheduling will “start checking from”.

Time Travel Recommendation

The recommendation here is to make sure you deploy your BizTalk application and bindings at a point in time on the domain that is before the point when testing will execute in terms of the domain time.  In the above scenario if I had deployed the application at 5pm on 08-03-2016 on the domain time then I wouldnt not have had any problems.

 

BizTalk 360

In BizTalk 360 there are a number of things which are captured which are recorded against a time. Some of these maybe affected by the time roll. While they will not affect the running of your BizTalk system some of the logged information may make issues harder to troubleshoot.

Advanced Message Box Viewer

You may find that advanced message box viewer becomes difficult to work with. This is because the events which are logged go into the database. If you have gone back in time you may find events which happened on the previous time the domain was on this date. You will not be able to differentiate between the dates.

Audit & Governance

The audit and governance module of BizTalk 360 records what administrative actions may have been taken. If you move the time backwards then there may be some overlap in the governance activities which are recorded in BizTalk 360.

Time Travel Recommendation

As a general rule BizTalk 360 has two kinds of data. Data which it reads from BizTalk and data which it stores to track data over time for changes or monitoring purposes. To ensure BizTalk 360 gives you its normal experience during date moves (in particular when going back in time) it is advisable to purge some of the data from BizTalk 360. You can do this manually via the user interface by in BizTalk 360 or there is a stored procedure in BizTalk 360 which can be used too. Please note however it is not recommended to use the stored procedure in a production environment, only consider this for test environments and if you are happy to take the risk of an unsupported approach. The stored procedure is called:

Executing the Time Travel

In general, the execution path for this scenario is probably the same as the other option but it is recommended to purge some data too so that you are not going to get into a confusing situation.

Simple Approach

1. Move the domain time
2. Re-synchronise the time for your BizTalk and SQL Machines
3. Restart the servers
4. Purge BizTalk tracking data
5. Clean BizTalk message box
6. Purge BizTalk 360 data

More Detailed Approach

1. Stop the SQL Agent
2. Stop your BizTalk Host Instances
3. Stop Enterprise Single Sign on
4. Stop Rules Engine Update Service
5. Stop MSDTC on all servers
6. Move the domain time
7. Re-synchronise the time for your BizTalk and SQL Machines
8. Start MSDTC on all servers
9. Start Enterprise Single Sign on
10. Start Rules Engine Update Service
11. Start BizTalk Host Instances
12. Restart IIS
13. Purge BizTalk tracking data
14. Clean BizTalk message box
15. Purge BizTalk 360 data

 

BizTalk Deployments

A number of the considerations listed above come down to how you do your deployment of BizTalk application changes.  Often you will need to deploy changes to your BizTalk applications in the middle of the test window.  Doing these changes without consideration for the point in time tends to lead to one or more of the above issues.

Time Travel Recommendation

With the above in mind I have tended to follow the below process to deploy BizTalk changes.  If we pretend the date of the environment today is 11-02-2016 and my test window runs from 08-02-2016 to 16-02-2016.  The process would be:

Step 1

Step 1 involves taking the BizTalk server out of the time sync and setting it to a point in time beyond the end of the test window.  In this case we would set it to 17-02-2016.  To do this I would run the following commands:

net stop vmictimesync
w32tm /configure /syncfromflags:No /update
net stop w32time
net start w32time
date 02-17-16

This script will stop the Hyper-v Time sync service because I am running on an Azure VM and I want to make sure my VM can go behind the actual time if required.  I will then disassociate w23tm from synching with a time server and set the date to 17-02-2016

Step 2

I will now undeploy the BizTalk applications which I need to undeploy.

Step 3

I now need to move the time of the server to a point before the start of the test window.  To do this I use the following script.

date 02-07-16

Step 4

I will now deploy my BizTalk changes

Step 5

I now need to configure w32tm to sync up with the domain time and synchronise itself.  I also need to start the Hyper-V Time Synchronisation Service back up again.  The below script will do this.

w32tm /config /syncfromflags:domhier /update
net stop w32time
net start w32time
w32tm /resync /rediscover
net start vmictimesync

Step 6

Restart BizTalk hosts

 

Caveat

If I can just caveat the above that we were using a single server playing the role of BizTalk and SQL which was joined to a domain that was hosted in Azure.  It is possible a more complex BizTalk infrastructure setup will require additional steps.

 

Summary

In summary you can successfully complete time travel testing with BizTalk if you use a little bit of thought about how things in your architecture might be affected and how you will execute the time travel. There are also some products out there in the marketplace which can help you by simulating time servers which may make it easier. Unfortunately I have not worked on a project where we have had one of these available but it would be great to hear about experiences of others who may have used one.

 

More Info

Below are some useful links.