Recently I did a talk about using App Insights at the integration bootcamp and I have also been troubleshooting some performance issues too. During the process of troubleshooting I felt that some of the guidance wasnt the easiest to follow around increasing and decreasing the telemetry being sent from functions to app insights. I wanted to put together a post just walking through an example scenario.
Also a big shout out to Sean Feldman (https://twitter.com/sfeldman) for an assist helping me work out why some of my app settings werent working as expected.
The scenario from the bootcamp is shown in the diagram below. In this post I want to look at the bits highlighted by the red boxes which are the function apps playing the role of my API backend and the processing of messages to my data platform from the Service Bus queue.
The problem I had was how to tune the settings for App Insights when I want to troubleshoot some performance issues to increase and decrease the amount of telemetry being sent to App Insights. I wanted to be able to find a balance for cost vs having the data I need to troubleshoot and be able to easily modify the settings without needing to do re-deployments.
Lets explore a couple of scenarios below.
Test 1 – Default Settings
In my 2 function apps they both have host.json files with logging sections to configure the app insights configuration. I have decided that by default I will have some telemetry configured to give me a decent feel of how the application is running. I have enabled sampling though and set it to apply to everything except exceptions. I have also set the max items per second quite low so it will start sampling under quite low load. The default sampling I would choose would usually depend on how confident I am about the component.
Below is an example of the host.json from the service bus processing functions.
"logging": {
"fileLoggingMode": "debugOnly",
"logLevel": {
"default": "Warning",
"Host.Results": "Information",
"Function": "Information",
"Host.Aggregator": "Warning",
//My Custom Function Configuration goes here
"Function.DataPlatformBridge": "Information"
},
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"maxTelemetryItemsPerSecond": 2,
"excludedTypes": "Exception"
},
"enableLiveMetrics": true,
"enableDependencyTracking": true,
"enablePerformanceCountersCollection": true,
"httpAutoCollectionOptions": {
"enableHttpTriggerExtendedInfoCollection": true,
"enableW3CDistributedTracing": true,
"enableResponseHeaderInjection": true
}
}
},Test Scenario
I will now run 3000 messages through the system at 10 thread concurrency to my API and then let the background functions process at a reasonable concurrency too.
Checking the use of App Insights
I wanted to be able to check how much app insights has been used from my test. I can run the below query which will give me a summary across the logs showing how many events have been logged and what the % of telemetry that has been captured.
union *
| where timestamp > ago(1h)
| summarize RetainedPercentage = 100/avg(itemCount), count() by bin(timestamp, 1h), itemTypeIf you look at the images below you will be able to see that in particular the retained percentage shows that sampling has kicked in when I pushed a bit of load to the scenario and telemetry messages have been being dropped. One point to note here is the load for my test isnt that high so the retained % will still be relatively high. As the load increases on the functions the amount of telemetry that gets dropped will increase and the retained % should go down. You can see that below where the service bus processor has lower % retained because its processing more in parallel.
The review from the API functions was
The review from the backend functions was
Test 2 – Max Logging
The next step is that I have a performance issue I need to troubleshoot and I want to increase the amount of telemetry being sent to app insights so that I have more data to help me analyze the system performance. With a function app I can configure app settings to override the setup of the app insights logging configuration at runtime so I can change the telemetry collection without having to re-deploy the application. I can then run the az functionapp config appsettings set command from the Azure CLI to update my function app and increase the telemetry I want to capture.
Reconfigure Function Apps to increase Telemetry
To make it easier to set lots of settings at once you can also send the command a file which is read and all of the settings in the file are upserted. Below is the commands I used. Note the file with the list of settings I used are at the bottom of the article.
az login
az account set --subscription "[TBC]"
$resourceGroup = "Bootcamp_Demo"
#Backend Functions
$functionApp = "[Backend Function App]"
az functionapp config appsettings set --name $functionApp --resource-group $resourceGroup --settings "@Backend.AppInsights.MaxLogging.json"
#API Functions
$functionApp = "[API Function App]
az functionapp config appsettings set --name $functionApp --resource-group $resourceGroup --settings "@API.AppInsights.MaxLogging.json"
You can see before I run the script my app settings look like the below.
After I have ran the script you can see there is a bunch of new settings added. This will configure the settings I need for deeper dive troubleshooting. One point to note here is that the granularity of the settings mean that I can increase the logging for individual functions too.
Test Scenario
Next I run the test scenario again and analyse the results with my query to review the amount of events logged.
Checking the use of App Insights
If we run the kusto query I ran earlier against the logs for the same small load spike then we see the following data was logged.
The review from the API functions was
The review from the backend functions was
We can see the key observation here is that the retained % for telemetry is 100% as we are no longer dropping events. This gives us a lot of information for analysing performance and we can deep dive into requests, traces and dependencies in particular which will be troubleshooting. For this kind of scenario it is likely the dependencies and trace events are where you will start running up a cost on App Insights. You may have many telemetry events per function execution.
Test 3 – Min Logging
In the real world you might find a scenario where the app you have deployed has the telemetry configuration set too high and you want to tune it right down. This may be so you can reduce the cost of app insights and maximize the performance of your app. Remember the more telemetry you send the bigger the overhead on your application.
In this case I want to tune the configuration to log a very small amount of info but not turn it off.
Reconfigure Function Apps to increase Telemetry
To do this we will run the same commands as earlier but this time I have a different settings file with the settings configured to a minimum level. Im going to log exceptions and performance counters but not much else. Again you can see the files at the bottom of the article.
#az login
#az account set --subscription "[TBC]"
$resourceGroup = "Bootcamp_Demo"
#Backend Functions
$functionApp = "[Backend Function App]
az functionapp config appsettings set --name $functionApp --resource-group $resourceGroup --settings "@Backend.AppInsights.MinLogging.json"
#API Functions
$functionApp = "[API Function App]
az functionapp config appsettings set --name $functionApp --resource-group $resourceGroup --settings "@API.AppInsights.MinLogging.json"
You can see now that after the script has ran, many of the settings are now disabled or set to Error level.
Test Scenario
Next I run the test scenario again and analyse the results with my query to review the amount of events logged.
Checking the use of App Insights
If we now check the query to see what has been logged we can see that API Functions & Backend Processing Functions both show results like below now where you can see no request or trace telemetry is being dropped and not sent to App Insights. This tuning of the app settings has reconfigured app insights to turn off a lot of the telemetry but we are still getting things like performance counters coming through.
Test 4 – Reset to Default
Finally now that my tests are complete I want to return the telemetry configuration back to the default. I can run the below script to remove the app settings I have added.
$resourceGroup = "Bootcamp_Demo"
#Backend Functions
#=================
$functionApp = "[Backend Functions]"
[Array]$settingNameArray = @()
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__default"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Host.Results"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge.User"
az functionapp config appsettings delete --name $functionApp --resource-group $resourceGroup --setting-names $settingNameArray
#API Functions
#=============
$functionApp = "[API Functions]"
[Array]$settingNameArray = @()
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking"
$settingNameArray += "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__default"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Host.Results"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate"
$settingNameArray += "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate.User"
az functionapp config appsettings delete --name $functionApp --resource-group $resourceGroup --setting-names $settingNameArrayRemoving all of these app settings will return the function app to the starting state as per the host.json
Point to Note
Restart
Remember that when you update the app settings it will restart your function app to refresh the settings.
Kill Switch
If I want to completely turn off the app insights telemetry I can also remove the telemetry connection string settings from the app settings too and that will mean no data goes to app insights.
Conclusion
In this example I wanted to show how you can tune up and down the amount of telemetry data you send to App Insights to help troubleshoot performance issues in your system without having to re-deploy components to modify the host.json. I found that the documentation from Microsoft showed some of the settings you can set but it didnt explain that clearly how to use them and what they mean so hopefully this walk through will be useful to people to see this in action.
In my scenario I did a relatively extreme tuning of everything to max or min. If you look over the files we used to update the app settings then it would be quite easy to tune just specific areas. For example if there was one function in particular you were having problems with you could modify just the telemetry for that function while leaving the others on a different logging level.
Reference
Max Logging Files
API Functions
[
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__default",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Results",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate.User",
"slotSetting": false,
"value": "Information"
}
]Background Functions
[
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__default",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Results",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge",
"slotSetting": false,
"value": "Information"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge.User",
"slotSetting": false,
"value": "Information"
}
]Min Logging Files
API Functions
[
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__default",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Results",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.PlantTagUpdate.User",
"slotSetting": false,
"value": "Error"
}
]Background Functions
[
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__samplingSettings__isEnabled",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableLiveMetrics",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enableDependencyTracking",
"slotSetting": false,
"value": "false"
},
{
"name": "AzureFunctionsJobHost__logging__applicationInsights__enablePerformanceCountersCollection",
"slotSetting": false,
"value": "true"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__default",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Results",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Host.Aggregator",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge",
"slotSetting": false,
"value": "Error"
},
{
"name": "AzureFunctionsJobHost__logging__logLevel__Function.DataPlatformBridge.User",
"slotSetting": false,
"value": "Error"
}
]
