This week we had a little issue with some of our service connections in Azure DevOps which talk to Azure for deployments. It had us stumped for a short while and actually we missed a trick as the answer is in the Microsoft documentation but it wasnt that obvious to join the dots to get to the right fix which was actually quite stupid really but I thought id make some notes on this for when it happens again.
Symptoms
- The Builds started failing with an error on the key vault secret retrieve Get secrets failed. “Error: Could not fetch access token for Azure. Verify if the Service Principal used is valid and not expired”
- You look in Azure AD at the service principals app registration and you can see the secret has expired
The error message is pretty good and you know straight away what is wrong. The problem is however if you create a new one and go to Azure DevOps there isnt a way to update the secret!
This was because when the service principal was setup there is a decision to make by the user to use “Automatic” and “Manual” as shown below.
Automatic is recommended because this does a lot of the work for you, but it might actually bite you 2 years later as we found out. Automatic will use the user who is setting up the service connection and will go and create an app registration inside Azure AD and setup a secret and register it in Azure DevOps. This is done behind the scenes. If you choose “manual” you setup the Service Principal yourself and register the information in Azure DevOps.
Now we have the problem when the secret is expired and needs to be updated. We found that in the documentation there is a section about “Automatically created service principal secret has expired” and followed that by clicking the Verify button, but it didnt work. We were getting the below error.
At this point we made a silly mistake because we completely missed that the next bit of advice further down the documentation tells you how to fix this but we were trying to workout why the verify button wasnt working. Anyway the fix to this was:
- add a character of text to the description field on the service connection in Azure
- Click save
- This will refresh the service principal and create a new secret for you
- Open the service connection back up
- Click verify
- It now all works
Note you do need appropriate permissions when doing this too which might be an extra couple of hoops for some people
There are 2 lessons to learn from this:
- Read more documentation than just the specific bit you think is your issue because we didnt spot the error message when clicking verify was the next item after the advice for our original error message
- Dont use the automatic setup for the service connection, its easier in the short term but long term it might be a pain when things need renewing
Hope this helps someone else not fall into the stupid mistake I did, the doc we used is below
