Having used Azure DevOps builds for quite a while using self hosted and microsoft hosted build agents, one of the scenarios I have not had the change to use the option of using an Azure VM Scale set as a self hosted build agent. When I looked at this there were a couple of things I struggled with and I wanted to make a few notes which I think might help others.

First off the 2 links below are really useful at giving you some background info on how to do it:

I followed these resources and got most of the way there but when I tried to get a build working I found there was a few additional hoops I had to jump through and I wanted to take the opportunity to go over these. Before we jump into that I just wanted to touch for a second on the “why” point about using the scale set.

Why use a scale set

The key benefit to the Microsoft hosted build agent is that its simple and easy to use and will let you cover many many scenarios.

The self hosted build agent is great if you need a specific server setup for running your build but the trade off is that its a single machine so you lose the option to have a truly clean machine easily for each build.

The scale set is a blend of the best of both. I can setup a scale set and then allow Azure DevOps to take control of the scale set and it can be configured to use a new clean instance for each build by re-imaging the VM back to the clean state. You can also allow the pool to dynamically scale up to support running a lot builds and then it can also scale back down to zero which is great for cost reduction and means it can be cheaper than having your hosted VM running all of the time.

For our scenario we have a solution which utilises a lot of the private networking options which makes it harder to do a complete CI/CD process without being connected to the network. With the scale set we can have the vm’s within the scale set connected to a subnet in the virtual network.

From a security perspective you also dont need your scale set to be publicly visible and you dont even need to be able to log into it so it ticks quite a few boxes here.

You can also setup a VM with all of the stuff you want installed and then image it and use that image to be the foundation for each instance that is created.

Where’s the challenge

All of the above sounds great but the are some challenges. We wanted to take the approach of using one of the out of the box windows images rather than using an image we made ourself but the out of the box images are a little more locked down and dont have the same software installed.

I wanted to share some notes on what I had to do to make it run the tasks I needed.

Tip 1 – Setting up the Scale Set

The first thing I did was try to setup the scale set in the portal and I actually just made my life harder. There are a couple of settings which Azure DevOps recommends which I hadnt set right. Also the examples mainly seemed to cover a linux example.

The below azure cli command will setup a scale set using the base Windows 2019 image with a Standard B2 size with the right things I needed for my demo.

$resourceGroup = "BLOG_BUILDAGENT"
$name = "blog-ms-buildagent"
$image = "Win2019Datacenter"
$adminUsername = "[Add here]"
$adminPassword = "[add here]"
$vmSize = "Standard_B2s"


az vmss create --resource-group $resourceGroup --name $name --image $image --admin-username $adminUsername --admin-password $adminPassword --vm-sku $vmSize --instance-count 2 --disable-overprovision --upgrade-policy-mode manual --load-balancer '""'

Tip 2 – How do I get the DevOps agent on the Scale set and configure it

The first thing I didnt quite understand was how to setup the devops agent on the VM. I had tried setting this up by hand a few times and wasnt doing this right and my machines kept failing. Its actually really easy to do it correctly, if you setup the scale set correctly like I mention in tip 1 then when you add the Agent Pool using the option of scale set and point to your existing scale set it will automatically add the Azure DevOps pipeline extension and it will provide the right configuration settings.

I think where I went wrong originally was to not set the scale set up properly then this step kept failing.

Tip 3 – Removing your Agent Pool in Azure DevOps

One bit I had problems with until I realised what I was doing wrong was when I setup the agent pool and did it wrong, I was deleting it then trying to add it again and it kept saying that the pool was still there.

What I did wrong was to be adding and removing the agent pool at the project level. When I remove it I thought that it was caching the pool and it kept saying I cant add a pool because it already exists but when I had deleted it, it was only removed from the project and not the organisation.

Notice the 2 url formats below. If you need to replace the agent pool make sure you do it at org level otherwise your removing it from the project but its still there for other projects.

Organisation level – https://dev.azure.com/ [Org Name] /_settings/agentpools

Project level – https://dev.azure.com/[Org Name]/[Project Name]/_settings/agentqueues

Tip 4 – Interactive Tests

I fell into this gotcha without thinking about it properly. When I configured my build agent I had just ticked run interactive tests because it sounds cool and who wouldnt want to be able to do that right!

What this does however is make the build run with interactive permissions rather than as a service. The effect of this is you wont be able to run commands which require elevated permissions and some things you might want to do in the pipeline will get an error. Below is the

When I was troubleshooting this I came across a very handy little idea and Ill appologies I cant remember where I found it to shout out to the person who did the script, but I added a powershell task to my pipeline with the below script which tells me if the build user has admin permission which helped me workout that I had misconfigured the agent.



$CurrentID = [System.Security.Principal.WindowsIdentity]::GetCurrent()

$CurrentPrincipal = new-object System.Security.Principal.WindowsPrincipal($CurrentID)

$adminRole = [System.Security.Principal.WindowsBuiltInRole]::Administrator


if ($CurrentPrincipal.IsInRole($adminRole)) {
    write-host "Yes we are running elevated."
}else{
    write-host "No this is a normal user session."
}

Tip 5 – What software do you need

I think one of the key things for our decision about the build agent image depends on what software we need. If you need something like Visual Studio on the agent then you probably want to create your own image and install this first because these will take a long time to install each time the machine is span up.

In this scenario you may want to think about the option of using the cloud hosted agent for these stages of your pipeline or having your own image with all of this software pre-installed.

There is then the option to install software either in your pipeline or via a custom script extension on the instance setup on the scale set.

Next Steps

We managed to get a scale set up and running as a self hosted build agent quite well once we had understood the above considerations.

We did have some problems with running a few tasks which needed additional steps to be performed which are already pre-covered if your using the image from the Microsoft hosted build agent which I will cover in their own articles.

 

Buy Me A Coffee