Azure Batch (Service) Automation using AzureDevOps CI/CD pipeline

Jul 01, 2020 Azure Batch, Azure Monitor, Azure CI/CD, 4992 Views

One of my current projects is working with Azure Batch, which we use to run a to processing large calculations. I saw most of the developers were deploying this application via a manual process. Every-time they have to run Microsoft .net solution on their machines to generate zip file, manually set the version, and do this every time we needed a push to our Dev or prod environments.

I took the opportunity to automate this entire process end to end. Currently, not much documentation available pertaining to the straight forward approach, Hope this will help others as well.

The agenda behind writing this article is to achieve Azure batch service deployment along with the required packages. Not many articles available over the internet in order to achieve CI/CD automation using the Azure DevOps pipeline.

My goal was to automate deployment and to do this, we need to look at Batch clients. Microsoft splits the API structure into two different clients:

Management: Manage Batch accounts, storage, and account keys
Service: Manage to compute nodes, pools, tasks, and jobs.

Requirements

We need a couple of Azure Resources to get this to work.

Azure Batch Account
Azure Storage Account
Azure Batch Account:
while creating the Azure batch account it needs to be followed in below workflow.

New Azure batch account creation
New Application creation
Application package assignment
New Job creation
New job schedule creation
Nodes restart

In order to achieve this, I have created a few ARM templates & 1 zip file (with .exe — dt.zip) which will do our work seamlessly.

all these templates have been uploaded at:

As a first step, I have created a Build pipeline

Click on 1published artifact and make sure it has all required files

Now, let's start creating a Release Pipeline, as a part of this step we are going to achieve in the following order

Need to update variables
Create new Azure storage
Create new Azure batch
Create a new Application
Upload New application package
Create new job
Create new job schedule
Restart nodes

a. As a part of updating variables, this creates a new release job and click on Variables and update key-value pair based on your Org requirements.

Make sure you update: Resource group name which is related to your Azure.

resourcegroupname
batchapplicationversion: One of the reasons for using this variable is because every time I generate a new .exe file I want to CI/CD pipeline to pick a new version automatically as a counter.
Location: this is the location where you want to deploy Azure Batch.

As once you trigger a new release it will start creating below components

Now let's look into step by step to show how it really creates in Azure

Storage account creation

2. AzureBlob file copy: this step will upload all required deployment and exe(zip) file to Blob storage.

3. New Azure Batch Service Deployment: this service will get deploy using the “Deployment-application1.json” file.

4. Azure Batch Application creation: this will create a new application.

templateContainerUri $(templateContainerUri) -templateContainerSasToken $(templateContainerSasToken) -batchAccountName $(batchAccountName) -batchAccountPoolName $(batchAccountPoolName) -applicationStorageAccountName $(applicationStorageAccountName)

In the Azure portal.

5. Azure batch application package creation: this will create and associate a new zip file to the application, click on Pools → click on respective PoolID → go to the application package & then you can see the associated application and versioned zip file.

6. Azure batch job creation: The next step is to create a batch job in order to process all your application requests.

Release pipeline view:

Azure View: click on jobs and then it will show all the jobs which are associated with that application pool1

7. The new package assignment is optional based on application to application.

8. The new schedule for newly created jobs:

9. The next step is for Nodes restart, the first time when you create applications. Go to Pools → click on Pool ID → click on Nodes it will show the Name.