CI/CD Pipeline for Azure Search Service

What is Azure Search:

Microsoft Azure Search is a component of the Microsoft Azure Cloud Platform providing indexing and querying capabilities for data uploaded to Microsoft servers. The Search as a service framework is intended to provide developers with complex search capabilities for mobile and web development while hiding infrastructure requirements and search algorithm complexities. Azure Search is a recent addition to Microsoft’s Infrastructure as a Service (IaaS) approach.

Azure Search is an API based service that provides REST APIs via protocols such as OData or integrated libraries such as the .NET SDK. Primarily the service consists of the creation of data indexes and search requests within the index.

Data to be searched is uploaded into logical containers called indexes. An interface schema is created as part of the logical index container that provides the API hooks used to return search results with additional features integrated into Azure Search. Azure Search provides two different indexing engines: Microsofts own proprietary natural language processing technology or Apache Lucene analyzers. The Microsoft search engine is ostensibly built on Elasticsearch.

Problem Statement:

Let's assume as any Developer might have created some views in Database but after a couple of sprints as the requirement changes and you may have to start making changes to Views in Database. And eventually, those views will contain Indexes/Indexers & some times coming from a new Data source(s) as well. Now as the changes are coming very frequently and you have to keep changing in the Database and then you may have to recreate them very often. During this entire process when you do manually you end up causing Human error. So in order to avoid I came up with a solution to mitigate all these issues.

Solution:

The solution is to deploy the changes automatically in the Database and then the same changes to be applied to Azure Search. some of the components which we need as pre-requisite are:

• Azure SQL DB with some views.
• Azure Search SDK
• Microsoft .Net Solution
• Azure DevOps Pipeline

1. So first thing I have just created Azure SQL DB with some views. In our case, I have created 2 views.

2. Now as a developer while building the application for the first time you might have added few columns onto first view such as “Id”, “Code”, “name”, “city” etc.

3. Now you got a new requirement from your business saying you may need to add few new additional columns onto the new Index, then Instead of adding it directly to the Database and avoid unnecessary access, I am going to achieve this using Microsoft .Net Solution.

4. So I have created a new as a prerequisite I have created Microsoft .Net core solution. In this project, I have specifically added 2 .cs files

• Index1.cs
• Index2.cs

5. Now lets open first .cs file and start adding it in .cs file which we want to deploy it in Azure Cognitive Search (Formerly Azure Search).

second Index2.cs file:

Finally Program.cs file and this is the main which is going to call for .cs files.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Threading.Tasks;
using System.Xml.Linq;
using Microsoft.Azure.Search;
using Microsoft.Azure.Search.Models;
using Microsoft.Extensions.Configuration;
using Microsoft.Rest.Azure;

namespace AzureSearch
{
 public sealed class Program
 {

public static void Main(string[] args)
 {

IConfiguration configuration = null;
 var builder = new ConfigurationBuilder().AddJsonFile(“appSettings.json”);
 configuration = builder.Build();
 if (configuration[“SearchServiceName”] == “Put your search service name here”)
 {
 Console.Error.WriteLine(“Specify SearchServiceName in appsettings.json”);
 Environment.Exit(-1);
 }

if (configuration[“SearchServiceAdminApiKey”] == “Put your primary or secondary API key here”)
 {
 Console.Error.WriteLine(“Specify SearchServiceAdminApiKey in appsettings.json”);
 Environment.Exit(-1);
 }

if (configuration[“AzureSQLConnectionString”] == “Put your Azure SQL database connection string here”)
 {
 Console.Error.WriteLine(“Specify AzureSQLConnectionString in appsettings.json”);
 Environment.Exit(-1);
 }

SearchServiceClient searchService = new SearchServiceClient(searchServiceName: configuration[“SearchServiceName”], credentials: new SearchCredentials(configuration[“SearchServiceAdminApiKey”]));
 Console.WriteLine(“Creating Index Begin … ‘index1’”);
 var index = new Index
 (
 name: “index1”,
 fields: FieldBuilder.BuildForType<index1>(),
 suggesters: new List<Suggester> { new Suggester { Name = “sg”, SourceFields = new List<string> { “code”, “site2ndCode”, “siteTypeName”, “name”, “city”, “street” } } },
 corsOptions: new CorsOptions(new List<string> { “*” }, 300)
 );

bool exists = searchService.Indexes.Exists(index.Name);
 if (exists)
 {
 searchService.Indexes.Delete(index.Name);
 }

searchService.Indexes.Create(index);
 Console.WriteLine(“Creating Index End … ‘index1’”);
 Console.WriteLine(“Creating Datasource Begin …’dsource1'”);
 DataSource dataSource = DataSource.AzureSql(
 name: “firstdatasource”,
 sqlConnectionString: configuration[“AzureSQLConnectionString”],
 tableOrViewName: “INDEX_FIRST_VIEW”, description: null);

searchService.DataSources.CreateOrUpdate(dataSource);
 Console.WriteLine(“Creating Datasource End …’dsource1'”);
 Console.WriteLine(“Creating Azure SQL Indexer Begin …’indexer1'”);
 Indexer indexer = new Indexer(
 name: “indexer1”,
 dataSourceName: dataSource.Name,
 targetIndexName: index.Name);

exists = searchService.Indexers.Exists(indexer.Name);
 if (exists)
 {
 searchService.Indexers.Reset(indexer.Name);
 }
 searchService.Indexers.CreateOrUpdate(indexer);
 Console.WriteLine(“Creating Azure SQL Indexer End …’indexer1'”);
 Console.WriteLine(“Running Azure SQL Indexer Begin … ‘indexer1’”);
 try
 {
 searchService.Indexers.Run(indexer.Name);
 }
 catch (CloudException e) when (e.Response.StatusCode == (HttpStatusCode)429)
 {
 Console.WriteLine(“Failed to run indexer: {0}”, e.Response.Content);
 }
 Console.WriteLine(“Running Azure SQL Indexer End … ‘indexer1’”);

Console.WriteLine(“”);
 Console.WriteLine(“”);
 Environment.Exit(0);
 }
 }
}

 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —  — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

6. Now please make changes in your Appsettings.json file with these parameters.

• SearchServiceName: where you want to deploy new indexes/Indexers etc
• SearchServiceAdminKey: This is the Primary key of your Azure Search Service.
• AzureSQLConnectionString: this contains your Data source, Database name along with DB userId/Password.

7. Now let's try to test it, here in this example we just created only 6 columns for Index 1 and 5 columns for Index2. If I open my Azure Search service currently you can see it is blank and doesn’t have any Indexes/ Indexers and no Data Sources.

8. Now let's try pushing it from our Visual Studio. → as I have already added all the required info at point 6.

Let me execute it.

After the console get’s finish you may see new updated Indexes, Indexers with new Data sources.

after a few mins, it will show the actual Index document count and storage size.

After a few mins, the status will get the change to Success

Continuous Integration for Azure Search Service

Now we just tested this entirely using Visual studio → Now If we need to automate this for to deploy the same to any higher environments we will be creating Azure DevOps CI/CD pipeline. As soon as Developers adds/removes any new Indexes and check-in the code Build pipeline task will start.

1. First, I need to upload the code onto the Repository, so in my case, I have checked-in the code onto Master Branch.
2. Now, let's start creating our first Continuous Integration Pipeline.
3. First, I have copy files from my source to the artifact staging directory.

4. Then I need to start with the process of Restore, Build and Publish of my Microsoft .Net solution, Need to mention the path of my AzureSearch project location.

5. A couple of important considerations while Building and Publishing .Net core solution

for Build → Pass the arguments as 

and for Publish → Pass the arguments as

In my variables, I have set “BuildConfiguration” as Debug mode.

6. Next, I have updated the Appsettings.json file with my Environment Azure Search name, along with the Subscription and Resource group name.

7. Next, when you generate Microsoft .net core solution by default it creates an Appsettings.json file with a different structure so we have to add new extra parameters.

8. The next step is to replace variables in the AppSettings.json file where you want to deploy it could be your UAT/Staging/Prod environment.So I am using replace variables PowerShell task

Finally, I am publishing the artifact

when you expand your artifact location you should be seeing the .exe file and this file we will be deploying onto our Prod environment.

Continuous Deployment for Azure Search Service

Now as a part of Continuous Deployment → I will go back to my Releases → create a new release pipeline → Added newly created Artifact which I have to push it to the Production environment.

Now, let's check the tasks in the Prod stage.

first I will be downloading the artifacts and then I am executing .exe file using the Azure PowerShell task.

For Azure Search PowerShell I need the exact location of the AzureSearch.exe file and same path I have to paste it in the Azure Power search command line.

The entire execution may take 2–3mins max.

After the execution again you may see Indexes/Indexers and new Datasource being created in the Production (higher) environment. [same as point #8]

Sample Index.cs and program.cs file is uploaded at:

https://github.com/v6prask/AzureSearchsolution

Let me know your thoughts and comments at prashanth.kumar.ms@outlook.com