What is Azure Databricks?

1. Pre-requisite Knowledge –

Before we start with the understanding of what is Azure Databricks, we should know –

• Basic knowledge of cloud computing and its services
• Basic knowledge of Microsoft Azure
• Basic knowledge of analytics and its services.

2. Background –

I would like to explain the short introduction of the ‘Apache Spark-based analytics platform’ before jump into Azure data bricks.

Apache Spark-based analytics platform –

• It is an open-source parallel processing framework and fast clustering computing system.
• It is leading platform large scale SQL data source, batch processing, stream processing, and machine learning (ML)
• It is a great platform for big data distributed processing frameworks.
• Spark can be deployed in a variety of ways
• It has native bindings for the Java, Scala, Python, and R programming languages, and supports SQL, streaming data, machine learning, and graph processing.

3. Introduction –

• This is an enhanced platform of ‘Apache Spark-based analytics’ for Azure cloud meaning data bricks works on the ‘Apache Spark-based analytics’ which is the most advanced high-performance processing engine in the market now.
• It also provides a great platform to bring data scientists, data engineers, and business analysts together.
• It provides an end-to-end solution for all types of data, analytics and builds artificial intelligence (AI).
• Azure data brick Apache Spark environment set-up takes a few minutes only.
• It supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch, and scikit-learn.

Image Source – Microsoft Docs

• Most of the time, the raw/structured data is pushed by using Azure Data Factory or real-time by any other technique like Kafka to the Azure.
• This data stored into the Azure storage like a blob or data lake etc.
• Azure data bricks this data from one or multiple data stores in Azure and turn in to insights using Spark.
• Azure data bricks have tight integration with Azure data stores like ‘SQL Data Warehouse, Cosmos DB, Data Lake Store, and Blob Storage’ as well as the BI tool like Power BI to view and share the impactful insights.
  
  
                          Image Source – Microsoft docs

4. Azure Data Factory Tangible Benefits –

• Fully managed Apache Spark clusters in the cloud –
  1. It has a secure and reliable production environment in the Azure cloud.
  2. The environment is managed and supported by Spark experts in the Azure cloud.
  3. We can create clusters in seconds, auto-scale the clusters.
  4. Use secure data integration capabilities on top of Spark.
  5. We can access the clusters using REST APIs.
• Databricks Runtime - With the Serverless option data scientists iterate quickly as a team.
• It is tightly integrated with Azure and Spark.
• It is the collaborative and integrated environment, Azure Databricks streamlines the process of exploring data, prototyping, and running data-driven applications in Spark.
• It has enterprise security like integration with Azure Active Directory, role-based access, etc.

5. Reference Links –
   • https://azure.microsoft.com/en-in/services/databricks/
   • https://azure.microsoft.com/en-in/resources/videos/azure-databricks-overview/
   • https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks
   • https://databricks.com/blog/2017/11/15/introducing-azure-databricks.html
   • https://databricks.com/blog/2017/11/15/a-technical-overview-of-azure-databricks.html

Conclusion - In this article, we have learned the overview of Azure Data bricks, its services.