Azure Data Factory to Azure Databricks passwordless Integration using Managed Identity without Contributor role

Azure Data Factory(ADF)  : Industry leading Enterprise grade Data Integration service from Microsoft Azure

 

As of this article date, ADF supports 90+ built-in Connectors ranging from  on-prem ,SaaS and multi cloud data sources

 

Azure Databricks: Cloud scale big data analytics and advanced machine learning platform based on code first approach

 

 

How to invoke Databricks notebook remotely from ADF without access tokens:

 Solution explained in the link below by my colleague shows detailed steps on how to configure  Managed Identity access over access tokens

Azure Databricks activities now support Managed Identity authentication - Microsoft Tech Community

Challenge:

   But regulated industry customers generally do not grant Contributor access to any identity as it comes with high privileged access . Hence customers will create custom contributor role with minimal privileges. Unfortunately this will not work with the Managed Identities

Workaround/Hack :

  As a workaround , customers can add the managed identity object ID of the ADF onto Databricks workspace directly using SCIM API eliminating the need for Contributor access at the control plane/IAM blade

Here are the step by step procedure how to do the same

Step 1 : Create a Linked Service in ADF for Databricks:

When you pick “Managed Service Identity “ You could notice the Auth fails because the identity doesn’t have access on Databricks yet.

 

Step 2: Add the Managed identity of ADF as a user in Databricks Workspace using SCIM Service Principal API

 For more details on SCIM API , refer here

 

I use postman collection to test APIs.

Step 3: Ensure the User appears in the Workspace,

Step 4 : Test the connectivity now

Step 5: Now pick the Linked Service and specify the Databricks notebook location