Custom CA certificate to Encrypt Databricks worker nodes traffic

Here is the high level Databricks security architecture. Ref.https://databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html 

 

Though the transport layer security is implemented using TLS, Often times, customers ask the ability to use their own custom CA certificates to use for encxrypting traffic between Databricks cluster nodes instead of Databricks CA certs.

Refer: 

https://docs.microsoft.com/en-us/azure/databricks/security/encryption/encrypt-otw

Here is the step by step procedure to implement the same

 

1.  Bring your own Custom CA cert to a blobstore

2.  Just run this script which will put it in init script. Change the JVM path as necessary

dbutils.fs.put("/databricks/init/cert.sh", """#!/bin/bash

cat < /usr/local/share/ca-certificates/myca.crt

-----BEGIN CERTIFICATE-----

MIIFszL3r6kpPCCiMrCeL3CSmayKqdEKm2HSotsLIvVUK/5HOmc2

uyuyLlcdP7IVuQvpAHYe7u/mtCn8Yp4=

-----END CERTIFICATE-----

EOF

update-ca-certificates

keytool -import -trustcacerts -alias MY-CA \

  -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts \

  -file /etc/ssl/certs/myca.pem \

  -storepass changeit \

  -noprompt

echo "export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> /databricks/spark/conf/spark-env.sh

""")