Commits Promotion Between GitHub and Azure Databricks

One of the projects I'm currently working on is utilizing Azure Databricks for its machine learning component. The machine learning engineers working on the project wanted to use external IDEs for the development. Unfortunately, using external IDEs doesn't remove all needs for developing or testing directly in Azure Databricks. As we wanted our GitHub repository to be the only source of truth, we had to establish a commits promotion approach that would enable that.

Azure Databricks has support for Git integration, so we've decided to start by using it to integrate Azure Databricks with GitHub.

Configuring GitHub Credentials in Azure Databricks

The first step in setting up Git integration with Azure Databricks is credentials configuration. This is something that every engineer needs to do independently, to enable syncing workspace with a specific branch. It requires the following actions:

1. Login to GitHub, click the profile picture and go to Settings and then Developer settings at the bottom.
2. On the Settings / Developer settings switch to Personal access tokens and click Generate new token.

3. Fill in the form:

   • Provide a recognizable Note for the token.
   • Set the Expiration corresponding to the expected time of work on the project.

   • Select the repo scope.