Azure Databricks Delta Lake Bulk Load with Azure Datalake Access Control Lists

[AStasi]

It is great to see the ability to stage data for bulk loading into Databricks in s3 and ADLS.  Previously this only appeared to allow staging in Databricks DBFS. 

However the current connector included in Designer 2022.1 has a key gap in functionality with ADLS Gen 2 

The only authentication method provided to the ADLS storage is though a shared key. 

Shared keys provide access that is

1. Too broad and allows access to the entire storage
2. Limited or no auditability and traceability to who is using it 

We do not provide users the shared key for the ADLS storage, thus our users cannot take advantage of this new feature. 

The preferred method of authentication to ADLS would be

1. RBAC - Role Based Access Control
2. ACL - Access Control lists

Either of these options can be provided though a service principal with a tenant id, client id and client secret as inputs to the bulk load tool

This request would specifically be to allow the ACL authentication. ACL would help empower our our self service data analysts and data scientists who could have access to a specific container. 

For example

storageAccount/Container/directory

The ACL access in this tool would allow the Alteryx tool to follow the same access patterns where fine grained access was provided at the directory level and not at the storage account or container level.  This would provide self service analysts and data scientists to use Alteryx as they need within their directory without providing higher level access.  

Access control model for Azure Data Lake Storage Gen2 | Microsoft Docs