This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I tried Apache Spark Direct connection to Databricks. It gives an error ‘DBFS Path not specified’. While configuring the direct connection, there is no field to give any DBFS path and to that matter all of the Hive tables that are created in our Databricks are using Azure Data lake instead of DBFS default storage (blob).
If you set up an Apache Spark On Databricks In-Database connection, you can then load .csv or .avro from your Databricks environment and run Spark code on it.
This likely won't give you all the functionality you need, as you mentioned you are using Hive tables created in Azure Data Lake. If this is the case, I'd recommending emailing firstname.lastname@example.org to look into the error you are getting using the Spark Direct Tool standalone.