Databricks Data Stream In Tool found bug

[ChristinaFPE]

I'm trying to write to a Databricks table. I've created the In-DB connection, and made a successful connection. I'm able to use the Connect In-DB tool to read the table I want. I can write to the table using the Write Data In-DB tool. However, when I try to write to the table using the Data Stream In tool, I get an error saying I've found a bug. Why can I write with the Write Data In-DB tool but not the Data Stream In tool?

I receive the same "found bug" error no matter what configuration I choose for the Data Stream In tool. I've tried all 3 drop down options: "Create Temporary Table", "Create New Table", and "Overwrite Table (Drop)". And for the "Create New Table" and "Overwrite Table (Drop)", I have tried putting just the table name, the schema.table name, and the database.schema.table name.

[DanM]

@ChristinaFPE  please put in a ticket with our Support team and mention you have posted in the Community. This will help the team track the issue and get it resolved with Product sooner.

DanM

[apathetichell]

There are lots of reasons you may be seeing this behavior - some of which relate to specific driver settings in your odbc. I can tell you that datastream-in works to write to Databricks on my system. But I can also tell you that I've seen that bug error before.

A few quick clarifications 1) you are staging locally (vs in an s3) - and this is set up in your in-db connection.

2) can you share a screen shot of your odbc 64/advanced options config?

3) I use overwrite table/drop (which also creates new tables with an ignorable warning). I use a quoted schema.table syntax. In my in-db write connection string I have no / at the end of the url for the endpoint. I vaguely remember that may have caused this issue. I'm running 21.4 because it works.

[ChristinaFPE]

1) I'm not sure where staging is taking place. Do you know how I could check this?

2) Here is my ODBC64 Advanced Options screen for Databricks:

3) "Overwrite Table (Drop)" gives me the same bug error. Initially, I was having issues with the Write Data In-DB tool until I removed the "/" at the end of the URL for the endpoint, but it did not fix the Data Stream In tool.

[apathetichell]

o.k. - so I'm running an older versio nof the driver - but I do have some different selections worth trying out:

1) uncheck enable translation for CTAS

2) uncheck invalid session autorecover

3) check use native query

4) check fast sql prepare

5) check driver config take precedence

My hunch is 1) could trigger this. Not a wildly huge fan of Alteryx documentation but they even mention that that box should be unchecked - https://help.alteryx.com/current/en/designer/data-sources/databricks.html#databricks

[ChristinaFPE]

I'm still getting the bug error even with those settings changed. It's almost like the tool is making a connection but unable to complete the transaction, but the error message is too vague to figure out what's going on.

When I run the tool with "Overwrite Table (Drop)", it deletes my table that's in Databricks, but doesn't recreate it, and if I put just the schema.table, I get a warning that it can't find the table so it's talking to Databricks, just not completing the entire task.

[apathetichell]

Do you have a default db/catalog name set up in your driver? I think the driver used to require that default was the base catalog name (and in your case it's 'sandbox_automation' - can you make sure that's set in your driver level - and then try running with the schema.tablename as your identifier?

and sorry - there's a ton of specifics for this and lots of reasons this could happen. if you want to downgrate to an earlier version of alteryx/driver - I do know mine works for datastream-in. I run 1039.

[apathetichell]

ONE MORE! What format are you writing to - use CSV! I vaguely remember that Avro didn't work.

[ChristinaFPE]

I set the database at the driver level to be my catalog name (sandbox_automation) which didn't work. And I have the In-DB Connection set Databricks Bulk Loader (CSV). Neither of these two solutions worked :( I've opened a ticket with Alteryx so hopefully we can get this solved. I really appreciate all your suggestions, @apathetichell! Thank you!

[ChristinaFPE]

Unfortunately, the version of Alteryx/drivers on my machine are set by IT and I cannot change those.