I am feeding my source data into 'Dynamic Input In-DB' tool after doing some filters. Sometimes, I get zero records after applying filters and as a result Dynamic In-Db tool throws error. How can i update my workflow to use a test data to feed into Dynamic In-DB; when there is no record from source data?
can you stream out and use the test tool setting the expecting number to 0?
something like this
@ayush_mishra would a test tool not work in this case?
nope
ok what about a conditional runner from the crew macros, would that help?
http://www.chaosreignswithin.com/p/macros.html
Hi @ayush_mishra, what is the error you're getting? I don't get any errors when I output a sql query with no records
@OllieClarke snapshot below. I am applying filters in upstream process which contains inputs (line with unique/duplicate tool) for In-DB tool. Text input has my SQL query.
Ah @ayush_mishra the issue looks like column references in your sql don't exist in the table? The error is the
invalid identifier "CONCAT_SHIPMENT_LIST"
You might need a step which checks whether the SQL you're generating is valid. You can use SHOW COLUMNS in snowflake to list available columns, and then check whether the query you're generating works against it?
@aatalai @OllieClarke thanks for sharing your suggestions. I used Union tool and passed test data through Text Input before In-db tool and filtered it out after In-db process.