Scheduled Gallery Jobs | BigQuery Simba | In-DB | Data-Stream Out| Zero 0 Records

[JMB001]

Hello,

Server Version: 2023.1.1.123

Designer 2023.1.1.281 Patch 3

SIMBA ODBC Driver for Google BigQuery v3.0.2.1004 (64-bit)

Problem: scheduled job shows Data Stream Out tool with 0 records while records do exist in queried table.

First Noticed: 8/22/2024

Frequency: sporadic

Workflows effected: seems random, one day its one workflow, next week it's another

Diagnostics:

Source Table Schedule Issue? I looked into the refresh times for the source tables and didn't find any overlap issues with the schedules.

Filter Logic in the workflow? The issue doesn't appear to be the data since I can re-run the workflow manually on the same node a couple hours after zero records were streamed out and there will be records that stream out. It will also produce records one day but not the next.

Any ideas on how to ferret out what's going on?

Is this like the laundry monster eating my socks in BigQuery? 🤣

[fmvizcaino · Accepted solution]

Hi @JMB001 ,

I don`t know much about BigQuery ODBC's advanced settings, so I won`t say anything about it other than to increase the log details to see if you find anything there.

One idea is to have a simple batch macro triggered only if no records are coming from the inDB input. This doesn`t solve your issue, but it will be good to know if the connection will work seconds later. Having two different types of inputs, will help identify if the issue is in the inDB tool or the connection in general.

Best,

Fernando Vizcaino

[JMB001]

Looping in @jacob_esposito, @STP  - was thinking of your control container workflow too.

Thanks @fmvizcaino for the tips on how to start.