How to paginate Data coming from the DB (Not API)

Question

Hello everyone!

I have been trying to come up with a solution to this one for sometime but haven't yet managed.

I need to retrieve all the data from a table that has close to 1B records, but since the DB doesn't allow me to retrieve it all at once I have to paginate this and get 10k records by query run.

I tried using the "Dynamic input tool", but by the looks of it, it only replaces the clause once, and I have to run it multiple times.

Is it possible to do this using a batch macro, if so, how could I set it up?

I'm not using an API route, I'm connected directly inside the database.

AkimasaKajitani · Accepted Answer

Hi @Daniel_Figueredo

It is possible to do it using batch macro.

This is sample workflow.

I make "between" clause at Formula tool.

After that, I overwrite SQL at Batch macro of Update tool.

At my sample workflow, the data of database have record number column as "Row ID".

If you will use the between clause, the data required record number column.