Batch Macro to run multiple API calls seems to not run sequentially

[Cremo]

Hi all, 

I used to work with batch macros several times, and always worked as expected. However, I am facing an issue while trying to face the following use-case:

I have a list of different API REST calls, I want to run them one after the other, sequentially. This is why I thought the batch macro could be the correct choice. The problem is that I have 900 API calls, and the workflows runs without any problem, but only for a random amount of times and consequently returning only a subset of the API results. The weirdest part is that the number of output records changes from one run to another, appearing to be random (once it runs for 763 records, another time for 812, and so on). 

Once I debug the missing API calls and try to test them in the macro, they always run correctly. My guess and fear is that the 900 calls are not run sequentially one after the other, and that some calls happen in the same time causing one of the two to fail. Is there a way to force the WF to process each row at a time, waiting for the previous one to complete?

I am sorry but I cannot share the workflow and the URL calls for security reasons. I am attaching some configuration screenshots.

Thank you in advance for your time, and sorry in case of a non-clear explanation. Happy to discuss it further.

 

 

 

[binuacs]

@Cremo remove the field URL Request from the "group by" tab and only update the "questions" tab and try again? "

[BS_THE_ANALYST · Accepted solution]

@Cremo  I agree with binuacs. That trick normally works for me. I've added another layer of security in, just for peace of mind. Remap each url to an Row ID, then allow one of them through at a time. Check the workflow I've attached as an example.

All the best,

BS

[Qiu]

@Cremo 
I also often do the same with @BS_THE_ANALYST 

Add a RecordID and send each record one by one. 😁

[Cremo]

Thank you all for your valuable answer. I tried all of yours solutions, unfortunately without success. The returned records are always less then the original ones, and changing at every iteration. The trick with the RecordID seemed promising, however it did not work..

[BS_THE_ANALYST]

@Cremo the record ID trick ensures that they will run one-by-one.

For some of the API calls that supposedly didn't yield results, are the API calls definitely returning information?

Did you use Binuacs' suggestion and alter the Group By Tab on your batch macro to say none and only have it on your Questions tab?

FYI, a batch macro is practically a For Loop. It's iterative. It will not start the next one until it's finished the current block.

Either you're throwing a whole bunch of rows through together into the the batch macro (by accident) and something unexpected is happening for that reason or the api calls are unsuccessful for some rows?

Beyond that, I can't see why.

All the best,

BS

[binuacs]

@Cremo Also check your result window to see any warning/error messages. 

[Gaurav_Dhama_]

You can make the macro to have a text input (instead of macro input) which a control parameter will change at each run. This way you can be sure that only one API is running per batch.

You can perform the grouping operation outside the batch macro using summarization to create unique APIs passing into the macro, if required.

[apathetichell]

A few things:

1) I don't see a reason for you to do what you are doing in a batch macro. You can run a download tool x times where x is the number of records in your datastream. No reason to have a batch macro here. 

2) Turn off amp - because.

3) use the questions prompt ONLY - don't use a group by for your batch macro.

4) are you sure your issue is with your downlod tool - not your filter tool dropping records where you don't return a 200 - or whatever you are looking for there?

[Cremo]

After a lot of debugging I found out the issue.

FYI: At the beginning, the batch macro was not running sequentially, some of the URL were running in the same iteration and therefore some of them were not processed and were missing from the output. 

After your suggestions, the batch macro were running as expected (one record per iteration). However, sometimes too many requests were sent in a short time frame, and some records were not processed with a HTTP response of 'too many requests'. No error were thrown by the Workflow, and the issue from the output point of view was exactly the same as before: a subset of records were processed, changing at each iteration.

The fact that the misbehaviour was the same led me to think that the problem was the same, but there were two different issues. I fixed the second one by including a "wait" behaviour in the macro to avoid sending too many request in a sort time range.

However, your suggestions helped me solve the problem, so thank you really a lot for your time and kindness. Such a great community