Unable to send more than 7K records using API post request (download tool) with Json body

[Nagendra_Vuppala]

Hello,

We are trying to send an API Post request to an external software/tool. And we are unable to push more than 7K records. We are using the body as Json and please find the following configurations of Download tool.

URL Field - is API endpoint link

Output to a field - String

Payload: HTTP Action - POST

Take Query String/Body from field - is Json file

Connection:  We are using a username and password as authentication

Maximum Connections : 4

I cannot send you the endpoints and Json file due to confidentiality. We have used the external's tools Swagger page to push more than 7K records with the same user name and password that we configured to download tool and it seems to be working fine. So, it's a clear sign that we do not have any network is open to push as many records as needed and the third party software confirms that there are no limitations in sending the records. And everything works fine when we send less than 7K records.

Can you please confirm if there are any limitations set in the performance of Download tools?

[ArtApa]

Hi @Nagendra_Vuppala - There is no limitation in the Download tool. Most likely there is a limitation in your Data Source. For example, a number of records per minute or number of records per API call.

You can try either to split your data set in batches and send them to your destination with multiple API calls or you can use Throttle in your Download.

[Nagendra_Vuppala]

Hi @ArtApa,

I strongly feel that there is a limitation in pushing a Json file via API using download tool. Please note that I am sending only one request i.e. one Json file which has the details of 10K+ records. And I experienced that the option "Take Query String/Body from Field" under connection tab of Download tool can handle only limited information or characters. In my case it's just 7K lines/records with more than 25 fields.

On the other hand it works when I split the Json files in batches. But the user has to download the batches from the third party software and manually combine all the files to perform the reconciliation which is leading to a lot of manual work. I feel that we will have to implement a python or R script to push these bulk records and not rely on the Download tool due to it's limitations. Can you please confirm? Also the throttle doesn't work because we are just sending one Json embeded in one cell. The moment we split it into different cells, it's going as different batches.

[apathetichell]

@Nagendra_Vuppala  - The limitation is in the specific API you are pinging which has a set max size to the payload or the number of records processed. If you provide more information on the API - perhaps someone can help? In addition you could be facing a filed size limit - ie the string size you are using for your body is too small. Perhaps you need to split your post into multiple calls?

[Nagendra_Vuppala]

Hi @apathetichell - Thanks for your swift response!

As mentioned in my initial request, there are no limitations set in the API that I use. I have tried to push more than 10K+ records using the same user name and password on Postman application and it seems to be working. This may be because of the string size limitation as you said. Currency, the string length of my Json is 13860453. Can you please let me know the maximum size a cell can handle to perform Post API request? Unfortunately, I cannot send you the endpoint of the service provider on community.

I am happy to share my contact details for a catch-up.

[apathetichell]

divide this up to multiple rows/api posts.