There was an unknown error executing the application. The engine reported status Error

[morr-co]

I have an app published to our Gallery that fails after about 3.5 to 4 hours of processing with the message "There was an unknown error executing the application. The engine reported status Error". To troubleshoot:

1. I have pulled the job results from the Gallery API but there is no detailed messaging.
2. I checked our Server configuration and the job timeout is set to 0.

Has anyone else encountered this? What was the solution?

[RishiK]

@morr-co I would advise having a look at the workflow log on the Server. Do you have access to this?

What is your workflow / app doing? - Are you accessing databases or accessing other networks?

[morr-co]

Hi @RishiK - my suspicion is that the machine is running out of memory as the error always occurs after about 3.5 or 4 hours. We are currently modifying the workflows to run the files individually rather than all at once. There were no details when I pulled the job status from the Gallery API but I think you are referring to the logs in the MongoDB?

[RishiK]

Hi @morr-co I am referring to the workflow logs that generated when the workflow is run. If you are processing file by file then latency may be an issue also?

Can I have a look at the workflow ie. to understand what its doing functionally?

Another option I was thinking about was to run the workflow with Performance Profiling turned on so you can understand what the expensive parts of the process is (https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Performance-Profiling-in-v10/td-p/3771)

It is worth understanding what else is running on the Server at the same time too.

How does it behave when run in Alteryx Designer?

[morr-co]

Hi @RishiK - thank you for the response and offering to take a look. Unfortunately, I cannot share the workflow for security reasons. It is also a bit complex - containing several layers of macros. It may take awhile to decipher what it actually does. The high-level view of it is:

1. Two zipped folders are downloaded from an SFTP server to the local directory where they are then unzipped.
2. A batch macro reads and reformats each unzipped file. The output of the batch macro is the consolidated (unioned) data set from all of the files (about 95K records).
3. The data from the file is joined to data queried from a database input to determine which records exist, or do not exist in the database. The join informs which records need to be created, updated, or disabled in the database.
4. We cannot write directly to the database for internal reasons, we are only permitted to submit updates via our product's REST API. This requires about 100-125K API requests (there is no bulk endpoint). 

I did pull the log from the last failed run like you suggested. These are the two messages at the time the job failed:

S:\Alteryx\Service\AlteryxService_Client\Persistence_MongoDB.cpp: 1498. Mongo error: "Document 0 is too large for the cluster. Document is 34797213 bytes, max is 16777216.: generic server error" code: <mongodb:18>

AlteryxService_SvcReportEvent: App <AlteryxService> message <Worker <alteryx-alpha|B3168603> There was an error processing queueId <5fe0ddb4445b0000300078ae>: Mongo error: "Document 0 is too large for the cluster. Document is 34797213 bytes, max is 16777

We have been able to run it completely in Designer without issue. Only when it is run from the Gallery does it fail.

[RishiK]

@morr-co I think this issue is specific to the Alteryx Server and Mongo DB itself.

Reading these links:

https://community.alteryx.com/t5/Alteryx-Server-Discussions/Dynamic-Input-tool-causes-quot-Document-...

https://community.alteryx.com/t5/Alteryx-Server-Discussions/Document-0-is-too-large-for-the-cluster-...

they may give you some ideas.

Are you able to run the workflow in Designer that is installed on the Server Machine?

[morr-co]

I was finally able to produce some of the errors running the files asynchronously and it was in fact related to memory. We converted the job to run the files in smaller batches and were able to have the whole thing complete successfully.

[RishiK]

@morr-co I am really happy that this has been solved for you somehow.