Schedule API 500 Internal Server Error

[kgalbert]

Hello!

I had a fun workflow that used a bunch of API calls to summarize everything that happened on the server yesterday and then email it to me.  This ran fine on 2021.4 for about 2 months.

We upgraded to 2023.1 yesterday and the workflow stopped working.  Looking at the details now, most of the API calls are working except for /v3/schedules/

This doesn't work when the server runs the workflows nor in designer itself.

What can I check on the server side to troubleshoot this?

The API error is pretty generic: 

{
"message": "An error has occurred."
}

HTTP/1.1 500 Internal Server Error
Content-Length: 43
Content-Type: application/json; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Content-Type, Accept, Authorization
Access-Control-Max-Age: 1728000
Date: Fri, 21 Feb 2025 21:49:35 GMT

This same workflow is calling these APIs successfully:

/v3/users

/v3/workflows

/admin/v1/workflows/jobs

/v1/workflows

Any insights are appreciated!

I even tried doing a smaller mockup workflow that just has 3 tools; Server API Authenticate, Record ID, and Server API GET.  This mockup can successfully run /v3/workflows, but just changing the selection the Action tab to /v3/schedules/ brings back the 500 error.

Thanks,
Ken

[kgalbert]

Hi everyone,

I have some updates:

Hello All,

Updated Issue: The Schedule tab on the Admin side doesn't list any schedules (gallery/ #!/ admin/ schedules).  The API endpoint /v3/schedules/ returns an error 500.  The schedules themselves run normally.  Looking further into the schedule tab, it's actually just calling a different API and also getting a 500 error.

Background: We upgraded from 2021.4 to 2023.1 recently and this wasn't an issue before.   The API worked fine.  I reviewed the service logs and it had errors with 14 ids from AS_Schedules.  However, when I look at the mongodb, those IDs aren't on the table.  If it go through AS_Queue data, I can find what workflow they're related to and that workflow still exists, but not sure if deleting the workflow will help.

Example Service Log error: S:\Alteryx\Service\AlteryxService_Client\src\Persistence_MongoDB.cpp: 1578. PersistenceContainer_MongoDBImpl_Get_Error: Record identifier is invalid <624c8be57c61000087005e04> collection <AS_Schedules>

Next Steps: What should they be?  I'm thinking that if I knew what API /v3/schedules/ or the admin schedules page were actually doing I could figure out the intersection between where these 'bad records' sit and how the system is trying to access them.  Obviously it's not going directly to AS_Schedules or it would work because the bad records aren't there.

[fmvizcaino]

Hi @kgalbert ,

I had a similar issue before and deleting the bad data from Mongo solved the problem. 

I would also inspect the browser page and see the exact error message you are receiving in the network tab. Maybe you can confirm if the error is indeed with the workflow that had the schedule deleted.

Just an example showing how to get to the network tab. https://help.fullstory.com/hc/en-us/articles/360020623654-How-do-I-use-the-Network-view-in-Dev-Tools

Best,

Fernando V.

[kgalbert]

Hey Fernando,

Thanks for replying!  So I tracked down the error in the network tab and it's just as useful as the API error:

{data: null, exceptionName: "UnknownException", innerExceptionMessage: "",…}

data: null

exceptionName: "UnknownException"

innerExceptionMessage: ""

message: "Unknown error.\ nNullReferenceException\nRequestID: 63dbd0ce63b84250bc469eb07d7cd563"

Maybe a little more helpful since it says Reference Exception.  It's trying to reference something that isn't there.  

Can you give me any hints on what bad data would look like in Mongo?  Just missing fields or like scrambled fields?

Thanks,
Ken

[fmvizcaino]

@kgalbert , Alteryx has its internal API requests and it is very difficult for us to know what exactly is happening in the background. 

However, the relationship between the collections is often added to JSON files inside the MongoDB collections, which might cause the issue you are having. A good test is, of course, to delete the workflow that used to have the schedule.

Another thing you could try doing is reindexing your MongoDB. This is a 'known' process we sometimes do after an upgrade or when we see weird things happening in the Server. Important to mention that this won`t break anything, only fix issues.

https://knowledge.alteryx.com/index/s/article/How-To-and-When-to-run-a-manual-reindex

Best,

Fernando V.

[kgalbert]

Hey Fernando,

I deleted all 14 offending workflows and no improvement.  Looking at re-indexing now.

Thanks,
Ken

[kgalbert]

Reindex didn't work.  Doesn't look like Schedules is an index that gets rebuilt by looking at the results of the re-indexing workflow.  

* - edit - The AS_Schedule collection doesn't have an index to rebuild.

[fmvizcaino]

How are you connecting to your MongoDB? ODBC, robo3t, alteryx connector? I wonder if the records are not showing up just because they are sort of corrupted but they are there. Since the schedule endpoint is not working, I`m almost sure something is wrong with the schedule collection.

This schedule page issue happened to me once, and I don`t remember it being this difficult to fix. 

Also, which endpoint are you currently using? The one starting with v3 or v1?

Best,

Fernando Vizcaino

[kgalbert · Accepted solution]

Hey Fernando,

Thanks for checking back in.  I'm connecting using robo3t.  End points are a mix of v3 and v1.  Schedule is v3.  I opened a ticket with Alteryx support who explained how these errors happen.  Which is basically:

a) schedule a workflow and then delete that workflow 

b) schedule a workflow and something weird happens to that workflow

I also found a way of narrowing down the problems:

Run the schedule API once for each user using the ownerId parameter.  We have about 60 users and 4 of them were getting 500 errors with the rest on 200 whether they had anything scheduled or not.

2 of the users were easily fixed because they had schedules related to deleted workflows

The next user turned into a bit of a blood bath.  We deleted all their schedules --> still 500.  We deleted all their workflows --> still 500.  We then joined the AS_Schedules table to the AS_Queues tables.  Queues has all of the workflows executions ever on the server.  One workflow stood out because the workflow was owned by someone else and scheduled by someone else, but this user at one time had that workflow scheduled.  We found there was still an old schedule related to that workflow and deleted that too and now they're not 500 anymore.  Even though that schedule DID NOT have this user on it.

The last user is actually myself.  I'm the most active user and have many more schedules and workflows. So I'm still digging around.  Since deleting all of the schedules for user #3 didn't solve it, so I'm hesitant to try that.  It's also difficult to just join AS_Schedules to a list of workflows because of versioning.  I think the AS_Schedules table has the original version id so need to link them back to the current version from /v3/workflows.

I'll post again once this is fully solved.

Thanks!
Ken

[kgalbert]

Hey everybody,

Final update as it's fixed now.  So going through my scheduled workflows I deleted the test schedules that I created with the Alteryx Support people and that fixed my account.  So.. I'm guessing that my issue was related to something we deleted with Users 1,2 or 3.  Not sure.

Best of luck to anyone with the problem in the future.

Thanks,
Ken