Error: Lock obtain timed out: MongoDB.Lucene.MongoDocumentLock

[MichaelAd]

Error: Lock obtain timed out: MongoDB.Lucene.MongoDocumentLock
Typically, this happens when Alteryx Server crashes during a write operation on MongoDB. 

Prerequisites

• Alteryx Server
  • All
• Admin permissions to run Alteryx System Settings
• Additional Tools - Robo3T would make this process easier, however you can perform all steps using the Windows Command Prompt

Pre-check
Before starting with the below steps, it is imperative to make sure that the underlying issue is not related to a MongoDB schema / migration issue occurring from a Server upgrade. If you recently upgraded, review Gallery logs for entries like these:
 

idea Skyscrapers
 

If the logs do NOT have sections looking similar to the above, specifically "Queue worker sleeping for 1 second", there is a more serious issue ongoing. Please create a new Support case so a Customer Support Engineer can investigate a failed MongoDB migration. Do not proceed with the below instructions.

Symptoms

This is a list of some issues commonly encountered that can be solved using the below steps. Note, not all symptoms need to be present for this article.

 

• Newly uploaded workflows do not appear in Gallery. This occurs even after deleting the browser cache and reloading the page (hard refresh).
• Old Workflows that have been deleted still appear in Gallery.
• Newly added Users in the Admin > Permissions section do not appear (Windows Authentication only).
• New users do not appear in search results
• Newly created Collections do not appear in the Collections page

Diagnosis

The following entries in the Gallery logs will point to the issue discussed in this article.

2020-05-11 15:53:06.564593,FATAL,18,QueueWorker,RunOnce,,,,LD-LT-241,,,,,,Lock obtain timed out: MongoDB.Lucene.MongoDocumentLock->Retry in (1) seconds...,"Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: MongoDB.Lucene.MongoDocumentLock-context, String name)-

Cause

While AlteryxService is writing to MongoDB, it sets the LuceneFs lock, to make sure that MongoDB is synchronized correctly with the writing operation to prevent database corruption. As part of the MongoDB shutdown procedure whenever AlteryxService is stopped correctly, the LuceneFs lock is cleared.
If  however, the service running MongoDB has an unclean shutdown (crash) while the LuceneFs lock is set, the LuceneFs lock remains in place even after AlteryxService stopped. This will typically happen when the crash occurs during a longer writing operation. This lock prevents AlteryxService from starting unless cleared manually.

Please note if you have to repeatedly clear the LuceneFs lock, it is imperative to investigate the root cause of the issue.

Deleting the Locks

Please note that the pre-work differs depending whether an embedded MongoDB or a user-managed MongoDB is being used. Most users will use an embedded MongoDB. You can confirm the type in Alteryx System Settings: Controller > Persistence > Database type.
 

Pre-work for embedded MongoDB

1. Open Alteryx System Settings.
2. Navigate to Controller > Persistence
3. Copy Password (not Admin Password) from Persistence and save it for later use

idea Skyscrapers

4. Open Command Prompt as an Administrator
5. Change directories to the bin folder of your Alteryx installation. Default location is "C:\Program Files\Alteryx\bin"

cd "C:\Program Files\Alteryx\bin"

Pre-work for user managed MongoDB

1. Open Alteryx System Settings.
2. Navigate to Controller > Persistence
3. Copy the Admin Password  from Persistence and save it for later use

Please note that for Server 2020.1+ there is an alternative option connecting to MongoDB, the advanced connection string. It has the format mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]].
If the customer uses a separate MongoDB connection for Gallery copy the credentials from Gallery > Persistence.

4. Open Command Prompt as an Administrator
5. Change directories to the bin folder of your Alteryx installation. Default location is "C:\Program Files\Alteryx\bin"

cd "C:\Program Files\Alteryx\bin"

 

Deleting the locks

1. Connect to MongoDB

mongo -u user -p PASTE_PASSWORD -host localhost:27018 AlteryxGallery_Lucene

You should see a welcome message to show you got in successfully

2. Enter the following command: (it should return 1-2 locks)

db.luceneFs.locks.find()

3. Remove the records with this command:

   db.luceneFs.locks.remove({})

Confirm you resolved the issue

1. Publish a simple workflow to Gallery
2. It should appear in Gallery (do not use View in Browser button in Designer).
3. If this does not resolve the issue, a reindex may be needed.