Alteryx Server Knowledge Base
Definitive answers from Server experts.Troubleshooting a Failed MongoDB Migration - Server 2018.1-2019.2
Issue
As documented in the Community article Alteryx Install Wizard | 2018.1,Alteryx Server version 2018.1 includes a new version of the embedded MongoDB (MongoDB functions as the server's persistence layer). If you are upgrading your Alteryx Server from an older version to 2018.1 or greater, your MongoDB will be migrated from Mongo 3.0 to Mongo 3.4 as part of the process.
Some users have reported the following error after beginning the migration :
Unhandled exception has occurred in your application.
at System.Diagnostics.Process.EnsureState(State state)
at System.Diagnostics.Process.get_ExitCode()
at MongoDbUpgradeTo30.MongoDBManager.HandleMongoExit(String dbPath)
at MongoDbUpgradeTo30.MongoDBManager.StartMongo(String mongoExeName, String dbPath, String port, Boolean withAuth, Boolean setMmapFlag)
at MongoDbUpgradeTo30.Migrator.StartNewMongo()
at MongoDbUpgradeTo30.Migrator.PerformMigration()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at MongoDbUpgradeTo30.MainForm.StartMigration_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
---> (Inner Exception #0) System.InvalidOperationException: Process must exit before requested information can be determined.
at System.Diagnostics.Process.EnsureState(State state)
at System.Diagnostics.Process.get_ExitCode()
at MongoDbUpgradeTo30.MongoDBManager.HandleMongoExit(String dbPath)
at MongoDbUpgradeTo30.MongoDBManager.StartMongo(String mongoExeName, String dbPath, String port, Boolean withAuth, Boolean setMmapFlag)
at MongoDbUpgradeTo30.Migrator.StartNewMongo()
at MongoDbUpgradeTo30.Migrator.PerformMigration()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
After hitting "Continue"the installer freezes on the following migration window.
Eventually,to complete the upgrade, the user will click "Cancel".
While the server may still run on the MongoDB 3.0, this may cause unexpected behavior in the long term. Should you encounter this error, it is important to troubleshoot it and make sure that your MongoDB is able to successfully migrate.
Environment
- Alteryx Server
- Upgrading from a version < 2018.1 to a version≥ 2018.1
Diagnosis
If this error is encountered, the server itself will upgrade, but fail to restart the service.
To determine which troubleshooting steps to take, we first need to check if the migration was actually successful or not :
- Is there a MongoDB_34 folder under %ProgramData%\Alteryx\Service\Persistence?
- If there is a MongoDB_34 folder, check the migration.log file found under %ProgramData%\Alteryx\Service\Persistence\MongoDB_34 :
A "successful" migration should have the following:
--Starting: C:\Program Files\Alteryx\bin\mongod3_0_4.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB" --port 27019 --storageEngine mmapv1 --Starting: mongo3_0_4.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"
This section starts up the original 3.0.4 database with the 3.0.4 executable and gets some stats on it. You'll see a list of collections and a count of items in them.
--Starting: mongodump3_0_4.exe --host localhost --port 27019 --username user --password BB8B81A9539E3A6BF3CF9360ADAF13DBD90F6B21 --out "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
... {backup logs}
--Stopping MongoDb
--Starting: mongo3_0_4.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"
This section runs a Mongo dump out to the PreUpgrade folder specified in the migration utility. You should see that the databases get dumped out to bson files. Hopefully, we won't see any errors in there. Once the backup completes the MongoDB service is stopped and the .js script verifies the MongoDB is not responding.
--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34" --port 27019 --storageEngine mmapv1 --Starting: mongorestore.exe --host localhost --port 27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
... {restore logs}
--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\createUsers.js"
... {user creation logs}
--Stopping MongoDb
--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"
We then start Mongo 3.4 server, and restore to the new MongoDB_34 directory from the previous backup that was just taken.Again, we should hope not to see any errors in here.
Then, the createUsers.js re-creates all the user accounts for each database.
The MongoDB is then shut down again, and the .js script verifies the service is down.
--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34" --port 27019 --storageEngine mmapv1 --Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"
The new MongoDB is then started up again with Mongo 3.4, and we run getstats.js again, giving us a list of the collections and the count of the documents, similar to the first step.
If everything looks good up to this point, the migration has likely been successful.
- Are the System Settings pointing out the new MongoDB folder (MongoDB_34)?
If everything looks good in the logs, but the service didn't start or something happened after this point, we should be able to manually change the persistence directory to point at the new MongoDB_34 directory, and in theory, the service should start up fine.
- What version of Mongo is running after the attempt?
In the persistence directory of the target database, there's an "ASMongoDBVersion.bin" that seems to match up with the version of Mongo.
Solution A
If the log contains all the information mentioned previously and the migration seems successful, but there’s an error starting the service:
- Make sure that the Persistence folder is set to MongoDB_34 and not MongoDB.
- Restart the service.
Solution B
If the migration log doesn’t match the description above, run the following utility to give migration another attempt :
C:\Program Files\Alteryx\bin\MongoDbUpgradeTo30.exe
If there had been some previous attempt to upgrade from Mongo 3.0.4 to 3.4, then it's possible that the mongoDB is already at 3.4, but the migration software cannot tell. Referring to the second Spoiler above, if human intervention was ever required to manually restore a backed-up version of the mongoDB, then C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\ASMongoDBVersion.bin may not have been updated. It will still say the version is 3.0.4. If you ever have to manually restore a backup to complete a migration to 3.4, be sure to replace the text in that file with
MongoDB shell version v3.4.10-62-g960c883
git version: 960c883bf7ef7b7f0f72fc99f79b0fb110006e55
OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
allocator: tcmalloc
modules: none
build environment:
distmod: 2008plus-ssl
distarch: x86_64
target_arch: x86_64
If the mongoDB has been migrated, but that file not corrected, when you subsequently run an upgrade, the installer will attempt to (again) migrate the database to 3.4, but that won't work because the database is already WiredTiger, so the old (3.0) mongo executable is incompatible. If you attempt to run MongoDBUpgradeto30.exe, it will also fail. In either case, the error message will look like this:
Could not start the previous version of MongDB: The MongoDB database failed to start with exit code: 100
The solution in this case is to verify independently that the mongoDB migration has actually occurred. One way is to go to C:\Program Files\Alteryx\bin and run
mongod --version
another way is to view the contents of the mongo collection versions and verify that the schema matches the desired software version.
If you have a good reason to believe that the mongoDB is already version 3.4, then you can update the C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\ASMongoDBVersion.bin as noted above, then make sure the RuntimeSettings.xml/System Settings points to the correct mongoDB, and finally restart the service.
No migration is needed, and assuming the rest of the upgrade was successful, it's fine to ignore the "failed migration" message. If you have any doubts, you can run MongoDBUpgradeto30.exe again, and see that it now errors with the message that the mongoDB has already been migrated. Restart the AlteryxService, and run as usual.
