Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Alteryx Server Knowledge Base

Definitive answers from Server experts.

Troubleshooting a failed MongoDB migration - Server 2019.3

SophiaF
Alteryx
Alteryx
Created

Issue

Asdocumented in the Server release notes (Server 2019.3), Alteryx Server version 2019.3 includes a new version of the embeddedMongoDB (MongoDB functions as the server'spersistence layer). If you are upgrading your Alteryx Server from 2018.1 or greater*, your MongoDB will be migrated from Mongo 3.4 to Mongo 4.0 as part of the process.

*If upgrading from pre-2018.1, you will need to upgrade to a version between 2018.1-2019.2 before upgrading to 2019.3.

Some usershave reported the following error after the migration :

Inconsistencies between the original data and the new data may indicate a migration error. Check the C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0\migration.log for any errors and contact Alteryx Customer Support for assistance. 

migration mongo.png

While the server may still run on the MongoDB 3.4, this may cause unexpected behavior in the long term. Should you encounter this error, it is important to troubleshoot it and make sure that your MongoDB is able to successfully migrate.

Environment

  • Alteryx Server
    • Upgrading from a version 2018.1 to version 2019.3

Diagnosis

If this error is encountered, the server itself will upgrade, but fail to start the service.

To determine which troubleshooting steps to take, we first need to check if the migration was actually successful or not :

  • Confirm there a MongoDB_4_0 folder underthe path in the Alteryx System Settingsunder Controller >Persistence > Data Folder.
    • If there is a MongoDB_4_0 folder, check the migration.log file found under the Data Folder path.Check to see if it matches the successful migration shown here in the Spoiler:

Spoiler (Highlight to read)

A "successful" migration should have the following:

First, you see the original 3.4 database start with the 3.4 executable. Then,getstats.js is run - it will showa list of collections and a count of items in them.

--Starting: C:\Program Files\Alteryx\bin\mongod3_4.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB" --port 27018
--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"
{"databases": [ {"name": "AlteryxService", "collections": [ {"name": "AS_App_Chunks", "count": 290} , {"name": "AS_App_Chunks.Files", "count": 27}
...


Next, Mongo will dump out to the PreUpgrade folder specified in the migration utility. Once the backup completes the MongoDB service is stopped with the shutdown.js. You will seeMongoDB is not responding because it was shut down successfully -any errors referencing the server being down, or the connection being forcibly closed is expected.

--Starting: mongodump3_4.exe --host localhost --port 27018 --username user --password xxxxxxxxxxxxxxxxxxxxxxxxxxx --out "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
2019-09-04T11:21:45.557+0100 writing admin.system.users to
2019-09-04T11:21:45.559+0100 done dumping admin.system.users (4 documents)
2019-09-04T11:21:45.559+0100 writing admin.system.version to
2019-09-04T11:21:45.560+0100 done dumping admin.system.version (2 documents)
...

--Stopping MongoDb
--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"
...

2019-09-04T11:21:48.413+0100 I NETWORK [thread1] Socket recv() An existing connection was forcibly closed by the remote host. 127.0.0.1:27018
2019-09-04T11:21:48.413+0100 I NETWORK [thread1] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27018]
server should be down...


Then Mongo is started again, and restored to the new MongoDB_4_0 directory from the previous backup that was just taken.Again, we should not see any errors in here.
Then, the createUsers.js re-creates all the user accounts for each database.
The MongoDB is then shut down via theshutdown.js again. Again -any errors referencing the server being down, or the connection being forcibly closed is expected.

--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongorestore.exe --host localhost --port 27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
... {restore happens}
2019-09-04T11:21:59.059+0100 done

--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\createUsers.js"
MongoDB shell version v4.0.10
connecting to: mongodb://localhost:27018/test?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b355d9b1-d9b0-4b6f-816f-158da213dcaa") }
MongoDB server version: 4.0.10
Successfully added user: {
"user" : "user",
"roles" : [
"dbOwner",
"hostManager",
"dbAdminAnyDatabase",
"userAdminAnyDatabase",
"backup",
"restore"
]
}
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }

--Stopping MongoDb
--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"

...

server should be down...
2019-09-04T11:22:01.726+0100 I NETWORK [js] trying reconnect to localhost:27018 failed

2019-09-04T11:22:02.728+0100 I QUERY [js] Failed to end session { id: UUID("129ef6b5-c6a8-40d0-8801-5987cb980e00") } due to SocketException: socket exception [CONNECT_ERROR] server [couldn't connect to server localhost:27018, connection attempt failed: SocketException: Error connecting to localhost:27018 (127.0.0.1:27018) :: caused by :: No connection could be made because the target machine actively refused it.]

The new MongoDB_4_0 is then started up again with Mongo 4.0, and we run getstats.js again, giving us a list of the collections and the count of the documents, similar to the first step.

--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongo.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"



If everything looks good up to this point, the migration has likely been successful - continue to check the points below. If not, please see Solution B.

A "successful" migration should have the following:First, you see the original 3.4 database start with the 3.4 executable. Then,getstats.js is run - it will showa list of collections and a count of items in them. --Starting: C:\Program Files\Alteryx\bin\mongod3_4.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB" --port 27018--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js" {"databases": [ {"name": "AlteryxService", "collections": [ {"name": "AS_App_Chunks", "count": 290} , {"name": "AS_App_Chunks.Files", "count": 27}... Next, Mongo will dump out to the PreUpgrade folder specified in the migration utility. Once the backup completes the MongoDB service is stopped with the shutdown.js. You will seeMongoDB is not responding because it was shut down successfully -any errors referencing the server being down, or the connection being forcibly closed is expected. --Starting: mongodump3_4.exe --host localhost --port 27018 --username user --password xxxxxxxxxxxxxxxxxxxxxxxxxxx --out "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"2019-09-04T11:21:45.557+0100 writing admin.system.users to 2019-09-04T11:21:45.559+0100 done dumping admin.system.users (4 documents)2019-09-04T11:21:45.559+0100 writing admin.system.version to 2019-09-04T11:21:45.560+0100 done dumping admin.system.version (2 documents)...--Stopping MongoDb--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"...2019-09-04T11:21:48.413+0100 I NETWORK [thread1] Socket recv() An existing connection was forcibly closed by the remote host. 127.0.0.1:27018 2019-09-04T11:21:48.413+0100 I NETWORK [thread1] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27018] server should be down... Then Mongo is started again, and restored to the new MongoDB_4_0 directory from the previous backup that was just taken.Again, we should not see any errors in here.Then, the createUsers.js re-creates all the user accounts for each database.The MongoDB is then shut down via theshutdown.js again. Again -any errors referencing the server being down, or the connection being forcibly closed is expected. --Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongorestore.exe --host localhost --port 27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"... {restore happens}2019-09-04T11:21:59.059+0100 done--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\createUsers.js"MongoDB shell version v4.0.10connecting to: mongodb://localhost:27018/test?gssapiServiceName=mongodbImplicit session: session { "id" : UUID("b355d9b1-d9b0-4b6f-816f-158da213dcaa") }MongoDB server version: 4.0.10Successfully added user: {"user" : "user","roles" : ["dbOwner","hostManager","dbAdminAnyDatabase","userAdminAnyDatabase","backup","restore"]}Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }--Stopping MongoDb--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"...server should be down...2019-09-04T11:22:01.726+0100 I NETWORK [js] trying reconnect to localhost:27018 failed2019-09-04T11:22:02.728+0100 I QUERY [js] Failed to end session { id: UUID("129ef6b5-c6a8-40d0-8801-5987cb980e00") } due to SocketException: socket exception [CONNECT_ERROR] server [couldn't connect to server localhost:27018, connection attempt failed: SocketException: Error connecting to localhost:27018 (127.0.0.1:27018) :: caused by :: No connection could be made because the target machine actively refused it.] The new MongoDB_4_0 is then started up again with Mongo 4.0, and we run getstats.js again, giving us a list of the collections and the count of the documents, similar to the first step. --Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongo.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js" If everything looks good up to this point, the migration has likely been successful - continue to check the points below. If not, please see Solution B.
  • Are the Alteryx System Settings under Controller >Persistence > Data Folderpointing to the new MongoDB folder (MongoDB_4_0)?

If everything looks good in the logs, but the service didn't start or something happened after this point, we should be able to manually change the persistence directory to point at the new MongoDB_4_0 directory, and in theory, the service should start up fine. See Solution A for steps.

  • What version of Mongo is running after the attempt?

In the persistence directory of the target database, there's an ASMongoDBVersion.binfile that seems to match up with the version of Mongo. If this does not show version 4.0, see Solution B.

Solution A

If the log contains all the information mentioned previously and the migration seems successful, but there’s an error starting the service:

  1. Check that the Data Folder underController > Persistence is set to theMongoDB_4_0 folder. If is it not, edit it to point to this new folder.
    systemsettings.png
  2. Restart the Alteryx Service.

Solution B

If the migration log doesn’t match the description above, give the migration another attempt:

1. Delete the folders MongoDB_PreUpgrade and MongoDB_4_0

2. Run the MongoDbUpgradeTo30.exe found here:

%ALTERYX_INSTALL_DIRECTORY%\bin\MongoDbUpgradeTo30.exe

mongoupgrade.png

If the MongoDB_4_0 folder is still empty with the exception of the migration.log, then check the MongoDB_PreUpgrade folder for the ASMongoDBVersion.bin file. If it is not there:

  1. Copy the file from your old MongoDB folder and paste it into the MongoDB_PreUpgrade folder.
  2. Open the file and change the version number to 4.0.10.
  3. Close the file and re-attempt the upgrade.

As always, don't hesitate to contact us over atCustomer Supportif you run into any trouble.

Comments
varunvarma87
8 - Asteroid

Had the mentioned error pop-up. Solution-A worked ! Thank you.

Toby_work
6 - Meteoroid
This makes sense for the Controller but what do we do when it shows up on a Worker?
SophiaF
Alteryx
Alteryx

@Toby_work the process should only run if 1) System Settings are set to a Controller 2) and it points to a database. I would confirm in System Settings that you have only Worker checked and no database set. You may also want to confirm by checking the file that System Settings pulls from directly - %PROGRAMDATA%\Alteryx\RuntimeSettings.XML

Toby_work
6 - Meteoroid

The existing version was 2019.1.6. We upgraded to 2019.3. I recently became the admin and this is my first install/upgrade. One Controller (with the Gallery on the same server), one Worker. The Worker was (and still is) set up as so:

< oh you've got to be kidding, I can't paste an image? >

Well, the Setup Type is "Custom", "Enable Worker". Snippet from the Worker file D:\Program Files\Alteryx\bin\RuntimeData\RuntimeSettings.xml where I think you're talking about in the .xml file: < EmbeddedMongoDBArguments/ > < EmbeddedMongoDBDatabaseName >AlteryxService < EmbeddedMongoDBEnabled >False< /EmbeddedMongoDBEnabled > < EmbeddedMongoDBLogPath/ > < EmbeddedMongoDBPasswordEncrypted/ > < EmbeddedMongoDBPort >27018< /EmbeddedMongoDBPort > < EmbeddedMongoDBReplSetEnabled >False< /EmbeddedMongoDBReplSetEnabled > < EmbeddedMongoDBReplSetName >ASReplSet< /EmbeddedMongoDBReplSetName > < EmbeddedMongoDBRootPath /> < EmbeddedMongoDBServerName >localhost< /EmbeddedMongoDBServerName > < EmbeddedMongoDBUserName >user< /EmbeddedMongoDBUserName > < GalleryEnabled>False< /GalleryEnabled >

Toby_work
6 - Meteoroid

Let's back up.  I'm a recent admin so I'm not extremely familiar with the server software.  The existing version was 2019.1 and we upgraded to 2019.3.  This is in a test environment.  There is one Controller with a Gallery; there is one Worker.  This is the Worker:

 

< Multiple attempts to add an image failed >

 

So the Setup Type is Custom and I only have Enable Worker set.

 

 

-<SystemSettings>
-<Controller>
<ControllerEnabled>False</ControllerEnabled>
<EmbeddedMongoDBEnabled>True</EmbeddedMongoDBEnabled>
<EmbeddedMongoDBRootPath>D:\ProgramData\Alteryx\Service\Persistence\MongoDB</EmbeddedMongoDBRootPath>
<InsightServiceEnabled>True</InsightServiceEnabled>
<LoggingEnabled>True</LoggingEnabled>
<LoggingPath>D:\ProgramData\Alteryx\Service\AlteryxServiceLog.log</LoggingPath>
<MapTileServiceEnabled>False</MapTileServiceEnabled>
<ServerSecretEncrypted>-</ServerSecretEncrypted>
<SQLitePath>D:\ProgramData\Alteryx\Service\Persistence</SQLitePath>
<WebInterfaceEnabled>True</WebInterfaceEnabled>
<WebInterfaceStagingPath>D:\ProgramData\Alteryx\Service\Staging</WebInterfaceStagingPath>
</Controller>

 

 

Controller:

 

 

-<SystemSettings>
-<Authentication>
<ServiceProviderEntityID><a href="<a href="<a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a>" target="_blank"><a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a>>" target="_blank"><a href="<a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a>" target="_blank"><a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a</a>>>>
</Authentication>
-<Controller>
<CleanupQueueInputsTimeToLive>43200</CleanupQueueInputsTimeToLive>
<CleanupQueueTimeToLive>10080</CleanupQueueTimeToLive>
<CleanupSchedulerTimeToLive>20160</CleanupSchedulerTimeToLive>
<EmbeddedMongoDBEnabled>True</EmbeddedMongoDBEnabled>
<EmbeddedMongoDBRootPath>E:\ProgramData\Alteryx\Service\Persistence\MongoDB_34_4_0</EmbeddedMongoDBRootPath>
<GalleryEnabled>True</GalleryEnabled>
<InsightServiceEnabled>True</InsightServiceEnabled>
<LoggingEnabled>True</LoggingEnabled>
<LoggingPath>E:\ProgramData\Alteryx\Service\AlteryxServiceLog.log</LoggingPath>
<LoggingSeverityThreshold>7</LoggingSeverityThreshold>
<ServerSecretEncrypted>--</ServerSecretEncrypted>
<SQLitePath>D:\ProgramData\Alteryx\Service\Persistence</SQLitePath>
<StorageKeysEncrypted>--</StorageKeysEncrypted>
<WebInterfaceEnabled>True</WebInterfaceEnabled>
<WebInterfaceStagingPath>E:\ProgramData\Alteryx\Service\Staging</WebInterfaceStagingPath>
</Controller>

 

 

 Given this I'm confident the database is NOT being referenced by the Worker.

Garabujo7
Alteryx
Alteryx

Great article @SophiaF, very useful just saved my life and in the easiest and comprehensive way possible.