09-04-2019 10:19 AM - edited 08-03-2021 02:05 PM
Issue
Asdocumented in the Server release notes (Server 2019.3), Alteryx Server version 2019.3 includes a new version of the embeddedMongoDB (MongoDB functions as the server'spersistence layer). If you are upgrading your Alteryx Server from 2018.1 or greater*, your MongoDB will be migrated from Mongo 3.4 to Mongo 4.0 as part of the process.
*If upgrading from pre-2018.1, you will need to upgrade to a version between 2018.1-2019.2 before upgrading to 2019.3.
Some usershave reported the following error after the migration :
Inconsistencies between the original data and the new data may indicate a migration error. Check the C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0\migration.log for any errors and contact Alteryx Customer Support for assistance.
While the server may still run on the MongoDB 3.4, this may cause unexpected behavior in the long term. Should you encounter this error, it is important to troubleshoot it and make sure that your MongoDB is able to successfully migrate.
Environment
Diagnosis
If this error is encountered, the server itself will upgrade, but fail to start the service.
To determine which troubleshooting steps to take, we first need to check if the migration was actually successful or not :
A "successful" migration should have the following:
First, you see the original 3.4 database start with the 3.4 executable. Then,getstats.js is run - it will showa list of collections and a count of items in them.
--Starting: C:\Program Files\Alteryx\bin\mongod3_4.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB" --port 27018
--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"
{"databases": [ {"name": "AlteryxService", "collections": [ {"name": "AS_App_Chunks", "count": 290} , {"name": "AS_App_Chunks.Files", "count": 27}
...
Next, Mongo will dump out to the PreUpgrade folder specified in the migration utility. Once the backup completes the MongoDB service is stopped with the shutdown.js. You will seeMongoDB is not responding because it was shut down successfully -any errors referencing the server being down, or the connection being forcibly closed is expected.
--Starting: mongodump3_4.exe --host localhost --port 27018 --username user --password xxxxxxxxxxxxxxxxxxxxxxxxxxx --out "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
2019-09-04T11:21:45.557+0100 writing admin.system.users to
2019-09-04T11:21:45.559+0100 done dumping admin.system.users (4 documents)
2019-09-04T11:21:45.559+0100 writing admin.system.version to
2019-09-04T11:21:45.560+0100 done dumping admin.system.version (2 documents)
...
--Stopping MongoDb
--Starting: mongo3_4.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"
...
2019-09-04T11:21:48.413+0100 I NETWORK [thread1] Socket recv() An existing connection was forcibly closed by the remote host. 127.0.0.1:27018
2019-09-04T11:21:48.413+0100 I NETWORK [thread1] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27018]
server should be down...
Then Mongo is started again, and restored to the new MongoDB_4_0 directory from the previous backup that was just taken.Again, we should not see any errors in here.
Then, the createUsers.js re-creates all the user accounts for each database.
The MongoDB is then shut down via theshutdown.js again. Again -any errors referencing the server being down, or the connection being forcibly closed is expected.
--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongorestore.exe --host localhost --port 27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_PreUpgrade"
... {restore happens}
2019-09-04T11:21:59.059+0100 done
--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_34\createUsers.js"
MongoDB shell version v4.0.10
connecting to: mongodb://localhost:27018/test?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b355d9b1-d9b0-4b6f-816f-158da213dcaa") }
MongoDB server version: 4.0.10
Successfully added user: {
"user" : "user",
"roles" : [
"dbOwner",
"hostManager",
"dbAdminAnyDatabase",
"userAdminAnyDatabase",
"backup",
"restore"
]
}
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }
Successfully added user: { "user" : "user", "roles" : [ "readWrite", "userAdmin" ] }
--Stopping MongoDb
--Starting: mongo.exe localhost:27019 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\shutdown.js"
...
server should be down...
2019-09-04T11:22:01.726+0100 I NETWORK [js] trying reconnect to localhost:27018 failed
2019-09-04T11:22:02.728+0100 I QUERY [js] Failed to end session { id: UUID("129ef6b5-c6a8-40d0-8801-5987cb980e00") } due to SocketException: socket exception [CONNECT_ERROR] server [couldn't connect to server localhost:27018, connection attempt failed: SocketException: Error connecting to localhost:27018 (127.0.0.1:27018) :: caused by :: No connection could be made because the target machine actively refused it.]
The new MongoDB_4_0 is then started up again with Mongo 4.0, and we run getstats.js again, giving us a list of the collections and the count of the documents, similar to the first step.
--Starting: C:\Program Files\Alteryx\bin\mongod.exe --dbpath "C:\ProgramData\Alteryx\Service\Persistence\MongoDB_4_0" --port 27018 --Starting: mongo.exe localhost:27018 "C:\ProgramData\Alteryx\Service\Persistence\MongoDB\getstats.js"
If everything looks good up to this point, the migration has likely been successful - continue to check the points below. If not, please see Solution B.
If everything looks good in the logs, but the service didn't start or something happened after this point, we should be able to manually change the persistence directory to point at the new MongoDB_4_0 directory, and in theory, the service should start up fine. See Solution A for steps.
In the persistence directory of the target database, there's an ASMongoDBVersion.binfile that seems to match up with the version of Mongo. If this does not show version 4.0, see Solution B.
Solution A
If the log contains all the information mentioned previously and the migration seems successful, but there’s an error starting the service:
Solution B
If the migration log doesn’t match the description above, give the migration another attempt:
1. Delete the folders MongoDB_PreUpgrade and MongoDB_4_0
2. Run the MongoDbUpgradeTo30.exe found here:
%ALTERYX_INSTALL_DIRECTORY%\bin\MongoDbUpgradeTo30.exe
If the MongoDB_4_0 folder is still empty with the exception of the migration.log, then check the MongoDB_PreUpgrade folder for the ASMongoDBVersion.bin file. If it is not there:
As always, don't hesitate to contact us over atCustomer Supportif you run into any trouble.
Had the mentioned error pop-up. Solution-A worked ! Thank you.
@Toby_work the process should only run if 1) System Settings are set to a Controller 2) and it points to a database. I would confirm in System Settings that you have only Worker checked and no database set. You may also want to confirm by checking the file that System Settings pulls from directly - %PROGRAMDATA%\Alteryx\RuntimeSettings.XML
The existing version was 2019.1.6. We upgraded to 2019.3. I recently became the admin and this is my first install/upgrade. One Controller (with the Gallery on the same server), one Worker. The Worker was (and still is) set up as so:
< oh you've got to be kidding, I can't paste an image? >
Well, the Setup Type is "Custom", "Enable Worker". Snippet from the Worker file D:\Program Files\Alteryx\bin\RuntimeData\RuntimeSettings.xml where I think you're talking about in the .xml file: < EmbeddedMongoDBArguments/ > < EmbeddedMongoDBDatabaseName >AlteryxService < EmbeddedMongoDBEnabled >False< /EmbeddedMongoDBEnabled > < EmbeddedMongoDBLogPath/ > < EmbeddedMongoDBPasswordEncrypted/ > < EmbeddedMongoDBPort >27018< /EmbeddedMongoDBPort > < EmbeddedMongoDBReplSetEnabled >False< /EmbeddedMongoDBReplSetEnabled > < EmbeddedMongoDBReplSetName >ASReplSet< /EmbeddedMongoDBReplSetName > < EmbeddedMongoDBRootPath /> < EmbeddedMongoDBServerName >localhost< /EmbeddedMongoDBServerName > < EmbeddedMongoDBUserName >user< /EmbeddedMongoDBUserName > < GalleryEnabled>False< /GalleryEnabled >
Let's back up. I'm a recent admin so I'm not extremely familiar with the server software. The existing version was 2019.1 and we upgraded to 2019.3. This is in a test environment. There is one Controller with a Gallery; there is one Worker. This is the Worker:
< Multiple attempts to add an image failed >
So the Setup Type is Custom and I only have Enable Worker set.
-<SystemSettings>
-<Controller>
<ControllerEnabled>False</ControllerEnabled>
<EmbeddedMongoDBEnabled>True</EmbeddedMongoDBEnabled>
<EmbeddedMongoDBRootPath>D:\ProgramData\Alteryx\Service\Persistence\MongoDB</EmbeddedMongoDBRootPath>
<InsightServiceEnabled>True</InsightServiceEnabled>
<LoggingEnabled>True</LoggingEnabled>
<LoggingPath>D:\ProgramData\Alteryx\Service\AlteryxServiceLog.log</LoggingPath>
<MapTileServiceEnabled>False</MapTileServiceEnabled>
<ServerSecretEncrypted>-</ServerSecretEncrypted>
<SQLitePath>D:\ProgramData\Alteryx\Service\Persistence</SQLitePath>
<WebInterfaceEnabled>True</WebInterfaceEnabled>
<WebInterfaceStagingPath>D:\ProgramData\Alteryx\Service\Staging</WebInterfaceStagingPath>
</Controller>
Controller:
-<SystemSettings>
-<Authentication>
<ServiceProviderEntityID><a href="<a href="<a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a>" target="_blank"><a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a>>" target="_blank"><a href="<a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a>" target="_blank"><a href="http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a" target="_blank">http://ourCorpLocation.com/aas</ServiceProviderEntityID</a</a</a>>>>
</Authentication>
-<Controller>
<CleanupQueueInputsTimeToLive>43200</CleanupQueueInputsTimeToLive>
<CleanupQueueTimeToLive>10080</CleanupQueueTimeToLive>
<CleanupSchedulerTimeToLive>20160</CleanupSchedulerTimeToLive>
<EmbeddedMongoDBEnabled>True</EmbeddedMongoDBEnabled>
<EmbeddedMongoDBRootPath>E:\ProgramData\Alteryx\Service\Persistence\MongoDB_34_4_0</EmbeddedMongoDBRootPath>
<GalleryEnabled>True</GalleryEnabled>
<InsightServiceEnabled>True</InsightServiceEnabled>
<LoggingEnabled>True</LoggingEnabled>
<LoggingPath>E:\ProgramData\Alteryx\Service\AlteryxServiceLog.log</LoggingPath>
<LoggingSeverityThreshold>7</LoggingSeverityThreshold>
<ServerSecretEncrypted>--</ServerSecretEncrypted>
<SQLitePath>D:\ProgramData\Alteryx\Service\Persistence</SQLitePath>
<StorageKeysEncrypted>--</StorageKeysEncrypted>
<WebInterfaceEnabled>True</WebInterfaceEnabled>
<WebInterfaceStagingPath>E:\ProgramData\Alteryx\Service\Staging</WebInterfaceStagingPath>
</Controller>
Given this I'm confident the database is NOT being referenced by the Worker.
Great article @SophiaF, very useful just saved my life and in the easiest and comprehensive way possible.