community
cancel
Showing results for 
Search instead for 
Did you mean: 

Engine Works Blog

Under the hood of Alteryx: tips, tricks and how-to's.
Alteryx
Alteryx

Introduction

At the core of the Server is an executable called AlteryxService.exe (the “Service”), which acts as the Server's central nervous system.  Its duties range from serving module interfaces and results to the Gallery, to rendering map tiles for the Map Input Tool in Designer, to orchestrating module execution with the Alteryx Engine for the Scheduler and Gallery.

 

By default, the Alteryx Service is installed as Windows Service, a type of process that typically starts when a computer is booted and remains active until the machine is stopped.  These processes are managed by the Service Control Manager (SCM) in Windows, and common examples include Windows system processes, virus scanners, and drivers for “on demand” scanners/printers.

 

You can view the Windows Services currently installed on your system by opening Task manager (taskmgr.exe) and switching to the “Services” tab:

taskmgr_services.png

 

Although the Service is intended to run as a Windows Service, it also supports a rich command line interface which is exceedingly useful for automation, maintenance and troubleshooting.  The purpose of this post is to introduce the Service command line with concrete examples demonstrating a few of its more interesting capabilities.

 

Follow-up posts will explore some of these capabilities in much greater detail.  But, for now, let’s set the stage and start with some basics.

 

A word of caution

As with any sharp tool, the Service command line must be used with great care.  When experimenting with the Service command line, work in a non-production (sandboxed) environment if possible, and take a few minutes to back up the system before making changes.

 

The Service command line

To access the Service command line, open a command shell (cmd.exe) and navigate to the Alteryx installation directory (by default, c:\Program Files\Alteryx\bin).  Since most of the options require an Administrator-level login, when using the Service command line you’ll generally want to start the command shell as an Administrator.

 

To see the available command line options, type ‘AlteryxService /?’ at the command prompt:

sl_command_line_options.png

Note that a default shell width of 130 columns is encouraged when using the Service command line

 

This post deliberately glosses over the self-explanatory command line options, but we’ll explore a few of the more interesting commands, including scheduling from the command line, Mongo dump/restore, and the service “test” mode.

 

Scheduling from the command line

One of my favorite options on the Service command line is addtoqueue, which schedules a module for immediate execution.  The syntax of the command is straightforward, requiring the path to the module to be queued, the server (Controller) on which to schedule the module, and that Controller’s server secret:

addtoqueue=path,host[:port],secret

 

Functionally, this command is equivalent to scheduling a module from Designer using the “Run the workflow from its original location…” option selected:

schedule_as_local_module.png

 

 

 

 

 

 

 

But, what’s neat about this command line flag is it enables automation (the secret to the universe), and can be used as a quick command line smoke-test for a new Server installation or at the core of more sophisticated scripts.  As a concrete example, consider the benchmarking app introduced in a previous post.  That app uses a Run Command tool in Designer to call a batch (bat) script, which, in turn, queues a collection of modules to a particular Server instance using AlteryxService.exe addtoqueue:

 

:start_scheduling
set /a numIterations=numIterations-1
for %%m in (%workflowDir%\*.yxmd) do (
echo scheduling %%m
start /wait /d %alteryxBinPath% AlteryxService.exe addtoqueue="%%m",%server%,%serverSecret%
)

 

To use this command on your local system, fire up a command shell (preferably as an Administrator), and locate the module you want to schedule.  You'll need the security token for the Server you're scheduling to, or if the target Controller is localhost, you can collect the token on the command line using the getsecret flag as shown here:

sl_addtoqueue.png

 

After scheduling the module on the command line, the module will be placed into the execution queue on that Controller, and processed by the next available worker.

 

Backing up and restoring Embedded Mongo

By default, the Service uses an “Embedded” Mongo instance for its data storage, which is a Mongo instance that the Service bootstraps on first run and uses thereafter.  We encourage users to back up this Mongo instance regularly, and we provide a command line option on the Service to make it easier.

 

Using the emongodump option on the Service, we can safely dump the current Embedded Mongo database to a target folder on the filesystem.  Before we can do so, however, we must stop the Service running as a System process.

 

We can stop the Service either from the command line, or via the task manager.  To stop from the command line, open an Administrator command shell and ask the Service Control Manager to stop the service using sc stop AlteryxService:

sc_stop_service.png

sc start, sc stop and sc query are useful commands when working with system services

 

Or, from the task manager, right-click on the AlteryxService service and choose “Stop”:

taskmgr_stop_service.png

 

Remember, stopping the AlteryxService means stopping the Server itself, so please tred lightly to avoid interrupting users.

 

emongodump

After stopping the Service, we can use the emongodump command to export the Mongo database.  From the Administrator command shell, navigate to the Alteryx installation directory and execute the emongodump command, which requires the full path to an empty folder:

emongodump=TargetDirectory

 

Depending on the size of your database, this command may take a few seconds to a few minutes to complete:

sl_emongodump.png

If the path you're writing to contains spaces, remember to wrap the path in quotations on the command line

 

emongorestore

In the [unlikely] event of an unrecoverable Mongo failure, you can restore the Mongo database from a backup using the emongorestore command.  This command requires the path to a folder containing a Mongo dump, and a path to which it should be restored:

emongorestore=SourcePath,TargetPath

 

In this example, we’ll restore from the backup we just created to the default Service persistence folder:

sl_emongorestore.png

 

Note that the emongorestore command must be used with great caution as it will overwrite the contents of the target folder!

 

Service Test mode

Instead of executing in the background as a system service, the Service can run as an interactive, console-style application.  In this "test" mode, the Service runs as the current Windows user (typically the Administrator), and supports exactly the same functionality as a Service running as a System Service.  This mode is typically used for testing and troubleshooting, details of which we’ll cover in future blog posts.

 

As with the Embedded Mongo dump and restore (emongodump and emongorestore), we must ask the Service Control Manager to stop the Service prior to starting a new instance in test mode.  After the system-level instance of the Service is stopped, start an interactive instance of the Service by navigating to your Alteryx installation directory in a command shell and adding the test flag to the Service command line:

sl_test_mode.png

The console shows the "logical services" the Alteryx Service starts to support the current configuration

 

Note that, depending on how your Server is configured, you may see two other console windows appear, one for the Mongo daemon and one for the Gallery.  To exit the Alteryx Service running in test mode, activate the Service console window and press any key on the keyboard.

 

Note that closing the Gallery or Mongo window will not terminate the Service, and serves as a demonstration of some of the built-in resliency mechanisms in the Service itself.

 

Closing

This brief overview of the Service command line sets the stage for more advanced discussions about automation, setup and troubleshooting, so stay tuned for more Server-related posts.  Thank you kindly for reading and feel free to reach out with comments and questions.

Stephen Ahlgren

Steve is a principal developer at Alteryx on the Emerging Capabilities team, working largely in C++ but dabbling in other languages and technologies. His contributions include backend components of the Alteryx Server, Hadoop connectors (HDFS/Avro), JavaScript integration in both the Designer and Engine (CEF), and the new Spark Direct functionality.

Steve is a principal developer at Alteryx on the Emerging Capabilities team, working largely in C++ but dabbling in other languages and technologies. His contributions include backend components of the Alteryx Server, Hadoop connectors (HDFS/Avro), JavaScript integration in both the Designer and Engine (CEF), and the new Spark Direct functionality.

Comments

Hi Steve,

 

Regarding the scheduling of workflows using AlteryxService.exe, how do you set the schedule to be "In Scheduler DB" instead of "On disk"? It seems like by default it is an "On Disk" schedule.

Alteryx
Alteryx

Hi @josecavalheiro,

 

Thanks for the note.  And yes, you are correct that that addtoqueue will run the module on disk instead of adding to the DB.  We've improved this functionality in Alteryx 10.5 to also allow the module to be persisted in the DB, which also means the command now correctly support modules *(.yxmd), apps (*.yxwz) and app packages (*.yxzp).

 

In 10.5, the addtoqueue command line flag now looks like this:

sl_command_line.png

Adding that final "true" flag will result in the module/app to be persisted in the DB, and leaving it absent or setting to "false" will result in the run-from-disk behavior as in previous Alteryx releases.

 

The improved functionality also makes multi-node Server benchmarking much easier as it expands the breadth of modules/apps that can be used in the benchmark and also eliminates the need to copy the modules to each worker before the test.  Concretely, the core of the benchmark script now iterates over all *.yxmd, *.yxwz and *.yxzp, adding each to the DB instead of running from disk:

for %%m in (%workflowDir%\*.yxmd %workflowDir%\*.yxwz %workflowDir%\*.yxzp) do (
echo scheduling %%m
start /wait /d %alteryxBinPath% AlteryxService.exe addtoqueue="%%m",%server%,%serverSecret%,true

 

Best,

Steve A

 

 

Asteroid

 I created a session where I stopped the service and went to restart the service but the command prompt window says a seesion has already started.  I check the Services window and it's saying the service is stopped.  I then restart if from the services window and I'm on my way.  Thing is I want to automate t so I don't have to restart the service manually.  Do I need to have a timer in my batch file to give the service time to shut down before I restart?  Any help is appreciated.

 

Brad

Asteroid

Umm... nevermind.  I put in a TIMEOUT \T 300 \NOBREAK and that did the trick.

Alteryx
Alteryx

Hi @brad_j_crep,

 

 

Adding the explicit "timeout \t" is certainly a workaround but will result in a guaranteed sleep state, which will slow down throughput.  I expect you could turn that passive sleep into an "active" sleep by looping until a query to "sc status AlteryxService" shows that it is, indeed, completely stopped.

 

Another option is to use "net stop AlteryxService" instead of "sc stop AlteryxService" as the former is synchronous and the latter is asynchronous.

 

Best,

Steve

Alteryx
Alteryx

Hi Steve,

Can you optionally supply wizard parameters when adding to queue via

 

addtoqueue=path,host[:port],secret

 

Thanks!

Alteryx
Alteryx

Hey @DanH,

 

That's a great idea and I encourage you to add it to the product suggestions area in Community.  Tagging @MattB so he's aware of the idea.

 

One workaround suggestion might be to use a chained app with the first setting up parameters used by the second?  That way it would be self-contained and could be executed via the current addtoqueue method.  Just a thought...

 

Best,

Steve

Meteor

I have a chained workflow that is calling another packaged workflow using the AlteryxService command line option.  That is:

 

C:\Program Files\Alteryx\bin\AlteryxService.exe

addtoqueue="\\...\SCM Visit List.yxzp",<My Server Name>,<My Server Token>,true

 

This works great.  However, it uploads a new workflow to the scheduler database each time it's called.  This is a workflow I'd like to schedule to run every hour and so it will result in a huge number of duplicate 'Workflows' being created within the scheduler database (e.g., see attached screen).  Is there any way to run a packaged workflow from the command line that's already in the scheduler DB?  

 

Thank you for any guidance. 

 

AlteryxScheduledWorkflows.png

Alteryx
Alteryx

@klonergan Thanks for checking in...

 

It looks like your YXZP is on UNC path, if that's the case then perhaps it would work for you to continue using addtoqueue, but instead of adding the workflow to the DB just choose to run in place?  So, instead of passing "true" as the final command-line flag you would pass "false" to run in place.

 

Tagging @MattH so he's aware of the conversation.

Meteor

Hi @SteveA

Thanks for your reply.  It seems you can't run a packaged workflow from disk.  That is, you get the following error if the flag is set to 'false':

"Add to queue failed:
Workflow package (yxzp) must be added to database, unable to run from disk"

 

 

Alteryx
Alteryx

@klonergan Of course you are correct... I forgot about the yxzp restriction!

 

So a couple of concrete suggestions then:

1. Could you go to a standard chained app execution where the second app in the chain is an actual app instead of running the Service command line?

2. -or- could you unpack the resources from that yxzp and have them available to the module contained within the yxzp?  So, for example, if there's macros or data in the package you could have them reside in a folder alongside the module itself.  That way, when you run the module in place via addtoqueue, I believe the dependencies will be found.

Meteor

Thanks Steve.  I really appreciate the quick and helpful responses within this community. 

Following your first suggestion, I can convert the workflows to apps and then use the On Success - Run Another Analytic App option.  That works when run in desktop, but the second app won't trigger when I schedule the first app on server (either when run from disk or imported into scheduling database).  We have just a scheduling license and not full-blown Gallery etc.

The second option also works, but the original issue I was asking about occurs with this as well - it creates a new record in the 'Workflows' tab every time the workflow is run (i.e., every hour) rather than just queuing an existing workflow.   

I wanted to avoid this as it's messy, but I can combine these together to avoid chaining in this case ... 

Hi @SteveA,

 

This is a great post which has been super helpful on plenty of occasions! 

 

I am using the "addtoqueue" service which has made the scheduling a lot more dynamic and flexible (even solved an issue I had regarding running workflows stored in a network drive). 

 

The issue I am seeing is that the controller is not assigning any of these tasks to the worker nodes which is creating a queue on my controller. Is this expected behavior? Is there an easy way to get around this ?

 

Best,

dK

^^ Disregard. Services for workers were not running at that time. 

 

What is interesting is when the queue runs workflows on the shared network drive on the controller, it runs fine. But when it designates to the worker node, the worker can not find the workflow in the network drive (says the Service log). 

Alteryx
Alteryx

@The_Dev_Kev_Env Thank you for the note, glad you enjoyed the post!

 

If you're running jobs from a UNC path that will work as long as all workers have access to that path when logged in as your Run-As user.  If you aren't using a Run-As user, that's fine too, but in that case the system account needs access to the UNC path.

 

Good luck!

Hi Steve,

 

Thanks for the response. We were able to resolve the issue. The "Run As" system setting is used when the scheduler finds the workflow and has the engine run it. But the scheduler uses by default the system account to go out and find the workflow to be run. We had to go into the services and change the properties of the Alteryx Service to use the specific account that had access to the network drive.

 

Best,

Kevin

Alteryx Partner

Hi @SteveA,

I have been trying to call an alteryx workflow. I have been using the addtoqueue=path,localhost,getsecret. It keeps giving me an html error saying Add to queue failed: Server Error: 503 service unavailable. Are there things that can be checked that might cause this problem? I have already tried scheduling the same workflow from the designer and it worked there. I am also running the command prompt as administrator.

 

Thanks,

Skyler

Alteryx
Alteryx

Hi @sdavis

 

Just a hunch, but if you're getting a 503 plus an html response I'm going to guess you're connecting to your Gallery machine instead of the AlteryxService directly?  Or perhaps another web process (such as an IIS-hosted server)?

 

To test, paste this in your browser:

http://localhost/AlteryxService/status

 

Do you get a response?  If not, perhaps your Server is running on a different machine?

Alteryx Partner

Hi @SteveA

When putting the url into the browser it returns an html page saying service unavailable. The service is also showed as running in the task manager.

Alteryx
Alteryx

@sdavis Then perhaps your Server is running on a different port.  If you have access to the file "C:\ProgramData\Alteryx\RuntimeSettings.xml", look for a tag under the "Controller" section (top of the file) called "ServicePort".

 

 

Then you should be able to use the URL:

http://localhost:<service_port>/AlteryxService/Status

 

And if you get a response from that, update your addtoqueue command line accordingly with localhost:<service_port>.

 

Best,

Steve A

Alteryx Partner

hi @SteveA,

It is still having the problem and giving the same error.

Alteryx
Alteryx

@sdavis

 

 It sounds like you may be addressing the wrong machine? Or perhaps the machine you are targeting isn't actually running the AlteryxService.

 

To verify that the Service is running, open a browser and point to the machine :

http://<machine_name>/AlteryxService/status

 

Tagging @MattH

Meteoroid

Hello,

Having some trouble with the addtoqueue command.

I can successfully execute a workflow (located on a network drive) to run from the Alteryx DB using addtoqueue=path,host,secret,True

However, cannot get the same workflow to run from it's location on disk by simply using addtoqueue=path,host,secret

While this second command works when explicitly executed from the Alteryx controller machine, I cannot get it to run from my local machine. I also tried scheduling the workflow via the controller (from my local machine) using the following configuration, but alas no joy.

Picture1.png


It seems to be a permission issue, but I'm not sure what configuration changes are required to get the 2nd command to work from a non-controller machine.

Note: The controller has been set to "Run As" an Alteryx Service account, which also has read/write access to the network drive referenced in the workflow.

Thanks,
Ben

Tagging in @SteveA and @MattH

Meteoroid

Hello.

I'm having some trouble with the addtoqueue command.

While I can run a workflow on the Alteryx DB using addtoqueue=path,host,secret,True
I cannot get it to run from it's location on a network drive using addtoqueue=path,host,secret

We tried running the 2nd command directly from the Alteryx Controller machine, and it worked. However, it doesn't run from my local machine. I also tried running from my machine via the controller using the configuration below, but alas no joy.

Picture1.png

So this seems to be a permission issue, but I don't know what configuration changes are required to get the 2nd command to run from a non-controller machine.

Note: The controller is set to "Run As" an Alteryx Tech account, which has been given access to the network drive referenced in the workflow.

Thanks,
Ben

Tagging in  @MattH &  @SteveA

Alteryx
Alteryx

@benjamesdavis -

 

The addtoqueue should work with a local path (d:\path\to\module.yxmd) or a UNC network path (\\machine\path\to\module.yxmd).

 

A few suggestions to verify:

1. If you are using an alias to a network mount, ensure that the alias is valid for the worker node(s) in your server setup.  For example, if you use a path z:\path\to\module.yxmd, the z: mount point on the worker(s) must be valid.

2. Regardless of the path, the resource must be readable by the service account, meaning whatever account is running the AlteryxService.exe process.  This is not to be confused with the Run-As account (which is the account under which the module will be executed).

3. Use quotes around the path if there are spaces as in "d:\path\to a module with spaces.yxmd".

 

Best,

Steve A

Meteoroid

Thanks for the reply @SteveA. We're not remapping the network drive to an alias, and we already have quotation marks around the path, so I think I can rule out 1 and 3.

How can you tell which account is running the AlteryxService.exe process? Would this be the main Controller account or would it be whatever machine the cmd prompt is run on?

Atom

thanks for a great post @stevea. Open questions here regarding addtoqueue pointing to published workflows.

 

How would i go about running workflow2 upon success of workflow1 while both exist in the gallery/mongoDB not a locally?

 

Example: I have 2 workflows. Before we had server we ran them manually using the list runner from Crew macro. Now that we have server, we want to publish these workflows to the server and utilise monitoring and logging from the server. When using the addtoqueue parameter it looks like (from the posts above) that I can only point to and actual yxmd file on a drive, and not point it to run a workflow i have published to my gallery? Is this correctly understood? How would i trigger a workflow that is published?

 

Im open for any input as i have searched for a solutoin for a long time. Thanks

Alteryx
Alteryx

@Henry_Dw Apologies for the delayed response...

 

The command line option addtoqueue is explicitly for loading a local (ie on disk) workflow into the job queue and executing immediately, either "in place" meaning where it lies on the filesystem, or from the database as any other job on the Server would be.

 

Form your problem description it sounds like you might benefit from using our Gallery APIs, which are higher-level APIs exposed by the Gallery for performing common workflow-related tasks such as queuing a workflow for execution (with optional runtime configuration) and retrieving its results.

 

The general help for the Gallery APIs is available on the Gallery itself at https://gallery.alteryx.com/api-docs/.  Note that if you are on a local server you can access the interactive APIs...

 

There are several Community threads on using the Gallery APIs, this one in particular discusses consuming the APIs from within an app itself which sounds like it might be up your alley https://gallery.alteryx.com/api-docs/.

 

Best,

Steve A

Atom

Thanks for your reply and great input @steveA

 

Just to be make sure I understand correctly. Is there no way for me as and end user, to schedule e.g. 3 workflows where the workflows only runs if the predecessor succeeds, without using the Gallery API?

 

It seems like a quite standard process, to chuck workflows up in logical bits instead of having one gigantic workflow, and running them in sequence. 

 

Would there be any other way, beside storing them locally on disk?

 

The reason I want them to be run from Mongo is so that we can have everything in one place (Mongo). Also, in regards to logging I don't know if there will be a difference in running workflows on disk or directly from mongo?

 

Lastly I'm not even sure I would be allowed to get cmd access as an end-user.

 

Thanks for all your input, really appreciated!

Alteryx
Alteryx

Hi again @Henry_Dw 

 

You may also be able to take advantage of Chained Apps to achieve your goal.  There's a good writeup available here on Community.

Alteryx Certified Partner

I am using AlteryxEngineCMD to execute workflows with app value XMLs. It doesn't seem like using

addtoqueue=path,host[:port],secret

will allow me to pass the app value XML as an argument. Can you please clarify this? If the addtoqueue method allows a parameter, how does one use it? If the addtoqueue method doesn't allow the parameter, will this be on the roadmap?

 

Some reference links on how I am using AlteryxEngineCMD:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Alteryx-Workflow-execution-from-Comman...

https://help.alteryx.com/10.1/index.htm#Command_Line.htm

Alteryx
Alteryx

@kksieski This article applies to the Alteryx Server (AlteryxService.exe) and not the standalone engine (AlteryxEngineCmd.exe) as you describe.

 

The article you linked has examples to run the engine command line... hope that helps!

 

Best,

Steve A

Meteoroid

Is is possible to define a worker with AddToQueue?

Alteryx
Alteryx

@CLindsay No it is not possible to preferentially route a job to a particular worker via the command line addtoqueue method.

Meteoroid

@SteveA Thanks for the response. 

Meteoroid

I want to be able to review outcomes of workflows that are being added to queue through the Alteryx Service. I have been using the engine logs but I have a problem with logs that are still being written to by the engine. I need to be able to determine if the module failed so I can retrigger it, but we have some workflows that can take hours to run. When this happens I cannot read in all the logs from the day because it will stop reading in at the log that is "being accessed by another user". 

 

I plan to add hundreds of workflows to queue a day, many of which have multiple external dependencies so I am using a scheduled workflow to sweep for these dependencies and release workflows to the queue when their respective predecessors or dependencies are complete. As mentioned above, I want to use the engine logs to determine the success of my daily automation but cannot do so when the logs are not complete. When a workflow is added to queue and then sent to a Worker, is there a way to actually see what module/workflow the worker engine is executing in real time? 

Alteryx
Alteryx

Hi @CLindsay -

 

There are several ways to achieve your goal, including iteratively parsing the server logs and tracking what jobs are in flight, using the public Gallery APIs, or accessing the underlying DB tables directly.


To access the DB tables directly, I would suggest starting with the app "MeasureServerPerformance" attached to this blog post https://community.alteryx.com/t5/Engine-Works-Blog/Measuring-and-Scaling-a-Private-Server/ba-p/8786.

 

If you open that app and start digging in, you'll find macros that extract information from mongo and monitor running jobs via their queue status (mongo_queue_extractor and mongo_queue_monitor, respectfully).  Hopefully these will be good starting points for your needs.

 

Best,

Steve A

Alteryx Certified Partner

Hi @SteveA ,

 

your message "

@kksieski This article applies to the Alteryx Server (AlteryxService.exe) and not the standalone engine (AlteryxEngineCmd.exe) as you describe.

 

The article you linked has examples to run the engine command line... hope that helps!

"

misses the point of my question. I use the AlteryxEngineCmd.exe and I know the examples and the article I linked. The reason I linked it is to highlight how it can take an XML parameter. The point of my question is "Can the AlteryxService.exe take an XML parameter?" Please reread my post.

Alteryx
Alteryx

@kksieski Apologies for the misunderstanding! The short answer is no, we don't yet support passing the runtime config XML with the addtoqueue command line.

 

Tagging @TanyaS so she is aware of the request.


Best,

Steve A

Labels