Macros: Batch vs. Iterative

[RichardRogers]

Hi Alteryx Pro's,

I would really appreciate being able to ask a few simple questions on macro's:

- The typical usage is that a standard workflow calls either a batch or iterative macro, correct?

- I have created a batch macro, that parses one item, it works correctly, and I will be calling it from the iterative macro. Is this feasible?

- The main difference between a batch macro and an iterative macro is that the batch macro performs work on one item, whereas the iterative macro loops until a condition is met, correct?

- Is it possible to setup an iterative macro that accepts a list of numeric id's as a parameter from the calling workflow? If so, would I still use a 'Macro Input' tool?

Thank you for your advice/pointers.

Richard Rogers

[MarqueeCrew]

@RichardRogers,

Standard Macros I think are the most common.  They encapsulate the logic of a workflow and allow for configuration around the use case.  If you are parsing 1 field, you could TRANSPOSE the incoming data (Name + Value pair) and parse the value for all fields.  I wouldn't use a batch macro for this function.

For your question about iterative macro that accepts a list of numeric id's, I wonder what you're trying to achieve.  If you had a filter or join before the logic, you could push through the desired records for the macro and union the macro output with the un-selected input data.

Macros are powerful, but a good container can replace the added complexity of a macro if it isn't being replicated by many processes.

Cheers,

Mark

[patrick_digan]

@RichardRogers I'll give you my two cents. Full disclosure: I love batch macros, so I apologize for any bias that comes through :)

---

- The typical usage is that a standard workflow calls either a batch or iterative macro, correct?

---

Macros can be called from anywhere and nested as much as you want. I have a workflow that contains a standard macro, which contains an iterative macro, which contains a batch macro. A workflow contains lot of tools, and macros are just a collection of tools.

---

- I have created a batch macro, that parses one item, it works correctly, and I will be calling it from the iterative macro. Is this feasible?

---

Yep!

---

- The main difference between a batch macro and an iterative macro is that the batch macro performs work on one item, whereas the iterative macro loops until a condition is met, correct?

---

Here is the way I think about it: a batch macro is basically my default choice. It allows me to use control parameters which a standard and iterative macro don't. When you're using control parameters, it does indeed send one item at a time as you correctly pointed out. I would choose to use an iterative macro when either 1) I need to join my macro output (as opposed to union which is essentially what a batch macro outputs) or 2) my output from one run needs to be used again. My second scenario is your looping until a condition is met point.

---

- Is it possible to setup an iterative macro that accepts a list of numeric id's as a parameter from the calling workflow? If so, would I still use a 'Macro Input' tool?

---

I think I would need more info here about what you want to do with that parameter. Iterative macros can get data through macro inputs or regular input tools. Only the batch macro adds the ability to use the control parameter, which can come in handy.

[RichardRogers]

Hi gents,

Thank you for your responses.

I made a couple of major mistakes in my original post. When I wrote "Batch macro", I meant standard macro.

I have attached the main workflow, I think it will clearly show what I'm trying to accomplish.

I haven't attached the standard "GetAuthToken" or "ParseTCXData" macros I created. They are called by the main workflow.

This workflow is working correctly, as it is saved right now.

You will see the comments in the main workflow.

There are essentially six tools that need to be executed once for each row that is output by "SelectRidesForProcessing".

The workflow currently works only if one ride is output by "SelectRidesForProcessing". If more than one row is returned, processing creates errors.

Once I encapsulate the final six tools in a macro, I will also need to figure out how to alter the logic of the final "Write to file" tool.

I need to "Overwrite" if RowNumber = 1, otherwise append.

If you could shed some light on how I should structure this, I would be very, very grateful.

Richard 

[patrick_digan]

@RichardRogers Can you attach the ParseTCXData macro? You can private message me if you don't want to post it publicly. 

Here is where I'm at right now: The download tool can handle multiple URLs, so I just want to see how your parsing the xml with that macro. I think the path of least resistance would be to put the output tool and parsetcxdata inside a batch macro and then just write to your tde once. You batch macro would have a control parameter that you would map to the TCXdata field. Then your batch macro would output the data to a file followed by parsing it. Then you would have a macro output.  Alternatively, it make more sense to just tweak your parse macro so that it can handle all the data at once.

[RichardRogers]

Hi Patrick,

Thank you for your response.

I appreciate that the download tool can handle multiple URLs, but I can only download the TCX data for one ride at a time.

In other words, I cannot have the TCX data for more than one ride in one XML file.

I've uploaded 160 rides to the RWGPS website over the years, so I have 160 XML files containing TCX data after download.

Now I need to parse the TCX data one file at a time.

I agree with you - I will append to the TDE.

I'll figure out how to create an empty TDE with the structure I need, and keep that 'read-only' and in a separate folder.

The main problem I need to resolve currently is how to pass the filename at runtime (from a directory tool) to the macro.

Thank you Patrick,

Richard

[patrick_digan]

@RichardRogers I found your macro over at another post, so I was able to accomplish what I think you are after in the attached. Here are some notes:

1) I was able to get the data without authentication, so I took that part out. Feel free to add it back in.

2) I removed the filter before your second download tool so that it grabs all the tcx files one after the other. Note that it will make 107 consecutive hits on the RWGPS server, so you may want to put a sample tool before the download tool when you're testing and limit it to a couple records.

3) I took what was in your tcx parser and replicated it with xmlparse tools in your main workflow. 

4) I used a recordID tool and sort to get everything back in order at the end.

Let me know if you have any questions! I tested it for the first 2 records and it seemed to be working great.

[patrick_digan · Accepted solution]

@RichardRogers For what it's worth, I'm attaching an example if you wanted to use a batch macro and send the records through the second download tool 1 by 1. I first used the ParseData1 macro to use the download tool and give it one URL at a time. I like to use the %Question.Name% method to pass variables, as opposed to using action tools. Note that you can use the workflow-Configuration-Workflow tab to put in test values. Note that I've changed the download tool to output to a temp file. Then I send the filename to ParseTCX macro where it reads in the tempfile that we just created. This batch macro basically unions all the data and sends it out of the batch macro. Then I just connected it to your tde output. As before, you may want to put a sample tool in before the batch macro if you don't want to send all 100+ records through while you're testing.

Let me know if you have any questions!

[RichardRogers]

Hi Patrick,

Thanks a lot for creating the Batch macro. I'm still figuring it out, but it's settling in my mind.

I'm making good progress, due to your help. Much, much, appreciated!!!

Richard