Dynamically Changing URL for Download Tool

[RichardRogers]

Hi folks,

I have a Text Input tool on my workflow, the output goes to the Download tool.

The "GetRideURL" column is set to: "https://ridewithgps.com/trips/{RideId}.tcx?sub_format=history".

I have a sort tool that outputs multiple RideIds.

How do I alter the workflow so that the RideIds from the sort tool get passed to the Text/Download tool?

There must be something simple I am not seeing.

I've looked in the "Basic", "Headers", "Payload", and "Connection" tabs for the Download tool.

Thank you for your help,

Richard Rogers

[david_fetters]

You could do this two ways.  The first (and slightly more complicated) would be to create a batch macro that you pass RideIDs into, and use to replace the text {RideID} in your text input tool.  Then you can parse the results inside the macro and return only the information you need.  This is useful mostly when you need to access the data in a lot of different macros and want the convenience of building the HTML parsing in a single location.

The easier way would be to append each RideID to your Text Input Tool that contains the 'Template URL' "https://ridewithgps.com/trips/{RideId}.tcx?sub_format=history" which will create one row per RideID.  Then pass it through a formula tool where you use the Replace() function to fill your 'template' URL with the correct RideID and then pass that into your download tool.  An example of that function would be Replace([TemplateURL], "{RideId}", [RideIDValue]).

Each row passed into the download tool will trigger a download from that website.  Be mindful of rate limiting if you plan to pass a bunch of these at one time, or you might end up banned.  Consider using the Throttle tool to slow down your requests to only a few per second.  And be sure to read the terms and conditions of their website, as many sites don't permit scraping their data in a programmatic way.  I'm not sure about ridewithgps.com, and i'm not your lawyer!  Do with this knowledge what you will :)

Edit: I looked at your workflow and it doesn't seem like your question and workflow are the same.  The workflow seems to indicate you're trying to login to a website and then grab pages of results based on a different URL than the one you provided.  What are you looking to pull off here?  (if you're trying to pull down all results for a bunch of different users as your workflow suggests, you might want to more elegantly figure out how many pages of results you need to go through to get all of them).

[RichardRogers]

Hi David,
Thank you for your help, I appreciate it, and I would appreciate it if I can ask you two more questions.
The first download tool retrieves the list of RideId's, just for my UserId.I am a cyclist, and I've uploaded quite a few rides to RideWithGPS over the years.They have a JSON/REST api that is documented here: https://ridewithgps.com/api.
The first download tool is successfully retrieving my RideIds.I pass them to the sort tool (named SortOnRideId) so that they are in ascending and chronological sequence.
Using the second download tool (DownloadTCX), I want to retrieve all the detailed TCX (XML) data for one RideId.(FWIW, on TCX format: https://en.wikipedia.org/wiki/Training_Center_XML)

GetRide is the text tool, with column GetRideDataURL set to "https://ridewithgps.com/trips/{RideId}.tcx?sub_format=history".I have a wired connection from GetRide to the Formula tool, FormatRideId.FormatRideId has an output column "Formatted" = "Replace([GetRideDataURL], "{RideId}", "5959775")".
There is a wired connection from FormatRideId to DownloadTCX.This should retrieve the TCX data for my first ride.
There are two problems:
- The first is that the Formula tool has a hard coded value for RideId.
I have looked high and low as to how to retrieve the RideId's output from SortOnRideId, and I don't know how to reference the output of a previous step.
I thought that something like this would work: "Formatted" = "Replace([GetRideDataURL], "{RideId}", [SortOnRideId.RideId])".
- The DownloadTCX tool is establishing a new connection, WITHOUT the credentials that I supplied with the first download tool. Is there a way that I can have it use the same connection as the first download? If not, I will have to log back in, using username and password, meaning I will have to somehow set that up again within the workflow.

I look forward to any advice/pointers you have.
In the meantime, I will lookup what these wireless connections are.
Thank you,
Richard Rogers

[david_fetters · Accepted solution]

Hey Richard, it looks like i didn't see the second half of your workflow on my laptop when I glanced at it (sorry!).  I've added a formula tool that prepares the URL for each row of RideID by using this formula:

"https://ridewithgps.com/trips/"+[RideID]+".tcx?sub_format=history"

That should be all you need if we assume RideID is properly formatted.

I also added a throttle tool to slow your queries down (tweak the settings however you like, it's at 1 per second).

Let me know if this works?

Edit: Ooops, RideID is a number, so to make the formula work I had to add the ToString() function.  I reuploaded the corrected version (which works when I test it).

The new formula is: "https://ridewithgps.com/trips/" + ToString([RideID]) + ".tcx?sub_format=history"

Also, you may want to change your password to that site since you included it in the sample :)

[RichardRogers]

Thank you for your help, and the advice on the password!