Download file wont extract URL's inputed in text file

Hi, I'm unsure why the download tool won't download the URL's properly and extract them and output them. Is there anything I can do to fix this? Why might this issue be occurring? I've attached the workflow below, any help would be greatly appreciated!

New Workflow3.yxmd

Download

Accepted answers

TheOC

@zaina1498 okay so that's literally mostly everything I can think of ruled out as to why it would run on my machine and not your own. And not a problem at all for the help!

As far as the python tool - I have yet to touch that tool, but as far as I understand it will just run python scripts, you would be looking at something like Beautiful Soup , to run for each URL brought in.

What I would like to try first, is a Batch Macro, just to see if it works, or gives any clues, running through them iteratively.

Please see attached a workflow with a macro, you may need to import the macro

If you have to bring in the macro, then it should have both connectors from the original file, and the only thing that needs changing is the control parameter in the questions tab:

Give that a whirl, hopefully it works! I've tested it on my side and it works fine!

Zaina1498 batch.yxmd

Zaina1498.yxmc

All comments

atcodedog05

Hi @zaina1498

I tried you workflow. It seems to work.

What exactly is the issue?

TheOC

Hi @zaina1498

I had a look at it looked like some of your request headers were coming back as wrong.

I have added a user-agent to the headers field of the request, and this seems to have fixed the issue on my side. Does it fix it for you too?

Attached below is the workflow

Zaina1498.yxmd

atcodedog05

Hi @TheOC ,

Thats powerful tool you have got there can you explain more about adding a user-agent to the headers field of the request

zaina1498

TheOC thank you for getting back to me! However it still wouldn't run for me. It stops after a while of it running and gives me an error when I run the workflow you attached.

TheOC

very interesting, my workflow works with no problem at all, i'm curious as to why it wouldn't for yourself!

zaina1498

Do you think it could be a difference in Alteryx settings or versions?

TheOC

Hi @zaina1498 ,
Very potentially, What version are you on with Alteryx? I'm not quite on the newest so ill try updating to your version and seeing if I get the same issue!

zaina1498

Hi @TheOC ,

Would greatly appreciate it if you can! I have the Alteryx Designer 2020.3 x64 version

TheOC

@zaina1498 Hi,

I am on the same version, and weirdly I do not get this issue. There is only a few other things I can think of, but I'm not that sure at this point.

Has the machine you're running it off got a stable internet connection?

Is it potentially a firewall stopping you doing the requests?

Can you troubleshoot by running my attached workflow? This should just run for one of the rows, just to see if that works.

As you can see from the screenshot, my workflow is a success and displays a result:

Zaina1498.yxmd

zaina1498

Hi @TheOC,

This is my work laptop so I'm not sure about potential firewalls. And yes your new workflow does run for me, for the one link but if i increase it i get the same error, So frustrated why this seems to happen. Do you think I could extract the same info. I need from the python tool. Not sure how to use that tool because I've never used it before.

TheOC

This one is weird to me, to be honest!

I have put a filter on, and as far as my side of things goes, none of the download headers are null:

if you go into the sample tool in the last workflow and change it to 50, or 100, does it still work?

zaina1498

Nope! It didnt even go through 30. Do you think I could extract the same info. I need from the python tool. Not sure how to use that tool because I've never used it before.

TheOC

You definitely could get the data from the python tool, but I suspect you will run into the same issue. Just to check, do you happen to know how much RAM your machine has? and have you been find running big workflows before?

zaina1498

I have a 16GB Ram and I have run other workflows before which also go through a lot of URL's like this one, but there didn't seem to be a problem with that. Would you be able to explain how I would implement the same thing using the Python tool? Again, thank you so much for your help!

TheOC

If you have to bring in the macro, then it should have both connectors from the original file, and the only thing that needs changing is the control parameter in the questions tab:

Give that a whirl, hopefully it works! I've tested it on my side and it works fine!

Zaina1498 batch.yxmd

Zaina1498.yxmc

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703