how to append data frame output in python tool

knozawa

Hello,

Would it be possible to append data frame output in python tool?

I'm writing a web scrape program using a python tool.  This python tool contains a loop to scrape multiple pages from the website.

df = pandas.DataFrame.append({"html_page":[html_page]}) --> TypeError: append() missing 1 required positional argument: 'other'

for i in range (0,2):
page += i

webpage = webpage + str(page)
if __name__ == "__main__":
    main()

Alteryx.write(df,1)

clmc9601

Hi @knozawa,

Thanks for the indentations and longer code! Much easier to follow.

Ok, so that TypeError is misleading and actually means that you're trying to output an empty dataframe. Alteryx & Python refuse to output an empty dataframe. I think the empty dataframe is happening because you're not storing the changes made to df and because you're trying to use df as a global variable. All variables created within a function are local variables by default.

Since you're defining main() in your code, here are a couple of changes I'd make:

data = Alteryx.read("#1")
df = pandas.DataFrame()

def main(): 
    wd = webdriver.Chrome("C:/Program Files/Alteryx/bin/Plugins/chromedriver.exe")
    wd.get(webpage)
...
    # And grab the page whole HTML source
    html_page = wd.page_source

    # Attempt to close chromium instance
    wd.quit()
    
    return html_page # by returning the results, you can append it to a global dataframe external to the function
    
for x in range(0, 2):
    webpage = data['file'][x]
    print(webpage)
    if __name__ == "__main__":
        df = df.append({'html_page':main()}, ignore_index = True) # you have to redefine the df variable. if you just do df.append, the change is executed and then lost

#Write the data frame to Alteryx workflow for downstream processing
Alteryx.write(df,1)

I hope this helps! I tested as much as I could without having those exact packages, but there might still be syntax errors.

I look forward to hearing from you.