Directory Creation Tooling

[SeanAdams]

Every time we create a file output - you first have to check if the folder exists - and if not then create it.

Currently it's quite onerous to do a directory create - especially with all the error trapping to make this production safe - and everyone is reinventing the wheel in their own companies.

Given the commonality of this need - could we add a tool that allows you to check for existance of a directory and attempt to create it (with nested directories and useful status / error descriptions to act upon)

[Claje]

Would also be in favor of supporting this natively within Output Data - don't really see a reason to have two tools when a path could be dynamically generated at runtime (and may or may not require a directory check/create)

[SeanAdams]

:-) Like the way you think @Claje !   
So then there are two asks:

a) add this to the output tool as base functionality
b) add directory management tools anyway because this is a core part of data and data prep and ETL

[Claje]

I'll agree to that! :)

[Hollingsworth]

@SeanAdams, this is an idea worth spreading!

I'm here mostly in hopes that comments lend weight to an idea.

Secondly, I'd be in your debt if you would share the directory creation code that you developed for this. I've built a crude DOS command runner (save commands to a batch file and then run the batch file). However, currently I'm just ignoring the errors it gives if the directory already exists. I'd love something a little more elegant.

And I also second the motion for better file management tools in Alteryx. In the Swiss Army knife of Alteryx this is up there with the corkscrew...not everyone needs it, but the substitutes are really ugly to use.

[SeanAdams]

Hey @Hollingsworth 

Here's the code for the directory creator in Python.  

If you look in the first section after the imports, you'll see the error checking to make sure that there are two columns which are needed for this - "RootFolder" and "FinalFolder" - final folder is the actual folder being created, and root folder is the parent.

Much of this is straight-forward - only thing which took a while to figure out is that if you use mkdir then it only does one level of depth at a time, but if you use os.makedirs then it goes an arbitrary depth (e.g. creates subfolders under subfolders if you need).

Hopefully this helps!

------------------------------------------------------

#Bring in the packages you need
from ayx import Package
from ayx import Alteryx
import os
import pandas as pd

#check if inputs are in the right place
dfInput = Alteryx.read('#1')

# Check if the dataframe has the required fields
lstInputFields = list(dfInput.columns.values)
if not('RootFolder' in lstInputFields):
    raise ValueError('RootFolder column does not exist on input columns')
if not('FinalFolder' in lstInputFields):
    raise ValueError('FinalFolder column does not exist on input columns')   

# Create a flag for the new folder
dfInput['FolderCreated'] = False

#Then go through a loop to create the folder
for index,row in dfInput.iterrows():
    #get values from data frame
    folderRoot = row['RootFolder']
    finalFolder = row['FinalFolder']

    # validate if the root folder exists
    if not(os.path.isdir(folderRoot)):
        raise NotADirectoryError("Root folder " + folderRoot + " does not exist")

    # Create the sub-folder
    if os.path.isdir(finalFolder):
        #folder already exists
        dfInput.loc[index, 'FolderCreated'] = True
    else:
        try:
            os.makedirs(finalFolder)
        except:
            raise NotADirectoryError("Subfolder " + finalFolder + " cannot be created")
            dfInput.loc[index, 'FolderCreated'] = False
        else:
            dfInput.loc[index, 'FolderCreated'] = True

#Output the final values
Alteryx.write(dfInput,1)

[JTCairns]

@Hollingsworth 

You could always add an "IF EXIST/ IF NOT EXIST" clause to your DOS command. Avoids the errors. 

[SeanAdams]

Agree with your thinking @JTCairns  - the challenge with the DOS commands though is that we are generally fairly hesitant to allow folks to punch out to the command line within Alteryx when they publish to the server;  for fear that they do something silly to our servers - the command line tool gives unfettered access to a lot of your server, and so we don't generally allow cmd tools on our workflows in the gallery.

I do agree with you in principle though, and on the desktop this is a good solution.

[machazthegamer]

@SeanAdams @Claje 

The render tool already creates a folder if it does not exist (if you tick the group data into separate reports box and give alteryx a completely new path)though this might be a side effect.See sample workflow here https://www.dropbox.com/s/zak8i460v455hhj/create%20path.yxmd?dl=0

So Alteryx team please do not remove this ability when you add the feature requested here: 

[SeanAdams]

great catch @machazthegamer  - I didn't know that - so that's a curious workaround!

Thanks for posting this!

[Hollingsworth]

@JTCairns, I finally incorporated the IF EXIST code and it works much better now. Thanks for the tip!