Find and Delete duplicate source files from System

Hi All,

Within a directory on my machine, there are over 5000 files containing 280 GB. I discovered almost 34 GB of duplicate files by using the Directory and Unique tools.

How can I use Alteryx to help me remove duplicate source files from my system? Please share any workflows you have that can be completed with any scripting language.

Workflow

Developer

Help

Accepted answers

Prometheus

@EN6924 Here's a workflow for you to modify. The batch file won't attach, but this is the script: /c move /y "C:\Users\YourName\YourPath\Fake Data.xlsx" "C:\Users\YourName\YourNEWPath\Fake Data.xlsx"

Move Files Example.yxzp

All comments

Prometheus

@EN6924 Are you saying you have files that are named differently but have the same data in them on your computer? If they're automatically saved and versioned, like File.xlsx vs File(1).xlsx, then you can call those original files to move to a folder full of files you want to keep using the Run Command tool in a batch macro. You just need a list of the files you want to keep. If you don't have a list of files but they're named from automatic versioning, you can pull in that data using a Directory tool and send that to the batch macro.

EN6924

Hi Prometheus

On my system, I have several files with the same name, but they are placed in different folders.

I configured my workflow in the similar way after finding a post on the Alteryx community. Approximately 17 GB of files were removed from my machine after running it. However, the remaining files remained in the system, and there are still 394 duplicate records and files.

I'm getting error messages -

Select (11) cmd: "del "E:\Samsung\15th June 2023\Pn\ABC.mp4..." could not be fully converted from a WString to a String.

Select (11) cmd: Field Conversion Error Limit Reached.

Designer x64 Finished running New Workflow1.yxmd in 5.5 seconds with 11 field conversion errors

Sharing the workflow here. Need help to fix it properly.

New Workflow1.yxzp

Prometheus

@EN6924 The first error is saying that there's probably special characters that aren't recognized in String but are recognized in WString. Take a look at the Data Types help page. I use it all the time to try to understand the differences between specific data types that are generally the same, such as String, WString, and V_WString: https://help.alteryx.com/20231/designer/data-types#:~:text=Alteryx%20supports%20string%2C%20numeric%2C%20date,of%20data%20in%20a%20column. This thread talks about this error: https://community.alteryx.com/t5/Alteryx-Designer-Desktop-Discussions/could-not-be-fully-converted-from-wstring-to-a-string/m-p/61470#M22377. The second error comes about because Alteryx is trying to convert a character recognized by the original data type (WString) and not recognized by the second data type (String). Any time you manually alter data types with a Select tool, you have to keep in mind the limitations caused by the differences between the data types. Alteryx will attempt to convert those characters, but if it can't because of the limitations of the data type you want, it'll mostly likely turn them to nulls. Take for instance the value "ABC." If you try to change the data type of the column that contains "ABC" to an integer, you'll get a null and the warning, after so many records, that the conversion error limit was reached.

EN6924

Hi Prometheus

I built a workflow which copies the duplicate files and paste them to a new folder.

But I want to cut+paste the files, that is delete the duplicate files from source location and move them to the new folder.

Can you help me here?

Copy files from one location to another.yxmd

Prometheus

@EN6924 Here's a script I've used to move files: /c move /y "Current Folder full path with file name and extension" "New Folder full path with file name and extension" . After that, you can run the script to delete the files from the old folder: /c del /y "Current Folder full path with file name and extension".

EN6924

Hi Prometheus

Can you share a sample workflow with the script included?

Prometheus

@EN6924 Here's a workflow for you to modify. The batch file won't attach, but this is the script: /c move /y "C:\Users\YourName\YourPath\Fake Data.xlsx" "C:\Users\YourName\YourNEWPath\Fake Data.xlsx"

Move Files Example.yxzp

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703