This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I am experiencing a strange issue with this portion of my workflow:
I am importing about 10 different files, with mostly identical column headers. I then union them and pass them on to the rest of the workflow.
However, the workflow freezes at 88% after the union every time I run it. Here are some very strange things, though:
If I delete any of the connections between the Union, Block Until Done, Data Cleansing, Find Replace, and the BB Date tools, the workflow will complete successfully (up until the severed connection, of course).
If the workflow connects to the BB Date tool in any way, it will freeze at the Block Until Done tool.
Since the Union tool never reaches 100%, I am not sure what is causing the hangup by having it connect to the BB Date tool. If I remove all of those tools, the workflow again completes just fine.
I have tried letting the workflow run for 10 minutes, thinking it might just be working in the background, but it never passes the 88% mark at the Block Until Done tool.
How would I track down the cause and potentially work around this?
1. Do you need to use the BB date tool? Are your dates in more than just a couple of formats that can't be handled with if-then logic in a formula tool? If each input has its own date format, can you deal with it before you get to the union?
2. Why do you have the Block Until Done with only one output?. The point of this tool is to let multiple streams complete sequentially. With only one output, it becomes redundant.
Unfortunately, I do need to use the BB Date tool as the incoming data is quite "dirty." Some input files even use more than one date format within the same data set.
I added the Block Until Done tool as a part of troubleshooting this issue. The issue persists without that tool as well.
I do believe it's the BB Date tool that is causing the bottleneck, though. I allowed the workflow to run overnight and it's now at 90%, with my single BB Date Tool reporting 23%. I do see it is still working as the number of items processed is increasing, albeit slowly.
My entire dataset at the Union tool is only 68,000 rows, though, so I am not sure why the BB Date tool is taking so long.
I would be happy to know of alternatives to that tool since that is obviously not going to be a viable option for me.
I would consider building up a list of input date formats and continuing with your IF/THEN logic.
Another option would be to use the BBTool on each input leg. From what I've found out about it, it uses nested macros to scan the data to find the most common input formats. This sounds like the kind of algorithm that increases in run time exponentially as the data set grows. In this case, breaking the data down in to 10 chunks would cause it to run 10 or more times faster.