Optional override for record limit on updated browse tool data profile.

[helmick2003usap]

Added in Alteryx Version 2020.3, the Browse tool no longer shows a profile of the complete dataset (it is capped when the record data size reached 300MB).

My proposed solution is an optional override of the record size limit on the browse tool (which will make the profiling take longer, but actually profile the entire dataset).  I would also like a general user setting to set the default behavior of the browse tool to either be limited or unlimited.

Below is the newly included documentation of the Data Profiling Limit, which I'm proposing can be overridden.

Data Profiling Limit
Data Profiling in the Browse tool is capped at 300 MB. This allows you to process very large datasets faster. For each record in the incoming dataset, we process the record and add the record size to a counter. Once the counter reaches 300 MB, we stop processing records.

It is important to note that there is no specific number of records that we can process. This depends on the dataset since a record size can range from 1 byte to a few thousand bytes. This record size is different from the file size, displayed in the Results grid and Data Profiling Holistic View. The file size is generally different since it has been compressed to optimize spacing.

In other words, 300 MB of record size is not the same as 300 MB of file size.

This new tool can cause confusion when looking at the data profile (e.g. if you expect the sum to be $3 million, but the browse tool is only showing 2% of your total records in the profile tool, the profile sum may only show $60 thousand).

The sampled version with a cutoff of 300MB is rarely useful if you are using browse tools to get a quick sense of the variable profiles on medium sized datasets (around 1 million records) since this rarely will fit into the 300MB record size limit.

An example can be shown in the image below, where the dataset contains 855,085 records, but the browse tool is profiling only the first 20,338.

 

Again, being able to override this 300MB record size limit would fix the problem created in the 2020.3 change to the browse tool.

[bsanders2009]

I use data sets with >5 million rows commonly (work for a big company), and this limitation makes the Browse tool effectively useless to me.  Like others, I used to use this all the time to get a quick view of my data and an idea of what's in various fields, so now I have to use the Summarize tool or Basic Data Profile or something, which is far less convenient.  Making the 300MB cap optional would be my top enhancement to Alteryx right now.

[JL]

Really wish this one would get more attention.