topic Re: Bug with CSV files containing quoted fields in Dev Space

Bug with CSV files containing quoted fields

360Andy — Tue, 23 Jan 2018 10:15:25 GMT

We have found what appears to be a bug in Alteryx.

If you load data from a CSV file containing quoted fields, then any blank fields get a value from a previous record.

The simplest way to replicate the bug is using the SampleOutput component included in the SDKSample. This exhibits the problem out-of-the-box, with a workflow as simple as an Input Data component linked to the SampleOutput component.

Input example1.csv:
"10","MR W DAYTON","PALMER AIR CHARTERS, INC","SUITE 106","7350 AIRPORT ROAD","WILMINGTON","DELAWARE","19801","212 249 7586"
"11","MR K SIPPLE","FEIDER COMPANIES INC","POST OFFICE BOX 56259","FEIDER PARKWAY","MINNEAPOLIS","MINNESOTA","55459",""
"12","TAMMY TASAN","ANDERSEN LAMB INC","2300 MARIE VICTORIN/05CAZ","LONGUEUIL QUEBEC","","","",""

Output example1output.csv (extraneous fields highlighted):
10,MR W DAYTON,PALMER AIR CHARTERS, INC,SUITE 106,7350 AIRPORT ROAD,WILMINGTON,DELAWARE,19801,212 249 7586
11,MR K SIPPLE,FEIDER COMPANIES INC,POST OFFICE BOX 56259,FEIDER PARKWAY,MINNEAPOLIS,MINNESOTA,55459,212 249 7586
12,TAMMY TASAN,ANDERSEN LAMB INC,2300 MARIE VICTORIN/05CAZ,LONGUEUIL QUEBEC,MINNEAPOLIS,MINNESOTA,55459,212 249 7586

Does anyone know of a workaround for this issue?

Re: Bug with CSV files containing quoted fields

cmcclellan — Tue, 23 Jan 2018 10:31:51 GMT

I think it must be a problem with the SampleOutput macro. I don't have that in my installation, but I used a Browse tool and checked the Input tool and they are both reading the data perfectly.

Re: Bug with CSV files containing quoted fields

360Andy — Tue, 23 Jan 2018 10:52:58 GMT

SampleOutput is not a macro, it's C++ SDK example.

The problem is in the SDK class FieldBase. The accessor methods to fetch a field value. E.g. GetAsWString() return the value from a previous record when they should return null.

There is nothing that I can see wrong with the SampleOutput code:

for (unsigned x=0; x<m_recordInfoIn.NumFields(); x++)
{
const FieldBase* pField = m_recordInfoIn[x];
if (x > 0)
m_outfile << m_separator;
TFieldVal<WStringVal> field = pField->GetAsWString(pRecord);
if (!field.bIsNull)
m_outfile << field.value.pValue;
}

It's just that the FieldBase class is doing something wrong.

The Browse tool doesn't have this problem, so it is obviously doing something different, but using these FieldBase class methods is the only way for a Custom Component developer to access the field values.

Re: Bug with CSV files containing quoted fields

MichaelCh — Wed, 24 Jan 2018 16:03:00 GMT

What does II_PushRecord look like on the SampleOutput tool?

Re: Bug with CSV files containing quoted fields

360Andy — Wed, 24 Jan 2018 16:07:40 GMT

That's where the code above was taken from. Here's the whole method:

long SampleOutputInterface::II_PushRecord(const RecordData * pRecord)
{
    // Our input is providing us with a new record to process.

    // If this is the first record, then we need to open (or create) the
    // file, then write the field names as the first record.
    if (!m_outfile.is_open())
    {
        m_outfile.open(ConvertToAString(m_strFilename));
        if (!m_outfile.is_open())
            throw Error(L"Cannot open the specified file for writing");

        // Write the field headers.
        for (unsigned x=0; x<m_recordInfoIn.NumFields(); x++)
        {
            const FieldBase* pField = m_recordInfoIn[x];
            if (x > 0)
                m_outfile << m_separator;
            m_outfile << pField->GetFieldName().c_str();
        }
        m_outfile << L"\n";
    }

    // Write the record.
    for (unsigned x=0; x<m_recordInfoIn.NumFields(); x++)
    {
        const FieldBase* pField = m_recordInfoIn[x];
        if (x > 0)
            m_outfile << m_separator;
        TFieldVal<WStringVal> field = pField->GetAsWString(pRecord);
        if (!field.bIsNull)
            m_outfile << field.value.pValue;
    }
    m_outfile << L"\n";

    // Return 1 to signify that we processed the incoming record properly.
    return 1;
}

Re: Bug with CSV files containing quoted fields

MichaelCh — Wed, 24 Jan 2018 17:25:00 GMT

There's something fishy going on here, but, like yourself, I can't tell what it is. My suggestion would be to debug it and see what the values are at each step of that for loop and go from there--maybe you'll figure it out, but if not you'll be able to narrow the problem down to something more specific.

Re: Bug with CSV files containing quoted fields

MichaelCh — Wed, 24 Jan 2018 18:24:59 GMT

@Nedand I looked into this a little bit more. We both feel like there must be something else going on here, but I did come away with one idea. What happens if you replace

        if (!field.bIsNull)
            m_outfile << field.value.pValue;

with

        if (!field.bIsNull)
            m_outfile.write(field.value.pValue, field.value.nLength);

Re: Bug with CSV files containing quoted fields

cmcclellan — Wed, 24 Jan 2018 21:07:43 GMT

I'm probably stating the bleedingly obvious here and I don't know C++ at all, so can't offer some code suggestions.

BUT .... the problem is basically that the old values are being "carried over" to the next record. I can't see the code that wipes out the values for the previous record before loading in the values for the next record.

Would the solution be as simple as adding some code that simply loops through all the fields and clears them before adding the new values in ?

Re: Bug with CSV files containing quoted fields

MichaelCh — Wed, 24 Jan 2018 21:31:07 GMT

The API for the record classes in the C++ SDK is a little odd. The fields are being used to pull data out of the pRecord, which represents the data passed in from the upstream field. So it doesn't need to be reset in the tool because the upstream tool has already passed new data in. The problem here looks like some kind of caching issue in the FieldBase object. My suspicion (as implied by my last post) is that the string returned from GetAsWString is somehow not being null-terminated correctly, but that the length is still correct. If that's not it, I'm not sure what's going on.

Re: Bug with CSV files containing quoted fields

360Andy — Thu, 25 Jan 2018 09:49:41 GMT

Thanks for your help guys.

I have stepped through the SampleOutput code but can't find any way to detect that the value being returned is bogus.

Looking at the documentation for FieldBase there is a method called GetNull():

"This function determines whether the field's value in the specified RecordData structure is NULL."

I changed the SampleOutput code to use this but it doesn't make any difference.

        if (!pField->GetNull(pRecord))
        {
            TFieldVal<WStringVal> field = pField->GetAsWString(pRecord);
            if (!field.bIsNull)
                m_outfile << field.value.pValue;
        }

(Edit: I thought I'd found the solution here, but actually I'd already tried this back in September)

Re: Bug with CSV files containing quoted fields

360Andy — Thu, 25 Jan 2018 09:50:20 GMT

This doesn't help either:

m_outfile.write(field.value.pValue, field.value.nLength);

In my own component I'm already testing everything available:

    if (!pField->GetNull(pRecord))
    {
        // Get the field value
        TFieldVal<WStringVal> fieldVal = pField->GetAsWString(pRecord);

        if (!fieldVal.bIsNull && fieldVal.value.nLength > 0)

None of it helps :(

Re: Bug with CSV files containing quoted fields

MichaelCh — Thu, 25 Jan 2018 16:30:39 GMT

This looks like a bug in the SDK. I've made @TashaA aware of it and she is looking into it.

Re: Bug with CSV files containing quoted fields

360Andy — Thu, 25 Jan 2018 16:32:02 GMT

Thanks. Good luck, @TashaA

Re: Bug with CSV files containing quoted fields

360Andy — Mon, 12 Mar 2018 15:33:11 GMT

Any progress on this?

Re: Bug with CSV files containing quoted fields

360Andy — Tue, 20 Mar 2018 15:19:52 GMT

@TashaA, at first we thought this problem only affected data from CSV input files with quoted fields, but we are now seeing it in other circumstances.

We saw it today occur between the output of one of our components and the input of the next.

Upstream the source data was from a tab delimited file without quoted fields.

The problem seems to go away if I switched the input file from Latin1 to UTF-8.

Any news on when we can get a version of the SDK with this bug fixed?

Ta.

Re: Bug with CSV files containing quoted fields

TashaA — Tue, 20 Mar 2018 17:44:15 GMT

Thanks for the additional information @360Andy. At this time, I do not have an estimated time for a fix. I will be working with our groups here to see if we can provide any additional troubleshooting or work-around steps.

I am sorry you are experiencing trouble with this, feel free to continue to share any additional items you notice as you have done today.

Best,

Tasha

Re: Bug with CSV files containing quoted fields

360Andy — Wed, 30 May 2018 07:49:33 GMT

.@TashaA, Any news on this? Our sales people keep running up against this problem with prospects and it is very embarrassing for them as it looks like a problem with our component. It is such a severe problem that it destroys any confidence in us.

Re: Bug with CSV files containing quoted fields

TashaA — Wed, 13 Jun 2018 16:41:12 GMT

Greetings @360Andy,

Some news on this front. Good news-we have validated the bug, the not-so-good news-we do not have an immediate remedy for this. I will keep this thread and you posted when we know that we have a fix planned for release.

-Tasha

Re: Bug with CSV files containing quoted fields

360Andy — Wed, 19 Sep 2018 14:35:28 GMT

Any update on this? Is there any sort of workaround I can use?

This is constantly causing problems.

Re: Bug with CSV files containing quoted fields

360Andy — Wed, 14 Nov 2018 14:46:18 GMT

Hey @TashaA, any news?

I see the is a new release 2018.4.

My fingers are crossed ;)