Error capturing and logging

[Rohit_Bajaj]

Hi All,

I wanted to know is there any easy way to capture various errors which might occur during moving data from one place to another.

Even capturing this in form of error logs and later parsing in a readable format should be ok.

The requirement is to capture basic data movement technical errors (and not functional errors). Some of the examples might be -

1) Data Truncation Errors.

2) Precision Related Errors etc.

These errors can occur in any of the transformations, starting from input till output and in between transformations.

Is there also a way to increase the logging level to capture details at record level, error type level.

Any help in this regard is highly appreciated.

Thanks,

Rohit

[Philip]

Hi Rohit

I have a small guide that I'm working on. It's still in draft form, but can send it to you later this evening.

It's been created out of removing the "Black Box" out of data transformations so that data consumers know exactly what's been done to their data, problems with the data, and an overall data quality score. With this information, they are able to express the confidence they can put on any products created from the data set.

Let me know if you're interested in the draft guide.

[Rohit_Bajaj]

Hi Philip,

Yes it sounds like it should be helpful, if we are able to track down data changes which are happening when data moves through an Alteryx workflow.

If the changes are desired - like some business transformation etc, we can ignore those and take those which are not desired for e.g. - Truncation, Precision, Conversion etc for my use case.

Please share the guide with me.

Thanks for your help.

Regards,

Rohit

[NeilR]

---

Is there also a way to increase the logging level

---

To increase the number of conversion errors logged go to the Runtime tab in the Workflow Configuration panel (see image below). Here you can increase or remove the default limit of 10.

If you're interested in creating custom logging, take a look at the Message tool. 

[NeilR]

Also, in User Settings, you can enable a logging directory which you may find useful.

[Rohit_Bajaj]

Thanks Neil for the reply.

I am already having the logging enabled under Defaults -> Logging directory.

Can you please guide as in what kind of errors are classified/caught by 'Limit Conversion Errors'.

By definition it reads the following ...

Conversion Errors

Conversion errors are reported when Alteryx cannot convert data from one field type to another. You can set a threshold for the number of Conversion errors to allow per workflow in the Workflow Properties window. Conversion Error reporting was introduced with Alteryx version 2.5. Therefore workflows ...

ConversionErrors.htm

Does it also captures truncation related errors as well or is only limited to conversion errors?

When going through the hyperlink (message tool) ... I was able to get the following information.

• Formula (13): Code: "12 Alteryx Blue" was truncated

I will try exploring the Message tool from my side and see if I can catch hold of undesired effects at workflow level.

In my case I am trying to move data from one SFDC object to another one, but since SFDC does not expose its database for any kind of minus/assertion queries - in particular when Objects are not related by master child/lookup, we are relying on the Alteryx (being our end to end data migration tool) logs to capture any/all kind of errors.

At summary level, record counts can be obtained from source and target, but we wanted to be sure at field level as well.

Generally speaking we wanted to achieve this at workflow level with limited/minimal impact to performance - to the extent possible.

Any other thoughts in this regards are most welcome.

Thanking you for the help.

Best Regards,

Rohit

[NeilR]

---

Can you please guide as in what kind of errors are classified/caught by 'Limit Conversion Errors'.

...

Does it also captures truncation related errors as well or is only limited to conversion errors?

---

From here: 

Conv Errors: (Orange) Data cannot be converted from one field type to another, or data cannot fit in the field specification.

So if you have String 'abc' and you try to convert to Byte with a Select tool you will get the following conversion error:

abc is not a number.

If you have string '9999999999999999999999' and you try to convert to Int64 with a Select tool you will get the following conversion error:

9999999999999999999999 does not fit in an Int64.

If you create a new string field of size 10 and in the expression set it to '9999999999999999999999' you will get the following conversion error:

  "9999999999999999999999" was truncated

So yes, you get some truncation related errors.

---

Any other thoughts in this regards are most welcome.

---

I don't know if any of this will actually apply in you're case since it sounds like you're moving data straight from a Salesforce Input tool to a Salesforce Output tool, so you're relying on the Salesforce API for errors. 

[Rohit_Bajaj]

I tried changing few settings with a dummy workflow but was not able to accomplish Alteryx report the truncation errors.

I believe there should be something fairly simple which I am overlooking.

Can you please have a look at the attachments and guide further.

[NeilR · Accepted solution]

It appears as though shortening field length in a select tool does not result in truncation warnings. So a couple suggestions would be:

1. use a formula tool to create a new field of the desired length and set it equal to the long field, as attached. you will get the truncation warning.
2. proactively calculate the max length of a field to see if it's greater than the desired length. you can even output a custom warning, as attached.

If you think it's appropriate for the select tool to output the truncation warning, I'd advise you to post your suggestion to the Ideas section.