Job when run in Spark Engine it fails with below error ERROR spark.Transformer: Transformer error: java.lang.RuntimeException: Expected only partition pruning predicates: upper(NUMBER_OF_MONTHS#19) IN (NOVEMBER 2018).toString()

[40c7b8a8e9e8e4f7e204]

Photon Job is successful

[Trifacta_Alumni · Accepted solution]

Hello Vijay,

The error is due to Hive Partition columns are case-sensitive as in https://issues.apache.org/jira/browse/SPARK-19490, please change the column name (NUMBER_OF_MONTHS) to lower case and run the job again.

Thanks,

Pavan