When using the append to existing option, you are able to control the field mappings using the "Append Field Map" setting. the default is to just append based on field position, so you will want to make sure your file matches the Alteryx output. The other alternative would be to do a custom mapping and then you can map specific fields from the Alteryx output to columns of the Excel sheet.
I think the issue with Jet Engine database is due to versions of Excel. The Microsoft ACE engine should be used when working with Excel 2007 or later files (.xlsx as opposed to .xls), while the Jet Engine is used for earlier Excel formats.