YXDB - CReW Inspired Idea about YXDB

[MarqueeCrew]

Hey YXDB Bosses,

Let's move forward with our YXDB.  Maybe give AMP a real edge over e1.  Here are some things that could may YXDB super-powered:

• Metadata
  • Workflow information about what created that poorly named output file.
  • When was the file originally created/updated.
  • SORT order.  If there is a sort order for the data, what is it?
• Other stuff
  • INDEX.  Currently you get spatial indexes (or you can opt out).  If I want to search through a 100+MM record file, it is a sequential read of all of the data.  With an index I could grab data without the expense of a calgary file creation.  Don't go crazy on the indexing option, just allow users to set 1+ fields as index (takes more time to write).  
  • I'm sure that you've been asked before, but CREATE DIRECTORY if the output directory doesn't exist.
• Old School - Crazy Idea
  • Generation Data Groups (GDG)
    This will likely make @NicoleJ 's eyes roll 🙄 but back in the days, we could write our data to the SAME filename and the old data became 1 version older.  You could read the (0) version of the file or read from 1, 2, 3 or more previous versions of the data using the same name (e.g. .\Customers|||3).  The write of the output file would do all of the backing up of the data (easy to use) and when the initial defined limit expires, the data drops off.

Just a little more craziness from me

cheers

[TonyaS]

Thanks @MarqueeCrew 

Nicole and I will review these next time we meet and figure out who owns what and get them into the Intake list for consideration. 

We'll talk about this one too: https://community.alteryx.com/t5/Alteryx-Designer-Ideas/YXDB-Source-Information/idi-p/944599 (Likely Engine and we'll have to review security implications on some of the metadata like path and userid). 

[TonyaS]

Mark,

Thanks again for all the ideas. 

Nicole and I reviewed all 6 of these ideas and and we are definitely going to consider the INDEXing suggestion for our Roadmap. 

We do already have a lot of metadata and were not sure what you meant by the SORT order. Did you mean a field and whether it was asc or desc or were you talking about the different sort ordering between e1 and AMP? How would you use that? 

The other ones about file/dir management and versioning sound like they could be Data Connectivity ideas, I will run those by that team. 

[MarqueeCrew]

I'm happy to meet with you this week. 

[TonyaS]

Thanks Mark but we're probably a bit early on that, we will make sure to include you when we are closer to starting work on it to get additional input and feedback.