Engine Works
Under the hood of Alteryx: tips, tricks and how-tos.
This article is part of the Alteryx Obscura Blog series. Alteryx Obscura is the Inspire session dedicated to all things non-analytics, highlighting what the software “can” do rather than what it’s “supposed” to do.
The Interpreterator (aka what’s the most obscure way to Rick Roll Inspire?)
Source: Fandom
Motivation
This year was my 15th Inspire and 4th year hosting Alteryx Obscura, so I was super motivated to create an epic Obscura project. After noodling on it for a while, it came in a flash that there was only one clear solution to my problem – to Rick Roll Inspire attendees. The question then was what is the most obtuse way possible? The answer, of course, was dead simple: Use Alteryx!
Thanks to the seminal work of @clmc9601 and @MarqueeCrew from Obscura 2023 called “Music is the shorthand of Alteryx,” I knew it was possible for Alteryx to play musical notes. In their Obscura entry, they assigned a note to each tool on the canvas, thus “playing” the canvas as a song. The Rick Roll takes this idea a step further, because instead of playing random notes, we want to play a song comprising notes in a specified order. Sounds simple enough, but it turns out to be a Hard problem.
But, now that we have the idea and a copy of Alteryx Designer, the race to create the Interpreterator begins. Off we go!
Step one: Figure out how to play sequential notes
The crux problem of the Interpreterator is having Alteryx create something that can eventually be Interpreterated into music. Alteryx is amazing at generating data, so it seems logical to create a data stream corresponding to the correct sequence of notes, and then “play” that data as the Rick Roll. Sure, it’s a decent idea that’s easily done in Alteryx, but it’s waaaaay too simple for Obscura.
Fortunately, there’s a harder way to do it!
When Alteryx runs a workflow, in addition to producing data, the Engine also generates a commentary about the workflow. That commentary comes in the form of the Engine output log, which represents the Engine’s train of thoughts while it’s processing the workflow. The Engine isn’t entirely smart, so it only knows how to speak in three ways – messages, warnings, and errors:
It’s critical to note that these Engine outputs have two important qualities:
- The messages come in different lengths.
- The Engine emits messages deterministically, meaning the order of messages is always the same, i.e. corresponding to tools on the canvas (note that this is only true for the e1 Engine).
Armed with those two details, I knew this project had to be possible.
Step two: Find error messages
The next challenge then, of course, was to enumerate the possible messages (message, warning, field conversion error, etc.) the Engine could possibly produce during any given workflow run. Given that I have access to the source code for the Engine, this task included a straightforward Alteryx workflow to find all the places in our Engine source code where the Engine is asked to emit a message (Dynamic Input for the win!). Thanks to Rule #1 of Obscura (“There are no rules”), using the source code is well within the limits of Obscura 😊.
The messages are formatted in the code as strings, and are either fixed length (meaning, every time they are written they are always the same number of characters), or they are variable length thanks to optional parameters that get populated when the message is requested (for example, using a field name or file name). As an example, consider this snippet showing two messages from the Append Fields tool. The first message is nominally 44 characters long but has a placeholder in it (marked by the “@1”) which will ultimately change the message length depending on the exact value of the placeholder. The second message (“There were no records present in the source”), on the other hand, has no placeholders and is therefore always 43 characters long:
After creating my spreadsheet of messages harvested from the e1 Engine source code, I knew I had a solid project idea. I did not know if it was possible to pull it off though ☹️. The desired algorithm, however, was simple:
- Create a workflow that generates a precise sequence of Engine output messages.
- Control the exact length of each message so it corresponds to some concept of a “note”.
- Map the message lengths to actual notes.
- Play the notes out loud.
- Smirk & Profit.
Step three: Find and prepare Rick Roll music
Next, I decided to skip the Hard part of the project (i.e. the note generation) and focus instead on finding some music. Thankfully, the internet machine is littered with Rick Astley music, so I settled on an example that included the lyrics and corresponding notes:
At this point in the project, I figured I would use the PowerShell function [console]::beep to play each of the notes, and it requires specifying the frequency of each note. Therefore, I had to map the notes in the music to their corresponding frequencies using a lookup table like this:
This was easily done using the Alteryx workflow “DetermineNotes.yxmd” which I’ve attached to this post for reference. As you can see, the workflow takes the text of the music as one input, the frequency lookup table as the second input, and it merges the two together to create the song as a sequence of frequencies:
Note too that the workflow plays the corresponding notes using PowerShell, but it may or may not work on your machine. More on that problem later!
Step four: Create an Engine log corresponding to the song
By far the most complex problem in this project is generating an Engine log that can be interpreterated as notes. The idea is quite simple – the length of an output message corresponds to a note in the song, and the sequence of the messages corresponds to stanzas in the song. We can visualize the “shape” of the Engine output log by placing vertical bars at the end of each message:
The question then is how to match the “shape” of the Engine log with the “shape” of the song itself. The answer is simple – A LOT of experimentation.
Using the harvested Engine message templates as a guide, I started dragging & dropping like a madman, experimenting with different chains of tools to try and create some semblance of a song. There are tens of tools to choose from, but the trick is not only choosing the right tool but figuring out how to trigger the correct Engine message of the correct length, for example, as an informational message, warning, or field conversion error:
The final workflow is called “Remix.yxmd” and is a pile of pure Alteryx gibberish. As you’ll see in the workflow, the tools are roughly organized into three chains and individual tools are labeled by their note index and “target” note, i.e. the note that we want to generate.
Note that Text Input tools are extremely useful because they can start each chain and will also generate a consistent output message whose length depends on the number of input records. The Text to Columns is similar because, in a warning message, it will emit the field it had difficulty parsing. Since we can control the length of the misbehaving input field, it’s useful for “tuning” the length of the output message to match a target note.
Step five: Interpreterate and play some music!
Nonsensical workflow in hand, it was time to build the actual Interpreterator. As a reminder, the mission of the Interpreterator is to convert the engine output log generated by Remix.yxmd into our coveted Rick Roll. As you can see in the attached “Interpreterator.yxmd”, the workflow itself is quite simple and roughly split into two parts.
The first part of the workflow joins the Engine output log with the expected notes from the song for comparison. This was critical for the experimentation phase when I iterated on the order of tools in the parent workflow that would produce the correct messages (aka notes):
The Engine messages are then fed through a Tile Tool, which converts the length of each message into a corresponding tile. The tile index is then zero-based into the note index. As you can see by the Tile Tool configuration, the message length thresholds are very carefully chosen to segment the messages correctly into the target note buckets. This turned out to be a very tricky process, but each successful note brought more laughter so it was all worth it:
After generating the list of notes, the Interpreterator feeds the note indices into a macro called the “Frequency Modulator.” The job of the Frequency Modulator, of course, is to modulate frequencies, and is inspired by Marvin the Martian’s Earth-destroying Space Modulator from Bugs Bunny (see this video).
Source: Wikipedia
As originally designed, the Frequency Modulator joins the note indices with corresponding frequencies, and plays each note in sequence using the PowerShell method [console]::beep. This method worked perfectly in testing, but when my computer was attached to my larger monitor via an HDMI cable, or played through headphones, the output was spotty at best. At this point I was imagining the inevitable disaster of the music not playing correctly before the live audience at Inspire so I was nervous the whole project was crashing to a halt.
My goal with this project was to write it in Base-A (plus a little PowerShell but who’s counting…), but at this point it was time to rip off the purist bandaid and invoke a Python solution. Following the lead of @clmc9601 and @MarqueeCrew, I used the Python SCAMP library for note playback, requiring just a couple of lines of Python in a custom Python tool:
So, given that the Frequency Modulator now relies on Python, it’s basically Base-A 😊.
Conclusion
In summary, this was a super fun Alteryx Obscura project and definitely in the spirit of the event. The resulting workflows are very simple but required a lot of trial and error to get correct, and the final output is as good of a Rick Roll as I could imagine. Not perfect, but gets the point across.
Thanks for reading and for continuing to support Alteryx Obscura. Obscura is a grassroots effort hosted by Alteryx Engineering to celebrate all the things that Alteryx “can” do rather than what it’s “supposed” to do, and it’s becoming a cornerstone of Inspire. Hope to see you all there!
