Alter Everything Podcast
A podcast about data science and analytics culture.Podcast Guide
For a full list of episodes, guests, and topics, check out our episode guide.
Go to Guide
In this episode of Alter Everything, we chat with Eric Soden and JT Morris from Alteryx partner Capitalize about the practical applications and limitations of generative AI. They discuss ideal use cases for large language models, the importance of balancing generative AI with traditional analytics techniques, and strategies for scaling AI capabilities in enterprise environments. Eric and JT also share real-world examples and insights into achieving productivity gains and ROI with generative AI, along with the importance of maintaining data quality and explicability in AI processes.
Panelists
- JT Morris, Senior Manager, Advanced Analytics Practice Lead @ Capitalize, @JTMorris, LinkedIn
- Eric Soden, Co-founder and Managing Partner @ Capitalize, @esoden, LinkedIn
- Megan Bowers, Sr. Content Manager @ Alteryx - @MeganBowers, LinkedIn
Topics
Transcription
Ep 186 Capitalize - Alteryx + Generative AI Use Cases
[00:00:00] Introduction to the Podcast and Guests
[00:00:00] Megan Bowers: Welcome to Alter Everything, a podcast about data science and analytics culture. I'm Megan Bowers, and today I am talking with Eric Soden and JT Morris from Alteryx partner Capitalize. In this episode, we chat about what generative AI is good for, what it's not good for, and use cases that break out of the chatbot productivity box by scaling the power of AI with Alteryx. Let's get started. Eric. JT, it's great to have you on our show today. Thanks so much for joining. Could you each give a quick introduction to yourself for our listeners? We can start with Eric.
[00:00:43] Eric Soden: Sure. Yeah, absolutely. So Eric Soden, managing partner and one of the founders of Capitalized Analytics. My whole background has been data warehousing, ETL, process
[00:00:51] JT Morris: automation. Now a lot of AI ML. T Morris Leader Advanced Analytics practice here at Capitalize. So I've been here to Capitalize coming on about six years next month, but background in math, computer science, but a lot of focus on helping clients extract insights from their data. JT is the brains of the operation.
[00:01:08] Megan Bowers: Awesome. Well, it's good to have you both here to weigh in on some of these questions and looking forward to chatting more about AI and large language models.
[00:01:17] Understanding Generative AI and Its Use Cases
[00:01:17] Megan Bowers: I think a good place to start would be to talk about some use cases for large language models. Just from your experience working with clients. When should we look to leverage LLMs and when should we look to leverage other approaches for data challenges?
[00:01:35] JT Morris: So to me, I feel like there's three areas that I really think about when we're looking for generative AI, large language model use cases. Number one, just anytime we're working with lots of unstructured data. So this could be text, data, documents, transcripts, things that we need to summarize, analyze process quickly. Second would be anytime we have some sort of natural language interaction. Most commonly people think about things like chat box, but this could also be search interfaces, things along those lines where you're typing in a question or a quest in natural language and you're expecting a response back. And then the third times when we need to generate content. This could be we need to generate emails, summaries, we wanna generate code, and I think we can go into a lot more detail as to all the different use cases that fit into those three kind of major buckets. But those are the big areas that come to mind for me.
[00:02:30] Eric Soden: I'll take it from a slightly different angle then.
[00:02:32] Personal Productivity and AI at Scale
[00:02:32] Eric Soden: I think you have a lot of people who are seeing massive assistance in personal productivity. I need to write an email, I need to do a post. I need to summarize something personally and with the, with J Chat, GPT hitting us in 2022, that was a big change I think in just personal productivity. And I think a lot of the things that we're seeing, and a lot of the things I hear people talking about are personal productivity things. Quicker to analyze something quicker to do research, quicker, to get an answer to a question, have an AI take notes at a meeting, things that we would potentially have done ourselves personally. The things maybe I'm the most excited about as I think of Alteryx and generative AI or RP and generative AI, or AI at scale or in an enterprise is how can we utilize it? Not for single requests, but how do we use it where we're calling it hundreds of times or thousands of times. Rapidly in succession. So how do we secure generative AI and how do we lock it down so that our proprietary content can't get out of our walls and that we're not sending things out into the cloud and we don't know where it's going? So for me, I think personal productivity is the easiest. It's the stuff that everyone is doing every day. I don't, there aren't very many people I know right now who aren't using it at least once a day for something. Even if it's grammar and spelling or summarization or clarification or something like that. But the stuff that I've really found fun lately is how can we take advantage of it at a, in a scale that would've been impossible before.
[00:04:08] Megan Bowers: Yeah, definitely. And I'm excited to get into that later in the episode too, this idea of scaling it, something that talked a little bit on the show, but not a ton. So looking forward to getting into that. And I mean, from your guys' perspective.
[00:04:23] Market Hype and Realistic Expectations
[00:04:23] Megan Bowers: Do you think that AI is overhyped? Like is there a ton of market hype around it and you're not seeing as many use cases, or do you feel like it's really something that's been like empowering for the customers you work with?
[00:04:37] Eric Soden: I'll take that one from maybe more the leadership and executive side of things. I think there's, there's a ton of hype and a ton of excitement and a ton of hope that's being put into it, but without a lot of knowledge of what its capabilities are. So it's a very new tool and it, it feels like magic. It's like the first time you use the internet, first time you used an iPhone. Hard to believe that yesterday we didn't have it. And today we do. And the level of excitement is very high. And with that becomes, I think the negative is becomes a lot of hype. That excitement can turn into, we have to do something and we have to do it now and we don't know what we're gonna do. We don't know why we're gonna do it, but we're dammit we're doing something. And I think the. The danger in that is doing the wrong things or investing in the wrong places and not really understanding what is it, what is this thing gonna be good at? What, where could I get value? Just because you're using it doesn't mean it's valuable. You could be, you could spend a hundred thousand dollars implementing something and get $10,000 of value back and say, you're using AI. I don't. I wouldn't applaud you for that. I would say, well, that was a waste of money. Now, at the same time, if you find the right use cases, and if you really focus on. The problem, the challenge, the thing we have to do. And then we say, now let's look at our entire toolbox of which generative AI LLMs are. One of the tools in the toolbox, so is linear regression, logistic regression, kans analysis on like the data science side. So are data prep and blend and SQL and Python and just standard coding techniques and Power BI and Alteryx and all these different tools exist and you shouldn't. Get so excited about your new table saw. You only wanna use table saws going forward because you would make some really weird woodworking projects if the only tool you wanted to use was a table saw. And so I, to me, the danger isn't the technology, it isn't really the hype around the technology. It is that we can become so, so myopically focused, put on the blinders on everything else that's available to us, because darn it, we wanna use the new cool thing. That is a lot of the conversations I'm having is I think, I think there's a lot of leaders who want it for the sake of having it. There's a lot of leaders who are like, how could I possibly use this? I really am excited about it and I feel like it's a game changer. And then there's, there are some who are hesitant and on the wait and see train and saying how much of this is real, how much of it is in it?
And I think that it's dangerous if you're not at least experimenting and using it as personal productivity. And I think there's a danger if you are. Spending millions of dollars without a real challenge or problem or something that you can say, I'm gonna spend a million dollars to get 10 million. That's why I'm spending a million. Wouldn't it be cool if is probably my least favorite project when someone says, alright Eric, I've got an idea for you. Wouldn't it be cool if, and pretty much it's almost like saying, Hey, I had a dream last night where it's like, I'm checked out. I don't really need to know the rest of where this is going because I know it's, I know it's not gonna be real. And so that's my perspective. I think talking to leaders who are trying to figure this out. JT, I guess from the more implementation or technical side, if you wanna grab that. Question.
[00:07:49] JT Morris: Yeah.
[00:07:49] Challenges and Limitations of Generative AI
[00:07:49] JT Morris: And I think you hit it right on the head when you said they're not magic. And I think a lot of people assume that they are. And just like anything else, it's another tool in your toolbox, right? The real value is when you combine those with structured processes and automation platforms like Alteryx, but with real use cases. And so a lots of the conversations I have today with so many people, they, they wanna use AI, they want to begin to incorporate this in their workflows, but especially the danger comes from people who. Probably don't have much of a background in this space, or they just don't really know where you draw the lines on. What is AI, what is ML? What is traditional, just statistics. Some people think they need the latest and greatest large language model, but really what they might need is just a simple forecast, right?
And what we try to do is help people understand that although large language models are great and there's a lot of cool things that they can do, there's a lot of other things that they want to or need to accomplish that can be done with some more traditional methods that have been around for a little. While
[00:08:48] Eric Soden: I hear a lot of people saying, I wanna replace entire roles, entire jobs, and I think that's where we're like, how do you, how do you go into ChatGPT right now and say, go ahead and attach to my payroll system, my ERP system, pull out all accounts payable and match that to purchase orders and make sure that we have enough funding for that, and make sure that the people who are in the payroll system and that their reporting structure are such that they were able to approve that. That's not a query you can issue to ChatGPT. Right now, it doesn't know how to access your ERP system. It doesn't know where in your ERP system this data would be. It doesn't know what your payroll system is. It doesn't know how to access it. If it's on the cloud, it doesn't know API keys. There's so much it doesn't know. That doesn't mean that you couldn't construct a very optimal way of doing that type of analysis at scale and over time using many different methods of which, you know, we may use some coding methods. We may use generative AI for certain things. So that to me right now anyway, is the danger of the hype is when you expect it to do things that are fundamentally, at the moment impossible. Now, five years from now, 10 years from now, maybe it will, maybe we will just, uh, type a query and say, go fetch this out of all my systems, and it'll just go and it knows how to do it. It's just not right now. And so I think that's the danger is be careful with your expectations because. If you expect too much from it, you're gonna be let down if you expect the right stuff from it, and you combine that with the right knowledge and right expertise, you can do some really crazy things that were impossible or virtually impossible just a few years ago.
[00:10:25] Megan Bowers: Totally. I kinda wanna go back to one of the things you said, JT, about people recognizing or having trouble recognizing when something is maybe better suited for statistics or a machine learning model. I'm curious, like what kinds of things make you think, oh, this would be a really good generative AI use case, like this type of data or maybe this type of problem. When is it that those kind of alarm bells are going off that generative AI would be a good fit for that project?
[00:10:52] JT Morris: So there's a couple. I think key triggers that come to mind whenever we're looking for. Okay. Where would be a good fit for generative AI? One is when we are dealing with information overload, it's so many roles involve reading, summarizing, adjusting, making sense of large amounts of text. Right. So like going back to. Emails. This could be financial statements, earnings reports, transcripts, large amounts of texts that we have to be able to read, process, and make sense of and take action on. So that information overload, that's probably number one. Any situation where we need mass personalization. So where we want to make personalized content for an individual. So maybe that's personalized emails, personalized recommendations, right? Or personalized outreach. I mean, you want to try to tailor these things at scale to a lot of different individuals. Maybe that's based on their buying behavior or their buying patterns, right?
Or certain trends and activity that we've been seeing, and we want to be able to do this. Scale. So typically, you know, in the past that you either could go cast a broad net and have a pretty standard approach of doing things, or you can take the time to try to personalize things to your customer or the people that you're working with. But now this just allows you to do that a lot more quickly. And I think another area too is anytime we need to be able to take unstructured information, this could be, uh, data that's sitting in PDF documents, word documents, and need to be able to get that into a more structured format. Those are some pretty key areas that we see coming up quite a bit.
[00:12:28] Megan Bowers: That makes a lot of sense. And then I'm also curious about, are there times when generative AI could create more challenges when you're looking at a project I.
[00:12:38] JT Morris: I
think one of the things you have to look out for is having overconfidence in those outputs. Um, and I think it's similar to, you can't believe everything that you read on the internet. You should take that same approach whenever you're looking at some of these results from a large language model, they're not always right. They can come up with some very convincing. Results. But going back to what Eric was mentioning before, I think we're still at a point where a lot of these use cases, there still needs to be a human in the loop at some point in that process, or at least some standard checks, ways of placing those guardrails to ensure that you can trust those responses that you're getting. Those are things that you, you need to look out for when you're working with these types of models. One is also two, just being able to ground these in information that's hopefully relevant. To your business or what you're doing, right? So these models are typically trained on mountains and mountains of data and have access to lots of information that's out there. But it may not always be information that's exactly relevant to what you're needing or what you are doing. So you want to make sure that the information that is feeding these models is accurate and relevant to you and what you're looking to accomplish. And just to kind of wrap that up to, uh, a lot of it comes down to data quality. This goes back to traditional ML and predictive analytics is that garbage in equals garbage out. So if you're feeding in poor, bad data that's not vetted and has good structure and quality, you're not gonna get great results coming out on the other side. So these are all things that you need to think about and look out for when you're using large language models or beginning to incorporate them into your processes.
[00:14:17] Eric Soden: I've got two, two things to add to that. Well, the one when I think things get dangerous is when you need high position, high accuracy and high repeatability, because every time you ask it a question, it responds differently. It very rarely gives you the same response twice. And so if you're doing things with financial data, there's not a lot of times where it's okay if the number's 5 million or if it's 5.7 million or if it's 8 million, depending upon how you decided to do your analysis. That accuracy and precision and repeatability is something you need to be very careful with because just because you tested it today and it worked, doesn't mean when you test it tomorrow it's gonna work the same way. And is that okay? In many cases, it will be, if it's a, if it's a marketing thing, if it's an email that you're writing, if it's a summarization, it might not make any difference if it does it exactly the same way twice. If you're dealing with things that are high precision, high accuracy, these things can be pretty tricky to get to work. Um, and then the other thing, so you have the foundational models, the ones that we think of like Gemini and OpenAI and and things like that. And then you have of course, your own models or models that you augment these models with. JT mentioned data quality. I think of this a lot like teaching a person and we tell it. We confuse it by giving information that is not correct and we give it historical facts that are not correct and we start giving it documents and contracts and say that we have all these contracts in place, but we don't. How's it supposed to have any idea whether or not what we gave it is? Right? It, oh, it believes what we gave it is, right? Because we're the people who are building it, so it doesn't have the ability to know whether you're feeding it garbage or you're feeding it something great. The difference I think is unlike the dashboard that shows something drastically wrong, I. This just becomes part of the model. It becomes part of the memory of this model, and you can never take it. I, in fact, I don't know how you get it outta there, whether it's right or wrong, it's changed. And how you unprogram that is, is pretty tricky. If you're using RAG or something like that might be a little different because you just remove it from that database. But if you're fine tuning the model or fundamentally modifying the model, once you teach it, it's, it begins
[00:16:29] JT Morris: to believe it. One more thing too is explainability as well. So there's a lot of times when you wanna be able to explain how and why you're getting the results that you're getting. Um, especially if we're looking at something like a forecast or we're looking at why are we going to give this person a loan or not. So when we're having these decisions made based on a prediction of some sort. Although the accuracy of the prediction is important, the ability to explain how it got to that result is almost just as important in a lot of cases. And so sometimes what you see with these models is a little bit of a trade off between. Explainability and precision. Sometimes the more accurate these models are, the harder it is to explain how it exactly got to the results that it's getting. And I think even if you talk to a lot of experts, they still can't always fully explain how the model arrives at the outcome that it's getting and then trying to explain that to a board of directors, right?
So then it can right quickly become very difficult there in, in cases where you need that high level of explainability. Some of those more traditional approaches tend to to work out a little bit better.
[00:17:36] Eric Soden: And I think with all the machine learning stuff, we're worrying about bias. So you start making hiring and firing decisions, or you start making decisions that you could potentially get in trouble or be sued because of, or you start writing legal papers or you start submitting things to the SEC. If they come back and ask how you did this, or why you did it, or what the decision making process was, and you are legally bound to be able to explain that, you can get into a pretty precarious situation with some of this stuff. So I think that's right. I think accuracy, precision, the ability to fully articulate exactly how it came to it. And I think some of the reasoning models are helping with some of that too, because they're telling you how it's coming up with the decision and the types of analysis and research it's doing. But um. There's a lot to think through, and if you're under external audit and they want you to show that you have controls in place where things cannot go off the rails, all of that as, as you automate these things beyond just, again, personal productivity, you've gotta be able to answer that. Whether you're using SQL, Python, Alteryx, Power BI, or generative AI. The auditors need to know how you came to your conclusions, what steps happened, the fact that those steps are repeatable, that they're documented. And I think the external auditors and internal audit and everybody involved has some real interesting governance, learning curves to go through to try to figure out how to control the outputs of these things and the decision making that we're gonna do based on that output.
[00:19:04] Megan Bowers: Yeah, a lot of good considerations that you guys just mentioned. So for our listeners who. In the data field and they have the AI chat bot thing down. They were using it for productivity. They're using it personally. What do you think their next thing to try should be? What should they be doing with generative AI?
[00:19:25] Practical Applications and Future Potential
[00:19:25] JT Morris: Yeah, I think Eric started to touch on it a little bit earlier on, but it's starting to think about these things in batches rather than just in those chat interfaces. I think most people interact, or first exposure to LMS is through chat interface, like chat, pt. I. But it's limited in its scope and its scalability, and it's more focused on the dialogue than being able to really take these things and harness them at scale. So how do we take that next step beyond just those q and a style chatbots and really starting to use these as jitt transformers? That's what they are. At the end of the day, you're using the LLM like an engine where you're taking your input data and using it to generate in an output that is entirely new. So instead of one input resume and get one output, it's one data set can give me a thousand responses or outputs. This is where we've started to use tools like Alteryx to be able to feed in hundreds and thousands of rows of data with a specific request. So maybe it's, I'm working in HR and I need to review a thousand different resumes and summarize some of the key information in these resumes. I can pull all that either structured or unstructured data in. Pass that along to something like OpenAI or Gemini or Claude, provide a prompt of what I want it to do that can be specific by each individual record, and then be able to get all of those results back. And so this allows you to really add a dramatic amount of scale to what you're doing. So this could be. Writing a thousand emails or reviewing a thousand different documents and providing summaries. How do we go from just that one-to-one dialogue interface to really being able to do these things multiple times over and over again?
[00:21:13] Eric Soden: You, you have employee surveys already. You probably, if you're in hr, you probably already work with that data and you might work with it in Alteryx. So now think about how could I take these complex text. Prompts or these complicated many different answers. And then figure out, based on all the answers to all these different questions, is this person likely to quit my company or not? And that would be a question that'd be very difficult to answer using traditional ML or using, certainly using just SQL or something. But it's pretty straightforward with generative AI and I think. Think about something you wish you could do a thousand times. Try to get yourself to be able to do it one time using chat, GPT or Gemini or Claude. And if you can get it to work, then think, okay, if I can pass that same set of prompts and that same data or the same information that I have to this thing, maybe I can do it 5,000 times. Because we have 5,000 employees and they all have this survey, so I don't wanna copy and paste 5,000 times. That's crazy. That's the old school days, right? Start sticking it back in Excel. Like ideally, I'm just gonna process 5,000 of them. And there are things that you do today that you need to do more than once or many times, and you don't wanna just turn generative AI into the next manual process that everybody has to do once a week, once a month, whatever You wanna think, okay, how do I incorporate. The power of generative AI to do this resume review, to do these employee review summaries, to do these product review classifications. How do I do this as part of my process in real time? It can run 24 hours a day. It doesn't matter if there's one, two, or 2000. And try to incorporate it into your Power BI stuff. That's what I've been posting a lot of is incorporating generative AI into your workflows that already exist. Your Alteryx workflows, your UiPath workflows, your Databricks workflows, your Adaptive Planning workflows, et cetera. Use it in a way that isn't necessarily totally transformational. It's just an evolutionary step in what you already do, and I think you'll find really impressive. Immediate results that you might not be the next keynote speaker because of it, but it's real. It's working, it's in production, and there aren't a lot of people who can say that right now.
[00:23:47] JT Morris: Tools like Alteryx kind of remove that need for you to have a super complex infrastructure to build out things like this. You could build a relatively simple workflow that you then set up to run on a schedule. You has it send you notifications, you're combining it with other data that you're working with in your workflows and scheduling it to run, to give you those results on a daily, hourly basis. Uh, and then you can put this in the hands of some non-technical users, folks who are working in HR or finance or operations, um, and they can begin to apply these generative AI methods to things that they're doing on a day to day that I think a lot of people think, oh, I need to be a developer or data scientists to be able to even start thinking about anything like this. But really it makes it a lot more achievable for a lot of different teams.
[00:24:37] Megan Bowers: I really like all the examples and that concept of, of building, building upon existing workflows or building from small use cases or what you think might be small but adds a little productivity. I feel like those are great places to start and I would really love if listeners listening to this. Have ideas or they have those building blocks or they've started, leave a comment. Tell us what you're doing with generative AI or generative AI in Alteryx. Even better. We'd love to just hear some examples from listeners too.
[00:25:09] Maximizing ROI with Generative AI
[00:25:09] Megan Bowers: But yeah, I think a good place to end would be to just chat a little bit about how you guys are seeing ROI or like real value out of generative AI and some of these designer use cases.
[00:25:21] Eric Soden: I, I haven't seen anything personally. Where the ROI is billions of dollars or whatever the things that I've seen are augmentations, adding on new functionality to things that we either wouldn't have been able to do before or that are hyper personalized. Now, like I said, send out a thousand emails, each email personalized to the person you're sending it to. Addressing them, talking about their situation, talking about the products they purchased, talking about the review, they left. And to me that hyper personalization, that stuff is super, super exciting and I think there's a ton of ROI to be had there. I think being able to very quickly summarize and I think on the pro personal productivity thing, I don't know what percent more product productive people who use this a lot are, but it's gotta be 10, 20% depending upon the person. I, I think that's really impactful, especially in aggregate as you think about your entire company. But I think the best is yet to come. Just like when the internet launched, just like when the iPhone launched. The things that happened five years later or 10 years later, as more innovations piled on top of the next, piled on top of the next. People pushed boundaries, figured out new things and tried things. Some of them that worked, some of them that didn't, some of them that they thought worked and then found out later that they didn't probably caused them some pain. There's also, I think there's gonna be a lot more MySpace than Facebook's. The thing that we think is the best thing right now may not be the thing that's the best thing. 10 years from now, and that's okay. And my, my thing is go try things. Do things in a way, don't bet everything on it. Try to figure out how you iterate, how you increment, how you make small gains daily or weekly, and incorporate this into everything you do. And if you have children, I really hope that parents are getting their kids involved and helping them because now this is the next big wave. Your kids and your high school kids, and. You've got to help them figure out how to use this in a way that's obviously safe, but also beneficial for them in their future.
[00:27:26] JT Morris: I would say if you're a leader of a team or a director and you're trying to think about what are some ways that we can be using this, probably the simplest way I could put it is, where is my team spending time reading, writing? Analyzing or rephrasing things, that is really gonna be a bulk of the use cases where it can be applied, and we can go through a million different examples from the emails to reviewing resumes, to creating reports to taking PDF documents and turning them into usable data and insights. But it's going to be a lot of those situations where humans are spending a lot of time. Processing text with their brains, the gray matter, and being able to turn that into something that's usable or actionable. And in these ways, I think it can really make people faster at what they're doing, but without necessarily totally taking people outta loop.
[00:28:20] Conclusion and Final Thoughts
[00:28:20] Megan Bowers: Well, it's been really fun to have you on our show, and thanks so much for sharing all your knowledge.
[00:28:25] Eric Soden: Absolutely. Thanks for having us.
[00:28:27] JT Morris: Great to be here.
[00:28:29] Megan Bowers: Thanks for listening to learn more about Eric and JT and the work they're doing. Head over to our show notes on alteryx.com/podcast. And if you like this episode, leave us a review. See you next time.
This episode was produced by Megan Bowers (@MeganBowers), Mike Cusic (@mikecusic), and Matt Rotundo (@AlteryxMatt). Special thanks to @andyuttley for the theme music track, and @mikecusic for our album artwork.
