Workflow Optimisation - Searching Transcripts for Keywords

Question

Hi there!

I've built a workflow which searches transcripts for certain keywords (attached a watered down example to show my process)

The workflow works perfectly, however we've reached almost a million transcripts now and there's probably more keywords we'd like to search for going forwards

Does anyone have any suggestions on how to optimise/improve this workflow so it won't take a massive amount of processing power/time to run?

Thanks in advance :)

Searching Transcripts for Keywords Example.yxmd

IraWatt · Answer

@CHarrison,

I've attached the example macro workflow:

Should be super efficient hopefully , like @MarqueeCrew said you should be able to compare the workflows efficiency here:

Have to tell us which is the fastest on your real data @CHarrison !

Example.yxzp

IraWatt · Answer

Hey @CHarrison,

I think this is the most efficient and possibly the simplest way. The regex generated from the keywords lets you tokenise just the keywords. Could just put this in a macro and your sorted!

The regex generated makes sure that each word is it is just the full keyword (punctuation is allowed)!

Any questions or issues please ask :)
HTH!
Ira

MarqueeCrew · Answer

@IraWatt ,

In the runtime settings you can look at performance tuning.  But if you AMP the workflow, it runs so fast that I'd be surprised if the tuning will amount to anything.  I think in this case you can afford the "cost" of tokenizing the transcripts with the simplification of the workflow.  I also think that this approach gives better metrics.

Cheers,

Mark