I need to search text for matches against a list of key words.
Each row of text contains 3 text fields. Text_1 is searched for hits. If a hit is registered, it receives a score of 1 and the entire row passed to an output union tool (with no need to search Text_2 or Text_3). Entries that fail to generate a hit in the Text_1 filter are passed to the Text_2 filter where Text_2 is searched for keywords hits... At the end, I union the results of the 3 filter passes and Score tells me which filter generated the hit.
As currently configured. Score tells me which Text field generated the hit, but I do not know which keyword or keywords generated the hit.
How could you generate an output that tells you which keywords generated the hit -- and as an extra bonus a field that included the 5 words before and 5 words after the keyword?
The sample text below contains the keywords PLANES and unmanned air systems. Other keywords might include tacos, Cervelo, or EW. A text field may generate zero hits; it may generate multiple separate hits; or it may generate multiple hits of the same keyword.
A generic field...
| TEXT FIELD |
| Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod PLANES incididunt ut labore et dolore magna aliqua. Id porta nibh venenatis cras sed felis eget. Amet est placerat in egestas. Sed augue lacus viverra vitae congue. At lectus urna duis convallis convallis. Neque unmanned air systems uisque egestas diam in. Ornare lectus sit amet est placerat in. Elementum nibh tellus molestie nunc non blandit massa. Dictum non consectetur a erat nam at. Cras tincidunt lobortis feugiat vivamus at augue eget arcu. Egestas integer eget aliquet nibh praesent tristique magna sit amet. Elit ullamcorper dignissim cras tincidunt lobortis feugiat vivamus at augue. Molestie a planes at erat pellentesque adipiscing. Sagittis nisl starship kloi rhoncus urna neque viverra justo. Mi proin sed libero enim sed faucibus turpis. |
A real world example where the keyword might be hypersonic
| Similarly, advancements in material design, processing and manufacturing are enabling novel material architectures that can further enhance performance and resilience in structures such as leading edges, windows and apertures, propulsion systems, and space structures. Exemplar areas of research within the Materials for Extreme Environments thrust include the following: 1) high temperature materials for hypersonic platforms; 2) high temperature window and aperture materials; 3) radiation and/or electromagnetic pulse (EMP) hardened electronics for space platforms; and 4) coatings for platform survivability in corrosive environments. |
Ya know @hellyars, in the Biz, this is known as scope creep😋
Key words are now key phrases. "Extreme thrust" is there to show the negative case of the words in the text being present, but not contiguous.
There's still work to be done on this. For instance, it doesn't handle partial and full matches of the same key phrase. But, as my advanced differential equations teacher was fond of saying after putting the trivial substitutions into the equations, "The rest is left as an exercise for the student"
Dan
@hellyars, the 5-words-before-and-after requirement immediately made me think of regex (although for a non-regex solution, I'd use the same method @danilang shared).
Full regex string = '.*?((?:\w+\W+){0,5})(keyword1|keyword2|keyword3|etc)((?:\W+\w+){0,5}).*
- Here, I'm using \w and \W which mean "word" and "non-word" characters. In regex, a "word" character is a letter or number (and a "non-word" character is everything else, such as spaces and punctuation).
- .*? = match any character, any number of times
- capturing group #1: ((?:\w+\W+){0,5}) = match one or more word characters (\w) followed by one or more non-word characters (\W); find that pattern 0-5 times. The ?: means I'm just using parentheses to group the \w+\W+ pattern together, but don't actually want to capture each individual match; I only want to capture the collection of 5.
- capturing group #2: (keyword1|keyword2|keyword3|etc) = match one of the phrases in the keyword group (the vertical bar = "or")
- capturing group #3: ((?:\W+\w+){0,5}) = match one or more non-word characters, then one or more word characters; do that 0-5 times
- .* = match any character, any number of times
This might need some tweaking as well, depending on your specific requirements. For example, this method counts the numbered bullets in text #1 as "words" (since they are numbers), and it would count a hyphenated word as multiple words ("not-so-distant" in the third text counts as 3 words).
Results (I had a little fun highlighting the found keyword):
