Challenge #205: Taynalysis

A solution to last week’s challenge can be found here!

Taynalysis: Analyzing Taylor Swift Lyrics

Whether you love her or hate her, Taylor Swift is one of the most popular musicians and songwriters of our time and her lyrics have been the source of both criticism and praise.
Her lyrics have also been the subject of a wide spectrum of analyses. Get in on the taynalysis by finding the top 10 words she uses in each album and then find out if her transformation from being a country singer to a pop star has affected the repetiveness of her albums.

Can't get enough of Taylor's lyrics? Take this Weekly Challenge up a notch by adding your own levels of analysis and reporting.

Check out these analyses for some swifspiration:
Text Analysis and Data Visualiztion with Taylor Swift Songs
Using Machine Learning to Analyze Taylor Swift's Lyrics

How Has Taylor Swift's Word Choice Changed Over Time?

giphy

*Note: As of 3-3-20 we uploaded a new start file. We realized the way we handled some grouping options and contractions created the discrepancy some people were experiencing.

challenge_205_start_file.yxzp

challenge_205_solution.yxmd

Parse

Join

Preparation

Data Preparation

Core

Transform

Basic

Accepted answers

All comments

Supergiant

Spoiler

The unique line numbers don't match up but I think the logic still serves its purpose.

challenge_205_JY.N.yxzp

gautiergodard

solution

challenge_205_start_file solution.yxmd

DataKilroy

Spoiler

challenge_205_start_file.yxzp

daiphuongngo

Challenge #205 Taynalysis p2.jpg

Challenge #205 Taynalysis p1.jpg

Challenge #205 Taynalysis.yxmd

JoshuaB

Updating from my old profilel

challenge_205_solution_jb.yxmd

PeterHajnal

...

Solved - challenge_205_start_file.yxmd

TungThanhHo

my solution
a little different to the solution

challenge_205_Thanh_Solution.yxmd

martinding

Spoiler

challenge_205_mysol.yxmd

Jose_Flores22

my attempt

challenge_205_start_file.yxzp

ZacharyRyanTIL

i knew this challenge was trouble when it walked in

challenge_205_start_file.yxmd

gerda

Spoiler

i knew you were trouble when you walked in

challenge_205_GS.yxmd

alexhenson

Spoiler

trevor_harvey

Spoiler

JoachimCaronTIL

Here is my solution

challenge_205_solution_JC.yxmd

alfredcsy

challenge_205_start_file.yxmd

FinnCharlton

Spoiler

challenge_205_complete.yxmd

CammyPhillips

Spoiler

KORKIS3

Spoiler

Simple Solution

DanielG

Solved with very minor descrepancies on a few unique album lines. Without diving in line by line to review every last character, I think my solve accomplishes the basic task at hand. I cant justify spending more time on this than on my actual work at this point. haha.

Glad to be done with Taylor Swift.

challenge_205_finished.yxmd

lucysinclair62

Spoiler

Solution under spoiler tag

olga_strubbe

Spoiler

This was a nice challenge, though a bit confusing, I agree with the others.

KKSzpakowski

Hi thanks for the challenge.
I was able to complete it.

Regards
K

done.bmp

dougbaron

What a pain. I had to manipulate a bunch of data cleansing to get it to come to the expected output. I seemed the numbers would be off by 1 on one album, so I would change cleansing options and it would be corrected but 2 others would then be off by 1. Also, getting the word could was an issue. I found an unidentified character � that it wouldn't let me put in a formula, and it threw off the word count order.

Spoiler

challenge_205_final_file.yxzp

ShankerV

Had the thrill of solving!!!

Spoiler

challenge_205_start_file.yxmd

Ed2000

Spoiler

challenge_205_start_file.yxmd

longyanChen

challenge_205_start_file.yxzp

Dani_Lin

solved

challenge_205_DML_Solution.yxmd

gaoa

This is a fun challenge and I learned many new features of tools! 😉 NOTES in Spoiler below. 💡

Spoiler

1) Lowercase and ReplaceCharacter formula: lowercase(replacechar([lyric],'",.:;?!()',"")): I used Data Cleansing tool at first to remove leading whitespace, punctuation and modify upper case but did not work exactly as expected. I assume the discrepancy is from single quote punctuation getting removed by Data Cleansing tool but not by formula.
2) Text to Column: learned using Delimiter \s to represent space and split to rows. I used Reg Ex to tokenize entire words to rows but as a result some abbreviated words containing single quotes, such as "that's", ended up splitting into two words, "that" and "s". So when counting the top words the result included single letter words like "s", "ll", etc.
3) Summarize: count distinct - useful when counting unique values. The Unique tool returned a different result.
4) Join: left: lyric and right stop_words: this is so smart because the left output shows all lyrics without stop words which is what we want and can proceed to next steps, joined output shows lyrics matching stop words and right output shows stop words "leftovers" that don't match any lyric so these are the data we don't need.
5) Again: Sort -- two sorting orders and using dictionary order; Sample -- first n rows and grouping by column; Summarize -- concatenate string. add space in concat_string -- replacechar([Concat_lyric],","," ").

challenge_205_start_file_gaoa.yxmd

akorte1

Done

Heinrich_Preuß

Spoiler

challenge_205_start_file.yxmd