```python
import pandas as pd
import ms3
from  ms3.utils import *
import os
from ms3 import Score
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from scipy.spatial import distance
import numpy as np
from scipy import stats
import pretty_midi
import statistics
import torch.nn
from torch import nn
from torch.optim import SGD

# include directory
home_dir = '/home/nulpe/Desktop/Tresillo/'
```

```javascript
%%javascript
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}
```

    <IPython.core.display.Javascript object>

# Table of contents

1.  [Towards a Rhythmical Definition of the Tressilio beat and a Tracing of it in Popular Music](#Towards-a-Rhythmical-Definition-of-the-Tressilio-beat-and-a-Tracing-of-it-in-Popular-Music)
    1.  [Introduction & Research Question](#1\)-Introduction-&-Research-Question)
        1.  [Research question:](#1.1\)-Research-question:)
        2.  [Assumptions](#1.2\)-Assumptions)
        3.  [Data Representation](#1.3\)-Data-Representation)
    2.  [Methods](#2\)-Methods)
        1.  [Definition of the Tresillo rhythm](#2.1\)-Definition-of-the-Tresillo-rhythm)
        2.  [Rhythm histograms and vectors](#2.2\)-Rhythm-histograms-and-vectors)
        3.  [Evaluating different similarity measures](#2.3\)-Evaluating-different-similarity-measures)
            1.  [Rhythm similarity measured with cosine similarity, using tresillo template as centre.](#2.3.1\)-Rhythm-similarity-measured-with-cosine-similarity,-using-tresillo-template-as-centre.)
            2.  [Rhythm similarity measured with cosine similarity, using the centroid of tresillo songs as centre.](#2.3.2\)-Rhythm-similarity-measured-with-cosine-similarity,-using-the-centroid-of-tresillo-songs-as-centre.)
            3.  [Rhythm similarity measured with parameterized cosine simularity, using tresillo template as centre.](#2.3.3\)-Rhythm-similarity-measured-with-parameterized-cosine-simularity,-using-tresillo-template-as-centre.)
            4.  [Rhythm similarity measured with parameterized cosine similarity, using the centroid of tresillo songs as centre.](#2.3.4\)-Rhythm-similarity-measured-with-parameterized-cosine-similarity,-using-the-centroid-of-tresillo-songs-as-centre.)
    3.  [Final Results](#3\)-Final-Results)
        1.  [Onset histograms and rhythm vectors](#3.1\)-Onset-histograms-and-rhythm-vectors)
        2.  [Tresilio-ness with Cosine Similarity](#3.2\)-Tresilio-ness-with-Cosine-Similarity)
        3.  [Different models to find similarity with tresillo.](#3.3\)-Different-models-to-find-similarity-with-tresillo.)
            1.  [Rhythm similarity measured with cosine similarity, using the tresillo template as centre.](#3.3.1\)-Rhythm-similarity-measured-with-cosine-similarity,-using-the-tresillo-template-as-centre.)
            2.  [Rhythm similarity measured with cosine similarity, using the centroid of tresillo songs as centre.](#3.3.2\)-Rhythm-similarity-measured-with-cosine-similarity,-using-the-centroid-of-tresillo-songs-as-centre.)
            3.  [Rhythm similarity measured with parameterized cosine similarity, using tresillo template as centre.](#3.3.3\)-Rhythm-similarity-measured-with-parameterized-cosine-similarity,-using-tresillo-template-as-centre.)
            4.  [Rhythm similarity measured with parameterized cosine similarity, using the centroid of tresillo songs as centre.](#3.3.4\)-Rhythm-similarity-measured-with-parameterized-cosine-similarity,-using-the-centroid-of-tresillo-songs-as-centre.)
        4.  [Tresilio-ness over Time](#3.4\)-Tresilio-ness-over-Time)
    4.  [Outlook on final interpretation](#4\)-Outlook-on-final-interpretation)
    5.  [References](#References)

# Towards a Rhythmical Definition of the Tressilio beat and a Tracing of it in Popular Music

## 1) Introduction & Research Question

### 1.1) Research question:
"Can we compute to which extent a defined rhythm, which we refer to as Tresillo rhythm, is used in a given pop song and if so can we measure the intensity of Tresillo rhythm use in top-20-billboard songs over the past twenty years?"

**Discussion:**
In our project we would like to discuss the use of a rhythm, which we refer to as 'Tresillo rhythm', in the popular music of the last 20 years. We define this rhythm in our project, given secondary literature, and thus obtain a precise notation and formulization of the Tresillo rhythm. Given this definition we can then compute the similarity between the Tresillo rhythm and the rhythm of a given pop song. Thus, we hope to obtain a similarity coefficient which measures how similar the rythm of a given pop song is to our self defined Tresillo rythm. Given the computed similarity coefficients, we hope to measure the use of the Tresillo rhythm in the top 20 billboard songs of the past 20 years (1999-2019).

### 1.2) Assumptions

- The pop songs found in the billboard charts mostly come with a simple melodic and rhythmic structure without a lot of variation troughout the song. We assume that for most pop songs present in the billboard charts, one can identify one dominant rhythm per song.
- For our presntation of songs in rhythm space, we also assume that the majority of the songs in the billboard charts is in the time signature of 4/4. Our data exploration justifies this assumption. Nine out of 442 songs in the dataset are in 3/4 or 6/8, those are not considered in the analysis.
- We assume that a time sample of about 30 seconds is suitable to present the main rythm of a pop song. This assumptions is related to the difficultys of finding puplic available midi data for recent pop music. While we couldn't find a data source providing the midi data of the full songs, we managed to find midi data for shorter samples of the songs. The average sample length is 44.1 sec.

### 1.3) Data Representation
Initially our data is represented in the MIDI file format. The representation of music in the MIDI format has the advantage, that often several voices of different instruments are represented in such files.
To obtain a list of onsets of every musical event in a given song, we have to convert our MIDI (.midi) files to Musescore (.mscx) files.
To convert and further analyze our files, we will use the [ms3](https://pypi.org/project/ms3/) python library.
To convert a directory of .midi files to .mscx files we use following command:

```python
# Time intensive task
path_midi = home_dir + 'dataset/project_midi/tresillo/'
target = home_dir + 'dataset/project_mscx/mscx_tresillos_billboard/'

dir_list = os.listdir(path_midi)

for el in dir_list:
    convert(path_midi+el, target+el[:-4]+'.mscx', MS='musescore3')
```

## 2) Methods

### 2.1) Definition of the Tresillo rhythm
We notate a clean version of a rhythm wich we from now on use as our defintion for the clean tresillo rhythm, in the following context also called tresillo template(see also (Floyd, 1999)).

The rhythm pattern consists of a dotted eighth note, followed by a sixteenth note, an eighth rest and an eighth note and is repeated two times in a 4/4 bar. The rytm pattern is beeing used as a own rythm or as a rythmic part of a more complex rythm, for example the "clave" pattern or the ragaetton rythm (Toussaint, 2004).
By notating the rythm in MuseScore 3 and saving it as a .mscx file, we can use it our data processing pipeline described in the following sections.

### 2.2) Rhythm histograms and vectors

To be able to measure the similarity between rhythm we must have a clear definition and thus following representation of rhythm. In general, one can define rhythm as "a series of onsets and durations of musical events.” (Rohrmeier, 2020). In our specific case however, we are interested in the dominat and repeating rhythm of a given song. Therefore we prefer a narrower definition of rhythm as “repeated recurrences in alternate heavy and light beats” (Chin and Wu, 1992). To furthermore simplify our data, we assume that the main rhythm of a song can be defined by the onsets  of its musical events (notes).
To obtain a representation of the dominate rhythm of a song, we preceed to aggregate all musical onset to one bar. Collapsing all musical onsets to one bar and thus obtaining onset 'histograms' is a common pratice and has been used beside others to analyze western classical music (Palmer and Krumhansl, 1990) and american folk music (Huron and Ommen, 2006).
With the onset histogram of a song we can compile a n dimensional vector for each song, which we refer to as a 'rhythm vector'. Given that the number of songs with meters other than 4/4 is negible, we only consider songs with a 4/4 meter in our analysis.  Given that we only consider songs with 4/4 meters, we obtain for each song a 16 dimensional vector.

### 2.3) Evaluating different similarity measures
In this section we try out different similarity measures to choose the best one. A good similarity measure would have a high similarity for all songs which have a tresillo pattern and a low similarity for songs which do not have such a pattern.
To measure this, we use 'Similarity Goodness' $S^*$, which is the ratio of mean similarity in songs which have tresillo and mean similarity of songs which don't.
\begin{equation}
S^* = \frac{\frac{\sum{\text{similarity(Songs with tresillo)}}}{n}}{\frac{\sum{\text{similarity(Songs without tresillo)}}}{m}}
\end{equation}
`n` and `m` denote the number of songs in each of respective categories.

In each of the subsections we calculate relavant statistics by dividing the similarity methods based on the following criterions.
* Template based center vs Centroid computed from known tresillo songs
* Parameterized vs non parameterized distance function

In each subsection in section 3, we will compute the mean similarity and it's standard deviation.

At the end of the section we compare the $S^*$ (Model Goodness) for all of the above subsections.

_NOTE: Both the datasets used to evaluate $S^*$ are from the validation/test set and not used as part of training/finding the centroid._

#### 2.3.1) Rhythm similarity measured with cosine similarity, using tresillo template as centre.
Given the 16 dimensional rhythm vectors we obtain following the methode described above, we can now compute simple similarity metrics.
In rhythm analysis a common similarity metric which is used to calculate the similarity between two rhythm vectors is the cosine distance (see: Panteli et al., 2014; Parry and Essa, 2003).  The cosine similarity metric is scale invariante, which as such is interesting for rhythm similarity given that thus only relative frequencies of onsets are important and not absolute frequencies.
The cosine distance between two vectors A and B is defined as following:
\begin{equation}
\cos ({\bf A},{\bf B})= {{\bf A}*{\bf B} \over \|{\bf A}\| \|{\bf B}\|} = \frac{ \sum_{i=1}^{n}{{\bf a}_i{\bf b}_i} }{ \sqrt{\sum_{i=1}^{n}{({\bf a}_i)^2}} \sqrt{\sum_{i=1}^{n}{({\bf b}_i)^2}} }
\end{equation}

Given the definition of the cosine similarity we can now compute the similarity between our self defined Tresillo rhythm and the billboard songs.
First, however we will validate this similarity metric by testing it on our self compiled list of songs which do comprise a Tresillo rhythm and songs which do not comprise a Tresillo rhythm. We then compute the mean 'Tresillo-ness' (similarity to Tresillo rhythm) of both samples. By employing the Bootstrapping method we can also obtain a measurement of uncertainty, as provided by 2.5% and 97.5% confidence intervals.

#### 2.3.2) Rhythm similarity measured with cosine similarity, using the centroid of tresillo songs as centre.

We compute the centroid by taking the mean of all the tresillo song vectors. This centroid is then used as the centre of `tresillo-ness` in the model. The cosine similarity is the same as 2.3.1.

\begin{equation}
\text{centroid}_i = \frac{\underset{j: \text{1 to n}}\sum{A_{i}^{j}}}{n}
\end{equation}

Where,
* $\textbf{centroid}_i$ is the ith dimension of the centroid.
* $A^j$ is the $j^{th}$ song

#### 2.3.3) Rhythm similarity measured with parameterized cosine simularity, using tresillo template as centre.

In 2.3.5 and 2.3.6, we define a parameterized model (Zhihua Zhang, 2003) for computing the similarity between tresillo and the tresillo-ness center.
We define the parameterized cosine similarity
\begin{equation}
\cos_{\Theta} ({\bf A},{\bf B})= {{\bf A_{\Theta}}*{\bf B_{\Theta}} \over \|{\bf A_{\Theta}}\| \|{\bf B_{\Theta}}\|} = \frac{ \sum_{i=1}^{n}{{\bf a}_i.{\theta_i}{\bf b}_i.{\theta_i}} }{ \sqrt{\sum_{i=1}^{n}{({\bf a}_i.{\theta_i})^2}} \sqrt{\sum_{i=1}^{n}{({\bf b}_i.{\theta_i})^2}} }
\end{equation}

We learn this theta by optimizing the over the objective function
\begin{equation}
\underset{\Theta}{\mathrm{argmin}} \frac{cos_{\Theta}(A', T)}{cos_{\Theta}(A, T)}
\end{equation}

Where,
* **A'** is the set of songs with tresillo __not__ present in them
* **A** is the set of songs with tresillo present in them
* **T** is the center point for tresilloness. In this case it is the vector corresponding to the template tresillo vector described in 3.1
* $\textbf{cos}_{\Theta}$(*) is the parameterized cosine similarity

This is a linear transformation of the rhythm space which is expanding/contracting each axis of this space in order to have the best model goodness.

We use the validation set for both A' and A to avoid overfitting in the model.

#### 2.3.4) Rhythm similarity measured with parameterized cosine similarity, using the centroid of tresillo songs as centre.

In this section we combine the method used in 2.3.2 and 2.3.3, i.e. use a parameterized cosine similarity with centroid as the tresillo center.

T = Centroid as defined in 2.3.2

## 3) Final Results

### 3.1) Onset histograms and rhythm vectors
In this first part we will use onset histograms to compute rhythm vectors.
To obtain the onset histogram of a given song, we use the notes representation provided by the [ms3](https://pypi.org/project/ms3/) libary and colapse all musical onsets to one bar. In the example below we will compile the histograms for our self defined 'Vanilla Tresillo' and for the example song 'shape of you' by Ed Sheran. Then we will proceed to compute the rhythm vectors for both songs

```python
#paths to both examples
shape_of_you = home_dir+'dataset/project_mscx/mscx_tresillos_billboard/Shape of you-Ed Sheran.mscx'
vanilla_tresillo = home_dir+'dataset/project_mscx/mscx_tresillos/Vanilla_Tresillo.mscx'

# get the note scores of both examples
df_shape_of_you = Score(shape_of_you).mscx.notes
df_vanilla_tresillo = Score(vanilla_tresillo).mscx.notes

#calculate quarter note position for each note
df_shape_of_you['quarter_beats'] = (df_shape_of_you.mc_onset*16).astype('int32')
df_vanilla_tresillo['quarter_beats'] = (df_vanilla_tresillo.mc_onset*16).astype('int32')

fig, ax = plt.subplots(1,2, figsize=(12,3))

ax[0].hist(df_shape_of_you['quarter_beats'], bins=16)
ax[1].hist(df_vanilla_tresillo['quarter_beats'], bins=16)
ax[0].set_xlabel('quarter_beats')
ax[1].set_xlabel('quarter_beats')
ax[0].xaxis.set_ticks(np.arange(0, 16, 1))
ax[1].xaxis.set_ticks(np.arange(0, 16, 1))
ax[0].set_ylabel('count')
ax[0].set_title('Shape of you')
ax[1].set_title('Synthetic Tresillo')

#mention somewhere that we are working with quarter notes
df_shape_of_you
plt.savefig('plots/onset_histogram_example.png', dpi=160)
```

In a next step we want to compile rhythm of each song given this notion of histograms. E.g: every dimension incorporates the absolute frequency of onsets on one given quarter note. This is done as follows:

```python
rhythm_vector_shape_you = df_shape_of_you.groupby(['quarter_beats'])['mn'].agg(['count'])
rhythm_vector_shape_you  = rhythm_vector_shape_you.reindex(list(range(0,16)),fill_value=0).T
rhythm_vector_shape_you
```

      quarter_beats
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15

      count
      206
      33
      44
      119
      72
      37
      169
      12
      200
      29
      75
      112
      75
      27
      167
      10

In the assumption that the rhythm vectors of distinct voices within a song might include information we want to preserve, we compiled one rhythm vector per instrument in a song as follows:

```python
# Define instruments
shape_of_you_score = Score(shape_of_you)
instrument_dict = {}
for key in shape_of_you_score.mscx.metadata['parts']:
    for staff in shape_of_you_score.mscx.metadata['parts'][key].keys():
            instrument_dict[staff] = key

#staff to voice/instruments
df_shape_of_you['instrument'] = [instrument_dict[el] if el in instrument_dict else 'na' for el in df_shape_of_you.staff]

#compute rhythm vectors per voice
rhythm_vector_shape_you_instruments = df_shape_of_you.groupby(['instrument','quarter_beats'])['mn'].agg(['count'])
rhythm_vector_shape_you_instruments = rhythm_vector_shape_you_instruments.groupby(level=0).apply(lambda x: x.reset_index(level = 0).drop(['instrument'],axis=1).reindex(list(range(0,16)),fill_value=0).T)
rhythm_vector_shape_you_instruments = rhythm_vector_shape_you_instruments.reset_index()
rhythm_vector_shape_you_instruments
```

      quarter_beats
      instrument
      level_1
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15

      0
      Grand Piano
      count
      66
      0
      0
      0
      0
      0
      0
      0
      66
      0
      0
      0
      0
      0
      0
      0

      1
      Marimba, untitled
      count
      66
      0
      0
      53
      14
      0
      63
      0
      68
      0
      0
      57
      17
      0
      63
      0

      2
      Melodic Drum
      count
      0
      0
      33
      36
      13
      19
      39
      0
      0
      0
      50
      28
      10
      19
      38
      0

      3
      Overdrive Gtr
      count
      6
      2
      1
      1
      4
      4
      6
      2
      6
      1
      6
      2
      6
      1
      6
      0

      4
      Percussion
      count
      24
      23
      0
      2
      19
      2
      22
      2
      21
      21
      1
      2
      19
      2
      23
      2

      5
      Tenor Sax
      count
      20
      8
      10
      8
      16
      12
      16
      8
      16
      7
      18
      6
      15
      5
      14
      8

      6
      Woodblock
      count
      2
      0
      0
      2
      0
      0
      2
      0
      2
      0
      0
      1
      2
      0
      2
      0

      7
      Xylophone
      count
      22
      0
      0
      17
      6
      0
      21
      0
      21
      0
      0
      16
      6
      0
      21
      0

If we want to compile rhythm vectors (per voice) for all mscx files in one directory we can use following loop:

```python
# NOTE: Time intensive task
def rythm_vectors(in_dir, out_dir):
    list_sheet_music = os.listdir(in_dir)
    df_rythm_vectors =[]

    for idx, el in enumerate(list_sheet_music):
        if el[-4:] == 'mscx':

            #Get notes with onsets
            s = Score(dir_sheet_music+el)
            df = s.mscx.notes

            # Define instruments
            instrument_dict = {}
            for key in s.mscx.metadata['parts']:
                for staff in s.mscx.metadata['parts'][key].keys():
                    instrument_dict[staff] = key

            #staff to instruments
            df['instrument'] = [instrument_dict[el] if el in instrument_dict else 'na' for el in df.staff]

            # define quarter beat
            df['quarter_beats'] = (df.mc_onset*16).astype('int32')

            #make rythm matrix & data frame
            df_histogram = df.groupby(['instrument','quarter_beats'])['mn'].agg(['count'])
            df_histogram = df_histogram.groupby(level=0).apply(lambda x: x.reset_index(level = 0).drop(['instrument'],axis=1).reindex(list(range(0,16)),fill_value=0).T)
            df_histogram = df_histogram.reset_index()

            df_histogram.insert(loc=0, column='song_artist', value=el[:-5])

            #concat to big rythm vector df
            if len(df_rythm_vectors) == 0: df_rythm_vectors = df_histogram

            df_rythm_vectors = pd.concat([df_rythm_vectors,df_histogram], axis=0)

    df_rythm_vectors.to_csv(out_dir, index = False)

dir_sheet_music = home_dir + '/dataset/project_mscx/mscx_billboard/'
out_dir = home_dir + '/dataset/rythm_vectors/rythm_vectors_billboard.csv'
rythm_vectors(dir_sheet_music, out_dir)
```

### 3.2) Tresilio-ness with Cosine Similarity
In a first naive analysis we will employ the cosine similarity measurement to assess if the vector of a given song is similar to our defined Tresillo rhythm. We will need some helper functions.
Following two helper functions will help us to collapse all rhythms of all instruments to one rhythm vector per song.
The second function calculates the cosine similarity between a pandas data frame of rhythm vectors and one single rhythm vector.
The third function calculates the 2.5% and the 97.5% confidence intervals of the distribution of a mean of a given data set. This function will allow us to assess, how big the uncerntaingty is in our data set and if the means of two distributions are indeed significantly different.

```python
# Calculate and add the cosine similarity to the dataframe
def calc_cosine_sim(rythm_vectors, tresillo_vector):
    rythm_vectors['cosine_sim_tresillo'] = cosine_similarity(rythm_vectors.loc[:,"0":"15"],tresillo_vector)
    return rythm_vectors

# Calculate and add the cosine similarity using the trained model, to the dataframe.
def calc_cosine_sim_model(rythm_vectors, tresillo_vector, model):
    rythm_vectors['model_cosine_sim_tresillo'] = model.similarity(rythm_vectors.to_numpy()[:,:16],tresillo_vector).cpu().detach().numpy()
    return rythm_vectors

# Collapse all voices into a single voice by taking the mean over the respective beat.
def collapse_normalize_vectors(df_rythm_vectors):
    rythm_vector_collaped = df_rythm_vectors.groupby(['song_artist']).agg(['sum'])
    rythm_vector_collaped.columns = rythm_vector_collaped.columns.droplevel(1)
    rythm_vector_collaped = rythm_vector_collaped.drop(['instrument', 'level_1'],axis=1)
    rythm_vector_collaped["sum"] = rythm_vector_collaped.sum(axis=1)
    rythm_vector_collaped = rythm_vector_collaped.loc[:,"0":"15"].div(rythm_vector_collaped["sum"], axis=0)
    return rythm_vector_collaped

def bootstrap_CI(data, nbr_draws):
    means = np.zeros(nbr_draws)
    data = np.array(data)

    for n in range(nbr_draws):
        indices = np.random.randint(0, len(data), len(data))
        data_tmp = data[indices]
        means[n] = np.nanmean(data_tmp)
    return [np.nanpercentile(means, 2.5),np.nanpercentile(means, 97.5)]
```

Now lets read in the rhythm vector of our Tresillo template.

```python
df_synt_tresillo = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillio.csv') # read in all defined 'tresillos' and variations
df_vanilla_tresillo = df_synt_tresillo[df_synt_tresillo['song_artist']=='Vanilla_Tresillo'].loc[: ,"0":"15"] # only use the 16d vector of our vanilla_tresillo
vector_vanilla_tresillo = np.asarray(df_vanilla_tresillo)
vector_vanilla_tresillo
```

    array([[20,  0,  0, 20,  0,  0, 20,  0, 20,  0,  0, 20,  0,  0, 20,  0]])

We will now calculate the cosine similarity between our Tresillo template vector and between a hand selected set of Tresillo songs.
This set of songs has been selected by us ourself and none of those 'validation' songs  are included in the billboard data sets.

```python
tresillo_test_set_vectors = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillo_not_billboard.csv')
#tresillo_test_set_vectors = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillos_billboard.csv')
tresillo_test_set_vectors = collapse_normalize_vectors(tresillo_test_set_vectors)
tresillo_test_set_vectors = calc_cosine_sim(tresillo_test_set_vectors, vector_vanilla_tresillo)
print(tresillo_test_set_vectors['cosine_sim_tresillo'])
```

    song_artist
    Chandelier Sia                             0.605698
    Cold Water Major Lazer                     0.896750
    Hips Don't Lie Shakira                     0.819003
    I Don't Care Ed Sheeran & Justin Bieber    0.876542
    One Dance - Drake & Wizkid, Kyla Reid      0.717454
    Sorry Justin Bieber                        0.853424
    Name: cosine_sim_tresillo, dtype: float64

Let's calculate the mean Tresillo-ness and also the 2.5% and 97.5% confidence intervals of the mean as obtained by bootstrapping.

```python
print('mean Tresillo-ness in the test set: ', tresillo_test_set_vectors['cosine_sim_tresillo'].mean())
lower_tresillo_ci, upper_tresillo_ci = bootstrap_CI(tresillo_test_set_vectors['cosine_sim_tresillo'], 100)
print('tresillo upper and lower ci on 100 draws: ', lower_tresillo_ci, upper_tresillo_ci)
```

    mean Tresillo-ness in the test set:  0.7948119135522188
    tresillo upper and lower ci on 100 draws:  0.7168127641658357 0.8594804989254289

Let's do exact same thing, but with a comparison data set of songs which we kno to not include any Tresillo songs.
Also calculating the mean and the confidence intervals of the not Tresillo songs.

```python
non_tresillio_vectors =  pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_not_tresillo_validation.csv')
#non_tresillio_vectors = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_no_tresillos_4_4.csv')

non_tresillio_vectors = collapse_normalize_vectors(non_tresillio_vectors)
non_tresillio_vectors = calc_cosine_sim(non_tresillio_vectors, vector_vanilla_tresillo)
print(non_tresillio_vectors['cosine_sim_tresillo'])

print('mean Tresillo-ness in the test set: ', non_tresillio_vectors['cosine_sim_tresillo'].mean())
lower_non_tresillo_ci, upper_non_tresillo_ci = bootstrap_CI(non_tresillio_vectors['cosine_sim_tresillo'], 100)
print('tresillo upper and lower ci on 100 draws: ', lower_non_tresillo_ci, upper_non_tresillo_ci)
```

    song_artist
    Dress You Up           0.666450
    Fear of the Dark       0.689443
    Folsom Prison Blues    0.657568
    Immigrant Song         0.584275
    Jailhouse Rock         0.593940
    One Step Closer        0.523399
    Sweet Child O'Mine     0.724700
    Sweet Home Alabama     0.534338
    Name: cosine_sim_tresillo, dtype: float64
    mean Tresillo-ness in the test set:  0.6217641548092555
    tresillo upper and lower ci on 100 draws:  0.5751519746156368 0.6632489551053067

Already looking at the means of the two samples and their confidence intervalls, they seem to be significantly different.
However let's also calculate the t-test statistics to ensure that the distributions are actually different.

```python
model_goodness_1 = tresillo_test_set_vectors['cosine_sim_tresillo'].mean()/non_tresillio_vectors['cosine_sim_tresillo'].mean()
```

```python
ttest_pvalue = stats.ttest_ind(tresillo_test_set_vectors['cosine_sim_tresillo'], non_tresillio_vectors['cosine_sim_tresillo']).pvalue
print('p value that the means are the same: ', ttest_pvalue)
```

    p value that the means are the same:  0.004412600322146463

**Discussion:**
Looking at the the Tresillio cosine similarity metrics of both data sets and comparing the distributions of the means to each other (confidence intervals and t-test), we see that their mean Tresillo-ness is indeed significantly diffferent.
However, we also see that 1) this Tresillo-ness measurement seems to be quite noisy 2) The distribution of the two samples are not as far apart as we would like them to be.
The noisiness of this metric, can be especially seen, if one looks at the individual cosine Tresillo-ness values of the songs. Here we see that there is great variance in Tresillo-ness in the hand selected Tresillo data set. Values can be as small as 0.6, which corresponds to the value of a none Tresillo song.
Furthermore, it seems that also songs wich we classified to have no Tresillo, have a cosine Tresillo-ness up to 0.72.
In general we can state that the cosine similarity methods seems to identify a ceratin Tresillo-ness, however it is questionable how robust this method is.

### 3.3) Different models to find similarity with tresillo.
In the previous section we explored the one of consine similarity to visualize the similarity of known songs. In this section we will argue about the other possible metrics for doing the same task. More about this has been described in section 2.3

```python
# Loading necessary data
df_tresillo_not_billb = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillo_not_billboard.csv')
df_not_tre_validation = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_not_tresillo_validation.csv')
df_rythm_vectors_train = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_no_tresillos_4_4.csv')
df_rythm_vectors = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillos_billboard.csv')

tresillo_train = collapse_normalize_vectors(df_rythm_vectors)
tresillo_test = collapse_normalize_vectors(df_tresillo_not_billb)
df_not_tre_validation = collapse_normalize_vectors(df_not_tre_validation)
non_tresillio_vectors = collapse_normalize_vectors(df_rythm_vectors_train)

np_tresillo_train = tresillo_train.to_numpy()
np_tresillo_test = tresillo_test.to_numpy()
np_not_tre_validation = df_not_tre_validation.to_numpy()
np_non_tresillo = non_tresillio_vectors.to_numpy()

df_synt_tresillo = pd.read_csv(home_dir + '/dataset/rythm_vectors/rythm_vectors_tresillio.csv')
df_synt_tresillo = df_synt_tresillo[df_synt_tresillo['song_artist']!='Raggetone'].drop(['instrument','level_1'], axis=1)
df_synt_tresillo.index = df_synt_tresillo.song_artist
df_synt_tresillo.drop(['song_artist'], axis=1)
df_synt_tresillo["sum"] = df_synt_tresillo.sum(axis=1)
df_synt_tresillo = df_synt_tresillo.loc[:,"0":"15"].div(df_synt_tresillo["sum"], axis=0)
vanilla_tresillo_vector = np.asarray(df_synt_tresillo.loc['Vanilla_Tresillo',"0":"15"]).reshape(-1, 1).T
```

#### 3.3.1) Rhythm similarity measured with cosine similarity, using the tresillo template as centre.
In the previous section 3.2 we used the tresillo template as the reference point to compute similarity with. We can compute the model goodness by taking the ratio of similarity of tresillo songs to songs which don't have tresillo.

```python
print(f"Model Goodness: {model_goodness_1}")
```

    Model Goodness: 1.2783173610193896

#### 3.3.2) Rhythm similarity measured with cosine similarity, using the centroid of tresillo songs as centre.

```python
centroid = tresillo_train.sum(axis=0)/tresillo_train.shape[0]
centroid = np.array(centroid).reshape(1,-1)
sim_present_train = cosine_similarity(tresillo_train, centroid)
```

```python
sim_present = cosine_similarity(np_tresillo_test, centroid)
print(f"Mean and Standard dev of Similarity for tresillo set: {round(np.mean(sim_present), 4)} +- {round(np.std(sim_present), 4)}")
sim_not_present = cosine_similarity(np_non_tresillo, np.array(centroid).reshape(1,-1))
print(f"Mean and Standard dev of Similarity for NOT-tresillo set: {round(np.mean(sim_not_present), 4)} +- {round(np.std(sim_not_present), 4)}")
model_goodness_2 = round(np.mean(sim_present) / np.mean(sim_not_present), 4)
print(f"Model Goodness: {model_goodness_2}")
```

    Mean and Standard dev of Similarity for tresillo set: 0.9335 +- 0.0344
    Mean and Standard dev of Similarity for NOT-tresillo set: 0.917 +- 0.0181
    Model Goodness: 1.0179

**Discussion:**

Using the centroid of known tresillo songs as a reference point decreases the gap between songs with and without tresillo and hence it will not be a good measure to analyse the billboard dataset with. The mean similarity in both the sets is also very similar.

#### 3.3.3) Rhythm similarity measured with parameterized cosine similarity, using tresillo template as centre.

```python
class ParameterizedDistance(nn.Module):
    def __init__(self, theta, device, distance_function):
        super().__init__()
        self.device = device
        if len(theta.shape) == 1:
            theta = theta.reshape(1, -1)
        self.theta = torch.nn.Parameter(torch.from_numpy(theta), requires_grad=True).to(device)
        self.distance_function = distance_function

    def forward(self, tresillo_vectors, not_tresillo_vectors, vanila_tresillo_vector):
        assert isinstance(tresillo_vectors, np.ndarray)
        assert isinstance(not_tresillo_vectors, np.ndarray)
        assert isinstance(vanila_tresillo_vector, np.ndarray)
        assert self.theta.shape[1] == tresillo_vectors.shape[1] == not_tresillo_vectors.shape[1] == \
               vanila_tresillo_vector.shape[1]

        not_tresillo_vectors = torch.from_numpy(not_tresillo_vectors).to(self.device)
        tresillo_vectors = torch.from_numpy(tresillo_vectors).to(self.device)
        vanila_tresillo_vector = torch.from_numpy(vanila_tresillo_vector).to(self.device)

        parameterized_vector_not_tresillo = self.theta * not_tresillo_vectors
        parameterized_vector_tresillo = self.theta * tresillo_vectors
        parameterized_vector_vanilla = self.theta * vanila_tresillo_vector

        cosine_similarity_not_t = torch.mean(
            self.distance_function(parameterized_vector_not_tresillo, parameterized_vector_vanilla))
        cosine_similarity_t = torch.mean(
            self.distance_function(parameterized_vector_tresillo, parameterized_vector_vanilla))

        assert cosine_similarity_t.cpu().detach().numpy() != 0, "0 Similarity between Tresillo set and Vanilla-Tresillo Beat"

        return cosine_similarity_not_t / cosine_similarity_t

    def similarity(self, x, y):
        assert isinstance(x, np.ndarray)
        assert isinstance(y, np.ndarray)
        assert self.theta.shape[1] == x.shape[1] == y.shape[1]

        x = torch.from_numpy(x).to(self.device)
        y = torch.from_numpy(y).to(self.device)

        x = self.theta * x
        y = self.theta * y
        return self.distance_function(x, y)
```

```python

```

```python
device = torch.device('cpu')
model1 = ParameterizedDistance(np.random.rand(1, 16), device, torch.cosine_similarity)
optim = SGD(model1.parameters(), lr=1e-3, weight_decay=1e-4)
optim.zero_grad()
model1.train()
prev_validation_ratio = float('inf')

for i in range(200):
    loss = model1(np_tresillo_train, np_non_tresillo, vanilla_tresillo_vector)

    loss.backward()
    optim.step()
    with torch.no_grad():
        validation_ratio = model1(np_tresillo_test, np_not_tre_validation, vanilla_tresillo_vector)
        if validation_ratio > prev_validation_ratio:
            break
        prev_validation_ratio = validation_ratio
```

```python
model_goodness_3 = 1/validation_ratio
model_goodness_3 = model_goodness_3.cpu().detach().item()
similarity = model1.similarity(np_tresillo_test, vanilla_tresillo_vector).cpu().detach().numpy()
print(f"Mean and Standard dev of Similarity for tresillo set: {round(np.mean(similarity), 4)} +- {round(np.std(similarity), 4)}")
similarity_not_tre = model1.similarity(np_not_tre_validation, vanilla_tresillo_vector).cpu().detach().numpy()
print(f"Mean and Standard dev of Similarity for NOT tresillo set: {round(np.mean(similarity_not_tre), 4)} +- {round(np.std(similarity_not_tre), 4)}")
print(f"Model Goodness: {round(model_goodness_3, 4)}")
```

    Mean and Standard dev of Similarity for tresillo set: 0.5434 +- 0.2254
    Mean and Standard dev of Similarity for NOT tresillo set: 0.2459 +- 0.1169
    Model Goodness: 2.2098

Since this is out best result so far, let's try to interpret what the parameters are hinting towards.

```python
import matplotlib.pyplot as plt
y = model1.theta.cpu().detach().numpy().reshape(-1)
x = np.arange(16)
fig = plt.figure(figsize=(12, 6), dpi=80)
ax = fig.add_axes([0,0,1,1])
plt.xticks(x)
ax.bar(x,y)
ax.set_xlabel('Theta/Scaling-factor for each axis')
ax.set_ylabel('Value for theta')
ax.set_title('The theta values from model')
plt.show()
```

**Discussion:**

This method provides the best gap between both the sets. The parameters of the similarity function hint that it is better to not consider the value of 0th and the 8th beat as both the songs with/without tresillo, which are in 4/4 meter tend to have high values in the rhythm space for these beats. It gives a high value for the 3rd axis which is the second beat in the tresillo rhythm.

#### 3.3.4) Rhythm similarity measured with parameterized cosine similarity, using the centroid of tresillo songs as centre.

```python
device = torch.device('cpu')
model = ParameterizedDistance(np.random.rand(1, 16), device, torch.cosine_similarity)
optim = SGD(model.parameters(), lr=1e-2, weight_decay=1e-4)
optim.zero_grad()
model.train()
prev_validation_ratio = float('inf')
for i in range(100):
    loss = model(np_tresillo_train, np_non_tresillo, centroid)
    loss.backward()
    optim.step()
    with torch.no_grad():
        validation_ratio = model(np_tresillo_test, np_not_tre_validation, centroid)
        if validation_ratio > prev_validation_ratio:
            break
        prev_validation_ratio = validation_ratio
```

```python
model_goodness_4 = 1/validation_ratio
model_goodness_4 = model_goodness_4.cpu().detach().item()
similarity = model.similarity(np_tresillo_test, vanilla_tresillo_vector).cpu().detach().numpy()
print(f"Mean Similarity +- Standard Dev: {round(np.mean(similarity), 4)} +- {round(np.std(similarity), 4)}")
similarity_not_tre = model.similarity(np_not_tre_validation, vanilla_tresillo_vector).cpu().detach().numpy()
print(f"Mean Similarity +- Standard Dev: {round(np.mean(similarity_not_tre), 4)} +- {round(np.std(similarity_not_tre), 4)}")
print(f"Model Goodness: {round(model_goodness_4, 4)}")
```

    Mean Similarity +- Standard Dev: 0.852 +- 0.1391
    Mean Similarity +- Standard Dev: 0.5883 +- 0.2731
    Model Goodness: 1.2277

**Discussion:**
Although the parameterization does improve the model goodness, using the template teresill rhythm

**Visualizing the model good for different models described above**

```python
import plotly.express as px
fig = plt.figure(figsize=(12, 6), dpi=80)
ax = fig.add_axes([0,0,1,1])
models = ["2.3.1", "2.3.2", "2.3.3", "2.3.4"]
goodness = [model_goodness_1, model_goodness_2, model_goodness_3, model_goodness_4]
ax.bar(models,goodness)
ax.set_xlabel('Models as described in the respective sections')
ax.set_ylabel('Models goodness ratio')
ax.set_title('Model goodness ratio')
plt.show()

print(model_goodness_1, model_goodness_2, model_goodness_3, model_goodness_4)
```

2.3.1) Rhythm similarity measured with cosine simularity, using tresillo template as centre.

2.3.2) Rhythm similarity measured with cosine simularity, using the centroid of tresillo songs as centre.

2.3.3) Rhythm similarity measured with parameterized cosine simularity, using the centroid of tresillo songs as centre.

2.3.4) Rhythm similarity measured with parameterized cosine simularity, using tresillo template as centre.

### 3.4) Tresilio-ness over Time

Now that we explored several ways to compute Tresillo-ness, we can calculate the Tresilo-ness of our billboard data set.
First let us load the rhythm vectors of our billboard songs and let's also merge it to the metadata of the billboard songs.
The metadata of the billboard songs includes time signatures with which we can Tresillo-ness over time.

```python
df_billboard = pd.read_csv(home_dir + 'dataset/rythm_vectors/rythm_vectors_billboard.csv')
df_billboard_meta = pd.read_csv(home_dir + 'dataset/billboard_1999-2019_unique_top_20_selection.csv')

#calculate cosine simularity
df_billboard = collapse_normalize_vectors(df_billboard)
df_billboard_sim = calc_cosine_sim(df_billboard, vector_vanilla_tresillo)
df_billboard_sim = calc_cosine_sim_model(df_billboard, vector_vanilla_tresillo, model)

#prepare for merge with meta data
df_billboard_sim['song_artist'] = df_billboard_sim.index
df_billboard_sim['song'] = df_billboard_sim.song_artist.apply(lambda x: x.split('_')[0])
df_billboard_sim['artist'] = df_billboard_sim.song_artist.apply(lambda x: x.split('_')[1][:-1])
df_billboard_sim = df_billboard_sim.drop(['song_artist'], axis=1)

#merge data frames
df_billboard_merged = df_billboard_sim.merge(df_billboard_meta, left_on=['song','artist'], right_on=['Name', 'Artists'], how='left')

df_billboard_merged[df_billboard_merged['cosine_sim_tresillo']>0.8]

df_billboard_reduced = df_billboard_merged[['Name', 'Artists', 'Peak.position', 'Week', 'Genre', 'cosine_sim_tresillo', 'model_cosine_sim_tresillo']]
#let's look at some songs with very high tresillo-ness

print(df_billboard_reduced.sort_values('cosine_sim_tresillo', ascending=False).head(10))

print(df_billboard_reduced.sort_values('model_cosine_sim_tresillo', ascending=False).head(10))

# Nan is Magic by rude https://www.youtube.com/watch?v=PIh2xe4jnpk
```

                        Name        Artists  Peak.position        Week  \
    423  Where Have You Been        Rihanna           18.0  2012-05-25
    389           Turn Me On   Kevin Lyttle           20.0  2004-06-23
    319         Shape Of You     Ed Sheeran            1.0  2017-01-28
    89         Cheap Thrills            Sia           19.0  2016-06-18
    387     Treat You Better   Shawn Mendes           20.0  2016-07-30
    293            Rather Be   Clean Bandit           19.0  2014-08-08
    248             Miss You        Aaliyah           15.0  2002-12-24
    261            New Rules       Dua Lipa           19.0  2017-12-02
    58                 Blame  Calvin Harris           19.0  2014-11-14
    302               Royals          Lorde           17.0  2013-08-30

                                                     Genre  cosine_sim_tresillo  \
    423                              House,Electro-Pop,Pop             0.947448
    389                                                Pop             0.937243
    319                    Tropical House,Dancehall,Pop,UK             0.915565
    89           Jamaica,Remix,Australia,Rap,Synth-Pop,Pop             0.908945
    387                                         Canada,Pop             0.883772
    293  Electro-Pop,Deep House,Eurodance,Dance-Pop,Cla...             0.861075
    248                                               R&;B             0.847090
    261  Power Pop,Electronic,Electro-Pop,Tropical Hous...             0.838867
    58                                   Electronic,UK,Pop             0.828732
    302  Alternative,Art Pop,Adult Alternative,New Zeal...             0.827160

         model_cosine_sim_tresillo
    423                   0.993943
    389                   0.981877
    319                   0.978163
    89                    0.990468
    387                   0.969584
    293                   0.886355
    248                   0.909887
    261                   0.979894
    58                    0.538262
    302                   0.919284
                        Name           Artists  Peak.position        Week  \
    423  Where Have You Been           Rihanna           18.0  2012-05-25
    89         Cheap Thrills               Sia           19.0  2016-06-18
    389           Turn Me On      Kevin Lyttle           20.0  2004-06-23
    261            New Rules          Dua Lipa           19.0  2017-12-02
    319         Shape Of You        Ed Sheeran            1.0  2017-01-28
    387     Treat You Better      Shawn Mendes           20.0  2016-07-30
    53   Best Day Of My Life  American Authors           16.0  2014-03-21
    7       A Thousand Miles   Vanessa Carlton           16.0  2002-04-02
    11             Alejandro         Lady Gaga           20.0  2010-05-06
    374        The Way I Are         Timbaland           18.0  2007-07-04

                                                     Genre  cosine_sim_tresillo  \
    423                              House,Electro-Pop,Pop             0.947448
    89           Jamaica,Remix,Australia,Rap,Synth-Pop,Pop             0.908945
    389                                                Pop             0.937243
    261  Power Pop,Electronic,Electro-Pop,Tropical Hous...             0.838867
    319                    Tropical House,Dancehall,Pop,UK             0.915565
    387                                         Canada,Pop             0.883772
    53   Adult Alternative,Adult Contemporary,Pop-Rock,...             0.826479
    7                           Singer-Songwriter,Pop,Rock             0.793010
    11   Electronic,Electro-Pop,Synth-Pop,Eurodance,Dan...             0.756820
    374                                                Rap             0.697078

         model_cosine_sim_tresillo
    423                   0.993943
    89                    0.990468
    389                   0.981877
    261                   0.979894
    319                   0.978163
    387                   0.969584
    53                    0.958528
    7                     0.948718
    11                    0.947575
    374                   0.947043

Let's plot weekly cosine Tresillo-ness

```python
df_bilboard_merged_weekly = df_billboard_merged.groupby(['Week'])['cosine_sim_tresillo'].agg(['mean']).reset_index()
df_bilboard_merged_weekly_model = df_billboard_merged.groupby(['Week'])['model_cosine_sim_tresillo'].agg(['mean']).reset_index()

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_figheight(7)
fig.set_figwidth(17)
fig.suptitle('Weekly Cosine Tresillo-ness in the Top 20 Billboards', fontsize=16)
ax1.plot(pd.to_datetime(df_bilboard_merged_weekly.Week), df_bilboard_merged_weekly['mean'])
ax1.set_title('Cosine Similarity')
ax1.set_xlabel('Time')
ax1.set_ylabel('Cosine Tresillo-ness')
ax2.plot(pd.to_datetime(df_bilboard_merged_weekly_model.Week), df_bilboard_merged_weekly_model['mean'])
ax2.set_title('Parameterized Cosine Similarity')
ax2.set_xlabel('Time')
ax2.set_ylabel('Cosine Tresillo-ness')
```

The time trend above is very noisy and hard to read. By calculating a 4 weeks moveing average we might get smoother results.

```python
df_bilboard_merged_weekly['rolling_mean'] = df_bilboard_merged_weekly.iloc[:,1].rolling(window=4).mean()
df_bilboard_merged_weekly_model['rolling_mean'] = df_bilboard_merged_weekly_model.iloc[:,1].rolling(window=4).mean()

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_figheight(7)
fig.set_figwidth(17)
fig.suptitle('4 Weeks Moving Average Cosine Tresillo-ness in the Top 20 Billboards', fontsize=16)
ax1.plot(pd.to_datetime(df_bilboard_merged_weekly.Week), df_bilboard_merged_weekly['rolling_mean'])
ax1.set_title('Cosine Similarity')
ax1.set_xlabel('Time')
ax1.set_ylabel('Cosine Tresillo-ness')
ax2.plot(pd.to_datetime(df_bilboard_merged_weekly_model.Week), df_bilboard_merged_weekly_model['rolling_mean'])
ax2.set_title('Parameterized Cosine Similarity')
ax2.set_xlabel('Time')
ax2.set_ylabel('Cosine Tresillo-ness')
```

```python
df_bilboard_merged_weekly['rolling_mean_yearly'] = df_bilboard_merged_weekly.iloc[:,1].rolling(window=52).mean()
df_bilboard_merged_weekly_model['rolling_mean_yearly'] = df_bilboard_merged_weekly_model.iloc[:,1].rolling(window=52).mean()

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_figheight(7)
fig.set_figwidth(17)
fig.suptitle('52 Weeks Moving Average Cosine Tresillo-ness in the Top 20 Billboards', fontsize=16)
ax1.plot(pd.to_datetime(df_bilboard_merged_weekly.Week), df_bilboard_merged_weekly['rolling_mean_yearly'])
ax1.set_title('Cosine Similarity')
ax1.set_xlabel('Time')
ax1.set_ylabel('Cosine Tresillo-ness')
ax2.plot(pd.to_datetime(df_bilboard_merged_weekly_model.Week), df_bilboard_merged_weekly_model['rolling_mean_yearly'])
ax2.set_title('Parameterized Cosine Similarity')
ax2.set_xlabel('Time')
ax2.set_ylabel('Cosine Tresillo-ness')
```

### To plot confidence intervals

```python

```

```python

```

```python
week_windows = [(df_bilboard_merged_weekly['Week'][index], df_bilboard_merged_weekly['Week'][index+1]) for index in range(len(df_bilboard_merged_weekly)-1)]

lower_ci_cosine = []
upper_ci_cosine = []
lower_ci_model= []
upper_ci_model = []
for el in week_windows:
    sample = df_billboard_merged[(df_billboard_merged['Week']>=el[0]) & (df_billboard_merged['Week']<=el[1])]

    ci_cosine = bootstrap_CI(sample['cosine_sim_tresillo'], 1000)
    ci_model = bootstrap_CI(sample['model_cosine_sim_tresillo'], 1000)

    lower_ci_cosine.append(ci_cosine[0])
    upper_ci_cosine.append(ci_cosine[1])

    lower_ci_model.append(ci_model[0])
    upper_ci_model.append(ci_model[1])
```

```python

df_bilboard_merged_weekly.loc[1:, 'lower_ci'] = lower_ci_cosine
df_bilboard_merged_weekly.loc[1:, 'upper_ci'] = upper_ci_cosine

df_bilboard_merged_weekly_model.loc[1:, 'lower_ci'] = lower_ci_model
df_bilboard_merged_weekly_model.loc[1:, 'upper_ci'] = upper_ci_model

print(np.mean(np.asarray(lower_ci_cosine[:10])- np.asarray(upper_ci_cosine[:10])))
print(np.mean(np.asarray(lower_ci_cosine[10:])- np.asarray(upper_ci_cosine[10:])))

df_bilboard_merged_weekly_model
```

    -0.08203447598113545
    -0.10173506910248475

      Week
      mean
      lower_ci
      upper_ci

      0
      1999-07-12
      0.426478
      NaN
      NaN

      1
      1999-08-02
      0.558625
      0.223335
      0.735339

      2
      1999-08-16
      0.340538
      0.270729
      0.846520

      3
      1999-08-23
      0.623450
      0.340538
      0.623450

      4
      1999-08-30
      0.109573
      0.109573
      0.623450

      ...
      ...
      ...
      ...
      ...

      358
      2018-09-08
      0.772530
      0.772530
      0.824255

      359
      2018-12-22
      0.374472
      0.374472
      0.772530

      360
      2019-01-05
      0.194019
      0.194019
      0.374472

      361
      2019-03-30
      0.624726
      0.194019
      0.624726

      362
      2019-05-25
      0.453827
      0.453827
      0.624726

363 rows × 4 columns

```python

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_figheight(7)
fig.set_figwidth(17)
fig.suptitle('Weekly Cosine Tresillo-ness in the Top 20 Billboards', fontsize=16)
ax1.plot(pd.to_datetime(df_bilboard_merged_weekly.Week), df_bilboard_merged_weekly['mean'])
ax1.fill_between(pd.to_datetime(df_bilboard_merged_weekly.Week), (df_bilboard_merged_weekly['lower_ci']), (df_bilboard_merged_weekly['upper_ci']), color='b', alpha=.1)

ax1.set_title('Cosine Similarity')
ax1.set_xlabel('Time')
ax1.set_ylabel('Cosine Tresillo-ness')
ax2.plot(pd.to_datetime(df_bilboard_merged_weekly_model.Week), df_bilboard_merged_weekly_model['mean'])
ax2.fill_between(pd.to_datetime(df_bilboard_merged_weekly_model.Week), (df_bilboard_merged_weekly_model['lower_ci']), (df_bilboard_merged_weekly_model['upper_ci']), color='b', alpha=.1)

ax2.set_title('Parameterized Cosine Similarity')
ax2.set_xlabel('Time')
ax2.set_ylabel('Cosine Tresillo-ness')
plt.savefig('plots/tresillo_ness_over_week.png', dpi=160)
```

```python
week_windows = [(df_bilboard_merged_weekly['Week'][index], df_bilboard_merged_weekly['Week'][index+51]) for index in range(len(df_bilboard_merged_weekly)-51)]

lower_ci_cosine = []
upper_ci_cosine = []
lower_ci_model= []
upper_ci_model = []
for el in week_windows:
    sample = df_billboard_merged[(df_billboard_merged['Week']>=el[0]) & (df_billboard_merged['Week']<=el[1])]

    ci_cosine = bootstrap_CI(sample['cosine_sim_tresillo'], 1000)
    ci_model = bootstrap_CI(sample['model_cosine_sim_tresillo'], 1000)

    lower_ci_cosine.append(ci_cosine[0])
    upper_ci_cosine.append(ci_cosine[1])

    lower_ci_model.append(ci_model[0])
    upper_ci_model.append(ci_model[1])
```

```python
df_bilboard_merged_weekly['rolling_mean_yearly'] = df_bilboard_merged_weekly.iloc[:,1].rolling(window=52).mean()
df_bilboard_merged_weekly_model['rolling_mean_yearly'] = df_bilboard_merged_weekly_model.iloc[:,1].rolling(window=52).mean()

df_bilboard_merged_weekly.loc[51:, 'lower_ci'] = lower_ci_cosine
df_bilboard_merged_weekly.loc[51:, 'upper_ci'] = upper_ci_cosine

df_bilboard_merged_weekly_model.loc[51:, 'lower_ci'] = lower_ci_model
df_bilboard_merged_weekly_model.loc[51:, 'upper_ci'] = upper_ci_model

print(np.mean(np.asarray(lower_ci_cosine[:10])- np.asarray(upper_ci_cosine[:10])))
print(np.mean(np.asarray(lower_ci_cosine[10:])- np.asarray(upper_ci_cosine[10:])))

```

    -0.03230255394969321
    -0.039347252551720314

```python

fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_figheight(7)
fig.set_figwidth(17)
fig.suptitle('52 Weeks Moving Average Cosine Tresillo-ness in the Top 20 Billboards', fontsize=16)
ax1.plot(pd.to_datetime(df_bilboard_merged_weekly.Week), df_bilboard_merged_weekly['rolling_mean_yearly'])
ax1.fill_between(pd.to_datetime(df_bilboard_merged_weekly.Week), (df_bilboard_merged_weekly['lower_ci']), (df_bilboard_merged_weekly['upper_ci']), color='b', alpha=.1)
ax1.set_title('Cosine Similarity')
ax1.set_xlabel('Time')
ax1.set_ylabel('Cosine Tresillo-ness')
ax2.plot(pd.to_datetime(df_bilboard_merged_weekly_model.Week), df_bilboard_merged_weekly_model['rolling_mean_yearly'])
ax2.fill_between(pd.to_datetime(df_bilboard_merged_weekly_model.Week), (df_bilboard_merged_weekly_model['lower_ci']), (df_bilboard_merged_weekly_model['upper_ci']), color='b', alpha=.1)
ax2.set_title('Parameterized Cosine Similarity')
ax2.set_xlabel('Time')
ax2.set_ylabel('Cosine Tresillo-ness')
plt.savefig('plots/tresillo_ness_over_year.png', dpi=160)
```

```python
dir_sheet_music = home_dir + '/dataset/project_mscx/mscx_billboard/'

list_sheet_music = os.listdir(dir_sheet_music)
df_rythm_vectors =[]

df_agg_collapsed = []
for idx, el in enumerate(list_sheet_music[:10]):
    if el[-4:] == 'mscx':

        #Get notes with onsets
        s = Score(dir_sheet_music+el)
        df = s.mscx.notes
        df['beat_onset'] =df.mc+df.mc_onset
        #df_collapsed = df.groupby(['beat_onset'])['mn_onset'].agg(['count'])
        df_collapsed = df

        if len(df_agg_collapsed) == 0 : df_agg_collapsed = df_collapsed
        else: df_agg_collapsed = pd.concat([df_agg_collapsed, df_collapsed], axis=0)

```

```python
# adapt it so to
values = df_agg_collapsed[:10]['mc_onset']*12

values = (df_agg_collapsed['mc_onset']*128).astype('int32')

fig = plt.figure(figsize=(12, 6), dpi=80)
plt.hist(values, bins=128)
plt.locator_params(axis='x', nbins=32)
plt.xlabel('Frequency of Onsets')
plt.ylabel('Onset time (in 1/128)')
plt.xlim([0, 130])
plt.title('Frequency of onsets of musical events notated in 1/128')

plt.savefig('plots/onset_frequency_128.png', dpi=160)

df_agg_collapsed
```

## 4) Outlook on final interpretation

Looking at the finally generated plots we do not recognize any clear trend in the use of the Tresillo rhythm over time. This might have several reasons and is not unintuitive.
Assuming that we can trust our methodology and that we actually manage to measure the intensity of Tresillo use over time, we would then have to conclude that there is no trend in Tresillo use over the last 20 years. In a final analysis we would further investigate by which songs some of these peaks are caused and if they validate or question our methods.
As part of the interpretation, we'd also like to question if we can indeed trust our methodology and the thereby the produced results. This question can be discussed by looking at the obtained results and asking ourselves if the results seem intuitive. Furthermore, we also have to critically reflect on the assumptions on which our methods are based. For instance, we will have to question if the number of train samples is sufficient to obtain a trustable model.

## References

- Chin,  F.  and  Wu,  S.  (1992).   An  efficient  algorithm  for  rhythm-finding.Computer MusicJournal, 16(2):35–44.
- Dixon,  S.,  Gouyon,  F.,  Widmer,  G.,  et  al.  (2004).   Towards  characterisation  of  music  viarhythmic patterns.  InISMIR.
- Huron, D. and Ommen, A. (2006).  An empirical study of syncopation in american popularmusic, 1890–1939.Music Theory Spectrum, 28(2):211–231.
- Palmer, C. and Krumhansl, C. L. (1990).  Mental representations for musical meter.Journalof Experimental Psychology: Human Perception and Performance, 16(4):728.
- Panteli,  M.,  Bogaards,  N.,  Honingh,  A.  K.,  et  al.  (2014).   Modeling  rhythm  similarity  forelectronic dance music.  InISMIR, pages 537–542.
- Parry, M. and Essa, I. (2003).  Rhythmic similarity through elaboration.
- Pohle,  T.,  Schnitzer,  D.,  Schedl,  M.,  Knees,  P.,  and  Widmer,  G.  (2009).   On  rhythm  andgeneral music similarity.  InISMIR, pages 525–530.
- Rohrmeier, M. (2020).  Towards a formalization of musical rhythm.  InProc. of the 21st Int.Society for Music Information Retrieval Conf
- Floyd, Samuel A. "Black music in the circum-Caribbean." American Music (1999): 5-7.
- Toussaint, Godfried. "The geometry of musical rhythm." Japanese Conference on Discrete and Computational Geometry. Springer, Berlin, Heidelberg, 2004, page 199.
- Zhihua Zhang, James T. Kwok and Dit-Yan Yeung. (2003). Parametric  Distance Metric Learning with Label Information