_Note: This Python notebook assumes that this is your first time to see/work on a Python code. It also assumes that you are new to recommendation systems, but familiar with it as a high-level concept._

# VANILLA COLLABORTIVE FILTERING

_Prepared for EMBA 2022 by EF Legara_

---

Here, we use Python libraries to help us with some of our computations and data visualization. Think of them as a collection of pre-written formulae that we can utilize so we don't have to explicitly write the formulae ourselves. The following libraries are used. You can click on the link to find out more about them.

- [pandas](https://pandas.pydata.org/) - Python data analysis library
- [matplotlib](https://matplotlib.org/) - a plotting library for Python
- [seaborn](https://seaborn.pydata.org/) - for statistical visualization
- [numpy](https://numpy.org/) - for numerical computing in Python
- [sklearn](https://scikit-learn.org/stable/) - ML in Python

Below is the "code cell" that we run to load the libraries.

```python
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import seaborn as sns

%matplotlib inline
```

---
## Loading the Data
_This data was generated from the EMBA 2019 class._

`df` is the variable we use for our dataframe or table. You may use other variable names, of course.

Here, we load the data stored in an MS in Excel file named `movie recom.xlsx` to a table that we call `df` (a dataframe).

```python
df = pd.read_excel('./data/movie recom.xlsx')
```

Let's take a look at what's inside `df`. The `.head()` function  just tells Python to print the first five rows of the dataframe.

```python
df.head()
```

      Name
      Matrix
      Inception
      Titanic
      Amélie
      Love Actually
      Terminator
      Elysium
      Avatar

      0
      Person 1
      5
      5
      4
      0
      0
      3
      0
      5

      1
      Person 2
      5
      5
      1
      4
      3
      4
      2
      4

      2
      Person 3
      0
      0
      3
      4
      5
      0
      0
      0

      3
      Person 4
      4
      4
      4
      0
      3
      3
      2
      5

      4
      Person 5
      1
      0
      3
      0
      0
      3
      2
      4

The table above currently has nine columns which include the **Name** column and the eight other columns for the movies. For ease of analysis, we can set the _index_ of each row in the table using the **Name** of clients or users. You can think of the index as the row number or row name.

```python
df = df.set_index('Name')
```

Here's how are dataframe looks like.

```python
df.head()
```

      Matrix
      Inception
      Titanic
      Amélie
      Love Actually
      Terminator
      Elysium
      Avatar

      Name

      Person 1
      5
      5
      4
      0
      0
      3
      0
      5

      Person 2
      5
      5
      1
      4
      3
      4
      2
      4

      Person 3
      0
      0
      3
      4
      5
      0
      0
      0

      Person 4
      4
      4
      4
      0
      3
      3
      2
      5

      Person 5
      1
      0
      3
      0
      0
      3
      2
      4

As you can see above, **Name** is not a column (or variable) in the table anymore, but instead an _index_ (or a row name).

The resulting dataframe now purely consists of values that are numerical; in this case, the ratings given by each person (row name or index) to the set of movies (columns).

We can view the values of the pandas dataframe by running the code cell below.

```python
df.values
```

    array([[5, 5, 4, 0, 0, 3, 0, 5],
           [5, 5, 1, 4, 3, 4, 2, 4],
           [0, 0, 3, 4, 5, 0, 0, 0],
           [4, 4, 4, 0, 3, 3, 2, 5],
           [1, 0, 3, 0, 0, 3, 2, 4],
           [5, 5, 2, 3, 0, 3, 1, 4],
           [5, 5, 2, 0, 1, 3, 0, 5],
           [5, 5, 4, 3, 3, 4, 0, 3],
           [4, 5, 4, 0, 4, 4, 5, 4],
           [5, 5, 3, 0, 3, 4, 4, 5],
           [5, 5, 5, 5, 0, 2, 0, 3],
           [4, 5, 3, 2, 5, 3, 2, 1],
           [5, 5, 5, 4, 5, 3, 3, 4],
           [4, 5, 4, 0, 4, 4, 1, 4],
           [4, 5, 2, 0, 1, 3, 3, 5],
           [4, 3, 3, 0, 4, 3, 0, 4],
           [5, 5, 4, 5, 3, 2, 0, 4],
           [4, 0, 2, 0, 4, 4, 0, 3],
           [1, 3, 3, 0, 3, 3, 0, 5],
           [4, 0, 5, 0, 4, 4, 4, 5]])

These are essentially the numbers in our dataframe, but displayed as an $N$-by-$N$ array of numbers, where $N=20$  is the total number of users and where each `[ ... ]` lists the ratings a person has given to the movies he/she has seen. The order, of course, matters. The first element or number in the list `[ ... ]` corresponds to the movie *The Matrix*, while the last corresponds to *Avatar*.

Let's then have a look at how Person 1 (`p1`) and Person 2 (`p2`) rated the movies.

```python
p1 = list(df.loc['Person 1'])
p2 = list(df.loc['Person 2'])

print('p1 [The Matrix, Inception, ..., Elysium, Avatar]:', p1)
print('p2: ', p2)
```

    p1 [The Matrix, Inception, ..., Elysium, Avatar]: [5, 5, 4, 0, 0, 3, 0, 5]
    p2:  [5, 5, 1, 4, 3, 4, 2, 4]

Indeed, the values reflect the ratings in our raw data for persons 1 and 2.

```python
df.head(n=2)
```

      Matrix
      Inception
      Titanic
      Amélie
      Love Actually
      Terminator
      Elysium
      Avatar

      Name

      Person 1
      5
      5
      4
      0
      0
      3
      0
      5

      Person 2
      5
      5
      1
      4
      3
      4
      2
      4

---
## Matrix Visualization

Let's explore if we can also already identify some patterns by just visualizing the raw data of ratings. We can use seaborn's `heatmap` function to visualize our dataframe `df`.

```python
# the choice of colormap
cmap = sns.cm.rocket_r

# indicate figure size
plt.figure(figsize=(10,8))

# draw heatmap
g = sns.heatmap(df.values);

# set tick labes for both x & y axes
g.set_xticklabels(df.columns, rotation=90);
g.set_yticklabels(df.index, rotation=0);
```

That is actually still a bit tough to read. We cannot accurately tell yet if there are strong connections, correlations, and/or similarities between movies/users.

---

## Collaborative Filtering

### User-based CF
Now, let's implement [**collaborative filtering**](https://en.wikipedia.org/wiki/Collaborative_filtering). With collaborative filtering, we can explore how users are similar to each other based on the ratings they gave movies using the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) measure. If you can recall,

\begin{equation}
\cos (X,Y) = \frac{\sum_{i=1}^{n} X_i Y_i}{\sqrt {\sum_{i=1}^{n}X_i^2}\sqrt {\sum_{i=1}^{n}Y_i^2}}.
\end{equation}

That is, the similarity of Person $X$ and Person $Y$, in terms of their taste in movies, can be computed by the ratings they gave to the movies ($1$ to $n=8$) where movie 1 corresponds to _The Matrix_, movie 2 to _Inception_, movie 3 to _Titanic_, and so on and so forth.

We define a function `compute_cosine_similarity( )` below that computes the cosine similarity given the above equation.

```python
def compute_cosine_similarity (p1, p2):
    num_movies = len(p1)
    sumprod = 0
    for i in range(num_movies):
        sumprod = sumprod + (p1[i] * p2[i])
    numerator = sumprod
    denominator = np.sqrt(sum([r1*r1 for r1 in p1])) * np.sqrt(sum([r2*r2 for r2 in p2]))
    return numerator/denominator
```

Let's use the function above to check how similar (or not) _Person 1_ and _Person 2_ are.

```python
p1 = list(df.loc['Person 1'])
p2 = list(df.loc['Person 2'])
print ('Person 1 and Person 2 are {:.2f}% similar.'.format(compute_cosine_similarity(p1, p2)*100))
```

    Person 1 and Person 2 are 81.26% similar.

Check similarity between Person 4 and Person 3.

```python
p1 = list(df.loc['Person 4'])
p2 = list(df.loc['Person 3'])
print ('Person 4 and Person 3 are {:.2f}% similar.'.format(compute_cosine_similarity(p1, p2)*100))
```

    Person 4 and Person 3 are 39.18% similar.

#### Using a Python library

As mentioned at the beginning, the good news is that we don't have to manually perform this pairwise computation of cosine similarity. We didn't even need to write the function ourselves. Python has a function via the `sklearn` library that computes the cosine similarity of the whole matrix for us.

It's a one liner as you can see in the code cell below.

```python
user_similarity_matrix = cosine_similarity(df.values)
```

The result is a 20 x 20 matrix where the numbers of rows and columns correspond to the number of users (in this case, 20) in the data. The element in each cell of the matrix gives us the cosine similarity between two persons.

```python
user_similarity_matrix.size
```

    400

```python
user_similarity_matrix
```

    array([[1.        , 0.81262362, 0.16970563, 0.92338052, 0.73658951,
            0.92219816, 0.97519805, 0.89077845, 0.81566396, 0.88548292,
            0.85605599, 0.7362357 , 0.80833162, 0.90329585, 0.92219816,
            0.87757241, 0.83984125, 0.70420284, 0.83820084, 0.72117107],
           [0.81262362, 1.        , 0.45434411, 0.86281799, 0.60522753,
            0.94150762, 0.87139535, 0.93221259, 0.85360419, 0.89586351,
            0.84445278, 0.8916428 , 0.9258201 , 0.87188993, 0.88141139,
            0.85104977, 0.92296269, 0.75009757, 0.76802458, 0.71684223],
           [0.16970563, 0.45434411, 1.        , 0.39175718, 0.20380987,
            0.26983141, 0.16489697, 0.52828266, 0.39691115, 0.30357866,
            0.46563307, 0.61591788, 0.6466323 , 0.4395538 , 0.16489697,
            0.47356802, 0.60676739, 0.47078588, 0.43105272, 0.46358632],
           [0.92338052, 0.86281799, 0.39175718, 1.        , 0.80501129,
            0.85915255, 0.92440465, 0.90409231, 0.95383309, 0.97272271,
            0.78177899, 0.86175089, 0.9214786 , 0.9765879 , 0.94615534,
            0.95960518, 0.85229309, 0.82758732, 0.92512561, 0.89365259],
           [0.73658951, 0.60522753, 0.20380987, 0.80501129, 1.        ,
            0.64499491, 0.67894201, 0.62883731, 0.75838508, 0.77340406,
            0.5724164 , 0.49813548, 0.66679486, 0.71543897, 0.76380977,
            0.70262025, 0.57008771, 0.69707851, 0.79311554, 0.88484517],
           [0.92219816, 0.94150762, 0.26983141, 0.85915255, 0.64499491,
            1.        , 0.93258427, 0.92391739, 0.79952449, 0.87224365,
            0.92736078, 0.79139995, 0.86548464, 0.84423998, 0.91011236,
            0.80782688, 0.91925919, 0.65145034, 0.74040926, 0.65523412],
           [0.97519805, 0.87139535, 0.16489697, 0.92440465, 0.67894201,
            0.93258427, 1.        , 0.89345857, 0.82741488, 0.91016729,
            0.80770132, 0.76941662, 0.81355557, 0.91630925, 0.94382022,
            0.90574529, 0.84184789, 0.74645352, 0.84810515, 0.70487307],
           [0.89077845, 0.93221259, 0.52828266, 0.90409231, 0.62883731,
            0.92391739, 0.89345857, 1.        , 0.84846992, 0.87383999,
            0.91906812, 0.92369422, 0.94629488, 0.93962636, 0.84269388,
            0.9179821 , 0.96180895, 0.79714108, 0.82717961, 0.7445818 ],
           [0.81566396, 0.85360419, 0.39691115, 0.95383309, 0.75838508,
            0.79952449, 0.82741488, 0.84846992, 1.        , 0.98058068,
            0.70130676, 0.90037213, 0.91662704, 0.93706146, 0.92967964,
            0.88108325, 0.77662155, 0.76361125, 0.8353986 , 0.8871553 ],
           [0.88548292, 0.89586351, 0.30357866, 0.97272271, 0.77340406,
            0.87224365, 0.91016729, 0.87383999, 0.98058068, 1.        ,
            0.74043756, 0.87182912, 0.90556796, 0.94693149, 0.97653365,
            0.90886009, 0.80833162, 0.7901857 , 0.85194275, 0.87121613],
           [0.85605599, 0.84445278, 0.46563307, 0.78177899, 0.5724164 ,
            0.92736078, 0.80770132, 0.91906812, 0.70130676, 0.74043756,
            1.        , 0.77063086, 0.86794777, 0.77665255, 0.75784322,
            0.73865061, 0.95321997, 0.56610073, 0.66904135, 0.59912476],
           [0.7362357 , 0.8916428 , 0.61591788, 0.86175089, 0.49813548,
            0.79139995, 0.76941662, 0.92369422, 0.90037213, 0.87182912,
            0.77063086, 1.        , 0.94826761, 0.91653063, 0.79139995,
            0.87407914, 0.87087481, 0.75677794, 0.75065008, 0.73810763],
           [0.80833162, 0.9258201 , 0.6466323 , 0.9214786 , 0.66679486,
            0.86548464, 0.81355557, 0.94629488, 0.91662704, 0.90556796,
            0.86794777, 0.94826761, 1.        , 0.91993984, 0.84817495,
            0.89566859, 0.94660211, 0.77360839, 0.81919184, 0.83354383],
           [0.90329585, 0.87188993, 0.4395538 , 0.9765879 , 0.71543897,
            0.84423998, 0.91630925, 0.93962636, 0.93706146, 0.94693149,
            0.77665255, 0.91653063, 0.91993984, 1.        , 0.90601364,
            0.97574355, 0.86005887, 0.8456508 , 0.92515071, 0.83691715],
           [0.92219816, 0.88141139, 0.16489697, 0.94615534, 0.76380977,
            0.91011236, 0.94382022, 0.84269388, 0.92967964, 0.97653365,
            0.75784322, 0.79139995, 0.84817495, 0.90601364, 1.        ,
            0.85678609, 0.79346582, 0.69216599, 0.83464317, 0.78429539],
           [0.87757241, 0.85104977, 0.47356802, 0.95960518, 0.70262025,
            0.80782688, 0.90574529, 0.9179821 , 0.88108325, 0.90886009,
            0.73865061, 0.87407914, 0.89566859, 0.97574355, 0.85678609,
            1.        , 0.85381497, 0.91663438, 0.92387682, 0.85436615],
           [0.83984125, 0.92296269, 0.60676739, 0.85229309, 0.57008771,
            0.91925919, 0.84184789, 0.96180895, 0.77662155, 0.80833162,
            0.95321997, 0.87087481, 0.94660211, 0.86005887, 0.79346582,
            0.85381497, 1.        , 0.70128687, 0.77676265, 0.68398557],
           [0.70420284, 0.75009757, 0.47078588, 0.82758732, 0.69707851,
            0.65145034, 0.74645352, 0.79714108, 0.76361125, 0.7901857 ,
            0.56610073, 0.75677794, 0.77360839, 0.8456508 , 0.69216599,
            0.91663438, 0.70128687, 1.        , 0.7967743 , 0.87539793],
           [0.83820084, 0.76802458, 0.43105272, 0.92512561, 0.79311554,
            0.74040926, 0.84810515, 0.82717961, 0.8353986 , 0.85194275,
            0.66904135, 0.75065008, 0.81919184, 0.92515071, 0.83464317,
            0.92387682, 0.77676265, 0.7967743 , 1.        , 0.80883632],
           [0.72117107, 0.71684223, 0.46358632, 0.89365259, 0.88484517,
            0.65523412, 0.70487307, 0.7445818 , 0.8871553 , 0.87121613,
            0.59912476, 0.73810763, 0.83354383, 0.83691715, 0.78429539,
            0.85436615, 0.68398557, 0.87539793, 0.80883632, 1.        ]])

Okay. That's a bit tough to read. We can use a heatmap instead. We may view it as a dataframe

```python
pd.DataFrame(user_similarity_matrix, index = df.index, columns = list(df.index))
```

      Person 1
      Person 2
      Person 3
      Person 4
      Person 5
      Person 6
      Person 7
      Person 8
      Person 9
      Person 10
      Person 11
      Person 12
      Person 13
      Person 14
      Person 15
      Person 16
      Person 17
      Person 18
      Person 19
      Person 20

      Name

      Person 1
      1.000000
      0.812624
      0.169706
      0.923381
      0.736590
      0.922198
      0.975198
      0.890778
      0.815664
      0.885483
      0.856056
      0.736236
      0.808332
      0.903296
      0.922198
      0.877572
      0.839841
      0.704203
      0.838201
      0.721171

      Person 2
      0.812624
      1.000000
      0.454344
      0.862818
      0.605228
      0.941508
      0.871395
      0.932213
      0.853604
      0.895864
      0.844453
      0.891643
      0.925820
      0.871890
      0.881411
      0.851050
      0.922963
      0.750098
      0.768025
      0.716842

      Person 3
      0.169706
      0.454344
      1.000000
      0.391757
      0.203810
      0.269831
      0.164897
      0.528283
      0.396911
      0.303579
      0.465633
      0.615918
      0.646632
      0.439554
      0.164897
      0.473568
      0.606767
      0.470786
      0.431053
      0.463586

      Person 4
      0.923381
      0.862818
      0.391757
      1.000000
      0.805011
      0.859153
      0.924405
      0.904092
      0.953833
      0.972723
      0.781779
      0.861751
      0.921479
      0.976588
      0.946155
      0.959605
      0.852293
      0.827587
      0.925126
      0.893653

      Person 5
      0.736590
      0.605228
      0.203810
      0.805011
      1.000000
      0.644995
      0.678942
      0.628837
      0.758385
      0.773404
      0.572416
      0.498135
      0.666795
      0.715439
      0.763810
      0.702620
      0.570088
      0.697079
      0.793116
      0.884845

      Person 6
      0.922198
      0.941508
      0.269831
      0.859153
      0.644995
      1.000000
      0.932584
      0.923917
      0.799524
      0.872244
      0.927361
      0.791400
      0.865485
      0.844240
      0.910112
      0.807827
      0.919259
      0.651450
      0.740409
      0.655234

      Person 7
      0.975198
      0.871395
      0.164897
      0.924405
      0.678942
      0.932584
      1.000000
      0.893459
      0.827415
      0.910167
      0.807701
      0.769417
      0.813556
      0.916309
      0.943820
      0.905745
      0.841848
      0.746454
      0.848105
      0.704873

      Person 8
      0.890778
      0.932213
      0.528283
      0.904092
      0.628837
      0.923917
      0.893459
      1.000000
      0.848470
      0.873840
      0.919068
      0.923694
      0.946295
      0.939626
      0.842694
      0.917982
      0.961809
      0.797141
      0.827180
      0.744582

      Person 9
      0.815664
      0.853604
      0.396911
      0.953833
      0.758385
      0.799524
      0.827415
      0.848470
      1.000000
      0.980581
      0.701307
      0.900372
      0.916627
      0.937061
      0.929680
      0.881083
      0.776622
      0.763611
      0.835399
      0.887155

      Person 10
      0.885483
      0.895864
      0.303579
      0.972723
      0.773404
      0.872244
      0.910167
      0.873840
      0.980581
      1.000000
      0.740438
      0.871829
      0.905568
      0.946931
      0.976534
      0.908860
      0.808332
      0.790186
      0.851943
      0.871216

      Person 11
      0.856056
      0.844453
      0.465633
      0.781779
      0.572416
      0.927361
      0.807701
      0.919068
      0.701307
      0.740438
      1.000000
      0.770631
      0.867948
      0.776653
      0.757843
      0.738651
      0.953220
      0.566101
      0.669041
      0.599125

      Person 12
      0.736236
      0.891643
      0.615918
      0.861751
      0.498135
      0.791400
      0.769417
      0.923694
      0.900372
      0.871829
      0.770631
      1.000000
      0.948268
      0.916531
      0.791400
      0.874079
      0.870875
      0.756778
      0.750650
      0.738108

      Person 13
      0.808332
      0.925820
      0.646632
      0.921479
      0.666795
      0.865485
      0.813556
      0.946295
      0.916627
      0.905568
      0.867948
      0.948268
      1.000000
      0.919940
      0.848175
      0.895669
      0.946602
      0.773608
      0.819192
      0.833544

      Person 14
      0.903296
      0.871890
      0.439554
      0.976588
      0.715439
      0.844240
      0.916309
      0.939626
      0.937061
      0.946931
      0.776653
      0.916531
      0.919940
      1.000000
      0.906014
      0.975744
      0.860059
      0.845651
      0.925151
      0.836917

      Person 15
      0.922198
      0.881411
      0.164897
      0.946155
      0.763810
      0.910112
      0.943820
      0.842694
      0.929680
      0.976534
      0.757843
      0.791400
      0.848175
      0.906014
      1.000000
      0.856786
      0.793466
      0.692166
      0.834643
      0.784295

      Person 16
      0.877572
      0.851050
      0.473568
      0.959605
      0.702620
      0.807827
      0.905745
      0.917982
      0.881083
      0.908860
      0.738651
      0.874079
      0.895669
      0.975744
      0.856786
      1.000000
      0.853815
      0.916634
      0.923877
      0.854366

      Person 17
      0.839841
      0.922963
      0.606767
      0.852293
      0.570088
      0.919259
      0.841848
      0.961809
      0.776622
      0.808332
      0.953220
      0.870875
      0.946602
      0.860059
      0.793466
      0.853815
      1.000000
      0.701287
      0.776763
      0.683986

      Person 18
      0.704203
      0.750098
      0.470786
      0.827587
      0.697079
      0.651450
      0.746454
      0.797141
      0.763611
      0.790186
      0.566101
      0.756778
      0.773608
      0.845651
      0.692166
      0.916634
      0.701287
      1.000000
      0.796774
      0.875398

      Person 19
      0.838201
      0.768025
      0.431053
      0.925126
      0.793116
      0.740409
      0.848105
      0.827180
      0.835399
      0.851943
      0.669041
      0.750650
      0.819192
      0.925151
      0.834643
      0.923877
      0.776763
      0.796774
      1.000000
      0.808836

      Person 20
      0.721171
      0.716842
      0.463586
      0.893653
      0.884845
      0.655234
      0.704873
      0.744582
      0.887155
      0.871216
      0.599125
      0.738108
      0.833544
      0.836917
      0.784295
      0.854366
      0.683986
      0.875398
      0.808836
      1.000000

From the dataframe above, we can see that Person 1 is 81.26% similar to Person 2 while Person 3 is only 39.18% similar to Person 4.

We can display the dataframe above as a heatmap. The darker the cell color is the more similar two persons are.

```python
plt.figure(figsize=(10,10))
g = sns.heatmap(user_similarity_matrix, cmap = cmap)
g.set_xticklabels(df.index, rotation=90);
g.set_yticklabels(df.index, rotation=0);
```

### Item-based CF

How about instead of user-based, we use item-based?

The method/approach is exactly the same. We'll just transpose our original dataframe; i.e., our rows are now the movie names, while the columns are the individual users.

```python
df_t = df.transpose()
```

```python
df_t
```

      Name
      Person 1
      Person 2
      Person 3
      Person 4
      Person 5
      Person 6
      Person 7
      Person 8
      Person 9
      Person 10
      Person 11
      Person 12
      Person 13
      Person 14
      Person 15
      Person 16
      Person 17
      Person 18
      Person 19
      Person 20

      Matrix
      5
      5
      0
      4
      1
      5
      5
      5
      4
      5
      5
      4
      5
      4
      4
      4
      5
      4
      1
      4

      Inception
      5
      5
      0
      4
      0
      5
      5
      5
      5
      5
      5
      5
      5
      5
      5
      3
      5
      0
      3
      0

      Titanic
      4
      1
      3
      4
      3
      2
      2
      4
      4
      3
      5
      3
      5
      4
      2
      3
      4
      2
      3
      5

      Amélie
      0
      4
      4
      0
      0
      3
      0
      3
      0
      0
      5
      2
      4
      0
      0
      0
      5
      0
      0
      0

      Love Actually
      0
      3
      5
      3
      0
      0
      1
      3
      4
      3
      0
      5
      5
      4
      1
      4
      3
      4
      3
      4

      Terminator
      3
      4
      0
      3
      3
      3
      3
      4
      4
      4
      2
      3
      3
      4
      3
      3
      2
      4
      3
      4

      Elysium
      0
      2
      0
      2
      2
      1
      0
      0
      5
      4
      0
      2
      3
      1
      3
      0
      0
      0
      0
      4

      Avatar
      5
      4
      0
      5
      4
      4
      5
      3
      4
      5
      3
      1
      4
      4
      5
      4
      4
      3
      5
      5

```python
movie_similarity_matrix = cosine_similarity(df_t)
plt.figure(figsize=(10,10))
g = sns.heatmap(movie_similarity_matrix, cmap = cmap, annot=True)
g.set_xticklabels(df_t.index, rotation=90);
g.set_yticklabels(df_t.index, rotation=0);
```

See if you can confirm the following statements:
- Among the other movies in the list and according to the preferences of EMBA 2019 students, _Avatar_ is most similar to _Terminator_.
- Among the other movies in the list and according to the preferences of EMBA 2019 students, _The Matrix_ is most similar to both _Inception_ and _Terminator_.

---
## Going back to the Users
Can we maybe cluster the users instead based on the ratings they gave the eight (8) movies, at most?

Definitely. And one of the most basic methods we can use is [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering). With $k$-means, we need to indicate the number of clusters $k$ we want our users to be grouped into.

Let us once again use `sklearn` to implement k-means.

```python
from sklearn.cluster import KMeans
```

```python
# setting k = 4 means that we're expecting 4 groupings in the EMBA 2019 cohort based
# on their movie ratings.
k = 4
kmeans = KMeans(n_clusters=k).fit(df.values)
```

```python
cluster_labels = kmeans.labels_
```

Below, we print the groupings.

```python
groupings = {}
for c in range(k):
    groupings[c] = []

persons = list(df.index)
i = 0
for i in range(len(persons)):
    groupings[cluster_labels[i]].append(persons[i])

for group, people in groupings.items():
    print ('Group #{}: {}'.format(group, people))
```

    Group #0: ['Person 1', 'Person 4', 'Person 7', 'Person 9', 'Person 10', 'Person 14', 'Person 15', 'Person 16', 'Person 19']
    Group #1: ['Person 2', 'Person 6', 'Person 8', 'Person 11', 'Person 12', 'Person 13', 'Person 17']
    Group #2: ['Person 5', 'Person 18', 'Person 20']
    Group #3: ['Person 3']

$k$-means is actually one of the most basic and (arguably) one of the more popular _unsupervised_ machine learning algorithms out there.

### Assigning _new_ users to existing groups.
With a model generated using our data, we can now assign (or "predict") new users to the generated clusters.

Say, for example, our new user has given our movies the following ratings:
- The Matrix: 0
- Inception: 0
- Titanic: 3
- Amelie: 2
- Love Actually: 5
- Terminator: 5
- Elysium: 5
- Avatar: 3

```python
kmeans.predict([[0,0,3,2,5,5,5,3]])
```

    array([2], dtype=int32)

What if we have another new user?

- The Matrix: 5
- Inception: 5
- Titanic: 5
- Amelie: 2
- Love Actually: 5
- Terminator: 1
- Elysium: 1
- Avatar: 3

```python
kmeans.predict([[5,5,5,2,5,1,1,3]])
```

    array([1], dtype=int32)

And another one?

```python
kmeans.predict([[1,1,5,2,5,1,1,3]])
```

    array([3], dtype=int32)

---

## Summary and Some Notes

- We showed how to load a MS Excel file into a dataframe.
- We implemented collaborative filtering in Python:
    - user-user
    - item-item
- We introduced (for demo) $k$-means clustering, an unsupervised ML model, to cluster users based on what they've watched and their taste in movies.
    - For the $k$-means clustering demo above, we explicitly chose the size of $k$. There are, of course, ways on how to systematically find the "right" number of clusters (e.g., using intertia plots), but they are beyond the scope of this intro to Python and Vanilla Collaborative Filtering method.
    - Once you learn more about the math/science behind $k$-means, you'll also realize that the membership of clusters may change for every run. This is because $k$-means starts by _randomly_ creating $k$ centroids, which are the points it uses to cluster the data points, which is based on their distance from the centroids.


# MLSS2019: Bayesian Deep Learning

In this tutorial we will uncertainty estimation can be
used in active learning or expert-in-the-loop pipelines.

The plan of the tutorial
1. [Imports and definitions](#Imports-and-definitions)
2. [Bayesian Active Learning with images](#Bayesian-Active-Learning-with-images)
   1. [The model](#The-model)
   2. [the Acquisition Function](#the-Acquisition-Function)
   3. [Data and the Oracle](#Data-and-the-Oracle)
   4. [the Active Learning loop](#the-Active-Learning-loop)
   5. [The baseline](#The-baseline)
3. [Bayesian Active Learning by Disagreement](#Bayesian-Active-Learning-by-Disagreement)
   1. [Points of improvement: batch-vs-single](#Points-of-improvement:-batch-vs-single)
   2. [Points of improvement: bias](#Points-of-improvement:-bias)

**(note)**
* to view documentation on something  type in `something?` (with one question mark)
* to view code of something type in `something??` (with two question marks).

## Imports and definitions

In this section we import necessary modules and functions and
define the computational device.

First, we install some boilerplate service code for this tutorial.

```python
!pip install -q --upgrade git+https://github.com/mlss-skoltech/tutorials.git#subdirectory=bayesian_deep_learning
```

Next, numpy for computing, matplotlib for plotting and tqdm for progress bars.

```python
import tqdm
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
```

For deep learning stuff will be using [pytorch](https://pytorch.org/).

If you are unfamiliar with it, it is basically like `numpy` with autograd,
native GPU support, and tools for building training and serializing models.

There are good introductory tutorials on `pytorch`, like this
[one](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html).

```python
import torch
import torch.nn.functional as F

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
```

Next we import the boilerplate code.

* a procedure that implements a minibatch SGD **fit** loop
* a function, that **evaluates** the model on the provided dataset

```python
from mlss2019bdl import fit, predict
```

The algorithm to sample a random function is:
* for $b = 1... B$ do:

  1. draw an independent realization $f_b\colon \mathcal{X} \to \mathcal{Y}$
  with from the process $\{f_\omega\}_{\omega \sim q(\omega)}$
  2. get $\hat{y}_{bi} = f_b(\tilde{x}_i)$ for $i=1 .. m$

```python
from mlss2019bdl.bdl import freeze, unfreeze

def sample_function(model, dataset, n_draws=1, verbose=False):
    """Draw a realization of a random function."""
    outputs = []
    for _ in tqdm.tqdm(range(n_draws), disable=not verbose):
        freeze(model)

        outputs.append(predict(model, dataset))

    unfreeze(model)

    return torch.stack(outputs, dim=0)
```

Sample the class probabilities $p(y_x = k \mid x, \omega, m)$
with $\omega \sim q(\omega)$ by a model that **outputs raw class
logit scores**.

```python
def sample_proba(model, dataset, n_draws=1):
    logits = sample_function(model, dataset, n_draws=n_draws)

    return F.softmax(logits, dim=-1)
```

Get the predictive posterior class probabilities
$$
p(y_x = k \mid x, m)
%     = \mathbb{E}_{\omega \sim q(\omega)}
%         p(y_x = k \mid x, \omega, m)
    \approx \frac1{\lvert \mathcal{W} \rvert}
        \sum_{\omega \in \mathcal{W}}
            p(y_x = k \mid x, \omega, m)
    \,, $$
with $\mathcal{W}$ -- iid draws from $q(\omega)$.

```python
def predict_proba(model, dataset, n_draws=1):
    proba = sample_proba(model, dataset, n_draws=n_draws)

    return proba.mean(dim=0)
```

Gat the maximum a posteriori class label **(MAP)**: $
\hat{y}_x
    = \arg \max_k \mathbb{E}_{\omega \sim q(\omega)}
        p(y_x = k \mid x, \omega, m)
$

```python
def predict_label(model, dataset, n_draws=1):
    proba = predict_proba(model, dataset, n_draws=n_draws)

    return proba.argmax(dim=-1)
```

We will need some functionality from scikit

```python
from sklearn.metrics import confusion_matrix

def evaluate(model, dataset, n_draws=1):
    assert isinstance(dataset, TensorDataset)

    predicted = predict_label(model, dataset, n_draws=n_draws)

    target = dataset.tensors[1].cpu().numpy()
    return confusion_matrix(target, predicted.cpu().numpy())
```

A function to plot images in a small dataset.

```python
from mlss2019bdl.flex import plot
from torch.utils.data import TensorDataset
from IPython.display import clear_output

def display(images, n_col=None, title=None, figsize=None, refresh=False):
    if isinstance(images, TensorDataset):
        images, targets = images.tensors

    if refresh:
        clear_output(True)

    fig, ax = plt.subplots(1, 1, figsize=figsize)
    plot(ax, images, n_col=n_col, cmap=plt.cm.bone)
    if title is not None:
        ax.set_title(title)

    plt.show()
    plt.close()
```

## Bayesian Active Learning with images

* Data labelling is costly and time consuming
* unlabeled instances are essentially free

**Goal** Achieve high performance with fewer labels by
identifying the best instances to learn from

Essential blocks of active learning:

* a **model** $m$ capable of quantifying uncertainty (preferably a Bayesian model)
* an **acquisition function** $a\colon \mathcal{M} \times \mathcal{X}^* \to \mathbb{R}$
  that for any finite set of inputs $S\subset \mathcal{X}$ quantifies their usefulness
  to the model $m\in \mathcal{M}$
* a labelling **oracle**, e.g. a human expert

### The model

We reuse the `DropoutLinear` from the first part.

```python
from torch.nn import Module, Sequential
from torch.nn import AvgPool2d, LeakyReLU
from torch.nn import Linear, Conv2d

from mlss2019bdl.bdl import DropoutLinear, DropoutConv2d

class MNISTModel(Module):
    def __init__(self, p=0.5):
        super().__init__()

        self.head = Sequential(
            Conv2d(1, 32, 3, 1),
            LeakyReLU(),
            DropoutConv2d(32, 64, 3, 1, p=p),
            LeakyReLU(),
            AvgPool2d(2),
        )

        self.tail = Sequential(
            DropoutLinear(12 * 12 * 64, 128, p=p),
            LeakyReLU(),
            DropoutLinear(128, 10, p=p),
        )

    def forward(self, input):
        """Take images and compute their class logits."""
        x = self.head(input)
        return self.tail(x.flatten(1))
```

### the Acquisition Function

There are many acquisition criteria (borrowed from [Gal17a](http://proceedings.mlr.press/v70/gal17a.html)):
* Classification
  * Posterior predictive entropy
  * Posterior Mutual Information
  * Variance ratios
  * BALD

* Regression
  * predictive variance

... and there is always the baseline **random acquisition**

```python
random_state = np.random.RandomState(812_760_351)

def random_acquisition(dataset, model, n_request=1, n_draws=1):
    indices = random_state.choice(len(dataset), size=n_request)

    return torch.from_numpy(indices).to(device)
```

### Data and the Oracle

Prepare the datasets from the `train` part of
[MNIST](http://yann.lecun.com/exdb/mnist/)
(or [Kuzushiji-MNIST](https://github.com/rois-codh/kmnist)):
* ($\mathcal{S}_\mathrm{train}$) initial **training**: $30$ images
* ($\mathcal{S}_\mathrm{valid}$) our **validation**:
  $5000$ images, stratified
* ($\mathcal{S}_\mathrm{pool}$) acquisition **pool**:
  $5000$ of the unused images, skewed to class $0$

The true test sample of MNIST is in $\mathcal{S}_\mathrm{test}$ -- we
will use it to evaluate the final performance.

```python
from mlss2019bdl.dataset import get_dataset

S_train, S_pool, S_valid, S_test = get_dataset(
    n_train=30,
    n_valid=5000,
    n_pool=5000,
    name="MNIST",  # "KMNIST"
    path="./data",
    random_state=722_257_201)
```

* `query_oracle(ix, D)` **request** the instances in `D` at the specified
  indices `ix` into a dataset and **remove** from them from `D`

* `merge(*datasets, [out=])` merge the datasets, creting a new one, or replacing `out`

```python
from mlss2019bdl.dataset import collect as query_oracle
```

### the Active Learning loop

1. fit $m$ on $\mathcal{S}_{\mathrm{labelled}}$

2. get exact (or approximate) $$
    \mathcal{S}^* \in \arg \max\limits_{S \subseteq \mathcal{S}_\mathrm{unlabelled}}
        a(m, S)
$$ satisfying **budget constraints** and **without** access to targets
(constraints, like $\lvert S \rvert \leq \ell$ or other economically motivated ones).

3. request the **oracle** to provide labels for each $x\in \mathcal{S}^*$

4. update $
\mathcal{S}_{\mathrm{labelled}}
    \leftarrow \mathcal{S}^*
        \cup \mathcal{S}_{\mathrm{labelled}}
$ and goto 1.

```python
import copy
from mlss2019bdl.dataset import merge

def active_learn(S_train,
                 S_pool,
                 S_valid,
                 acquire_fn,
                 n_budget=150,
                 n_max_request=3,
                 n_draws=11,
                 n_epochs=200,
                 p=0.5,
                 weight_decay=1e-2):

    model = MNISTModel(p=p).to(device)

    scores, balances = [], []
    S_train, S_pool = copy.deepcopy(S_train), copy.deepcopy(S_pool)
    while True:
        # 1. fit on train
        l2_reg = weight_decay * (1 - p) / max(len(S_train), 1)

        model = fit(model, S_train, batch_size=32, criterion="cross_entropy",
                    weight_decay=l2_reg, n_epochs=n_epochs)

        # (optional) keep track of scores and plot the train dataset
        scores.append(evaluate(model, S_valid, n_draws))
        balances.append(np.bincount(S_train.tensors[1], minlength=10))

        accuracy = scores[-1].diagonal().sum() / scores[-1].sum()
        title = f"(n_train) {len(S_train)} (Acc.) {accuracy:.1%}"
        display(S_train, n_col=30, figsize=(15, 5), title=title, refresh=True)

        # 2-3. request new data from pool, if within budget
        n_request = min(n_budget - len(S_train), n_max_request)
        if n_request <= 0:
            break

        indices = acquire_fn(S_pool, model, n_request=n_request, n_draws=n_draws)

        # 4. update the train dataset
        S_requested = query_oracle(indices, S_pool)
        S_train = merge(S_train, S_requested)

    return model, S_train, np.stack(scores, axis=0), np.stack(balances, axis=0)
```

* `collect(ix, D)` **collect** the instances in `D` at the specified
  indices `ix` into a dataset and **remove** from them from `D`

* `merge(*datasets, [out=])` merge the datasets, creting a new one, or replacing `out`

### The baseline

How powerful will our model with random acquisition get under a total budget of $150$ images?

```python
baseline = active_learn(
    S_train,
    S_pool,
    S_valid,
    random_acquisition,
    n_draws=21,
    n_budget=150,
    n_max_request=3,
    n_epochs=200,
)
```

Let's see the dynamics of the accuracy ...

```python
def accuracy(scores):
    tp = scores.diagonal(axis1=-2, axis2=-1)
    return tp.sum(-1) / scores.sum((-2, -1))
```

```python
model_rand, train_rand, scores_rand, balances_rand = baseline

fig, ax = plt.subplots(1, 1, figsize=(12, 7))
ax.plot(accuracy(scores_rand), label='Accuracy (random)', lw=2)

ax.legend()
plt.show()
```

..., and the frequency of each class in $\mathcal{S}_\mathrm{train}$.

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 7))

lines = ax.plot(balances_rand, lw=2)
plt.legend(lines, list(range(10)), ncol=2);
```

## Bayesian Active Learning by Disagreement

Bayesian Active Learning by Disagreement, or **BALD** criterion, is
based on the posterior mutual information between model's predictions
$y_x$ at some point $x$ and its parameters $\omega$:

$$\begin{align}
    a(m, S)
        &= \sum_{x\in S} a(m, \{x\})
        \\
    a(m, \{x\})
        &= \mathbb{I}(y_x; \omega \mid x, m, D)
\end{align}
    \,, \tag{bald} $$

with the [**Mutual Information**](https://en.wikipedia.org/wiki/Mutual_information#Relation_to_Kullback%E2%80%93Leibler_divergence)
(**MI**)
$$
    \mathbb{I}(y_x; \omega \mid x, m, D)
        = \mathbb{H}\bigl(
            \mathbb{E}_{\omega \sim q(\omega\mid m, D)}
                p(y_x \,\mid\, x, \omega, m, D)
        \bigr)
        - \mathbb{E}_{\omega \sim q(\omega\mid m, D)}
            \mathbb{H}\bigl(
                p(y_x \,\mid\, x, \omega, m, D)
            \bigr)
    \,, \tag{mi} $$

and the [(differential) **entropy**](https://en.wikipedia.org/wiki/Differential_entropy#Differential_entropies_for_various_distributions)
(all densities and/or probability mass functions can be conditional):

$$
    \mathbb{H}(p(y))
        = - \mathbb{E}_{y\sim p} \log p(y)
    \,. $$

#### (task) Implementing the acquisition function

Note that $a(m, S)$ is additively separable in $S$, i.e.
equals $\sum_{x\in S} a(m, \{x\})$. This implies

$$
\begin{align}
    \max_{S \subseteq \mathcal{S}_\mathrm{unlabelled}} a(m, S)
        &= \max_{z \in \mathcal{S}_\mathrm{unlabelled}}
            \max_{F \in \mathcal{S}_\mathrm{unlabelled} \setminus \{z\}}
            \sum_{x\in F \cup \{x\}} a(m, \{x\})
        \\
        &= \max_{z \in \mathcal{S}_\mathrm{unlabelled}}
            a(m, \{z\})
            + \max_{F \in \mathcal{S}_\mathrm{unlabelled} \setminus \{z\}}
                \sum_{x\in F} a(m, \{x\})
\end{align}
    \,. $$

Therefore selecting the $\ell$ `most interesting` points from
$\mathcal{S}_\mathrm{unlabelled}$ is trivial.

The acquisition function that we implement has interface
identical to `random_acquisition` but uses BALD to choose
instances.

```python
def BALD_acquisition(dataset, model, n_request=1, n_draws=1):
    proba = sample_proba(model, dataset, n_draws=n_draws)

    ## Exercise: implement BALD

    pass
```

#### (task) implementing entropy

For categorical (discrete) random variables $y \sim \mathcal{Cat}(\mathbf{p})$,
$\mathbf{p} \in \{ \mu \in [0, 1]^d \colon \sum_k \mu_k = 1\}$, the entropy is

$$
    \mathbb{H}(p(y))
        = - \mathbb{E}_{y\sim p(y)} \log p(y)
        = - \sum_k p_k \log p_k
    \,. $$

**(note)** although in calculus $0 \cdot \log 0 = 0$ (because
$\lim_{p\downarrow 0} p \cdot \log p = 0$), in floating point
arithmetic $0 \cdot \log 0 = \mathrm{NaN}$. So you need to add
some **really tiny float number** to the argument of $\log$.

```python
def categorical_entropy(proba):
    """Compute the entropy along the last dimension."""

    ## Exercise: the probabilities sum to one along the last axis.
    #  Please, compute their entropy.

    pass
```

#### (task) implementing mutual information

Consider a tensor $p_{bik}$ of probabilities $p(y_{x_i}=k \mid x_i, \omega_b, m, D)$
with $\omega_b \sim q(\omega \mid m, D)$ with $\mathcal{W} = (\omega_b)_{b=1}^B$
being iid draws from $q(\omega \mid m, D)$.

Let's implement a procedure that computes the Monte Carlo estimate of the
posterior predictive distribution, its **entropy** and **mutual information**

$$
    \mathbb{I}_\mathrm{MC}(y_x; \omega \mid x, m, D)
        = \mathbb{H}\bigl(
            \hat{p}(y_x\mid x, m, D)
        \bigr)
        - \frac1{\lvert \mathcal{W} \rvert} \sum_{\omega\in \mathcal{W}}
            \mathbb{H}\bigl(
                p(y_x \,\mid\, x, \omega, m, D)
            \bigr)
    \,, \tag{mi-mc} $$
where
$$
\hat{p}(y_x\mid x, m, D)
    = \frac1{\lvert \mathcal{W} \rvert} \sum_{\omega\in \mathcal{W}}
        \,p(y_x \mid x, \omega, m, D)
    \,. $$

```python
def mutual_information(proba):
    ## Exercise: compute a Monte Carlo estimator of the predictive
    ##   distribution, its entropy and MI `H E_w p(., w) - E_w H p(., w)`

    pass
```

How powerful will our model with **BALD** acquisition, if we can afford no more than $150$ images?

```python
bald_results = active_learn(
    S_train,
    S_pool,
    S_valid,
    BALD_acquisition,
    n_draws=21,
    n_budget=150,
    n_max_request=3,
    n_epochs=200,
)
```

Let's see the dynamics of the accuracy ...

```python
model_bald, train_bald, scores_bald, balances_bald = bald_results

fig, ax = plt.subplots(1, 1, figsize=(12, 7))

ax.plot(accuracy(scores_rand), label='Accuracy (random)', lw=2)
ax.plot(accuracy(scores_bald), label='Accuracy (BALD)', lw=2)

ax.legend()
plt.show()
```

..., and the frequency of each class in $\mathcal{S}_\mathrm{train}$.

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 7))

lines = ax.plot(balances_bald, lw=2)
plt.legend(lines, list(range(10)), ncol=2);
```

#### Class performance

The *one-versus-rest* precision / recall scores on
$\mathcal{S}_\mathrm{valid}$. For binary classification:

$$ \begin{align}
\mathrm{Precision}
    &= \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}
        \approx \mathbb{P}(y = 1 \mid \hat{y} = 1)
    \,, \\
\mathrm{Recall}
    &= \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}
        \approx \mathbb{P}(\hat{y} = 1 \mid y = 1)
    \,.
\end{align}$$

```python
import pandas as pd

def pr_scores(score_matrix):
    tp = score_matrix.diagonal(axis1=-2, axis2=-1)
    fp, fn = score_matrix.sum(axis=-2) - tp, score_matrix.sum(axis=-1) - tp

    return pd.DataFrame({
        "precision": {l: f"{p:.2%}" for l, p in enumerate(tp / (tp + fp))},
        "recall": {l: f"{p:.2%}" for l, p in enumerate(tp / (tp + fn))},
    })
```

Let's see the performance on the test set

```python
scores = {}
scores["rand"] = evaluate(model_rand, S_test, n_draws=21)
scores["bald"] = evaluate(model_bald, S_test, n_draws=21)
```

```python
df = pd.concat({
    name: pr_scores(score)
    for name, score in scores.items()
}, axis=1).T

df.swaplevel().sort_index()
```

#### Question(s) (to work on in your spare time)

* Run the experiments on the `KMNIST` dataset

* Replicate figure 1 from [Gat et al. (2017): p. 4](http://proceedings.mlr.press/v70/gal17a.html).
  You will need to re-run each experiment several times $11$, recording
  the accuracy dynamics of each, then compare the mean and $25\%$-$75\%$
  quantiles as they evolve with the size of the training sample.

### (optional) Points of improvement: batch-vs-single

A drawback of the `pointwise` top-$\ell$ procedure above is that, although
it acquires individually informative instances, altogether they might end
up **being** `jointly poorly informative`. This can be corrected, if we
would seek the highest mutual information among finite sets $
S \subseteq \mathcal{S}_\mathrm{unlabelled}
$ of size $\ell$.

Such acquisition function is called **batch-BALD**
([Kirsch et al.; 2019](https://arxiv.org/abs/1906.08158.pdf)):

$$\begin{align}
    a(m, S)
        &= \mathbb{I}\bigl((y_x)_{x\in S}; \omega \mid S, m \bigr)
        = \mathbb{H} \bigl(
            \mathbb{E}_{\omega \sim q(\omega\mid m)} p\bigl((y_x)_{x\in S}\mid S, \omega, m \bigr)
        \bigr)
        - \mathbb{E}_{\omega \sim q(\omega\mid m)} H\bigl(
            p\bigl((y_x)_{x\in S}\mid S, \omega, m \bigr)
        \bigr)
\end{align}
    \,. \tag{batch-bald} $$

This criterion requires combinatorially growing number of computations and
memory, however there are working solutions like random sampling of subsets
$\mathcal{S}$ from $\mathcal{S}_\mathrm{unlabelled}$ or greedily maximizing
of this **submodular** criterion.

### (optional) Points of improvement: bias

The first term in the **MC** estimate of the mutual information is the
so-called **plug-in** estimator of the entropy:

$$
    \hat{H}
        = \mathbb{H}(\hat{p}) = - \sum_k \hat{p}_k \log \hat{p}_k
    \,, $$

where $\hat{p}_k = \tfrac1B \sum_b p_{bk}$ is the full sample estimator
of the probabilities.

It is known that this plug-in estimate is biased
(see [blog: Nowozin, 2015](http://www.nowozin.net/sebastian/blog/estimating-discrete-entropy-part-1.html)
and references therein, also this [notebook](https://colab.research.google.com/drive/1z9ZDNM6NFmuFnU28d8UO0Qymbd2LiNJW)).
In order to correct for small-sample bias we can use
[jackknife resampling](https://en.wikipedia.org/wiki/Jackknife_resampling).
It derives an estimate of the finite sample bias from the leave-one-out
estimators of the entropy and is relatively computationally cheap
(see [blog: Nowozin, 2015](http://www.nowozin.net/sebastian/blog/estimating-discrete-entropy-part-2.html),
[Miller, R. G. (1974)](http://www.math.ntu.edu.tw/~hchen/teaching/LargeSample/references/Miller74jackknife.pdf) and these [notes](http://people.bu.edu/aimcinto/jackknife.pdf)).

The jackknife correction of a plug-in estimator $\mathbb{H}(\cdot)$
is computed thus: given a sample $(p_b)_{b=1}^B$ with $p_b$ -- discrete distribution on $1..K$
* for each $b=1.. B$
  * get the leave-one-out estimator: $\hat{p}_k^{-b} = \tfrac1{B-1} \sum_{j\neq b} p_{jk}$
  * compute the plug-in entropy estimator: $\hat{H}_{-b} = \mathbb{H}(\hat{p}^{-b})$
* then compute the bias-corrected entropy estimator $
\hat{H}_J
    = \hat{H} + (B - 1) \bigl\{
        \hat{H} - \tfrac1B \sum_b \hat{H}^{-b}
    \bigr\}
$

**(note)** when we knock the $i$-th data point out of the sample mean
$\mu = \tfrac1n \sum_i x_i$ and recompute the mean $\mu_{-i}$ we get
the following relation
$$ \mu_{-i}
    = \frac1{n-1} \sum_{j\neq i} x_j
    = \frac{n}{n-1} \mu - \tfrac1{n-1} x_i
    = \mu + \frac{\mu - x_i}{n-1}
    \,. $$
This makes it possible to quickly compute leave-one-out estimators of
discrete probability distribution.

#### (task*) Unbiased estimator of entropy and mutual information

Try to efficiently implement a bias-corrected acquisition
function, and see it is worth the effort.

```python
def BALD_jknf_acquisition(dataset, model, n_request=1, n_draws=1):
    proba = sample_proba(model, dataset, n_draws=n_draws)

    ## Exercise: MC estimate of the predictive distribution, entropy and MI
    ##  mutual information `H E_w p(., w) - E_w H p(., w)` with jackknife
    ##  correction.

    pass
```

Let's see ...

```python
jknf_results = active_learn(
    S_train,
    S_pool,
    S_valid,
    BALD_jknf_acquisition,
    n_draws=21,
    n_budget=150,
    n_max_request=3,
    n_epochs=200,
)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 7))

model_jknf, train_jknf, scores_jknf, balances_jknf = jknf_results
ax.plot(accuracy(scores_rand), label='Accuracy (random)', lw=2)
ax.plot(accuracy(scores_bald), label='Accuracy (BALD)', lw=2)
ax.plot(accuracy(scores_jknf), label='Accuracy (BALD-jknf)', lw=2)

ax.legend()
plt.show()
```

```python
fig, ax = plt.subplots(1, 1, figsize=(12, 7))

lines = ax.plot(balances_jknf, lw=2)
plt.legend(lines, list(range(10)), ncol=2);
```

Machine Learning Summer School 2019 (Moscow, Russia)/tutorials/bayesian_deep_learning/Bayesian Deep Learning part2.ipynb


## Devito CFD Tutorial series

The following series of notebook tutorials will demonstrate the use of Devito and it's SymPy-based API to solve a set of classic examples from Computational Fluid Dynamics (CFD). The tutorials are based on the excellent tutorial series _CFD Python: 12 steps to Navier-Stokes_ by Lorena Barba and focus on the implementation with Devito rather than pure CFD or finite difference theory. For a refresher on how to implement 2D finite difference solvers for CFD problems, please see the original tutorial series here: <br>
http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/

### Example 1: Linear convection in 2D

Lets start with a simple 2D convection example - step 5 in the original blog. This will already allow us to demonstrate a lot about the use of Devito's symbolic data objects and how to use them to build a simple operator directly from the symbolic notation of the equation. The governing equation we will implement in this tutorial is:

$$\frac{\partial u}{\partial t}+c\frac{\partial u}{\partial x} + c\frac{\partial u}{\partial y} = 0$$

In order to implement this equation we first discretize it using forward differences in time and backward differences in space. Just as the original tutorial, we will use $u_{i,j}^n$ to denote a finite difference stencil point with $i$ and $j$ denoting spatial indices and $n$ denoting the time index. So, after re-arranging the discretized equation for the forward stencil point in time we get

$$u_{i,j}^{n+1} = u_{i,j}^n-c \frac{\Delta t}{\Delta x}(u_{i,j}^n-u_{i-1,j}^n)-c \frac{\Delta t}{\Delta y}(u_{i,j}^n-u_{i,j-1}^n)$$

Using this, we can start deriving the computational stencil for this equation. Let's first look at the original _pure Python_ implementation of the linear convection flow - but first we import our tools and define some parameters:

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 81
ny = 81
nt = 100
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
print("dx %s, dy %s" % (dx, dy))
sigma = .2
dt = sigma * dx
```

    dx 0.025, dy 0.025

**A small note on style:** Throughout this tutorial series we will use utility functions to plot the various 2D functions and data sets we deal with. These are all taken from the original tutorial series, but have been slightly modified for our purposes. One of the differences readers might find is that the original series uses _(y, x)_ indexing for 2d data arrays, whereas many of the examples have been adapted to use _(x, y)_ notation in our tutorials.

So, let's start by creating a simple 2D function and initialising it with a "hat function". We will use that initialisation function a lot, so it comes from our utility scripts:

```python
#NBVAL_IGNORE_OUTPUT

# Create field and assign initial conditions
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)

# Plot initial condition
plot_field(u)
```

Now we can repeat the _pure NumPy_ solve from the original tutorial, where we use NumPy array operations to speed up the computation. Note that we skip the derivation of the stencil used to implement our convection equation, as we are going to walk though this process using the Devito API later in this tutorial.

```python
# Repeat initialisation, so we can re-run the cell
init_hat(field=u, dx=dx, dy=dy, value=2.)

for n in range(nt + 1):
    # Copy previous result into a new buffer
    un = u.copy()

    # Update the new result with a 3-point stencil
    u[1:, 1:] = (un[1:, 1:] - (c * dt / dy * (un[1:, 1:] - un[1:, :-1])) -
                              (c * dt / dx * (un[1:, 1:] - un[:-1, 1:])))

    # Apply boundary conditions.
    u[0, :] = 1.  # left
    u[-1, :] = 1. # right
    u[:, 0] = 1.  # bottom
    u[:, -1] = 1. # top
    # Note that in the above expressions the NumPy index -1 corresponds to the final point of the array along the indexed dimension,
    #    i.e. here u[-1, :] is equivalent to u[80, :].

```

```python
#NBVAL_IGNORE_OUTPUT

# A small sanity check for auto-testing
assert (u[45:55, 45:55] > 1.8).all()
u_ref = u.copy()

plot_field(u)
```

Hooray, the wave moved! Now, this little example is already very concise from a notational point of view and it teaches us quite a few key points about how to perform finite difference stencil computation via NumPy:
* Due to the backward differencing scheme in space (more later) we use only three stencil point in this example:  $u^{n}_{i, j}$, $u^{n}_{i-1, j}$ and $u^{n}_{i, j-1}$. These can be identified in the code through the array indices and correspond to `un[1:, 1:]`, `un[:-1, 1:]` and `un[1:, :-1]` respectively.
* Two buffers for array data are used throughout: `un[...]` is read from, while `u[...]` is updated, where the line `un = u.copy()` performs a deep copy of the field to switch buffers between timesteps. Note that in some other finite difference tutorials the cost of this copy operation is sometimes amortised by using two pre-allocated buffers and switching the indices of them explicitly.
* The final four lines within the loop code show us how to implement simple Dirichlet boundary condition by simply setting a value on the outermost rows and columns of our cartesian grid.

You may have noticed that the hat has not only moved to a different location, but has also changed its shape into a smooth bump. This is a little surprising, as the correct solution of the convection equation would be movement without shape change. The smooth shape is caused by numerical diffusion, a well-known limitation of low-order finite difference schemes. We will discuss this issue and some solutions later in this tutorial.

#### Devito implementation
Now we want to re-create the above example via a Devito operator. To do this, we can start by defining our computational grid and creating a function `u` as a symbolic `devito.TimeFunction`. The core thing to note here is that this is one of Devito's symbolic functions, which have a dual role in the creation of finite difference solvers:
* They behave symbolically like `sympy.Function` objects, so that we can construct derivatives and use them in symbolic expressions, thus inheriting all the power of automated symbolic manipulation that SymPy provides.
* They act as containers for user data by providing a `.data` property that wraps automatically allocated memory space in a neat NumPy array.

The particular `TimeFunction` type that we will declare our variable $u$ as in this case is aware of the fact that we will want to implement a timestepping algorithm with it. So the object `u` will declare two buffers of shape `(nx, ny)` for us, as defined by the `Grid` object, and present them as `u.data[0]` and `u.data[1]`. Let's fill the initial buffer with some data and look at it:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
# Note u.data[0] == u.data[0,:,:]
plot_field(u.data[0])
```

Nice. Now we can look at deriving our 3-point stencil using the symbolic capabilities given to our function $u$ by SymPy. For this we will first construct our derivative terms in space and time. For the forward derivative in time we can easily use Devito's shorthand notation `u.dt` to denote the first derivative in time and `u.dxl` and `u.dyl` to denote the space derivatives. Note that the `l` means were using the "left" or backward difference here to adhere to the discretization used in the original tutorials.

From the resulting terms we can then create a `sympy.Equation` object that contains the fully discretized equation, but from a neat high-level notation, as shown below.

```python
from devito import Eq

# Specify the `interior` flag so that the stencil is only
# applied to the interior of the domain.
eq = Eq(u.dt + c*u.dxl + c*u.dyl, subdomain=grid.interior)

print(eq)
```

    Eq(Derivative(u(t, x, y), x) + Derivative(u(t, x, y), y) + Derivative(u(t, x, y), t), 0)

The above step resulted in a fully discretized version of our equation, which includes place-holder symbols for the spacing in time (`dt`) and space (`h`). These symbols are based on an internal convention and will later be replaced when we build an operator.

But before we can build an operator, we first need to change our discretized expression so that we are updating the forward stencil point in our timestepping scheme - Devito provides another short-hand notation for this: `u.forward`. For the actual symbolic reordering, SymPy comes to the rescue with the `solve` utility that we can use to re-organise our equation.

```python
from devito import solve
from sympy import nsimplify, pprint

stencil = solve(eq, u.forward)

# nsimplify: 1.0*x = x
pprint(nsimplify(stencil))
```

       ⎛  ∂                ∂                u(t, x, y)⎞
    dt⋅⎜- ──(u(t, x, y)) - ──(u(t, x, y)) + ──────────⎟
       ⎝  ∂x               ∂y                   dt    ⎠

The careful reader will note that this is equivalent to the symbolic expression for the RHS of the NumPy code we showed earlier - only with `dx` and `dy` denoted as `h_x` and `h_y` respectively, while `u(t, x, y)`, `u(t, x - h, y)` and `u(t, x, y - h)` denote the equivalent of $u^{n}_{i, j}$, $u^{n}_{i-1, j}$ and $u^{n}_{i, j-1}$ analytically and `u[1:, 1:]`, `u[:-1, 1:]` and `u[1:, :-1]` in the numpy implementation respectively.

We can now use this stencil expression to create an operator to apply to our data object:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our initial condition in both buffers.
# This is required to avoid 0s propagating into
# our solution, which has a background value of 1.
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=u.data[1], dx=dx, dy=dy, value=2.)

# Create an operator that updates the forward stencil point
op = Operator(Eq(u.forward, stencil, subdomain=grid.interior))

# Apply the operator for a number of timesteps
op(time=nt, dt=dt)

plot_field(u.data[0, :, :])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

Great, that looks to have done the same thing as the original NumPy example, so we seem to be doing something right, at least.

**A note on performance:** During the code generation phase of the previous operator Devito has introduced several performance optimisations. We will ignore this aspect for now, also because our example is tiny - but for large runs where performance matters, these optimisations will make a Devito kernel run very fast in comparison to raw Python/NumPy.

Now, despite getting a correct looking result, there is still one problem with the above operator: It doesn't set any boundary conditions as part of the time loop. We also note that the operator includes a time loop, but at this point Devito doesn't actually provide any language constructs to explicitly define different types of boundary conditions (Devito is probably still a kind of prototype at this point). Luckily though, Devito provides a backdoor for us to insert custom expression in the so-called "indexed" or "low-level" API that allow us to encode the Dirichlet boundary condition of the original example.

#### The "indexed" or low-level API
The `TimeFunction` field we created earlier behaves symbolically like a `sympy.Function` object with the appropriate indices, eg. `u(t, x, y)`. If we take a simple first-order derivative of that we have a term that includes the spacing variable `h`, which Devito uses as the default for encoding $dx$ or $dy$. For example, `u.dx` simply expands to `-u(t, x, y)/h + u(t, x + h, y)/h`.

Now, when the `Operator` creates explicit C-code from that expression, it at some point "lowers" that expression by resolving explicit data accesses (or indices) into our grid by transforming it into a `sympy.Indexed` object. During this process all occurrences of `h` in data accesses get replaced with integers, so that the expression now looks like `-u[t, x, y]/h + u[t, x + 1, y]/h`. This is the "indexed" notation and we can create custom expression of the same kind by explicitly writing `u[...]`, that is with indices in square-bracket notation. These custom expressions can then be injected into our operator like this:

```python
#NBVAL_IGNORE_OUTPUT

# Reset our data field and ICs in both buffers
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=u.data[1], dx=dx, dy=dy, value=2.)

# For defining BCs, we want to explicitly set rows/columns in our field
# We can use Devito's "indexed" notation to do this:
x, y = grid.dimensions
t = grid.stepping_dim
bc_left = Eq(u[t + 1, 0, y], 1.)
bc_right = Eq(u[t + 1, nx-1, y], 1.)
bc_top = Eq(u[t + 1, x, ny-1], 1.)
bc_bottom = Eq(u[t + 1, x, 0], 1.)

# Now combine the BC expressions with the stencil to form operator
expressions = [Eq(u.forward, stencil)]
expressions += [bc_left, bc_right, bc_top, bc_bottom]
op = Operator(expressions=expressions, opt=None, openmp=False)  # <-- Turn off performance optimisations
op(time=nt, dt=dt)

plot_field(u.data[0])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

You might have noticed that we used the argument `opt=None` in the creation of the previous operator. This suppresses the various performance optimisation steps in the code-generation pipeline, which makes the auto-generated C code much easier to look at. So, for the brave, let's have a little peek under the hood...

```python
print(op.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #define START_TIMER(S) struct timeval start_ ## S , end_ ## S ; gettimeofday(&start_ ## S , NULL);
    #define STOP_TIMER(S,T) gettimeofday(&end_ ## S, NULL); T->S += (double)(end_ ## S .tv_sec-start_ ## S.tv_sec)+(double)(end_ ## S .tv_usec-start_ ## S .tv_usec)/1000000;

    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"

    struct dataobj
    {
      void *restrict data;
      int * size;
      int * npsize;
      int * dsize;
      int * hsize;
      int * hofs;
      int * oofs;
    } ;

    struct profiler
    {
      double section0;
      double section1;
      double section2;
    } ;

    int Kernel(const float dt, const float h_x, const float h_y, struct dataobj *restrict u_vec, const int time_M, const int time_m, const int x_M, const int x_m, const int y_M, const int y_m, struct profiler * timers)
    {
      float (*restrict u)[u_vec->size[1]][u_vec->size[2]] __attribute__ ((aligned (64))) = (float (*)[u_vec->size[1]][u_vec->size[2]]) u_vec->data;

      for (int time = time_m, t0 = (time)%(2), t1 = (time + 1)%(2); time <= time_M; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        /* Begin section0 */
        START_TIMER(section0)
        for (int x = x_m; x <= x_M; x += 1)
        {
          for (int y = y_m; y <= y_M; y += 1)
          {
            u[t1][x + 1][y + 1] = dt*(-(-u[t0][x][y + 1]/h_x + u[t0][x + 1][y + 1]/h_x) - (-u[t0][x + 1][y]/h_y + u[t0][x + 1][y + 1]/h_y) + u[t0][x + 1][y + 1]/dt);
          }
        }
        STOP_TIMER(section0,timers)
        /* End section0 */

        /* Begin section1 */
        START_TIMER(section1)
        for (int y = y_m; y <= y_M; y += 1)
        {
          u[t1][1][y + 1] = 1.00000000000000F;

          u[t1][81][y + 1] = 1.00000000000000F;
        }
        STOP_TIMER(section1,timers)
        /* End section1 */

        /* Begin section2 */
        START_TIMER(section2)
        for (int x = x_m; x <= x_M; x += 1)
        {
          u[t1][x + 1][81] = 1.00000000000000F;

          u[t1][x + 1][1] = 1.00000000000000F;
        }
        STOP_TIMER(section2,timers)
        /* End section2 */
      }

      return 0;
    }

```python

```


```python
%%%HTML

<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>
<a href="https://github.com/a1rb4Ck/MECA654"></a>
```

<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
# MECA654, March 2018, Pierre Nagorny
# Polytech Annecy-Chambéry, SYMME lab, Savoie Mont Blanc University

from IPython.display import display, HTML, IFrame, Math
from IPython.core.interactiveshell import InteractiveShell
import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt  # Loading Pyplot
from scipy.interpolate import interp1d
# Setting Matplotlib render backend
%matplotlib notebook
# %matplotlib nbagg
# %pylab inline
from sympy import *  # Loading Sympy for symbolic calculus
from sympy.physics.vector import *  # Everything needed for vector calculs, ReferenceFrame, dynamic
from sympy.physics.vector import init_vprinting  # To print time varying vectors

init_printing()
init_vprinting(pretty_print=True)

# Display full output in Jupyter Notebook
InteractiveShell.ast_node_interactivity = "all"
```

__MECA654, March 2018, Polytech Annecy-Chambéry__
[This interactive notebook is on Github](https://github.com/a1rb4Ck/MECA654)

## Exercise 7 - Spherical robot
### Speeds composition and acceleration calculus

A robot is composed of 3 bodies.
The mechanical links between bodies are :
$0 / 1$ : pivot with axis $O \vec{y_1}$
$1 / 2$ : pivot with axis $O \vec{z_1}$
$2 / 3$ : linear slide with axis $O \vec{x_2}$

Those links are controlled by three motors. Thus angles $\theta_1$, $\theta_2$ and the lenght $r$ are variable.

```python
IFrame('http://www.glowscript.org/#/user/a1rb4Ck/folder/MECA654/program/sphericalrobot', width=640, height=562)
```

```python
# dynamicsymbols is a shortcut function to create undefined functions of time within SymPy
r, theta_1, theta_2, t, OM = symbols('r theta_1 theta_2 t OM')
theta_1, theta_2 = dynamicsymbols('theta_1 theta_2')
theta_1dot = dynamicsymbols('theta_1', 1)
theta_2dot = dynamicsymbols('theta_2', 1)

# Our reference frames for each body:
B0 = ReferenceFrame('B0', indices=['i', 'j', 'k'])
B1 = ReferenceFrame('B1', indices=['i', 'j', 'k'])
B2 = ReferenceFrame('B2', indices=['i', 'j', 'k'])

# We define space transformation relations in our problem:
B1.orient(B0, 'Axis', [theta_1, B0.y])  # Rotation of axis (O, y1), same as (O, y0)
B2.orient(B1, 'Axis', [-theta_2, B1.z])  # Rotation of axis (O, z1)

# We can easily get the DCM of any reference frame!
# B1.dcm(B0)
```

**1. What is the movement of $(3)$ in relation to $(2)$?** _It's a translation of $r$._

---
**Method for solving**
1. Differentiate the $\overrightarrow{OM}$ vector in the main reference frame, here it is $(2)$
2. Express the vector in the asked reference frame, here it is $(1)$
---

- Differentiate $\overrightarrow{V_{23}(M)}$ in the $(2)$ reference frame:

```python
# We define the OM vector translation
OM = r(t) * B2['i']
display(Math(r'\overrightarrow{OM}=%s \ %s' % (latex(OM), latex('in \ the \ B_2 \ base'))))
```

$$\overrightarrow{OM}=r\mathbf{\hat{b2}_{i}} \ in \ the \ B_2 \ base$$

```python
# Differentiate a vector with respect to a variable in a reference frame:
# V_23 = OM.diff(t, B2)
V_23 = OM.dt(B2)  # The reference frame for derivation is (2) because we want V_23 !
display(Math(r'\overrightarrow{V_{23}(M)}=%s \ %s' % (latex(V_23), latex('in \ the \ B_2 \ base'))))
```

$$\overrightarrow{V_{23}(M)}=\dot{r}\mathbf{\hat{b2}_{i}} \ in \ the \ B_2 \ base$$

- Express $\overrightarrow{V_{23}(M)}$ in the $(1)$ reference frame:

```python
# We project the x2 vector in the B1 reference frame
# x_2 = cos(theta_2) * x_1 + sin(theta_2) * y_1 + 0 * z_1
# y_2 = -sin(theta_2) * x_1 + cos(theta_2) * y_1 + 0 * z_1
# z_2 = z_1

# Express the vector that we originally wrote in the B2 frame, in the B1 frame:
# V_23.express(B1)  # V_23 in the B1 base
display(Math(r'\overrightarrow{V_{23}(M)}=%s \ %s' % (latex(V_23.express(B1)), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{V_{23}(M)}=\operatorname{cos}\left(\theta_{2}\right) \dot{r}\mathbf{\hat{b1}_{i}} -  \operatorname{sin}\left(\theta_{2}\right) \dot{r}\mathbf{\hat{b1}_{j}} \ in \ the \ B_1 \ base$$

**2. What is the movement of (2) in relation to (1)?** _It's a rotation of angle $\theta_2$._
- Differentiate $\overrightarrow{V_{12}(M)}$ in the $(1)$ reference frame:

```python
# We define the OM rotation movement
OM = r * B2.x
display(Math(r'\overrightarrow{OM}=%s \ %s' % (latex(OM), latex('in \ the \ B_2 \ base'))))
```

$$\overrightarrow{OM}=r\mathbf{\hat{b2}_{i}} \ in \ the \ B_2 \ base$$

```python
# Angular velocity of B2 with respect to B1
# B2.ang_vel_in(B1)
display(Math(r'\Omega_2=%s \ %s' % (latex(B2.ang_vel_in(B1)), latex('angular \ velocity \ of \ B_2 \ with \ respect \ to \ B_1'))))
```

$$\Omega_2=-  \dot{\theta}_{2}\mathbf{\hat{b1}_{k}} \ angular \ velocity \ of \ B_2 \ with \ respect \ to \ B_1$$

```python
V_12 = OM.dt(B1)  # The reference frame for derivation is (1) because we want V_12!
# V_12  # V_12 in the B2 base
display(Math(r'\overrightarrow{V_{12}(M)}=%s \ %s' % (latex(V_12), latex('in \ the \ B_2 \ base'))))
```

$$\overrightarrow{V_{12}(M)}=-  r \dot{\theta}_{2}\mathbf{\hat{b2}_{j}} \ in \ the \ B_2 \ base$$

- Express $\overrightarrow{V_{12}(M)}$ in the $(1)$ reference frame:

```python
# We project the y2 vector in the B1 reference frame
# V_12.express(B1)  # V_12 in the B1 base
display(Math(r'\overrightarrow{V_{12}(M)}=%s \ %s' % (latex(V_12.express(B1)), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{V_{12}(M)}=-  r \operatorname{sin}\left(\theta_{2}\right) \dot{\theta}_{2}\mathbf{\hat{b1}_{i}} -  r \operatorname{cos}\left(\theta_{2}\right) \dot{\theta}_{2}\mathbf{\hat{b1}_{j}} \ in \ the \ B_1 \ base$$

**3. What is the movement of $(1)$ in relation to $(0)$?** _It's a rotation of angle $\theta_1$_
- Differentiate $\overrightarrow{V_{01}(M)}$ in the $(0)$ reference frame:

```python
# We define the OM rotation movement
OM = B1.x
display(Math(r'\overrightarrow{OM}=%s \ %s' % (latex(OM), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{OM}=\mathbf{\hat{b1}_{i}} \ in \ the \ B_1 \ base$$

```python
V_01 = OM.dt(B0)  # The reference frame for derivation is (0) because we want V_01!
display(Math(r'\overrightarrow{V_{01}(M)}=%s \ %s' % (latex(V_01), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{V_{01}(M)}=-  \dot{\theta}_{1}\mathbf{\hat{b1}_{k}} \ in \ the \ B_1 \ base$$

**4. What is the speed of M in the reference frame $(0)$?**
- $\overrightarrow{V_{03}(M)} = \overrightarrow{V_{01}(M)} + \overrightarrow{V_{12}(M)} + \overrightarrow{V_{23}(M)}$ in the same reference frame, here we choose $(1)$ :

```python
V_03 = V_01.express(B1) + V_12.express(B1) + V_23.express(B1)
# V_03.to_matrix(B1)
display(Math(r'\overrightarrow{V_{03}(M)}=%s \ %s' % (latex(V_03.express(B1)), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{V_{03}(M)}=(- r \operatorname{sin}\left(\theta_{2}\right) \dot{\theta}_{2} + \operatorname{cos}\left(\theta_{2}\right) \dot{r})\mathbf{\hat{b1}_{i}} + (- r \operatorname{cos}\left(\theta_{2}\right) \dot{\theta}_{2} - \operatorname{sin}\left(\theta_{2}\right) \dot{r})\mathbf{\hat{b1}_{j}} -  \dot{\theta}_{1}\mathbf{\hat{b1}_{k}} \ in \ the \ B_1 \ base$$

- Express $\overrightarrow{V_{03}(M)}$ in the $(0)$ reference frame:

```python
# V_03.to_matrix(B0)
display(Math(r'\overrightarrow{V_{03}(M)}=%s \ %s' % (latex(V_03.express(B0)), latex('in \ the \ B_0 \ base'))))
```

$$\overrightarrow{V_{03}(M)}=(\left(- r \operatorname{sin}\left(\theta_{2}\right) \dot{\theta}_{2} + \operatorname{cos}\left(\theta_{2}\right) \dot{r}\right) \operatorname{cos}\left(\theta_{1}\right) - \operatorname{sin}\left(\theta_{1}\right) \dot{\theta}_{1})\mathbf{\hat{b0}_{i}} + (- r \operatorname{cos}\left(\theta_{2}\right) \dot{\theta}_{2} - \operatorname{sin}\left(\theta_{2}\right) \dot{r})\mathbf{\hat{b0}_{j}} + (- \left(- r \operatorname{sin}\left(\theta_{2}\right) \dot{\theta}_{2} + \operatorname{cos}\left(\theta_{2}\right) \dot{r}\right) \operatorname{sin}\left(\theta_{1}\right) - \operatorname{cos}\left(\theta_{1}\right) \dot{\theta}_{1})\mathbf{\hat{b0}_{k}} \ in \ the \ B_0 \ base$$

**5. What is the torseur of $(3)$ in relation to $(0)$ ?**
- $\Gamma_{03}(M) = \Gamma_{01}(M) + \Gamma_{12}(M) + \Gamma_{23}(M)$ in the (0) reference frame:
- $\Gamma_{03}(M) = V_{03}(O) + \Omega_{03} \wedge \overrightarrow{OM}$

```python
# We use matrix calculus with the reference frame change formula:
# V_03(M) = V_03(O) + Rot_03 ^ OM

V_03_O = Matrix([diff(r(t)) * cos(theta_2), diff(r(t)) * sin(theta_2), 0])  # in the B1 base
display(Math(r'\overrightarrow{V_{03}(M)}=%s \ %s' % (latex(V_03_O), latex('in \ the \ B_1 \ base'))))

Rot_03 = Matrix([0, diff(theta_1), diff(theta_2)])  # in the B1 base
display(Math(r'\overrightarrow{\Omega_{03}(M)}=%s \ %s' % (latex(Rot_03), latex('in \ the \ B_1 \ base'))))

# print('V_03(M) = V_03(O) + Rot_03 ^ OM')
OM = Matrix([r * cos(theta_2), r * sin(theta_2), 0])
V_03_M_B1 = V_03_O + Rot_03.cross(OM)  # .cross(OM) is the same as ^OM
display(Math(r'\overrightarrow{V_{03}(M)}=%s \ %s' % (latex(V_03_M_B1), latex('in \ the \ B_1 \ base'))))
```

$$\overrightarrow{V_{03}(M)}=\left[\begin{matrix}\cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\\\sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\\0\end{matrix}\right] \ in \ the \ B_1 \ base$$

$$\overrightarrow{\Omega_{03}(M)}=\left[\begin{matrix}0\\\frac{d}{d t} \theta_{1}{\left (t \right )}\\\frac{d}{d t} \theta_{2}{\left (t \right )}\end{matrix}\right] \ in \ the \ B_1 \ base$$

$$\overrightarrow{V_{03}(M)}=\left[\begin{matrix}- r \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\\r \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\\- r \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} \theta_{1}{\left (t \right )}\end{matrix}\right] \ in \ the \ B_1 \ base$$

**6. What is the acceleration of $M$ in relation to $(0)$ ?**

```python
# print("V_03''")
# V_03.dt(B0).to_matrix(B0)
display(Math(r'\overrightarrow{A_{03}(M)}=%s \ %s' % (latex(V_03.dt(B0).to_matrix(B0)), latex('in \ the \ B_0 \ base'))))
```

$$\overrightarrow{A_{03}(M)}=\left[\begin{matrix}\left(- \left(- r \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\right) \frac{d}{d t} \theta_{1}{\left (t \right )} - \frac{d^{2}}{d t^{2}}  \theta_{1}{\left (t \right )}\right) \sin{\left (\theta_{1}{\left (t \right )} \right )} + \left(- r \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  \theta_{2}{\left (t \right )} - r \cos{\left (\theta_{2}{\left (t \right )} \right )} \left(\frac{d}{d t} \theta_{2}{\left (t \right )}\right)^{2} - \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  r{\left (t \right )} - \left(\frac{d}{d t} \theta_{1}{\left (t \right )}\right)^{2}\right) \cos{\left (\theta_{1}{\left (t \right )} \right )}\\r \sin{\left (\theta_{2}{\left (t \right )} \right )} \left(\frac{d}{d t} \theta_{2}{\left (t \right )}\right)^{2} - r \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  \theta_{2}{\left (t \right )} - \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  r{\left (t \right )} - \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )} \frac{d}{d t} \theta_{2}{\left (t \right )}\\\left(- \left(- r \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )}\right) \frac{d}{d t} \theta_{1}{\left (t \right )} - \frac{d^{2}}{d t^{2}}  \theta_{1}{\left (t \right )}\right) \cos{\left (\theta_{1}{\left (t \right )} \right )} - \left(- r \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  \theta_{2}{\left (t \right )} - r \cos{\left (\theta_{2}{\left (t \right )} \right )} \left(\frac{d}{d t} \theta_{2}{\left (t \right )}\right)^{2} - \sin{\left (\theta_{2}{\left (t \right )} \right )} \frac{d}{d t} r{\left (t \right )} \frac{d}{d t} \theta_{2}{\left (t \right )} + \cos{\left (\theta_{2}{\left (t \right )} \right )} \frac{d^{2}}{d t^{2}}  r{\left (t \right )} - \left(\frac{d}{d t} \theta_{1}{\left (t \right )}\right)^{2}\right) \sin{\left (\theta_{1}{\left (t \right )} \right )}\end{matrix}\right] \ in \ the \ B_0 \ base$$

```python
# TODO: Why don't we find the same results ??!
print("\n= = = = = = = = = = = = = = = = = = = = = = = =")
print("Solve the exercise with another method:\n")
# Another method with O and M points:
# Define the Origin Point
O = Point('O')
O.set_vel(B0, 0)  # Set velocity of O
O.set_acc(B0, 0)  # Set acceleration of O

# Define the M point from the origin
M = O.locatenew('M', r(t) * B2.x)
# Check the OM vector:
print("OM vector definition:")
M.pos_from(O)

# Set velocity of M in the B2 reference frame
print("M velocity in the B2 frame: aka V_23 in B2")
M.set_vel(B2, M.pos_from(O).dt(B2))
M.vel(B2)
print("M velocity in the B1 frame: aka V_23 in B1")
M.vel(B2).express(B1)

print("M velocity in the B1 frame: aka V_12 in B1")
M.set_vel(B1, (r * B2.x).dt(B1))  # Set velocity in the B1 reference frame
M.vel(B1).express(B1)

print("M velocity in the B0 frame: aka V_01 in B0")
M.set_vel(B0, (B1.x).dt(B0))  # Set velocity in the B1 reference frame
M.vel(B0).express(B0)
print("M velocity in the B1 frame: aka V_01 in B1")
M.vel(B0).express(B1)

print("M acceleration in the B1 frame:")
M.acc(B1).to_matrix(B1)

print("M acceleration in the B0 frame:")
trigsimp(M.acc(B0).to_matrix(B0))
```

```python
%%HTML

<footer id="attribution" style="float:right; color:#999; background:#fff;">
Created with Jupyter, Sympy, GlowScript, Numpy   -   MECA654   -   Polytech Annecy-Chambéry   -   Pierre Nagorny   -   March 2018
</footer>
```

Created with Jupyter, Sympy, GlowScript, Numpy   -   MECA654   -   Polytech Annecy-Chambéry   -   Pierre Nagorny   -   March 2018