```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir( os.path.join('..', 'notebook_format') )
from formats import load_style
load_style(css_style = 'custom2.css')
```

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2

# pip install python-dotenv
# for the dotenv package, it is
# for loading personal API key, as we'll
# see, but you don't have to use it
import json
import requests
from subprocess import call
from dotenv import load_dotenv
from IPython.display import HTML
from IPython.display import Image
from IPython.display import display
from sklearn.metrics import mean_squared_error
from sklearn.metrics import pairwise_distances

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn,requests
```

    Ethen 2017-03-23 22:10:20

    CPython 3.5.2
    IPython 5.3.0

    numpy 1.12.1
    pandas 0.19.2
    matplotlib 2.0.0
    sklearn 0.18
    requests 2.13.0

# Alternating Least Squares with Weighted Regularization

Recommedantion system is a popular topic in recent years, what is does (or its goal) is to seek to predict the "rating" or "preference" that a user would give to an item. Given the rating, the business will then recommend preferable new items to the user for further purchases. The type of recommendation system that we will be discussing is known as **collaborative filtering**, where the features of the user (e.g. age, gender) or the item (e.g. perishable or not) itself do not play any role in the algorithm. Instead, we rely soley on the ratings that a user gave to the item.

We start by loading some sample data to make this a bit more concrete. For this introduction, we'll be using the [MovieLens dataset](http://grouplens.org/datasets/movielens/). The website has datasets of various sizes, but we just start with the smallest one [MovieLens 100K Dataset](http://files.grouplens.org/datasets/movielens/ml-100k.zip). This dataset consists of 100,000 movie ratings by users (on a 1-5 scale). The main data set `u.data` file consists of four columns (tab-delimited), including user-id (starting at 1), item-id (starting at 1), rating, and timestamp (we won't be using this field).

```python
# download the dataset if it isn't in the same folder
file_dir = 'ml-100k'
file_path = os.path.join(file_dir, 'u.data')
if not os.path.isdir(file_dir):
    call(['curl', '-O', 'http://files.grouplens.org/datasets/movielens/ml-100k.zip'])
    call(['unzip', 'ml-100k.zip'])

names = ['user_id', 'item_id', 'rating', 'timestamp']
df = pd.read_csv(file_path, sep = '\t', names = names)
print(df.shape)
df.head()
```

    (100000, 4)

      user_id
      item_id
      rating
      timestamp

      0
      196
      242
      3
      881250949

      1
      186
      302
      3
      891717742

      2
      22
      377
      1
      878887116

      3
      244
      51
      2
      880606923

      4
      166
      346
      1
      886397596

As we can see, the only data that we have is how each user interacted or rated each item. Given this information, collaborative filtering will start by constructing a user-item matrix with each distinct user being the row, item being the column and the value for each cell will simply be the rating that the user gave to the item. Apart from building the matrix, we will also print out some other information to help us understand our data a bit better.

```python
# create the rating matrix r_{ui}, remember to
# subract the user and item id by 1 since
# the indices starts from 0
n_users = df['user_id'].unique().shape[0]
n_items = df['item_id'].unique().shape[0]
ratings = np.zeros((n_users, n_items))
for row in df.itertuples(index = False):
    ratings[row.user_id - 1, row.item_id - 1] = row.rating

# compute the non-zero elements in the rating matrix
matrix_size = np.prod(ratings.shape)
interaction = np.flatnonzero(ratings).shape[0]
sparsity = 100 * (interaction / matrix_size)

print('dimension: ', ratings.shape)
print( 'sparsity: {:.1f}%'.format(sparsity) )
ratings
```

    dimension:  (943, 1682)
    sparsity: 6.3%

    array([[ 5.,  3.,  4., ...,  0.,  0.,  0.],
           [ 4.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 5.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  5.,  0., ...,  0.,  0.,  0.]])

```python
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

plt.hist( np.sum(ratings != 0, axis = 1), histtype = 'stepfilled', bins = 30,
          alpha = 0.85, label = '# of ratings', color = '#7A68A6', normed = True )
plt.axvline(x = 10, color = 'black', linestyle = '--')
plt.legend(loc = "upper right")
plt.show()
```

From the information above, we know there are 943 unique users, 1682 unique items. Within the rating matrix, only 6.3% of the cells have a value, although we filled in empty ratings as 0, we should not assume these values to truly be zero. More appropriately, they are just missing entries. This kind of sparsity is extremely common in recommendation system, where people seldom give ratings to things that they have purchased. One thing to note is that if the sparsity of the matrix is below 1% (rule of thumb), then the dataset might be too sparse to perform any sort of modeling.

Next the histogram tells us that every user has given at least more than 10 ratings, we will use this to perform the train/test split of the data for testing the algorithm that we'll build later.

One tricky thing about splitting the data into training and testing is that: In supervise machine learning we normally build the trainining and testing holdout set by randomly splitting the rows. In those cases, this idea works, because we have a model with features/target that we are trying to fit a function to. For recommender systems with collaborative filtering (no features), this just won't work anymore, because all of the items/users need to be available when the model is first built. So what we do instead is mask a random sample of the user/item ratings to validate and compare how well the recommender system did in predicting the ratings of those masked values. In our case, given we already know each user has given more than 10 ratings, what we'll do is for every user, we remove 10 of the item ratings and and assign them to the test set. The testing matrix is printed below, as hopefully, you can see that some of the values are indeed different from the original rating matrix.

```python
def create_train_test(ratings):
    """
    split into training and test sets,
    remove 10 ratings from each user
    and assign them to the test set
    """
    test = np.zeros(ratings.shape)
    train = ratings.copy()
    for user in range(ratings.shape[0]):
        test_index = np.random.choice( np.flatnonzero(ratings[user]),
                                       size = 10, replace = False )
        train[user, test_index] = 0.0
        test[user, test_index] = ratings[user, test_index]

    # assert that training and testing set are truly disjoint
    assert np.all(train * test == 0)
    return train, test

train, test = create_train_test(ratings)
del ratings
train
```

    array([[ 5.,  3.,  4., ...,  0.,  0.,  0.],
           [ 4.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 5.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  5.,  0., ...,  0.,  0.,  0.]])

## Matrix Factorization

Now that the data preprocessing part has been taken care of, let's get to the more exciting part, the algorithm. The algorithm that we'll introduce today is Matrix Factorization.

Recall that we had a user-item matrix, $R$ where nonzero elements of the matrix are ratings that a user has given an item. What Matrix Factorization does is it decomposes a large matrix into products of matrices, namely, $R = U \times V$. See the picture below taken from a quick [google search](https://www.google.com/search?q=matrix+factorization&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiD8Ku02_vQAhXGsVQKHdpSAW4Q_AUICCgB&biw=1433&bih=732#imgrc=aFtUeC0wtJCkgM%3A) for a better understanding:

Matrix factorization assumes that:

- Each user can be described by $d$ features. For example, feature 1 might be a referring to how much each user likes disney movies.
- Each item, movie in this case, can be described by an analagous set of $d$ features. To correspond to the above example, feature 1 for the movie might be a number that says how close the movie is to a disney movie.

After learning the two smaller matrices, all we have to do is to perform a matrix multiplication of the two matrices and the end result will be a our approximation for the rating the user would give that item (movie).

The cool thing about this is that, we do not know what these features are nor do we have to determine them beforehand, which is why these features are often refer to as latent features. Next, we also don't know how many latent features are optimal for the task at hand. Thus, we simply have to use random search or grid search or other fancy techniques to perform hyperparameter tuning and determine the best number for $d$.

Given all those information, the next question is: how do we learn the user matrix, $U$, and item matrix, $V$? Well, like a lot of machine learning algorithm, by minimizing a loss function.

We start by denoting our $d$ feature user into math by letting a user $u$ take the form of a $1 \times d$-dimensional vector $\textbf{x}_{u}$. These for often times referred to as latent vectors or low-dimensional embeddings. Similarly, an item *i* can be represented by a $1 \times d$-dimensional vector $\textbf{y}_{i}$. And the rating that we predict user $u$ will give for item $i$ is just the dot product of the two vectors

\begin{align}
\hat r_{ui} &= \textbf{x}_{u} \textbf{y}_{i}^{T} = \sum\limits_{d} x_{ud}y_{di}
\end{align}

Where $\hat r_{ui}$ represents our prediction for the true rating $r_{ui}$. Next, we will choose a objective function to minimize the square of the difference between all ratings in our dataset ($S$) and our predictions. This produces a objective function of the form:

\begin{align}
L &= \sum\limits_{u,i \in S}( r_{ui} - \textbf{x}_{u} \textbf{y}_{i}^{T} )^{2} + \lambda \big( \sum\limits_{u} \left\Vert \textbf{x}_{u} \right\Vert^{2} + \sum\limits_{i} \left\Vert \textbf{y}_{i} \right\Vert^{2} \big)
\end{align}

Note that we've added on two $L_{2}$ regularization terms, with $\lambda$ controlling the strength at the end to prevent overfitting of the user and item vectors. $\lambda$, is another hyperparameter that we'll have to search for to determine the best value. Regularization can a can topic of itself, and if you're confused by this, consider checking out [this documentation](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/regularization/regularization.ipynb) that covers it a bit more.

Now that we formalize our objective function, we'll introduce the **Alternating Least Squares with Weighted Regularization (ALS-WR)** method for optimizing it. The way it works is we start by treating one set of latent vectors as constant. For this example, we'll pick the item vectors, $\textbf{y}_{i}$. We then take the derivative of the loss function with respect to the other set of vectors, the user vectors, $\textbf{x}_{u}$ and solve for the non-constant vectors (the user vectors).

\begin{align}
\frac{\partial L}{\partial \textbf{x}_{u}}
&\implies - 2 \sum\limits_{i}(r_{ui} - \textbf{x}_{u} \textbf{y}_{i}^{T} ) \textbf{y}_{i} + 2 \lambda \textbf{x}_{u} = 0 \\
&\implies -(\textbf{r}_{u} - \textbf{x}_{u} Y^{T} )Y + \lambda \textbf{x}_{u} = 0 \\
&\implies \textbf{x}_{u} (Y^{T}Y + \lambda I) = \textbf{r}_{u}Y \\
&\implies \textbf{x}_{u} = \textbf{r}_{u}Y (Y^{T}Y + \lambda I)^{-1}
\end{align}

To clarify it a bit, let us assume that we have $m$ users and $n$ items, so our ratings matrix is $m \times n$.

- The row vector $\textbf{r}_{u}$ just represents users *u*'s row from the ratings matrix with all the ratings for all the items (so it has dimension $1 \times n$)
-  We introduce the symbol $Y$, with dimensions $n \times d$, to represent all item row vectors vertically stacked on each other
- Lastly, $I$ is just the identity matrix which has dimension $d \times d$ to ensure the matrix multiplication's dimensionality will be correct when we add the $\lambda$

Now comes the alternating part: With these newly updated user vectors in hand, in the next round, we hold them as constant, and take the derivative of the loss function with respect to the previously constant vectors (the item vectors). As the derivation for the item vectors is quite similar, we will simply list out the end formula:

\begin{align}
\frac{\partial L}{\partial \textbf{y}_{i}}
&\implies \textbf{y}_{i} = \textbf{r}_{i}X (X^{T}X + \lambda I)^{-1}
\end{align}

Then we alternate back and forth and carry out this two-step process until convergence. The reason we alternate is, optimizing user latent vectors, $U$, and item latent vectors $V$ simultaneously is hard to solve. If we fix $U$ or $V$ and tackle one problem at a time, we potentially turn it into a easier sub-problem. Just to summarize it, ALS works by:

1. Initialize the user latent vectors, $U$, and item latent vectors $V$ with randomly
2. Fix $U$ and solve for $V$
3. Fix $V$ and solve for $U$
4. Repeat step 2 and 3 until convergence

Now that we have our equations, let's program this thing up! We'll set the model to use 20 factors and a regularization value of 0.01 (chosen at random) and train it for 100 iterations and compute the mean square error on the train and test set to assess algorithm convergence.

```python
class ExplicitMF:
    """
    Train a matrix factorization model using Alternating Least Squares
    to predict empty entries in a matrix

    Parameters
    ----------
    n_iters : int
        number of iterations to train the algorithm

    n_factors : int
        number of latent factors to use in matrix
        factorization model, some machine-learning libraries
        denote this as rank

    reg : float
        regularization term for item/user latent factors,
        since lambda is a keyword in python we use reg instead
    """
    def __init__(self, n_iters, n_factors, reg):
        self.reg = reg
        self.n_iters = n_iters
        self.n_factors = n_factors

    def fit(self, train, test):
        """
        pass in training and testing at the same time to record
        model convergence, assuming both dataset is in the form
        of User x Item matrix with cells as ratings
        """
        self.n_user, self.n_item = train.shape
        self.user_factors = np.random.random((self.n_user, self.n_factors))
        self.item_factors = np.random.random((self.n_item, self.n_factors))

        # record the training and testing mse for every iteration
        # to show convergence later (usually, not worth it for production)
        self.test_mse_record  = []
        self.train_mse_record = []
        for _ in range(self.n_iters):
            self.user_factors = self._als_step(train, self.user_factors, self.item_factors)
            self.item_factors = self._als_step(train.T, self.item_factors, self.user_factors)
            predictions = self.predict()
            test_mse  = self.compute_mse(test, predictions)
            train_mse = self.compute_mse(train, predictions)
            self.test_mse_record.append(test_mse)
            self.train_mse_record.append(train_mse)

        return self

    def _als_step(self, ratings, solve_vecs, fixed_vecs):
        """
        when updating the user matrix,
        the item matrix is the fixed vector and vice versa
        """
        A = fixed_vecs.T.dot(fixed_vecs) + np.eye(self.n_factors) * self.reg
        b = ratings.dot(fixed_vecs)
        A_inv = np.linalg.inv(A)
        solve_vecs = b.dot(A_inv)
        return solve_vecs

    def predict(self):
        """predict ratings for every user and item"""
        pred = self.user_factors.dot(self.item_factors.T)
        return pred

    @staticmethod
    def compute_mse(y_true, y_pred):
        """ignore zero terms prior to comparing the mse"""
        mask = np.nonzero(y_true)
        mse = mean_squared_error(y_true[mask], y_pred[mask])
        return mse
```

```python
def plot_learning_curve(model):
    """visualize the training/testing loss"""
    linewidth = 3
    plt.plot(model.test_mse_record, label = 'Test', linewidth = linewidth)
    plt.plot(model.train_mse_record, label = 'Train', linewidth = linewidth)
    plt.xlabel('iterations')
    plt.ylabel('MSE')
    plt.legend(loc = 'best')
```

```python
als = ExplicitMF(n_iters = 100, n_factors = 40, reg = 0.01)
als.fit(train, test)
plot_learning_curve(als)
```

From the result, we can see ALS converges after a few sweeps, which is one of the main reason for its popularity. Fast, thus scalable to bigger dataset.

## Validating the Recommendation

After training our model, we'll actually look at the recommended result to see if our recommendation algorithm is any good or not. The way we do this is: The MovieLens dataset contains a file called `u.item` with information about each movie. To put the cherry on top, it turns out that there is a website called [themoviedb.org](http://www.themoviedb.org) which has a free API. If we have the IMDB "movie id" for a movie, then we can use this API to return the its poster. Note that you will have to sign up and request for an API key if you wish to reproduce the example below.

```python
# Load in movies' information, stored in the u.item (pipe delimited)
# file, the first element is the id, and the fourth element is the
# movie's IMDB url
idx_to_movie = {}
item_path = os.path.join(file_dir, 'u.item')

with open(item_path, encoding = 'latin1') as f:
    for line in f.readlines():
        info = line.split('|')
        idx = int(info[0]) - 1
        movie = info[4]
        idx_to_movie[idx] = movie

idx_to_movie[0]
```

    'http://us.imdb.com/M/title-exact?Toy%20Story%20(1995)'

For example, if we paste the sample url above into our browser, then our url will get redirected. The resulting url contains the IMDB movie ID as the last information in the url starting with "tt". For example, the redirected url for Toy Story is http://www.imdb.com/title/tt0114709/, and the corresponding IMDB movie ID is therefore tt0114709.

```python
# stripping the movie id with requests
response = requests.get(idx_to_movie[0])
movie_id = response.url.split('/')[-2]
print(movie_id)
```

    tt0114709

From the API documentation:

To build an image URL, we will need 3 pieces of data. The base_url, size and file_path. Simply combine them all and you will have a fully qualified URL. Here’s an example URL:
https://image.tmdb.org/t/p/w500/8uO0gUM8aNqYLs1OsTBQiXu0fEv.jpg

So to get the base_url, we send a request to the https://api.themoviedb.org/3/configuration using our API key as the parameter, after that we load the information into a dictionary and access the `base_url` key under the `images` key (the base_url is nested under images), and we'll also append the poster size at the end of the base_url.

```python
# here I'm simply loading my API key
# from a .env file, and using dotenv package
# to obtain the value of the variable, you may
# simply change the api_key variable to your key
load_dotenv('.env')
api_key = os.environ.get('API_KEY')

# request the base_url
params = {'api_key': api_key}
response = requests.get('http://api.themoviedb.org/3/configuration', params = params)

# for the poster there are different sizes
# 'w92', 'w154', 'w185', 'w342', 'w500', 'w780', 'original',
# simply change it to the one you prefer
base_url = json.loads(response.text)['images']['base_url'] + 'w185'
base_url
```

    'http://image.tmdb.org/t/p/w185'

Next to get the file_path to the poster image, the have to send another request to http://api.themoviedb.org/3/movie/{}/images, where we have to fill in the movie_id for {}. Then the actual path will be stored under the `file_path` key under the `posters` key (accessing the posters key will give you a list of path, we'll simply use the first one).

```python
image_url = 'http://api.themoviedb.org/3/movie/{}/images'.format(movie_id)
response  = requests.get(image_url, params = params)
file_path = json.loads(response.text)['posters'][0]['file_path']
file_path
```

    '/uMZqKhT4YA6mqo2yczoznv7IDmv.jpg'

After that we can simply concatenate the base_url and the file path and display the poster image.

```python
full_path = base_url + file_path
Image(full_path)
```

Unfortunately, there are also other edge-cases, where we have to search for the title and grab the poster image there. e.g.

```python
movie_url = 'http://us.imdb.com/M/title-exact?Fox%20and%20the%20Hound,%20The%20(1981)'
response  = requests.get(movie_url)

# see how the url is different this time,
# we have to strip out the title between the = sign
# and pass it to the movie search API using the title
# as a query parameter
print(response.url)
query = response.url.split('=')[-2]
params['query'] = query
response = requests.get('http://api.themoviedb.org/3/search/movie', params = params)
json.loads(response.text)
file_path = json.loads(response.text)['results'][0]['poster_path']
full_path = base_url + file_path
Image(full_path)
```

    http://www.imdb.com/find?q=Fox%20and%20the%20Hound,%20The&s=tt

In the following section, we'll train the model again, use [themoviedb.org](http://www.themoviedb.org)'s API to grab the movie posters and visualize the top 5 most similar movies (measured by cosine distance) for some input movie. These similar movies will serve as our recommendation for a given input movie.

Side note, the images are displayed using HTML so multiple images can be displayed in one line, idea taken from this [Stackoverflow question](http://stackoverflow.com/questions/19471814/display-multiple-images-in-one-ipython-notebook-cell/27795087#27795087).

```python
class Recommendation:
    """
    Train the ALS model to be able to
    recommend the top-k similar movies for a given query

    Parameters
    ----------
    k: int
        number of top similar movies to recommend for
        a given query

    model: ExplicitMF
        the recommendation algorithm

    idx_to_movie: dictionary
        a dictionary mapping movie id to their imdb url
    """
    def __init__(self, model, k, idx_to_movie):
        self.k = k
        self.model = model
        self.idx_to_movie = idx_to_movie

    def fit(self, train, test):
        # train the model and compute the pairwise cosine distance
        self.model.fit(train, test)
        self.sim_mat = pairwise_distances(self.model.item_factors, metric = 'cosine')

        # also get the base url for the API
        # assuming API key is stored in .env file
        load_dotenv('.env')
        api_key = os.environ.get('API_KEY')
        self.params = {'api_key': api_key}
        response = requests.get('http://api.themoviedb.org/3/configuration',
                                params = self.params)
        self.base_url = json.loads(response.text)['images']['base_url'] + 'w185'
        return self

    def recommend(self, query):
        movie_url = idx_to_movie[query]
        input_poster = self._get_poster(movie_url)
        input_image = ''.format(input_poster)
        recommended_images = self._get_top_k_recommendation(query)

        # display query movie and for that movie top-k recommendations
        display( HTML('<font size=5> Input </font>') )
        input_image = ''.format(input_poster)
        display( HTML(input_image) )
        display( HTML('<font size=5> Recommendation </font>') )
        display( HTML(recommended_images) )

    def _get_poster(self, movie_url):
        """get the poster image for a given movie url"""
        try:
            response  = requests.get(movie_url)
            movie_id  = response.url.split('/')[-2]
            image_url = 'http://api.themoviedb.org/3/movie/{}/images'.format(movie_id)
            response  = requests.get(image_url, params = self.params)
            file_path = json.loads(response.text)['posters'][0]['file_path']
        except KeyError:
            try:
                response = requests.get(movie_url)
                query = response.url.split('=')[-2]
                self.params['query'] = query
                response = requests.get('http://api.themoviedb.org/3/search/movie',
                                        params = self.params)
                json.loads(response.text)
                file_path = json.loads(response.text)['results'][0]['poster_path']
                self.params.pop('query')
            except KeyError:
                # sometimes the url just doesn't work
                # return an empty string to not mess up the display
                file_path = ''

        full_path = self.base_url + file_path
        return full_path

    def _get_top_k_recommendation(self, query):
        """
        sort the similarity vector to get the most similar,
        remember the smaller the cosine, the more similar they are
        but the query item itself will always be the most similar,
        so exclude that from the output
        """
        sim = self.sim_mat[query]
        top_k_item_idx = np.argsort(sim)[1:self.k + 1]
        recommended_images = ''
        for item_idx in top_k_item_idx:
            movie_url = self.idx_to_movie[item_idx]
            recommended_poster = self._get_poster(movie_url)
            recommended_images += """

                                  """.format(recommended_poster)
        return recommended_images
```

```python
als = ExplicitMF(n_iters = 100, n_factors = 40, reg = 0.01)
recsys = Recommendation(model = als, k = 5, idx_to_movie = idx_to_movie)
recsys = recsys.fit(train, test)
```

```python
query = 1
recsys.recommend(query)
```

 Input

 Recommendation

```python
query = 500
recsys.recommend(query)
```

 Input

 Recommendation

I'll leave it up to you to determine whether these recommendations are good or not (results will slightly differ across different runs). In practice A/B testing is most likely a better way to judge whether these recommendations are useful, but that's a topic for some other day.

# Reference

- [Blog: Intro to Recommender Systems: Collaborative Filtering](http://blog.ethanrosenthal.com/2015/11/02/intro-to-collaborative-filtering/)
- [Blog: Explicit Matrix Factorization: ALS, SGD, and All That Jazz](http://blog.ethanrosenthal.com/2016/01/09/explicit-matrix-factorization-sgd-als/)
- [Quora: What is the Alternating Least Squares method in recommendation systems?](https://www.quora.com/What-is-the-Alternating-Least-Squares-method-in-recommendation-systems)


```python
pip install tensorflow==2.4.1 tensorflow-quantum
```

    Collecting tensorflow==2.4.1
      Downloading tensorflow-2.4.1-cp37-cp37m-manylinux2010_x86_64.whl (394.3 MB)
         |████████████████████████████████| 394.3 MB 13 kB/s
    Collecting tensorflow-quantum
      Downloading tensorflow_quantum-0.5.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (7.8 MB)
         |████████████████████████████████| 7.8 MB 22.7 MB/s
    Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (1.1.0)
    Collecting wrapt~=1.12.1
      Downloading wrapt-1.12.1.tar.gz (27 kB)
    Collecting grpcio~=1.32.0
      Downloading grpcio-1.32.0-cp37-cp37m-manylinux2014_x86_64.whl (3.8 MB)
         |████████████████████████████████| 3.8 MB 41.1 MB/s
    Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (3.3.0)
    Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (0.2.0)
    Collecting flatbuffers~=1.12.0
      Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
    Requirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (1.19.5)
    Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (1.15.0)
    Requirement already satisfied: tensorboard~=2.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (2.7.0)
    Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (1.1.2)
    Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (0.12.0)
    Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (1.6.3)
    Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (3.17.3)
    Collecting tensorflow-estimator<2.5.0,>=2.4.0
      Downloading tensorflow_estimator-2.4.0-py2.py3-none-any.whl (462 kB)
         |████████████████████████████████| 462 kB 77.7 MB/s
    Collecting typing-extensions~=3.7.4
      Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
    Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.4.1) (0.37.1)
    Collecting h5py~=2.10.0
      Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
         |████████████████████████████████| 2.9 MB 38.3 MB/s
    Collecting gast==0.3.3
      Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (57.4.0)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (2.23.0)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (1.8.1)
    Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (0.6.1)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (1.0.1)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (0.4.6)
    Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (1.35.0)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow==2.4.1) (3.3.6)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.4->tensorflow==2.4.1) (4.2.4)
    Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.4->tensorflow==2.4.1) (4.8)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.4->tensorflow==2.4.1) (0.2.8)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow==2.4.1) (1.3.0)
    Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.4->tensorflow==2.4.1) (4.10.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.4->tensorflow==2.4.1) (3.7.0)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.4->tensorflow==2.4.1) (0.4.8)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow==2.4.1) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow==2.4.1) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow==2.4.1) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow==2.4.1) (3.0.4)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow==2.4.1) (3.1.1)
    Collecting sympy==1.5
      Downloading sympy-1.5-py2.py3-none-any.whl (5.6 MB)
         |████████████████████████████████| 5.6 MB 27.2 MB/s
    Collecting google-api-core==1.21.0
      Downloading google_api_core-1.21.0-py2.py3-none-any.whl (90 kB)
         |████████████████████████████████| 90 kB 10.6 MB/s
    Collecting protobuf>=3.9.2
      Downloading protobuf-3.13.0-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB)
         |████████████████████████████████| 1.3 MB 38.5 MB/s
    Collecting googleapis-common-protos==1.52.0
      Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
         |████████████████████████████████| 100 kB 9.3 MB/s
    Collecting cirq==0.11.0
      Downloading cirq-0.11.0-py3-none-any.whl (7.6 kB)
    Collecting google-auth<3,>=1.6.3
      Downloading google_auth-1.18.0-py2.py3-none-any.whl (90 kB)
         |████████████████████████████████| 90 kB 9.8 MB/s
    Collecting cirq-google==0.11.0
      Downloading cirq_google-0.11.0-py3-none-any.whl (380 kB)
         |████████████████████████████████| 380 kB 64.0 MB/s
    Collecting cirq-core==0.11.0
      Downloading cirq_core-0.11.0-py3-none-any.whl (1.5 MB)
         |████████████████████████████████| 1.5 MB 49.6 MB/s
    Requirement already satisfied: sortedcontainers~=2.0 in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (2.4.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (1.4.1)
    Requirement already satisfied: networkx~=2.4 in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (2.6.3)
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (4.62.3)
    Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (3.2.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (1.1.5)
    Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from cirq-google==0.11.0->cirq==0.11.0->tensorflow-quantum) (1.26.3)
    Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core==1.21.0->tensorflow-quantum) (2018.9)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy==1.5->tensorflow-quantum) (1.2.1)
    Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
      Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.6 MB/s
      Downloading google_api_core-1.31.4-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.6 MB/s
      Downloading google_api_core-1.31.3-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.4 MB/s
      Downloading google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.5 MB/s
      Downloading google_api_core-1.31.1-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.2 MB/s
      Downloading google_api_core-1.31.0-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.2 MB/s
      Downloading google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.3 MB/s
      Downloading google_api_core-1.29.0-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.3 MB/s
      Downloading google_api_core-1.28.0-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 1.2 MB/s
      Downloading google_api_core-1.27.0-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.2 MB/s
      Downloading google_api_core-1.26.2-py2.py3-none-any.whl (93 kB)
         |████████████████████████████████| 93 kB 1.2 MB/s
      Downloading google_api_core-1.26.1-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 1.2 MB/s
      Downloading google_api_core-1.26.0-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 1.1 MB/s
      Downloading google_api_core-1.25.1-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 221 kB/s
      Downloading google_api_core-1.25.0-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 164 kB/s
      Downloading google_api_core-1.24.1-py2.py3-none-any.whl (92 kB)
         |████████████████████████████████| 92 kB 11.0 MB/s
      Downloading google_api_core-1.24.0-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 9.7 MB/s
      Downloading google_api_core-1.23.0-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 10.6 MB/s
      Downloading google_api_core-1.22.4-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 9.3 MB/s
      Downloading google_api_core-1.22.3-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 9.1 MB/s
      Downloading google_api_core-1.22.2-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 10.1 MB/s
      Downloading google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 10.1 MB/s
      Downloading google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
         |████████████████████████████████| 91 kB 8.7 MB/s
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (2.8.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (3.0.6)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (0.11.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq-core==0.11.0->cirq==0.11.0->tensorflow-quantum) (1.3.2)
    Building wheels for collected packages: wrapt
      Building wheel for wrapt (setup.py) ... done
      Created wheel for wrapt: filename=wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl size=68716 sha256=b11797ca1ee6ac2c6d9cbf1cc04efe0c4d8603599985496242fe1a5398b359ef
      Stored in directory: /root/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6
    Successfully built wrapt
    Installing collected packages: protobuf, googleapis-common-protos, google-auth, typing-extensions, sympy, grpcio, google-api-core, cirq-core, cirq-google, wrapt, tensorflow-estimator, h5py, gast, flatbuffers, cirq, tensorflow-quantum, tensorflow
      Attempting uninstall: protobuf
        Found existing installation: protobuf 3.17.3
        Uninstalling protobuf-3.17.3:
          Successfully uninstalled protobuf-3.17.3
      Attempting uninstall: googleapis-common-protos
        Found existing installation: googleapis-common-protos 1.54.0
        Uninstalling googleapis-common-protos-1.54.0:
          Successfully uninstalled googleapis-common-protos-1.54.0
      Attempting uninstall: google-auth
        Found existing installation: google-auth 1.35.0
        Uninstalling google-auth-1.35.0:
          Successfully uninstalled google-auth-1.35.0
      Attempting uninstall: typing-extensions
        Found existing installation: typing-extensions 3.10.0.2
        Uninstalling typing-extensions-3.10.0.2:
          Successfully uninstalled typing-extensions-3.10.0.2
      Attempting uninstall: sympy
        Found existing installation: sympy 1.7.1
        Uninstalling sympy-1.7.1:
          Successfully uninstalled sympy-1.7.1
      Attempting uninstall: grpcio
        Found existing installation: grpcio 1.43.0
        Uninstalling grpcio-1.43.0:
          Successfully uninstalled grpcio-1.43.0
      Attempting uninstall: google-api-core
        Found existing installation: google-api-core 1.26.3
        Uninstalling google-api-core-1.26.3:
          Successfully uninstalled google-api-core-1.26.3
      Attempting uninstall: wrapt
        Found existing installation: wrapt 1.13.3
        Uninstalling wrapt-1.13.3:
          Successfully uninstalled wrapt-1.13.3
      Attempting uninstall: tensorflow-estimator
        Found existing installation: tensorflow-estimator 2.7.0
        Uninstalling tensorflow-estimator-2.7.0:
          Successfully uninstalled tensorflow-estimator-2.7.0
      Attempting uninstall: h5py
        Found existing installation: h5py 3.1.0
        Uninstalling h5py-3.1.0:
          Successfully uninstalled h5py-3.1.0
      Attempting uninstall: gast
        Found existing installation: gast 0.4.0
        Uninstalling gast-0.4.0:
          Successfully uninstalled gast-0.4.0
      Attempting uninstall: flatbuffers
        Found existing installation: flatbuffers 2.0
        Uninstalling flatbuffers-2.0:
          Successfully uninstalled flatbuffers-2.0
      Attempting uninstall: tensorflow
        Found existing installation: tensorflow 2.7.0
        Uninstalling tensorflow-2.7.0:
          Successfully uninstalled tensorflow-2.7.0
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    pydata-google-auth 1.3.0 requires google-auth<3.0dev,>=1.25.0; python_version >= "3.6", but you have google-auth 1.18.0 which is incompatible.
    Successfully installed cirq-0.11.0 cirq-core-0.11.0 cirq-google-0.11.0 flatbuffers-1.12 gast-0.3.3 google-api-core-1.21.0 google-auth-1.18.0 googleapis-common-protos-1.52.0 grpcio-1.32.0 h5py-2.10.0 protobuf-3.13.0 sympy-1.5 tensorflow-2.4.1 tensorflow-estimator-2.4.0 tensorflow-quantum-0.5.1 typing-extensions-3.7.4.3 wrapt-1.12.1

```python
# Update package resources to account for version changes.
import importlib, pkg_resources
importlib.reload(pkg_resources)
```

    <module 'pkg_resources' from '/usr/local/lib/python3.7/dist-packages/pkg_resources/__init__.py'>

```python
#Mounting a google drive
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

```python
#This code part depends where your eecs598 file
import sys
sys.path.append('/content/drive/My Drive')
```

```python
import eecs598
import torch
import torchvision
import matplotlib.pyplot as plt
import statistics
import numpy as np
```

```python
import tensorflow as tf
import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np
import seaborn as sns
import collections

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

```python

# Control qrid size for visualization
plt.rcParams['figure.figsize'] = (10.0, 8.0)
plt.rcParams['font.size'] = 16
```

```python
#Data preparation
x_train, y_train, x_test, y_test = eecs598.data.cifar10()

print('Training set:', )
print('  data shape:', x_train.shape)
print('  labels shape: ', y_train.shape)
print('Test set:')
print('  data shape: ', x_test.shape)
print('  labels shape', y_test.shape)
```

    Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./cifar-10-python.tar.gz

      0%|          | 0/170498071 [00:00<?, ?it/s]

    Extracting ./cifar-10-python.tar.gz to .
    Training set:
      data shape: torch.Size([50000, 3, 32, 32])
      labels shape:  torch.Size([50000])
    Test set:
      data shape:  torch.Size([10000, 3, 32, 32])
      labels shape torch.Size([10000])

```python
#Visualize the dataset
import random
from torchvision.utils import make_grid

classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
samples_per_class = 12
samples = []
for y, cls in enumerate(classes):
    plt.text(-4, 34 * y + 18, cls, ha='right')
    idxs, = (y_train == y).nonzero(as_tuple=True)
    for i in range(samples_per_class):
        idx = idxs[random.randrange(idxs.shape[0])].item()
        samples.append(x_train[idx])
img = torchvision.utils.make_grid(samples, nrow=samples_per_class)
plt.imshow(eecs598.tensor_to_image(img))
plt.axis('off')
plt.show()
#The label data is just a list of 10,000 numbers ranging from 0 to 9, which corresponds to each of the 10 classes in CIFAR-10.

#airplane : 0 automobile : 1 bird : 2 cat : 3 deer : 4 dog : 5 frog : 6 horse : 7 ship : 8 truck : 9
```

```python
from tensorflow.keras.datasets import cifar10
(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()
```

    Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    170500096/170498071 [==============================] - 11s 0us/step

```python
y_train=train_labels.flatten()
y_test=test_labels.flatten()
# We have reduced the dimension
```

```python
x_train=tf.image.rgb_to_grayscale(train_images)
x_test=tf.image.rgb_to_grayscale(test_images)
#convert grayscale
```

```python
#Filter the dataset to keep just the cat  and frog, remove the other classes
def filter_36(x, y):
    keep = (y == 3) | (y == 6)
    x, y = x[keep], y[keep]
    y = y == 3
    return x,y
```

```python
x_train, y_train = filter_36(x_train, y_train)
x_test, y_test = filter_36(x_test, y_test)

print("Number of filtered training examples:", len(x_train))
print("Number of filtered test examples:", len(x_test))
```

    Number of filtered training examples: 10000
    Number of filtered test examples: 2000

```python
plt.imshow(x_train[0, :, :, 0])
plt.colorbar()
```

```python
 #Resize the image 36x36 to down to 4x4
x_train_s = tf.image.resize(x_train, (4,4)).numpy()
x_test_s = tf.image.resize(x_test, (4,4)).numpy()
```

```python
#Encode the data as quantum circuits¶
#The first step is to convert to a binary encoding.
THRESHOLD = 0.5

x_train_bin = np.array(x_train_s > THRESHOLD, dtype=np.float32)
x_test_bin = np.array(x_test_s > THRESHOLD, dtype=np.float32)
#The qubits at pixel indices with values that exceed a threshold, are rotated through an gate.
```

```python
def convert_to_circuit(image):
    """Encode truncated classical image into quantum datapoint."""
    values = np.ndarray.flatten(image)
    qubits = cirq.GridQubit.rect(4, 4)
    circuit = cirq.Circuit()
    for i, value in enumerate(values):
        if value:

            circuit.append(cirq.X(qubits[i]))

    return circuit

x_train_circ = [convert_to_circuit(x) for x in x_train_bin]
x_test_circ = [convert_to_circuit(x) for x in x_test_bin]

```

```python
SVGCircuit(x_train_circ[0])

```

```python
#Convert  Cirq circuits to tensors for tfq
x_train_tfcirc = tfq.convert_to_tensor(x_train_circ)
x_test_tfcirc = tfq.convert_to_tensor(x_test_circ)
```

```python
#Quantum neural network¶
#add a layer of these gates to a circuit
class CircuitLayerBuilder():
    def __init__(self, data_qubits, readout):
        self.data_qubits = data_qubits
        self.readout = readout

    def add_layer(self, circuit, gate, prefix):
        for i, qubit in enumerate(self.data_qubits):
            symbol = sympy.Symbol(prefix + '-' + str(i))
            circuit.append(gate(qubit, self.readout)**symbol)

```

```python
demo_builder = CircuitLayerBuilder(data_qubits = cirq.GridQubit.rect(4,1),
                                   readout=cirq.GridQubit(-1,-1))

circuit = cirq.Circuit()
demo_builder.add_layer(circuit, gate = cirq.XX, prefix='xx')
SVGCircuit(circuit)
```

```python
def create_quantum_model():
    """Create a QNN model circuit and readout operation to go along with it."""
    data_qubits = cirq.GridQubit.rect(4, 4)  # a 4x4 grid.
    readout = cirq.GridQubit(-1, -1)         # a single qubit at [-1,-1]
    circuit = cirq.Circuit()

    # Prepare the readout qubit.
    circuit.append(cirq.X(readout))
    circuit.append(cirq.H(readout))

    builder = CircuitLayerBuilder(
        data_qubits = data_qubits,
        readout=readout)

    # Then add layers (experiment by adding more).
    builder.add_layer(circuit, cirq.XX, "xx1")
    builder.add_layer(circuit, cirq.ZZ, "zz1")

    # Finally, prepare the readout qubit.
    circuit.append(cirq.H(readout))

    return circuit, cirq.Z(readout)
```

```python
model_circuit, model_readout = create_quantum_model()

```

```python
model = tf.keras.Sequential([
    # The input is the data-circuit, encoded as a tf.string
    tf.keras.layers.Input(shape=(), dtype=tf.string),
    # The PQC layer returns the expected value of the readout gate, range [-1,1].
    tfq.layers.PQC(model_circuit, model_readout),
])
```

```python
y_train_hinge = 2.0*y_train-1.0
y_test_hinge = 2.0*y_test-1.0
```

```python
def hinge_accuracy(y_true, y_pred):
    y_true = tf.squeeze(y_true) > 0.0
    y_pred = tf.squeeze(y_pred) > 0.0
    result = tf.cast(y_true == y_pred, tf.float32)

    return tf.reduce_mean(result)
```

```python
model.compile(
    loss=tf.keras.losses.Hinge(),
    optimizer=tf.keras.optimizers.Adam(),
    metrics=[hinge_accuracy])
```

```python
print(model.summary())
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    pqc_1 (PQC)                  (None, 1)                 32
    =================================================================
    Total params: 32
    Trainable params: 32
    Non-trainable params: 0
    _________________________________________________________________
    None

```python
EPOCHS = 3
BATCH_SIZE = 128

NUM_EXAMPLES = len(x_train_tfcirc)
```

```python
x_train_tfcirc_sub = x_train_tfcirc[:NUM_EXAMPLES]
y_train_hinge_sub = y_train_hinge[:NUM_EXAMPLES]

```

```python
qnn_history = model.fit(
      x_train_tfcirc_sub, y_train_hinge_sub,
      batch_size=32,
      epochs=EPOCHS,
      verbose=1,
      validation_data=(x_test_tfcirc, y_test_hinge))

qnn_results = model.evaluate(x_test_tfcirc, y_test)
```

    Epoch 1/3
    313/313 [==============================] - 733s 2s/step - loss: 0.9999 - hinge_accuracy: 0.5018 - val_loss: 0.9999 - val_hinge_accuracy: 0.4990
    Epoch 2/3
    313/313 [==============================] - 734s 2s/step - loss: 1.0007 - hinge_accuracy: 0.4879 - val_loss: 0.9999 - val_hinge_accuracy: 0.4995
    Epoch 3/3
    313/313 [==============================] - 736s 2s/step - loss: 1.0001 - hinge_accuracy: 0.5023 - val_loss: 1.0000 - val_hinge_accuracy: 0.4995
    63/63 [==============================] - 25s 401ms/step - loss: 1.0000 - hinge_accuracy: 0.4995

```python
qnn_accuracy = qnn_results[1]
qnn_accuracy
```

    0.4995039701461792

```python
model.predict_classes(x_train_tfcirc[0:7])
```

    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/sequential.py:450: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).
      warnings.warn('`model.predict_classes()` is deprecated and '

    array([[0],
           [0],
           [0],
           [0],
           [0],
           [0],
           [0]], dtype=int32)

```python
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.gridspec import GridSpec

def format_axes(fig):
    for i, ax in enumerate(fig.axes):
        ax.tick_params(labelbottom=False, labelleft=False)

fig = plt.figure(figsize=(10, 10))

gs = GridSpec(3, 3, figure=fig)
ax1 = fig.add_subplot(gs[0, 0])
# identical to ax1 = plt.subplot(gs.new_subplotspec((0, 0), colspan=3))

for i in range(3):
  for j in range(3):
    ax = fig.add_subplot(gs[i, j])

    ax.imshow(x_train[i+j, :, :, 0])

fig.suptitle("GridSpec")
format_axes(fig)

plt.show()
```

```python
plt.plot(qnn_history.history['hinge_accuracy'], label='QNN')
#plt.plot(hybrid_history.history['val_custom_accuracy'], label='Hybrid CNN')
#plt.title('Quantum vs Hybrid CNN performance')
plt.xlabel('Epochs')
plt.legend()
plt.ylabel('Validation Accuracy')
plt.show()
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# IPython basics: beyond plain Python

This short notebook will give you a quick taste of some of IPython's capabilities. IPython is the default "kernel" for most Jupyter installations (the kernel is the process that executes the code you type into notebook cells, and it's typically based on a single programming language).

IPython started its life in 2001 as a terminal-based application that offered enhancements over the default Python interactive prompt. You can still use it that way, but here we'll use it through the notebook environment in JupyterLab.

The focus of IPython is on your Interactive experience, and its features are tailored to the typical workflow of everyday scientific computation and data analysis. In a nutshell, IPython is an interactive Python prompt that:

* Starts by being a strict superset of Python: any valid Python code should behave similarly, at most returning cosmetically improved representations of information, but not different results.

* Offers features to make it easy to run small code fragments (typed directly or as external scripts with the `%run` command) and iterate on their results. Variables from execution stay defined in the kernel and you can incrementally build the state of your analysis.

* Facilitates exploration of your Python environment with the `?` operator and a powerful and extensible tab-completion system.

* Provides direct access to the underlying operating system via a syntax that's not valid Python but is convenient to type: `!cmd arg1 arg2...` is executed by the underlying OS instead of as Python.

* Provides an extensible system of special commands known as "magics," that can perform all manner of transformations on their input, letting you do anything with user input ranging from defining one-off aliases for common operations, to blending multiple programming languages in a single session. Magic commands are marked by one or two `%` characters at their start.

This notebook is a small introduction to the features of IPython. Our [IPython in depth
tutorial](https://github.com/ipython/ipython-in-depth) contains many more details on these topics and much more.

## First things first: running code, getting help

In the notebook, to run a cell of code, hit `Shift-Enter`. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end.  Alternately, you can use:

- `Alt-Enter` to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).
- `Control-Enter` executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don't need to keep permanently.

## Discovering what you need within IPython itself

IPython tries to make it easy for you to discover things about all your Python tools, objects, variables and libraries (including IPython itself, obviously) with a few keystrokes, and while staying focused on the task at hand. The first key you will want to use is `?`, which can be summoned by itself:

### Asking for help with `?`

```python
?
```

Typing `object_name?` will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.

```python
import collections
collections.namedtuple?
```

    Signature:
    collections.namedtuple(
        typename,
        field_names,
        *,
        rename=False,
        defaults=None,
        module=None,
    )
    Docstring:
    Returns a new subclass of tuple with named fields.

    >>> Point = namedtuple('Point', ['x', 'y'])
    >>> Point.__doc__                   # docstring for the new class
    'Point(x, y)'
    >>> p = Point(11, y=22)             # instantiate with positional args or keywords
    >>> p[0] + p[1]                     # indexable like a plain tuple
    33
    >>> x, y = p                        # unpack like a regular tuple
    >>> x, y
    (11, 22)
    >>> p.x + p.y                       # fields also accessible by name
    33
    >>> d = p._asdict()                 # convert to a dictionary
    >>> d['x']
    11
    >>> Point(**d)                      # convert from a dictionary
    Point(x=11, y=22)
    >>> p._replace(x=100)               # _replace() is like str.replace() but targets named fields
    Point(x=100, y=22)
    File:      ~/local/conda/lib/python3.9/collections/__init__.py
    Type:      function

If you use _two_ `??` characters, IPython will attempt to dig deeper into the object and show you its source code, if available:

```python
collections.Counter.most_common??
```

    Signature: collections.Counter.most_common(self, n=None)
    Source:
        def most_common(self, n=None):
            '''List the n most common elements and their counts from the most
            common to the least.  If n is None, then list all element counts.

            >>> Counter('abracadabra').most_common(3)
            [('a', 5), ('b', 2), ('r', 2)]

            '''
            # Emulate Bag.sortedByCount from Smalltalk
            if n is None:
                return sorted(self.items(), key=_itemgetter(1), reverse=True)
            return _heapq.nlargest(n, self.items(), key=_itemgetter(1))
    File:      ~/local/conda/lib/python3.9/collections/__init__.py
    Type:      function

The `?` character can also be used in conjunction with a shell-like wildcard `*` as a _search_ tool to find names that match a substring in your variables (or the namespace of any library or object):

```python
*int*?
```

    FloatingPointError
    breakpoint
    int
    print

### Tab completion

Tab completion, especially for attributes, is a convenient way to explore the structure of any object you’re dealing with. Simply type `object_name.<TAB>` to view the object’s attributes. Besides Python objects and keywords, tab completion also works on file and directory names.

```python
collections.
```

Before we move on, this command (whose special `%` syntax we explain below) gives you access to an IPython quick reference card:

```python
%quickref
```

## Accessing the underlying operating system

Any text prepended with a `!` is sent directly to the underlying OS:

```python
!pwd
```

    /Users/fperez/research/jupytearth/jupyter-earth/docs/interactive

and this syntax can be used in assignments, conveniently capturing command output into a Python variable you can then process further:

```python
files = !ls
print("My current directory's files:")
print(files)
```

    My current directory's files:
    ['ipython-beyond-python.ipynb']

Conversely, Python variables can be expanded when calling system commands. For simple values, `$var` expands the variable into its text form:

```python
!echo $files
```

    [ipython-beyond-python.ipynb]

and more complex expressions can be captured with `{expr}`:

```python
!echo {files[0].upper()}
```

    IPYTHON-BEYOND-PYTHON.IPYNB

Note that all this is available even in multiline blocks. While not as portable as pure Python code to do the same (it will only run in IPython and not as a regular Python script), it can be an extremely convenient way to weave together messy scripting logic that is quite common in data analysis activities:

```python
import os
for i,f in enumerate(files):
    if f.endswith('ipynb'):
        !echo {"%02d" % i} - "{os.path.splitext(f)[0]}"
    else:
        print('--')
```

    00 - ipython-beyond-python

## Beyond Python: magic functions

The IPyhton 'magic' functions are a set of commands, invoked by prepending one or two `%` signs to their name, that live in a namespace separate from your normal Python variables and provide a more command-like interface.  They take flags with `--` and arguments without quotes, parentheses or commas. The motivation behind this system is two-fold:

- To provide an orthogonal namespace for controlling IPython itself and exposing other system-oriented functionality.

- To expose a calling mode that requires minimal verbosity and typing while working interactively.  Thus the inspiration taken from the classic Unix shell style for commands.

```python
%magic
```

Line vs cell magics:

```python
%timeit list(range(1000))
```

    11 µs ± 122 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
list(range(10))
list(range(100))
```

    1.01 µs ± 25.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Like with `!cmd` above, you can capture the return value of any magic by assigning it. For example, with the `-o` flag, `%timeit` returns its output as a data structure in addition to printing the summary, so you can later do computations with it.

```python
t = %timeit -o list(range(100))
t.best
```

    786 ns ± 25.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

    7.442348900003708e-07

```python
t.best
```

    7.325789290007379e-07

_Note:_ there is no generic mechanism for capturing the output of a cell magic into a variable, though you can work around this if needed by grabbing the `_` variable that automatically stores the last computed result after running a cell magic whose return value you may want.

Line magics can be used even inside code blocks:

```python
for i in range(1, 5):
    size = i*100
    print('size:', size, end=' ')
    %timeit list(range(size))
```

    size: 100 100000 loops, best of 3: 1.86 µs per loop
    size: 200 100000 loops, best of 3: 2.49 µs per loop
    size: 300 100000 loops, best of 3: 4.04 µs per loop
    size: 400 100000 loops, best of 3: 6.21 µs per loop

Magics can do anything they want with their input, so it doesn't have to be valid Python. Note that the `bash` magic passes _all_ of its input unmodified to Bash, so `$var` isn't expanded assuming that `var` is a Python variable as `!cmd` does. This lets you paste unmodified Shell scripting code into a cell and you are guaranteed that it will run in a separate Bash process as if it was a script executed at the command line.

```bash
%%bash
echo "My shell is:" $SHELL
echo "My disk usage is:"
df -h
```

    My shell is: /bin/bash
    My disk usage is:
    Filesystem                                                                 Size   Used  Avail Capacity  iused       ifree %iused  Mounted on
    /dev/disk1s5s1                                                            1.8Ti   14Gi  432Gi     4%   553757 19538475003    0%   /
    devfs                                                                     199Ki  199Ki    0Bi   100%      690           0  100%   /dev
    /dev/disk1s4                                                              1.8Ti  1.0Gi  432Gi     1%        3 19539028757    0%   /System/Volumes/VM
    /dev/disk1s2                                                              1.8Ti  281Mi  432Gi     1%      814 19539027946    0%   /System/Volumes/Preboot
    /dev/disk1s6                                                              1.8Ti  264Ki  432Gi     1%       19 19539028741    0%   /System/Volumes/Update
    /dev/disk1s1                                                              1.8Ti  1.4Ti  432Gi    77%  3890131 19535138629    0%   /System/Volumes/Data
    map auto_home                                                               0Bi    0Bi    0Bi   100%        0           0  100%   /System/Volumes/Data/home
    /dev/disk2s2                                                              1.8Ti  1.3Ti  481Gi    75% 32911853  4262055426    1%   /Volumes/WD Time Machine Backup
    /dev/disk2s3                                                              2.7Ti  610Gi  2.1Ti    22%   675768  4294291511    0%   /Volumes/WDSilverData
    /Library/PreferencePanes/Witch.prefPane/Contents/Helpers/witchdaemon.app  1.8Ti  1.4Ti  385Gi    80%  3730872 19535297888    0%   /private/var/folders/qh/gvcvjg9x2qj0py4gwxt65x3c0010dw/T/AppTranslocation/1A4BB6CE-9FE1-4B38-9920-C49FE0DDAB87
    /dev/disk3s1                                                              603Mi  396Mi  207Mi    66%      638  4294966641    0%   /Volumes/Slack

You can also capture stdout/err from these subprocesses into Python variables, instead of letting them go directly to stdout/err

```bash
%%bash
echo "hi, stdout"
echo "hello, stderr" >&2
```

    hi, stdout

    hello, stderr

```bash
%%bash --out output --err error
echo "hi, stdout"
echo "hello, stderr" >&2
```

```python
print(error)
print(output)
```

    hello, stderr

    hi, stdout

Another interesting cell magic: create any file you want locally from the notebook:

```python
%%writefile test.txt
This is a test file!
It can contain anything I want...

And more...
```

    Writing test.txt

```python
!cat test.txt
```

    This is a test file!
    It can contain anything I want...

    And more...

Most magic functions have help built into them which you can summon with the usual `?` operator:

```python
%cd?
```

    Docstring:
    Change the current working directory.

    This command automatically maintains an internal list of directories
    you visit during your IPython session, in the variable _dh. The
    command %dhist shows this history nicely formatted. You can also
    do 'cd -<tab>' to see directory history conveniently.

    Usage:

      cd 'dir': changes to directory 'dir'.

      cd -: changes to the last visited directory.

      cd -<n>: changes to the n-th directory in the directory history.

      cd --foo: change to directory that matches 'foo' in history

      cd -b <bookmark_name>: jump to a bookmark set by %bookmark
         (note: cd <bookmark_name> is enough if there is no
          directory <bookmark_name>, but a bookmark with the name exists.)
          'cd -b <tab>' allows you to tab-complete bookmark names.

    Options:

    -q: quiet.  Do not print the working directory after the cd command is
    executed.  By default IPython's cd command does print this directory,
    since the default prompts do not display path information.

    Note that !cd doesn't work for this purpose because the shell where
    !command runs is immediately discarded after executing 'command'.

    Examples
    --------
    ::

      In [10]: cd parent/child
      /home/tsuser/parent/child
    File:      ~/local/conda/lib/python3.9/site-packages/IPython/core/magics/osm.py

The magic system is easily extensible, you can define your own magic functions with a few lines of code, [as explained in the documentation](https://ipython.readthedocs.io/en/stable/config/custommagics.html).

To close this section, let's see what other magics are currently defined in the system. You should explore these, as they can be quite useful in everyday work, and they can give you ideas for creating new ones to meet your own needs.

```python
%lsmagic -h
```

    Available line magics:
    %alias  %alias_magic  %autoawait  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %cl  %clear  %colors  %conda  %config  %connect_info  %cp  %d  %dd  %debug  %dhist  %dirs  %dl  %doctest_mode  %dp  %dus  %dx  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %pip  %popd  %pprint  %precision  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

    Available cell magics:
    %%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

    Automagic is ON, % prefix IS NOT needed for line magics.

## Multi-language integration

Through the Magic system, IPython makes it possible to create rich integrations with other programming languages. Note that this is separate from the use of entirely separate _kernels_ for Jupyter itself, that are dedicated to executing code in other languages. Here we focus on making calls to other languages from within a mostly-Python, IPython-based session.

We've already seen the use of `%%bash` above, and you may want to poke at the `%%script` magic if you'd like to run other arbitrary scripts.  Since this is meant to be a very brief intro we only show one more language integration here, with Fortran. But in [this example](https://nbviewer.jupyter.org/gist/fperez/5b49246af4e340c37549265a90894ce6/polyglot-ds.ipynb) you can see how to interleave within one single session others, including R, Julia and Cython.

To use the Fortran magic we load an extension we've previously installed:

```python
%load_ext fortranmagic
```

And with this, we now can type arbitrary Fortran functions that will be compiled, linked and imported for us using the [`f2py`](https://numpy.org/doc/stable/f2py) tool that is central to the SciPy ecosystem:

```python
%%fortran
subroutine sinx2(x, y, n)
    real, intent(in), dimension(n) :: x
    real, intent(out), dimension(n) :: y
    !intent(hide) :: n
    y = sin(x**2)
end subroutine sinx2
```

After running the above cell, we have a new function in Python called `sinx2` that computes $\sin(x^2)$ and that understands natively numpy arrays. It even has a docstring!

```python
sinx2?
```

    Call signature: sinx2(*args, **kwargs)
    Type:           fortran
    String form:    <fortran object>
    Docstring:
    y = sinx2(x,[n])

    Wrapper for ``sinx2``.

    Parameters
    ----------
    x : input rank-1 array('f') with bounds (n)

    Other Parameters
    ----------------
    n : input int, optional
        Default: len(x)

    Returns
    -------
    y : rank-1 array('f') with bounds (n)

We can now use it as any other Python function:

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
x = np.linspace(0, 2*np.pi, 200)
y = sinx2(x)
plt.plot(x, y);
```

## Rich object displays

In Python, objects can define how to display themselves both for print statements, via the special `__str__` method, and with a potentially richer representation with `__repr__`.  IPython takes this idea pretty far, allowing you to declare multiple display representations of your objects, that can include images, mathematical representations in LaTeX and arbitrarily rich HTML and JavaScript.

For example, the [SymPy library for symbolic computing](https://sympy.org) uses this system to display mathematical expressions:

```python
from sympy import symbols, I, pi, exp, Rational

x, y = symbols("x y")
Rational(3,2)*pi + exp(I*x) / (x**2 + y)
```

$\displaystyle \frac{3 \pi}{2} + \frac{e^{i x}}{x^{2} + y}$

This is the system that Pandas takes advantage of to display datframes as a readable, HTML table:

```python
import pandas as pd

pd.DataFrame(dict(x=[3,2,1], y=[10,20,30]))
```

      x
      y

      0
      3
      10

      1
      2
      20

      2
      1
      30

The [Xarray library](https://xarray.pydata.org/en/stable/examples/multidimensional-coords.html) makes particularly good use of this system, to provide very informative displays of datasets that describe the structure of the data immediately with a collapsible HTML view. Given the complexity of datasets frequently encountered in the geosciences and that Xarray can read, this can be a very useful feature when interactively exploring data:

```python
import xarray as xr

xr.tutorial.open_dataset('rasm').load()
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.Dataset>
Dimensions:  (time: 36, x: 275, y: 205)
Coordinates:
  * time     (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00
    xc       (y, x) float64 189.2 189.4 189.6 189.7 ... 17.65 17.4 17.15 16.91
    yc       (y, x) float64 16.53 16.78 17.02 17.27 ... 28.26 28.01 27.76 27.51
Dimensions without coordinates: x, y
Data variables:
    Tair     (time, y, x) float64 nan nan nan nan nan ... 29.8 28.66 28.19 28.21
Attributes:
    title:                     /workspace/jhamman/processed/R1002RBRxaaa01a/l...
    institution:               U.W.
    source:                    RACM R1002RBRxaaa01a
    output_frequency:          daily
    output_mode:               averaged
    convention:                CF-1.4
    references:                Based on the initial model of Liang et al., 19...
    comment:                   Output from the Variable Infiltration Capacity...
    nco_openmp_thread_number:  1
    NCO:                       netCDF Operators version 4.7.9 (Homepage = htt...
    history:                   Fri Aug  7 17:57:38 2020: ncatted -a bounds,,d...

xarray.Dataset

<input id='section-15afd890-f6ed-48e3-8dc5-a3d6956431b6' class='xr-section-summary-in' type='checkbox' disabled >Dimensions:

time: 36

x: 275

y: 205

<input id='section-b94413e6-b52a-467d-b6cd-d389aeb73f30' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (3)

time

(time)

object

1980-09-16 12:00:00 ... 1983-08-...
<input id='attrs-1a5a357e-50a7-4e8f-8c9d-2df711fe3d89' class='xr-var-attrs-in' type='checkbox' ><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-c238fda8-5b5a-4006-b5dc-ffaf4d095a6c' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

long_name :

time

type_preferred :

int

array([cftime.DatetimeNoLeap(1980, 9, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1980, 10, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1980, 11, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1980, 12, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 1, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 3, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 4, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 5, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 6, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 7, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 8, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 9, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 10, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 11, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1981, 12, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 1, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 3, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 4, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 5, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 6, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 7, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 8, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 9, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 10, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 11, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1982, 12, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 1, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 3, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 4, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 5, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 6, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 7, 17, 0, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(1983, 8, 17, 0, 0, 0, 0, has_year_zero=True)],
      dtype=object)

xc

(y, x)

float64

189.2 189.4 189.6 ... 17.15 16.91
<input id='attrs-ac254b13-7410-46b0-809f-82cc98b23230' class='xr-var-attrs-in' type='checkbox' ><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-0834c074-5eeb-40b2-8b3a-4a264c44efef' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

long_name :

longitude of grid cell center

units :

degrees_east

array([[189.22293223, 189.38990916, 189.55836619, ..., 293.77906088,
        294.0279241 , 294.27439931],
       [188.96836986, 189.13470591, 189.30253733, ..., 294.05584005,
        294.30444387, 294.55065969],
       [188.71234264, 188.87800731, 189.04515208, ..., 294.335053  ,
        294.58337453, 294.8292928 ],
       ...,
       [124.04724025, 123.88362026, 123.71852016, ...,  16.83171831,
         16.58436953,  16.33949649],
       [123.78686428, 123.62254238, 123.45672512, ...,  17.11814486,
         16.87043749,  16.62518298],
       [123.52798366, 123.36295986, 123.1964407 , ...,  17.40209947,
         17.1540526 ,  16.90845095]])

yc

(y, x)

float64

16.53 16.78 17.02 ... 27.76 27.51
<input id='attrs-5d713a9f-2ee3-443f-8535-0d1e7d0f0174' class='xr-var-attrs-in' type='checkbox' ><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-f00cf79a-342c-4fa3-8085-d4cacccabb98' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

long_name :

latitude of grid cell center

units :

degrees_north

array([[16.53498637, 16.7784556 , 17.02222429, ..., 27.36301592,
        27.11811045, 26.87289026],
       [16.69397341, 16.93865381, 17.18364512, ..., 27.5847719 ,
        27.33821848, 27.0913656 ],
       [16.85219179, 17.09808909, 17.34430872, ..., 27.80584314,
        27.55764558, 27.30915621],
       ...,
       [17.31179033, 17.56124674, 17.81104646, ..., 28.4502485 ,
        28.19718339, 27.94384744],
       [17.15589701, 17.40414034, 17.65272318, ..., 28.23129632,
        27.97989251, 27.72821596],
       [16.99919497, 17.24622904, 17.49358736, ..., 28.01160028,
        27.76185586, 27.51182726]])

<input id='section-419c39c1-b2e4-45d4-bdd0-e7fff31b4e75' class='xr-section-summary-in' type='checkbox'  checked>Data variables: (1)

Tair

(time, y, x)

float64

nan nan nan ... 28.66 28.19 28.21
<input id='attrs-4667fadf-5f4b-4a49-adeb-fcfdf51905cf' class='xr-var-attrs-in' type='checkbox' ><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-4ea58b7a-36bb-4c50-a5f6-e60383a263e5' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

units :

C

long_name :

Surface air temperature

type_preferred :

double

time_rep :

instantaneous

array([[[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        ...,
        [        nan,         nan,         nan, ..., 27.03290153,
         27.03125761, 27.33531541],
        [        nan,         nan,         nan, ..., 27.2784053 ,
         26.80261869, 27.08603517],
        [        nan,         nan,         nan, ..., 27.02344402,
         26.56473862, 26.73064933]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
...
        [        nan,         nan,         nan, ..., 27.8597472 ,
         27.82928439, 28.09249224],
        [        nan,         nan,         nan, ..., 27.89704094,
         27.31104941, 27.67387171],
        [        nan,         nan,         nan, ..., 27.46837113,
         27.0088944 , 27.23017976]],

       [[        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        [        nan,         nan,         nan, ...,         nan,
                 nan,         nan],
        ...,
        [        nan,         nan,         nan, ..., 28.95929072,
         28.87672039, 29.04890862],
        [        nan,         nan,         nan, ..., 29.036132  ,
         28.42273578, 28.68721201],
        [        nan,         nan,         nan, ..., 28.66381585,
         28.18595533, 28.20753022]]])

<input id='section-7fd4725f-4017-44df-bcf5-675bdad36f2f' class='xr-section-summary-in' type='checkbox'  >Attributes: (11)

title :

/workspace/jhamman/processed/R1002RBRxaaa01a/lnd/temp/R1002RBRxaaa01a.vic.ha.1979-09-01.nc

institution :

U.W.

source :

RACM R1002RBRxaaa01a

output_frequency :

daily

output_mode :

averaged

convention :

CF-1.4

references :

Based on the initial model of Liang et al., 1994, JGR, 99, 14,415- 14,429.

comment :

Output from the Variable Infiltration Capacity (VIC) model.

nco_openmp_thread_number :

1

NCO :

netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)

history :

Fri Aug  7 17:57:38 2020: ncatted -a bounds,,d,, rasm.nc
Tue Dec 27 14:15:22 2016: ncatted -a dimensions,,d,, rasm.nc rasm.nc
Tue Dec 27 13:38:40 2016: ncks -3 rasm.nc rasm.nc
history deleted for brevity

Again, we're only scratching the surface of these capabilities here - you can read more in the IPython docs and in the [IPython in depth tutorial](https://github.com/ipython/ipython-in-depth), particularly in the notebooks that illustrate the [rich output](https://nbviewer.jupyter.org/github/ipython/ipython-in-depth/blob/master/examples/IPython%20Kernel/Rich%20Output.ipynb) and [custom display logic](https://nbviewer.jupyter.org/github/ipython/ipython-in-depth/blob/master/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb) that powers these capabilities.


```python
import pandas as pd
from PIL import Image, ImageDraw
from pathlib import Path
import os
import numpy as np
import matplotlib.pyplot as plt
import cv2 as cv
```

```python
def get_frame_from_video(frame, video):
    frame = frame - 1
    !ffmpeg \
        -hide_banner \
        -loglevel fatal \
        -nostats \
        -i $video -vf "select=eq(n\,$frame)" -vframes 1 frame.png
    img = Image.open('frame.png')
    os.remove('frame.png')
    return img

def annotate_frame(img, xc, yc, r, col = (57, 255, 20)):
    draw = ImageDraw.Draw(img)
    for x, y in zip(xc, yc):
        draw.ellipse((x-r, y-r, x+r, y+r), fill=col, outline='black')
    return img

# code from: https://www.kaggle.com/robikscube/nfl-helmet-assignment-getting-started-guide
def add_track_features(tracks, fps=59.94, snap_frame=10):
    """
    Add column features helpful for syncing with video data.
    """
    tracks = tracks.copy()
    tracks["game_play"] = (
        tracks["gameKey"].astype("str")
        + "_"
        + tracks["playID"].astype("str").str.zfill(6)
    )
    tracks["time"] = pd.to_datetime(tracks["time"])
    snap_dict = (
        tracks.query('event == "ball_snap"')
        .groupby("game_play")["time"]
        .first()
        .to_dict()
    )
    tracks["snap"] = tracks["game_play"].map(snap_dict)
    tracks["isSnap"] = tracks["snap"] == tracks["time"]
    tracks["team"] = tracks["player"].str[0].replace("H", "Home").replace("V", "Away")
    tracks["snap_offset"] = (tracks["time"] - tracks["snap"]).astype(
        "timedelta64[ms]"
    ) / 1_000
    # Estimated video frame
    tracks["est_frame"] = (
        ((tracks["snap_offset"] * fps) + snap_frame).round().astype("int")
    )
    return tracks

def add_video_features(videos):
    videos['game_play'] = videos['video_frame'].apply(lambda x: '_'.join(x.split('_')[:2]))
    videos['camera'] = videos['video_frame'].apply(lambda x: x.split('_')[2])
    videos['frame'] = videos['video_frame'].apply(lambda x: x.split('_')[-1])
    videos['xc'] = (videos['left'] + videos['width']/2).astype(int).values
    videos['yc'] = (videos['top'] + videos['height']/2).astype(int).values
    return videos

def annotate_field(xc, yc, player, r = 10, width = 3, col = [(27, 3, 163), (255, 7, 58)], crop = None):
    field = Image.open('../input/nflhelmet-helper-dataset/field.png')
    w, h = field.size
    zero = (68,68)
    fs = (2424,1100)
    draw = ImageDraw.Draw(field)
    xc, yc = xc*fs[0]/120 + zero[0], (1 - yc/53.3)*fs[1] + zero[1]
    for x, y, p in zip(xc, yc, player):
        c = col[0] if p[0] == 'H' else col[1]
        draw.ellipse((x-r, y-r, x+r, y+r), fill=c, width=width, outline = 'black')
    if isinstance(crop, float):
#         cp = [xc.min() - crop*w, yc.min() - crop*h, xc.max() + crop*w, yc.max() + crop*h]
        cp = [xc.min() - crop*w, 0, xc.max() + crop*2*w, h]
        return field.crop(cp)
    else:
        return field

class show_play_with_tracking():

    def __init__(self, video_df = None, track_df = None):
        if video_df is None:
            video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
        self.video_df = add_video_features(video_df)
        if track_df is None:
            tracking_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_player_tracking.csv')
            tracking_df = add_track_features(tracking_df)
        self.tracking_df = tracking_df.query("est_frame > 0")

    def __call__(self, game_play, frame, img_size = 800, video_folder = '../input/nfl-health-and-safety-helmet-assignment/train/'):

        camera = 'Sideline'
        frame_side = get_frame_from_video(frame, video_folder + game_play + '_' + camera + '.mp4')
        df = self.video_df.query(f"game_play == '{game_play}' and frame == '{frame}' and camera == '{camera}'")
        frame_side = annotate_frame(frame_side, df.xc, df.yc, 10)

        camera = 'Endzone'
        frame_end = get_frame_from_video(frame, video_folder + game_play + '_' + camera + '.mp4')
        df = self.video_df.query(f"game_play == '{game_play}' and frame == '{frame}' and camera == '{camera}'")
        frame_end = annotate_frame(frame_end, df.xc, df.yc, 10)

        frames = self.tracking_df['est_frame'].values
        if frame not in frames:
            index = np.absolute(frames-frame).argmin()
            frame = frames[index]
        df = self.tracking_df.query(f"game_play == '{game_play}' and est_frame == {frame}")
        field = annotate_field(df.x, df.y, df.player, 20, crop = 0.01)

        wf, hf = field.size
        wc, hc = frame_side.size
        field = field.resize((int(wf*2*hc/hf), 2*hc))
        wf, hf = field.size

        img = Image.new('RGB', (wf+wc+20, 2*hc+20))
        img.paste(im=field, box=(5, 10))
        img.paste(im=frame_side, box=(wf+15, 5))
        img.paste(im=frame_end, box=(wf+15, hc+15))
        img.thumbnail((img_size,img_size))
        return img

spwt = show_play_with_tracking()

# TODO, add interpolation of tracking_df and replace nearest
class get_keypoints():

    def __init__(self, video_df = None, track_df = None):
        if video_df is None:
            video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
        self.video_df = add_video_features(video_df)
        if track_df is None:
            tracking_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_player_tracking.csv')
            tracking_df = add_track_features(tracking_df)
        self.tracking_df = tracking_df.query("est_frame > 0")

    def __call__(self, game_play, frame, normalized = True):
        keypoints = dict()
        keypoints['Sideline'] = self.video_df.query(
            f"game_play == '{game_play}' and frame == '{frame}' and camera == 'Sideline'")[['xc', 'yc']].values
        keypoints['Endzone'] = self.video_df.query(
            f"game_play == '{game_play}' and frame == '{frame}' and camera == 'Endzone'")[['xc', 'yc']].values

        frames = self.tracking_df['est_frame'].values
        if frame not in frames:
            index = np.absolute(frames-frame).argmin()
            frame = frames[index]
        keypoints['Tracking'] = self.tracking_df.query(
            f"game_play == '{game_play}' and est_frame == {frame}")[['x', 'y']].values

        if normalized:
            for k, v in keypoints.items():
                keypoints[k] = (v - v.min(axis = 0)) / (v.max(axis = 0) - v.min(axis = 0))

        keypoints['Sideline'][:,1] = 1-keypoints['Sideline'][:,1]

        self.keypoints = keypoints

        return keypoints

    def plot(self, add_no = False):
        if not hasattr(self, 'keypoints'):
            print('you must run the function first...')
        else:
            kp = self.keypoints
            plt.figure(figsize=(6, 6))
            plt.scatter(kp['Endzone'][:,0], kp['Endzone'][:,1], marker = 'x', color = 'red', label = 'Endzone')
            plt.scatter(kp['Sideline'][:,0], kp['Sideline'][:,1], marker = '^', color = 'red', label = 'Sideline')
            plt.scatter(kp['Tracking'][:,0], kp['Tracking'][:,1], marker = 'o', color = 'green', label = 'Tracking')
            plt.legend();

get_kp = get_keypoints()
```

```python
spwt('57583_000082', 10)
```

```python
videoframes
```

What I wanted to do is to map each player from tracking data to the camera data like this:

The same data can be normalized and viewed on the same plot:

```python
k = get_kp('57583_000082', 12)
get_kp.plot(True)
```

# Point Cloud Registration

So, a good portion of this comp comes down to point-cloud registration. There are plenty of methods that perform this taks. For example, OpenCV have `cv.findHomography`. As you can see below, this doesn't perform well. The main reason is that although the function uses Random sample consensus (RANSAC) to filter outliers, the clouds of points must be roughly alinged already.

```python
srcPoints = k['Tracking'].astype('float32').reshape(-1,1,2)
dstPoints = k['Sideline'].astype('float32').reshape(-1,1,2)
M, mask = cv.findHomography(srcPoints, dstPoints, cv.RANSAC)
print(mask.sum())
tfmdPoints = cv.perspectiveTransform(srcPoints,M)
plt.scatter(srcPoints[:,0,0], srcPoints[:,0,1], marker = 'o', color = 'red', label = 'source')
plt.scatter(dstPoints[:,0,0], dstPoints[:,0,1], marker = '^', color = 'green', label = 'target')
plt.scatter(tfmdPoints[:,0,0], tfmdPoints[:,0,1], marker = 'o', color = 'blue', label = 'result')
plt.legend();
```

# Pure Pytorch implementation (using gradient descent)

So, isntead of using open CV, I decided to do this my won using pytorch. The idea here is to solve the folowing problem:

\begin{equation}
\begin{bmatrix} x^{'} \\ y^{'} \\ 1 \end{bmatrix} = H \begin{bmatrix} x \\ y \\ 1 \end{bmatrix} = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}
\end{equation}

where $H$ is a transformation matrix (that I want to find) and $(x', y')$ is the transformed coorinates of a point $(x, y)$.

The optimization problem is to find $H$ such as $(x', y')$ is as close as possible to some ground truth $(x_t, y_t)$ cloud of points.

```python
import torch
```

```python
def min_mse(preds, targets):
    d = torch.cdist(preds.squeeze(2), targets.squeeze(2))
    loss = (d.min(dim = 1).values**2).mean().sqrt()
    return loss
```

```python
def step(src, trg, m, lr = 3e-3, prt = True):
    preds = torch.matmul(m, src) # Homography transform
    loss = min_mse(preds, trg)   # mse between the closes pair of points
    if prt: print(f'loss: {(loss.item()):.5f}')
    loss.backward()
    m.data -= lr * m.grad.data
    m.grad = None
```

```python
def fit_predict(src, trg, init_rot = 0, lr = 3e-3, n_steps = 10000, verbose = True):
    t = np.pi * init_rot / 180
    m = torch.tensor([[np.cos(t),-np.sin(t), 0],
                      [np.sin(t), np.cos(t), 0],
                      [        0,         0, 1]], dtype = torch.double)
    m.requires_grad_()
    for i in range(n_steps):
        if i % (n_steps//10) and verbose:
            step(src, trg, m, lr=lr, prt=False)
        else:
            step(src, trg, m, lr=lr)

    with torch.no_grad():
        tfm = torch.matmul(m, src)

    if verbose:
        plt.scatter(src[:,0], src[:,1], marker = 'o', color = 'red', label = 'source')
        plt.scatter(trg[:,0], trg[:,1], marker = '^', color = 'green', label = 'target')
        plt.scatter(tfm[:,0], tfm[:,1], marker = 'o', color = 'blue', label = 'result')
        plt.legend();

    return tfm
```

```python
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg)
```

```python
k = get_kp('57597_000658', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg)
```

```python
k = get_kp('57781_000252', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg, 0)
```

```python
spwt('57781_000252', 1)
```

```python
spwt
```

```python
video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
video_df = add_video_features(video_df)
video_df.query(f"game_play == '57781_000252' and frame == '1' and camera == 'Sideline'").head(10)
```

```python
video_df = pd.read_csv('../input/nfl-health-and-safety-helmet-assignment/train_baseline_helmets.csv')
video_df = video_df.query('conf > 0.8')
video_df = add_video_features(video_df)
get_kp_highconf = get_keypoints(video_df)
```

```python
k = get_kp_highconf('57781_000252', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Sideline']), torch.ones(len(k['Sideline'])).unsqueeze(1)], axis = -1).unsqueeze(2)
tfm = fit_predict(src, trg, 0)
```

```python
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Endzone']), torch.ones(len(k['Endzone'])).unsqueeze(1)], axis = -1).unsqueeze(2)
print(src.shape, trg.shape)
tfm = fit_predict(src, trg)
```

```python
k = get_kp('57583_000082', 1, True)
src = torch.cat([torch.tensor(k['Tracking']), torch.ones(len(k['Tracking'])).unsqueeze(1)], axis = -1).unsqueeze(2)
trg = torch.cat([torch.tensor(k['Endzone']), torch.ones(len(k['Endzone'])).unsqueeze(1)], axis = -1).unsqueeze(2)
print(src.shape, trg.shape)
tfm = fit_predict(src, trg, 90)
```