#  CS109A Introduction to Data Science

## Lab 3:  Scikit-learn for Regression

**Harvard University**

**Fall 2018**

**Instructors:** Pavlos Protopapas and Kevin Rader

**Authors:** David Sondak, Will Claybaugh, Eleni Kaxiras

---

```python
# RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

---

# Table of Contents

 Array creation and reshape

 Some plotting

 Simple linear regression

 $k$-nearest neighbors

## Learning Goals

Overall description and goal for the lab.

By the end of this lab, you should be able to:
* Understand array reshaping
* Review how to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lecture 4 and maps on to homework 2 (and beyond).**

```python
# import the necessary libraries
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
import seaborn as sns
```

## Simple Linear Regression
Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (10 min)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

    numpy.ndarray

```python
x_train.shape
```

    (3,)

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

    (3, 1)

```python
xx = np.array([[1,3,5],[6,2,1]])
xx.shape
```

    (2, 3)

```python
xx = xx.reshape(3,-1)
xx
```

    array([[1, 3],
           [5, 6],
           [2, 1]])

```python
# make a simple scatterplot
x_train = np.array([1,2,3])
y_train = np.array([2,2,4])
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)
```

```python
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')
```

#### Formulae
Linear regression is special among the models we study beuase it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

From the re-aranged second equation we can see that the best-fit line  passes through $(\bar{x},\bar{y})$, the center of mass of the data

From any of the first equations, we can see that the slope of the line has to do with whether or not an x value that is above/below the center of mass is typically paired with a y value that is likewise above/below, or typically paired with one that is opposite.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise (5 min)

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

    (3, 2)

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

    (3, 1)

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

    () ()

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:3.2f} + {1:3.2f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

    The best-fit line is 0.67 + 1.00 * x
    The best fit is 0.6666666666666665

Exercise (5 min)

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])

```

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

    Reshaping features array.
    Reshaping observations array.
    The best-fit line is 0.666667 + 1.000000 * x

Exercise (5 min)

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');
```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

### Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print("The regression coefficients from the statsmodels package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_sm, beta1_sm))
```

    [[1. 1.]
     [1. 2.]
     [1. 3.]]
    The regression coefficients from the statsmodels package are: beta_0 = 0.666667 and beta_1 = 1.000000

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

    OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.750
    Model:                            OLS   Adj. R-squared:                  0.500
    Method:                 Least Squares   F-statistic:                     3.000
    Date:                Mon, 01 Jun 2020   Prob (F-statistic):              0.333
    Time:                        18:49:07   Log-Likelihood:                -2.0007
    No. Observations:                   3   AIC:                             8.001
    Df Residuals:                       1   BIC:                             6.199
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6667      1.247      0.535      0.687     -15.181      16.514
    x1             1.0000      0.577      1.732      0.333      -6.336       8.336
    ==============================================================================
    Omnibus:                          nan   Durbin-Watson:                   3.000
    Prob(Omnibus):                    nan   Jarque-Bera (JB):                0.531
    Skew:                          -0.707   Prob(JB):                        0.767
    Kurtosis:                       1.500   Cond. No.                         6.79
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

    The regression coefficients from the sklearn package are: beta_0 = 0.666667 and beta_1 = 1.000000

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The shape of things in `scikit-learn`
Before diving right in to a "real" problem, we really ought to discuss more of the details of `sklearn`.  We do this now.  Along the way, we'll import the real-world dataset.

`Scikit-learn` is the main `python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split`. It can be used in `python` by the incantation `import sklearn`.

In scikit-learn, an **estimator** is a Python object that implements the methods fit(X, y) and predict(T)

Let's see the structure of `scikit-learn` needed to make these fits. `.fit` always takes two arguments:
```python
  estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

Critically, `Xtrain` must be in the form of an *array of arrays* (or a 2x2 array) with the inner arrays each corresponding to one sample, and whose elements correspond to the feature values for that sample (visuals coming in a moment).

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn`
We begin by loading up the `mtcars` dataset and cleaning it up a little bit.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("/Users/jamillaakhund-zade/CS109A/2019-CS109A/content/labs/lab03/data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

    Shape of full dataset is: (32, 12)
    Shape of training dataset is: (25, 12)
    Shape of test dataset is: (7, 12)

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
```

Notice the shape of `y_train`.

```python
np.shape(y_train)
```

    (25,)

Another way to see the shape is to use the shape method.

```python
y_train.shape
```

    (25,)

This is *not* an "array of arrays".  That's okay!  Remember, `sklearn` requires an array of arrays only for the predictor array!  You will have to pay close attention to this in the exercises later.

For now, let's discuss two ways out of this debacle.  All we'll do is get `y_train` to be an array of arrays.  This doesn't hurt anything because `sklearn` doesn't care too much about the shape of `y_train`.

First, let's reshape `y_train` to be an array of arrays using the `reshape` method.  We want the first dimension of `y_train` to be size $25$ and the second dimension to be size $1$.

```python
y_train_reshape = y_train.values.reshape(y_train.shape[0], 1)
```

```python
y_train_reshape.shape
```

    (25, 1)

Notice that `y_train.shape[0]` gives the size of the first dimension.

There's an even easier way to get the correct shape right from the beginning.

```python
y_train_reshape = traindf[['mpg']]
```

```python
y_train_reshape.shape
```

    (25, 1)

Finally, there is a nice shortcut to reshaping an array.  `numpy` can infer a dimension based on the other dimensions specified.

```python
y_train_reshape = y_train.values.reshape(-1,1)
y_train_reshape.shape
```

    (25, 1)

In this case, we said the second dimension should be size $1$.  Since the requirement of the `reshape()` method is that the requested dimensions be compatible, `numpy` decides the the first dimension must be size $25$.

What would the `.shape` return if we did `y_train.values.reshape(-1,5)`?

Okay, enough of that.  The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

### Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise (10 min)

* Pick one variable to use as a predictor for simple linear regression.  Create a markdown cell below and discuss your reasons.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
# Your code here
```

```python
y_mpg = dfcars.mpg
x_wt = dfcars.wt
x_hp = dfcars.hp

fig_wt, ax_wt = plt.subplots(1,1, figsize=(10,6))
ax_wt.scatter(x_wt, y_mpg)
ax_wt.set_xlabel(r'Car Weight')
ax_wt.set_ylabel(r'Car MPG')

fig_hp, ax_hp = plt.subplots(1,1, figsize=(10,6))
ax_hp.scatter(x_hp, y_mpg)
ax_hp.set_xlabel(r'Car HP')
ax_hp.set_ylabel(r'Car MPG')

```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

**Hints:**
* Use the following to perform the analysis:
```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
```

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("/Users/jamillaakhund-zade/CS109A/2019-CS109A/content/labs/lab03/data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)

#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(r2)
```

    0.6879761857596272

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

    0.7701379909791617
    12.475985659918837
    7.773697766387512
    Coefficients:
     -5.336941400557081 36.93731031351841

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
```

## $k$-nearest neighbors

Great, so we did a simple linear regression on the car data.

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.  Let's use $5$ nearest neighbors.

```python
from sklearn.neighbors import KNeighborsRegressor
knnreg = KNeighborsRegressor(n_neighbors=5)
```

```python
knnreg.fit(X_train, y_train)
r2 = knnreg.score(X_test, y_test)
r2
```

    0.6992188728167332

Exercise

What is the $R^{2}$ score on the training set?

```python
# Your code here
```

```python
# solution
knnreg.score(X_train, y_train)
```

    0.8718092469742431

Lets vary the number of neighbors and see what we get.

```python
regdict = {}
# Do a bunch of KNN regressions
for k in [1, 2, 4, 6, 8, 10, 15]:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    regdict[k] = knnreg # Store the regressors in a dictionary
```

```python
# Now let's plot it all
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
for k in [1, 2, 6, 10, 15]:
    predictions = regdict[k].predict(xgrid.reshape(100,1))
    if k in [1, 6, 15]:
        ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere. Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    knnreg = KNeighborsRegressor(n_neighbors=k) # Create KNN model
    knnreg.fit(X_train, y_train) # Fit the model to training data
    score_train = knnreg.score(X_train, y_train) # Calculate R^2 score
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Why do we get a perfect $R^2$ at k=1?

Exercise (5 min)

* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
ks = range(1, 15) # Grid of k's
scores_test = [] # R2 scores
for k in ks:
    knnreg = KNeighborsRegressor(n_neighbors=k) # Create KNN model
    knnreg.fit(X_train, y_train) # Fit the model to training data
    score_test = knnreg.score(X_test, y_test) # Calculate R^2 score
    scores_test.append(score_test)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_test,'o-', ms=12)
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

```python

```


# A Robust Peak Numbering Algorithm

## Problem description

Signal output of a Multi-Pixel Photon Counter (MPPC) is best seen by this scope screenshot provided by Hamamatsu.

The most conspicuous feature in the screenshot is that the amplitudes of signals are integral multiples of a unit amplitude. In this example, the green trace has an amplitude 2 times as large as that of the orange trace. If one integrates the area under the signals, the area values will concentrate at integral multiples of a certain area, forming peaks with a equal distance. Below is an example.

```python
# my own modules
import os, sys
sys.path.insert(0, '../FEBDAQMULTx2/data_analysis/utilities')
import common_tools

# standard imports
from matplotlib import cm
import matplotlib.pyplot as plt
import numpy as np

# quiet warnings
import warnings
warnings.filterwarnings('ignore')

bv58_thr205 = common_tools.MPPCLine('sample_data/robust_peak_numbering/20210301_124140_dark_rate_feb0_ch0_thr205.0.root', 1, 0, prom=100, pcb_half=0)
bv58_thr205.adc_spectrum()
```

### Trigger threshold
The data acquisition system is triggered whenever the signal is larger than a predefined level called threshold, or equivalently here, DAC. This DAC effectively is a minimum area. All events with an area smaller than this minimum area are simply ignore by the system. The effect on the ADC spectra is that there will be peaks on the low ADC side removed by the DAC. Below is an example of data taken with a high DAC value.

```python
bv58_thr244 = common_tools.MPPCLine('sample_data/robust_peak_numbering/20210225_172920_mppc_volt58.0_thr244_gain52_temp20.0.root', 1, 0, prom=100, pcb_half=0)
bv58_thr244.adc_spectrum()
```

Obviously, several peaks on the left are removed by a higher DAC.
### Peak numbering convention
If the DAC is low enough, the first peak seen is the area under the baseline, or the pedestal peak. A number 0 is assigned to the pedestal peak. The equidistance peaks to the right are sequentially numbered 1, 2,... etc.
### The question
Given a dataset taken with an arbitrary threshold, determine the number of peaks removed by that threshold.
## Numbering peaks with auxiliary data
The gain, or the distance between peaks, has a linear relationship with the bias voltage applied to the MPPCs. Since a MPPC is diode, there is a breakdown voltage, beyond which a diode becomes conductive, and electrons undergo the avalanche effect, resulting in internal signal multiplication.

The gain is known to be proportional to the overvoltage, or the bias voltage less the breakdown voltage. To find the correct peak numbering for a certain bias voltage, one can seek help from data taken with different bias voltages. Since we know that peaks are equidistant, we can use certain peak finding algorithm to identify peak positions in ADC unit, and number the found peaks sequentially starting from 0. Below is an example of peak ADC vs peak number.

```python
data_files = [
    'sample_data/robust_peak_numbering/20210301_112158_dark_rate_feb0_ch0_thr205.0.root',
    'sample_data/robust_peak_numbering/20210301_124140_dark_rate_feb0_ch0_thr205.0.root',
    'sample_data/robust_peak_numbering/20210301_142803_dark_rate_feb0_ch0_thr205.0.root',
    'sample_data/robust_peak_numbering/20210301_163042_dark_rate_feb0_ch0_thr205.0.root'
]

my_lines = common_tools.MPPCLines(data_files, 1, 0, prom=100, pcb_half=0)
my_lines.plot_line_group(my_lines.mppc_lines)
```

It's clearly seen that different bias voltages result in different slope, or equivalently, gain. Also note that if peaks are all numbered correctly, the lines should all originate from the same point at 0 peak ID.
### A first algorithm to renumber peaks
A first simple idea can be something like this.
- For each line, shift it by some integral units to the left or right. Each combinations of the shifted lines form a *configuration*.
- For each configuration, take any two lines, and find their intersection point. All intersection points thus found form a point cloud.
- Take the average distance between any two points in the cloud. The configuration with the smallest average distance is the solution.

Let's see how this algorithm works.

```python
my_lines.enumerate_peaks()
```

The first plot is before the algorithm is executed, and the second one is after. The red cross is the centroid of the point cloud of intersection points. It does not have a 0 x-coordinate, so some guesswork has to be done. Besides, in certain datasets, this algorithm just gives the wrong answer (based on some educated guess).

A more robust algorithm is required.

## A formula linking peak ADC, photoelectron number (PE), and bias voltage
Since we know that the gain is proportional to overvoltage, we have

$$
\begin{align}
\frac{\Delta ADC}{\Delta PE} \sim (V-V_{bd}) \\
\Rightarrow \frac{\Delta ADC}{\Delta PE} = \beta (V-V_{bd})
\end{align}
$$
Integrate to get
$$
\boxed{ADC=\beta (V-V_{bd})PE+\gamma}
$$
Here, $\beta$ is the proportional constant, $V$ the bias voltage, $V_{bd}$ the breakdown voltage, and $\gamma$ a constant for pedestal ADC.

Of course, in the mathematical model, I have made the assumption that the pedestal ADC does not depend on the applied voltage, and this turns out to be a good approximation for peak numbering purpose.
### Visualize the ADC function
We can visualize the ADC as a function of PE and bias voltage with $\beta=8$, $V_{bd}=52$, and $\gamma=180$.

```python
sys.path.insert(0, '../FEBDAQMULTx2/data_analysis/tests')
from draw_adc_surface import adc_value

fig = plt.figure(num=1, clear=True)
ax = fig.add_subplot(1, 1, 1, projection='3d')

# create the meshgrid with corresponding ranges
(x, y) = np.meshgrid(np.linspace(0, 8, 9), np.linspace(51, 61, 11))
z = adc_value(x, y, beta=8, vbd=52, gamma=180)

# plot the surface
ax.plot_surface(x, y, z, cmap=cm.hot)
ax.set(xlabel='photoelectron', ylabel='bias voltage (V)', zlabel='ADC',
       title='ADC as a function of #PE and bias voltage')

fig.tight_layout()
```

Each dot in the ADC vs. PE plots corresponds to a point in this 3D space, $(PE,V_{bd},ADC)$. By fitting the ADC surface to the 3D points, relevant physical parameters can be estimated.
### Finding the best numbering configuration and parameter initialization
Before fit the ADC function, the best numbering configuration has to be found. The idea is like this.
For a configuration:
- For each line with a certain bias voltage, there is a ADC value at $PE=0$.
- With $N$ lines, there are $N$ pedestal ADC values.
- Since it is assumed that all lines eminate from a single point, the configuration with the least pedestal ADC spread is the best solution.
- To make this algorithm more robust, instead of using different $\beta$ values for each line from individual line fits, an average number is used for all lines, $\beta=\sum_i \beta_i/N$, where $\beta_i=slope_i/(V_i-V_{bd})$.

Here is an example pedestal distribution before and after configuration search.

```python
import peak_numbering

my_pn = peak_numbering.fitting_algorithm(data_files, 1, ch=0)
my_pn.fit_peak_numbering(shift_limit=4)
my_pn.plot_gamma_spread()
```

### Refit physical parameters
With the best configuration found in the previous paragraph, the ADC surface function can be fit to the corrected $(PE,V_{bd},ADC)$ points for estimating the physical parameters $\beta$, $V_{bd}$, and $\gamma$.

In the end, the ADC vs PE plot can be redrawn to visualize the final result.

```python
my_pn.refit_physics_parameters()
my_pn.plot_adc_vs_peak_number()
```

## Conclusion
From the final plot, one can see that this algorithm produces fitted lines originating from the same point at 0 PE. Besides, it is very resistant to the errors in the found peak ADC values, since this algorithm utilizes all peaks from all bias voltages for one fit, as opposed to the first algorithm which treats datasets with different bias voltages independently.


# FDT in climate science (fluctuation dissipation theorem)

FDT: The use of fluctuation dissipation theorem can be traced back to Annus mirabilis in 1905. The year that Albert Einstein published 5 legendary paper which changed how we see the world. However, I am not going to talk about "Relativity". Instead, this project will focus on FDT, a theorem was applied to explain Brownian motion (or other fields in Statistical mechanics). FDT has a lot of names in climate science and you might be able to name a few of them, such as (1) linear response function, (2) Jacobian Matrix, (3) Green's function, (4) linear inverse model or (5) Markov train.

Before I jump into the details of FDT, I would like give you a general picture of the physical assumption over different dynamical models (or dynamical systems). All of the dynamical models we use today are based on different physical assumption. For example, when we use a global climate model with cumulus parameterization, we are assuming a quasi-equilibrium state of short-lived cumulus cloud, which nearly strikes balance with large-scale tendency. When we use cloud permitting model, we can use vertical wind shear and static stability to approximate the large eddies in the boundary layer (i.e. boundary layer scheme). In addition, in a large eddy simulation (cloud resolving), we assume the physical processes (i.e. condensation) are relatively fast compared to the lifecycle of cloud and turbulence. Thus, when we predict the "resolved scale systems" (e.g. those can be seen by the grid), the "unresovled scales" are parameterized. A good news is, most of time, these assumptions are quite robust (except some cases going through cross-scale gray zones). This indicates that if we want to forecast the large-scale systems, we don't need to do deterministic forecast for the small-scale systems. Insead, what we need is the statistics of these systems (i.e. parameterization).

Then...what is the assumption of FDT? FDT can ba regarded as a reduced-dimension dynamical model, in which the time series of normal modes are predicted. Typically, these normal modes are red in spectrum (i.e. Gaussian or linear), while high frequency systems (those can't be described by the normal mode) are white in spectrum (i.e. random process). Just like the physical parameterizations mentioned above, these high frequency systems are parameterized in FDT. Thus, what we really care is when the predictable signal (e.g. those normal modes) is greater than the stochastic process (i.e. unpredictable components).

Here, I use a very simple O.D.E as an example.

$
\begin{equation}
\frac{d}{dt}\mathbf x= \mathbf A \mathbf x+\mathbf\epsilon \label{eq:1}\tag{1}
\end{equation}
$

In equation $\ref{eq:1}$, $\mathbf{x}$ is the predictable components (i.e. the time series of normal modes), which has a dimension of $n\times1$ and $\epsilon$ is the sotchastic part (i.e. random white noise) with the same dimension as $\mathbf{x}$. If we drop the unpredictable part, we can have the general solution of $\ref{eq:1}$.

$
\begin{equation}
\mathbf{x}_{\tau}=e^{\mathbf A \tau} \mathbf{x}_{0} \label{eq:2}\tag{2}
\end{equation}
$

In equation $\eqref{eq:2}$, $e^{\mathbf A \tau}$ is so-called "propagator operator", which carries the information of $\mathbf{x}$ from $t=0$ to $t=\tau$ and has a dimension of $n\times n$. If you still remember the high school mathematics, we will find the $e^{\mathbf A \tau}$ matrix is just like the matrix we use for coordinate transform (i.e. Jacobian matrix). For example, you have a vector $(a,b)$ and you want to rotate this vector counter-clockwise for $\theta$ degrees, what does the formulation look like ?

$
\begin{equation}
\begin{bmatrix}
a'\\
b'
\end{bmatrix}=
\begin{bmatrix}
cos(\theta) & -sin(\theta) \\
sin(\theta) & cos(\theta)
\end{bmatrix}
\begin{bmatrix}
a\\
b
\end{bmatrix} \label{eq:3}\tag{3}
\end{equation}
$

$\mathbf{x}_{0}$ is just like the vector $(a,b)$ and $\mathbf{x}_{\tau}$ is similar to vector $(a',b')$. $e^{\mathbf A \tau}$ is the matrix we use for coordinate transform. Similar formulations can be found in a lot of places, such as stastistical test, fourier transform, Laplace transform...etc (I will write another discussion about these analogs).

Apparently, the key is to derive the propagator operator $e^{\mathbf A \tau}$, which enables us to develop this simplified model. I will provide details in the following part.

***
# Developing a simple climate model from observational data
To solve $e^{\mathbf A \tau}$ in $\eqref{eq:2}$ numerically, we can use inverse approach. (of course the forward approach is another option as well, which is computationally less expensive but also characterized by some drawbacks ). The first step to solve $e^{\mathbf A \tau}$ is, we multiply both sides of $\eqref{eq:2}$ by $\mathbf{x}_{0}^T$.

$
\begin{equation}
\mathbf{x}_{\tau}\mathbf{x}_{0}^{T}=e^{\mathbf A \tau} \mathbf{x}_{0} \mathbf{x}_{0}^{T}  \label{eq:4} \tag{4}
\end{equation}
$

then we can have

$
\begin{equation}
\mathbf{C}_\tau=e^{\mathbf A \tau} \mathbf{C}_0 \label{eq:5}\tag{5}
\end{equation}
$

where $\mathbf{x}_{\tau}\mathbf{x}_{0}^{T}=\mathbf{C}_\tau$ and $\mathbf{x}_{0} \mathbf{x}_{0}^{T}=\mathbf{C}_0$. $\mathbf{C}_\tau$ and  $\mathbf{C}_0$ are covariance matrices of $\mathbf{x}$ between (1)$t=\tau$ and $t=0$ and (2) $t=0$ and $t=0$. Now, we can find all three matrices in $\eqref{eq:5}$ have dimension of $n\times n$. This means we can iverse any matrix in $\eqref{eq:5}$ directly as long as the det is not 0. Luckily, using normal modes ensures that the det of $\mathbf{C}_0$ won't be 0. Thus, multiplying both sides of $\eqref{eq:5}$ by the inversed $\mathbf{C}_0 $, we can ultimately get $e^{\mathbf A \tau}$. Woohoo! (Looks easy right?).

$
\begin{equation}
\mathbf{C}_\tau \mathbf{C}_0^{-1}=e^{\mathbf A \tau}  \label{eq:6}\tag{6}
\end{equation}
$

***
# Limitation of Inverse modeling: $\tau$ test
Although the steps of driving $e^{\mathbf A \tau}$ look simple, there is still a very important criteria. Let's go back to the assumption at the very beginning. From $\eqref{eq:1}$ to $\eqref{eq:2}$, we drop $\epsilon$. The reason we can drop this term is that...

$
\begin{equation}
(\frac{d}{dt}\mathbf x)\mathbf x^{T}= \mathbf A \mathbf x\mathbf x^{T}+\mathbf\epsilon \mathbf x^{T}\label{eq:7} \tag{7}
\end{equation}
$

the last term of $\eqref{eq:7}$ is nearlly zero because $\epsilon$ is white in spectrum (you will only get mean 0 (or nearly zero) when you multiply a matrix with a white noise matrix). This indicates that there is no information of $\epsilon$ passed from fomer time step to next time step. On the other hand, everything left (i.e. the predictable part, $\mathbf{x}$) should be red. This assumption gives us the following way to test our data.

Specifically, we can examine if a dataset can be used in FDT by checking the diagonal elements of $\mathbf{C}_\tau \mathbf{C}_0^{-1}$ in $\eqref{eq:6}$. Because if the general solution in $\eqref{eq:4}$ hold for every $\tau$, we will only have one $\mathbf{A}$ matrix regardless of the value of $\tau$, which means...

$
\begin{equation}
tr(\mathbf A)=tr(\mathbf{ln}(\mathbf{C}_\tau \mathbf{C}_0^{-1})/\tau)=const
\label{eq:8}
\tag{8}
\end{equation}
$

If we find $\eqref{eq:8}$ doesn't hold for every $\tau$, we can't apply FDT to a given dataset.

```python

```


# Quantum Amplitude Amplification

Quantum amplitude amplification is an algorithm that amplifies the amplitude of a particular state in a superposition state.
It is the core of Grover's algorithm, which will be explained later.

## Explanation of algorithm

### Amplitude amplification
Regarding the word "amplitude", the coefficients for each state are called **amplitude** and the probability is called **probability amplitude**.

Let's look at the following equation.

$$
H \lvert 0 \rangle \otimes H \lvert 0 \rangle = \frac{1}{2} (\lvert 00 \rangle + \lvert 01 \rangle + \lvert10 \rangle + \lvert 11 \rangle) \xrightarrow{\text{Amplitude amplification}} \lvert 00 \rangle
$$

In this case, the probability amplitudes on the left side of the arrow are all $1/4$, while the state on the right side has probability amplitude $1$ for $00$.

Amplitude amplification, as the word implies, is an algorithm that amplifies this probability amplitude as shown above.

The following figure illustrates the calculations.

参考: https://en.wikipedia.org/wiki/Grover%27s_algorithm

First, let's explain the symbols.
$\lvert s\rangle$ is an arbitrary initial state.
For example, $\lvert s\rangle$ can be the superposition of all states. Such a initial state is used in Grover's algorithm, explained later.

$$
\lvert s \rangle = \otimes^n  H  \lvert 0 \rangle =\frac{1}{\sqrt{2^n}}\sum^{2^n}_{x\in \{0, 1\}^n}\lvert x \rangle
$$

Of these, $\lvert \omega\rangle$ is the state we want to amplify.

$$
\lvert \omega \rangle = \frac{1}{\sqrt{2^n}} \lvert 00...010...00\rangle
$$

Suppose that the $x$-th number contains a 1.

$\lvert s'\rangle$ is a vector of $\lvert s\rangle$ minus $\lvert \omega\rangle$.

$$
\lvert s' \rangle = \lvert s \rangle - \lvert \omega \rangle = \lvert \omega^{\perp} \rangle
$$

You can see that it is perpendicular to $\lvert \omega\rangle$.

$U_{\omega}$ is a matrix that inverts $\psi$ on the $\lvert s'\rangle$ axis.
In other words, rotate $\psi$ by $-\phi$.

$$
U_{\omega} \lvert \psi \rangle = \cos(-\phi) \tilde{\lvert s' \rangle} + \sin(-\phi) \tilde{\lvert \omega \rangle} = \cos(\phi) \tilde{\lvert s' \rangle} - \sin(\phi) \tilde{\lvert \omega \rangle}
$$

$U_s$ is a matrix that inverts $\psi$ around $\lvert s\rangle$.

$$
U_s \lvert \psi \rangle = \cos \bigg\{ \frac{\theta}{2} + \big(\frac{\theta}{2} - \phi \big) \bigg\} \tilde{\lvert s' \rangle} + \sin \bigg\{ \frac{\theta}{2} + \big(\frac{\theta}{2} - \phi \big) \bigg\}\tilde{\lvert \omega \rangle} = \cos(\theta - \phi)\tilde{\lvert s' \rangle} + \sin(\theta - \phi) \tilde{\lvert \omega \rangle}
$$

### Overview
The algorithm is outlined below.

1. Invert $\lvert s\rangle$ around $\lvert s' \rangle$ using $U_{\omega}$.

2. Invert $U_{\omega}\lvert s\rangle$ around $\lvert s\rangle$ using $U_s$.

We will explain the above process in detail.

#### 1. Invert around $\lvert s' \rangle$

Using $\theta$ as shown in the figure above, we describe $\lvert s \rangle$ as following.

$$
\lvert s \rangle = \cos\bigl(\frac{\theta}{2}\bigr) \tilde{\lvert s' \rangle} - \sin\bigl(\frac{\theta}{2}\bigr) \tilde{\lvert \omega \rangle}
$$

As a same time,

$$
\cos\bigl( \frac{\theta}{2} \bigr) = \sqrt{\frac{2^n-1}{2^n}},\ \ \ \  \sin\bigl( \frac{\theta}{2} \bigr) = \sqrt{\frac{1}{2^n}}
$$

Using $U_{\omega}$, fold $\lvert s \rangle$ around $\lvert s' \rangle$.
From the above figure, described as follows

$$
U_{\omega} \lvert s \rangle = \cos\bigl(-\frac{\theta}{2}\bigr)\tilde{\lvert s' \rangle} + \sin\bigl(-\frac{\theta}{2}\bigr)\tilde{\lvert \omega \rangle} = \cos\bigl(\frac{\theta}{2}\bigr)\tilde{\lvert s' \rangle} - \sin\bigl(\frac{\theta}{2}\bigr)\tilde{\lvert \omega \rangle}
$$

Since we are only working on $\lvert \omega \rangle$ for this operation, we can see that $U_{\omega}$ represents Oracle as described above.

#### 2. Invert around $\lvert s \rangle$
Using $U_s$, fold $U_{\omega}\lvert s \rangle$ around $\lvert s \rangle$.

$$
U_s U_{\omega} \lvert s\rangle = U_s\biggl( \cos\bigl(-\frac{\theta}{2}\bigr)\tilde{\lvert s' \rangle} + \sin\bigl(-\frac{\theta}{2}\bigr)\tilde{\lvert \omega \rangle}  \biggr)
$$

Since we need to rotate $2\theta$ here,

$$
U_s U_{\omega} \lvert s\rangle =  \cos\bigl(\frac{3}{2}\theta\bigr)\tilde{\lvert s' \rangle} + \sin\bigl(\frac{3}{2}\theta\bigr)\tilde{\lvert \omega \rangle}
$$

Specifically, $\cos$ and $\sin$ are obtained from the additive theorem.

$$
\cos \frac{3}{2}\theta = \bigl( 1-\frac{4}{2^n} \bigr) \sqrt{\frac{2^n-1}{2^n}},\ \ \ \ \sin \frac{3}{2}\theta = \bigl( 3-\frac{4}{2^n} \bigr) \sqrt{\frac{1}{2^n}}
$$

Thus, using $\lvert s' \rangle$, $\lvert \omega \rangle$, we get

$$
U_s U_{\omega} \lvert s\rangle =  \bigl(1 - \frac{4}{2^n}\bigr)\lvert s' \rangle + \bigl(3 - \frac{4}{2^n}\bigr)\lvert \omega \rangle
$$

This operation makes $\lvert \omega \rangle$ of the $2^n$ amplitudes about three times larger than the others.
We were able to amplify the amplitude by the above.

## Circuit

Let's implement it using blueqat.

Here we see 2qubits quantum amplitude amplification algorithm.
Combination of 2 binary numbers are 00, 01, 10 and 11. Now we want to amplify a specific state.
The state correponds to $\lvert \omega \rangle$.

#### 1. Invert $\lvert s \rangle$ around $\lvert s' \rangle$ using $U_{\omega}$.

First, we need to realize a gate $U_{\omega}$ that reverses the state on the axis $\lvert s' \rangle$ perpendicular to $\lvert \omega \rangle$.

To do it, we prepare a diagonal matrix which is one element is -1 and others are 1.
By using H gate, CZ gate and S gate we can realize it.

Now we see the circuit one by one. We can check the unitary matrix of the circuit by running .run_with_sympy_unitary(). And we usually start from the diagonal matrix of CZ gate first and changing it.

```python
from blueqat import Circuit
```

```python
'''
#marking on 11

-------*-----
-------Z-----
'''

Circuit(2).cz[0,1].run_with_sympy_unitary()
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & -1\end{matrix}\right]$

```python
'''
#marking on 01

----S--*--S---
-------Z-------
'''

Circuit(2).s[0].cz[0,1].s[0].run_with_sympy_unitary()
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & -1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
'''
#marking on 10

--------*------
----S--Z--S---
'''

Circuit(2).s[1].cz[0,1].s[1].run_with_sympy_unitary()
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & -1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python

'''
#00

----S--*--S--
----S--Z--S--
'''

Circuit(2).s[:].cz[0,1].s[:].run_with_sympy_unitary()
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & -1 & 0 & 0\\0 & 0 & -1 & 0\\0 & 0 & 0 & -1\end{matrix}\right]$

The result is inversed on the sign. We can think about global phase that flips all the minus sign to plus and plus sign to minus.

#### 2. Invert $U_{\omega}\lvert s \rangle$ around $\lvert s \rangle$ using $U_s$.

Here we think about $U_s$.

If we separate $\lvert s \rangle = \lvert s' \rangle + \lvert \omega \rangle$ from the definition of $\lvert s \rangle$, then $U_{\omega}$ is simply

$$
U_{\omega} (\lvert s' \rangle + \lvert \omega \rangle) = \lvert s' \rangle - \lvert \omega \rangle
$$

This means that we can use gates that change sign only for certain states, such as Z and CZ gates.

Thinking geometrically from the above figure, $U_s$ is described as follows.

$$
\begin{align}
U_s U_{\omega} \lvert s \rangle &= 2(\langle s \lvert U_{\omega} \rvert s \rangle \rvert s \rangle - U_{\omega}\lvert s \rangle) + U_{\omega}\lvert s \rangle \\
&= 2\lvert s \rangle \langle s \lvert U_{\omega} \rvert s \rangle - U_{\omega}\lvert s \rangle \\
&= (2\lvert s\rangle \langle s\rvert - I) U_{\omega}\lvert s \rangle
\end{align}
$$

So $U_s = 2\lvert s\rangle \langle s\rvert - I$.

Furthermore, in the case initial state $\lvert s \rangle = \otimes^n  H  \lvert 0 \rangle$, $U_s$ can be decomposed as follows.

$$
2\lvert s\rangle \langle s\rvert - I = 2H^{\otimes n}\lvert 0^n\rangle \langle 0^n\rvert H^{\otimes n} - I = H^{\otimes n} (2\lvert 0^n\rangle \langle 0^n\rvert - I) H^{\otimes n}\ \ \ \ (\lvert 0^n \rangle = \lvert 00\cdots 00 \rangle)
$$

Here, we can write $2\lvert 0^n\rangle \langle 0^n\rvert - I$ as

$$
2\lvert 0^n\rangle \langle 0^n\rvert - I =
    \begin{pmatrix}
      -1 & 0 & 0 & \ldots & 0 & 0 \\
      0 & 1 & 0 & \ldots & 0 & 0 \\
      0 & 0 & 1 & \ldots & 0 & 0 \\
      \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
      0 & 0 & 0 & \ldots & 1 & 0 \\
      0 & 0 & 0 & \ldots & 0 & 1
    \end{pmatrix}
$$

We can describe it as following.
$$
XZX =
    \begin{pmatrix}
      -1 & 0 \\
      0 & 1
    \end{pmatrix}
$$

$$
2\lvert 0^n\rangle \langle 0^n\rvert - I = X^{\otimes n}C^n ZX^{\otimes n}
$$

It can be implemented with blueqat.

```python
'''
--H-X-*-X-H--
--H-X-Z-X-H--
'''

Circuit(2).h[:].x[:].cz[0,1].x[:].h[:].run_with_sympy_unitary()
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & - \frac{1}{2} & - \frac{1}{2} & - \frac{1}{2}\\- \frac{1}{2} & \frac{1}{2} & - \frac{1}{2} & - \frac{1}{2}\\- \frac{1}{2} & - \frac{1}{2} & \frac{1}{2} & - \frac{1}{2}\\- \frac{1}{2} & - \frac{1}{2} & - \frac{1}{2} & \frac{1}{2}\end{matrix}\right]$

From the above, we were able to rewrite the amplitude amplification oracle with simple gate sets.

Now we have the total circuit.

```python
from blueqat import Circuit

#Amplitude amplification
a = Circuit(2).h[:].x[:].cz[0,1].x[:].h[:].m[:]

'''
#00 Circuit
--H--S--*--S----H-X-*-X-H--
--H--S--Z--S----H-X-Z-X-H--
'''

(Circuit(2).h[:].s[:].cz[0,1].s[:] + a).run(shots=100)
```

    Counter({'00': 100})

```python
'''
#01 Circuit
--H-----*-------H-X-*-X-H--
--H--S--Z--S---H-X-Z-X-H--
'''

(Circuit(2).h[:].s[1].cz[0,1].s[1] + a).run(shots=100)
```

    Counter({'01': 100})

```python
'''
#10 Circuit
--H--S--*--S----H-X-*-X-H--
--H-----Z--------H-X-Z-X-H--
'''
(Circuit(2).h[:].s[0].cz[0,1].s[0] + a).run(shots=100)
```

    Counter({'10': 100})

```python
'''
#11 Circuit
--H-----*-------H-X-*-X-H--
--H-----Z-------H-X-Z-X-H--
'''
(Circuit(2).h[:].cz[0,1] + a).run(shots=100)
```

    Counter({'11': 100})

From the above, we were able to amplify the probability of a particular state being measured.

# Grover's algorithm
Grover's algorithm is a search algorithm that finds out a data entry efficiently from a list of unstructured data.

This algorithm is based on [quantum amplitude amplification](./110_amplitude_amplification_en.ipynb).

The algorithm iterates through quantum amplitude amplification to bring the initial state $\lvert s\rangle$ closer to the state we are looking for $\lvert \omega\rangle$.

$$
\lvert s \rangle = \cos\bigl(\frac{\theta}{2}\bigr) \tilde{\lvert s' \rangle} - \sin\bigl(\frac{\theta}{2}\bigr) \tilde{\lvert \omega \rangle}
$$

From the above equation, if we perform amplitude amplification once, we get

$$
U_s U_{\omega} \lvert s\rangle =  \cos\bigl(\frac{3}{2}\theta\bigr)\tilde{\lvert s' \rangle} + \sin\bigl(\frac{3}{2}\theta\bigr)\tilde{\lvert \omega \rangle}
$$

Therefore, if we perform amplitude amplification $n$ times, we get

$$
\cos\bigl(\frac{1}{2}\theta + n\theta \bigr)\tilde{\lvert s' \rangle} + \sin\bigl(\frac{1}{2}\theta + n\theta\bigr)\tilde{\lvert \omega \rangle} = \cos\biggl\{\bigl(\frac{1}{2} + n \bigr)\theta \biggr\} \tilde{\lvert s' \rangle} + \sin\biggl\{\bigl(\frac{1}{2} + n \bigr)\theta \biggr\}\tilde{\lvert \omega \rangle}
$$

As the amplitude is amplified, the initial state $\lvert s\rangle$ in the above figure becomes closer and closer to $\lvert \omega\rangle$.
In other words, the probability amplitude of $\omega$ goes to 1.
However, if you keep repeating the process, it will just keep rotating, so you need to think about the number of times you want to repeat it.

Since we want to set the probability amplitude of $\omega$ to 1,

$$
\sin^2\biggl\{\bigl(\frac{1}{2} + k \bigr)\theta \biggr\} < 1 \xrightarrow{\sin >0}\bigl(\frac{1}{2} + k \bigr)\theta < \frac{\pi}{2} \Leftrightarrow k<\frac{\pi}{2\theta} - \frac{1}{2}
$$

From the above, the number of times the amplitude is amplified is determined and we can measure $\lvert \omega\rangle$ with high probability.

In the example implementation of amplitude amplification, the measurement probability of $\lvert \omega \rangle$ was set to 1 with one amplitude amplification.
In the initial state $\lvert s\rangle$, which is an equally superposition state of two qubits, $\theta = \frac{\pi}{3}$.
In this case, we can see from the above equation that a single amplitude amplification results in $U_s U_{\omega}\lvert s\rangle = \tilde{\lvert \omega \rangle}$.
Thus, we were effectively running Grover's algorithm.

```python

```