```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

```

```python
plt.style.use(['ggplot'])
```

# Create Data

 Generate some data with:
\begin{equation} \theta_0= 4 \end{equation}
\begin{equation} \theta_1= 3 \end{equation}

Add some Gaussian noise to the data

```python
X = 2 * np.random.rand(100,1)
y = 4 +3 * X+np.random.randn(100,1)
```

Let's plot our data to check the relation between X and Y

```python

plt.plot(X,y,'b.')
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
_ =plt.axis([0,2,0,15])
```

#  Analytical way of Linear Regression

```python
X_b = np.c_[np.ones((100,1)),X]
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
print(theta_best)
```

    [[3.87706422]
     [2.96335497]]

This is close to our real thetas 4 and 3. It cannot be accurate due to the noise I have introduced in data

```python
X_new = np.array([[0],[2]])
X_new_b = np.c_[np.ones((2,1)),X_new]
y_predict = X_new_b.dot(theta_best)
y_predict
```

    array([[3.87706422],
           [9.80377417]])

Let's plot prediction line with calculated:theta

```python
plt.plot(X_new,y_predict,'r-')
plt.plot(X,y,'b.')
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0,2,0,15])

```

# Gradient Descent

## Cost Function & Gradients

 The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have  to be derived from the cost functions

Cost
\begin{equation}
J(\theta) = 1/2m \sum_{i=1}^{m} (h(\theta)^{(i)} - y^{(i)})^2
\end{equation}

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_0 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_1 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_2 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_j -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}

```python

def  cal_cost(theta,X,y):
    '''

    Calculates the cost for given X and Y. The following shows and example of a single dimensional X
    theta = Vector of thetas
    X     = Row of X's np.zeros((2,j))
    y     = Actual y's np.zeros((2,1))

    where:
        j is the no of features
    '''

    m = len(y)

    predictions = X.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

```

```python
def gradient_descent(X,y,theta,learning_rate=0.01,iterations=100):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    theta_history = np.zeros((iterations,2))
    for it in range(iterations):

        prediction = np.dot(X,theta)

        theta = theta -(1/m)*learning_rate*( X.T.dot((prediction - y)))
        theta_history[it,:] =theta.T
        cost_history[it]  = cal_cost(theta,X,y)

    return theta, cost_history, theta_history

```

 Let's start with 1000 iterations and a learning rate of 0.01. Start with theta from a Gaussian distribution

```python
lr =0.01
n_iter = 1000

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history,theta_history = gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.635,
    Theta1:          3.172
    Final cost/MSE:  6584.880

 Let's plot the cost history over iterations

```python
fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('J(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),cost_history,'b.')
```

 After around 150 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. Let us zoom in till iteration 200 and see the curve

```python

fig,ax = plt.subplots(figsize=(10,8))
_=ax.plot(range(200),cost_history[:200],'b.')
```

It is worth while to note that the cost drops faster initially and then the gain in cost reduction is not as much

### It would be great to see the effect of different learning rates and iterations together

### Let us  build a function which can show the effects together and also show how gradient decent actually is working

```python

def plot_GD(n_iter,lr,ax,ax1=None):
     """
     n_iter = no of iterations
     lr = Learning Rate
     ax = Axis to plot the Gradient Descent
     ax1 = Axis to plot cost_history vs Iterations plot

     """
     _ = ax.plot(X,y,'b.')
     theta = np.random.randn(2,1)

     tr =0.1
     cost_history = np.zeros(n_iter)
     for i in range(n_iter):
        pred_prev = X_b.dot(theta)
        theta,h,_ = gradient_descent(X_b,y,theta,lr,1)
        pred = X_b.dot(theta)

        cost_history[i] = h[0]

        if ((i % 25 == 0) ):
            _ = ax.plot(X,pred,'r-',alpha=tr)
            if tr < 0.8:
                tr = tr+0.2
     if not ax1== None:
        _ = ax1.plot(range(n_iter),cost_history,'b.')
```

### Plot the graphs for different iterations and learning rates combination

```python
fig = plt.figure(figsize=(30,25),dpi=200)
fig.subplots_adjust(hspace=0.4, wspace=0.4)

it_lr =[(2000,0.001),(500,0.01),(200,0.05),(100,0.1)]
count =0
for n_iter, lr in it_lr:
    count += 1

    ax = fig.add_subplot(4, 2, count)
    count += 1

    ax1 = fig.add_subplot(4,2,count)

    ax.set_title("lr:{}".format(lr))
    ax1.set_title("Iterations:{}".format(n_iter))
    plot_GD(n_iter,lr,ax,ax1)

```

 See how useful it is to visualize the effect of learning rates and iterations on gradient descent. The red lines show how the gradient descent starts and then slowly gets closer to the final value

## You can always plot Indiviual graphs to zoom in

```python
_,ax = plt.subplots(figsize=(14,10))
plot_GD(100,0.1,ax)
```

# Stochastic Gradient Descent

```python
def stocashtic_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)

    for it in range(iterations):
        cost =0.0
        for i in range(m):
            rand_ind = np.random.randint(0,m)
            X_i = X[rand_ind,:].reshape(1,X.shape[1])
            y_i = y[rand_ind].reshape(1,1)
            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```python
lr =0.5
n_iter = 50

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history = stocashtic_gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.878,
    Theta1:          3.017
    Final cost/MSE:  57.764

```python
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```

# Mini Batch Gradient Descent

```python
def minibatch_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10,batch_size =20):
    '''
    X    = Matrix of X without added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    n_batches = int(m/batch_size)

    for it in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            X_i = X[i:i+batch_size]
            y_i = y[i:i+batch_size]

            X_i = np.c_[np.ones(len(X_i)),X_i]

            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```python
lr =0.1
n_iter = 200

theta = np.random.randn(2,1)

theta,cost_history = minibatch_gradient_descent(X,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.885,
    Theta1:          2.961
    Final cost/MSE:  1296.250

```python
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```

```python

```

Day 24 - Gradient Descent Manual/GradientDescent.ipynb


# Simulation of variance of OLS parameters
Recently, I've spend some time to understand what *variance* of parameters actually means. If I understand it correctly, the variance of an estimate refers to how much we expect our estimate ($\hat{\beta}$) to vary from the true population parameter ($\beta$). In OLS, two terms contribute to the variance estimate: the unexplained variance and the design variance.

Unexplained variance, also referred to as $\hat{\sigma}^{2}$, is defined as:

\begin{align}
\hat{\sigma}^{2} = \frac{\sum_{i=1}^{N} (y_{i} - \hat{y}_{i})^{2}}{N - K}
\end{align}

The design variance, for any contrast $c$, is defined as:

\begin{align}
desvar = c(X'X)^{-1}c'
\end{align}

As such, the variance of any contrast of beta-parameter(s) is defined as:

\begin{align}
\mathrm{var}[c\beta] = \frac{\sum_{i=1}^{N} (y_{i} - \hat{y}_{i})^{2}}{N - K} c(X'X)^{-1}c'
\end{align}

While mathematically, I understand *how* variance is computed. But it's conceptual meaning -- i.e. the expected squared deviation from the true population parameter -- is still a bit muddy for me. As such, I decided to simulate the process, which often helps me to understand statistical issues.

First, some imports.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Now, let's define an arbitrary *true* model of some dependent variable $y$. Let's assume the true function is:

\begin{align}
y = 10 + 5x
\end{align}

```python
x = np.arange(100)
y = 10 + 5 * x
plt.plot(x, y)
plt.title("True function of y")
plt.show()
```

Let's now add some random gaussian noise ($\mu = 0, \sigma=1$) to our dependent variable and estimate the parameter of the intercept and $x$ using linear regression. We'll also calculate the variance of this estimate (using the contrast-vector `[1, 0]` for the intercept-parameter and `[0, 1]` for the $x$-parameter). We'll iterate this process 10,000 times and keep track of the parameter-estimates and their variance-estimates.

```python
iters = 50000
betas = np.zeros((iters, 2))
vars = np.zeros((iters, 2))
c = np.array([[1, 0], [0, 1]])
X = np.hstack((np.ones((100, 1)), x[:, np.newaxis]))

for i in range(iters):
    y2 = y + np.random.normal(0, 1, 100) * 100
    betas[i, :] = np.linalg.lstsq(X, y2)[0]
    y_hat = X.dot(betas[-1, :])
    sigma_hat = np.sum((y_hat - y2) ** 2) / (X.shape[0] - X.shape[1])
    desvar = c.dot(np.linalg.pinv(X.T.dot(X))).dot(c.T)
    vars[i, :] = np.diag(sigma_hat * desvar)
```

Let's plot the histograms of the two parameter-estimates:

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.hist(betas[:, 0], bins=50)
plt.title(r"$\hat{\beta}_{0}$", fontsize=20)

plt.subplot(1, 2, 2)
plt.hist(betas[:, 1], bins=50)
plt.title(r"$\hat{\beta}_{x}$", fontsize=20)

plt.show()
```

Now, the central limit theorem (CTL) predicts that the mean of this distribution should approximate the true parameters (I think). Let's check that:

```python
betas.mean(axis=0)
```

    array([ 10.10447338,   4.99799692])

Now, let's calculate the variance of our parameter-estimates across the simulations:

```python
betas.var(axis=0)
```

    array([  3.94466169e+02,   1.19993340e-01])

This estimate from our simulation should then (I think) approximate the average variance estimate across simulations:

```python
vars.mean(axis=0)
```

    array([  3.90509682e+03,   1.18930922e+00])

... which seems indeed the case!

```python
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
plt.hist(vars[:, 0], bins=50)
plt.title(r"$\hat{\mathrm{var}[\beta}_{0}]$", fontsize=20)

plt.subplot(1, 2, 2)
plt.hist(vars[:, 1], bins=50)
plt.title(r"$\hat{\mathrm{var}[\beta}_{x}]$", fontsize=20)

plt.show()
```


# Z-Score

A z-score measures how many standard deviations above or below the mean a data point is.

>
\begin{align}
z = \frac{data\ point - mean}{standard\ deviation} \\
\end{align}

## Important facts

- A positive z-score says the data point is above average.
- A negative z-score says the data point is below average.
- A z-score close to 0 says the data point is close to average.

__Source:__ [Khan Academy](https://www.khanacademy.org/math/statistics-probability/modeling-distributions-of-data/z-scores/a/z-scores-review)

## Code

### Packages

```python
import numpy as np
```

### Function

```python
def z_score(sample, data_point):
    # data point needs to come from the sample
    assert data_point in sample
    avg = np.mean(sample)
    sd = np.std(sample)
    z_score = (data_point - avg)/sd
    return z_score
```

### Input

```python
sample = np.random.randn(30)
sample
```

    array([-0.59877513, -0.51223475, -1.40174708, -1.86207935, -0.10035927,
            0.12655593,  0.32697515, -0.35919666, -1.865143  , -0.27180892,
            0.42933215, -0.44281586, -0.30340738,  1.17923063, -0.09077022,
           -0.60174546,  0.34032583, -0.58059233, -1.90742451, -1.00759512,
            0.73630683, -0.13672296,  1.54279119, -0.4798372 ,  1.41396989,
            0.87904765, -0.9446896 ,  1.02256431,  0.77302047, -0.15333515])

```python
# grab the first data point
first_data_point = sample[0]
first_data_point
```

    -0.59877512996891891

### Output

```python
z_score(sample, first_data_point)
```

    -0.4795330537549915


# Package

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import yfinance as yf
import yahoofinancials
from IPython.core.display import HTML
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
pd.options.plotting.backend = 'plotly'
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# Random Walk

Perhatikan suatu partikel bergerak sepanjang garis, berangkat dari titik 0. Setiap satu satuan waktu partikel bergerak ke kiri atau ke kanan sejauh satu satuan jarak, dengan peluang ke kiri atau ke kanan diberikan oleh

\begin{equation}
p_{i,i+1} = \frac{1}{2} = p_{i,i-1}, \quad i = 0, \pm 1, \pm 2, \dots
\end{equation}

Selanjutnya proses dipercepat dengan mengambil langkah yang semakin pendek dan juga selang waktu yang semakin pendek. Untuk tiap satuan waktu $\Delta t$ melangkah ke kiri atau ke kanan sejauh $\Delta x$, dengan peluang yang sama besar.

Misalkan $X(t)$ posisi partikel pada saat $t$ dengan $t = n \Delta t (n=0,1,2,\dots)$, maka
\begin{equation}
X(t) = 0 + X_1 \Delta x + X_2 \Delta x + \dots + X_n \Delta x
\end{equation}

```python
n  = 500 # Banyak Langkah
sigma = 1 # Volatilitas
T = 3 # Total Waktu
dt = T/n # Delta t
dx = sigma*dt**0.5 # Delta x
t_i = np.linspace(0, T, n) # Diskritisasi dari t
X_i = 2*(np.random.rand(1,n)<0.5) - 1 # Nilai x_1 adalah 1 atau -1
X = dx*np.cumsum(X_i)

plt.plot(t_i, X)
plt.grid(True)
plt.xlabel('t')
plt.ylabel('X(t): posisi Partikel')
plt.title('Simulasi Perjalanan Acak Vertikel Simetri 1 Dimensi')
plt.show()
```

# Peubah Acak Log Normal

```python
x = np.linspace(0.01,4,500)
mu = 0.05
sigma = 0.3
a = ((np.log(x)-mu)**2)/(2*sigma**2)
b = x*sigma*(2*np.pi)**0.5
y1 = np.exp(-a)/b

sigma = 0.5
a = ((np.log(x)-mu)**2)/(2*sigma**2)
b = x*sigma*(2*np.pi)**0.5
y2 = np.exp(-a)/b

plt.plot(x,y1, 'r-')
plt.plot(x,y2, 'b:')
plt.ylim([0, 1.5])
plt.xlabel('x')
plt.ylabel('f(x)')
plt.legend([r'$\sigma = 0.3$', r'$\sigma = 0.5$' ])
plt.title(r'Grafik Fungsi Padat Peluang Log Normal , $\mu$ = 0.05')
plt.show()
```

# Pemanfaatan pada model harga saham

```python
S = 6987
mu = 0.3
sigma = 0.3
L = 252
T = 1
dt = T/L
M = 1
```


# Factorization
Factorization is the process of restating an expression as the *product* of two expressions (in other words, expressions multiplied together).

For example, you can make the value **16** by performing the following multiplications of integer numbers:
- 1 x 16
- 2 x 8
- 4 x 4

Another way of saying this is that 1, 2, 4, 8, and 16 are all factors of 16.

## Factors of Polynomial Expressions
We can apply the same logic to polynomial expressions. For example, consider the following monomial expression:

\begin{equation}-6x^{2}y^{3} \end{equation}

You can get this value by performing the following multiplication:

\begin{equation}(2xy^{2})(-3xy) \end{equation}

Run the following Python code to test this with arbitrary ***x*** and ***y*** values:

```python
from random import randint
x = randint(1,100)
y = randint(1,100)

(2*x*y**2)*(-3*x*y) == -6*x**2*y**3
```

So, we can say that **2xy2** and **-3xy** are both factors of **-6x2y3**.

This also applies to polynomials with more than one term. For example, consider the following expression:

\begin{equation}(x + 2)(2x^{2} - 3y + 2) = 2x^{3} + 4x^{2} - 3xy + 2x - 6y + 4 \end{equation}

Based on this, **x+2** and **2x2 - 3y + 2** are both factors of **2x3 + 4x2 - 3xy + 2x - 6y + 4**.

(and if you don't believe me, you can try this with random values for x and y with the following Python code):

```python
from random import randint
x = randint(1,100)
y = randint(1,100)

(x + 2)*(2*x**2 - 3*y + 2) == 2*x**3 + 4*x**2 - 3*x*y + 2*x - 6*y + 4
```

## Greatest Common Factor
Of course, these may not be the only factors of **-6x2y3**, just as 8 and 2 are not the only factors of 16.

Additionally, 2 and 8 aren't just factors of 16; they're factors of other numbers too - for example, they're both factors of 24 (because 2 x 12 = 24 and 8 x 3 = 24). Which leads us to the question, what is the highest number that is a factor of both 16 and 24? Well, let's look at all the numbers that multiply evenly into 12 and all the numbers that multiply evenly into 24:

|   16   |   24   |
|--------|--------|
| 1 x 16 | 1 x 24 |
| 2 x **8**  | 2 x 12 |
|        | 3 x **8**  |
| 4 x 4  | 4 x 6  |

The highest value that is a multiple of both 16 and 24 is **8**, so 8 is the *Greatest Common Factor* (or GCF) of 16 and 24.

OK, let's apply that logic to the following expressions:

\begin{equation}15x^{2}y\;\;\;\;\;\;\;\;9xy^{3}\end{equation}

So what's the greatest common factor of these two expressions?

It helps to break the expressions into their consitituent components. Let's deal with the coefficients first; we have 15 and 9. The highest value that divides evenly into both of these is **3** (3 x 5 = 15 and 3 x 3 = 9).

Now let's look at the ***x*** terms; we have x2 and x. The highest value that divides evenly into both is these is **x** (*x* goes into *x* once and into *x*2 *x* times).

Finally, for our ***y*** terms, we have y and y3. The highest value that divides evenly into both is these is **y** (*y* goes into *y* once and into *y*3 *y•y* times).

Putting all of that together, the GCF of both of our expression is:

\begin{equation}3xy\end{equation}

An easy shortcut to identifying the GCF of an expression that includes variables with exponentials is that it will always consist of:
- The *largest* numeric factor of the numeric coefficients in the polynomial expressions (in this case 3)
- The *smallest* exponential of each variable (in this case, x and y, which technically are x1 and y1.

You can check your answer by dividing the original expressions by the GCF to find the coefficent expressions for the GCF (in other words, how many times the GCF divides into the original expression). The result, when multiplied by the GCF will always produce the original expression. So in this case, we need to perform the following divisions:

\begin{equation}\frac{15x^{2}y}{3xy}\;\;\;\;\;\;\;\;\frac{9xy^{3}}{3xy}\end{equation}

These fractions simplify to **5x** and **3y2**, giving us the following calculations to prove our factorization:

\begin{equation}3xy(5x) = 15x^{2}y\end{equation}
\begin{equation}3xy(3y^{2}) = 9xy^{3}\end{equation}

Let's try both of those in Python:

```python
from random import randint
x = randint(1,100)
y = randint(1,100)

print((3*x*y)*(5*x) == 15*x**2*y)
print((3*x*y)*(3*y**2) == 9*x*y**3)
```

## Distributing Factors
Let's look at another example. Here is a binomial expression:

\begin{equation}6x + 15y \end{equation}

To factor this, we need to find an expression that divides equally into both of these expressions. In this case, we can use **3** to factor the coefficents, because 3 • 2x = 6x and 3• 5y = 15y, so we can write our original expression as:

\begin{equation}6x + 15y = 3(2x) + 3(5y) \end{equation}

Now, remember the distributive property? It enables us to multiply each term of an expression by the same factor to calculate the product of the expression multiplied by the factor. We can *factor-out* the common factor in this expression to distribute it like this:

\begin{equation}6x + 15y = 3(2x) + 3(5y) = \mathbf{3(2x + 5y)} \end{equation}

Let's prove to ourselves that these all evaluate to the same thing:

```python
from random import randint
x = randint(1,100)
y = randint(1,100)

(6*x + 15*y) == (3*(2*x) + 3*(5*y)) == (3*(2*x + 5*y))
```

For something a little more complex, let's return to our previous example. Suppose we want to add our original 15x2y and 9xy3 expressions:

\begin{equation}15x^{2}y + 9xy^{3}\end{equation}

We've already calculated the common factor, so we know that:

\begin{equation}3xy(5x) = 15x^{2}y\end{equation}
\begin{equation}3xy(3y^{2}) = 9xy^{3}\end{equation}

Now we can factor-out the common factor to produce a single expression:

\begin{equation}15x^{2}y + 9xy^{3} = \mathbf{3xy(5x + 3y^{2})}\end{equation}

And here's the Python test code:

```python
from random import randint
x = randint(1,100)
y = randint(1,100)

(15*x**2*y + 9*x*y**3) == (3*x*y*(5*x + 3*y**2))
```

So you might be wondering what's so great about being able to distribute the common factor like this. The answer is that it can often be useful to apply a common factor to multiple terms in order to solve seemingly complex problems.

For example, consider this:

\begin{equation}x^{2} + y^{2} + z^{2} = 127\end{equation}

Now solve this equation:

\begin{equation}a = 5x^{2} + 5y^{2} + 5z^{2}\end{equation}

At first glance, this seems tricky because there are three unknown variables, and even though we know that their squares add up to 127, we don't know their individual values. However, we can distribute the common factor and apply what we *do* know. Let's restate the problem like this:

\begin{equation}a = 5(x^{2} + y^{2} + z^{2})\end{equation}

Now it becomes easier to solve, because we know that the expression in parenthesis is equal to 127, so actually our equation is:

\begin{equation}a = 5(127)\end{equation}

So ***a*** is 5 times 127, which is 635

## Formulae for Factoring Squares
There are some useful ways that you can employ factoring to deal with expressions that contain squared values (that is, values with an exponential of 2).

### Differences of Squares
Consider the following expression:

\begin{equation}x^{2} - 9\end{equation}

The constant *9* is 32, so we could rewrite this as:

\begin{equation}x^{2} - 3^{2}\end{equation}

Whenever you need to subtract one squared term from another, you can use an approach called the *difference of squares*, whereby we can factor *a2 - b2* as *(a - b)(a + b)*; so we can rewrite the expression as:

\begin{equation}(x - 3)(x + 3)\end{equation}

Run the code below to check this:

```python
from random import randint
x = randint(1,100)

(x**2 - 9) == (x - 3)*(x + 3)
```

### Perfect Squares
A *perfect square* is a number multiplied by itself, for example 3 multipled by 3 is 9, so 9 is a perfect square.

When working with equations, the ability to factor between polynomial expressions and binomial perfect square expressions can be a useful tool. For example, consider this expression:

\begin{equation}x^{2} + 10x + 25\end{equation}

We can use 5 as a common factor to rewrite this as:

\begin{equation}(x + 5)(x + 5)\end{equation}

So what happened here?

Well, first we found a common factor for our coefficients: 5 goes into 10 twice and into 25 five times (in other words, squared). Then we just expressed this factoring as a multiplication of two identical binomials *(x + 5)(x + 5)*.

Remember the rule for multiplication of polynomials is to multiple each term in the first polynomial by each term in the second polynomial and then add the results; so you can do this to verify the factorization:

- x • x = x2
- x • 5 = 5x
- 5 • x = 5x
- 5 • 5 = 25

When you combine the two 5x terms we get back to our original expression of x2 + 10x + 25.

Now we have an expression multipled by itself; in other words, a perfect square. We can therefore rewrite this as:

\begin{equation}(x + 5)^{2}\end{equation}

Factorization of perfect squares is a useful technique, as you'll see when we start to tackle quadratic equations in the next section. In fact, it's so useful that it's worth memorizing its formula:

\begin{equation}(a + b)^{2} = a^{2} + b^{2}+ 2ab \end{equation}

In our example, the *a* terms is ***x*** and the *b* terms is ***5***, and in standard form, our equation  *x2 + 10x + 25* is actually *a2 +  2ab + b2*. The operations are all additions, so the order isn't actually important!

Run the following code with random values for *a* and *b* to verify that the formula works:

```python
from random import randint
a = randint(1,100)
b = randint(1,100)

a**2 + b**2 + (2*a*b) == (a + b)**2
```

MathsToML/Module01-Equations, Graphs, and Functions/01-06-Factorization.ipynb


# Simulating a Yo-Yo

*Modeling and Simulation in Python*

Copyright 2021 Allen Downey

License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)

```python
# install Pint if necessary

try:
    import pint
except ImportError:
    !pip install pint
```

```python
# download modsim.py if necessary

from os.path import exists

filename = 'modsim.py'
if not exists(filename):
    from urllib.request import urlretrieve
    url = 'https://raw.githubusercontent.com/AllenDowney/ModSim/main/'
    local, _ = urlretrieve(url+filename, filename)
    print('Downloaded ' + local)
```

```python
# import functions from modsim

from modsim import *
```

## Yo-yo

Suppose you are holding a yo-yo with a length of string wound around its axle, and you drop it while holding the end of the string stationary. As gravity accelerates the yo-yo downward, tension in the string exerts a force upward. Since this force acts on a point offset from the center of mass, it exerts a torque that causes the yo-yo to spin.

The following diagram shows the forces on the yo-yo and the resulting torque. The outer shaded area shows the body of the yo-yo. The inner shaded area shows the rolled up string, the radius of which changes as the yo-yo unrolls.

In this system, we can't figure out the linear and angular acceleration independently; we have to solve a system of equations:

$\sum F = m a $

$\sum \tau = I \alpha$

where the summations indicate that we are adding up forces and torques.

As in the previous examples, linear and angular velocity are related because of the way the string unrolls:

$\frac{dy}{dt} = -r \frac{d \theta}{dt} $

In this example, the linear and angular accelerations have opposite sign.  As the yo-yo rotates counter-clockwise, $\theta$ increases and $y$, which is the length of the rolled part of the string, decreases.

Taking the derivative of both sides yields a similar relationship between linear and angular acceleration:

$\frac{d^2 y}{dt^2} = -r \frac{d^2 \theta}{dt^2} $

Which we can write more concisely:

$ a = -r \alpha $

This relationship is not a general law of nature; it is specific to scenarios like this where there is rolling without stretching or slipping.

Because of the way we've set up the problem, $y$ actually has two meanings: it represents the length of the rolled string and the height of the yo-yo, which decreases as the yo-yo falls.  Similarly, $a$ represents acceleration in the length of the rolled string and the height of the yo-yo.

We can compute the acceleration of the yo-yo by adding up the linear forces:

$\sum F = T - mg = ma $

Where $T$ is positive because the tension force points up, and $mg$ is negative because gravity points down.

Because gravity acts on the center of mass, it creates no torque, so the only torque is due to tension:

$\sum \tau = T r = I \alpha $

Positive (upward) tension yields positive (counter-clockwise) angular acceleration.

Now we have three equations in three unknowns, $T$, $a$, and $\alpha$, with $I$, $m$, $g$, and $r$ as known parameters.  We could solve these equations by hand, but we can also get SymPy to do it for us.

```python
from sympy import symbols, Eq, solve

T, a, alpha, I, m, g, r = symbols('T a alpha I m g r')
```

```python
eq1 = Eq(a, -r * alpha)
eq1
```

```python
eq2 = Eq(T - m * g, m * a)
eq2
```

```python
eq3 = Eq(T * r, I * alpha)
eq3
```

```python
soln = solve([eq1, eq2, eq3], [T, a, alpha])
```

```python
soln[T]
```

```python
soln[a]
```

```python
soln[alpha]
```

The results are

$T      = m g I / I^*   $

$a      = -m g r^2 / I^* $

$\alpha = m g r / I^*    $

where $I^*$ is the augmented moment of inertia, $I + m r^2$.

You can also see [the derivation of these equations in this video](https://www.youtube.com/watch?v=chC7xVDKl4Q).

We can use these equations for $a$ and $\alpha$ to write a slope function and simulate this system.

**Exercise:**  Simulate the descent of a yo-yo.  How long does it take to reach the end of the string?

Here are the system parameters:

```python
Rmin = 8e-3     # m
Rmax = 16e-3    # m
Rout = 35e-3    # m
mass = 50e-3    # kg
L = 1           # m
g = 9.8         # m / s**2
```

* `Rmin` is the radius of the axle.  `Rmax` is the radius of the axle plus rolled string.

* `Rout` is the radius of the yo-yo body.  `mass` is the total mass of the yo-yo, ignoring the string.

* `L` is the length of the string.

* `g` is the acceleration of gravity.

```python
1 / (Rmax)
```

Based on these parameters, we can compute the moment of inertia for the yo-yo, modeling it as a solid cylinder with uniform density ([see here](https://en.wikipedia.org/wiki/List_of_moments_of_inertia)).

In reality, the distribution of weight in a yo-yo is often designed to achieve desired effects.  But we'll keep it simple.

```python
I = mass * Rout**2 / 2
I
```

And we can compute `k`, which is the constant that determines how the radius of the spooled string decreases as it unwinds.

```python
k = (Rmax**2 - Rmin**2) / 2 / L
k
```

The state variables we'll use are angle, `theta`, angular velocity, `omega`, the length of the spooled string, `y`, and the linear velocity of the yo-yo, `v`.

Here is a `State` object with the the initial conditions.

```python
init = State(theta=0, omega=0, y=L, v=0)
```

And here's a `System` object with `init` and `t_end` (chosen to be longer than I expect for the yo-yo to drop 1 m).

```python
system = System(init=init, t_end=2)
```

Write a slope function for this system, using these results from the book:

$ r = \sqrt{2 k y + R_{min}^2} $

$ T      = m g I / I^*  $

$ a      = -m g r^2 / I^* $

$ \alpha  = m g r / I^*  $

where $I^*$ is the augmented moment of inertia, $I + m r^2$.

```python
# Solution goes here
```

Test your slope function with the initial conditions.
The results should be approximately

```
0, 180.5, 0, -2.9
```

```python
# Solution goes here
```

Notice that the initial acceleration is substantially smaller than `g` because the yo-yo has to start spinning before it can fall.

Write an event function that will stop the simulation when `y` is 0.

```python
# Solution goes here
```

Test your event function:

```python
# Solution goes here
```

Then run the simulation.

```python
# Solution goes here
```

Check the final state.  If things have gone according to plan, the final value of `y` should be close to 0.

```python
# Solution goes here
```

How long does it take for the yo-yo to fall 1 m?  Does the answer seem reasonable?

The following cells plot the results.

`theta` should increase and accelerate.

```python
results.theta.plot(color='C0', label='theta')
decorate(xlabel='Time (s)',
         ylabel='Angle (rad)')
```

`y` should decrease and accelerate down.

```python
results.y.plot(color='C1', label='y')

decorate(xlabel='Time (s)',
         ylabel='Length (m)')

```

Plot velocity as a function of time; is the acceleration constant?

```python
results.v.plot(label='velocity', color='C3')

decorate(xlabel='Time (s)',
         ylabel='Velocity (m/s)')
```

We can use `gradient` to estimate the derivative of `v`.  How does the acceleration of the yo-yo compare to `g`?

```python
a = gradient(results.v)
a.plot(label='acceleration', color='C4')
decorate(xlabel='Time (s)',
         ylabel='Acceleration (m/$s^2$)')
```

And we can use the formula for `r` to plot the radius of the spooled thread over time.

```python
r = np.sqrt(2*k*results.y + Rmin**2)
r.plot(label='radius')

decorate(xlabel='Time (s)',
         ylabel='Radius of spooled thread (m)')
```

```python

```


# Summary of Quantum Operations

```python
# Useful additional packages
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
from math import pi
```

```python
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister, execute
from qiskit.tools.visualization import circuit_drawer
from qiskit.quantum_info import state_fidelity
from qiskit import BasicAer

backend = BasicAer.get_backend('unitary_simulator')
```

## Multi-Qubit Gates

### Mathematical Preliminaries

The space of quantum computer grows exponential with the number of qubits. For $n$ qubits the complex vector space has dimensions $d=2^n$. To describe states of a multi-qubit system, the tensor product is used to "glue together" operators and basis vectors.

Let's start by considering a 2-qubit system. Given two operators $A$ and $B$ that each act on one qubit, the joint operator $A \otimes B$ acting on two qubits is

$$\begin{equation}
	A\otimes B =
	\begin{pmatrix}
		A_{00} \begin{pmatrix}
			B_{00} & B_{01} \\
			B_{10} & B_{11}
		\end{pmatrix} & A_{01} 	\begin{pmatrix}
				B_{00} & B_{01} \\
				B_{10} & B_{11}
			\end{pmatrix} \\
		A_{10} 	\begin{pmatrix}
					B_{00} & B_{01} \\
					B_{10} & B_{11}
				\end{pmatrix} & A_{11} 	\begin{pmatrix}
							B_{00} & B_{01} \\
							B_{10} & B_{11}
						\end{pmatrix}
	\end{pmatrix},
\end{equation}$$

where $A_{jk}$ and $B_{lm}$ are the matrix elements of $A$ and $B$, respectively.

Analogously, the basis vectors for the 2-qubit system are formed using the tensor product of basis vectors for a single qubit:
$$\begin{equation}\begin{split}
	\left|{00}\right\rangle &= \begin{pmatrix}
		1 \begin{pmatrix}
			1  \\
			0
		\end{pmatrix} \\
		0 \begin{pmatrix}
			1  \\
			0
		\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 1 \\ 0 \\ 0 \\0 \end{pmatrix}~~~\left|{01}\right\rangle = \begin{pmatrix}
	1 \begin{pmatrix}
	0 \\
	1
	\end{pmatrix} \\
	0 \begin{pmatrix}
	0  \\
	1
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix}0 \\ 1 \\ 0 \\ 0 \end{pmatrix}\end{split}
\end{equation}$$

$$\begin{equation}\begin{split}\left|{10}\right\rangle = \begin{pmatrix}
	0\begin{pmatrix}
	1  \\
	0
	\end{pmatrix} \\
	1\begin{pmatrix}
	1 \\
	0
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}~~~ 	\left|{11}\right\rangle = \begin{pmatrix}
	0 \begin{pmatrix}
	0  \\
	1
	\end{pmatrix} \\
	1\begin{pmatrix}
	0  \\
	1
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \\1 \end{pmatrix}\end{split}
\end{equation}.$$

Note we've introduced a shorthand for the tensor product of basis vectors, wherein $\left|0\right\rangle \otimes \left|0\right\rangle$ is written as $\left|00\right\rangle$. The state of an $n$-qubit system can described using the $n$-fold tensor product of single-qubit basis vectors. Notice that the basis vectors for a 2-qubit system are 4-dimensional; in general, the basis vectors of an $n$-qubit sytsem are $2^{n}$-dimensional, as noted earlier.

### Basis vector ordering in Qiskit

Within the physics community, the qubits of a multi-qubit systems are typically ordered with the first qubit on the left-most side of the tensor product and the last qubit on the right-most side. For instance, if the first qubit is in state $\left|0\right\rangle$ and second is in state $\left|1\right\rangle$, their joint state would be $\left|01\right\rangle$. Qiskit uses a slightly different ordering of the qubits, in which the qubits are represented from the most significant bit (MSB) on the left to the least significant bit (LSB) on the right (big-endian). This is similar to bitstring representation on classical computers, and enables easy conversion from bitstrings to integers after measurements are performed. For the example just given, the joint state would be represented as $\left|10\right\rangle$. Importantly, _this change in the representation of multi-qubit states affects the way multi-qubit gates are represented in Qiskit_, as discussed below.

The representation used in Qiskit enumerates the basis vectors in increasing order of the integers they represent. For instance, the basis vectors for a 2-qubit system would be ordered as $\left|00\right\rangle$, $\left|01\right\rangle$, $\left|10\right\rangle$, and $\left|11\right\rangle$. Thinking of the basis vectors as bit strings, they encode the integers 0,1,2 and 3, respectively.

### Controlled operations on qubits

A common multi-qubit gate involves the application of a gate to one qubit, conditioned on the state of another qubit. For instance, we might want to flip the state of the second qubit when the first qubit is in $\left|0\right\rangle$. Such gates are known as _controlled gates_. The standard multi-qubit gates consist of two-qubit gates and three-qubit gates. The two-qubit gates are:
- controlled Pauli gates
- controlled Hadamard gate
- controlled rotation gates
- controlled phase gate
- controlled u3 gate
- swap gate

The three-qubit gates are:
- Toffoli gate
- Fredkin gate

## Two-qubit gates

Most of the two-gates are of the controlled type (the SWAP gate being the exception). In general, a controlled two-qubit gate $C_{U}$ acts to apply the single-qubit unitary $U$ to the second qubit when the state of the first qubit is in $\left|1\right\rangle$. Suppose $U$ has a matrix representation

$$U = \begin{pmatrix} u_{00} & u_{01} \\ u_{10} & u_{11}\end{pmatrix}.$$

We can work out the action of $C_{U}$ as follows. Recall that the basis vectors for a two-qubit system are ordered as $\left|00\right\rangle, \left|01\right\rangle, \left|10\right\rangle, \left|11\right\rangle$. Suppose the **control qubit** is **qubit 0** (which, according to Qiskit's convention, is one the _right-hand_ side of the tensor product). If the control qubit is in $\left|1\right\rangle$, $U$ should be applied to the **target** (qubit 1, on the _left-hand_ side of the tensor product). Therefore, under the action of $C_{U}$, the basis vectors are transformed according to

$$\begin{align*}
C_{U}: \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle} &\rightarrow \underset{\text{qubit}~1}{U\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle} &\rightarrow \underset{\text{qubit}~1}{U\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle}\\
\end{align*}.$$

In matrix form, the action of $C_{U}$ is

$$\begin{equation}
	C_U = \begin{pmatrix}
	1 & 0 & 0 & 0 \\
	0 & u_{00} & 0 & u_{01} \\
	0 & 0 & 1 & 0 \\
	0 & u_{10} &0 & u_{11}
		\end{pmatrix}.
\end{equation}$$

To work out these matrix elements, let

$$C_{(jk), (lm)} = \left(\underset{\text{qubit}~1}{\left\langle j \right|} \otimes \underset{\text{qubit}~0}{\left\langle k \right|}\right) C_{U} \left(\underset{\text{qubit}~1}{\left| l \right\rangle} \otimes \underset{\text{qubit}~0}{\left| k \right\rangle}\right),$$

compute the action of $C_{U}$ (given above), and compute the inner products.

As shown in the examples below, this operation is implemented in Qiskit as `cU(q[0],q[1])`.

If **qubit 1 is the control and qubit 0 is the target**, then the basis vectors are transformed according to
$$\begin{align*}
C_{U}: \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|0\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|0\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{U\left|0\right\rangle}\\
C_{U}: \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{\left|1\right\rangle} &\rightarrow \underset{\text{qubit}~1}{\left|1\right\rangle}\otimes \underset{\text{qubit}~0}{U\left|1\right\rangle}\\
\end{align*},$$

which implies the matrix form of $C_{U}$ is
$$\begin{equation}
	C_U = \begin{pmatrix}
	1 & 0 & 0  & 0 \\
	0 & 1 & 0 & 0 \\
	0 & 0 & u_{00} & u_{01} \\
	0 & 0 & u_{10} & u_{11}
		\end{pmatrix}.
\end{equation}$$

```python
q = QuantumRegister(2)
```

### Controlled Pauli Gates

#### Controlled-X (or, controlled-NOT) gate
The controlled-not gate flips the `target` qubit when the control qubit is in the state $\left|1\right\rangle$. If we take the MSB as the control qubit (e.g. `cx(q[1],q[0])`), then the matrix would look like

$$
C_X =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 1 & 0
\end{pmatrix}.
$$

However, when the LSB is the control qubit, (e.g. `cx(q[0],q[1])`), this gate is equivalent to the following matrix:

$$
C_X =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 1 & 0\\
0 & 1 & 0 & 0
\end{pmatrix}.
$$

```python
qc = QuantumCircuit(q)
qc.cx(q[0],q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.+0.j, 1.+0.j],
           [0.+0.j, 0.+0.j, 1.+0.j, 0.+0.j],
           [0.+0.j, 1.+0.j, 0.+0.j, 0.+0.j]])

#### Controlled $Y$ gate

Apply the $Y$ gate to the target qubit if the control qubit is the MSB

$$
C_Y =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & -i\\
0 & 0 & i & 0
\end{pmatrix},
$$

or when the LSB is the control

$$
C_Y =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 0 & 0 & -i\\
0 & 0 & 1 & 0\\
0 & i & 0 & 0
\end{pmatrix}.
$$

```python
qc = QuantumCircuit(q)
qc.cy(q[0],q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.+0.j, 0.-1.j],
           [0.+0.j, 0.+0.j, 1.+0.j, 0.+0.j],
           [0.+0.j, 0.+1.j, 0.+0.j, 0.+0.j]])

#### Controlled $Z$ (or, controlled Phase) gate

Similarly, the controlled Z gate flips the phase of the target qubit if the control qubit is $\left|1\right\rangle$. The matrix looks the same regardless of whether the MSB or LSB is the control qubit:

$$
C_Z =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & -1
\end{pmatrix}
$$

```python
qc = QuantumCircuit(q)
qc.cz(q[0],q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[ 1.+0.j,  0.+0.j,  0.+0.j,  0.+0.j],
           [ 0.+0.j,  1.+0.j,  0.+0.j,  0.+0.j],
           [ 0.+0.j,  0.+0.j,  1.+0.j,  0.+0.j],
           [ 0.+0.j,  0.+0.j,  0.+0.j, -1.+0.j]])

### Controlled Hadamard gate

Apply $H$ gate to the target qubit if the control qubit is $\left|1\right\rangle$. Below is the case where the control is the LSB qubit.

$$
C_H =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}}\\
0 & 0 & 1 & 0\\
0 & \frac{1}{\sqrt{2}}  & 0& -\frac{1}{\sqrt{2}}
\end{pmatrix}
$$

```python
qc = QuantumCircuit(q)
qc.ch(q[0],q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[ 0.707+0.707j,  0.   +0.j   ,  0.   +0.j   ,  0.   +0.j   ],
           [ 0.   +0.j   ,  0.5  +0.5j  ,  0.   +0.j   ,  0.5  +0.5j  ],
           [ 0.   +0.j   ,  0.   +0.j   ,  0.707+0.707j,  0.   +0.j   ],
           [ 0.   +0.j   ,  0.5  +0.5j  ,  0.   +0.j   , -0.5  -0.5j  ]])

### Controlled rotation gates

#### Controlled rotation around Z-axis

Perform rotation around Z-axis on the target qubit if the control qubit (here LSB) is $\left|1\right\rangle$.

$$
C_{Rz}(\lambda) =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & e^{-i\lambda/2} & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & e^{i\lambda/2}
\end{pmatrix}
$$

```python
qc = QuantumCircuit(q)
qc.crz(pi/2,q[0],q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[1.   +0.j   , 0.   +0.j   , 0.   +0.j   , 0.   +0.j   ],
           [0.   +0.j   , 0.707-0.707j, 0.   +0.j   , 0.   +0.j   ],
           [0.   +0.j   , 0.   +0.j   , 1.   +0.j   , 0.   +0.j   ],
           [0.   +0.j   , 0.   +0.j   , 0.   +0.j   , 0.707+0.707j]])

### Controlled phase rotation

Perform a phase rotation if both qubits are in the $\left|11\right\rangle$ state. The matrix looks the same regardless of whether the MSB or LSB is the control qubit.

$$
C_{u1}(\lambda) =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 0 & 0 & e^{i\lambda}
\end{pmatrix}
$$

```python
qc = QuantumCircuit(q)
qc.cu1(pi/2,q[0], q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 1.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 1.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.+0.j, 0.+1.j]])

### Controlled $u3$ rotation

Perform controlled-$u3$ rotation on the target qubit if the control qubit (here LSB) is $\left|1\right\rangle$.

$$
C_{u3}(\theta, \phi, \lambda) \equiv
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & e^{-i(\phi+\lambda)/2}\cos(\theta/2) & 0 & -e^{-i(\phi-\lambda)/2}\sin(\theta/2)\\
0 & 0 & 1 & 0\\
0 & e^{i(\phi-\lambda)/2}\sin(\theta/2) & 0 & e^{i(\phi+\lambda)/2}\cos(\theta/2)
\end{pmatrix}.
$$

```python
qc = QuantumCircuit(q)
qc.cu3(pi/2, pi/2, pi/2, q[0], q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[ 1.   +0.j   ,  0.   +0.j   ,  0.   +0.j   ,  0.   +0.j   ],
           [ 0.   +0.j   ,  0.   -0.707j,  0.   +0.j   , -0.707+0.j   ],
           [ 0.   +0.j   ,  0.   +0.j   ,  1.   +0.j   ,  0.   +0.j   ],
           [ 0.   +0.j   ,  0.707+0.j   ,  0.   +0.j   ,  0.   +0.707j]])

### SWAP gate

The SWAP gate exchanges the two qubits. It transforms the basis vectors as

$$\left|00\right\rangle \rightarrow \left|00\right\rangle~,~\left|01\right\rangle \rightarrow \left|10\right\rangle~,~\left|10\right\rangle \rightarrow \left|01\right\rangle~,~\left|11\right\rangle \rightarrow \left|11\right\rangle,$$

which gives a matrix representation of the form

$$
\mathrm{SWAP} =
\begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 1
\end{pmatrix}.
$$

```python
qc = QuantumCircuit(q)
qc.swap(q[0], q[1])
qc.draw(output='mpl')
```

```python
job = execute(qc, backend)
job.result().get_unitary(qc, decimals=3)
```

    array([[1.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 1.+0.j, 0.+0.j],
           [0.+0.j, 1.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.+0.j, 1.+0.j]])

------------

```python
import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright
```

    UsageError: Line magic function `%qiskit_version_table` not found.

This code is a part of Qiskit
© Copyright IBM 2017, 2019.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.

Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals.

```python

```


# Multivariate Gaussian with full covariance

In this reading you will learn how you can use TensorFlow to specify any multivariate Gaussian distribution.

```python
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print("TF version:", tf.__version__)
print("TFP version:", tfp.__version__)
```

    TF version: 2.5.0
    TFP version: 0.13.0

So far, you've seen how to define multivariate Gaussian distributions using `tfd.MultivariateNormalDiag`. This class allows you to specify a multivariate Gaussian with a diagonal covariance matrix $\Sigma$.

In cases where the variance is the same for each component, i.e. $\Sigma = \sigma^2 I$, this is known as a _spherical_ or _isotropic_ Gaussian. This name comes from the spherical (or circular) contours of its probability density function, as you can see from the plot below for the two-dimensional case.

```python
# Plot the approximate density contours of a 2d spherical Gaussian

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

spherical_2d_gaussian = tfd.MultivariateNormalDiag(loc=[0., 0.])

N = 100000
x = spherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x=x1, y=x2, kind='kde', space=0);
```

As you know, a diagonal covariance matrix results in the components of the random vector being independent.

## Full covariance with `MultivariateNormalFullTriL`

You can define a full covariance Gaussian distribution in TensorFlow using the Distribution `tfd.MultivariateNormalTriL`.

Mathematically, the parameters of a multivariate Gaussian are a mean $\mu$ and a covariance matrix $\Sigma$, and so the `tfd.MultivariateNormalTriL` constructor requires two arguments:

- `loc`, a Tensor of floats corresponding to $\mu$,
- `scale_tril`, a a lower-triangular matrix $L$ such that $LL^T = \Sigma$.

For a $d$-dimensional random variable, the lower-triangular matrix $L$ looks like this:

\begin{equation}
    L = \begin{bmatrix}
            l_{1, 1} & 0 & 0 & \cdots & 0 \\
            l_{2, 1} & l_{2, 2} & 0 & \cdots & 0  \\
            l_{3, 1} & l_{3, 2} & l_{3, 3} & \cdots & 0  \\
            \vdots  & \vdots & \vdots & \ddots & \vdots \\
            l_{d, 1} & l_{d, 2} & l_{d, 3} & \cdots & l_{d, d}
        \end{bmatrix},
\end{equation}

where the diagonal entries are positive: $l_{i, i} > 0$ for $i=1,\ldots,d$.

Here is an example of creating a two-dimensional Gaussian with non-diagonal covariance:

```python
# Set the mean and covariance parameters

mu = [0., 0.]  # mean
scale_tril = [[1.,  0.],
              [0.6, 0.8]]

sigma = tf.matmul(tf.constant(scale_tril), tf.transpose(tf.constant(scale_tril)))  # covariance matrix
print(sigma)
```

    tf.Tensor(
    [[1.  0.6]
     [0.6 1. ]], shape=(2, 2), dtype=float32)

```python
# Create the 2D Gaussian with full covariance

nonspherical_2d_gaussian = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
nonspherical_2d_gaussian
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Check the Distribution mean

nonspherical_2d_gaussian.mean()
```

    <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>

```python
# Check the Distribution covariance

nonspherical_2d_gaussian.covariance()
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[1. , 0.6],
           [0.6, 1. ]], dtype=float32)>

```python
# Plot its approximate density contours

x = nonspherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x=x1, y=x2, kind='kde', space=0, color='r');
```

As you can see, the approximate density contours are now elliptical rather than circular. This is because the components of the Gaussian are correlated.

Also note that the marginal distributions (shown on the sides of the plot) are both univariate Gaussian distributions.

## The Cholesky decomposition

In the above example, we defined the lower triangular matrix $L$ and used that to build the multivariate Gaussian distribution. The covariance matrix is easily computed from $L$ as $\Sigma = LL^T$.

The reason that we define the multivariate Gaussian distribution in this way - as opposed to directly passing in the covariance matrix - is that not every matrix is a valid covariance matrix. The covariance matrix must have the following properties:

1. It is symmetric
2. It is positive (semi-)definite

_NB: A symmetric matrix $M \in \mathbb{R}^{d\times d}$ is positive semi-definite if it satisfies $b^TMb \ge 0$ for all nonzero $b\in\mathbb{R}^d$. If, in addition, we have $b^TMb = 0 \Rightarrow b=0$ then $M$ is positive definite._

The Cholesky decomposition is a useful way of writing a covariance matrix. The decomposition is described by this result:

> For every real-valued symmetric positive-definite matrix $M$, there is a unique lower-diagonal matrix $L$ that has  positive diagonal entries for which
>
> \begin{equation}
     LL^T = M
 \end{equation}
> This is called the _Cholesky decomposition_ of $M$.

This result shows us why Gaussian distributions with full covariance are completely represented by the `MultivariateNormalTriL` Distribution.

### `tf.linalg.cholesky`

In case you have a valid covariance matrix $\Sigma$ and would like to compute the lower triangular matrix $L$ above to instantiate a `MultivariateNormalTriL` object, this can be done with the `tf.linalg.cholesky` function.

```python
# Define a symmetric positive-definite matrix

sigma = [[10., 5.], [5., 10.]]
```

```python
# Compute the lower triangular matrix L from the Cholesky decomposition

scale_tril = tf.linalg.cholesky(sigma)
scale_tril
```

```python
# Check that LL^T = Sigma

tf.linalg.matmul(scale_tril, tf.transpose(scale_tril))
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[10.,  5.],
           [ 5., 10.]], dtype=float32)>

If the argument to the `tf.linalg.cholesky` is not positive definite, then it will fail:

```python
# Try to compute the Cholesky decomposition for a matrix with negative eigenvalues

bad_sigma = [[10., 11.], [11., 10.]]

try:
    scale_tril = tf.linalg.cholesky(bad_sigma)
except Exception as e:
    print(e)
```

    Cholesky decomposition was not successful. The input might not be valid. [Op:Cholesky]

### What about positive semi-definite matrices?

In cases where the matrix is only positive semi-definite, the Cholesky decomposition exists (if the diagonal entries of $L$ can be zero) but it is not unique.

For covariance matrices, this corresponds to the degenerate case where the probability density function collapses to a subspace of the event space. This is demonstrated in the following example:

```python
# Create a multivariate Gaussian with a positive semi-definite covariance matrix

psd_mvn = tfd.MultivariateNormalTriL(loc=[0., 0.], scale_tril=[[1., 0.], [0.4, 0.]])
psd_mvn
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Plot samples from this distribution

x = psd_mvn.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.title("Scatter plot of samples")
plt.scatter(x1, x2, alpha=0.5);
```

If the input to the function `tf.linalg.cholesky` is positive semi-definite but not positive definite, it will also fail:

```python
# Try to compute the Cholesky decomposition for a positive semi-definite matrix

another_bad_sigma = [[10., 0.], [0., 0.]]

try:
    scale_tril = tf.linalg.cholesky(another_bad_sigma)
except Exception as e:
    print(e)
```

    Cholesky decomposition was not successful. The input might not be valid. [Op:Cholesky]

In summary: if the covariance matrix $\Sigma$ for your multivariate Gaussian distribution is positive-definite, then an algorithm that computes the Cholesky decomposition of $\Sigma$ returns a lower-triangular matrix $L$ such that $LL^T = \Sigma$. This $L$ can then be passed as the `scale_tril` of `MultivariateNormalTriL`.

## Putting it all together

You are now ready to put everything that you have learned in this reading together.

To create a multivariate Gaussian distribution with full covariance you need to:

1. Specify parameters $\mu$ and either $\Sigma$ (a symmetric positive definite matrix) or $L$ (a lower triangular matrix with positive diagonal elements), such that $\Sigma = LL^T$.

2. If only $\Sigma$ is specified, compute `scale_tril = tf.linalg.cholesky(sigma)`.

3. Create the distribution: `multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)`.

```python
# Create a multivariate Gaussian distribution

mu = [1., 2., 3.]
sigma = [[0.5, 0.1, 0.1],
         [0.1,  1., 0.6],
         [0.1, 0.6, 2.]]

scale_tril = tf.linalg.cholesky(sigma)

multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
```

```python
# Check the covariance matrix

multivariate_normal.covariance()
```

    <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
    array([[0.49999997, 0.1       , 0.1       ],
           [0.1       , 1.0000001 , 0.6       ],
           [0.1       , 0.6       , 2.        ]], dtype=float32)>

```python
# Check the mean

multivariate_normal.mean()
```

    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([1., 2., 3.], dtype=float32)>

## Deprecated: `MultivariateNormalFullCovariance`

There was previously a class called `tfd.MultivariateNormalFullCovariance` which takes the full covariance matrix in its constructor, but this is being deprecated. Two reasons for this are:

* covariance matrices are symmetric, so specifying one directly involves passing redundant information, which involves writing unnecessary code.
* it is easier to enforce positive-definiteness through constraints on the elements of a decomposition than through a covariance matrix itself. The decomposition's only constraint is that its diagonal elements are positive, a condition that is easy to parameterize for.

### Further reading and resources
* https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MultivariateNormalTriL
* https://www.tensorflow.org/api_docs/python/tf/linalg/cholesky


# Pyomo - Getting started

Pyomo installation: see http://www.pyomo.org/installation

```
pip install pyomo
```

```python
from pyomo.environ import *
```

## Definition of the problem

$$
\begin{align}
    \min_{x_0,x_1} & \quad -x_0 + 4 x_1 \\
    \text{s.t.}    & \quad -3 x_0 + x_1 \leq 6 \\
                   & \quad -x_0 - 2 x_1 \geq -4 \\
                   & \quad  x_1 \geq -3
\end{align}
$$

## Solution

### Version 1: naive implementation

```python
model = ConcreteModel(name="Getting started")

c = [-1, 4]
b = [ 6, 4, 3]
A = [[-3,  1],
     [ 1,  2],
     [ 0, -1]]

N = list(range(len(c)))    # num decision variables
M = list(range(len(A)))    # num constraints

model.x = Var(N)

model.obj = Objective(expr = sum(c[n] * model.x[n] for n in N) )

m = 0
model.const_0 = Constraint(expr = sum(A[m][n] * model.x[n] for n in N) <= b[m] )
m = 1
model.const_1 = Constraint(expr = sum(A[m][n] * model.x[n] for n in N) <= b[m] )
m = 2
model.const_2 = Constraint(expr = sum(A[m][n] * model.x[n] for n in N) <= b[m] )

model.pprint()

# @tail:
print()
print("-" * 60)
print()

opt = SolverFactory('glpk')

results = opt.solve(model)

model.display()

print()
print("Optimal solution: ", [value(model.x[n]) for n in N])
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```

### Version 2: without construction rules

```python
model = ConcreteModel(name="Getting started")

c = [-1, 4]
b = [ 6, 4, 3]
A = [[-3,  1],
     [ 1,  2],
     [ 0, -1]]

N = list(range(len(c)))    # num decision variables
M = list(range(len(A)))    # num constraints

model.x = Var(N)

model.obj = Objective(expr = sum(c[n] * model.x[n] for n in N) )

for m in M:
    name = "const_{}".format(m)
    val = Constraint(expr = sum(A[m][n] * model.x[n] for n in N) <= b[m])
    model.add_component(name=name, val=val)

model.pprint()

# @tail:
print()
print("-" * 60)
print()

opt = SolverFactory('glpk')

results = opt.solve(model)

model.display()

print()
print("Optimal solution: ", [value(model.x[n]) for n in N])
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```

### Version 3: with construction rules

```python
model = ConcreteModel(name="Getting started")

c = [-1, 4]
b = [ 6, 4, 3]
A = [[-3,  1],
     [ 1,  2],
     [ 0, -1]]

N = list(range(len(c)))    # num decision variables
M = list(range(len(A)))    # num constraints

model.x = Var(N)

model.obj = Objective(expr = sum(c[n] * model.x[n] for n in N) )

def constraint_fn(model, m):
    return sum(A[m][n] * model.x[n] for n in N) <= b[m]

model.constraint = Constraint(M, rule=constraint_fn)

model.pprint()

# @tail:
print()
print("-" * 60)
print()

opt = SolverFactory('glpk')

results = opt.solve(model)

model.display()

print()
print("Optimal solution: ", [value(model.x[n]) for n in N])
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```

### Version 4: construction rules on Objective too

```python
model = ConcreteModel(name="Getting started")

c = [-1, 4]
b = [ 6, 4, 3]
A = [[-3,  1],
     [ 1,  2],
     [ 0, -1]]

N = list(range(len(c)))    # num decision variables
M = list(range(len(A)))    # num constraints

model.x = Var(N)

def objective_fn(model):
    return sum(c[n] * model.x[n] for n in N)

model.obj = Objective(rule=objective_fn)

def constraint_fn(model, m):
    return sum(A[m][n] * model.x[n] for n in N) <= b[m]

model.constraint = Constraint(M, rule=constraint_fn)

model.pprint()

# @tail:
print()
print("-" * 60)
print()

opt = SolverFactory('glpk')

results = opt.solve(model)

model.display()

print()
print("Optimal solution: ", [value(model.x[n]) for n in N])
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```


# Differences in Matrix Representation - SymPy vs. NumPy
SymPy stores entries in 1D arrays, while NumPy stores entries in 2D arrays.

```python
import sympy as sp
import numpy as np
sp.init_printing(use_unicode=True)
```

```python
#Sympy version of our demo matrix
As = sp.Matrix([[1,2,3], [4,5,6]])
As
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\end{matrix}\right]$

```python
#Numpy version of our demo matrix - Libraries are inter-compatible
An = np. array(As)
An
```

    array([[1, 2, 3],
           [4, 5, 6]], dtype=object)

```python
#We can't obtain the dimensions of the SymPy Matrix easily
len(As)
```

```python
#We can obtain the size of the NumPy Matrix:
size= [len(An), len(An[0])]
size
```

```python
#We can also obtain the size of the NumPy Matrix using shape property
An.shape
```

```python
An
```

    array([[1, 2, 3],
           [4, 5, 6]], dtype=object)

```python
An+1
```

    array([[2, 3, 4],
           [5, 6, 7]], dtype=object)

```python
As.T
```

$\displaystyle \left[\begin{matrix}1 & 4\\2 & 5\\3 & 6\end{matrix}\right]$

```python
An.T
```

    array([[1, 4],
           [2, 5],
           [3, 6]], dtype=object)

```python
3*As
```

$\displaystyle \left[\begin{matrix}3 & 6 & 9\\12 & 15 & 18\end{matrix}\right]$

```python
As/2
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & 1 & \frac{3}{2}\\2 & \frac{5}{2} & 3\end{matrix}\right]$

```python
Bs = 3*As
Bs
```

$\displaystyle \left[\begin{matrix}3 & 6 & 9\\12 & 15 & 18\end{matrix}\right]$

```python
#As is 2x3 Bs^T is 3x2, so the product is 2x2 ...
As*Bs.T
```

$\displaystyle \left[\begin{matrix}42 & 96\\96 & 231\end{matrix}\right]$

```python
#Try the following code, it will generate ShapeError: Matrix size mismatch: (2, 3) * (2, 3).
#As*Bs
```

```python
# You can use Numpy library as well
Bn = 3*An
Bn
```

    array([[3, 6, 9],
           [12, 15, 18]], dtype=object)

```python
# we use dot method in Numpy for Matrix multiplications
np.dot(An, Bn. T)
```

    array([[42, 96],
           [96, 231]], dtype=object)

```python
# if you use * for multiplication in Numpy, it is element-wise, it is NOT a matrix product
An*Bn
```

    array([[3, 12, 27],
           [48, 75, 108]], dtype=object)

# Matrix Inverse

```python
x = np.array([[1,2],[3,4]])
y = np.linalg.inv(x)
print (x)
print (y)
print (np.dot(x,y))
```

    [[1 2]
     [3 4]]
    [[-2.   1. ]
     [ 1.5 -0.5]]
    [[1.0000000e+00 0.0000000e+00]
     [8.8817842e-16 1.0000000e+00]]


# Solutions for the Lane-Emden with n=0

```python
from distutils.spawn import find_executable

import matplotlib.pyplot as plt
import numpy as np

import seaborn

rem = 12

seaborn.set(context='notebook', style='darkgrid')

plt.ioff()

plt.rc('lines', linewidth=1)
plt.rc('font', family='serif')
plt.rc('font', size=rem)
plt.rc('axes', titlepad=1.500*rem)
plt.rc('axes', titlesize=1.728*rem)
plt.rc('axes', labelsize=1.200*rem)
plt.rc('legend', fontsize=1.000*rem)
plt.rc('xtick', labelsize=0.833*rem)
plt.rc('ytick', labelsize=0.833*rem)

if find_executable('latex'):
    plt.rc('text', usetex=True)

material_palette = {
    -1: "#212121",
    0: "#F44336",
    1: "#E91E63",
    2: "#9C27B0",
    3: "#673AB7",
    4: "#3F51B5",
    5: "#2196F3",
    6: "#03A9F4",
    7: "#00BCD4",
    8: "#009688",
    9: "#4CAF50",
    10: "#8BC34A",
    11: "#CDDC39",
    12: "#FFEB3B",
    13: "#FFC107",
    14: "#FF9800",
    15: "#FF5722",
}
```

```python
%matplotlib inline
```

```python
from sympy import (
    Eq as Equation,
    Derivative,
    Function,
    Symbol,
    lambdify,
    simplify,
    symbols,
    dsolve,
    solve,
)
```

```python
n = Symbol("n")
xi = Symbol("xi")
theta = Function("theta")
```

```python
lhs = simplify(
    (1 / xi ** 2) * Derivative(
        (xi ** 2) * Derivative(theta(xi), xi), xi
    ).doit()
)
lhs
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi}$

```python
rhs = -theta(xi) ** n
rhs
```

$\displaystyle - \theta^{n}{\left(\xi \right)}$

```python
lane_endem_eq = Equation(lhs, rhs)
lane_endem_eq
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi} = - \theta^{n}{\left(\xi \right)}$

Reemplazando n=0:

```python
lane_endem_eq_0 = lane_endem_eq.subs(n, 0)
lane_endem_eq_0
```

$\displaystyle \frac{d^{2}}{d \xi^{2}} \theta{\left(\xi \right)} + \frac{2 \frac{d}{d \xi} \theta{\left(\xi \right)}}{\xi} = -1$

```python
solution = dsolve(lane_endem_eq_0, theta(xi))
solution
```

$\displaystyle \theta{\left(\xi \right)} = C_{1} + \frac{C_{2}}{\xi} - \frac{\xi^{2}}{6}$

```python
constants = solve(
    [
        simplify(xi * solution.rhs).subs(xi, 0),
        Derivative(
            simplify(xi * solution.rhs), xi
        ).doit().subs(xi, 0) - 1,
    ],
    symbols('C1 C2'),
)
constants
```

    {C1: 1, C2: 0}

$$C1= 1,  C2=0$$

```python
solution = solution.subs(constants)
solution
```

$\displaystyle \theta{\left(\xi \right)} = 1 - \frac{\xi^{2}}{6}$

```python
theta_zeros = solve(solution.rhs, xi)
theta_zeros
```

    [-sqrt(6), sqrt(6)]

```python
num_theta_f = lambdify(xi, solution.rhs, "numpy")
```

```python
n_xi = np.linspace(0, 10, 101)
n_theta = num_theta_f(n_xi)

fig = plt.figure(figsize=(11.25, 4.5), frameon=False)
axs = fig.add_subplot(1, 1, 1)

axs.plot(
    n_xi,
    n_theta,
    color=material_palette[1],
    label=r"$\theta_0(\xi)$"
)
axs.legend()
axs.set_title(r"Solution for Lane-Endem equation for $n=0$")
axs.set_xlim([0, 10])
axs.set_xlabel(r"$\xi$")
axs.set_ylim([-2, 2])
axs.set_ylabel(r"$\theta_0(\xi)$")
plt.tight_layout()
plt.show()
```

```python

```


[Qiita の解説記事](https://qiita.com/tobira-code/items/d76ed91a88f112b4a474)を参考にした ODE のサンプルノート。
はじめに必要なパッケージを import する.
- numpy で多次元配列データ処理
- matplotlib でデータ可視化
- scipy.integrate で積分

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import ode
```

常微分方程式 $y''=-y$ を解くために以下のように連立1階微分方程式に分解する.

\begin{align}
y'&=z \\
z'&=-y
\end{align}

コード上は $v\equiv(y,z)$ とする.
導関数を $v'\equiv f(t,v)$ とし、初期条件を $v(t=0)\equiv v_0 \equiv (1.0, 0.0)$ とすると以下のコードのようになる。

```python
def f(t, v): # t, y:v[0], y'=z:v[1]
    return [v[1], -v[0]] # y':return[0] y''=z':return[1]

v0 = [1.0, 0.0] # y0, y'0=z0
```

Nonstiff な微分方程式を解くためのアルゴリズム「8(5,3)次の Runge-Kutta 法(Dorman-Prince 法)」による関数 dop853 を利用する。

詳細は[こちら](https://org-technology.com/posts/ordinary-differential-equations.html).
公式ドキュメントは[こちら](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html).

```python
solver = ode(f)
solver.set_integrator(name="dop853")
solver.set_initial_value(v0)
```

    <scipy.integrate._ode.ode at 0x7f687f869470>

積分区間 $0<t<tw\equiv10\times(2\pi)$,差分量 $\Delta t = tw/1000$ で計算を実行.

$ts$ に時刻データ, $ys$ に計算結果を格納.

```python
tw = 10.0*2.0*np.pi
dt = tw / 1000;
t = 0.0
ts = []
ys = []
while solver.t < tw:
    solver.integrate(solver.t+dt)
    ts += [solver.t]
    ys += [solver.y[0]]
```

最後にデータのプロットを行い, 解析解 $\cos(t)$ との差から誤差の蓄積を計算する.

```python
plt.figure(0)
plt.plot(ts, ys)

plt.figure(1)
plt.plot(ts, np.cos(ts)-ys)

plt.show()
print(np.max(np.cos(ts)-ys))
```


# Importance sampling

Author: [Nipun Batra](https://nipunbatra.github.io/)

https://www.youtube.com/watch?v=TNZk8lo4e-Q&t=2733s

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rc
import seaborn as sns
import scipy.stats
%matplotlib inline

rc('font', size=16)
rc('text', usetex=True)
```

Consider the following likelihood and a Gaussian prior.

```python
# Likelihood
def l(theta):
    return 4+ np.sin(theta) - (theta**2)/3

prior = scipy.stats.norm(0).pdf
```

```python
x = np.linspace(-3, 3, 1000)

plt.plot(x, prior(x),label='Prior')
plt.plot(x, l(x),label='Likelihood')
plt.legend();
plt.xlabel('x');
```

We define a sampling distribution $q(x)$ as the following.

```python
q_rvs = scipy.stats.norm(loc=0, scale=10)
q = q_rvs.pdf

plt.plot(x, q(x), label='$q(x)$');
plt.xlabel('x');
plt.legend();
```

Let's draw a large number of samples from $q(x)$ distribution.

```python
q_samples = q_rvs.rvs(size=10000)

sns.kdeplot(q_samples);
plt.xlabel('x');
```

We can find the marginal likelihood $z$ using the following technique.

\begin{align}
z &= \int likelihood(x)\cdot prior(x)\cdot dx\\
z &= \int \frac{likelihood(x)\cdot prior(x)}{q(x)}\cdot q(x) \cdot dx\\
z &= \int w(x) \cdot q(x) \cdot dx\\
z &\approx \frac{1}{N}\sum\limits_{i=1}^{N}w(x)
\end{align}

```python
prior_eval_q = prior(q_samples)
likelihood_eval_q = l(q_samples)
z = (prior_eval_q*likelihood_eval_q/q_samples).mean()
z
```

    -0.508527841674989

## Importance sampling for linear regression

Likelihood for linear regression can be given as the following,

$p(\mathbf{y}|\theta) \sim \mathcal{N}(\theta \mathbf{x}, \sigma_n^2I)$

```python
theta_gt = 4
sigma_n = 1

# generate some samples

x = np.random.uniform(0, 1, size = 100)
y = np.random.normal(theta_gt*x, sigma_n)
```

```python
plt.scatter(x, y);
plt.xlabel('x');plt.ylabel('y');
```

We will use the following $q$ distribution in this problem.

```python
# Proposal

q_rvs = scipy.stats.norm(loc=3, scale=10)
q = q_rvs.pdf
```

Prior on $\theta$ is Standard Gaussian distribution.

```python
prior = scipy.stats.norm(0).pdf
```

Likelihood is given as the following,

```python
# Likelihood
def l(theta):
    return scipy.stats.multivariate_normal.pdf(y, mean=theta*x, cov=3*np.eye(len(x)))
```

```python
xu = np.linspace(2, 6, 1000)
k = []
for xt in xu:
    k.append(l(xt))
plt.plot(xu, k, label='likelihood');
plt.xlabel('x');
plt.legend();
```

```python
plt.hist(scipy.stats.norm(10*x, 1).pdf(y), density=False, bins=20);
```

Let us draw a few samples from the $q$ distribution.

```python
n_samples = 1000
q_samples = q_rvs.rvs(size=n_samples)
```

We calculate the $w(x)$ using the samples drawn from the $q$ distribution.

```python
plt.hist(q_samples)

w = np.zeros(n_samples)
for i in range(n_samples):
    theta_i = q_samples[i]
    likelihood_i = l(theta_i)
    prior_i = prior(theta_i)
    q_i = q_rvs.pdf(theta_i)
    w_i = likelihood_i*prior_i/q_i
    w[i] = w_i
```

It is easy to retrive marginal likelihood now.

```python
marginal_likelihood = np.mean(w)
```

Let us compute the full posterior distribution.

```python
post = np.zeros(n_samples)
for i in range(n_samples):
    theta_i = q_samples[i]
    likelihood_i = l(theta_i)
    prior_i = prior(theta_i)
    post_i = likelihood_i*prior_i
    post[i] = post_i/marginal_likelihood
```

We can visualize the posterior distribution as the following,

```python
idx = np.argsort(q_samples)
plt.plot(q_samples[idx], post[idx], label='posterior pdf');
plt.xlabel('x')
plt.legend()

print('approx posterior mean', np.mean(q_samples[np.where(post>1.25)]))
```


```python
from sympy import *
```

```python
x = Symbol('x')
y = Function('y')
ec = Eq(x**2*y(x).diff(x,2) - 3*x*y(x).diff(x) + 6*y(x),0)
```

```python
ec
```

$\displaystyle x^{2} \frac{d^{2}}{d x^{2}} y{\left(x \right)} - 3 x \frac{d}{d x} y{\left(x \right)} + 6 y{\left(x \right)} = 0$

```python
print('Ejercicio a ')
dsolve(ec)
```

    Ejercicio a

$\displaystyle y{\left(x \right)} = C_{2} e^{- 2 x} + \left(C_{1} + 2 x\right) e^{2 x}$

```python

ec = Eq(x**2*y(x).diff(x,2) + 5*x*y(x).diff(x) + 4*y(x),0)
```

```python
print('ejrcicio b')
ec = Eq(x**2*y(x).diff(x,2) + 5*x*y(x).diff(x) + 4*y(x),0)
dsolve(ec)
```

    ejrcicio b

$\displaystyle y{\left(x \right)} = \frac{C_{1} + C_{2} \log{\left(x \right)}}{x^{2}}$

```python
ec = Eq(x**2*y(x).diff(x,2) + 9*x*y(x).diff(x) +17*y(x),0)
print('Ejercicio C')
dsolve(ec)
```

    Ejercicio C

$\displaystyle y{\left(x \right)} = \frac{C_{1} \sin{\left(\log{\left(x \right)} \right)} + C_{2} \cos{\left(\log{\left(x \right)} \right)}}{x^{4}}$

```python
print('Ejercicio d')
ec = Eq(y(x).diff(x,2) - 4*y(x),8*exp(2*x))
ec
```

    Ejercicio d

$\displaystyle - 4 y{\left(x \right)} + \frac{d^{2}}{d x^{2}} y{\left(x \right)} = 8 e^{2 x}$

```python
dsolve(ec)
```

$\displaystyle y{\left(x \right)} = C_{2} e^{- 2 x} + \left(C_{1} + 2 x\right) e^{2 x}$

```python
print(' Ejercicio e ')
ec = Eq(y(x).diff(x,2)-4*y(x).diff(x)+ 4*y(x),exp(-2*x))
dsolve(ec)
```

     Ejercicio e

$\displaystyle y{\left(x \right)} = \left(C_{1} + C_{2} x\right) e^{2 x} + \frac{e^{- 2 x}}{16}$

```python

```


# 15.4. Computing exact probabilities and manipulating random variables

```python
from sympy import *
from sympy.stats import *
init_printing()
```

```python
X, Y = Die('X', 6), Die('Y', 6)
```

```python
P(Eq(X, 3))
```

```python
P(X > 3)
```

```python
P(X > Y)
```

```python
P(X + Y > 6, X < 5)
```

```python
Z = Normal('Z', 0, 1)  # Gaussian variable
```

```python
P(Z > pi)
```

```python
E(Z**2), variance(Z**2)
```

```python
f = density(Z)
```

```python
var('x')
f(x)
```

```python
%matplotlib inline
plot(f(x), (x, -6, 6))
```

```python
Eq(Integral(f(x), (x, pi, oo)),
   simplify(integrate(f(x), (x, pi, oo))))
```


```python
%%markdown
# Overview
In a linear regression, the outcome, $y$, is approximated by a hidden function of the type $\forall x \in S, f(x) = a + bx$,
and the error function is $J(x) = E_{x\in S} (f(x) - y)^2$

For a $n$ samples of $x$:
\begin{align}
J_{a,b}(x) = \frac{1}{n} \sum_{i=1}^{i=n} (f(x_i) - y_i)^2 = \frac{1}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i)^2
\end{align}

The gradient of $J(x)$ is then:
\begin{align}
\frac{\partial J(x)}{\partial a} &= \frac{2}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i) \\
\frac{\partial J(x)}{\partial b} &= \frac{2}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i)x_i
\end{align}

# References
## Gradient descent
* http://en.wikipedia.org/wiki/Gradient_descent
* http://en.wikipedia.org/wiki/Stochastic_gradient_descent#Example
## Formatting
* [Tex/$\LaTeX$ support in Jupyter Markdown](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html)
```

# Overview
In a linear regression, the outcome, $y$, is approximated by a hidden function of the type $\forall x \in S, f(x) = a + bx$,
and the error function is $J(x) = E_{x\in S} (f(x) - y)^2$

For a $n$ samples of $x$:
\begin{align}
J_{a,b}(x) = \frac{1}{n} \sum_{i=1}^{i=n} (f(x_i) - y_i)^2 = \frac{1}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i)^2
\end{align}

The gradient of $J(x)$ is then:
\begin{align}
\frac{\partial J(x)}{\partial a} &= \frac{2}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i) \\
\frac{\partial J(x)}{\partial b} &= \frac{2}{n} \sum_{i=1}^{i=n} (a + bx_i - y_i)x_i
\end{align}

# References
## Gradient descent
* http://en.wikipedia.org/wiki/Gradient_descent
* http://en.wikipedia.org/wiki/Stochastic_gradient_descent#Example
## Formatting
* [Tex/$\LaTeX$ support in Jupyter Markdown](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html)

```python
from mpl_toolkits import mplot3d
import numpy as np
import matplotlib.pyplot as plt

```

```python
# Error function for x = {1, 2}: 1/2 [(a+b-1)^2 + (a+2b-2)^2]
# Derivative:
#  a. dJa = (a+b-1) + (a+2b-2) = 2a + 3b - 3
#  b. dJb = (a+b-1) + 2(a+2b-2) = 3a + 5b - 5
#  a = 0, b = 0 => dJa = -3 ; dJb = -5
def error_function(a, b):
    total_error = 0.0
    n = 2
    for x in range(1, n):
        total_error += (a + b*x - x) ** 2

    return total_error / n

x = np.linspace(-5, 5, 30)
y = np.linspace(-5, 5, 30)

X, Y = np.meshgrid(x, y)
Z = error_function(X, Y)
```

```python
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(X, Y, Z, 50, cmap='binary')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
```

```python

```


# Logistic Regression Example Notebook
This example shows how to predict if a student passes a test using Logistic Regression.

The goal is to give a basic introduction to classification problems, so many things will be simplified.

Regarding notation, $a$ is a scalar, $\mathbf{a}$ is a vector (column vector), and $\mathbf{A}$ is a matrix.

# Imports

```python
import matplotlib.pyplot as plt  # Plotting
import numpy as np  # Array computation
import seaborn as sns  # Plotting
from sklearn.linear_model import LinearRegression  # Linear Regression model
from IPython.display import display, clear_output  # Plotting

%matplotlib inline
sns.set_theme()
sns.set_style("ticks")
```

# Create and Process data
We use fake data for this example. The input is the number of hours a student has studied for a test and the target is a binary label indicating if they have passed the test or not.

We want train_xs to be a matrix with shape ($m$, $n + 1$) with the first column being all ones (in our case $m = 12$ and $n = 1$):
\begin{align}
\mathbf{X} = \begin{bmatrix} x^1_{0} && \ldots && x^1_{n} \\ \vdots && \ddots && \vdots \\ x^m_{0} && \ldots && x^m_{n} \end{bmatrix}
\end{align}

We want train_ys to be a vector with shape ($m$, 1):
\begin{align}
\mathbf{y} = \begin{bmatrix} y^1 \\ \vdots \\ y^m \end{bmatrix}
\end{align}

```python
train_xs = np.array([1, 2, 3.5, 5, 6, 7.2, 8, 10.2, 12, 15.7, 17, 20]).reshape(-1, 1)
train_ys = np.array([0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1]).reshape(-1, 1)

# Plot data
sns.relplot(train_xs.flatten(), train_ys.flatten())
plt.title("Students Passing Exam")
plt.ylabel("Passed  Exam")
plt.xlabel("Hours Studied")
plt.xlim([0, 25])

```

This is a classification problem but for educational purposes let's try fitting a line to this data using sklearn.

```python
regressor = LinearRegression()
regressor = regressor.fit(train_xs, train_ys)  # Remove the extra features from x

plot_xs = np.linspace(-10, 30, 500)
plot_ys =  regressor.intercept_ + regressor.coef_[0] * plot_xs
sns.relplot(train_xs.flatten(), train_ys.flatten())
sns.lineplot(plot_xs, plot_ys, color="red",linestyle="--")
plt.title("Students Passing Exam")
plt.ylabel("Passed  Exam")
plt.xlabel("Hours Studied")
plt.xlim([-10, 30])
```

We quickly see that regression can't be directly used for classification in this way. Linear regression produces a line that is not restricted to 0 and 1.

Therefore, we need to define a different hypothesis.

```python
# Add extra feature to xs
ones = np.ones(shape=(12, 1))
train_xs = np.c_[ones, train_xs]  # Shape is now (12, 2)
```

# Hypothesis Definition
The hypothesis function $h$ is defined as follows for a single example $\mathbf{x}^k$:

\begin{align}
h_\theta(\mathbf{x}^k) & = g(\boldsymbol{\theta}^T\mathbf{x}^k) = \frac{1}{1 + e^{-\boldsymbol{\theta}^T\mathbf{x}^k}}
\end{align}

where $g$ is the logistic or sigmoid function:

\begin{align}
g(z) &= \frac{1}{1 + e^{-z}}
\end{align}

We want to implement a batched method which computes the hypothesis for all examples in one go. To do so, we can implement the following version:

\begin{align}
h_\theta(\mathbf{X}) & =  g(\mathbf{X} \boldsymbol{\theta}) = \begin{bmatrix} h_\theta(\mathbf{x}^1) \\ \vdots \\ h_\theta(\mathbf{x}^m) \end{bmatrix}
\end{align}

```python
def logistic(z):
    return 1/(1 + np.e**-z)

def hypothesis(thetas, input_data):
    return logistic(np.dot(input_data, thetas))
```

```python
# Plot logistic function
x_plot = np.linspace(-10, 10, 500)
y_plot = logistic(x_plot)

sns.lineplot(x_plot, y_plot)
```

We use the hypothesis to define the probabilities of the classes:

\begin{align}
P(y^k = 1 | \mathbf{x}^k; \boldsymbol{\theta}) &= h_\theta(\mathbf{x}^k) \\
P(y^k = 0 | \mathbf{x}^k; \boldsymbol{\theta}) &= 1 - h_\theta(\mathbf{x}^k) \\
P(y^k| \mathbf{x}^k; \boldsymbol{\theta}) &= (h_\theta(\mathbf{x}^k))^{y^{k}} (1- h_\theta(\mathbf{x}^k))^{1 - y^{k}}  \\
\end{align}

```python
def class_prob(thetas, input_data, class_n):
    hypothesis_estimate = hypothesis(thetas, input_data)
    return (hypothesis_estimate ** class_n) * (1-hypothesis_estimate)**(1-class_n)
```

# Learning Algorithm

## Cost Function
We want to maximise the likelihood of our training data:
\begin{align}
L(\boldsymbol\theta) &= P(\mathbf{y}| \mathbf{X}; \boldsymbol{\theta}) \\
                     &= \prod_{k=1}^m P(y^k| \mathbf{x}^k; \boldsymbol{\theta})
\end{align}

To make things easier, we will instead minimise the negative log likelihood:
\begin{align}
l(\boldsymbol\theta) &= -log(L(\boldsymbol\theta)) \\
                     &= -\left[\sum_{k=1}^m y^klog(h_\theta(\mathbf{x}^k)) + (1 - y^k)log(1 - h_\theta(\mathbf{x}^k))\right]
\end{align}

We can instead implement a vectorized form which returns the negative log likelihood per example:
\begin{align}
l(\boldsymbol\theta) &= -\left[\mathbf{y} * log(h_\theta(\mathbf{X})) + (\mathbf{1} - \mathbf{y}) * log(\mathbf{1} - h_\theta(\mathbf{X}))\right]
\end{align}

```python
def cost_function(thetas, input_data, target_data):
    hypothesis_result = hypothesis(thetas, input_data)
    return -(target_data * np.log(hypothesis_result) + (1 - target_data) * np.log(1 - hypothesis_result))
```

## Gradient Descent
Second step, define gradient descent per epoch, where each parameter $\theta_i$ is updated following:
\begin{align}
\theta_i & ≔ \theta_i - \alpha \frac{\partial}{\partial \theta_i} l(\boldsymbol\theta) \\
& ≔ \theta_i - \alpha \sum_{k=1}^m (h_\theta(\mathbf{x}^k) - y^k)x^k_i
\end{align}
As before, we can instead implement a vectorized operation:

\begin{align}
\boldsymbol{\theta} & ≔ \boldsymbol{\theta} - \alpha \mathbf{X}^T(h_\theta(\mathbf{X}) - \mathbf{y})
\end{align}

```python
def gradient_descent(thetas, input_data, target_data, alpha=0.1):
    """
    thetas: parameters for the hypothesis, expected shape is (2, 1)
    input_data: full matrix of input data, expected shape is (n_samples, 2)
    target_data: full vector of target data, expected shape is (n_samples, 1)
    alpha: learning rate, defaults to 0.1

    return: New parameters, shape is (2, 1)
    """
    return thetas - alpha * np.dot(input_data.T, hypothesis(thetas, input_data) - target_data)
```

# Train

```python
parameters = np.array([[0.01], [0.01]])

# Create array of xs to use for plotting
x_space = np.linspace(0, 25, 500).reshape(500, 1)
ones = np.ones(shape=(500, 1))
x_added = np.c_[ones, x_space]

def create_fig():
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)

    return fig, ax

def plot(ax, fig, parameters):
    current_curve = hypothesis(parameters, x_added)

    ax.cla()

    ax.set_xlim([0, 25])
    ax.set_ylim([-0.05, 1.05])
    ax.plot(x_space.flatten(), current_curve.flatten(), "r--")
    ax.scatter(train_xs[:, 1], train_ys.flatten())
    ax.hlines(0.5, 0, 25)
    ax.set_title("Students Passing Exam")
    ax.set_ylabel("Probability of Passing Exam")
    ax.set_xlabel("Hours Studied")
    display(fig)

    clear_output(wait = True)
    plt.pause(0.0001)

def train(parameters, n_epochs=300):
    # Create empty figure for plotting
    fig, ax = create_fig()

    # Iterate over n_epochs
    for epoch in range(n_epochs):
        parameters = gradient_descent(parameters, train_xs, train_ys, alpha=0.005)
        plot(ax, fig, parameters)

    return parameters

trained_parameters = train(parameters)
```

```python
# Compute accuracy for the training set
def class_accuracy(thetas, input_data, targets):
    probs = class_prob(thetas, input_data, targets)
    correct = (probs >= 0.5).sum()  # Using >= means that we say passing when the probability is 50%
    return correct/probs.size

print(f"Final training accuracy: {class_accuracy(trained_parameters, train_xs, train_ys):.2f}")
```

    Final training accuracy: 0.75

## Library Version

```python
import sklearn.linear_model

lr_classifier = sklearn.linear_model.LogisticRegression()
lr_classifier.fit(train_xs[:, 1:], train_ys)

sklearn_params = np.array([lr_classifier.intercept_[0], lr_classifier.coef_[0, 0]])  # Write the learned parameters as before
fig, ax = create_fig()
plot(ax, fig, sklearn_params)
```

```python
lr_acc = lr_classifier.score(train_xs[:, 1:], train_ys)
lr_acc
```

    0.75


```python
import numpy as np
import matplotlib.pyplot as plt
import torch
```

$\textbf{Definitions:}$ $\\$

$\mbox{---Gradient---}$ $\\$ $\\$
Vector formed by partial derivatives of scalar function, f(x) in which $x = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\x_n \end{bmatrix}$ $\\$
Gradient maps $\mathbb{R}^n \rightarrow \mathbb{R}$ $\\$

$$\nabla f(\mathbf{x})=\frac{\partial f(\mathbf{x})}{\partial x_1}\hat{x}_1+\frac{\partial f(\mathbf{x})}{\partial x_2}\hat{x}_2+\ldots+\frac{\partial f(\mathbf{x})}{\partial x_n}\hat{x}_n$$

$$\nabla f(x) = \left[\frac{\partial f}{\partial x_1}\frac{\partial f}{\partial x_2}\dots\frac{\partial f}{\partial x_n}\right]$$

Note: Input is a column vector, outputs a row vector. $\\$
Gradient is the rate of change wrt each dimension/component and corresponds to steepest slope due to linear independence used for gradient descent. $\\$

$\mbox{---Jacobian---}$ $\\$
Matrix formed by partial derivatives of vector function of scalar functions, maps $\mathbb{R}^n \rightarrow \mathbb{R}^m$ $\\$

$$J_\mathbf{f} = \frac{\partial (f_1,\ldots,f_m)}{\partial(x_1,\ldots,x_n)} = \left[
\begin{matrix}
\frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n}
\end{matrix}
\right]$$
The Jacobian is the gradient applied to multiple rows, commonly used as a change of basis/unit conversion:

$$\iiint_R f(x,y,z) \,dx\,dy\,dz = \iiint_S f(x(u,v,w),y(u,v,w),z(u,v,w))\left|\frac{\partial (x,y,z)}{\partial(u,v,w)}\right|\,du\,dv\,dw$$
Note: Gradient = Jacobian if $m = 1$.

$\mbox{---Hessian---}$ $\\$
Gradient applied to Gradient, Double Gradient: $\\$
$$\begin{align}D[\nabla f(\mathbf x)] &= D[D[f(\mathbf x)]]\\
&=\left(D\left[\frac{\partial f}{\partial x_1}\right]^T, \ldots, D\left[\frac{\partial f}{\partial x_n}\right]^T\right)\end{align}$$
Which expands to give us the Hessian matrix:
$$D^2[f(\mathbf x)]=\left(\begin{matrix}\frac{\partial^2 f}{\partial x_1^2} & \ldots & \frac{\partial^2 f}{\partial x_1\partial x_n}\\
\vdots & \ddots & \vdots \\
\frac{\partial^2 f}{\partial x_n\partial x_1}& \ldots & \frac{\partial^2 f}{\partial x_n^2}\end{matrix}\right)$$

Note: Inputs are column vectors, outputs are row vectors. (Transposed first because the first gradient outputs a row vector) $\\$
The Hessian represents the rate of change of gradient, analogous to curvature. Used to computationally determine the position of a min/max point in optimization, which is darn impossible to visualize past 2 dimensions.

$\textbf{Analytic Gradient:}$ $\\$
$\mbox{---Linear Form---}$ $\\$
$f(x) = a^T x$ $\\$
Component-wise derivative yields corresponding dot-product coefficent of each component k. $\\$
Assemble each partial derivatives into vector: $\\$
$\nabla f(x) = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} = a$

-General Linear Form: $\\$
$f(x) = a^Tx + b$ $\\$
$\nabla f(x) = a$

$\mbox{---Quadratic Form---}$ $\\$
$f(x) = x^T A x$ $\\$
Tracing through 2x2 example: $\\$
$\nabla f(x) = (A + A^T)x$ $\\$
For pd matrices, $A = A^T$ so: $\\$
$\nabla f(x) = 2Ax$

-General Quadratic Form, which builds from gradient of general linear form: $\\$
$f(x) = \frac{1}{2}x^T A x + b^Tx + c$ $\\$
$\nabla f(x) = \frac{1}{2}(A^T + A)x + b$ $\\$
For symmteric matrix A: $\\$
$\nabla f(x) = Ax + b$

-Mixed Quadratic Form: $\\$
$f(x,y) = x^T A y$ $\\$
Wrt x: $\nabla_x f(x,y) = Ay$ $\\$
Wrt y: $\nabla_y f(x,y) = A^Tx$ $\\$
Taking the right partial derivative, transpose. $\\$
Matrices are pd, so wrt y: $\nabla_y f(x,y) = Ax$

$\textbf{Analytic Hessian:}$ $\\$
Tracing through 2x2 example again: $\\$
$\mbox{---Linear Form---}$ $\\$
$f(x) = a^T x$ $\\$
$\nabla f(x)$ does not depend on x, so $\nabla^2 f(x) = 0$.

$\mbox{---Quadratic Form---}$ $\\$
$f(x) = x^T A x$ $\\$
$\nabla^2 f(x) = A + A^T$ $\\$
For symmteric matrix A: $\\$
$\nabla^2 f(x) = 2A$

Mixed Quadratic Form: $\\$
$f(x,y) = x^T A y$ $\\$
Wrt xx, yy: $H_{xx} = H_{yy} = 0$ $\\$
Wrt xy, yx: $H_{xy} = H_{yx} = 2A$

Simultaneous gradient descent (continuous time):
    $\dot x = -D_1f_1(x,y),\ \dot y = -D_2f_2(x,y)$, simgrad Jacobian
    $J(x,y) = \begin{bmatrix} D_1^2f_1(x,y) & D_{12}f_1(x,y) \\ D_{21}f_2(x,y) & D_2^2f_2(x,y) \end{bmatrix}$

(discrete time):
$x^+ = x - \gamma_x D_1f_1(x,y),\
y^+ = y - \gamma_y D_2f_2(x,y)$

```python

```

```python
m = 2
n = 2

#Random pd matrices, Cholesky form:
np.random.seed(0)
A1 = np.random.randn(n,n)
A1 = A1.T @ A1

A2 = np.random.randn(n,n)
A2 = A2.T @ A2

#Random matricies, not pd:
B1 = np.random.randn(n,m)
B2 = np.random.randn(n,m)
C1 = np.random.randn(m,m)
C2 = np.random.randn(m,m)

#Define e,h vectors:
e1 = np.random.randn(n)
e2 = np.random.randn(m)
h1 = np.random.randn(m)
h2 = np.random.randn(n)

#Convert Matrices into tensors
A1 = torch.tensor(A1, dtype = torch.float)
A2 = torch.tensor(A2, dtype = torch.float)
B1 = torch.tensor(B1, dtype = torch.float)
B2 = torch.tensor(B2, dtype = torch.float)
C1 = torch.tensor(C1, dtype = torch.float)
C2 = torch.tensor(C2, dtype = torch.float)
e1 = torch.tensor(e1, dtype = torch.float)
e2 = torch.tensor(e2, dtype = torch.float)
h1 = torch.tensor(h1, dtype = torch.float)
h2 = torch.tensor(h2, dtype = torch.float)

x1 = torch.ones((n, 1), requires_grad=True)
x2 = torch.ones((m, 1), requires_grad=True)

#Generic Quadratic Cost:
#B_ij, C_ij still rather vague
def f1(x1,x2):
    return (0.5 * x1.t() @ A1 @ x1) + (x1.t() @ B1 @ x2) + (0.5 * x2.t() @ C1 @ x2) + (e1.t() @ x1) + (h1.t() @ x2)

def f2(x1,x2):
    return (0.5 * x2.t() @ A2 @ x2) + (x2.t() @ B2 @ x1) + (0.5 * x1.t() @ C2 @ x1) + (e2.t() @ x2) + (h2.t() @ x1)
```

```python
#Analytical Gradient:
#D wrt x1:
def D1f1(x1,x2):
    return (A @ x1) + 0.5 * (B1 @ x2) + e1

def D1f2(x1,x2):
    return 0.5 * (B2.t() @ x2) + (C2 @ x1) + h2

#D wrt x2:
def D2f1(x1,x2):
    return 0.5 * (B1.t() @ x1) + (C1 @ x2) + h1

def D2f2(x1,x2):
    return (A2 @ x2) + 0.5 * (B2 @ x1) + e2

#Analytical Hessian:
#H wrt x1:
def H11f1(x1, x2):
    return A1

def H11f2(x1, x2):
    return C2

#H wrt x2:
def H22f1(x1, x2):
    return C1

def H22f2(x1, x2):
    return A2

```

```python
#Computational Gradient:
#tensors = [tensor.zero_grad() for tensor in tensors]

'''
-Possible solutions:
-Make functions just expressions
-use backward
-Seeing an example would be nice
'''
print(f1(x1,x2).grad)

#Computational Hessian:
#print(torch.autograd(x1).autograd(x1))
#print(torch.autograd(x2).autograd(x2))
```

    None

    /projects/ac66b0d1-d01e-42cc-9d9b-3799ed5ac4d1/.local/lib/python3.6/site-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
      warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "

```python

```