## Topics covered in this notebook:
1. What is Classification?.
2. Logistic Regression.
3. Optimal Weights.
    1. Closed form solution - Linear Discriminant Analysis(LDA).
    2. Cross-entropy function.
    3. Maximizing the likelihood.
    4. Gradient Descent.
4. L2 Regularization.
5. L1 Reguarization.
6. Advantages/Disadvantages.
7. References.

## 1. Classification Problem:

Consider a set of questions:
1. What digit is this? (MNIST data).
2. Is it a cat or a dog?
3. Will an online customer make a purchase?

What do all these problems in common?
1. Categorical **outputs**, called **labels** (Eg. Yes/No, Cat/Dog etc.).
2. Assigning each input vector to one of a finite number of labels is called **classification**.
3. **Binary classification** - Two possible labels. Simplest form of classification.
4. **Multi-class classification** - Multiple possible labels.

One such classifier is **Logistic Regression**.

**Example - Simple 2-dimenstional classification:**
Consider the following figure:

1. There are 2 categories 'purple' & 'yellow'.
2. These 2 categories are separated by the blue decision line (x-y = 0).
3. Lets consider a point (2,1) indicated by red on the figure. How can we classify this?
    1. h(x,y) = x - y = 2 - 1 = 1 > 0 -> Hence 'yellow'.
4. Lets consider a point (1,2) indicated by black on the figure. How can we classify this?
    1. h(x,y) = x - y = 1 - 2 = -1 < 0 -> Hence 'purple'.
4. Lets consider a point (1,1) indicated by green on the figure. How can we classify this?
    1. h(x,y) = x - y = 1 - 1 = 0 -> Don't know.

From the above example to we formalize the variables in the following way:

1. Call (x,y) -> (x1,x2).
2. Rename the constants to w.
3. Rename the bias/intercept term w0.
4. h(x) = w0 + w1x1 + w2x2. -> h is a **linear combination** of x.
5. Vector form: h(x) = wTx.
6. In 3-dimensions: line -> plane. >3 dimensions: hyperplane.

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

## 2. Logistic Regression:

Consider the schematic below from Simon Haykin's book on neural network:

Schematic for logistic regression looks lot like a single neuron.There are inputs and weights associated with each input. Each inputs are then multiplied and goes through a summing junction. The output of the summing junction goes through some non-linear function.

In the linear classification example above we looked at the sign of the output to classify each points. But in logistic regression we replace it with **sigmoid** or **logistic** function.

what does sigmond function look like?

1. In general: S - shaped curve.
2. Finite limit as x approaches infinitiy & negative infinity.
3. There are few variations of sigmoid function:
    1. Hyperbolic tangent - tanh(x) E (-1,1) & y-intercept is 0.
    2. Binary sigmoid:
    \begin{align}
    \large \sigma(z)\, &= \large \frac{1}{1\,+\,exp(-z)}\,  \epsilon \,(0,1)\,,\,y-intercept\,is\,0.5.\\
    \end{align}

#### Output of a logistic regression:
\begin{align}
\large \sigma(w^Tx)\, &= \large \frac{1}{1\,+\,exp(-w^Tx)}\,\\
\end{align}
1. If the inner product of w.dot(x) is positive we get a number close to +1.
2. If the inner product of w.dot(x) is negative we get a number close to 0.
3. If the inner product of w.dot(x) is zero we get 0.5.

So based on this we classify anything above 0.5 as class 1 and below 0.5 is class 0. This process of going from inputs to getting output from a sigmoid function is called **feedforward** process.

## 3. Optimal Weights:

### Special case of logistic regression:
In this case we can find a closed form solution. In general we cannot and we will have to use gradient descent.

1. Consider 2 Gaussian clouds.
2. Both have the same covariance but differnt mean.
3. Mulivariate Gaussian PDF:
\begin{align}
\large p(x)\, &= \large \frac{1}{(2 \pi)^D \lvert \large \Sigma \rvert}\, e^{-\frac {1}{2}(x-\mu)^T \Sigma^-1 (x-\mu)}\\
\end{align}

```python
import numpy as np
import matplotlib.pyplot as plt
% matplotlib inline

N = 100 # Number of samples
D = 2 # Number of features

X = np.random.randn(N,D)

# center the first 50 points at (-2,-2)
X[:50,:] = X[:50,:] - 2*np.ones((50,D))

# center the last 50 points at (2, 2)
X[50:,:] = X[50:,:] + 2*np.ones((50,D))

# labels: first 50 are 0, last 50 are 1
T = np.array([0]*50 + [1]*50)

plt.scatter(X[:,0], X[:,1], c=T, s=100, alpha=0.5)

# To be continued
print('To be continued')
```

### 3.A Closed Form Solution or LDA:
1. Lets look at **Bayes' Rule**:
\begin{align}
\large p(Y\,|\,X)\, &= \large \frac {p(X\,|\,Y)\,p(Y)}{p(X)}\\
\end{align}
Example:
\begin{align}
\large p(Y=1\,|\,X)\, &= \large \frac {p(X\,|\,Y=1)\,p(Y=1)}{p(X)}\\
\large p(Y=0\,|\,X)\, &= \large \frac {p(X\,|\,Y=0)\,p(Y=0)}{p(X)}\\
\end{align}
where:

    1. p(X | Y) is the Gaussian - Calculate it over all the data from the class Y.
    2. p(Y) is the frequencey estimate of Y. # class 1 appeared/ # total.

2. Manipulate Bayes' Rule & put it into logistic regression framework:

\begin{align}
\large p(y=1\,|\,x)\, &= \large \frac {p(x\,|\,y=1)\,p(y=1)}{p(x)} = \large \frac {p(x\,|\,y=1)\,p(y=1)}{p(x\,|\,y=1)p(y=1)\,+\, p(x\,|\,y=0)p(y=0)} \\
\end{align}

3. Divide top & bottom by p(x|y=1)p(y=1):

\begin{align}
\large p(y=1\,|\,x)\, = \large \frac {1}{1\,+\, \frac {p(x\,|\,y=0)p(y=0)}{p(x\,|\,y=1)p(y=1)}} \\
\end{align}

Looks a lot like **logistic regression sigmoid**.

\begin{align}
\large p(y=1\,|\,x)\, = \large \frac {1}{1\,+\, \large \frac {p(x\,|\,y=0)p(y=0)}{p(x\,|\,y=1)p(y=1)}} = \large \frac{1}{1\,+\,exp(-w^Tx)}\,\\
\end{align}

4. From the comparison above, we can write:

\begin{align}
\large (-w^Tx) = \large ln \left(\frac {p(x\,|\,y=0)p(y=0)}{p(x\,|\,y=1)p(y=1)}\right) \,\\
\end{align}

5. Let p(y=1) = 1 - $\alpha$ & p(y=0) = $\alpha$

\begin{align}
ln \left(\frac {p(x\,|\,y=0)p(y=0)}{p(x\,|\,y=1)p(y=1)}\right)\,&=\, ln(p(x\,|\,y=0)) \,+\, ln \alpha - ln\, p(x\,|\,y=1)\,-\,ln (1- \alpha)\\
&=  \frac{1}{(2 \pi)^D \lvert  \Sigma \rvert}\, e^{-\frac {1}{2}(x-\mu_0)^T \Sigma^-1 (x-\mu_0)} -
    \frac{1}{(2 \pi)^D \lvert  \Sigma \rvert}\, e^{-\frac {1}{2}(x-\mu_1)^T \Sigma^-1 (x-\mu_1)} +
    ln \frac {\alpha}{1-\alpha}\\
&= - \frac {1}{2} (x^T\Sigma^{-1}x - x^T\Sigma^{-1}\mu_0 - \mu_0^T\Sigma^{-1}x + \mu_0^T\Sigma^{-1}\mu_0) +
     \frac {1}{2} (x^T\Sigma^{-1}x - x^T\Sigma^{-1}\mu_1 - \mu_1^T\Sigma^{-1}x + \mu_1^T\Sigma^{-1}\mu_1) +
     ln \frac {\alpha}{1-\alpha}\\
&= \mu_0^T\Sigma^{-1}x - \mu_1^T\Sigma^{-1}x - \frac {1}{2} \mu_0^T\Sigma^{-1}\mu_0 + \mu_1^T\Sigma^{-1}\mu_1 + ln \frac{\alpha}    {1-\alpha}\\
&= (\mu_0^T \,-\, \mu_1^T)\Sigma^-1x - \frac {1}{2} \mu_0^T\Sigma^{-1}\mu_0 + \mu_1^T\Sigma^{-1}\mu_1 + ln \frac{\alpha}{1-        \alpha} \\
& = -(w^Tx + b)
\end{align}

where,

\begin{align}
\large w^T &= \large (\mu_1^T \,-\, \mu_0^T)\Sigma^{-1}x \\
\large b &=  \large \frac {1}{2} \mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1-+ ln \frac{\alpha}{1-\alpha} \\
\end{align}

Comments on the math above:
    1. 1/sqrt term cancels out.
    2. Quadratic terms cancels out.
    3. Covariance & Inverse are symmetric.

This method is also called as **Linear Discriminant Analysis (LDA)**.

Final Comments:
1. If covariance is diagonal matrix -> It is an instance of Naive Bayes.
2. If we have differenct covariances, it is called Quadratic Discriminant Analysis(QDA) as the quadratic term won't cancel out.
3. Above solution is valid only for special cases like this.

For the example above, if we calculate weights we get (4,4) & bias is 0.

```python
# Continuing from the above example:
# add a column of ones

ones = np.ones((N, 1))
Xb = np.concatenate((ones, X), axis=1)

def sigmoid(z):
    return 1/(1 + np.exp(-z))

# closed-form solution from the calculation above.
w = np.array([0, 4, 4])

# calculate the model output
z = Xb.dot(w)
Y = sigmoid(z)

plt.scatter(X[:,0], X[:,1], c=T, s=100, alpha=0.5)

x_axis = np.linspace(-6, 6, 100)
y_axis = -x_axis
plt.plot(x_axis, y_axis)
plt.show()
```

### 3.B Cross-Entropy Function:
1. In linear regression we used squared error as the cost function. Assumption here is Gaussian-distributed error.
2. Logistic error can't be Gaussian distributed because:
    1. Target is only 0/1.
    2. Output is only a number between 0-1.
3. We want 0:if correct, >0 if not correct, more wrong == bigger cost. This accomplished by using **cross-entropy error fucntion**.

\begin{align}
\large J \,=\, - [tlog(y) \,+\, (1-t)log(1-y)]\\
\end{align}

    1. For each input only one term matters. Because if t = 1 only 1st term matters & vice-versa.
    2. Also log(y) is always between 0 and $-\infty $. Hence the negative sign in the cost function.
    3. Example:
        1. t = 1, y = 1 -> 0.
        2. t = 0, y = 0 -> 0.
        3. t = 1, y = 0.9 -> 0.11.
        4. t = 1, y = 0.5 -> 0.69.
        5. t = 1, y = 0.1 -> 2.3. -> Notice how the error increase.
4. For multiple training samples -> we sum all the error:

\begin{align}
\large J \,=\, - \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n)\\
\end{align}

### 3.C Maximizing the Likelihood:
Consider a coin toss example:

p(H) = p

p(T) = 1-p

N = 10 -> 7H & 3T

Likelihood = p^7 (1-p)^3

Maximize Likelihood: Take derivative & set it to zero. We get p=0.7 that maximizes L. Notice this is same as p(H).

Apply this to Logistic Regression:

\begin{align}
\large p(y=1 \,|\, x) \,&=\, \large \sigma(w^TX) =\large y \\
\large L \,&=\, \large \prod_{n=1}^N\, y_n^{t_n} (1-y)^{t_n} \\
\end{align}

This works because if the target is 1 p = y else p = 1-y.
Take log likelihood:

\begin{align}
\large l \,=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n)\\
\end{align}

We get the same form as **Cross-Entropy** function. They are just negatives of each other. So maximizing the log-likelihood is same as minimizing the cross-enropy function.

### 3.D Gradient Descent:
If we assume the date is Gaussian-distributed with equal covariance, we can use the Bayes' method. But generally that is not the case. This is when gradient descent is very useful as it is more general.

\begin{align}
\large J \,=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n)\\
\end{align}

Split into 3 derivatives:

\begin{align}
\large \frac {\partial J}{\partial w_i} \,&=\, \large \Sigma_{n=1}^N\, \frac {\partial J}{\partial y_n}
                                                                      \frac {\partial y_n}{\partial a_n}
                                                                      \frac {\partial a_n}{\partial w_i}\\
\large a_n &= \large w^Tx_n\\
\end{align}

\begin{align}
\large \frac {\partial J}{\partial y_n} \,&=\, - \large \Sigma_{n=1}^N\, t_n \frac {1}{y_n} \,+\, (1-t_n) \frac {1}{1-y_n}(-1)\\
\end{align}

Calculate derivative of y w.r.t a:

\begin{align}
\large y_n \,&=\, \large \sigma(a_n) \,=\, \frac {1}{1+e^{-a_n}}\\
\large \frac {\partial y_n}{\partial a_n} \,&=\, \large \frac {-1}{(1+e^{-a_n})^2}(e^{-a_n})(-1)\\
                                           \,&=\, \large \frac {e^{-a_n}}{(1+e^{-a_n})^2}\\
                                           \,&=\, \large \frac {1}{(1+e^{-a_n})} \,\, \frac {e^{-a_n}}{(1+e^{-a_n})}\\
                                           \,&=\, \large y_n(1-y_n)\\
\end{align}

Calculate derivative of a w.r.t to w:

\begin{align}
\large a_n \,&=\, \large w^Tx_n\\
\large a_n \,&=\, \large w_0x_{n0} \,+\, w_1x_{n1} \,+\, ....\\
\large \frac {\partial a_n}{\partial w_i} \,&=\, \large x_{ni}
\end{align}

Put this all together:

\begin{align}
\large \frac {\partial J}{\partial w_i} \,&=\, - \large \Sigma_{n=1}^N\, t_n \frac {1}{y_n} y_n(1-y_n) x_{ni} \,-\,                                                              \frac {(1-t_n) }{1-y_n}
                                                 \large y_n(1-y_n)
                                                 \large x_{ni}\\
                                          &=\, - \large \Sigma_{n=1}^N\, t_n (1-y_n) x_{ni} \,-\,(1-t_n)
                                                 \large y_n
                                                 \large x_{ni}\\
                                           &=\   \large \Sigma_{n=1}^N\, (y_n \,-\, t_n)x_{ni}
\end{align}

If we vectorize:

\begin{align}
\large \frac {\partial J}{\partial w} \,&=\   \large \Sigma_{n=1}^N\, (y_n \,-\, t_n)x_{n}\\
\end{align}

Vectorize More:

\begin{align}
\large \frac {\partial J}{\partial w} \,&=\   \large X^T(Y-T)\\
\end{align}

Bias Term:

\begin{align}
\large \frac {\partial J}{\partial w_0} \,&=\   \large \Sigma_{n=1}^N\, (y_n \,-\, t_n)x_{n0} = \large \Sigma_{n=1}^N\, (y_n \,-\, t_n) \\
\end{align}

## 4. L2 Regularization:

Consider the example from section 3A - Closed form solution. From there,
1. We found the exact Bayesian solution was w = [0,4,4]
2. This represents a line 0 + 4x + 4y = 0 -> y = -x.
3. Slop of -1 & y-intercept: 0.
4. Why is solution (4,4)? Why not (1,1) or (10,10)? -> All these represent the same line.
    1. If you look at (0,4,4): y = 0.99966; J = -0.000335.
    2. If you look at (0,1,1): J = -0.12.
    3. If you look at (0,10,10): J = -2.06e-09.
    4. This explains why (0,4,4) was chosen. Also ideally we would want (0,inf, inf) for the cost to be 0.
5. Because of this logistic regression would still to go to w = (0,inf, inf).
6. We need regularization that penalizes large weights.

\begin{align}
\large J \,&=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n)\\
\large J_{reg} \,&=\, \large J + \left(\frac {\lambda}{2}\right)\lvert\lvert \large w^2\rvert\rvert\\
\end{align}

7. Now we won't try yo go to (0,10,10).
8. $\large \lambda$ - Smoothing parameter (~0.1,1).

### Solving for w:
1. Use gradient descent.
2. But we have to calculate the gradien for $\large \frac {\partial J_{reg}}{\partial w_{i}}$.
3. $\large \frac {\partial J_{reg}}{\partial w_{i}} = \large \frac {\partial J}{\partial w_{i}} + \large \frac {\partial reg-cost}{\partial w_{i}}$.
4. $\large \frac {\partial reg-cost}{\partial w_{i}} = \large \lambda w_i$.
5. In vector form: $\large \frac {\partial reg-cost}{\partial w} = \large \lambda w$.
6. Full form:

\begin{align}
\large \frac {\partial J_{reg}}{\partial w_{i}} = \large X^T(Y-T) + \lambda w
\end{align}

### Probabilistic Perspective:

1. Cross-entropy maximizes the likelihood, since J = -log(likelihood).
2. Definition of J with regularization:

\begin{align}
\large J \,&=\,  \large \, tlog(y) \,+\, (1-t)log(1-y) - \left(\frac {\lambda}{2}\right)\lvert\lvert \large w^2\rvert\rvert\\
\end{align}

3. Take exponent on both sides:

\begin{align}
\large e^{J} \,&=\, \large y^t(1-y)^{1-t}+ e^{\left(-\frac {\lambda}{2}\right)\lvert\lvert \large w^2\rvert\rvert}\\
\end{align}

4. Expoentiate the first part: Bernoulli(likelihood) t ~ Bernoulli(y).
5. Expoentiate the 2nd part: Gaussian(prior) w ~ N(0,1/$ \lambda$).
6. **Posterior $\propto$ Likelihood * Prior **.
7. Our 'prior belief' about w is that its Gaussian distributed with variance $\frac {1}{\large \lambda}$.
8. Without regulariztion we are maximizing the likelihood.
9. With regularization we are **maximizing the posterior or MAP**.

## 5. L1 Regularization
In general we want no. of features(D) << no. of samples(N). Also, there might be a case where some of the features are just constant or quasi-const, uncorrelated to output etc. In all these cases we could use L1 regularization to eliminate those features from the model.

1. Select a small number of important features that actually predict the trend.
2. Eliminate the noise influence on the output.
3. Similar to L2. It has a penalty term using L1 norm.

\begin{align}
\large J_{RIDGE} \,&=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n) + \left(\frac {\lambda}{2}\right)\lvert\lvert \large w^2\rvert\rvert\\
\large J_{LASSO} \,&=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n) + \left(\frac {\lambda}{2}\right)\lvert\lvert \large w\rvert\rvert\\
\end{align}

4. Similar to L2 this also puts a prior on w, so its also a MAP estimation of w. We had a gaussian distribution on the prior for L2. We have Laplace here:
\begin{align}
\large p(w) \, &= \large \frac {\lambda}{2} exp(- \lambda \lvert \large w\rvert) \\
\end{align}

5. Taking derivative of cost function:
\begin{align}
\large J \,&=\, \large \Sigma_{n=1}^N\, t_nlog(y_n) \,+\, (1-t_n)log(1-y_n) + \lvert\lvert \large w\rvert\rvert\\
\\
\large \frac {\partial J_{reg}}{\partial w_{i}} &= \large X^T(Y-T) + \lambda sign(w)\\
\end{align}

6. Use gradient descent and continue the same way.

## 6. Advantages & Disadvantages:
### Advantages:
1. Easily extended to mutiple classes.
2. Natural probabilistic view of class prerdictions.
3. Quick to train.
4. Fast at classification.
5. Good accuracy for many simple data sets.
6. Ressitant to overfitting.
7. Can interpret model coefficients as indicators of feature importance.

### Less good:
1. Linear decision boundary. Too simple for complex problems.
2. Only binary classification.

## 7. References:
1. An Introduction to Statistical Learning Textbook by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.
2. University of Michigan EECS 445 - Machine Learning Course (https://github.com/eecs445-f16/umich-eecs445-f16).

3. University of Toronto CSC 411 - Intro. to Machine Learning (http://www.cs.toronto.edu/~urtasun/courses/CSC411_Fall16/CSC411_Fall16.html).

4. Stanford CS109 - Intro. to proabability for computer scientists (https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/).

5. Few online courses on Udemy, Coursera etc.


# Animating a simple wave

We'll plot at various times a wave $u(x,t)$ that starts as a triangular shape as in Taylor Example 16.1, and then animate it.  We can imagine this as simulating a wave on a taut string.  Here $u$ is the transverse displacement (i.e., $y$ in our two-dimensional plots).  We are not solving the wave equation as a differential equation here, but starting with $u(x,0) \equiv u_0(x)$ and plotting the solution at time $t$:

$\begin{align}
  u(x,t) = \frac12 u_0(x - ct) + \frac12 u_0(x + ct)
  \;,
\end{align}$

which *is* the solution to the wave equation starting with $u_0(x)$ at time $t=0$.

We have various choices for animation in a Jupyter notebook.  We will consider two possibilities, which both use `FuncAnimation` from `matplotlib.animation`.
1. Make a javascript movie that is then displayed inline with a movie-playing widget using `HTML`.  We use
`%matplotlib inline` for this option and use `%%capture` to prevent the figure from displaying prematurely.
2. Update the figure in real time (so to speak) by using `%matplotlib notebook`, which creates active figures that we can modify after they are displayed.

We'll do the first option here. We should define at least one class for the animation.

v1: Created 25-Mar-2019. Last revised 27-Mar-2019 by Dick Furnstahl (furnstahl.1@osu.edu).

```python
%matplotlib inline
```

To use option 2: uncomment `%matplotlib notebook` here and `fig.show()` just after we define `anim`.  Comment  out `%%capture` and `HTML(anim.to_jshtml())` below.

```python
#%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt

from matplotlib import animation, rc
from IPython.display import HTML
```

First define functions for the $t=0$ wave function form (here a triangle) and for the subsequent shape at any time $t$ based on the wave speed `c_wave`.

```python
def u_0_triangle(x_pts, height=1., width=1.):
    """Returns a triangular wave of amplitude height and width 2*width.
    """
    y_pts = np.zeros(len(x_pts))  # set the y array to all zeros
    for i, x in enumerate(x_pts):
        if x < width and x >= 0.:
            y_pts[i] = -(height/width) * x + height
        elif x < 0 and x >= -width:
            y_pts[i] = (height/width) * x + height
        else:
            pass   # do nothing (everything else is zero already)
    return y_pts

```

```python
def u_triangle(x_pts, t, c_wave = 1., height=1., width=1.):
    """Returns the wave at time t resulting from a triangular wave of
       amplitude height and width 2*width at time t=0.  It is the
       superposition of two traveling waves moving to the left and the right.
    """
    y_pts = u_0_triangle(x_pts - c_wave * t) / 2. + \
            u_0_triangle(x_pts + c_wave * t) / 2.
    return y_pts
```

```python
# Set up the array of x points (whatever looks good)
x_min = -5.
x_max = +5.
delta_x = 0.01
x_pts = np.arange(x_min, x_max, delta_x)

```

First look at the initial ($t=0$) wave form.

```python
# Define the initial (t=0) wave form and the wave speed.
height = 1.
width = 1.
c_wave = 1.

# Make a figure showing the initial wave.
t_now = 0.

fig = plt.figure(figsize=(6,2), num='Triangular wave')
ax = fig.add_subplot(1,1,1)
ax.set_xlim(x_min, x_max)
gap = 0.1
ax.set_ylim(-gap, height + gap)
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'$u_0(x)$')
ax.set_title(rf'$t = {t_now:.1f}$')

line, = ax.plot(x_pts,
                u_triangle(x_pts, t_now, c_wave, height, width),
                color='blue', lw=2)

fig.tight_layout()

```

Next make some plots at an array of time points.

```python
t_array = np.array([0., 1,  2.,  3,  4.,  -2.])

fig_array = plt.figure(figsize=(12,6), num='Triangular wave')

for i, t_now in enumerate(t_array):
    ax_array = fig_array.add_subplot(3, 2, i+1)
    ax_array.set_xlim(x_min, x_max)
    gap = 0.1
    ax_array.set_ylim(-gap, height + gap)
    ax_array.set_xlabel(r'$x$')
    ax_array.set_ylabel(r'$u_0(x)$')
    ax_array.set_title(rf'$t = {t_now:.1f}$')

    ax_array.plot(x_pts,
                  u_triangle(x_pts, t_now, c_wave, height, width),
                  color='blue', lw=2)

fig_array.tight_layout()
fig_array.savefig('Taylor_Example_16p1_triangle_waves.png',
                   bbox_inches='tight')
```

Now it is time to animate!

```python
# Set up the t mesh for the animation.  The maximum value of t shown in
#  the movie will be t_min + delta_t * frame_number
t_min = 0.   # You can make this negative to see what happens before t=0!
t_max = 15.
delta_t = 0.1
t_pts = np.arange(t_min, t_max, delta_t)
```

We use the cell "magic" `%%capture` to keep the figure from being shown here.  If we didn't the animated version below would be blank.

```python
%%capture

fig_anim = plt.figure(figsize=(6,2), num='Triangular wave')
ax_anim = fig_anim.add_subplot(1,1,1)
ax_anim.set_xlim(x_min, x_max)
gap = 0.1
ax_anim.set_ylim(-gap, height + gap)

# By assigning the first return from plot to line_anim, we can later change
#  the values in the line.
line_anim, = ax_anim.plot(x_pts,
                          u_triangle(x_pts, t_min, c_wave, height, width),
                          color='blue', lw=2)

fig_anim.tight_layout()
```

```python
def animate_wave(i):
    """This is the function called by FuncAnimation to create each frame,
        numbered by i.  So each i corresponds to a point in the t_pts
        array, with index i.
    """
    t = t_pts[i]
    y_pts = u_triangle(x_pts, t, c_wave, height, width)

    line_anim.set_data(x_pts, y_pts)  # overwrite line_anim with new points
    return (line_anim,)   # this is needed for blit=True to work
```

```python
frame_interval = 40.  # time between frames
frame_number = 100    # number of frames to include (index of t_pts)
anim = animation.FuncAnimation(fig_anim,
                               animate_wave,
                               init_func=None,
                               frames=frame_number,
                               interval=frame_interval,
                               blit=True,
                               repeat=False)

#fig.show()
```

```python
HTML(anim.to_jshtml())

```

<link rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/
css/font-awesome.min.css">

    <input id="_anim_slider43209db1a6514510b20024e10eb5baa6" type="range" style="width:350px"
           name="points" min="0" max="1" step="1" value="0"
           onchange="anim43209db1a6514510b20024e10eb5baa6.set_frame(parseInt(this.value));"></input>

    <button onclick="anim43209db1a6514510b20024e10eb5baa6.slower()"></button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.first_frame()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.previous_frame()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.reverse_animation()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.pause_animation()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.play_animation()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.next_frame()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.last_frame()">
        </button>
    <button onclick="anim43209db1a6514510b20024e10eb5baa6.faster()"></button>
  <form action="#n" name="_anim_loop_select43209db1a6514510b20024e10eb5baa6" class="anim_control">
    <input type="radio" name="state"
           value="once" checked> Once </input>
    <input type="radio" name="state"
           value="loop" > Loop </input>
    <input type="radio" name="state"
           value="reflect" > Reflect </input>
  </form>

```python

```


# Lecture 14

## Systems of Differential Equations I:

### Linear Systems

```python
import numpy as np
import sympy as sp
import scipy.integrate
sp.init_printing()
##################################################
##### Matplotlib boilerplate for consistency #####
##################################################
from ipywidgets import interact
from ipywidgets import FloatSlider
from matplotlib import pyplot as plt

import cmath

%matplotlib inline

from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')

global_fig_width = 8
global_fig_height = global_fig_width / 1.61803399
font_size = 12

plt.rcParams['axes.axisbelow'] = True
plt.rcParams['axes.edgecolor'] = '0.8'
plt.rcParams['axes.grid'] = True
plt.rcParams['axes.labelpad'] = 8
plt.rcParams['axes.linewidth'] = 2
plt.rcParams['axes.titlepad'] = 16.0
plt.rcParams['axes.titlesize'] = font_size * 1.4
plt.rcParams['figure.figsize'] = (global_fig_width, global_fig_height)
plt.rcParams['font.sans-serif'] = ['Computer Modern Sans Serif', 'DejaVu Sans', 'sans-serif']
plt.rcParams['font.size'] = font_size
plt.rcParams['grid.color'] = '0.8'
plt.rcParams['grid.linestyle'] = 'dashed'
plt.rcParams['grid.linewidth'] = 2
plt.rcParams['lines.dash_capstyle'] = 'round'
plt.rcParams['lines.dashed_pattern'] = [1, 4]
plt.rcParams['xtick.labelsize'] = font_size
plt.rcParams['xtick.major.pad'] = 4
plt.rcParams['xtick.major.size'] = 0
plt.rcParams['ytick.labelsize'] = font_size
plt.rcParams['ytick.major.pad'] = 4
plt.rcParams['ytick.major.size'] = 0
##################################################
```

Our aim is to solve systems of equations of the form:

$$\frac{{\rm d}y_i}{{\rm d}t} = f_i(y_1,\ldots,y_n,t),$$

for $\displaystyle i=1,\ldots,n$.

Let us first consider the simplest form: a $2\times2$ **linear** system

  \begin{eqnarray*}
    \frac{{\rm d}x}{{\rm d}t} &=& ax+by,\\
    \frac{{\rm d}y}{{\rm d}t} &=& cx+dy.
  \end{eqnarray*}

## Motivation

- Lotka-Volterra (predator-prey)

- SIR model (epidemics)

- Chemical reaction networks

- Heart cell dynamics

In order to understand these systems, we must first understand coupled linear systems.

## Recap of eigenvalues

$A = \left(\begin{matrix} 1 & 1\\2 & 0  \end{matrix}\right)$

```python
A = sp.Matrix([[1, 1],[2, 0]])
A.eigenvals()

```

## Recap of eigenvectors

$A = \left(\begin{matrix} 1 & 1\\2 & 0  \end{matrix}\right)$

```python
A.eigenvects()

```

## Recap of diagonalisation

Recall that, for a matrix $A$ with **eigenvectors** $\mathbf{v}_1$ and $\mathbf{v}_2$, and **eigenvalues** $\lambda_1$ and $\lambda_2$, we can write a matrix of eigenvectors: $P = \left(\begin{matrix} \mathbf{v}_1 & \mathbf{v}_2  \end{matrix}\right)$. Then:

$$A = P \left(\begin{matrix} \lambda_1 & 0 \\ 0&
      \lambda_2  \end{matrix}\right) P^{-1}$$

(This is also true for general $n \times n$ matrices.)

In our example, $\qquad P = \left(\begin{matrix} -\frac{1}{2} & 1 \\
                                1& 1 \end{matrix}\right),\qquad\lambda_1 = -1,\qquad\lambda_2=2.$

## Coupled ODEs

For coupled system of first order linear differential equations of the form

  \begin{eqnarray*}
    \frac{{\rm d}x}{{\rm d}t} &=& ax+by,\\
    \frac{{\rm d}y}{{\rm d}t} &=& cx+dy.
  \end{eqnarray*}

we have three methods of analysing them mathematically:
  - Turn them into one second order equation (if we can solve second order!)
  - Divide one by other, to get one equation independent of $t$
  - Perform matrix diagonalisation (extends to $n \times n$ problems)

## Example

Solve
  \begin{eqnarray*}
    \frac{{\rm d}x}{{\rm d}t} &=& x+y,\\
    \frac{{\rm d}y}{{\rm d}t} &=& 2x.
  \end{eqnarray*}

Subject to
  \begin{eqnarray*}
    x(0) &=& 0,\\
    y(0) &=& 3.
  \end{eqnarray*}

## Method 1: Second order

We start with:

  \begin{eqnarray*}
    \frac{{\rm d}x}{{\rm d}t} &=& x+y,\\
    \frac{{\rm d}y}{{\rm d}t} &=& 2x.
  \end{eqnarray*}

We can convert that into a second order equation:

$$
\frac{{\rm d}^2x}{{\rm d}t^2}
=
\frac{{\rm d}x}{{\rm d}t} + \frac{{\rm d}y}{{\rm d}t}
=
\frac{{\rm d}x}{{\rm d}t} + 2x
\quad \quad\implies \quad\quad
\boxed{\frac{{\rm d}^2x}{{\rm d}t^2}
=
\frac{{\rm d}x}{{\rm d}t} + 2x}
$$

## Method 2: eliminate $t$

We start with:

  \begin{eqnarray*}
    \frac{{\rm d}x}{{\rm d}t} &=& x+y,\\
    \frac{{\rm d}y}{{\rm d}t} &=& 2x.
  \end{eqnarray*}

Then, dividing:

$$\frac{{\rm d}y}{{\rm d}x}
=
\frac{ \quad \frac{{\rm d}y}{{\rm d}t} \quad }{ \frac{{\rm d}x}{{\rm d}t} }
\quad \quad\implies \quad\quad
\boxed{\frac{{\rm d}y}{{\rm d}x}
=
\frac{2x}{x+y} }
$$

## Method 3: diagonalisation

Let $\mathbf{v}=\left(\begin{matrix}x\\y\end{matrix}\right),$

then
$$
    \frac{{\rm d}x}{{\rm d}t} = x+y,\quad
    \frac{{\rm d}y}{{\rm d}t} = 2x, \quad \implies \quad \boxed{\frac{{\rm d}\mathbf{v}}{{\rm d}t} = A\mathbf{v},}
$$

where  $A = \left(\begin{matrix} 1 & 1\\2 & 0  \end{matrix}\right).$

Substitute
$$
A = P \left(\begin{matrix} \lambda_1 & 0 \\ 0 &
      \lambda_2  \end{matrix}\right) P^{-1}.
$$

then
$$
\frac{{\rm d}\mathbf{v}}{{\rm d}t} = P \left(\begin{matrix} \lambda_1 & 0 \\ 0 &
       \lambda_2  \end{matrix}\right) \left( P^{-1}\mathbf{v} \right)\qquad
$$

or

$$
\frac{{\rm d}}{{\rm d}t}(P^{-1} \mathbf{v}) = \left(\begin{matrix} \lambda_1 & 0 \\ 0 &
       \lambda_2  \end{matrix}\right) \left( P^{-1}\mathbf{v} \right)
$$

We can now introduce a new variable $\;\mathbf{z} =    \left(\begin{matrix} z_1 \\ z_2\end{matrix}\right)   = P^{-1} \mathbf{v}\;$ so that:

$$
\frac{{\rm d}\mathbf{z}}{{\rm d}t} = \left(\begin{matrix} \lambda_1 & 0 \\ 0 &
       \lambda_2  \end{matrix}\right) \mathbf{z}.
$$

But now, because the matrix is diagonal, the system is no longer coupled.
The first equation **only** involves $z_1$ and the second **only** involves $z_2$, so we can solve each one individually:

$$\frac{{\rm d}z_1}{{\rm d}t} = \lambda_1 z_1 \qquad\implies\qquad z_1(t) = A\,e^{\lambda_1 t}$$

$$\frac{{\rm d}z_2}{{\rm d}t} = \lambda_2 z_2 \qquad\implies\qquad z_2(t) = B\,e^{\lambda_2 t}$$

Finally, we can substitute $z_1$ and $z_2$ back in terms of $x$ and $y$ to find the solution to the original coupled system:

We have

$$\mathbf{z} = \left(\begin{matrix} z_1 \\ z_2\end{matrix}\right) \quad = \quad P^{-1} \mathbf{v} \quad = \quad \frac{1}{3}\left(\begin{matrix} -2 & 2 \\ 2& 1  \end{matrix}\right) \left(\begin{matrix} x \\ y\end{matrix}\right) \quad = \quad \left(\begin{matrix} A\,e^{\lambda_1 t} \\ B\,e^{\lambda_2 t}\end{matrix}\right) $$

Rearranging, we now have two equations relating $x$ and $y$:

$$-2x + 2y = C\,e^{\lambda_1 t}$$

$$2x + y = D\,e^{\lambda_2 t}$$

where $C=3A$ and $D=3B$.  Using our initial conditions, $\,x(0)=0\,$ and $\,y(0)=3\,$ we find $C=6$ and $D=3$.

Finally, solving the simultaneous equations, we have a solution:

$$x(t) = -e^{\lambda_1 t} + e^{\lambda_2 t}$$

$$y(t) = 2e^{\lambda_1 t} + e^{\lambda_2 t}$$

## Summary

- Three methods to analytically solve systems of linear first
  order ODEs
- Best method depends on the system and what you need to ask about it


## Example
Let's begin with an example. Suppose that you conduct an opinion survey amongst pilots. In the survey, you ask them basic demographics (gender, race, etc) and whether they agree/disagree with a statement on a scale.

You then have a set of categorical data with which you can compare responses to questions between demographics (or responses). Here we can compare pilot gender with the sense of discrimination within the field.

```R
library("gmodels")

gender <- structure(c(2L, 2L, 1L, 2L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 1L,
1L, 2L, 2L, 1L, 2L, 1L, 2L), .Label = c("Female", "Male"), class = "factor")

discrim <- structure(c(3L, 4L, 3L, 3L, 3L, 2L, 4L, 3L, 2L, 3L, 3L, 2L, 3L,
3L, 3L, 3L, 2L, 3L, 4L, 3L), .Label = c("A Lot", "No", "Not at all",
"Yes"), class = "factor")

CrossTable(gender, discrim, prop.c=FALSE, prop.r=FALSE, chisq=FALSE, prop.t=FALSE, prop.chisq=FALSE)
```

       Cell Contents
    |-------------------------|
    |                       N |
    |-------------------------|

    Total Observations in Table:  20

                 | discrim
          gender |         No | Not at all |        Yes |  Row Total |
    -------------|------------|------------|------------|------------|
          Female |          2 |          4 |          2 |          8 |
    -------------|------------|------------|------------|------------|
            Male |          2 |          9 |          1 |         12 |
    -------------|------------|------------|------------|------------|
    Column Total |          4 |         13 |          3 |         20 |
    -------------|------------|------------|------------|------------|

In this representation we see response counts for each combination on the categoricals. When we add each count for each possible combination, the sum equals the total number of responses (n=20). From the table we can see that there were two (2) counts of females who responded 'No' to feeling discrimination.

What we also see is row and column totals for each category. Following the rows, we can see that the total number of women is eight (8) and the total number of men is twelve (12). From the columns, there were a total of four (4) 'No,' thirteen (13) 'Not at all,' three (3) 'Yes,' and zero (0) 'A lot.'

It is here in the proportions of counts in rows versus columns that we determine if there is a relationship between the categoricals.

If we assume that the responses to the question of discrimination are independent of gender, then then the number of female responding 'Yes' should be equal to the fraction of females multiplied by the number responding 'Yes.'

$$\begin{align}
n_{f,y} =& 3 \frac{8}{20}\\
=& 1.2
\end{align}$$

However, from the table we can see that there are actually more than 1.2 females who responded 'Yes.' The question then becomes, is this more likely to be from the same distribution (ie sample is independent of m/f and from the same distribution) or are they from different distributions.

## Chi squared
The chi squared test of independence is calculated as follows:

$$\begin{align}
\chi^2_{cell} =& \frac{\left( observed - expected \right)^2}{expected} \\
\chi^2 =& \sum_{all} \chi^2_{cell}
\end{align}$$

This value is then compared against the chi squared distribution to determine how likely that the sample is independent of gender. The chi squared distribution is dependent on the number of degrees of freedom (dof) of the measurement. The number used in selecting the distribution for dof is:

$$\begin{align}
dof =& \left( number of rows - 1\right) \left( number of columns -1 \right)
\end{align}$$

For our sample, we may recalculate the table to include the expected values and the chi squared value of each cell as well as the likelihood that the sample is independent of m/f:

```R
CrossTable(gender, discrim, prop.c=FALSE, prop.r=FALSE, chisq=TRUE, prop.t=FALSE, prop.chisq=TRUE, expected=TRUE)
```

    Warning message in chisq.test(t, correct = FALSE, ...):
    “Chi-squared approximation may be incorrect”

       Cell Contents
    |-------------------------|
    |                       N |
    |              Expected N |
    | Chi-square contribution |
    |-------------------------|

    Total Observations in Table:  20

                 | discrim
          gender |         No | Not at all |        Yes |  Row Total |
    -------------|------------|------------|------------|------------|
          Female |          2 |          4 |          2 |          8 |
                 |      1.600 |      5.200 |      1.200 |            |
                 |      0.100 |      0.277 |      0.533 |            |
    -------------|------------|------------|------------|------------|
            Male |          2 |          9 |          1 |         12 |
                 |      2.400 |      7.800 |      1.800 |            |
                 |      0.067 |      0.185 |      0.356 |            |
    -------------|------------|------------|------------|------------|
    Column Total |          4 |         13 |          3 |         20 |
    -------------|------------|------------|------------|------------|

    Statistics for All Table Factors

    Pearson's Chi-squared test
    ------------------------------------------------------------
    Chi^2 =  1.517094     d.f. =  2     p =  0.4683464

For each cell we now have count, expected value, and chi squared value. From the test statistic, the likelihood that the sample is independent of gender is 0.468. In this case we cannot reject the NULL.

Note the warning provided by the test, "Chi-squared approximation may be incorrect." Given the number of degrees of freedom, this is a very small sample. A statistical power test would suggest that for a moderate effect size of 0.3, a sample of n=108 would be needed.

```R
library("pwr")
pwr.chisq.test(w=0.3,N=NULL,df=2,sig.level=0.05,power=0.8)
```

         Chi squared power calculation

                  w = 0.3
                  N = 107.0521
                 df = 2
          sig.level = 0.05
              power = 0.8

    NOTE: N is the number of observations

```R

```


# The Error of our solution for the eccentric anomaly.

The idea is to calulate E in any time instant 't_i' using the approximate solution 'E_ap' and plug that E into the actual equation of E 'E_ac' to get a new time instant 't_f'. Then simply the error is equal to the difference between 't_f' and 't_i'.

```python
from sympy import *
init_printing()

E_ac, E_ap, n, e_c, t_i, t_f = symbols("E_ac E_ap n e_c t_i t_f")

exp1 = Eq(E_ac - e_c*sin(E_ac), n*t_f) # The approximate solution

E_ap = n*t_i + e_c*sin(n*t_i) # The actual equation
```

The approximate solution:

```python
Eq(symbols("E_ap"), E_ap)
```

The actual equation:

```python
exp1
```

Substitute E_ap into E_ac:

```python
expr2 = exp1.subs({E_ac:E_ap})
expr2
```

Solve for t_f:

```python
t_ff = solveset(expr2, t_f).args[0]
Eq(t_f, t_ff)
```

Error = t_f - t_i

```python
expr3 = t_ff - t_i
Eq(symbols('Error'), Eq(t_f - t_i, expr3.simplify()))
```

To get the maximum error, we can differentiate the error with respect to 't_i' and equalize it with zero.

```python
expr4 = expr3.diff(t_i).simplify()
expr4
```

```python
solveset(Eq(expr4, 0), t_i).args[1]
```

```python

```


# Analiza i projektiranje računalom - 3. domaća zadaća

U okviru ove domaće zadaće potrebno je implementirati sljedeće metode optimiranja:

---

## Funkcije cilja

### Funkcija 1 - Rosenbrockova "banana" funkcija

$$
\begin{equation}
    f_1 \left( \vec{x} \right) = 100 \left( x_1 - {x_0}^2 \right)^2 + \left( 1 - x_0 \right)^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        -1.9 & 2
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        1 & 1
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_1 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 2

$$
\begin{equation}
    f_2 \left( \vec{x} \right) = \left( x_0 - 4 \right)^2 + 4 \left( x_1 - 2 \right)^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        0.1 & 0.3
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        4 & 2
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_2 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 3

$$
\begin{equation}
    f_3 \left( \vec{x} \right) = \left( x_0 - 2 \right)^2 + \left( x_1 + 3 \right)^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        0 & 0
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        2 & -3
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_3 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 4

$$
\begin{equation}
    f_4 \left( \vec{x} \right) = \left( x_0 - 3 \right)^2 + {x_1}^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        0 & 0
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        3 & 0
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_4 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

## Priprema za izvođenje

```python
import os

CD_KEY = "--HW03_IN_ROOT"
```

```python
if (
    CD_KEY not in os.environ
    or os.environ[CD_KEY] is None
    or len(os.environ[CD_KEY]) == 0
    or os.environ[CD_KEY] == "false"
):
    %cd ..
else:
    print(os.getcwd())

os.environ[CD_KEY] = "true"
```

    /mnt/data/projekti/faks/AIPR/dz/dz-03

## Učitavanje paketa

```python
import copy
import sys
import warnings

import array_to_latex as a2l
from IPython.display import display, Markdown
from matplotlib import pyplot as plt
import numpy as np

from src.searches.box import box_search
from src.searches.constraint import (
    Constraint,
    constraints_to_function,
    find_inner_point
)
from src.searches.function import Function
from src.searches.gradient_descent import gradient_descent_search
from src.searches.hooke_jeeves import hooke_jeeves_search
from src.searches.nelder_mead import nelder_mead_simplex_search
from src.searches.newton_raphson import newton_raphson_search
```

```python
np.random.seed(21051208)
warnings.filterwarnings('ignore')
```

## Definicija pomoćnih funkcija

```python
def array_to_latex(array):
    return a2l.to_ltx(
        array,
        frmt="{:g}",
        arraytype="bmatrix",
        print_out=False
    )
```

## Definicija funkcija i početnih točaka

```python
f1 = Function(lambda x: 100 * (x[1] - x[0] ** 2) ** 2 + (1 - x[0]) ** 2)
f1_derivative = Function(
    lambda x: np.array(
        [
            2 * (200 * x[0] ** 3 - 200 * x[0] * x[1] + x[0] - 1),
            200 * (x[1] - x[0] ** 2)
        ]
    )
)
f1_hesse = Function(
    lambda x: np.array(
        [
            [2 * (600 * x[0] ** 2 - 200 * x[1] + 1), -400 * x[0]],
            [-400 * x[0], 200]
        ]
    )
)
f1_derivative.derivative = f1_hesse
f1.derivative = f1_derivative

f1_start = np.array([-1.9, 2])
```

```python
f2 = Function(lambda x: (x[0] - 4) ** 2 + 4 * (x[1] - 2) ** 2)
f2_derivative = Function(
    lambda x: np.array([2 * x[0] - 8, 8 * x[1] - 16])
)
f2_hesse = Function(
    lambda x: np.array(
        [
            [2, 0],
            [0, 8]
        ]
    )
)
f2_derivative.derivative = f2_hesse
f2.derivative = f2_derivative

f2_start = np.array([0.1, 0.3])
```

```python
f3 = Function(lambda x: (x[0] - 2) ** 2 + (x[1] + 3) ** 2)
f3_derivative = Function(
    lambda x: np.array([2 * x[0] - 4, 2 * x[1] + 6])
)
f3_hesse = Function(
    lambda x: np.array(
        [
            [2, 0],
            [0, 2]
        ]
    )
)
f3_derivative.derivative = f3_hesse
f3.derivative = f3_derivative

f3_start = np.array([0, 0])
```

```python
f4 = Function(lambda x: (x[0] - 3) ** 2 + x[1] ** 2)
f4_derivative = Function(
    lambda x: np.array([2 * x[0] - 6, 2 * x[1]])
)
f4_hesse = Function(
    lambda x: np.array(
        [
            [2, 0],
            [0, 2]
        ]
    )
)
f4_derivative.derivative = f4_hesse
f4.derivative = f4_derivative

f4_start = np.array([0, 0])
```

## Zadatci

### Zadatak 1

Primijenite postupak gradijentnog spusta na funkciju 3, uz i bez određivanje optimalnog iznosa koraka. Što možete zaključiti iz rezultata?

```python
t1_start = copy.deepcopy(f1_start)
```

#### Uz određivanje optimalnog iznosa

```python
t1_function_with_optimal_step = f3.get_new()

t1_result_with_optimal_step = gradient_descent_search(
    function=t1_function_with_optimal_step,
    start=t1_start,
    gradient_scaling="find optimal",
)
t1_value_with_optimal_step = t1_function_with_optimal_step(
    t1_result_with_optimal_step, dont_count=True
)
```

```python
display(
    Markdown(
        "**Postupak gradijentnog spusta** pronašao je minimum "
        f"u točki ${array_to_latex(t1_result_with_optimal_step)}$ "
        f"vrijednosti ${t1_value_with_optimal_step:g}$ "
        f"(uz ${t1_function_with_optimal_step.call_count}$ poziva funkcije "
        f"i ${t1_function_with_optimal_step.derivative.call_count}$ "
        "poziva gradijenta)."
    )
)
```

**Postupak gradijentnog spusta** pronašao je minimum u točki $\begin{bmatrix}
  2  & -3
\end{bmatrix}$ vrijednosti $1.04097e-26$ (uz $75$ poziva funkcije i $3$ poziva gradijenta).

#### Uz normalizaciju gradijenta

```python
t1_function_with_normalize = f3.get_new()

t1_result_with_normalize = gradient_descent_search(
    function=t1_function_with_normalize,
    start=t1_start,
    gradient_scaling="normalize",
)
t1_value_with_normalize = t1_function_with_normalize(
    t1_result_with_normalize, dont_count=True
)
```

    WARNING: Gradient descent timed out after 100 iterations passed with no improvement - result might not be a minimum.

```python
display(
    Markdown(
        "**Postupak gradijentnog spusta** pronašao je minimum "
        f"u točki ${array_to_latex(t1_result_with_normalize)}$ "
        f"vrijednosti ${t1_value_with_normalize:g}$ "
        f"(uz ${t1_function_with_normalize.call_count}$ poziva funkcije "
        f"i ${t1_function_with_normalize.derivative.call_count}$ "
        "poziva gradijenta)."
    )
)
```

**Postupak gradijentnog spusta** pronašao je minimum u točki $\begin{bmatrix}
  1.79019 & -2.73101
\end{bmatrix}$ vrijednosti $0.116373$ (uz $107$ poziva funkcije i $106$ poziva gradijenta).

#### Bez modifikacija gradijenta

```python
t1_function_vanilla = f3.get_new()

t1_result_vanilla = gradient_descent_search(
    function=t1_function_vanilla,
    start=t1_start,
    gradient_scaling="none",
    verbosity="full"
)
t1_value_vanilla = t1_function_vanilla(
    t1_result_vanilla, dont_count=True
)
```

    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21
    F(x' = [ 5.9 -8. ]) = 40.21
    F(x' = [-1.9  2. ]) = 40.21

    WARNING: Gradient descent timed out after 100 iterations passed with no improvement - result might not be a minimum.

```python
display(
    Markdown(
        "**Postupak gradijentnog spusta** pronašao je minimum "
        f"u točki ${array_to_latex(t1_result_vanilla)}$ "
        f"vrijednosti ${t1_value_vanilla:g}$ "
        f"(uz ${t1_function_vanilla.call_count}$ poziva funkcije "
        f"i ${t1_function_vanilla.derivative.call_count}$ "
        "poziva gradijenta)."
    )
)
```

**Postupak gradijentnog spusta** pronašao je minimum u točki $\begin{bmatrix}
 -1.9 &  2
\end{bmatrix}$ vrijednosti $40.21$ (uz $101$ poziva funkcije i $100$ poziva gradijenta).

**Odgovor**: Iz rezultata je jednostavno zaključiti da algoritam ne konvergira naivnim pristupom određivanja gradijenta. Konvergencija se događa samo kada odaberemo optimalan korak. U slučaju gdje ga normaliziramo, pretraga zapne. U slučaju kada ostavimo gradijent takav kakav je, vrtimo se u krug između 2 točke koje daju gradijente suprotnog smjera.

### Zadatak 2

Primijenite postupak gradijentnog spusta i Newton-Raphsonov postupak na funkcije 1 i 2 s određivanjem optimalnog iznosa koraka. Ispišite broj izračuna funkcije, gradijenta i Hesseove matrice.

```python
t2_f1_start = copy.deepcopy(f1_start)
t2_f2_start = copy.deepcopy(f2_start)
```

#### Funkcija 1

```python
t2_f1_gd = f1.get_new()
t2_f1_nr = f1.get_new()
```

##### Gradijentni spust

```python
t2_f1_gd_result = gradient_descent_search(
    function=t2_f1_gd,
    start=t2_f1_start,
    gradient_scaling="find optimal",
)
t2_f1_gd_value = t2_f1_gd(t2_f1_gd_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Postupak gradijentnog spusta** pronašao je minimum "
        f"u točki ${array_to_latex(t2_f1_gd_result)}$ "
        f"vrijednosti ${t2_f1_gd_value:g}$ "
        f"(uz ${t2_f1_gd.call_count}$ poziva funkcije "
        f"i ${t2_f1_gd.derivative.call_count}$ "
        "poziva gradijenta)."
    )
)
```

**Postupak gradijentnog spusta** pronašao je minimum u točki $\begin{bmatrix}
  1  &  1
\end{bmatrix}$ vrijednosti $1.06955e-12$ (uz $150961$ poziva funkcije i $4081$ poziva gradijenta).

##### Newton-Raphson

```python
t2_f1_nr_result = newton_raphson_search(
    function=t2_f1_nr,
    start=t2_f1_start,
    stride_scaling="find optimal",
)
t2_f1_nr_value = t2_f1_nr(t2_f1_nr_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Newton-Raphsonov postupak** pronašao je minimum "
        f"u točki ${array_to_latex(t2_f1_nr_result)}$ "
        f"vrijednosti ${t2_f1_nr_value:g}$ "
        f"(uz ${t2_f1_nr.call_count}$ poziva funkcije, "
        f"${t2_f1_nr.derivative.call_count}$ poziva gradijenta "
        f"i ${t2_f1_nr.derivative.derivative.call_count}$ "
        "poziva Hesseove matrice)."
    )
)
```

**Newton-Raphsonov postupak** pronašao je minimum u točki $\begin{bmatrix}
  1  &  1
\end{bmatrix}$ vrijednosti $3.62541e-13$ (uz $541$ poziva funkcije, $15$ poziva gradijenta i $15$ poziva Hesseove matrice).

#### Funkcija 2

```python
t2_f2_gd = f2.get_new()
t2_f2_nr = f2.get_new()
```

##### Gradijentni spust

```python
t2_f2_gd_result = gradient_descent_search(
    function=t2_f2_gd,
    start=t2_f2_start,
    gradient_scaling="find optimal",
)
t2_f2_gd_value = t2_f2_gd(t2_f2_gd_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Postupak gradijentnog spusta** pronašao je minimum "
        f"u točki ${array_to_latex(t2_f2_gd_result)}$ "
        f"vrijednosti ${t2_f2_gd_value:g}$ "
        f"(uz ${t2_f2_gd.call_count}$ poziva funkcije "
        f"i ${t2_f2_gd.derivative.call_count}$ "
        "poziva gradijenta)."
    )
)
```

**Postupak gradijentnog spusta** pronašao je minimum u točki $\begin{bmatrix}
  4  &  2
\end{bmatrix}$ vrijednosti $1.33413e-13$ (uz $1000$ poziva funkcije i $28$ poziva gradijenta).

##### Newton-Raphson

```python
t2_f2_nr_result = newton_raphson_search(
    function=t2_f2_nr,
    start=t2_f2_start,
    stride_scaling="find optimal",
)
t2_f2_nr_value = t2_f2_nr(t2_f2_nr_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Newton-Raphsonov postupak** pronašao je minimum "
        f"u točki ${array_to_latex(t2_f2_nr_result)}$ "
        f"vrijednosti ${t2_f2_nr_value:g}$ "
        f"(uz ${t2_f2_nr.call_count}$ poziva funkcije, "
        f"${t2_f2_nr.derivative.call_count}$ poziva gradijenta "
        f"i ${t2_f2_nr.derivative.derivative.call_count}$ "
        "poziva Hesseove matrice)."
    )
)
```

**Newton-Raphsonov postupak** pronašao je minimum u točki $\begin{bmatrix}
  4  &  2
\end{bmatrix}$ vrijednosti $1.12834e-12$ (uz $39$ poziva funkcije, $2$ poziva gradijenta i $2$ poziva Hesseove matrice).

Kako se Newton-Raphsonov postupak ponaša na ovim funkcijama?

**Odgovor**: Vidimo da oba algoritma pronalaze minimume. U slučaju **Newton-Raphsonovog postupka** nalaze se nešto precizniji minimumi (vjerojatno radi malo drukčkije formulacije kriterija zaustavljanja), ali je veća razlika znatno manji broj poziva funkcija, gradijenata i Hesseove matrice.

### Zadatak 3

Primijenite postupak po Boxu na funkcije 1 i 2 uz implicitna ograničenja:

$$
\begin{equation}
x_1 - x_0 \geq 0 \\
2 - x_0 \geq 0 \\
\end{equation}
$$

i eksplicitna ograničenja prema kojima su sve varijable u intervalu $\left[ -100, 100 \right]$.

```python
t3_f1 = f1.get_new()
t3_f2 = f2.get_new()

t3_f1_start = copy.deepcopy(f1_start)
t3_f2_start = copy.deepcopy(f2_start)

t3_value_range = (-100, 100)
t3_constraints = [
    Constraint(lambda x: x[1] - x[0] >= 0),
    Constraint(lambda x: 2 - x[0] >= 0),
]
```

#### Funkcija 1

```python
t3_f1_result = box_search(
    function=t3_f1,
    start=t3_f1_start,
    value_range=t3_value_range,
    constraints=t3_constraints
)
t3_f1_value = t3_f1(t3_f1_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Postupak po Boxu** pronašao je minimum "
        f"u točki ${array_to_latex(t3_f1_result)}$ "
        f"vrijednosti ${t3_f1_value:g}$ "
        f"(uz ${t3_f1.call_count}$ poziva funkcije)."
    )
)
```

**Postupak po Boxu** pronašao je minimum u točki $\begin{bmatrix}
  0.0102172 &  0.0102182
\end{bmatrix}$ vrijednosti $0.989899$ (uz $1066$ poziva funkcije).

#### Funkcija 2

```python
t3_f2_result = box_search(
    function=t3_f2,
    start=t3_f2_start,
    value_range=t3_value_range,
    constraints=t3_constraints
)
t3_f2_value = t3_f2(t3_f2_result, dont_count=True)
```

```python
display(
    Markdown(
        "**Postupak po Boxu** pronašao je minimum "
        f"u točki ${array_to_latex(t3_f2_result)}$ "
        f"vrijednosti ${t3_f2_value:g}$ "
        f"(uz ${t3_f2.call_count}$ poziva funkcije)."
    )
)
```

**Postupak po Boxu** pronašao je minimum u točki $\begin{bmatrix}
  2  &  2.00044
\end{bmatrix}$ vrijednosti $4$ (uz $203$ poziva funkcije).

Mijenja li se položaj optimuma uz nametnuta ograničenja?

**Odgovor**: Kod funkcije 1 ne jer

$$
\begin{equation}
x_{min} = \begin{bmatrix}1 & 1\end{bmatrix} \\\\
%
(1 - 1 = 0) \geq 0 \\
(2 - 1 = 1) \geq 0 \\
\left( x_0 = 1 \right) \in \left[-100, 100\right] \\
\left( x_1 = 1 \right) \in \left[-100, 100\right]
\end{equation}
$$

Kod funkcije 2 se ipak mijenja optimum jer

$$
\begin{equation}
x_{min} = \begin{bmatrix}4 & 2\end{bmatrix} \\\\
%
(2 - 4 = -2) \not\geq 0 \\
(2 - 4 = -2) \not\geq 0 \\
\left( x_0 = 4 \right) \in \left[-100, 100\right] \\
\left( x_1 = 2 \right) \in \left[-100, 100\right]
\end{equation}
$$

Međutim, čak i za funkciju 1 ćemo imati problema ovisno o inicijalno odabranim točkama. Ponekad će minimum koji pronađemo biti $\begin{bmatrix}0.01 & 0.01\end{bmatrix}$, a ponekad $\begin{bmatrix}1 & 1\end{bmatrix}$ (pravi minimum).

### Zadatak 4

Primijenite postupak transformacije u problem bez ograničenja na funkcije 1 i 2 s ograničenjima iz prethodnog zadatka (zanemarite eksplicitna ograničenja).

```python
t4_epsilon = 1e-6

t4_eq_constraints = []
t4_geq_constraints = [
    lambda x: x[1] - x[0],
    lambda x: 2 - x[0]
]
t4_constraint_function = constraints_to_function(
    eq_constraints=t4_eq_constraints,
    geq_constraints=t4_geq_constraints
)

t4_f1 = f1.get_new()
t4_f2 = f2.get_new()

t4_f1_start = copy.deepcopy(f1_start)
t4_f2_start = copy.deepcopy(f2_start)
```

Novodobiveni problem optimizacije bez ograničenja minimizirajte koristeći postupak Hooke-Jeeves ili postupak simpleksa po Nelderu i Meadu.

#### Funkcija 1

```python
t4_f1_r = 1.

t4_f1_start_nms = copy.deepcopy(t4_f1_start)
t4_f1_last_point = copy.deepcopy(t4_f1_start_nms)
t4_f1_best_value = None
t4_f1_iterations_without_improvement = 0

while True:
    _t4_f1 = Function(
        lambda x: t4_f1(x) + t4_constraint_function(x, t4_f1_r)
    )

    t4_f1_start_nms = nelder_mead_simplex_search(
        function=_t4_f1,
        start=t4_f1_start_nms,
    )
    t4_f1_value = _t4_f1(t4_f1_start_nms)

    if t4_f1_best_value is None or t4_f1_value < t4_f1_best_value:
        t4_f1_best_value = t4_f1_value
        t4_f1_iterations_without_improvement = 0
    else:
        t4_f1_iterations_without_improvement += 1

    if (
        np.linalg.norm(t4_f1_last_point - t4_f1_start_nms) < t4_epsilon
        or t4_f1_iterations_without_improvement > 10
    ):
        break

    t4_f1_last_point = copy.deepcopy(t4_f1_start_nms)

    # Isto kao i t *= 10
    t4_f1_r /= 10

t4_f1_result_nms = copy.deepcopy(t4_f1_start_nms)
t4_f1_value_nms = t4_f1(t4_f1_result_nms, dont_count=True)
```

```python
display(
    Markdown(
        "**Nelder Mead simpleks** pronašao je minimum "
        f"u točki ${array_to_latex(t4_f1_result_nms)}$ "
        f"vrijednosti ${t4_f1_value_nms:g}$ "
        f"(uz ${t4_f1.call_count}$ poziva funkcije)."
    )
)
```

**Nelder Mead simpleks** pronašao je minimum u točki $\begin{bmatrix}
  0.0101644 &  0.0101649
\end{bmatrix}$ vrijednosti $0.989898$ (uz $1175$ poziva funkcije).

#### Funkcija 2

```python
t4_f2_r = 1.

t4_f2_start_nms = copy.deepcopy(t4_f2_start)
t4_f2_last_point = copy.deepcopy(t4_f2_start_nms)
t4_f2_best_value = None
t4_f2_iterations_without_improvement = 0

while True:
    _t4_f2 = Function(
        lambda x: t4_f2(x) + t4_constraint_function(x, t4_f2_r)
    )

    t4_f2_start_nms = nelder_mead_simplex_search(
        function=_t4_f2,
        start=t4_f2_start_nms
    )
    t4_f2_value = _t4_f2(t4_f2_start_nms)

    if t4_f2_best_value is None or t4_f2_value < t4_f2_best_value:
        t4_f2_best_value = t4_f2_value
        t4_f2_iterations_without_improvement = 0
    else:
        t4_f2_iterations_without_improvement += 1

    if (
        np.linalg.norm(t4_f2_last_point - t4_f2_start_nms) < t4_epsilon
        or t4_f2_iterations_without_improvement > 10
    ):
        break

    t4_f2_last_point = copy.deepcopy(t4_f2_start_nms)

    # Isto kao i t *= 10
    t4_f2_r /= 10

t4_f2_result_nms = copy.deepcopy(t4_f2_start_nms)
t4_f2_value_nms = t4_f2(t4_f2_result_nms, dont_count=True)
```

```python
display(
    Markdown(
        "**Nelder Mead simpleks** pronašao je minimum "
        f"u točki ${array_to_latex(t4_f2_result_nms)}$ "
        f"vrijednosti ${t4_f2_value_nms:g}$ "
        f"(uz ${t4_f2.call_count}$ poziva funkcije)."
    )
)
```

**Nelder Mead simpleks** pronašao je minimum u točki $\begin{bmatrix}
  2  &  2.00974
\end{bmatrix}$ vrijednosti $4.00038$ (uz $1156$ poziva funkcije).

Može li se uz zadanu početnu točku pronaći optimalno rješenje problema s ograničenjima?

**Odgovor**: Za funkciju 2 da, no za funkciju 1 bi trebali mali izmijeniti početnu točku.

Ako ne, probajte odabrati početnu točku iz koje je moguće pronaći rješenje.

```python
t4_f1_2 = t4_f1.get_new()

t4_f1_r_2 = 1.

t4_f1_start_nms_2 = np.array([1.9, 2])
t4_f1_last_point_2 = copy.deepcopy(t4_f1_start_nms_2)
t4_f1_best_value_2 = None
t4_f1_iterations_without_improvement_2 = 0

while True:
    _t4_f1_2 = Function(
        lambda x: t4_f1_2(x) + t4_constraint_function(x, t4_f1_r_2)
    )

    t4_f1_start_nms_2 = nelder_mead_simplex_search(
        function=_t4_f1_2,
        start=t4_f1_start_nms_2,
    )
    t4_f1_value_2 = _t4_f1_2(t4_f1_start_nms_2)

    if t4_f1_best_value_2 is None or t4_f1_value_2 < t4_f1_best_value_2:
        t4_f1_best_value_2 = t4_f1_value_2
        t4_f1_iterations_without_improvement_2 = 0
    else:
        t4_f1_iterations_without_improvement_2 += 1

    if (
        np.linalg.norm(t4_f1_last_point_2 - t4_f1_start_nms_2) < t4_epsilon
        or t4_f1_iterations_without_improvement_2 > 10
    ):
        break

    t4_f1_last_point_2 = copy.deepcopy(t4_f1_start_nms_2)

    # Isto kao i t *= 10
    t4_f1_r_2 /= 10

t4_f1_result_nms_2 = copy.deepcopy(t4_f1_start_nms_2)
t4_f1_value_nms_2 = t4_f1_2(t4_f1_result_nms_2, dont_count=True)
```

```python
display(
    Markdown(
        "**Nelder Mead simpleks** pronašao je minimum "
        f"u točki ${array_to_latex(t4_f1_result_nms_2)}$ "
        f"vrijednosti ${t4_f1_value_nms_2:g}$ "
        f"(uz ${t4_f1_2.call_count}$ poziva funkcije)."
    )
)
```

**Nelder Mead simpleks** pronašao je minimum u točki $\begin{bmatrix}
  1.00072 &  1.0015
\end{bmatrix}$ vrijednosti $7.46346e-07$ (uz $3955$ poziva funkcije).

### Zadatak 5

Za funkciju 4 s ograničenjima

$$
\begin{equation}
    3 - x_0 - x_1 \geq 0 \\
    3 + 1.5x_0 - x_1 \geq 0 \\
    x_1 - 1 = 0
\end{equation}
$$

probajte pronaći minimum koristeći postupak transformacije u problem bez ograničenja (također koristite Hooke-Jeeves ili postupak simpleksa po Nelderu i Meadu za minimizaciju). Probajte kao početnu točku postaviti neku točku koja ne zadovoljava ograničenja nejednakosti (primjerice točku (5,5)) te pomoću postupka pronalaženja unutarnje točke odredite drugu točku koja zadovoljava ograničenja nejednakosti te ju iskoristite kao početnu točku za postupak minimizacije.

```python
t5_epsilon = 1e-6

t5_eq_constraints = [
    lambda x: x[1] - 1
]
t5_geq_constraints = [
    lambda x: 3 - x[0] - x[1],
    lambda x: 3 + 1.5 * x[0] - x[1]
]
t5_constraint_function = constraints_to_function(
    eq_constraints=t5_eq_constraints,
    geq_constraints=t5_geq_constraints
)

t5_f4_1 = f4.get_new()
t5_f4_2 = f4.get_new()

t5_start_1 = np.array([5, 5])
t5_start_2 = find_inner_point(t5_start_1, geq_constraints=t5_geq_constraints)
```

##### Točka koja ne zadovoljava ograničenja

```python
t5_r_1 = 1.

t5_current_point_1 = copy.deepcopy(t5_start_1)
t5_last_point_1 = copy.deepcopy(t5_current_point_1)
t5_best_value_1 = None
t5_iterations_without_improvement_1 = 0

while True:
    _t5_f4_1 = Function(
        lambda x: t5_f4_1(x) + t5_constraint_function(x, t5_r_1)
    )
    t5_current_point_1 = hooke_jeeves_search(
        function=_t5_f4_1,
        start=t5_current_point_1,
        max_iterations=10
    )
    t5_value_1 = _t5_f4_1(t5_current_point_1)

    if t5_best_value_1 is None or t5_value_1 < t5_best_value_1:
        t5_best_value_1 = t5_value_1
        t5_iterations_without_improvement_1 = 0
    else:
        t5_iterations_without_improvement_1 += 1

    if (
        np.linalg.norm(t5_last_point_1 - t5_current_point_1) < t5_epsilon
        or t5_iterations_without_improvement_1 > 10
    ):
        break

    t5_last_point_1 = copy.deepcopy(t5_current_point_1)

    # Isto kao i t *= 10
    t5_r_1 /= 10

t5_result_1 = copy.deepcopy(t5_current_point_1)
t5_value_1 = t5_f4_1(t5_result_1, dont_count=True)
```

```python
display(
    Markdown(
        "**Hooke-Jeeves postupak** pronašao je minimum "
        f"u točki ${array_to_latex(t5_result_1)}$ "
        f"vrijednosti ${t5_value_1:g}$ "
        f"(uz ${t5_f4_1.call_count}$ poziva funkcije)."
    )
)
```

**Hooke-Jeeves postupak** pronašao je minimum u točki $\begin{bmatrix}
  5  &  5
\end{bmatrix}$ vrijednosti $29$ (uz $61$ poziva funkcije).

##### Točka koja zadovoljava ograničenja

```python
display(
    Markdown(
        "Novodobivena točka koja zadovoljava uvjete je "
        f"${array_to_latex(t5_start_2)}$."
    )
)
```

Novodobivena točka koja zadovoljava uvjete je $\begin{bmatrix}
 -0.000191118 &  2.99937
\end{bmatrix}$.

```python
t5_r_2 = 1.

t5_current_point_2 = copy.deepcopy(t5_start_2)
t5_last_point_2 = copy.deepcopy(t5_current_point_2)
t5_best_value_2 = None
t5_iterations_without_improvement_2 = 0

while True:
    _t5_f4_2 = Function(
        lambda x: t5_f4_2(x) + t5_constraint_function(x, t5_r_1)
    )
    t5_current_point_2 = hooke_jeeves_search(
        function=_t5_f4_2,
        start=t5_current_point_2,
    )
    t5_value_2 = _t5_f4_2(t5_current_point_2)

    if t5_best_value_2 is None or t5_value_2 < t5_best_value_2:
        t5_best_value_2 = t5_value_2
        t5_iterations_without_improvement_2 = 0
    else:
        t5_iterations_without_improvement_2 += 1

    if (
        np.linalg.norm(t5_last_point_2 - t5_current_point_2) < t5_epsilon
        or t5_iterations_without_improvement_2 > 100
    ):
        break

    t5_last_point_2 = copy.deepcopy(t5_current_point_2)

    # Isto kao i t *= 10
    t5_r_2 /= 10

t5_result_2 = copy.deepcopy(t5_current_point_2)
t5_value_2 = t5_f4_2(t5_result_2, dont_count=True)
```

```python
display(
    Markdown(
        "**Hooke-Jeeves postupak** pronašao je minimum "
        f"u točki ${array_to_latex(t5_result_2)}$ "
        f"vrijednosti ${t5_value_2:g}$ "
        f"(uz ${t5_f4_2.call_count}$ poziva funkcije)."
    )
)
```

**Hooke-Jeeves postupak** pronašao je minimum u točki $\begin{bmatrix}
  2.59204 & -0.575437
\end{bmatrix}$ vrijednosti $0.497556$ (uz $455$ poziva funkcije).

**Komentar**: Vidimo da bez sređivanja ograničenja algoritam ni približno ne konvergira, tj. ne može se ni pomaknuti iz početne točke. Kada uredimo početnu točku u neku koja poštuje ograničenja, dođemo puno bliže minimuma.


$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$
$\newcommand{\bra}[1]{\left\langle{#1}\right|}$
# Challenge 03: Minimum Makespan with Quantum Annealing

## Introduction

The core of this challenge is about translating a real-world problem into a mathematical representation that fits a quantum computer. In this challenge we encourage you to use two representations which even though equivalent, are used in different contexts.

The first is Quadratic Unconstrained Binary Optimization (QUBO) which is used primarily for the quantum annealers (D-Wave) and the other is Ising Hamiltonian, which is often used with an algorithm called Quantum Approximate Optimization Algorithm (QAOA).

This challenge might require spending some time going through the references (especially 1-3), as we are not aware of any "quick introductions" worth mentioning. If you have a hard time understanding something, feel free to reach out for help on [QOSF Slack](https://join.slack.com/t/qosf/shared_invite/zt-bw59w8b9-WJ~k0~FAMHukTZov4AnLfA).

Unless you have a prior background with these kinds of problems, we encourage you to start from QUBO and then perhaps trying Ising Hamiltonians as well, as there are more materials about it.

It is also worth mentioning that D-Wave allows you to use their machines for free for a limited amount of time, so you can try running your QUBO on a real machine after signing up [here](https://cloud.dwavesys.com/leap/signup/).

## Problem

Niranjan owns a catering service and receives orders one day in advance. Each of the chefs he employs is paid by the hour. In order to optimize the cost of running his business, every night, he looks at all of the dishes he needs to deliver for the next day and asks his employees to work the specific hours he requires them for.

Given an order queue ($Q$) of length $N$ with each order $i$ taking $L_i$ time to prepare, if there are $m$ chefs, design a QUBO/Ising Hamiltonian to help Niranjan find the shortest time ($T$) it would take for the $m$ chefs to prepare all $N$ dishes. Assume each chef can only work on one dish at a time and each dish is worked on by only one chef. We can also safely assume that at any given point, all chefs can be actively working on a dish.

Since Niranjan wants to ensure that he can run his algorithms on the near term devices, keep track and try to minimize the number of:
1. Gates & Qubits (for Ising Hamiltonian)
2. Variables (for QUBO. For embedding on a real machine also keep track of the lenghts of chains)

**Optional:**
For the advanced reader, you can also attempt to build the Cost Hamiltonian or Mixer Hamiltonian as described in [7,8].

## Examples

### Case #1:

*If,* $\quad N = 5 , \quad m = 1, \quad \text{queue} = \{2, 3, 7, 2, 1\} \quad$ *then,* $\quad T=15$

*since* there is only one chef, the shortest time it would take to complete all the orders is simply the sum of the times it would take for each individual order.

### Case #2:

*If,* $\quad N = 5 , \quad m = 2, \quad \text{queue} = \{2, 3, 7, 2, 1\} \quad$ *then,* $\quad T=8$

*since* there are two chefs $m_1$ and $m_2$, the distribution that yields the shortest time is:

$m_1 = \{2, 3, 2\}, \qquad m_2 = \{ 7, 1 \} $

### Case #3:

*If,* $\quad N = 5 , \quad m = 3, \quad \text{queue} = \{2, 3, 2, 2, 1\} \quad$ *then,* $\quad T=4$

*since* there are three chefs $m_1$, $m_2$ and $m_3$, the distribution that yields the shortest time is:

$m_1 = \{3 \}, \qquad m_2 = \{2,2 \}, \qquad m_3 = \{2,1 \} $

Note that there could be another possible distribution of tasks, but the shortest time does not change:

$m_1 = \{3,1 \} , \qquad m_2 = \{2,2 \} , \qquad m_3 = \{2 \} $

## Solution by peguerosdc

The third example in the previous section is what gave the hint and motivated this approach.

Defining $m_i$ as the queue of dishes assigned to chef i and defining the total cost of any queue as $cost(m_i)$, we first note that when a chef is assigned with the **optimal queue** $m_{optimal}$ (that is, the queue that satisfies $cost(m_{optimal})=T$), the rest of the chefs can be assigned any combination of the remaining dishes and they are all still valid solutions as long as they fall within the constraint that $cost(m_i) \leq T$ (because of the definition of $m_{optimal}$).

In both of the solutions of the previous example, the **optimal chef** (the one that is assigned with $m_{optimal}$) is always $m_2$, but that doesn't have to be the case as all the chefs are indistinguishable, so it really doesn't matter! That means we can pick any chef as the **optimal chef** (in the following, we will pick $m_1$) and try to minimize its cost subject the constraints that make it the true **optimal chef**:

1. Every dish can be prepared once and only once. This ensures that the solution found is a valid solution.
2. The time it takes to $m_1$ to cook all his dishes is enough time for the other chefs to also cook their dishes:

$$cost(m_1) = T \geq cost(m_i) \quad \text{for any} \quad i\neq 1$$

To encode this problem into a QUBO/Ising optimisation problem, we use unary encoding to represent the dishes $d_i$ present in a given order Q of N dishes:

$$ \ket{d_0} = \ket{1000...0_N}$$

$$ \ket{d_1} = \ket{0100...0_N}$$

And so on. So, if the order is given by $Q=\{2,3,2,2,1\}$ and chef $m_1$ is preparing the first and the third dishes, his queue/state will be given by:

$$ \ket{m_1} = \ket{10100}$$

And the state $\ket{\Psi}$ of a system of $m$ chefs will be given by the tensor product (the $\otimes$ symbol is omitted for the sake of readability) of $m$ of these states:

$$ \ket{\Psi} = \ket{m_1}\ket{m_2}...\ket{m_m} $$

To find the most optimal solution to $\ket{\Psi}$, we need to build an Ising/QUBO Hamiltonian where the energy represents the cost $m_1$ (which is the quantity we want to minimize) subject to the 2 constraints mentioned above.

### Objective function and constraints

The QUBO Hamiltonian is coded using D-Wave's library and it is described in terms of the binary variables $\{ m_{ij} \}$ which will be 1 if the qubit j of the chef i is in the state $\ket{1}$ and 0 if it is in the state $\ket{0}$.

It is important to mention that even though this library allows us to define the algorithm already in terms of $\{ m_{ij} \}$, we are describing each state in the basis of the $Z$ gate (that is, $\ket{1}$ represents a spin up state and $\ket{0}$ a spin down state), so if we wanted to run the algorithm on a real quantum machine (please someone correct me if I am wrong) or in some other simulators (like pyquil+Grove), we need to replace the variables $m_i$ with the following operators in terms of the $Z$ gate as their eigenvalues translate to the binary description we desire:

$$ m_i \rightarrow \hat{m_i} = \frac{1 - \hat{Z_i}}{2} \quad \text{because} \quad \hat{m_i}\ket{0_i} = 0, \quad \hat{m_i}\ket{1_i} = \ket{1_i}   $$

Following with the $\{ m_i \}$ description (again for the sake of readability), the cost of the optimal chef $m_1$ is encoded as:

$$ H_0 = \vec{Q} \cdot \vec{m_1} $$

Where $\vec{Q}$ is a vector of length $N$ where the entry $i$ corresponds to the cost of the dish $d_i$. Now, it is left to map the two constraints as energy penalties to ensure that the solution is going to be valid and optimal.

#### Constraint 1: every dish can be prepared once and only once

In terms of $\ket{\Psi}$, this constraint means that for any given dish j, only one $m_{ij}$ must be set for all posible values of i. This can be translated to an equality constraint with Lagrange multipliers:

$$ H_1 = \lambda_1 \sum_i^{N} \left( \sum_j^m m_{ji} -1 \right)^2 $$

#### Constraint 2: chef m1 must be the optimal chef

To consider this inequality constraint $cost(m_1) \geq cost(m_i)$ in our optimization problem, we need to translate it to an equality constraint with the aid of one slack variable:

$$ H_2 = \lambda_2 \sum_{i>1}^{m} \left( \vec{Q}\cdot (\vec{m_1} - \vec{m_i}) - s \right)^2 $$

This slack variable is bounded by the worst case scenario in which this constraint is violated, which is given by the case where no dishes are prepared by the optimal chef and all of them are prepared by any other chef $i$, so the value of $s$ is always gonna be capped by $cost(Q)$. Remembering that the algorithm is a **binary** algorithm, we need to encode the slack variable in the $log2( max(s) )$ binary variables that will be represented by ancilla qubits.

#### Hamiltonian

Putting everything together, the objective function is given by $H=H_0 + H_1 + H_2$:

$$ H = \vec{Q} \cdot \vec{m_1} + \lambda_1 \sum_i^{N} \left( \sum_j^m m_{ji} -1 \right)^2 + \lambda_2 \sum_{i>1}^{m} \left( \vec{Q}\cdot (\vec{m_1} - \vec{m_i}) - s \right)^2 $$

Which is trivially quantized to:

$$\hat{H} = \hat{\vec{Q}} \cdot \hat{\vec{m_1}} + \lambda_1 \sum_i^{N} \left( \sum_j^m \hat{m_{ji}} -1 \right)^2 + \lambda_2 \sum_{i>1}^{m} \left( \hat{\vec{Q}}\cdot (\hat{\vec{m_1}} - \hat{\vec{m_i}}) - \hat{s} \right)^2 $$

and its energy eigenvalues are given by:

$$E = cost(m_1) + \text{how much the constraints are violated}$$

Which means that the annealing process will (most likely) drive the system to the state where no constraints are violated and $cost(m_1)$ is at its minimum.

Weights $\lambda$ have to be big enough so that even in the worst case scenario (that is, the state for which the energy $E_0$ of $\hat{H_0}$ is maximum), the constraints of $H_1 + H_2$ are still enforced:

$$
\lambda > max(E_0) = sum(\text{dishes})
$$

So, we choose $\lambda=sum(\text{dishes})+1$

### Coding the solution

```python
# math libraries
import math
import numpy as np
# library to manipulate symbolic expressions
from sympy import symbols
# dimod "simulator"
from dimod import ExactSolver
# d-wave libraries
from dwave.system import DWaveSampler, EmbeddingComposite

class MinimumMakespanSolver():

    def __init__(self, dishes, chefs):
        self.dishes = dishes
        self.m = chefs
        self.N = len(dishes)
        # calculate the amount of slack qubits required per chef
        self.ancilla_per_chef = math.ceil(math.log2(sum(dishes)))
        # prepare variables to build the objective function
        amount_qubits = self.N * self.m + (self.m-1)*self.ancilla_per_chef
        self.syms = symbols(f"x:{amount_qubits}")
        # variable to store the QUBO
        self.qubo = None

    def get_index(self, dish, chef):
        """Get the index of the qubit corresponding to a chef's dish"""
        return self.N*chef + dish

    def get_bit(self, dish, chef):
        """Get the symbol of the qubit corresponding to a chef's dish"""
        return self.get_raw_bit(self.get_index(dish, chef))

    def get_raw_bit(self, index):
        """Get the symbol corresponding to qubit 'index' """
        return self.syms[index]

    def get_indexes_of_ancilla(self, chef):
        """Get the indexes of the slack/ancilla qubits used by a chef"""
        return [i for i in range(self.N*self.m + (chef-1)*self.ancilla_per_chef, self.N*self.m + chef*self.ancilla_per_chef) ]

    def cost_of_unprepared_dishes(self, chef):
        """Get the expression that maps to the cost of the state of a certain chef"""
        return sum([cost*self.get_bit(dish, chef) for dish,cost in enumerate(self.dishes)])

    def get_objective(self, weight=1):
        """Builds the objective function enforcing three constraints:

           1) Minimize the amount of dishes prepared by chef0 = chef_max

           2) No dishes can be cooked more than once

           3) chef_max should take the highest amount of time
        """
        obj = 0
        # 1 constraint: Minimize the amount of dishes prepared by chef0 = chef_max
        obj += self.cost_of_unprepared_dishes(0)
        # 2 constraint: no dishes can be cooked more than once
        cost_operators = 0
        for dish in range(self.N):
            op = -1
            for chef in range(self.m):
                op += self.get_bit(dish, chef)
            cost_operators += op**2
        obj += weight*cost_operators
        # 3 constraint: chef_max should take the highest amount of time
        cost_operators = 0
        for chef in range(1,self.m):
            # get slack binary variables
            slack = sum( [ (2**i)*self.get_raw_bit(ancilla) for i,ancilla in enumerate(self.get_indexes_of_ancilla(chef))] )
            # get total hamiltonian
            cost_operators += (self.cost_of_unprepared_dishes(0) - self.cost_of_unprepared_dishes(chef) - slack)**2
        obj += weight*cost_operators
        # return expression with all the constraints
        return obj

    def build(self, weight=None):
        """Builds the QUBO for the given problem. Returns the linear and quadratic coefficients.
        If no weight is passed, it is calculated with the worst case scenario
        """
        # get best weight
        weight = weight if weight else sum(self.dishes)+1
        # get the raw function to minimize
        obj = self.get_objective(weight)
        print(f"Raw expression: {obj}")
        # simplify one-variable quadratic terms as linear terms
        for s in self.syms:
            obj = obj.expand().replace(s**2, s)
        # decompose in factors
        terms = obj.expand().as_coefficients_dict()
        # build QUBO
        linear = dict()
        quadratic = dict()
        for term in terms:
            # check how to map each term
            readable = str(term)
            # if the term is in the shape a*xn*xm (that is, with two variables), it is a quadratic term
            if readable.count("x") == 2:
                x0,x1 = readable.split("*")
                quadratic[(x0,x1)] = terms[term]
            # if the term is in the shape a*xn (that is, with one variable), it is a linear term
            elif readable.count("x") == 1:
                linear[(readable, readable)] = terms[term]
        # store the QUBO
        self.qubo = {**linear, **quadratic}
        return self.qubo

    def solve(self, qubo=None, quantum=False):
        """Solves a given QUBO. To use the quantum mode, the D-Wave API keys must be set in the environment.

        Args
        ----
        qubo: coefficients of the problem to solve. If none given, the qubo stored in the class is used.
        quantum: True if meant to be run on a d-wave machine, False if meant to be solved with dimod.ExactSolver
        """
        # get the correct qubo to run
        qubo = qubo if qubo else (self.qubo if self.qubo else self.build())
        # run
        samples = EmbeddingComposite(DWaveSampler(solver={'topology__type': 'chimera'})).sample_qubo(qubo, num_reads=1000) if quantum else ExactSolver().sample_qubo(qubo)
        # get only results with lowest energy as they make up the solution
        results = samples.lowest()
        data = results.record.sample
        # sort the variables correctly
        sorted_data = np.zeros((len(data), self.m, self.N))
        for r,result in enumerate(data):
            for m in range(self.m):
                for j in range(self.N):
                    # get index of this dish
                    i = results.variables.index(f"x{self.get_index(j,m)}")
                    # get value
                    sorted_data[r][m][j] = self.dishes[j] if result[i] else 0.
        return sorted_data
```

And here are some useful functions to show the result:

```python
def visualize_solution(results, queue, m):
    for solution in results:
        for i,chef in enumerate(solution):
            print(f"Chef {i} cooked {chef} with cost {sum(chef)}")
        print("-------")
```

#### Case 1

```python
queue = [2,3,7,2,1]
m = 1
solver = MinimumMakespanSolver(dishes=queue, chefs=m)
solution = solver.solve(quantum=False)
print("\nSOLUTION:")
visualize_solution(solution, queue, m)
```

    Raw expression: 2*x0 + 3*x1 + 7*x2 + 2*x3 + x4 + 16*(x0 - 1)**2 + 16*(x1 - 1)**2 + 16*(x2 - 1)**2 + 16*(x3 - 1)**2 + 16*(x4 - 1)**2

    SOLUTION:
    Chef 0 cooked [2. 3. 7. 2. 1.] with cost 15.0
    -------

#### Case 2

```python
queue = [2,3,7,2,1]
m = 2
solver = MinimumMakespanSolver(dishes=queue, chefs=m)
solution = solver.solve(quantum=False)
print("\nSOLUTION:")
visualize_solution(solution, queue, m)
```

    Raw expression: 2*x0 + 3*x1 + 7*x2 + 2*x3 + x4 + 16*(x0 + x5 - 1)**2 + 16*(x1 + x6 - 1)**2 + 16*(x2 + x7 - 1)**2 + 16*(x3 + x8 - 1)**2 + 16*(x4 + x9 - 1)**2 + 16*(2*x0 + 3*x1 - x10 - 2*x11 - 4*x12 - 8*x13 + 7*x2 + 2*x3 + x4 - 2*x5 - 3*x6 - 7*x7 - 2*x8 - x9)**2

    SOLUTION:
    Chef 0 cooked [2. 3. 0. 2. 1.] with cost 8.0
    Chef 1 cooked [0. 0. 7. 0. 0.] with cost 7.0
    -------
    Chef 0 cooked [0. 0. 7. 0. 1.] with cost 8.0
    Chef 1 cooked [2. 3. 0. 2. 0.] with cost 7.0
    -------

#### Case 3

Note that this case can't be solved in a classical computer using `ExactSolver`. We need to go full quantum!

```python
queue = [2,3,2,2,1]
m = 3
solver = MinimumMakespanSolver(dishes=queue, chefs=m)
solution = solver.solve(quantum=False)
print("\nSOLUTION:")
visualize_solution(solution, queue, m)
```

```python
queue = [2,3,2,2,1]
m = 3
solver = MinimumMakespanSolver(dishes=queue, chefs=m)
solution = solver.solve(quantum=True)
print("\nSOLUTION:")
visualize_solution(solution, queue, m)
```

    Raw expression: 2*x0 + 3*x1 + 2*x2 + 2*x3 + x4 + 11*(x0 + x10 + x5 - 1)**2 + 11*(x1 + x11 + x6 - 1)**2 + 11*(x12 + x2 + x7 - 1)**2 + 11*(x13 + x3 + x8 - 1)**2 + 11*(x14 + x4 + x9 - 1)**2 + 11*(2*x0 + 3*x1 - 2*x10 - 3*x11 - 2*x12 - 2*x13 - x14 - x19 + 2*x2 - 2*x20 - 4*x21 - 8*x22 + 2*x3 + x4)**2 + 11*(2*x0 + 3*x1 - x15 - 2*x16 - 4*x17 - 8*x18 + 2*x2 + 2*x3 + x4 - 2*x5 - 3*x6 - 2*x7 - 2*x8 - x9)**2

    SOLUTION:
    Chef 0 cooked [2. 0. 0. 2. 0.] with cost 4.0
    Chef 1 cooked [0. 3. 0. 0. 0.] with cost 3.0
    Chef 2 cooked [0. 0. 2. 0. 1.] with cost 3.0
    -------

Note that because of the probabilistic nature of the quantum realm, the solution provided by a quantum computer will not always be the most optimal, but still it would be optimal. When I ran my code I was lucky the best solution was provided, but sometimes (just a few, really), the cost can be 5 (which is almost 4, but not 4).

## Resources

\[1\] [D-Wave Problem Solving Handbook](https://docs.dwavesys.com/docs/latest/doc_handbook.html)

\[2\] [Implementation of the Travelling Salesman Problem](https://github.com/mstechly/quantum_tsp_tutorials)

\[3\] [Ising formulations of many NP problems](https://arxiv.org/pdf/1302.5843.pdf)

\[4\] [D-Wave Examples](https://docs.ocean.dwavesys.com/en/stable/getting_started.html#examples)

\[5\] [Quantum Approximate Optimization Algorithm explained](https://www.mustythoughts.com/quantum-approximate-optimization-algorithm-explained)

\[6\] [Job Shop Scheduling Solver based on Quantum Annealing](https://arxiv.org/pdf/1506.08479.pdf)

### References

\[7\] [Quantum Algorithms for Scientific Computing and Approximate Optimization](https://arxiv.org/pdf/1805.03265.pdf)

\[8\] [From the Quantum Approximate Optimization Algorithm to a Quantum Alternating Operator Ansatz](https://arxiv.org/pdf/1709.03489.pdf)

### Other useful resources
\[9\] [D-Wave Getting Started with the System](https://docs.dwavesys.com/docs/latest/doc_getting_started.html)

\[10\] [Qiskit Converters for Quadratic Programs](https://qiskit.org/documentation/tutorials/optimization/2_converters_for_quadratic_programs.html)

\[11\] [D-Wave Support Difference between BQM, Ising, and QUBO problems?](https://support.dwavesys.com/hc/en-us/community/posts/360017439853-Difference-between-BQM-Ising-and-QUBO-problems-)

\[12\] [A QAOA solution to the traveling salesman problem using pyQuil](https://cs269q.stanford.edu/projects2019/radzihovsky_murphy_swofford_Y.pdf)


# volume of N-d parallelepiped in terms of its edge vectors

**Definition**: Let $\Pi$ be an $n$-dimensional parallelotope defined by edge vectors $\mathcal{B}=\left\{ \mathbf{v}_{1},\ \cdots,\ \mathbf{v}_{n}\right\}$. We define the volume of the parallelotope recursively as follows:

1. For a 1-dimensional parallelotope, we have $\mathrm{vol}(\Pi)$.

2. For an $n$-dimensional parallelotope with $n>1$, we define the _base_ of the parllelotope to be the volume of the $(n-1)$-dimensional parallelotope defined by edges $\mathcal{B}\backslash\left\{\mathbf{v}_{n}\right\}$, and we define the _height_ to be the length of the component of $\mathbf{v}_{n}$ orthogonal to every vector in the base. Then we define
$$\mathrm{vol}\left(\Pi\right)=\mathrm{base}\times\mathrm{height}.$$

**Theorem**: Let $\Pi$ be an $m$-dimensional parallelotope defined by edge vectors $\mathcal{B}=\left\{ \mathbf{v}_{1},\ \dots,\ \mathbf{v}_{m}\right\}$, where $\mathbf{v}_{i}\in\mathbb{R}^{n}$ for $n\ge m$. That is, we are looking at an $m$-dimensional parallelotope embedded inside $n$-dimensional space. Suppose that $A$ is the $m\times n$ matrix with row vectors $\mathcal{B}$, given by
$$A=\begin{pmatrix}\mathbf{v}_{1}^{\mathrm{T}}\\
\vdots\\
\mathbf{v}_{m}^{\mathrm{T}}\end{pmatrix}$$
Then the $m$-dimensional volume of the parallelotope is given by $$\left[\mathrm{vol}\left(\Pi\right)\right]^{2}=\det\left(AA^{\mathrm{T}}\right).$$

**Proof**: Note that $AA^{\mathrm{T}}$ is an $m\times m$ square matrix. Suppose that $m=1$. Then
$$\det AA^{\mathrm{T}}=\det\left(\mathbf{v}_{1}^{\mathrm{T}}\mathbf{v}_{1}\right)=\mathbf{v}_{1}\cdot\mathbf{v}_{1}=\|\mathbf{v}_{1}\|^{2}=\left[\mathrm{vol}_{1}(\mathbf{v}_{1})\right]^{2},$$
so the proposition holds for $m=1$. Now we induct on $m$. Suppose that the proposition holds for $m\ge 1$ and consider $m+1$. Letting $A_{m}$ denote the matrix containing the rows $\mathbf{v}_{1}$ to $\mathbf{v}_{m}$, we can write $A=A_{m+1}$ as

$$A=\begin{pmatrix}A_{m}\\
\mathbf{v}_{m+1}^{\mathrm{T}}
\end{pmatrix}$$

We may decompose $\mathbf{v}_{m+1}$ orthogonally as
$$\mathbf{v}_{m+1}=\mathbf{v}_{\perp}+\mathbf{v}_{\parallel},$$
where $\mathbf{v}_{\perp}$ lies in the orthogonal complement of the base (this is our height), i.e. $\mathbf{v}_{\perp}\cdot\mathbf{v}_{i}=0$ for $1\le i\le m$, and where $\mathbf{v}_{\parallel}\in\mathrm{span}\left\{ \mathbf{v}_{1},\ \cdots,\ \mathbf{v}_{m}\right\}$. Suppose that $$\mathbf{v}_{\parallel}=c_{1}\mathbf{v}_{1}+\cdots+c_{m}\mathbf{v}_{m}.$$
We apply a sequence of elementary row operations to $A$, adding a multiple $-c_{i}$ of row $i$ to row $m+1$ for each $1\le i\le m$. Write the resulting matrix as $B$, and we have
$$B=\begin{pmatrix}A_{m}\\
\mathbf{v}_{\perp}^{\mathrm{T}}
\end{pmatrix}=E_{m}\cdots E_{1}A,$$

where each $E_{i}$ is an elementary matrix adding a multiple of $i$ row to $m+1$ row.
\begin{equation}
    E_i =
    \begin{bmatrix}
        1 \\
        & 1 \\
        & & \ddots \\
        & -c_i & \cdots & 1
    \end{bmatrix}
\end{equation}
Notice that the above operation corresponds to shearing the parallelotope so that the last edge is perpendicular to the base. We see that these operations do not change the determinant as
$$\det\left(BB^{\mathrm{T}}\right)=\det\left(E_{m}\cdots E_{1}\left(AA^{\mathrm{T}}\right)E_{1}^{\mathrm{T}}\cdots E_{m}^{\mathrm{T}}\right)=\det\left(AA^{\mathrm{T}}\right).$$

Through block-multiplication, we get $BB^{\mathrm{T}}$ as
$$BB^{\mathrm{T}}=\begin{pmatrix}A_{m}\\
\mathbf{v}_{\perp}^{\mathrm{T}}
\end{pmatrix}\begin{pmatrix}A_{m}^{\mathrm{T}} & \mathbf{v}_{\perp}\end{pmatrix}=\begin{pmatrix}A_{m}A_{m}^{\mathrm{T}} & A_{m}\mathbf{v}_{\perp}\\
\mathbf{v}_{\perp}^{\mathrm{T}}A_{m}^{\mathrm{T}} & \mathbf{v}_{\perp}^{\mathrm{T}}\mathbf{v}_{\perp}
\end{pmatrix}=\begin{pmatrix}A_{m}A_{m}^{\mathrm{T}} & A_{m}\mathbf{v}_{\perp}\\
\left(A_{m}\mathbf{v}_{\perp}\right)^{\mathrm{T}} & \|\mathbf{v}_{\perp}\|^{2}
\end{pmatrix}.$$

Now notice that
$$A_{m}\mathbf{v}_{\perp}=\begin{pmatrix}\mathbf{v}_{1}^{\mathrm{T}}\\
\vdots\\
\mathbf{v}_{m}^{\mathrm{T}}
\end{pmatrix}\mathbf{v}_{\perp}=\begin{pmatrix}\mathbf{v}_{1}\cdot\mathbf{v}_{\perp}\\
\vdots\\
\mathbf{v}_{m}\cdot\mathbf{v}_{\perp}
\end{pmatrix}=\mathbf{0}.$$
Therefore we have
$$BB^{\mathrm{T}}=\begin{pmatrix}A_{m}A_{m}^{\mathrm{T}} & \mathbf{0}\\
\mathbf{0}^{\mathrm{T}} & \|\mathbf{v}_{\perp}\|^{2}
\end{pmatrix}$$

Taking the determinant, we therefore have
$$\det\left(BB^{\mathrm{T}}\right)=\|\mathbf{v}_{\perp}\|^{2}\det\left(A_{m}A_{m}^{\mathrm{T}}\right).$$

By definition, $\|\mathbf{v}_{\perp}\|$ is the height of the parallelotope, and by the induction hypothesis, $\det\left(A_{m}A_{m}^{\mathrm{T}}\right)$ is the square of the base. Therefore the result follows. $\square$

As a corollary, we can easily verify that the volume is independent of which $n-1$ vectors you choose to define your base, as it should be.

**Corollary**: The volume of a parallelotope is invariant of which base and height you choose.

**Proof**: Choosing a different base and height corresponds to permuting the rows of the matrix $A$. This does not change the magnitude of the resulting determinant. $\square$

Finally, note that the singular values are the absolute values of $A$ are precisely the square roots of the eigenvalues of $AA^\mathrm{T}$. Since the product of the eigenvalues is the determinant, it follows that the product of the singular values of $A$ gives the square root of $\det(AA^\mathrm{T})$, which is precisely the volume.

```python

```


## ２章
### パーセプトロン

\begin{eqnarray}
y
 =
  \begin{cases}
    0 & ( b + w_1x_1 + w_2x_2 \leqq \theta ) \\
    1 & ( b + w_1x_1 + w_2x_2 \gt \theta )
  \end{cases}
\end{eqnarray}
- 最終的に0か1を返す。
- $ x_1 $、$ x_2 $　は入力信号、$ w_1 $、$ w_2 $ は重み。
- 「重み」は電流でいう抵抗のようなもの。

| $ x_1 $ | $ x_2 $ | AND | NAND | OR | XOR |
|----|----|----|----|----|----|
| 0 | 0 | 0 | 1 | 0 | 0 |
| 1 | 0 | 0 | 1 | 1 | 1 |
| 0 | 1 | 0 | 1 | 1 | 1 |
| 1 | 1 | 1 | 0 | 1 | 0 |

```python
# パーセプトロンの実装
import numpy as np

def AND (x1 ,x2):
    x = np.array([x1, x2])
    w = np.array([0.5, 0.5])
    b = -0.7
    tmp = np.sum(w*x) + b
    if tmp <= 0:
        return 0
    else :
        return 1

def NAND (x1 ,x2):
    x = np.array([x1, x2])
    w = np.array([-0.5, -0.5])
    b = 0.7
    tmp = np.sum(w*x) + b
    if tmp <= 0:
        return 0
    else :
        return 1

def OR (x1 ,x2):
    x = np.array([x1, x2])
    w = np.array([0.5, 0.5])
    b = -0.2
    tmp = np.sum(w*x) + b
    if tmp <= 0:
        return 0
    else :
        return 1

data = np.array([[0,0], [0,1], [1,0], [1,1]])

for p in data:
    y = AND(p[0], p[1])
    print(str(p) + " -> " + str(y))
for p in data:
    y = NAND(p[0], p[1])
    print(str(p) + " -> " + str(y))
for p in data:
    y = OR(p[0], p[1])
    print(str(p) + " -> " + str(y))
```

    [0 0] -> 0
    [0 1] -> 0
    [1 0] -> 0
    [1 1] -> 1
    [0 0] -> 1
    [0 1] -> 1
    [1 0] -> 1
    [1 1] -> 0
    [0 0] -> 0
    [0 1] -> 1
    [1 0] -> 1
    [1 1] -> 1

### パーセプトロンの限界
単純パーセプトロンは AND NAND OR を表現できるが、XORは表現できない
非線形は表現できない

### 多層パーセプトロン
パーセプトロンは層を重ねることができる
XOR ゲートは単純なパーセプトロンの層を重ねることで表現できる

```python
def XOR (x1, x2):
    s1 = NAND(x1, x2)
    s2 = OR(x1, x2)
    y = AND(s1, s2)
    return y

for p in data:
    y = XOR(p[0], p[1])
    print(str(p) + " -> " + str(y))
```

    [0 0] -> 0
    [0 1] -> 1
    [1 0] -> 1
    [1 1] -> 0

### 数式の書き方
参考: https://easy-copy-mathjax.nakaken88.com/
\begin{align}
    y = ax
\end{align}

\begin{align}
\lim_{n \to \infty} n (\sqrt{n^2+1} -n) \\
\end{align}


#  Taylor problem 5.32

last revised: 12-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

**Replace ### by appropriate expressions.**

The equation for an underdamped oscillator, such as a mass on the end of a spring, takes the form

$\begin{align}
  x(t) = e^{-\beta t} [B_1 \cos(\omega_1 t) + B_2 \sin(\omega_1 t)]
\end{align}$

where

$\begin{align}
  \omega_1 = \sqrt{\omega_0^2 - \beta^2}
\end{align}$

and the mass is released from rest at position $x_0$ at $t=0$.

**Goal: plot $x(t)$ for $0 \leq t \leq 20$, with $x_0 = 1$, $\omega_0=1$, and $\beta = 0.$, 0.02, 0.1, 0.3, and 1.**

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def underdamped(t, beta, omega_0=1, x_0=1):
    """Solution x(t) for an underdamped harmonic oscillator."""
    omega_1 = np.sqrt(omega_0**2 - beta**2)
    B_1 = ### fill in the blank
    B_2 = ### fill in the blank
    return np.exp(-beta*t) \
             * ( B_1 * np.cos(omega_1*t) + B_2 * np.sin(omega_1*t) )
```

```python
t_pts = np.arange(0., 20., .01)
betas = [0., 0.02, 0.1, 0.3, 0.9999]

fig = plt.figure(figsize=(10,6))
# look up "python enumerate" to find out how this works!
for i, beta in enumerate(betas):
    ax = fig.add_subplot(2, 3, i+1)
    ax.plot(t_pts, underdamped(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)  # lightened black zero line

fig.tight_layout()
### add code to print the figure
```

## Bonus: Widgetized!

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

omega_0 = 1.

def plot_beta(beta):
    """Plot function for underdamped harmonic oscillator."""
    t_pts = np.arange(0., 20., .01)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(t_pts, underdamped(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)

    fig.tight_layout()

max_value = omega_0 - 0.0001
interact(plot_beta,
         beta=widgets.FloatSlider(min=0., max=max_value, step=0.01,
                                  value=0., readout_format='.2f',
                                  continuous_update=False));

```

Now let's allow for complex numbers!  This will enable us to take $\beta > \omega_0$.

```python
# numpy.lib.scimath version of sqrt handles complex numbers.
#  numpy exp, cos, and sin already can.
import numpy.lib.scimath as smath

def all_beta(t, beta, omega_0=1, x_0=1):
    """Solution x(t) for damped harmonic oscillator, allowing for overdamped
        as well as underdamped solution.
    """
    omega_1 = smath.sqrt(omega_0**2 - beta**2)
    return np.real( x_0 * np.exp(-beta*t) \
               * (np.cos(omega_1*t) + (beta/omega_1)*np.sin(omega_1*t)) )
```

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

omega_0 = 1.

def plot_all_beta(beta):
    """Plot of x(t) for damped harmonic oscillator, allowing for overdamped
       as well as underdamped cases."""
    t_pts = np.arange(0., 20., .01)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(t_pts, all_beta(t_pts, beta), color='blue')
    ax.set_title(rf'$\beta = {beta:.2f}$')
    ax.set_xlabel('t')
    ax.set_ylabel('x(t)')
    ax.set_ylim(-1.1,1.1)
    ax.axhline(0., color='black', alpha=0.3)

    fig.tight_layout()

interact(plot_all_beta,
         beta=widgets.FloatSlider(min=0., max=2, step=0.01,
                                  value=0., readout_format='.2f',
                                  continuous_update=False));

```