# Critical Points and Optimization
We've explored various techniques that we can use to calculate the derivative of a function at a specific *x* value; in other words, we can determine the *slope* of the line created by the function at any point on the line.

This ability to calculate the slope means that we can use derivatives to determine some interesting properties of the function.

## Function Direction at a Point
Consider the following function, which represents the trajectory of a ball that has been kicked on a football field:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Run the Python code below to graph this function and see the trajectory of the ball over a period of 10 seconds.

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

By looking at the graph of this function, you can see that it describes a parabola in which the ball rose in height before falling back to the ground. On the graph, it's fairly easy to see when the ball was rising and when it was falling.

Of course, we can also use  derivative to determine the slope of the function at any point. We can apply some of the rules we've discussed previously to determine the derivative function:

- We can add together the derivatives of the individual terms (***-10x2***, ***100x***, and ***3***) to find the derivative of the entire function.
- The *power* rule tells us that the derivative of ***-10x2*** is ***-10 • 2x***, which is ***-20x***.
- The *power* rule also tells us that the derivative of ***100x*** is ***100***.
- The derivative of any constant, such as ***3*** is ***0***.

So:

\begin{equation}k'(x) = -20x + 100 + 0 \end{equation}

Which of course simplifies to:

\begin{equation}k'(x) = -20x + 100 \end{equation}

Now we can use this derivative function to find the slope for any value of ***x***.

Run the cell below to see a graph of the function and its derivative function:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Look closely at the purple line representing the derivative function, and note that it is a constant  decreasing value - in other words, the slope of the function is reducing linearly as x increases. Even though the function value itself is increasing for the first half of the parabola (while the ball is rising), the slope is becoming less steep (the ball is not rising at such a high rate), until finally the ball reaches its apogee and the slope becomes negative (the ball begins falling).

Note also that the point where the derivative line crosses 0 on the y-axis is also the point where the function value stops increasing and starts decreasing. When the slope has a positive value, the function is increasing; and when the slope has a negative value, the function is decreasing.

The fact that the derivative line crosses 0 at the highest point of the function makes sense if you think about it logically. If you were to draw the tangent line representing the slope at each point, it would be rotating clockwise throughout the graph, initially pointing up and to the right as the ball rises, and turning until it is pointing down and right as the ball falls. At the highest point, the tangent line would be perfectly horizontal, representing a slope of 0.

Run the following code to visualize this:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot tangent slopes for x = 2, 5, and 8
x1 = 2
x2 = 5
x3 = 8
plt.plot([x1-1,x1+1],[k(x1)-(kd(x1)),k(x1)+(kd(x1))], color='red')
plt.plot([x2-1,x2+1],[k(x2)-(kd(x2)),k(x2)+(kd(x2))], color='red')
plt.plot([x3-1,x3+1],[k(x3)-(kd(x3)),k(x3)+(kd(x3))], color='red')

plt.show()
```

Now consider the following function, which represents the number of flowers growing in a flower bed before and after the spraying of a fertilizer:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the derivative values
yd = [wd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that the green line represents the function, showing the number of flowers for 10 days before and after the fertilizer treatment. Before treatment, the number of flowers was in decline, and after treatment the flower bed started to recover.

The derivative function is shown in purple, and once again shows a linear change in slope. This time, the slope is increasing at a constant rate; and once again, the derivative function line crosses 0 at the lowest point in the function line (in other words, the slope changed from negative to positive when the flowers started to recover).

## Critical Points
From what we've seen so far, it seems that there is a relationship between a function reaching an extreme value (a maximum or a minimum), and a derivative value of 0. This makes intuitive sense; the derivative represents the slope of the line, so when a function changes from a negative slope to a positive slope, or vice-versa, the derivative must pass through 0.

However, you need to be careful not to assume that just because the derivative is 0 at a given point, that this point represents the minimum or maximum of the function. For example, consider the following function:

\begin{equation}v(x) = x^{3} - 2x + 100 \end{equation}

Run the following Python code to visualize this function and its corresponding derivative function:

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (2*x) + 100

def vd(x):
    return 3*(x**2) - 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-1000, 2000, 100))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that in this case, the purple derivative function line passes through 0 as the green function line transitions from a *concave downwards* slope (a slope that is decreasing) to a *concave upwards* slope (a slope that is increasing). The slope flattens out to 0, forming a "saddle" before the it starts increasing.

What we can learn from this is that interesting things seem to happen to the function when the derivative is 0. We call points where the derivative crosses 0 *critical points*, because they indicate that the function is changing direction. When a function changes direction from positive to negative, it forms a peak (or a *local maximum*), when the function changes direction from negative to positive it forms a trough (or *local minimum*), and when it maintains the same overall direction but changes the concavity of the slope it creates an *inflexion point*.

## Finding Minima and Maxima
A common use of calculus is to find minimum and maximum points in a function. For example, we might want to find out how many seconds it took for the kicked football to reach its maximum height, or how long it took for our fertilizer to be effective in reversing the decline of flower growth.

We've seen that when a function changes direction to create a maximum peak or a minimum trough, the derivative of the function is 0, so a step towards finding these extreme points might be to simply find all of the points in the function where the derivative is 0. For example, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

From this, we've calculated the function for the derivative as:

\begin{equation}k'(x) = -20x + 100 \end{equation}

We can then solve the derivative equation for an f'(x) value of 0:

\begin{equation}-20x + 100 = 0 \end{equation}

We can remove the constant by subtracting 100 to both sides:

\begin{equation}-20x = -100 \end{equation}

Multiplying both sides by -1 gets rid of the negative values (this isn't strictly necessary, but makes the equation a little less confusing)

\begin{equation}20x = 100 \end{equation}

So:

\begin{equation}x = 5 \end{equation}

So we know that the derivative will be 0 when *x* is 5, but is this a minimum, a maximum, or neither? It could just be an inflexion point, or the entire function could be a constant value with a slope of 0) Without looking at the graph, it's difficult to tell.

## Second Order Derivatives
The solution to our problem is to find the derivative of the derivative! Until now, we've found the derivative of a function, and indicated it as ***f'(x)***. Technically, this is known as the *prime* derivative; and it describes the slope of the function. Since the derivative function is itself a function, we can find its derivative, which we call the *second order* (or sometimes just *second*) derivative. This is indicated like this: ***f''(x)***.

So, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Here's the function for the prime derivative:

\begin{equation}k'(x) = -20x + 100 \end{equation}

And using a combination of the power rule and the constant rule, here's the function for the second derivative:

\begin{equation}k''(x) = -20 \end{equation}

Now, without even drawing the graph, we can see that the second derivative has a constant value; so we know that the slope of the prime derivative is linear; and because it's a negative value, we know that it is decreasing. So when the prime derivative crosses 0, it we know that the slope of the function is decreasing linearly; so the point at *x=0* must be a maximum point.

Run the following code to plot the function, the prime derivative, and the second derivative for the kicked ball:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

def k2d(x):
    return -20

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the k'(x) values
yd = [kd(i) for i in x]

# Use the 2-derivative function to get the k''(x)
y2d = [k2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot  k'(x)
plt.plot(x,yd, color='purple')

# Plot k''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

Let's take the same approach for the flower bed problem. Here's the function:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

Using the power rule and constant rule, gives us the prime derivative function:

\begin{equation}w'(x) = 2x + 2 \end{equation}

Applying the power rule and constant rule to the prime derivative function gives us the second derivative function:

\begin{equation}w''(x) = 2 \end{equation}

Note that this time, the second derivative is a positive constant, so the prime derivative (which is the slope of the function) is increasing linearly. The point where the prime derivative crosses 0 must therefore be a minimum. Let's run the code below to check:

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

def w2d(x):
    return 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the w'(x) values
yd = [wd(i) for i in x]

# Use the 2-derivative function to get the w''(x) values
y2d = [w2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot w'(x)
plt.plot(x,yd, color='purple')

# Plot w''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

## Critical Points that are *Not* Maxima or Minima
Of course, it's possible for a function to form a "saddle" where the prime derivative is zero at a point that is not a minimum or maximum. Here's an example of a function like this:

\begin{equation}v(x) = x^{3} - 6x^{2} + 12x + 2 \end{equation}

And here's its prime derivative:

\begin{equation}v'(x) = 3x^{2} - 12x + 12 \end{equation}

Let's find a critical point where v'(x) = 0

\begin{equation}3x^{2} - 12x + 12 = 0 \end{equation}

Factor the x-terms

\begin{equation}3x(x - 4) = 12 \end{equation}

Divide both sides by 3:

\begin{equation}x(x - 4) = 4 \end{equation}

Factor the x terms back again

\begin{equation}x^{2} - 4x = 4 \end{equation}

Complete the square, step 1

\begin{equation}x^{2} - 4x + 4 = 0 \end{equation}

Complete the square, step 2

\begin{equation}(x - 2)^{2} = 0 \end{equation}

Find the square root:

\begin{equation}x - 2 = \pm\sqrt{0}\end{equation}

\begin{equation}x - 2 = +\sqrt{0} = 0, -\sqrt{0} = 0\end{equation}

v'(2) = 0 (only touches 0 once)

Is it a maximum or minimum? Let's find the second derivative:

\begin{equation}v''(x) = 6x - 12\end{equation}

So

\begin{equation}v''(2) = 0\end{equation}

So it's neither negative or positive, so it's not a maximum or minimum.

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (6*(x**2)) + (12*x) + 2

def vd(x):
    return (3*(x**2)) - (12*x) + 12

def v2d(x):
    return (3*(2*x)) - 12

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-5, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Use the derivative function to get the derivative values
y2d = [v2d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-2000, 2000, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot the derivative
plt.plot(x,y2d, color='magenta')

plt.show()

print ("v(2) = " + str(v(2)))

print ("v'(2) = " + str(vd(2)))

print ("v''(2) = " + str(v2d(2)))

```

## Optimization
The ability to use derivatives to find minima and maxima of a function makes it a useful tool for scenarios where you need to optimize a function for a specific variable.

### Defining Functions to be Optimized
For example, suppose you have decided to build an online video service that is based on a subscription model. You plan to charge a monthly subscription fee, and you want to make the most revenue possible. The problem is that customers are price-sensitive, so if you set the monthly fee too high, you'll deter some customers from signing up. Conversely, if you set the fee too low, you may get more customers, but at the cost of reduced revenue.

What you need is some kind of function that will tell you how many subscriptions you might expect to get based on a given fee. So you've done some research, and found a formula to indicate that the expected subscription volume (in thousands) can be calculated as 5-times the monthly fee subtracted from 100; or expressed as a function:

\begin{equation}s(x) = -5x + 100\end{equation}

What you actually want to optimize is monthly revenue, which is simply the number of subscribers multiplied by the fee:

\begin{equation}r(x) = s(x) \cdot x\end{equation}

We can combine ***s(x)*** into ***r(x)*** like this:

\begin{equation}r(x) = -5x^{2} + 100x\end{equation}

### Finding the Prime Derivative
The function ***r(x)*** will return the expected monthly revenue (in thousands) for any proposed fee (*x*). What we need to do now is to find the fee that yields the maximum revenue. Fortunately, we can use a derivative to do that.

First, we need to determine the prime derivative of ***r(x)***, and we can do that easily using the power rule:

\begin{equation}r'(x) = 2 \cdot -5x + 100\end{equation}

Which is:

\begin{equation}r'(x) = -10x + 100\end{equation}

### Find Critical Points
Now we need to find any critical points where the derivative is 0, as this could indicate a maximum:

\begin{equation}-10x + 100 = 0\end{equation}

Let's isolate the *x* term:

\begin{equation}-10x = -100\end{equation}

Both sides are negative, so we can mulitply both by -1 to make them positive without affecting the equation:

\begin{equation}10x = 100\end{equation}

Now we can divide both sides by 10 to isolate *x*:

\begin{equation}x = \frac{100}{10}\end{equation}

So:

\begin{equation}x = 10\end{equation}

#### Check for a Maximum
We now know that with an *x* value of of **10**, the derivative is 0; or put another way, when the fee is 10, the slope indicating the change in subscription volume is flat. This could potentially be a point where the change in subscription volume has peaked (in other words, a maximum); but it could also be a minimum or just an inflexion point where the rate of change transitions from negative to positive.

To be sure, we can check the second order derivative. We can calculate this by applying the power rule to the prime derivative:

\begin{equation}r''(x) = -10\end{equation}

Note that the second derivative is a constant with a negative value. It will be the same for any point, including our critical point at *x=10*:

\begin{equation}r''(10) = -10\end{equation}

A negative value for the second derivative tells us that the derivative slope is moving in a negative direction at the point where it is 0, so the function value must be at a maximum.

In other words, the optimal monthly fee for our online video service is 10 - this will generate the maximum monthly revenue.

Run the code below to show the function ***r(x)*** as a graph, and verify that the maximum point is at x = 10.

```python
%matplotlib inline

# Create function s
def s(x):
    return (-5*x) + 100

# Create function r
def r(x):
    return s(x) * x

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 21))

# Use the function to get the y values
y = [r(i) for i in x]

# Set up the graph
plt.xlabel('x (monthly fee)')
plt.ylabel('r(x) (revenue in $,000)')
plt.xticks(range(0,22, 1))
plt.yticks(range(0, 600, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

```python

```


# https://en.wikipedia.org/wiki/Finite_difference

Three forms are commonly considered: forward, backward, and central differences.[1][2][3]

A forward difference is an expression of the form

$$ \displaystyle \Delta _{h}[f](x)=f(x+\Delta x)-f(x).$$
Depending on the application, the spacing h may be variable or constant. When omitted, $\Delta x=h$ is taken to be 1: Δ[ f ](x) = Δ1[ f ](x).

A backward difference uses the function values at x and x − \Delta, instead of the values at x + \Delta and x:

$$ \displaystyle \nabla _{h}[f](x)=f(x)-f(x-\Delta x).$$

Finally, the central difference is given by

$$\displaystyle \delta _{h}[f](x) = f\left(x+{\tfrac {1}{2}}\Delta x\right)-f\left(x-{\tfrac {1}{2}}\Delta x \right) $$

The derivative of a function f at a point x is defined by the limit.

$$ f'(x)=\lim_{h\to 0} {\frac {f(x+h)-f(x)}{h}} $$

```python
# red dashes, blue squares and green triangles
#Example: [a,b], n
# https://matplotlib.org/users/pyplot_tutorial.html
import numpy as np
import matplotlib.pyplot as plt
a=0
b=1
n=3
deltax=(b-a)/n
deltax
# evenly sampled time at delta x intervals
x = np.arange(a, b+deltax, deltax)
#x = np.linspace(a, b, n+1)
x
x = np.linspace(-3, 3, 50)
y2 = x**2+1

plt.figure()
#set x limits
plt.xlim((0, 2))
plt.ylim((0, 3))

# set new sticks
new_sticks = np.linspace(0, 2, 5)
plt.xticks(new_sticks)
# set tick labels
plt.yticks(np.arange(0, 5, step=1))

# set line styles

l2, = plt.plot(x, y2, color='red', linewidth=1.0, linestyle='--', label='f(x)= x^2+1')

plt.legend(loc='upper left')

plt.show()
```

plot a secant line pass the points (0,1) and (1,2)

```python
import matplotlib.pyplot as plt
import numpy as np

def main():
    # x = np.linspace(-2,2,100)
    a=-2
    b=3
    divx=0.01
    x = np.arange(a, b, divx)
    x1=0
    p1 = int((x1-a)/divx)  #starts from zero
    deltax=1
    count_deltax=int(deltax/divx)
    p2 = p1+ count_deltax #starts from zero

    y1 = main_func(x)
    y2 = calculate_secant(x, y1, p1, p2)
    plot(x, y1, y2)
    plt.show()

def main_func(x):
    return x**2+1

def calculate_secant(x, y, p1, p2):
    points = [p1, p2]
    m, b = np.polyfit(x[points], y[points], 1)
    return m * x + b

def plot(x, y1, y2):
    plt.plot(x, y1)
    plt.plot(x, y2)
    #set x limits
    plt.xlim((-2, 2))
    #set x limits
    plt.ylim((0, 4))

main()
```

forward difference

```python
x=0
deltax=1
main_func(x+deltax)-main_func(x)
```

    1

```python
plot a tangent secant line pass the points (0,1)
```

```python
import matplotlib.pyplot as plt
import numpy as np

def main():
    # x = np.linspace(-2,2,100)
    a=-2
    b=3
    divx=0.01
    x = np.arange(a, b, divx)
    x1=1
    p1 = int((x1-a)/divx)  #starts from zero
    deltax=0.02
    count_deltax=int(deltax/divx)
    p2 = p1+ count_deltax #starts from zero

    y1 = main_func(x)
    y2 = calculate_secant(x, y1, p1, p2)
    plot(x, y1, y2)
    plt.show()

def main_func(x):
    return x**2+1

def calculate_secant(x, y, p1, p2):
    points = [p1, p2]
    m, b = np.polyfit(x[points], y[points], 1)
    return m * x + b

def plot(x, y1, y2):
    plt.plot(x, y1)
    plt.plot(x, y2)
    #set x limits
    plt.xlim((-2, 2))
    #set x limits
    plt.ylim((0, 4))

main()
```

```python
import matplotlib.pyplot as plt
import numpy as np

def main():
    # x = np.linspace(-2,2,100)
    a=-2
    b=3
    divx=0.01
    x = np.arange(a, b, divx)
    x1=1
    p1 = int((x1-a)/divx)  #starts from zero
    deltax=0.01
    count_deltax=int(deltax/divx)
    p2 = p1+ count_deltax #starts from zero

    y1 = main_func(x)
    y2 = calculate_secant(x, y1, p1, p2)
    plot(x, y1, y2)
    plt.show()

def main_func(x):
    return x**2+1

def calculate_secant(x, y, p1, p2):
    points = [p1, p2]
    m, b = np.polyfit(x[points], y[points], 1)
    print(m)
    return m * x + b

def plot(x, y1, y2):
    plt.plot(x, y1)
    plt.plot(x, y2)

    #set x limits
    plt.xlim((-2, 2))
    #set x limits
    plt.ylim((0, 4))

main()

```

```python
x=1
deltax=0.00000000001
(main_func(x+deltax)-main_func(x))/deltax
```

    2.000000165480742

### $$ f'(x)=\lim_{h\to 0} {\frac {f(x+h)-f(x)}{h}} $$
$$ f'(x)={\frac {f(1+2)-f(1)}{2}} = 4$$

The derivative of a function f at a point x is defined by the limit.

$$ f'(x)=\lim_{h\to 0} {\frac {f(x+h)-f(x)}{h}} $$

http://www.math.unl.edu/~s-bbockel1/833-notes/node23.html
forward difference approximation:
$$ f'(x)={\frac {f(x+h)-f(x)}{h}}+O(h) $$

$$ f'(1)=?$$

```python
from sympy import diff, Symbol, sin, tan, limit
x = Symbol('x')
diff(main_func(x), x)
limit(main_func(x), x, 1)
```

    2

```python
import matplotlib.pyplot as plt
import numpy as np

def main():
    # x = np.linspace(-2,2,100)
    a=-2
    b=3
    divx=0.01
    x = np.arange(a, b, divx)
    x1=1
    p1 = int((x1-a)/divx)  #starts from zero
    deltax=0.01
    count_deltax=int(deltax/divx)
    p2 = p1+ count_deltax #starts from zero

    y1 = main_func(x)
    y2 = calculate_secant(x, y1, p1, p2)
    plot(x, y1, y2)
    plt.show()

def main_func(x):
    return x**2+1

def calculate_secant(x, y, p1, p2):
    points = [p1, p2]
    m, b = np.polyfit(x[points], y[points], 1)
    print(m)
    return m * x + b

def plot(x, y1, y2):
    plt.plot(x, y1)
    plt.plot(x, y2)

    #set x limits
    plt.xlim((-2, 2))
    #set x limits
    plt.ylim((0, 4))

main()
```

A backward difference uses the function values at x and x − \Delta, instead of the values at x + \Delta and x:

$$ f'(x)=\lim_{h\to 0} {\tfrac{f(x)-f(x-\Delta x)}{\Delta x}}$$

```python
x=1
deltax=0.0001
(main_func(x)-main_func(x-deltax))/deltax
```

    1.9999000000003875

Finally, the central difference is given by

$$f'(x)=\lim_{h\to 0} {\tfrac {f\left(x+{\tfrac {1}{2}}\Delta x\right)-f\left(x-{\tfrac {1}{2}}\Delta x \right)}{\Delta x}} $$

```python
x=1
deltax=0.0001
(main_func(x+deltax*(1/2))-main_func(x-deltax*(1/2)))/deltax
```

    1.9999999999997797

```python

```


dS/dt=-bSI, dI/dt=bSI (uso b para beta)

```python
from sympy import *
from sympy.abc import S,I,t,b
```

```python
#puntos criticos
P=-b*S*I
Q=b*S*I
#establecer P(S,I)=0 y Q(S,I)=0
Peqn=Eq(P,0)
Qeqn=Eq(Q,0)
print(solve((Peqn,Qeqn),S,I))
#matriz Jacobiana
J11=diff(P,S)
J12=diff(P,I)
J21=diff(Q,S)
J22=diff(Q,I)
J=Matrix([[J11,J12],[J21,J22]])
pprint(J)
```

    [(0, I), (S, 0)]
    ⎡-I⋅b  -S⋅b⎤
    ⎢          ⎥
    ⎣I⋅b   S⋅b ⎦

```python
#J en el punto critico
Jc1=J.subs([(S,0),(I,I)])
pprint(Jc1.eigenvals())
pprint(Jc1.eigenvects())
Jc2=J.subs([(S,S),(I,0)])
pprint(Jc2.eigenvals())
pprint(Jc2.eigenvects())
```

    {0: 1, -I⋅b: 1}
    ⎡⎛      ⎡⎡0⎤⎤⎞  ⎛         ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜-I⋅b, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣1⎦⎦⎠  ⎝         ⎣⎣1 ⎦⎦⎠⎦
    {0: 1, S⋅b: 1}
    ⎡⎛      ⎡⎡1⎤⎤⎞  ⎛        ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜S⋅b, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣0⎦⎦⎠  ⎝        ⎣⎣1 ⎦⎦⎠⎦

Los puntos criticos son no hiperbolicos.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
import pylab as pl
import matplotlib
```

```python
b=1
def dx_dt(x,t):
    return [ -b*x[0]*x[1] , b*x[1]*x[0] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=20)
plt.ylabel('I',fontsize=20)
plt.tick_params(labelsize=12)
plt.ticklabel_format(style="sci", scilimits=(0,0))
plt.xlim(0,100000)
plt.ylim(0,100000)
#campo vectorial
X,Y=np.mgrid[0:100000:15j,0:100000:15j]
u=-b*X*Y
v=b*Y*X
pl.quiver(X,Y,u,v,color='dimgray')
plt.savefig("SIinf.pdf",bbox_inches='tight')
plt.show()
```

Analisis de Bifurcaciones

El punto critico del sistema no depende del parametro beta por lo que no cambia al variar beta.


# Multivariate Gaussian with full covariance

In this reading you will learn how you can use TensorFlow to specify any multivariate Gaussian distribution.

```python
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

print("TF version:", tf.__version__)
print("TFP version:", tfp.__version__)
```

    TF version: 2.3.0
    TFP version: 0.11.0

So far, you've seen how to define multivariate Gaussian distributions using `tfd.MultivariateNormalDiag`. This class allows you to specify a multivariate Gaussian with a diagonal covariance matrix $\Sigma$.

In cases where the variance is the same for each component, i.e. $\Sigma = \sigma^2 I$, this is known as a _spherical_ or _isotropic_ Gaussian. This name comes from the spherical (or circular) contours of its probability density function, as you can see from the plot below for the two-dimensional case.

```python
# Plot the approximate density contours of a 2d spherical Gaussian

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

spherical_2d_gaussian = tfd.MultivariateNormalDiag(loc=[0., 0.])

N = 100000
x = spherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x1, x2, kind='kde', space=0, );
```

As you know, a diagonal covariance matrix results in the components of the random vector being independent.

## Full covariance with `MultivariateNormalFullTriL`

You can define a full covariance Gaussian distribution in TensorFlow using the Distribution `tfd.MultivariateNormalTriL`.

Mathematically, the parameters of a multivariate Gaussian are a mean $\mu$ and a covariance matrix $\Sigma$, and so the `tfd.MultivariateNormalTriL` constructor requires two arguments:

- `loc`, a Tensor of floats corresponding to $\mu$,
- `scale_tril`, a a lower-triangular matrix $L$ such that $LL^T = \Sigma$.

For a $d$-dimensional random variable, the lower-triangular matrix $L$ looks like this:

\begin{equation}
    L = \begin{bmatrix}
            l_{1, 1} & 0 & 0 & \cdots & 0 \\
            l_{2, 1} & l_{2, 2} & 0 & \cdots & 0  \\
            l_{3, 1} & l_{3, 2} & l_{3, 3} & \cdots & 0  \\
            \vdots  & \vdots & \vdots & \ddots & \vdots \\
            l_{d, 1} & l_{d, 2} & l_{d, 3} & \cdots & l_{d, d}
        \end{bmatrix},
\end{equation}

where the diagonal entries are positive: $l_{i, i} > 0$ for $i=1,\ldots,d$.

Here is an example of creating a two-dimensional Gaussian with non-diagonal covariance:

```python
# Set the mean and covariance parameters

mu = [0., 0.]  # mean
scale_tril = [[1.,  0.],
              [0.6, 0.8]]

sigma = tf.matmul(tf.constant(scale_tril), tf.transpose(tf.constant(scale_tril)))  # covariance matrix
print(sigma)
```

    tf.Tensor(
    [[1.  0.6]
     [0.6 1. ]], shape=(2, 2), dtype=float32)

```python
# Create the 2D Gaussian with full covariance

nonspherical_2d_gaussian = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
nonspherical_2d_gaussian
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Check the Distribution mean

nonspherical_2d_gaussian.mean()
```

    <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>

```python
# Check the Distribution covariance

nonspherical_2d_gaussian.covariance()
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[1. , 0.6],
           [0.6, 1. ]], dtype=float32)>

```python
# Plot its approximate density contours

x = nonspherical_2d_gaussian.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
sns.jointplot(x1, x2, kind='kde', space=0, color='r');
```

As you can see, the approximate density contours are now elliptical rather than circular. This is because the components of the Gaussian are correlated.

Also note that the marginal distributions (shown on the sides of the plot) are both univariate Gaussian distributions.

## The Cholesky decomposition

In the above example, we defined the lower triangular matrix $L$ and used that to build the multivariate Gaussian distribution. The covariance matrix is easily computed from $L$ as $\Sigma = LL^T$.

The reason that we define the multivariate Gaussian distribution in this way - as opposed to directly passing in the covariance matrix - is that not every matrix is a valid covariance matrix. The covariance matrix must have the following properties:

1. It is symmetric
2. It is positive (semi-)definite

_NB: A symmetric matrix $M \in \mathbb{R}^{d\times d}$ is positive semi-definite if it satisfies $b^TMb \ge 0$ for all nonzero $b\in\mathbb{R}^d$. If, in addition, we have $b^TMb = 0 \Rightarrow b=0$ then $M$ is positive definite._

The Cholesky decomposition is a useful way of writing a covariance matrix. The decomposition is described by this result:

> For every real-valued symmetric positive-definite matrix $M$, there is a unique lower-diagonal matrix $L$ that has  positive diagonal entries for which
>
> \begin{equation}
     LL^T = M
 \end{equation}
> This is called the _Cholesky decomposition_ of $M$.

This result shows us why Gaussian distributions with full covariance are completely represented by the `MultivariateNormalTriL` Distribution.

### `tf.linalg.cholesky`

In case you have a valid covariance matrix $\Sigma$ and would like to compute the lower triangular matrix $L$ above to instantiate a `MultivariateNormalTriL` object, this can be done with the `tf.linalg.cholesky` function.

```python
# Define a symmetric positive-definite matrix

sigma = [[10., 5.], [5., 10.]]
```

```python
# Compute the lower triangular matrix L from the Cholesky decomposition

scale_tril = tf.linalg.cholesky(sigma)
scale_tril
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[3.1622777, 0.       ],
           [1.5811388, 2.7386127]], dtype=float32)>

```python
# Check that LL^T = Sigma

tf.linalg.matmul(scale_tril, tf.transpose(scale_tril))
```

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[10.      ,  5.      ],
           [ 5.      ,  9.999999]], dtype=float32)>

If the argument to the `tf.linalg.cholesky` is not positive definite, then it will fail:

```python
# Try to compute the Cholesky decomposition for a matrix with negative eigenvalues

bad_sigma = [[10., 11.], [11., 10.]]

try:
    scale_tril = tf.linalg.cholesky(bad_sigma)
except Exception as e:
    print(e)
```

    Cholesky decomposition was not successful. The input might not be valid. [Op:Cholesky]

### What about positive semi-definite matrices?

In cases where the matrix is only positive semi-definite, the Cholesky decomposition exists (if the diagonal entries of $L$ can be zero) but it is not unique.

For covariance matrices, this corresponds to the degenerate case where the probability density function collapses to a subspace of the event space. This is demonstrated in the following example:

```python
# Create a multivariate Gaussian with a positive semi-definite covariance matrix

psd_mvn = tfd.MultivariateNormalTriL(loc=[0., 0.], scale_tril=[[1., 0.], [0.4, 0.]])
psd_mvn
```

    <tfp.distributions.MultivariateNormalTriL 'MultivariateNormalTriL' batch_shape=[] event_shape=[2] dtype=float32>

```python
# Plot samples from this distribution

x = psd_mvn.sample(N)
x1 = x[:, 0]
x2 = x[:, 1]
plt.xlim(-5, 5)
plt.ylim(-5, 5)
plt.title("Scatter plot of samples")
plt.scatter(x1, x2, alpha=0.5);
```

If the input to the function `tf.linalg.cholesky` is positive semi-definite but not positive definite, it will also fail:

```python
# Try to compute the Cholesky decomposition for a positive semi-definite matrix

another_bad_sigma = [[10., 0.], [0., 0.]]

try:
    scale_tril = tf.linalg.cholesky(another_bad_sigma)
except Exception as e:
    print(e)
```

    Cholesky decomposition was not successful. The input might not be valid. [Op:Cholesky]

In summary: if the covariance matrix $\Sigma$ for your multivariate Gaussian distribution is positive-definite, then an algorithm that computes the Cholesky decomposition of $\Sigma$ returns a lower-triangular matrix $L$ such that $LL^T = \Sigma$. This $L$ can then be passed as the `scale_tril` of `MultivariateNormalTriL`.

## Putting it all together

You are now ready to put everything that you have learned in this reading together.

To create a multivariate Gaussian distribution with full covariance you need to:

1. Specify parameters $\mu$ and either $\Sigma$ (a symmetric positive definite matrix) or $L$ (a lower triangular matrix with positive diagonal elements), such that $\Sigma = LL^T$.

2. If only $\Sigma$ is specified, compute `scale_tril = tf.linalg.cholesky(sigma)`.

3. Create the distribution: `multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)`.

```python
# Create a multivariate Gaussian distribution

mu = [1., 2., 3.]
sigma = [[0.5, 0.1, 0.1],
         [0.1,  1., 0.6],
         [0.1, 0.6, 2.]]

scale_tril = tf.linalg.cholesky(sigma)

multivariate_normal = tfd.MultivariateNormalTriL(loc=mu, scale_tril=scale_tril)
```

```python
# Check the covariance matrix

multivariate_normal.covariance()
```

    <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
    array([[0.4999999 , 0.09999999, 0.09999999],
           [0.09999999, 1.0000001 , 0.6000001 ],
           [0.09999999, 0.6000001 , 2.0000002 ]], dtype=float32)>

```python
# Check the mean

multivariate_normal.mean()
```

    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([1., 2., 3.], dtype=float32)>

## Deprecated: `MultivariateNormalFullCovariance`

There was previously a class called `tfd.MultivariateNormalFullCovariance` which takes the full covariance matrix in its constructor, but this is being deprecated. Two reasons for this are:

* covariance matrices are symmetric, so specifying one directly involves passing redundant information, which involves writing unnecessary code.
* it is easier to enforce positive-definiteness through constraints on the elements of a decomposition than through a covariance matrix itself. The decomposition's only constraint is that its diagonal elements are positive, a condition that is easy to parameterize for.

### Further reading and resources
* https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MultivariateNormalTriL
* https://www.tensorflow.org/api_docs/python/tf/linalg/cholesky


# Integral simbólica usando SymPy

```python
import numpy as np
import sympy as sy
from sympy.utilities.lambdify import lambdify
from scipy.integrate import quad
from scipy.misc import derivative
```

```python
x = sy.Symbol('x')
f = sy.exp(-x)*sy.sin(3.0*x)
res_symbolic = sy.integrate(f)
res = sy.integrate(f, (x, 0, 2*sy.pi))

print(res.evalf())
```

    0.299439767180488

```python
f
```

$\displaystyle e^{- x} \sin{\left(3.0 x \right)}$

```python
res_symbolic
```

$\displaystyle - 0.1 e^{- x} \sin{\left(3.0 x \right)} - 0.3 e^{- x} \cos{\left(3.0 x \right)}$

```python
%timeit sy.integrate(f, (x, 0, 2*sy.pi))
```

    207 ms ± 9.18 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
# Lambdfy integrals
res_symbolic = sy.integrate(f)
integ = lambda x0, x1: res_symbolic.evalf(subs={x: x1}) - res_symbolic.evalf(subs={x: x0})
%timeit integ(0, 2*np.pi)
```

    1.53 ms ± 88.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
integ = lambda x0, x1: float(sy.integrate(f, (x, x0, x1)))
%timeit integ(0, 2*np.pi)
```

    171 ms ± 12.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

## Multi-varible

```python
x, y = sy.symbols('x, y')
f = x**2
h = y**2
g = f + h
g1 = g.subs(x,1)

```

$\displaystyle 5$

```python
# Integrate g(x,y)*dx
sy.integrate(g,x)
```

$\displaystyle \frac{x^{3}}{3} + x y^{2}$

```python
# Integrate g(x,y)*dy
sy.integrate(g,y)
```

$\displaystyle x^{2} y + \frac{y^{3}}{3}$

```python
# Double integral g(x,y)*dx*dy
sy.integrate(sy.integrate(g,x),y)
```

$\displaystyle \frac{x^{3} y}{3} + \frac{x y^{3}}{3}$

```python
# Double integral g(x,y)*dx*dy, xfrom 0 to 1, y from zero to 1
sy.integrate(sy.integrate(g,(x,0,1)),(y,0,1))
```

$\displaystyle \frac{2}{3}$

```python
# Evaluating the results
sy.integrate(sy.integrate(g,(x,0,1)),(y,0,1)).evalf()
```

$\displaystyle 0.666666666666667$

```python
# Show the symbolic
sy.Integral(sy.Integral(g,(x,0,1)),(y,0,1))
```

$\displaystyle \int\limits_{0}^{1}\int\limits_{0}^{1} \left(x^{2} + y^{2}\right)\, dx\, dy$

# Using Scipy to Numerical integrate defined functions

```python
def f(x):
    return np.exp(-x)*np.sin(3.0*x)

i, err = quad(f, 0, 2*np.pi)
print(i)
```

    0.29943976718048754

```python
%timeit i, err = quad(f, 0, 2*np.pi)
```

    9.72 µs ± 1.04 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
def foo(x, y):
  return(x**2 + x*y**3)

from scipy.misc import derivative
derivative(foo, 1, dx = 1e-6, args = (2, ))
```

    9.999999999621423

# Derivada simbólica

```python
x = sy.Symbol('x')
f = 3*x**2 + 1
f
```

$\displaystyle 3 x^{2} + 1$

```python
# Lambdfy derivatives
ddx = lambdify(x, sy.diff(f, x)) # creates a function that you can call
%timeit ddx(2)
```

    91.8 ns ± 10.2 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

```python
dx = sy.diff(f, x)
ddx = lambda x0: dx.subs(x, x0)
%timeit ddx(2)
```

    14.2 µs ± 345 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
# Derivada de segunda ordem
sy.diff(f, (x, 2))
```

$\displaystyle 6$

# Derivada numérica usando Scipy

```python
def f(x):
    return 3*x**2 + 1

%timeit derivative(f, 2)
```

    13.3 µs ± 401 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)


#   Numerical investigation of an oscillator
We consider a particle of mass $m = 1$ moving on the horizontal $x$ axis in a potential $U=\omega_{0}^2 x^{2}/2$,
subject to a friction force $F_{friction}=-2\gamma \dot{x}$ and a driving force $F_{driving}=\alpha\cos(t+\phi)$. The equation of motion is therefore of the general form

\begin{equation} \ddot{x}+2\gamma \dot{x}+ \omega_0^2x-\alpha\cos(t+\phi)=0  \qquad(1)\end{equation}

This is a harmonic oscillator with damping (friction) and driving. We want to study the solutions of this equation in terms of the free parameters of the problem. These are the initial position $x_{0}$, the initial velocity $\dot{x}_{0}$, the frequency $\omega_{0}$, and the parameters $\gamma$, $\alpha$, and $\phi$. Once we have the
solution we want to produce plots of the position, the velocity, the potential and kinetic energy,and the phase portrait (velocity in terms of position).
### - First step:
By denoting the velocity $v$ (i.e. $v = \dot{x}$), rewrite (1) as a system of first order ODEs in $x$ and $v$.

$\frac{dx}{dt} = v$

$\frac{dv}{dt}$ = -2$\gamma v$ - $w^{2}_{0}x$ + $\alpha\cos(t+\phi)$

### - Second step:
Let $\{t_k \}$ be a partition of $[0,60]$ such that $0=t_0<t_1<\cdots<t_{600}=60$ and $H$ be the constant length of the $k$-th subinterval (i.e. $H = t_k -t_{k-1}$).

Using odeint, write a Python function named Oscillator_func which takes $x_{0}$, $\dot{x}_{0}$, $\omega_{0}$, $\gamma$, $\alpha$ and  $\phi$ as inputs and return the arrays $x$ and $v$ (numerical solution of the system of ODEs obtained above) such that for a given index $k$ we have $x[k]= x(t_k)$ and $v[k] = v(t_k)$

```python
from scipy.integrate import odeint
from pylab import*
def F (s,t):
    [x,v] =s
    dxdt = v
    dvdt = -2*gamma*v -(w_0)**2*x + alpha*(np.cos(t+phi))
    return [dxdt,dvdt]
def Oscillator_func(x_0,x_dot_0,gamma,w_0,alpha,phi):
    s0 = [x_0,x_dot_0]
    N = (int((b-a)/H))+1
    t=np.linspace(0,60,N)
    sol = odeint(F,s0,t)
    x = sol[:,0]
    v = sol[:,1]
    return x,v,t

```

### - Third step:
You can then play with the numerical values ($x_{0}$, $\dot{x}_{0}$, $\omega_{0}$, $\gamma$, $\alpha$, $\phi$) to investigate different situations. From now on, we assume that

 \begin{equation} x_{0}=5,\qquad \dot{x}_{0}=0,\qquad \omega_{0}=0.8,\qquad \phi=10.\end{equation}

write a code which produces the following three graphs:

 - First graph:
position $x(t)$ and velocity $\dot{x}(t)$, both as a function of time.

 - Second graph:
 potential energy $U(t)$ and kinetic energy $K(t)$, both as a function of time.

 - Third graph:
 a parametric plot representing the position $x(t)$ in terms of the position $\dot{x}(t)$.

 #### a- for  $\alpha=0, \gamma=0$ (Harmonic oscillator)

This is the usual harmonic oscillator, with the position and the velocity oscillating for ever, as you
can see on this plot:

 The kinetic energy $K$ and the potential energy $U$ both oscillate for ever, and the total energy $E = K + U$ is equal to a constant at any time. This is conservation of energy.

 The phase portrait is closed, and the trajectories go for ever around this ellipse representing pairs
of points $x(t)$ and $\dot{x}(t)$. We see that $x(t)$ and $\dot{x}(t)$ never reach $0$.

```python
x_0 =5
x_dot_0 =0
gamma = 0
w_0 = 0.8
alpha =0
phi =10
a=0
b=60
H=0.001
x,v,t=Oscillator_func(x_0,x_dot_0,gamma,w_0,alpha,phi)
K = (1/2)*((w_0)**2)*(x**2)
U = (1/2)*(v**2)
plt.figure(figsize=(8,14))
plt.subplot(3,1,1)
plt.plot(t,x)
plt.plot(t,v)
plt.legend(['x(t)','v(t)'])
plt.subplot(3,1,2)
plt.plot(t,K)
plt.plot(t,U)
plt.legend(['Kinetic Energy','Potential Energy'])
plt.subplot(3,1,3)
plt.plot(x,v)
plt.show()
```

#### b- for  $\alpha=0, \gamma=0.1$ (Damped oscillator)

In this case we have $\gamma^2 < \omega_{0}^2$, which corresponds to the under-damped regime discussed in class. The solution has an oscillatoray behavior with an exponential damping, as you can see on this
plot:

Es expected, the mechanical energy is not conserved because of friction (this is why the motion
eventually stops). You can see on this plot that energy goes to zero:

In the phase portrait the trajectory starts at $x_{0}=5$ and $\dot{x}_{0}=0$, which is our initial condition, and
is attracted to the point $x = 0$ and $\dot{x}=0$. This is consistent with what we see above on the plots
of $x(t)$ and $\dot{x}(t)$.

```python
x_0 =5
x_dot_0 =0
w_0 = 0.8
alpha =0
phi =10
gamma = 0.1
x,v,t=Oscillator_func(x_0,x_dot_0,gamma,w_0,alpha,phi)
plt.figure(figsize=(10,16))
plt.subplot(3,1,1)
plt.plot (t,x)
plt.plot(t,v)
plt.legend(['x(t)','v(t)'])
plt.subplot(3,1,2)
plt.plot(t,K)
plt.plot(t,U)
plt.legend(['Kinetic Energy','Potential Energy'])
plt.subplot(3,1,3)
plt.plot(x,v)
plt.show()
```

#### c- for  $\alpha=1, \gamma=0.1$ (Damped oscillator with driving)

In this case the motion is initially damped, as in the previous case, but after some time the driving force takes over and becomes dominant, so the motion is again oscillatory, where the oscillations are dictated by the driving. This can be seen on the plot.

The energy is initially decreasing, as in the damped case, but then the driving starts to inject energy in the system, so energy increases once again and starts to oscillate between kinetic and potential.

In the phase portrait the trajectory starts at $x_{0}=5$ and $\dot{x}_{0}=0$, which is our initial condition. It is initially attracted to the center, because of the damping as in the previous case, but then we see that the trajectory does not converge towards $(0, 0)$, but goes back to a stable circle.

```python
x_0 =5
x_dot_0 =0
gamma = 0.1
w_0 = 0.8
alpha =1
phi =10
x,v,t=Oscillator_func(x_0,x_dot_0,gamma,w_0,alpha,phi)
K = (1/2)*(w_0)**2*x**2
U = (1/2)*v**2
plt.figure(figsize=(10,16))
plt.subplot(3,1,1)
plt.plot (t,x)
plt.plot(t,v)
plt.legend(['x(t)','v(t)'])
plt.subplot(3,1,2)
plt.plot(t,K)
plt.plot(t,U)
plt.legend(['Kinetic Energy','Potential Energy'])
plt.subplot(3,1,3)
plt.plot(x,v)
plt.show()
```

```python
{
 "cells": [],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}

```

    {'cells': [], 'metadata': {}, 'nbformat': 4, 'nbformat_minor': 2}

```python

```


```python
import sympy as sm
import numpy as np
import matplotlib.pyplot as pl
```

```python
def ryOptimization(
        f,
        c1,
        c2,
        xyBounds):

    x,y= sm.symbols('x,y')

    # sympy function --> numpy function
    z= sm.lambdify([x,y],f([x,y]))

    xx= np.linspace(-10,10,1001)
    yy= np.linspace(-10,10,1001)
    zz= z(xx,yy)

    xm, ym= np.meshgrid(xx,yy)
    zm= z(xm,ym)
    zz.shape, zm.shape

    y1= sm.solve(
            c1([x,y]),
            y)[0]
    y2= sm.solve(
            c2([x,y]),
            y)[0]
    y1= sm.lambdify(x, y1)
    y2= sm.lambdify(x, y2)
    yy1= y1(xx)
    yy2= y2(xx)
    [(xmin,xmax),(ymin,ymax)]= xyBounds

    y_ymin= ymin *np.ones_like(xx)
    y_ymax= ymax *np.ones_like(xx)
    x_xmin= xmin *np.ones_like(yy)
    x_xmax= xmax *np.ones_like(yy)

    y_0= np.zeros_like(xx)
    x_0= np.zeros_like(yy)

    ax= pl.axes(xlim=(-10,10),ylim=(-10,10))#, projection='3d')

    ax.contour(xm,ym,zm, 100, cmap='rainbow')

    ax.plot(xx,yy1,'r',
            xx,yy2,'g',
            linestyle='--'
        )

    ax.plot(xx, y_0,'w-',
            x_0,yy, 'w-',
            linewidth= 3,
            alpha= 0.3
        )

    ax.plot(xx,     y_ymin,
            xx,     y_ymax,
            x_xmin, yy,
            x_xmax, yy,
            linewidth= 1,
            linestyle= '--',
            color= 'gray'
        )

    for yyy in [yy1,yy2]:

        ax.fill_between(xx, y_ymax, yyy,
                        where= (yyy>=y_ymax),
                        alpha= 0.5,
                        color='white',
                        interpolate=True
                    )
        ax.fill_between(xx, y_ymin, yyy,
                        where= (yyy<=y_ymin),
                        alpha= 0.5,
                        color='white',
                        interpolate=True
                    )

    import scipy.optimize as sopt

    x0= [0, 0]

    opt= sopt.minimize(
        f,
        x0,
        #method= 'SLSQP',
        bounds= xyBounds,
        constraints=     Constraints
    )

    opt

    xopt= opt.x[0]
    yopt= opt.x[1]
    fopt= opt.fun

    ax.scatter(
        x= xopt,
        y= yopt,
        color= 'magenta',
        marker= 's'
    )

    ax.text(
        x= xopt,
        y= yopt,
        s= f'{xopt:.3f},{yopt:.3f}'
    )

    fLtx=  sm.latex(f([x,y]))
    c1Ltx= sm.latex(c1([x,y]))
    c2Ltx= sm.latex(c2([x,y]))

    titleStr= f'''
    Optimize:      f(x,y)= ${fLtx}$
    Subject to:    Constraints and Bounds
    opt= [x= {xopt:.3f}, y= {yopt:.3f}; f= {fopt:.3f}]
    '''

    ax.set_title(titleStr)
    ax.set_xlabel('x')
    ax.set_ylabel('y')

    infoStr= f'''
    $Objective: f(x,y)= {fLtx}$
    $c_1(x,y)= {c1Ltx} ≥ 0$
    $c_2(x,y)= {c2Ltx} ≥ 0$
    $x \in [{xmin},{xmax}]$
    $y \in [{ymin},{ymax}]$
    opt= [x= {xopt:.3f}, y= {yopt:.3f}; f= {fopt:.3f}]
    '''
    ax.text(
        x= -10,
        y= -10,
        s= infoStr
    )

    pl.show()
```

```python
def f(s):
    x, y= s
    z=  x + y

    return z
def c1(s):
    x,y= s
    z= x**2 + y**2 -1
    return z

def c2(s):
    x,y= s
    z= -(x**2 + y**2 -2)
    return z

Constraints= [
    {'fun':c1, 'type':'ineq'},
    {'fun':c2, 'type':'ineq'},
]

xyBounds= [(-5, 5),  # xmin, xmax
           (-5, 5)   # ymin, ymax
           ]
```

```python
ryOptimization(f,c1,c2, xyBounds)
```

```python

```


```python
%pylab inline
from sympy import symbols, Matrix
from sympy.parsing.sympy_parser import parse_expr
```

    Populating the interactive namespace from numpy and matplotlib

```python
expr = parse_expr("x*y")
expr
```

    x*y

```python
type(expr)
```

    sympy.core.mul.Mul

```python
type(parse_expr("Matrix([[1,2],[3,4]]) * t"))
```

    sympy.matrices.dense.MutableDenseMatrix

```python
type(parse_expr("cos(x)"))
```

    cos

```python
type(parse_expr("Eq(10,a)"))
```

    sympy.core.relational.Equality

```python
type(parse_expr("10>a"))
```

    sympy.core.relational.StrictGreaterThan

```python
type(parse_expr("10>=a"))
```

    sympy.core.relational.GreaterThan

```python
type(parse_expr("10<a"))
```

    sympy.core.relational.StrictLessThan

```python
type(parse_expr("10<=a"))
```

    sympy.core.relational.LessThan

```python
type(parse_expr("Eq(x^2 + x, x)"))
```

    sympy.core.relational.Equality

```python
type(parse_expr("10!"))
```

    sympy.core.numbers.Integer

```python
type(parse_expr("2*pi"))
```

    sympy.core.mul.Mul

```python
type(parse_expr("1/4"))
```

    sympy.core.numbers.Rational

```python
type(parse_expr("2.11"))
```

    sympy.core.numbers.Float

```python
def detect(expr):
    return {
        'is_constant':   'Integer' in str(type(expr))
                      or 'Float' in str(type(expr))
                      or 'Rational' in str(type(expr)),
        'is_equality':   'Equality' in str(type(expr)),
        'is_inequality': 'Inequality' in str(type(expr))
                      or 'Greater' in str(type(expr))
                      or 'Less' in str(type(expr)),
        'is_matrix':     'Matrix' in str(type(expr)),
        'dimension':     list(expr.shape) if 'Matrix' in str(type(expr)) else [0, 0]
    }
```

```python
detect(parse_expr('10!'))
```

    {'is_constant': True,
     'is_equality': False,
     'is_inequality': False,
     'is_matrix': False,
     'dimension': [0, 0]}

```python
L = ['Matrix([[1,2],[3,4]]) * t', '10>a', '10<a', '2.11', 'Eq(x^2 + x, x)']
for e in L:
    print(e)
    print(detect(parse_expr(e)))
    print()
```

    Matrix([[1,2],[3,4]]) * t
    {'is_constant': False, 'is_equality': False, 'is_inequality': False, 'is_matrix': True, 'dimension': [2, 2]}

    10>a
    {'is_constant': False, 'is_equality': False, 'is_inequality': True, 'is_matrix': False, 'dimension': [0, 0]}

    10<a
    {'is_constant': False, 'is_equality': False, 'is_inequality': True, 'is_matrix': False, 'dimension': [0, 0]}

    2.11
    {'is_constant': True, 'is_equality': False, 'is_inequality': False, 'is_matrix': False, 'dimension': [0, 0]}

    Eq(x^2 + x, x)
    {'is_constant': False, 'is_equality': True, 'is_inequality': False, 'is_matrix': False, 'dimension': [0, 0]}

```python
M = Matrix([[1,2],[3,4]])
N = Matrix([1,2,3,4])
O = Matrix([[1],[2],[3]])
P = Matrix([[1,2],[3,4],[5,6]])
M, N, O, P
```

    (Matrix([
     [1, 2],
     [3, 4]]), Matrix([
     [1],
     [2],
     [3],
     [4]]), Matrix([
     [1],
     [2],
     [3]]), Matrix([
     [1, 2],
     [3, 4],
     [5, 6]]))

```python
M.det() # requires a square matrix
```

    -2

```python
M.inv() # requires a square matrix
```

    Matrix([
    [ -2,    1],
    [3/2, -1/2]])

```python
M.QRdecomposition(), N.QRdecomposition(), O.QRdecomposition(), P.QRdecomposition()
```

    ((Matrix([
      [  sqrt(10)/10, 3*sqrt(10)/10],
      [3*sqrt(10)/10,  -sqrt(10)/10]]), Matrix([
      [sqrt(10), 7*sqrt(10)/5],
      [       0,   sqrt(10)/5]])), (Matrix([
      [  sqrt(30)/30],
      [  sqrt(30)/15],
      [  sqrt(30)/10],
      [2*sqrt(30)/15]]), Matrix([[sqrt(30)]])), (Matrix([
      [  sqrt(14)/14],
      [   sqrt(14)/7],
      [3*sqrt(14)/14]]), Matrix([[sqrt(14)]])), (Matrix([
      [  sqrt(35)/35, 13*sqrt(210)/210],
      [3*sqrt(35)/35,  2*sqrt(210)/105],
      [   sqrt(35)/7,    -sqrt(210)/42]]), Matrix([
      [sqrt(35), 44*sqrt(35)/35],
      [       0, 2*sqrt(210)/35]])))

```python
Q, R = M.QRdecomposition()
Q * R
```

    Matrix([
    [1, 2],
    [3, 4]])

```python
Q.T, P.T
```

    (Matrix([
     [  sqrt(10)/10, 3*sqrt(10)/10],
     [3*sqrt(10)/10,  -sqrt(10)/10]]), Matrix([
     [1, 3, 5],
     [2, 4, 6]]))

```python
Q.eigenvals() # requires square matrix
```

    {-1: 1, 1: 1}

```python
Q.eigenvects() # requires square matrix
```

    [(-1, 1, [Matrix([
       [-3*sqrt(10)/(10*(sqrt(10)/10 + 1))],
       [                                 1]])]), (1, 1, [Matrix([
       [-3*sqrt(10)/(10*(-1 + sqrt(10)/10))],
       [                                  1]])])]

```python
def splat(*s): return [a for a in s]
[v[2][0][0] for v in Q.eigenvects()]
```

    [-3*sqrt(10)/(10*(sqrt(10)/10 + 1)), -3*sqrt(10)/(10*(-1 + sqrt(10)/10))]

```python

```


# Derivatives with Gaussian Processes

```python
%matplotlib inline

import numpy as np
from scipy import stats
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd

import gptools  # Install from GitHub repo directly (not pip), if using python 3

fontsize = 9
mpl.rcParams['font.size'] = fontsize
mpl.rcParams['axes.labelsize'] = fontsize
mpl.rcParams['axes.titlesize'] = fontsize
mpl.rcParams['xtick.labelsize'] = fontsize
mpl.rcParams['ytick.labelsize'] = fontsize
mpl.rcParams['legend.title_fontsize'] = fontsize
mpl.rcParams['legend.fontsize'] = fontsize
mpl.rcParams['figure.titlesize'] = fontsize
mpl.rcParams['text.usetex'] = True
mpl.rcParams['figure.dpi'] = 130
mpl.rcParams['font.family'] = 'serif'
mpl.rcParams['ytick.direction'] = 'in'
mpl.rcParams['xtick.direction'] = 'in'
mpl.rc('savefig', transparent=False, bbox='tight', pad_inches=0.05, format='pdf')
```

    WARNING:root:Deprecation Warning: 'triangle' has been renamed to 'corner'. This shim should continue to work but you should use 'import corner' in new code. https://github.com/dfm/corner.py

## Standard Case

We have data and would like to find a function that best captures the relationship between the inputs and outputs.
A popular nonparametric tool for regression is a Gaussian process (GP).
One of the neat things about a GP is that you automatically get uncertainty quantification along with the line of best fit.
But what is less well known is that it also provides derivatives of the best fit line, and corresponding uncertainty estimates, for free!

### Fitting a function f(x)

Let's check out how it works. Start by creating some data with noise.

```python
color_68 = 'darkgrey'   # color for 1 sigma bands
color_95 = 'lightgrey'  # color for 2 sigma bands
```

```python
def f(x): # the function
    return x**2
```

```python
x = np.linspace(0, 1, 20)
x_star = np.linspace(0, 1, 100)

np.random.seed(1)

std = 0.1
y = f(x) + stats.norm(scale=std).rvs(len(x))
```

Now plot the true line and the data we will use to fit our curve.

```python
plt.plot(x, f(x), c='k', ls='--', label='true')
plt.errorbar(x, y, std, ls='', marker='o', label='data')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.title("The data")
plt.legend();
```

To use a GP, we need to convert the 1d `x` input into an `NxD` matrix (since GPs can live in a `D` dimensional space, in general).
This is a `D=1` dimensional space in this example.

```python
X = x[:, None]
X_star = x_star[:, None]
```

Begin by creating the kernel, which quantifies the correlation structure (smoothness properties) of the curves.
Then feed the kernel into a GP object, and fit it to the data we plotted above.
The radial basis function kernel (i.e., the Squared Exponential) has two hyperparameters that it depends on: the marginal variance and length scale.
Let's let the data decide which hyperparameters provide the best fit, via optimization.

```python
# rbf_kernel = gptools.SquaredExponentialKernel(initial_params=[1,1], fixed_params=[True, True])
rbf_kernel = gptools.SquaredExponentialKernel(initial_params=[2,2], param_bounds=[(1e-5, 2), (1e-2, 2)])
gp = gptools.GaussianProcess(rbf_kernel)
gp.add_data(X, y, err_y=std)
gp.optimize_hyperparameters(max_tries=3)
```

    (     fun: 32.11743012805012
          jac: array([ 2.67028809e-05, -2.86102295e-06])
      message: 'Optimization terminated successfully.'
         nfev: 40
          nit: 9
         njev: 9
       status: 0
      success: True
            x: array([0.97963566, 0.85580235]),
     8)

That's it! Now we can get the curve of best fit and its uncertainty.

```python
# plot the data
plt.plot(x, f(x), c='k', ls='--', label='true')
plt.errorbar(x, y, std, ls='', marker='o', label='data')

# plot GP
y_star, std_star = gp.predict(X_star, return_std=True)
plt.plot(x_star, y_star, c='r', label='interpolated')
plt.fill_between(x_star, y_star+std_star, y_star-std_star, color='lightgrey')

# plot the function
plt.plot(x, f(x), c='k', ls='--')

# labels
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.title("The function $f(x)$")
plt.legend();
```

### Estimating the derivative f'(x)

Since our function is simple, we can take its derivative analytically and compare to our approximation.

```python
stepSize = 1e-3
def fprime(x, n=1):
    if n == 0 : return f(x)
    #elif n == 1 : return 2*x # insert here analytic derivatives...
    #elif n == 2 : return 2*np.ones(len(x))
    #elif n > 2 : return np.zeros(len(x))
    #else : return None;
    return (fprime(x+stepSize,n-1)-fprime(x-stepSize,n-1))/(2.*stepSize) #... or simply compute numerically
```

`GPTools` allows for predicting derivatives very easily! Just use `n=1` in the `predict` method to compute the first derivative.

```python
derivOrder = 1 # order of the derivative
y_prime_star, std_prime_star = gp.predict(X_star, n=derivOrder, return_std=True)
```

```python
plt.plot(x_star, y_prime_star, label="prediction")
plt.fill_between(x_star, y_prime_star+std_prime_star, y_prime_star-std_prime_star, color='lightgrey')
plt.plot(x, fprime(x,derivOrder), c='k', ls='--', label="true")

plt.title("The derivative")
plt.xlabel('$x$');
plt.ylabel(f"$f(x)$'s derivative of order {derivOrder}")
plt.legend();
```

# Nuclear Matter Application

Our specific use case is similar to the example above: we fit a GP to data, in this case from a physics simulation.
But there is one additional source of uncertainty from the theory error. This will also involve us creating our own custom GP kernel!

Start again by getting some data.

```python
df = pd.read_csv('../data/all_matter_data.csv')
# Convert differences to total prediction at each MBPT order
mbpt_orders = ['Kin', 'MBPT_HF', 'MBPT_2', 'MBPT_3', 'MBPT_4']
df[mbpt_orders] = df[mbpt_orders].apply(np.cumsum, axis=1)
# 'total' is now unnecessary. Remove it.
df.pop('total');
```

```python
orders = np.array([0, 2, 3, 4])
# body = 'NN-only'
body = 'NN+3N'
Lambda = 450
fits = {450: [1, 7], 500: [4, 10]}
train1 = slice(None, None, 5)
valid1 = slice(2, None, 5)
# valid1 = np.array([i % 5 != 0 for i in range(len())])
[fit_n2lo, fit_n3lo] = fits[Lambda]

excluded = np.array([1])

savefigs = False

mask_fit = np.isin(df['fit'], fits[Lambda]) | np.isnan(df['fit'])

mask1 = \
    (df['Body'] == body) & \
    mask_fit & \
    (df['Lambda'] == Lambda)

# df_fit = df[mask_fit]
df_n = df[mask1 & (df['x'] == 0)]
df_s = df[mask1 & (df['x'] == 0.5)]

kf_n = df_n[df_n['OrderEFT'] == 'LO']['kf'].values
kf_s = df_s[df_s['OrderEFT'] == 'LO']['kf'].values
density = df_n[df_n['OrderEFT'] == 'LO']['n'].values
kf_d = kf_n.copy()

# valid1 = np.arange(len(kf_n)) % 5 != 0

Kf_n = kf_n[:, None]
Kf_s = kf_s[:, None]
Kf_d = kf_d[:, None]

y_n = np.array([df_n[df_n['OrderEFT'] == order]['MBPT_4'].values for order in df_n['OrderEFT'].unique()]).T
y_s = np.array([df_s[df_s['OrderEFT'] == order]['MBPT_4'].values for order in df_s['OrderEFT'].unique()]).T
y_d = y_n - y_s
```

Visualize it.

```python
for i, n in enumerate(orders):
    plt.plot(kf_s, y_s[:, i], label=fr'$y_{n}$')
plt.legend()
plt.xlabel(r'$k_f$ [fm$^{-1}$]')
plt.ylabel(r'$E/A$');
```

The kernel for our GP convergence model is not a simple RBF, it must be multiplied by factors related to the converges of the EFT:

\begin{align}
    \kappa(x, x';\bar c, \ell) = y_{\text{ref}}(x)y_{\text{ref}}(x') \frac{[Q(x)Q(x')]^{k+1}}{1-Q(x)Q(x')} \bar c^2 r(x,x';\ell)
\end{align}

where $\bar c^2 r(x,x';\ell)$ is the RBF kernel used above.
GPTools can handle products of kernels, so we just need to create a kernel object that represents the prefactor above.
I'm still not completely sure how to make a truly compatible kernel object, but the code below seems to do the trick.

```python
class CustomKernel(gptools.Kernel):
    """A Custom GPTools kernel that wraps an arbitrary function f with a compatible signature

    Parameters
    ----------
    f : callable
        A positive semidefinite kernel function that takes f(Xi, Xj, ni, nj) where ni and nj are
        integers for the number of derivatives to take with respect to Xi or Xj. It should return
        an array of Xi.shape[0]
    *args
        Args passed to the Kernel class
    **kwargs
        Kwargs passed to the Kernel class
    """

    def __init__(self, f, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.f = f

    def __call__(self, Xi, Xj, ni, nj, hyper_deriv=None, symmetric=False):
        return self.f(Xi, Xj, int(np.unique(ni)[0]), int(np.unique(nj)[0]))
```

We need to take arbitrary numbers of derivatives. Humans can make mistakes, let `SymPy` handle it.

```python
from sympy import symbols, diff, sqrt, lambdify

def kernel_scale_sympy(lowest_order=4, highest_order=None):
    """Creates a sympy object that is the convergence part of the GP kernel

    Parameters
    ----------
    lowest_order
    highest_order
    """
    k_f1, k_f2, y_ref, Lambda_b, = symbols('k_f1 k_f2 y_ref Lambda_b')
    hbar_c = 200
    Q1 = hbar_c * k_f1 / Lambda_b
    Q2 = hbar_c * k_f2 / Lambda_b
    num = (Q1 * Q2)**(lowest_order)
    if highest_order is not None:
        num = num - (Q1 * Q2)**highest_order
    kernel_scale = y_ref**2 * num / (1 - Q1*Q2)
    return k_f1, k_f2, Lambda_b, y_ref, kernel_scale

def eval_kernel_scale(Xi, Xj=None, ni=None, nj=None, breakdown=600, ref=16, lowest_order=4, highest_order=None):
    """Creates a matrix for the convergence part of the GP kernel.
    Compatible with the CustomKernel class signature.

    Parameters
    -----------
    Xi
    Xj
    ni
    nj
    breakdown
    ref
    lowest_order
    highest_order
    """
    if ni is None:
        ni = 0
    if nj is None:
        nj = 0
    k_f1, k_f2, Lambda_b, y_ref, kernel_scale = kernel_scale_sympy(
        lowest_order=lowest_order, highest_order=highest_order
    )
    expr = diff(kernel_scale, k_f1, ni, k_f2, nj)
    f = lambdify((k_f1, k_f2, Lambda_b, y_ref), expr, "numpy")
    if Xj is None:
        Xj = Xi
    K = f(Xi, Xj, breakdown, ref)
    K = K.astype('float')
    return np.squeeze(K)
```

Finally, we can create the kernel objects for our GP interpolation and derivatives. Assume some values for the hyperparameters, these would come from our convergence analysis of the observable coefficients.

```python
from functools import partial

k_max = 4
eval_kernel_lower = partial(eval_kernel_scale, lowest_order=0, highest_order=k_max)
eval_kernel_upper = partial(eval_kernel_scale, lowest_order=k_max+1)

matter_rbf_kernel = gptools.SquaredExponentialKernel(
    initial_params=[1,1], fixed_params=[True, True])

kernel_lower = CustomKernel(eval_kernel_lower) * matter_rbf_kernel
kernel_upper = CustomKernel(eval_kernel_upper) * matter_rbf_kernel
```

```python
kf_s_all = np.linspace(kf_s[0], kf_s[-1], 50)
Kf_s_all = kf_s_all[:, None]
```

Begin by interpolating the data points we plotted above. This does not include truncation error yet. Also compute its derivative as before.

```python
gp_lower = gptools.GaussianProcess(kernel_lower)
gp_lower.add_data(Kf_s[:], y_s[:,-1], err_y=1e-5)

y_s_star, std_s_star = gp_lower.predict(Kf_s_all, return_std=True)
y_s_star_prime, std_s_star_prime = gp_lower.predict(Kf_s_all, n=1, return_std=True)
```

But now we can create the additional source of uncertainty due to EFT truncation.
Create both the error term for the interpolant and for its derivative.

```python
zero_d = np.zeros(Kf_s_all.shape, dtype=int)
one_d = np.ones(Kf_s_all.shape, dtype=int)
std_upper_star = np.sqrt(kernel_upper(Kf_s_all, Kf_s_all, ni=zero_d, nj=zero_d))
std_upper_star_prime = np.sqrt(kernel_upper(Kf_s_all, Kf_s_all, ni=one_d, nj=one_d))

std_total_star = std_s_star + std_upper_star
std_total_star_prime = std_s_star_prime + std_upper_star_prime
```

Let's see how we did. Error bands represent 2 standard deviations.
The left plot shows the interpolant alone, with error bands that are so small they can't be seen.
The right plot is the same as the left, except truncation bands are added.

```python
fig, axes = plt.subplots(1, 2, figsize=(3.5, 2.7), sharey=True, sharex=True)
ax1, ax2 = axes
ax1.plot(kf_s, y_s[:, -1], ls='', marker='.', c='k');
ax1.plot(kf_s_all, y_s_star, lw=0.8)
ax1.fill_between(kf_s_all, y_s_star+2*std_s_star, y_s_star-2*std_s_star, color=color_95)
ax1.fill_between(kf_s_all, y_s_star+std_s_star, y_s_star-std_s_star, color=color_68)
ax1.set_title('Interpolant')
ax1.set_xlabel(r'$k_f$ [fm$^{-1}$]')

ax2.plot(kf_s, y_s[:, -1], ls='', marker='.', c='k');
ax2.plot(kf_s_all, y_s_star, lw=0.8)
ax2.fill_between(kf_s_all, y_s_star+2*std_total_star, y_s_star-2*std_total_star, color=color_95)
ax2.fill_between(kf_s_all, y_s_star+std_total_star, y_s_star-std_total_star, color=color_68)
ax2.set_title('+ Truncation')
ax2.set_xlabel(r'$k_f$ [fm$^{-1}$]')
fig.tight_layout(w_pad=0.25)

fig.savefig('saturation_with_error_bands')
```

Plot the derivatives with respect to $k_{f}$ similarly

```python
fig, axes = plt.subplots(1, 2, figsize=(3.5, 2.7), sharey=True, sharex=True)
ax1, ax2 = axes

ax1.plot(kf_s_all, y_s_star_prime, lw=0.8)
ax1.fill_between(kf_s_all, y_s_star_prime+2*std_s_star_prime, y_s_star_prime-2*std_s_star_prime, color=color_95)
ax1.fill_between(kf_s_all, y_s_star_prime+std_s_star_prime, y_s_star_prime-std_s_star_prime, color=color_68)
ax1.axhline(0, 0, 1, c='lightgrey', zorder=0)
ax1.set_title('Interpolant')
ax1.set_xlabel(r'$k_f$ [fm$^{-1}$]')

ax2.plot(kf_s_all, y_s_star_prime, lw=0.8)
ax2.fill_between(kf_s_all, y_s_star_prime+2*std_total_star_prime, y_s_star_prime-2*std_total_star_prime, color=color_95)
ax2.fill_between(kf_s_all, y_s_star_prime+std_total_star_prime, y_s_star_prime-std_total_star_prime, color=color_68)
ax2.axhline(0, 0, 1, c='lightgrey', zorder=0)
ax2.set_title('+ Truncation')
ax2.set_xlabel(r'$k_f$ [fm$^{-1}$]')

fig.suptitle('Derivative of $E/A$ wrt $k_{f}$', y=1.05)
fig.tight_layout(w_pad=0.25)
fig.savefig('pressure_with_error_bands')
```

Nice! This is, up to some easily handled scaling factors, the pressure! With truncation errors!

```python

```


# Clustering

Wikipedia: Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). Clustering is one of the main task of exploratory data mining, and a common technique for statistical data analysis, used in many fields, including machine learning, pattern recognition, image analysis, information retrieval, and bioinformatics.

Sources: http://scikit-learn.org/stable/modules/clustering.html

## K-means clustering

Source: C. M. Bishop *Pattern Recognition and Machine Learning*, Springer, 2006

Suppose we have a data set $X = \{x_1 , \cdots , x_N\}$ that consists of $N$ observations of a random $D$-dimensional Euclidean variable $x$. Our goal is to partition the data set into some number, $K$, of clusters, where we shall suppose for the moment that the value of $K$ is given. Intuitively, we might think of a cluster as comprising a group of data points whose inter-point distances are small compared to the distances to points outside of the cluster. We can formalize this notion by first introducing a set of $D$-dimensional vectors $\mu_k$, where $k = 1, \ldots, K$, in which $\mu_k$ is a **prototype** associated with the $k^{th}$ cluster. As we shall see shortly, we can think of the $\mu_k$ as representing the centres of the clusters. Our goal is then to find an assignment of data points to clusters, as well as a set of vectors $\{\mu_k\}$, such that the sum of the squares of the distances of each data point to its closest prototype vector $\mu_k$, is at a minimum.

It is convenient at this point to define some notation to describe the assignment of data points to clusters. For each data point $x_i$ , we introduce a corresponding set of binary indicator variables $r_{ik} \in \{0, 1\}$, where $k = 1, \ldots, K$, that describes which of the $K$ clusters the data point $x_i$ is assigned to, so that if data point $x_i$ is assigned to cluster $k$ then $r_{ik} = 1$, and $r_{ij} = 0$ for $j \neq k$. This is known as the 1-of-$K$ coding scheme. We can then define an objective function, denoted **inertia**, as

$$
J(r, \mu) = \sum_i^N \sum_k^K r_{ik} \|x_i - \mu_k\|_2^2
$$

which represents the sum of the squares of the Euclidean distances of each data point to its assigned vector $\mu_k$. Our goal is to find values for the $\{r_{ik}\}$ and the $\{\mu_k\}$ so as to minimize the function $J$. We can do this through an iterative procedure in which each iteration involves two successive steps corresponding to successive optimizations with respect to the $r_{ik}$ and the $\mu_k$ . First we choose some initial values for the $\mu_k$. Then in the first phase we minimize $J$ with respect to the $r_{ik}$, keeping the $\mu_k$ fixed. In the second phase we minimize $J$ with respect to the $\mu_k$, keeping $r_{ik}$ fixed. This two-stage optimization process is then repeated until convergence. We shall see that these two stages of updating $r_{ik}$ and $\mu_k$ correspond respectively to the expectation (E) and maximization (M) steps of the expectation-maximisation (EM) algorithm, and to emphasize this we shall use the terms E step and M step in the context of the $K$-means algorithm.

Consider first the determination of the $r_{ik}$ . Because $J$ in is a linear function of $r_{ik}$ , this optimization can be performed easily to give a closed form solution. The terms involving different $i$ are independent and so we can optimize for each $i$ separately by choosing $r_{ik}$ to be 1 for whichever value of $k$ gives the minimum value of $||x_i - \mu_k||^2$ . In other words, we simply assign the $i$th data point to the closest cluster centre. More formally, this can be expressed as

\begin{equation}
  r_{ik}=\begin{cases}
    1, & \text{if } k = \arg\min_j ||x_i - \mu_j||^2.\\
    0, & \text{otherwise}.
  \end{cases}
\end{equation}

Now consider the optimization of the $\mu_k$ with the $r_{ik}$ held fixed. The objective function $J$ is a quadratic function of $\mu_k$, and it can be minimized by setting its derivative with respect to $\mu_k$ to zero giving

$$
2 \sum_i r_{ik}(x_i - \mu_k) = 0
$$

which we can easily solve for $\mu_k$ to give

$$
\mu_k = \frac{\sum_i r_{ik}x_i}{\sum_i r_{ik}}.
$$

The denominator in this expression is equal to the number of points assigned to cluster $k$, and so this result has a simple interpretation, namely set $\mu_k$ equal to the mean of all of the data points $x_i$ assigned to cluster $k$. For this reason, the procedure is known as the $K$-means algorithm.

The two phases of re-assigning data points to clusters and re-computing the cluster means are repeated in turn until there is no further change in the assignments (or until some maximum number of iterations is exceeded). Because each phase reduces the value of the objective function $J$, convergence of the algorithm is assured. However, it may converge to a local rather than global minimum of $J$.

```python
from sklearn import cluster, datasets
import matplotlib.pyplot as plt
import seaborn as sns  # nice color
%matplotlib inline

iris = datasets.load_iris()
X = iris.data[:, :2]  # use only 'sepal length and sepal width'
y_iris = iris.target

km2 = cluster.KMeans(n_clusters=2).fit(X)
km3 = cluster.KMeans(n_clusters=3).fit(X)
km4 = cluster.KMeans(n_clusters=4).fit(X)

plt.figure(figsize=(9, 3))
plt.subplot(131)
plt.scatter(X[:, 0], X[:, 1], c=km2.labels_)
plt.title("K=2, J=%.2f" % km2.inertia_)

plt.subplot(132)
plt.scatter(X[:, 0], X[:, 1], c=km3.labels_)
plt.title("K=3, J=%.2f" % km3.inertia_)

plt.subplot(133)
plt.scatter(X[:, 0], X[:, 1], c=km4.labels_)#.astype(np.float))
plt.title("K=4, J=%.2f" % km4.inertia_)
```

### Exercises

#### 1. Analyse clusters

- Analyse the plot above visually. What would a good value of $K$ be?

- If you instead consider the inertia, the value of $J$, what would a good value of $K$ be?

- Explain why there is such difference.

- For $K=2$ why did $K$-means clustering not find the two "natural" clusters? See the assumptions of $K$-means:
http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_assumptions.html#example-cluster-plot-kmeans-assumptions-py

#### 2. Re-implement the $K$-means clustering algorithm (homework)

Write a function `kmeans(X, K)` that return an integer vector of the samples' labels.

## Gaussian mixture models

The Gaussian mixture model (GMM) is a simple linear superposition of Gaussian components over the data, aimed at providing a rich class of density models. We turn to a formulation of Gaussian mixtures in terms of discrete latent variables: the $K$ hidden classes to be discovered.

Differences compared to $K$-means:

- Whereas the $K$-means algorithm performs a hard assignment of data points to clusters, in which each data point is associated uniquely with one cluster, the GMM algorithm makes a soft assignment based on posterior probabilities.

- Whereas the classic $K$-means is only based on Euclidean distances, classic GMM use a Mahalanobis distances that can deal with non-spherical distributions. It should be noted that Mahalanobis could be plugged within an improved version of $K$-Means clustering. The Mahalanobis distance is unitless and scale-invariant, and takes into account the correlations of the data set.

The Gaussian mixture distribution can be written as a linear superposition of $K$ Gaussians in the form:

$$
p(x) = \sum_{k=1}^K \mathcal{N}(x \,|\, \mu_k, \Sigma_k)p(k),
$$

where:

- The $p(k)$ are the mixing coefficients also know as the class probability of class $k$, and they sum to one: $\sum_{k=1}^K p(k) = 1$.

- $\mathcal{N}(x \,|\, \mu_k, \Sigma_k) = p(x \,|\, k)$ is the conditional distribution of $x$ given a particular class $k$. It is the multivariate Gaussian distribution defined over a $P$-dimensional vector $x$ of continuous variables.

The goal is to maximize the log-likelihood of the GMM:

$$
\ln \prod_{i=1}^N p(x_i)= \ln \prod_{i=1}^N \left\{ \sum_{k=1}^K \mathcal{N}(x_i \,|\, \mu_k, \Sigma_k)p(k) \right\} = \sum_{i=1}^N \ln\left\{ \sum_{k=1}^K  \mathcal{N}(x_i \,|\, \mu_k, \Sigma_k) p(k) \right\}.
$$

To compute the classes parameters: $p(k), \mu_k, \Sigma_k$ we sum over all samples, by weighting each sample $i$ by its responsibility or contribution to class $k$: $p(k \,|\, x_i)$ such that for each point its contribution to all classes sum to one $\sum_k p(k \,|\, x_i) = 1$. This contribution is the conditional probability
of class $k$ given $x$: $p(k \,|\, x)$ (sometimes called the posterior). It can be computed using Bayes' rule:

\begin{align}
p(k \,|\, x) &= \frac{p(x \,|\, k)p(k)}{p(x)}\\
                 &= \frac{\mathcal{N}(x \,|\, \mu_k, \Sigma_k)p(k)}{\sum_{k=1}^K \mathcal{N}(x \,|\, \mu_k, \Sigma_k)p(k)}
\end{align}

Since the class parameters, $p(k)$, $\mu_k$ and $\Sigma_k$, depend on the responsibilities $p(k \,|\, x)$ and the responsibilities depend on class parameters, we need a two-step iterative algorithm: the expectation-maximization (EM) algorithm. We discuss this algorithm next.

### The expectation-maximization (EM) algorithm for Gaussian mixtures

Given a Gaussian mixture model, the goal is to maximize the likelihood function with respect to the parameters (comprised of the means and covariances of the components and the mixing coefficients).

Initialize the means $\mu_k$, covariances $\Sigma_k$ and mixing coefficients $p(k)$

1. **E step**. For each sample $i$, evaluate the responsibilities for each class $k$ using the current parameter values

$$
p(k \,|\, x_i) = \frac{\mathcal{N}(x_i \,|\, \mu_k, \Sigma_k)p(k)}{\sum_{k=1}^K \mathcal{N}(x_i \,|\, \mu_k, \Sigma_k)p(k)}
$$

2. **M step**. For each class, re-estimate the parameters using the current responsibilities

\begin{align}
\mu_k^{\text{new}}    &= \frac{1}{N_k} \sum_{i=1}^N p(k \,|\, x_i) x_i\\
\Sigma_k^{\text{new}} &= \frac{1}{N_k} \sum_{i=1}^N p(k \,|\, x_i) (x_i - \mu_k^{\text{new}}) (x_i - \mu_k^{\text{new}})^T\\
p^{\text{new}}(k)     &= \frac{N_k}{N}
\end{align}

3. Evaluate the log-likelihood

$$
    \sum_{i=1}^N \ln \left\{ \sum_{k=1}^K \mathcal{N}(x|\mu_k, \Sigma_k) p(k) \right\},
$$

and check for convergence of either the parameters or the log-likelihood. If the convergence criterion is not satisfied return to step 1.

```python
import numpy as np
from sklearn import datasets
import matplotlib.pyplot as plt
import seaborn as sns  # nice color
import sklearn
from sklearn.mixture import GaussianMixture

import pystatsml.plot_utils

colors = sns.color_palette()

iris = datasets.load_iris()
X = iris.data[:, :2]  # 'sepal length (cm)''sepal width (cm)'
y_iris = iris.target

gmm2 = GaussianMixture(n_components=2, covariance_type='full').fit(X)
gmm3 = GaussianMixture(n_components=3, covariance_type='full').fit(X)
gmm4 = GaussianMixture(n_components=4, covariance_type='full').fit(X)

plt.figure(figsize=(9, 3))
plt.subplot(131)
plt.scatter(X[:, 0], X[:, 1], c=[colors[lab] for lab in gmm2.predict(X)])#, color=colors)
for i in range(gmm2.covariances_.shape[0]):
    pystatsml.plot_utils.plot_cov_ellipse(cov=gmm2.covariances_[i, :], pos=gmm2.means_[i, :],
                     facecolor='none', linewidth=2, edgecolor=colors[i])
    plt.scatter(gmm2.means_[i, 0], gmm2.means_[i, 1], edgecolor=colors[i],
                marker="o", s=100, facecolor="w", linewidth=2)
plt.title("K=2")

plt.subplot(132)
plt.scatter(X[:, 0], X[:, 1], c=[colors[lab] for lab in gmm3.predict(X)])
for i in range(gmm3.covariances_.shape[0]):
    pystatsml.plot_utils.plot_cov_ellipse(cov=gmm3.covariances_[i, :], pos=gmm3.means_[i, :],
                     facecolor='none', linewidth=2, edgecolor=colors[i])
    plt.scatter(gmm3.means_[i, 0], gmm3.means_[i, 1], edgecolor=colors[i],
                marker="o", s=100, facecolor="w", linewidth=2)
plt.title("K=3")

plt.subplot(133)
plt.scatter(X[:, 0], X[:, 1], c=[colors[lab] for lab in gmm4.predict(X)])  # .astype(np.float))
for i in range(gmm4.covariances_.shape[0]):
    pystatsml.plot_utils.plot_cov_ellipse(cov=gmm4.covariances_[i, :], pos=gmm4.means_[i, :],
                     facecolor='none', linewidth=2, edgecolor=colors[i])
    plt.scatter(gmm4.means_[i, 0], gmm4.means_[i, 1], edgecolor=colors[i],
                marker="o", s=100, facecolor="w", linewidth=2)
_ = plt.title("K=4")
```

## Model selection

### Bayesian information criterion

In statistics, the Bayesian information criterion (BIC) is a criterion for model selection among a finite set of models; the model with the lowest BIC is preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).

```python
X = iris.data
y_iris = iris.target

bic = list()
#print(X)

ks = np.arange(1, 10)

for k in ks:
    gmm = GaussianMixture(n_components=k, covariance_type='full')
    gmm.fit(X)
    bic.append(gmm.bic(X))

k_chosen = ks[np.argmin(bic)]

plt.plot(ks, bic)
plt.xlabel("k")
plt.ylabel("BIC")

print("Choose k=", k_chosen)
```

## Hierarchical clustering

Hierarchical clustering is an approach to clustering that build hierarchies of clusters in two main approaches:

- **Agglomerative**: A *bottom-up* strategy, where each observation starts in their own cluster, and pairs of clusters are merged upwards in the hierarchy.

- **Divisive**: A *top-down* strategy, where all observations start out in the same cluster, and then the clusters are split recursively downwards in the hierarchy.

In order to decide which clusters to merge or to split, a measure of dissimilarity between clusters is introduced. More specific, this comprise a *distance* measure and a *linkage* criterion. The distance measure is just what it sounds like, and the linkage criterion is essentially a function of the distances between points, for instance the minimum distance between points in two clusters, the maximum distance between points in two clusters, the average distance between points in two clusters, etc. One particular linkage criterion, the Ward criterion, will be discussed next.

### Ward clustering

Ward clustering belongs to the family of agglomerative hierarchical clustering algorithms. This means that they are based on a "bottoms up" approach: each sample starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy.

In Ward clustering, the criterion for choosing the pair of clusters to merge at each step is the minimum variance criterion. Ward's minimum variance criterion minimizes the total within-cluster variance by each merge. To implement this method, at each step: find the pair of clusters that leads to minimum increase in total within-cluster variance after merging. This increase is a weighted squared distance between cluster centers.

The main advantage of agglomerative hierarchical clustering over $K$-means clustering is that you can benefit from known neighborhood information, for example, neighboring pixels in an image.

```python
from sklearn import cluster, datasets
import matplotlib.pyplot as plt
import seaborn as sns  # nice color

iris = datasets.load_iris()
X = iris.data[:, :2]  # 'sepal length (cm)''sepal width (cm)'
y_iris = iris.target

ward2 = cluster.AgglomerativeClustering(n_clusters=2, linkage='ward').fit(X)
ward3 = cluster.AgglomerativeClustering(n_clusters=3, linkage='ward').fit(X)
ward4 = cluster.AgglomerativeClustering(n_clusters=4, linkage='ward').fit(X)

plt.figure(figsize=(9, 3))
plt.subplot(131)
plt.scatter(X[:, 0], X[:, 1], c=ward2.labels_)
plt.title("K=2")

plt.subplot(132)
plt.scatter(X[:, 0], X[:, 1], c=ward3.labels_)
plt.title("K=3")

plt.subplot(133)
plt.scatter(X[:, 0], X[:, 1], c=ward4.labels_)  # .astype(np.float))
plt.title("K=4")
```

## Exercises

Perform clustering of the iris dataset based on all variables using Gaussian mixture models.
Use PCA to visualize clusters.


$\Large{\text{Kernel machines}}$

Let us first generate a synthetic data set to illustrate the behavior of different kernels.

```python
from sklearn.datasets import make_moons
import numpy as np
X,y = make_moons(n_samples=200, shuffle=True, noise=0.1, random_state=42)
print ('first five target values before changing 0 to -1')
# print (sum(y))
print (y[0:5])
y = np.where(y >=1,y,-1)
print('moon data shape:', np.shape(X))

#check the shape of moon target labels
print('moon target shape:', np.shape(y))
#We can print first 5 samples of moon data and check
print('Features of first five samples of moon data:')
print(X[0:5,])
print ('first five target values after changing 0 to -1')
print (y[0:5])
```

    first five target values before changing 0 to -1
    [0 0 0 1 1]
    moon data shape: (200, 2)
    moon target shape: (200,)
    Features of first five samples of moon data:
    [[-1.04942573  0.08444263]
     [ 0.92281755  0.45748851]
     [ 0.65678659  0.69959669]
     [ 1.1889402  -0.38652807]
     [ 0.28926455 -0.13774489]]
    first five target values after changing 0 to -1
    [-1 -1 -1  1  1]

```python
import pandas as pd
import matplotlib.pyplot as plt

# scatter plot, dots colored by class value

df = pd.DataFrame(dict(x=X[:,0], y=X[:,1], label=y))
colors = {-1:'red', 1:'blue'}
fig, ax = plt.subplots()
grouped = df.groupby('label')
for key, group in grouped:
   group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
plt.show()
```

$\large{\text{Kernel trick}}$

Fortunately, finding a suitable choice of $\Phi$ transformation can be by-passed if we can find a suitable kernel function $K$ such that the dot product $\Phi(\mathbf{u})^\top \Phi(\mathbf{z})$ can be represented as $K(\mathbf{u},\mathbf{z})$.

Indeed, this is possible using a result known as $\textbf{Mercer's theorem}$.

$\large{\text{Mercer's theorem}}$

$K$ satisfies:

$
\begin{align}
\int K(\mathbf{u}, \mathbf{z}) g(\mathbf{u}) g(\mathbf{z}) d\mathbf{u} d\mathbf{z} \geq 0
\end{align}
$

for any function $g$ satisfying $\int g^2(\mathbf{u}) d\mathbf{u} < \infty$, if and only if  $K$ corresponds to a unique transformation $\Phi$ such that $K( \mathbf{u}, \mathbf{z}) = \Phi(\mathbf{u})^\top \Phi(\mathbf{z})$.

Several possible choices of $K$ exist. Some examples are:

*   Polynomial kernel: $K(\mathbf{u}, \mathbf{z}) = (\mathbf{u}^\top \mathbf{z} + 1)^p$
*   Gaussian kernel (or) radial basis function (rbf) kernel: $K(\mathbf{u}, \mathbf{z}) = e^{-\frac{\|\mathbf{u}-\mathbf{z}\|^2}{2\sigma^2}}$

```python
np.random.seed(1000)
num_samples = len(X)
#Create an index array
indexarr = np.arange(num_samples) #index array
np.random.shuffle(indexarr) #shuffle the indices
#print('shuffled indices of samples:')
#print(indexarr)
```

```python
#Use the samples corresponding to first 80% of indexarr for training
num_train = int(0.8*num_samples)
#Use the remaining 20% samples for testing
num_test = num_samples-num_train
print('num_train: ',num_train, 'num_test: ', num_test)
```

    num_train:  160 num_test:  40

```python
#Use the first 80% of indexarr to create the train data features and train labels
train_X = X[indexarr[0:num_train]]
train_y = y[indexarr[0:num_train]]
print('shape of train data features:')
print(train_X.shape)
print('shape of train data labels')
print(train_y.shape)

# print (sum(x <= 0 for x in train_y))
```

    shape of train data features:
    (160, 2)
    shape of train data labels
    (160,)

```python
#Use remaining 20% of indexarr to create the test data and test labels
test_X = X[indexarr[num_train:num_samples]]
test_y = y[indexarr[num_train:num_samples]]
print('shape of test data features:')
print(test_X.shape)
print('shape of test data labels')
print(test_y.shape)
```

    shape of test data features:
    (40, 2)
    shape of test data labels
    (40,)

$\Large{\text{Finding best hyperparameter } \gamma \ \text{using cross validation}}$

Question: What is cross validation?

Answer: Cross validation (CV) is a technique employed to best tune the hyperparameters of a machine learning algorithm. The k-fold cross validation splits the data into k folds. The samples in $k-1$ folds are used in training phase and the remaining fold is treated as test set. The scores are computed at every iteration of CV and then the hyperparameter associated with highest average score is chosen for the testing purpose.

#Visualizing the working procedure of k-fold cross validation

Image credit:  $\texttt{https://scikit-learn.org/stable/_images/grid_search_cross_validation.png} $

```python
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

gammas = [0.0001,0.001, 0.01, 0.1, 1, 20, 40, 60, 80, 100]
kernels = ['sigmoid','linear', 'poly', 'rbf']
best_gamma = {}
cv_k = 5 #5-fold cross validation
for kernel_ in kernels:
  print (kernel_,'kernel')
  avg_score = np.zeros(len(gammas))
  for gamma in gammas:
    clf = SVC(kernel=kernel_, gamma=gamma, random_state=1)
    scores = cross_val_score(clf, train_X, train_y, cv=cv_k)
    avg_score[gammas.index(gamma)] = np.mean(scores)
    # print ('average score for kernel',kernel_, 'at gamma = ', gamma,'is',avg_score[gammas.index(gamma)])
  print (avg_score)
  max_score_index = np.argmax(avg_score)
  # print (max_score_index)
  best_gamma[kernel_] = gammas[int(max_score_index)]

print ('best hyperparameters = ', best_gamma)
```

    sigmoid kernel
    [0.5125  0.5125  0.8     0.85625 0.65625 0.575   0.5625  0.55625 0.55625
     0.55625]
    linear kernel
    [0.8625 0.8625 0.8625 0.8625 0.8625 0.8625 0.8625 0.8625 0.8625 0.8625]
    poly kernel
    [0.5125 0.5125 0.5125 0.6125 0.925  0.925  0.925  0.925  0.925  0.9125]
    rbf kernel
    [0.5125  0.5125  0.825   0.85625 0.98125 0.99375 0.99375 0.99375 1.
     1.     ]
    best hyperparameters =  {'sigmoid': 0.1, 'linear': 0.0001, 'poly': 1, 'rbf': 80}

Now let us try to use four different kernels with above learned hyperparameters on moon data set.

```python
# for creating a mesh to plot in
h=0.02 #mesh step size
num_samples = len(X)
x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, h),
                     np.arange(x2_min, x2_max, h))

from sklearn.svm import SVC
train_accuracy = {}
test_accuracy = {}
for kernel_ in kernels:
  clf = SVC(kernel=kernel_,gamma=best_gamma[kernel_],max_iter = 10000)
  clf_model = clf.fit(train_X,train_y.ravel())
  predicted_labels_train = clf_model.predict(train_X)
  # print(predicted_labels)
  train_error = np.sum(0.5*np.abs(predicted_labels_train-train_y.ravel()))/len(train_y)*100.0
  train_accuracy[kernel_] = 100.0-train_error

  predicted_labels_test = clf_model.predict(test_X)
  # print(predicted_labels)
  test_error = np.sum(0.5*np.abs(predicted_labels_test-test_y.ravel()))/len(test_y)*100.0
  test_accuracy[kernel_] = 100.0-test_error

  ## visualizing decision boundary
  Z = clf_model.predict(np.c_[xx1.ravel(), xx2.ravel()])
  # Put the result into a color plot
  Z = Z.reshape(xx1.shape)

  fig, (ax1, ax2) = plt.subplots(1,2, figsize=(16,6))
  ax1.contourf(xx1, xx2, Z, cmap=plt.cm.coolwarm, alpha=0.8)

  # Plot also the training points
  ax1.scatter(train_X[:, 0], train_X[:, 1], c=train_y, cmap=plt.cm.coolwarm)
  # plt.scatter(X[:int(num_samples/2),0],X[:int(num_samples/2),1], marker='o', color='red')
  # plt.scatter(X[int(num_samples/2):,0],X[int(num_samples/2):,1], marker='s', color='green')
  xlabel = 'x1' + str('\n')+'kernel='+str(kernel_)
  ax1.set_xlabel(xlabel)
  ax1.set_ylabel('x2')
  ax1.set_xlim(xx1.min(), xx1.max())
  ax1.set_ylim(xx2.min(), xx2.max())
  ax1.set_xticks(())
  ax1.set_yticks(())
  ax1.set_title('decision boundary with training points')

  #plot the test points along with decision boundaries
  ax2.contourf(xx1, xx2, Z, cmap=plt.cm.coolwarm, alpha=0.8)

  # Plot also the test points
  ax2.scatter(test_X[:, 0], test_X[:, 1], c=test_y, cmap=plt.cm.coolwarm)
  # plt.scatter(X[:int(num_samples/2),0],X[:int(num_samples/2),1], marker='o', color='red')
  # plt.scatter(X[int(num_samples/2):,0],X[int(num_samples/2):,1], marker='s', color='green')
  xlabel = 'x1' + str('\n')+'kernel='+str(kernel_)
  ax2.set_xlabel(xlabel)
  ax2.set_ylabel('x2')
  ax2.set_xlim(xx1.min(), xx1.max())
  ax2.set_ylim(xx2.min(), xx2.max())
  ax2.set_xticks(())
  ax2.set_yticks(())
  ax2.set_title('decision boundary with test points')

  plt.show()

print('train accuracy:', train_accuracy)
print('test accuracy:', test_accuracy)
```

#Let us now check the decision boundaries for RBF kernel with different $\gamma$ values

```python
# for creating a mesh to plot in
h=0.02 #mesh step size
x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, h),
                     np.arange(x2_min, x2_max, h))

from sklearn.svm import SVC
gamma = [0.0001,0.001, 0.01, 0.1, 1, 20, 40, 60, 80, 100]
train_accuracy = {}
test_accuracy = {}
for gamma_ in gamma:
  clf = SVC(kernel='rbf', gamma=gamma_, max_iter = 10000)
  clf_model = clf.fit(train_X,train_y.ravel())
  predicted_labels = clf_model.predict(train_X)
  # print(predicted_labels)
  train_error = np.sum(0.5*np.abs(predicted_labels-train_y.ravel()))/len(train_y)*100.0
  train_accuracy[gamma_] = 100.0-train_error
  predicted_test_labels = clf_model.predict(test_X)
  test_error = np.sum(0.5*np.abs(predicted_test_labels-test_y.ravel()))/len(test_y)*100.0
  test_accuracy[gamma_] = 100.0-test_error
  ## visualizing decision boundary
  Z = clf_model.predict(np.c_[xx1.ravel(), xx2.ravel()])
  # Put the result into a color plot
  Z = Z.reshape(xx1.shape)

  fig, (ax1, ax2) = plt.subplots(1,2, figsize=(16,6))
  ax1.contourf(xx1, xx2, Z, cmap=plt.cm.coolwarm, alpha=0.8)

  # Plot also the training points
  ax1.scatter(train_X[:, 0], train_X[:, 1], c=train_y, cmap=plt.cm.coolwarm)
  # plt.scatter(X[:int(num_samples/2),0],X[:int(num_samples/2),1], marker='o', color='red')
  # plt.scatter(X[int(num_samples/2):,0],X[int(num_samples/2):,1], marker='s', color='green')
  xlabel = 'x1 '+str('\n')+'kernel='+str(kernel_)+' gamma='+str(gamma_)
  ax1.set_xlabel(xlabel)
  ax1.set_ylabel('x2')
  ax1.set_xlim(xx1.min(), xx1.max())
  ax1.set_ylim(xx2.min(), xx2.max())
  ax1.set_xticks(())
  ax1.set_yticks(())
  ax1.set_title('decision boundary with training points')

  #plot the test points along with decision boundaries
  ax2.contourf(xx1, xx2, Z, cmap=plt.cm.coolwarm, alpha=0.8)

  # Plot also the training points
  ax2.scatter(test_X[:, 0], test_X[:, 1], c=test_y, cmap=plt.cm.coolwarm)
  # plt.scatter(X[:int(num_samples/2),0],X[:int(num_samples/2),1], marker='o', color='red')
  # plt.scatter(X[int(num_samples/2):,0],X[int(num_samples/2):,1], marker='s', color='green')
  xlabel = 'x1 '+str('\n')+'kernel='+str(kernel_)+' gamma='+str(gamma_)
  ax2.set_xlabel(xlabel)
  ax2.set_ylabel('x2')
  ax2.set_xlim(xx1.min(), xx1.max())
  ax2.set_ylim(xx2.min(), xx2.max())
  ax2.set_xticks(())
  ax2.set_yticks(())
  ax2.set_title('decision boundary with test points')

  plt.show()

print('train accuracy:', train_accuracy)
print ('test accuracy:',test_accuracy)
```


# ソルバを使う

**MatrixEquations.jl**を使って最適レギュレータを設計する．
公式ページ : [https://docs.juliahub.com/MatrixEquations/1uOBF/1.1.4/](https://docs.juliahub.com/MatrixEquations/1uOBF/1.1.4/)

```julia
using Plots
using LinearAlgebra
using MatrixEquations
```

システムを定義

$$
    \begin{align}
    \dot{x} &= Ax+Bu\\
    y &= Cx
    \end{align}\\
    A\in\mathbb{R}^{2\times2},~B\in\mathbb{R}^{2\times2},~C\in\mathbb{R}^{2\times2}
$$

```julia
x₀ = [1, 0.5]
A = [
    1.1 2
    -0.3 -1
]
B = [
    1 2
    0.847 3
]
C = [
    1. 0.
    0. 1.
];
```

安定性を調べてみる

```julia
eigvals(A)
```

    2-element Vector{Float64}:
     -0.6588723439378912
      0.7588723439378913

負の固有値を持つので不安定．
初期値応答を調べる．

```julia
t = 0:0.01:5
x = Vector{typeof(x₀)}(undef, length(t))
for i in 1:length(t)
    x[i] = exp(A .* t[i]) * x₀
end

x = permutedims(hcat(x...))
plot(t, x[:, 1], label="x1")
plot!(t, x[:, 2], label="x2")
plot!(xlabel="t")
savefig("lqr.png")
```

発散することがわかる．

## リカッチ方程式を解く
`MatrixEquations`の`arec`を使う．
ドキュメント : [https://docs.juliahub.com/MatrixEquations/1uOBF/1.1.4/autodocs/#MatrixEquations.arec](https://docs.juliahub.com/MatrixEquations/1uOBF/1.1.4/autodocs/#MatrixEquations.arec)

`arec`は次の連続代数リカッチ方程式を解く．

```julia
Q = diagm([10.0, 10.0])
R = diagm([1., 1.])
S = zero(B)
P, E, F = arec(A, B, R, Q, S);
```

戻り値は次の通り．
* `P` : リカッチ方程式の解
* `E` : 最適レギュレータの閉ループの固有値
* `F` : 最適フィードバック行列

```julia
P  # リカッチ方程式の解
```

    2×2 Matrix{Float64}:
      3.34568  -1.07266
     -1.07266   1.30235

```julia
E  # 閉ループの固有値
```

    2-element Vector{Float64}:
     -11.862446758953242
      -2.732479989130699

```julia
F  # 最適フィードバック行列
```

    2×2 Matrix{Float64}:
     2.43714  0.0304348
     3.47339  1.76174

最適フィードバック行列を入力に使ったときのシステムの時間発展を調べる．

```julia
Ā = A .- (B * F)  # 新しいA行列

x2 = Vector{typeof(x₀)}(undef, length(t))
u2 = Vector{typeof(x₀)}(undef, length(t))
for i in 1:length(t)
    x2[i] = exp(Ā .* t[i]) * x₀
    u2[i] = -F * x2[i]
end

x2 = permutedims(hcat(x2...))
figx = plot(t, x2[:, 1], label="x1")
plot!(figx, t, x2[:, 2], label="x2")
plot!(figx, xlabel="t")
savefig(figx, "lqr2.png")

u2 = permutedims(hcat(u2...))
figu = plot(t, u2[:, 1], label="u1")
plot!(figu, t, u2[:, 2], label="u2")
plot!(figu, xlabel="t")
savefig(figu, "lqr2u.png")
```

安定化されていることがわかる．