```python
# needed imports
import numpy as np
from numpy import empty
import matplotlib.pyplot as plt
```

# B-Splines

We recall the Cox-DeBoor formula, used as a definition for B-Splines:

The j-th B-spline of degree $p$ is defined by the recurrence relation:
\begin{align}
  N_j^p = \frac{t-t_j}{t_{j+p}-t_{j}} N_j^{p-1} + \frac{t_{j+p+1}-t}{t_{j+p+1}-t_{j+1}} N_{j+1}^{p-1},
  \label{eq:bspline-reccurence}
\end{align}

where $T=\{t_i\}_{0\leqslant i \leqslant m}$ is a sequence of non-decreasing real numbers, called the knot vector.

## Examples

**Example 1.**

We consider a linear B-Spline with the knot vector $T = [0, 1, 2]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return t
    if t >= 1 and t< 2: return 2-t
    return 0.

xs = np.linspace(0., 2., 200)
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.legend()
```

**Example 2.**

We consider a linear B-Spline with the knot vector $T = [0, 0, 1]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return 1-t
    return 0.

xs = np.linspace(0., 1., 200)
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.legend()
```

**Example 3.**

We consider a linear B-Spline with the knot vector $T = [0, 1, 1]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return t
    return 0.

xs = np.linspace(0., 1., 201)[:-1]
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.legend()
```

**Example 4.**

We consider linear B-Splines with the knot vector $T = [0, 0, 1, 1]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return 1-t
    return 0.

def N1_1(t):
    if t >= 0 and t< 1: return t
    return 0.

xs = np.linspace(0., 1., 201)[:-1]
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.plot(xs, [N1_1(x) for x in xs], label='$N_1^1$')
plt.legend()
```

**Example 5.**

We consider linear B-Splines with the knot vector $T = [0, 0, 1, 2]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return 1-t
    return 0.

def N1_1(t):
    if t >= 0 and t< 1: return t
    if t >= 1 and t< 2: return 2-t
    return 0.

xs = np.linspace(0., 2., 201)[:-1]
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.plot(xs, [N1_1(x) for x in xs], label='$N_1^1$')
plt.legend()
```

**Example 6.**

We consider a quadratic B-Spline with the knot vector $T = [0, 0, 1, 1]$

```python
def N2_0(t):
    if t >= 0 and t< 1: return 2*t*(1-t)
    return 0.

xs = np.linspace(0., 1., 201)[:-1]
plt.plot(xs, [N2_0(x) for x in xs], label='$N_0^2$')
plt.legend()
```

**Example 7.**

We consider a quadratic B-Spline with the knot vector $T = [0, 0, 1, 2]$

```python
T = [0, 0, 1, 2]
def N2_0(t):
    if t >= 0 and t< 1: return 2*t-3./2.*t**2
    if t >= 1 and t< 2: return 0.5*(2-t)**2
    return 0.

xs = np.linspace(0., 2., 200)
plt.plot(xs, [N2_0(x) for x in xs], label='$N_0^2$')
plt.legend()
```

**Example 8.**

We consider linear B-Splines with the knot vector $T = [0, 0, 1, 2, 3, 3]$

```python
def N1_0(t):
    if t >= 0 and t< 1: return 1-t
    return 0.

def N1_1(t):
    if t >= 0 and t< 1: return t
    if t >= 1 and t< 2: return 2-t
    return 0.

def N1_2(t):
    if t >= 1 and t< 2: return t-1
    if t >= 2 and t< 3: return 3-t
    return 0.

def N1_3(t):
    if t >= 2 and t< 3: return t-2
    return 0.

xs = np.linspace(0., 3., 201)[:-1]
plt.plot(xs, [N1_0(x) for x in xs], label='$N_0^1$')
plt.plot(xs, [N1_1(x) for x in xs], label='$N_1^1$')
plt.plot(xs, [N1_2(x) for x in xs], label='$N_2^1$')
plt.plot(xs, [N1_3(x) for x in xs], label='$N_3^1$')
plt.legend()
```

**Example 9.**

We consider quadratic B-Splines with the knot vector $T = [0, 0, 0, 1, 1, 1]$

```python
def N2_0(t):
    if t >= 0 and t< 1: return (1-t)**2
    return 0.

def N2_1(t):
    if t >= 0 and t< 1: return 2*t*(1-t)
    return 0.

def N2_2(t):
    if t >= 0 and t< 1: return t**2
    return 0.

xs = np.linspace(0., 1., 201)[:-1]
plt.plot(xs, [N2_0(x) for x in xs], label='$N_0^2$')
plt.plot(xs, [N2_1(x) for x in xs], label='$N_1^2$')
plt.plot(xs, [N2_2(x) for x in xs], label='$N_2^2$')
plt.legend()
```

## Evaluation of B-Splines

Given a knot sequence $T=\{t_i\}_{0\leqslant i \leqslant n + p}$, we are interested in the algorithmic evaluation of B-Splines of degree $p$. For a given real point $x$, it is done in two steps:

1. find the knot span index $j$, such that $x \in~ ] t_j,t_{j+1} [ $
2. evaluate all non-vanishing B-Splines $N_{j-p}^p, \cdots, N_j^p$

The first point is achieved by the function implemented by the following function:

```python
def find_span( knots, degree, x ):
    # Knot index at left/right boundary
    low  = degree
    high = 0
    high = len(knots)-1-degree

    # Check if point is exactly on left/right boundary, or outside domain
    if x <= knots[low ]: returnVal = low
    elif x >= knots[high]: returnVal = high-1
    else:
        # Perform binary search
        span = (low+high)//2
        while x < knots[span] or x >= knots[span+1]:
            if x < knots[span]:
                high = span
            else:
                low  = span
            span = (low+high)//2
        returnVal = span

    return returnVal
```

The second point is implemented by the following function, that returns all non-vanishing B-Splines at $x$

```python
def all_bsplines( knots, degree, x, span ):
    left   = empty( degree  , dtype=float )
    right  = empty( degree  , dtype=float )
    values = empty( degree+1, dtype=float )

    values[0] = 1.0
    for j in range(0,degree):
        left [j] = x - knots[span-j]
        right[j] = knots[span+1+j] - x
        saved    = 0.0
        for r in range(0,j+1):
            temp      = values[r] / (right[r] + left[j-r])
            values[r] = saved + right[r] * temp
            saved     = left[j-r] * temp
        values[j+1] = saved

    return values
```

The following function plots all B-Splines given a knot vector and a polynomial degree.

```python
def plot_splines(knots, degree, nx=100):
    xmin = knots[degree]
    xmax = knots[-degree-1]

    # grid points for evaluation
    xs = np.linspace(xmin,xmax,nx)

    # this is the number of the BSplines in the Schoenberg space
    N = len(knots) - degree - 1

    ys = np.zeros((N,nx), dtype=np.double)
    for ix,x in enumerate(xs):
        span = find_span( knots, degree, x )
        b    = all_bsplines( knots, degree, x, span )
        ys[span-degree:span+1, ix] = b[:]

    for i in range(0,N):
        plt.plot(xs,ys[i,:], label='$N_{}$'.format(i+1))
    plt.legend(loc=9, ncol=4)
```

### Knots vector families

There are two kind of **knots vectors**, called **clamped** and **unclamped**. Both families contains **uniform** and **non-uniform** sequences.

The following are examples of such knots vectors

#### Clamped knots (open knots vector)

##### uniform

```python
T = np.array([0, 0, 0, 1, 2, 3, 4, 5, 5, 5])

plot_splines(T, degree=2, nx=100)
```

```python
T = [-0.2, -0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 0.8]

plot_splines(T, degree=2, nx=100)
```

##### non-uniform

```python
T = [0, 0, 0, 1, 3, 4, 5, 5, 5]

plot_splines(T, degree=2, nx=100)
```

```python
T = [-0.2, -0.2, 0.4, 0.6, 0.8, 0.8]

plot_splines(T, degree=2, nx=100)
```

#### Unclamped knots

##### uniform

```python
T = [0, 1, 2, 3, 4, 5, 6, 7, 8]

plot_splines(T, degree=2, nx=100)
```

```python
T = [-0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0]

plot_splines(T, degree=2, nx=100)
```

##### non-uniform

```python
T = [0, 0, 3, 4, 7, 8, 9]

plot_splines(T, degree=2, nx=100)
```

```python
T = [-0.2, 0.2, 0.4, 0.6, 1.0, 2.0, 2.5]

plot_splines(T, degree=2, nx=100)
```

```python
# css style
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```

```python

```


# Attitude estimation

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
import matplotlib.pyplot as plt

from helpers import IMU, plot_compare
```

In this exercise you will use measurements from the IMU of the drone's attitude and body rates. The state and measurements are given by:

$$
\begin{align}
x_t= \begin{bmatrix} \theta \\ \phi \end{bmatrix} && z_t =\begin{bmatrix} \theta \\ \phi \\ q \\ p  \end{bmatrix}
\end{align}
$$

For this exercise, we will assume that the drone is close to the hover mode, so we can assume rotational rates in the body frame are the same as in the global frame.

$$
\dot{\phi} = p \\
\dot{\theta} = q
$$

For larger angles, you have to use the transformation matrix which was covered in previous lessons.

The equations which describe the attitude update are as follows:

$$
\hat{\theta}_t = \frac{\tau}{\tau + dt}\left( \hat{\theta}_{t-1} +  z_{t,\dot{\theta}} \times dt\right) + \frac{dt}{\tau + dt} z_{t,\theta} \\
\hat{\phi}_t = \frac{\tau}{\tau + dt}\left( \hat{\phi}_{t-1} +  z_{t,\dot{\phi}} \times dt\right) + \frac{dt}{\tau + dt} z_{t,\phi}
$$

```python
class ComplementaryFilter:
    def __init__(self,dt,tau):
        self.dt = dt
        self.tau = tau
        self.estimated_theta = 0.0
        self.estimated_phi   = 0.0

    def update(self,z):

        # TODO:
        #  implement the estimated pitch and roll

        self.estimated_theta = self.tau / (self.tau + self.dt) * (self.estimated_theta + z[2] * self.dt) \
                             + self.dt / (self.tau + self.dt) * z[0]
        self.estimated_phi = self.tau / (self.tau + self.dt) * (self.estimated_phi + z[3] * self.dt) \
                           + self.dt / (self.tau + self.dt) * z[1]

    def make_estimates(self,measurements):
        self.estimated_theta = 0.0
        self.estimated_phi = 0.0
        estimates = np.zeros((2, measurements.shape[1]))
        for i in range(measurements.shape[1]):
            z = measurements[:,i]
            self.update(z)
            est = np.array([self.estimated_theta, self.estimated_phi])
            estimates[:,i] = est
        return estimates
```

```python
# drone hovers for all N measurements
N = 2000
true_values = np.zeros((4, N))

# simulate sensor measurements
imu = IMU()
measurements = imu.make_measurements(true_values)
```

```python
# Parameters for complementary filter
#
# TODO - try modifying TAU.
#   What value gives you the best estimate?
#   What do your plots look like when TAU = DT * 100?
#   What about when TAU = DT * 1?

DT  = 0.01
TAU = DT * 1
cf  = ComplementaryFilter(DT, TAU)

# make estimates based on measurements
estimates = cf.make_estimates(measurements)

# integrating gyro directly
integrated_ests = np.zeros((2,N))
integrated_ests[0,:] = np.cumsum(measurements[2,:]) * DT
integrated_ests[1,:] = np.cumsum(measurements[3,:]) * DT
```

```python
# plot pitch
plot_compare(true_values,estimates,measurements,integrated_ests, DT, 0)
```

```python
# plot roll
plot_compare(true_values,estimates,measurements,integrated_ests, DT, 1)
```

[Solution](./Attitude%20estimation-Solution-Revised.ipynb)


# Root finding and optimization

## Introduction

In statistical applications, point estimation problems often boil down to maximize a function according to:

* maximize likelihood

* least squares

* maximum of a posterior distribution (i.e. bayesian)

When the function to be optimized is "smooth", we can reformulate optimization into a root finding problem.

**Trouble:** these problems often have no analytical solution.

## General setup

We will discuss 2 kinds of problems

* Root finding: solve $f(x) = 0$ for $x \in \mathbf{R}^d$, $d \geq 1$

* Optimization: maximize $g(x)$ for $x \in \mathbf{R}^d$, $d \geq 1$

We will address univariate and multivariate problems cases separately. We will look at methods that construct a sequence $\{ x_t : t \geq 0 \}$ designed to converge (as $t \xrightarrow{} \infty$) to the solution, denoted by $x^\star$.

### Theoretical considerations

* Under what conditions on $f$ (or $g$) and initial guess $x_0$ can we prove that $x_t \xrightarrow{} x^\star$?

* If $x_t \xrightarrow{} x^\star$, then how fast, i.e. what is its convergence order?

### Practical consideration

* How to write and implement the algorithm?

* Can't run the algorithm till $t = \infty$, so how to stop?

## Univariate problems

## Bisection (root-finding)

1. Find unique root $x^\star$ of $f$ in an interval $[a, b]$

* $f(a)f(b) \leq 0$ means a root exists between $a$ and $b$ (by the intermediate value theorem).

Pick an initial guess $x_0 = \frac{a + b}{2}$.

$x^\star$ must be in either $[ a, x_0 ]$ or $[ x_0, b ]$

Evaluate $f(x)$ at the end points to determine which one. The selected interval, call it $[a_1, b_1]$, is now just like the initial interval. Take $x_1 = \frac{a_1 + b_1}{2}$. Continue this process to construct a sequence $\{ x_t : t \geq 0 \}$.

### Algorithm

Assume $f(x)$ and the interval $[a, b]$ are given.

1. Set $x = \frac{a+b}{2}$

2. If $f(a)f(b) \leq 0$, then $b=x$, else $a=x$.

3. If "converged", then stop; otherwise, return to step 1.

The convergence criteria is usually something like $| x_{new} - x_{old} | < \epsilon$ where $\epsilon$ is a specified small number (e.g. $\epsilon = 10^{-8}$).
A relative convergence riteria might be better $\frac{|x_{new} - x_{old}|}{x_{old}} < \epsilon$.

As a reminder, the limit of a sequence is defined by: $\forall \epsilon > 0, \ni N > 0$ s.t. $|U_N - l| < \epsilon$ where $N \xrightarrow{} \infty$ is the number of items in the sequence and $l$ is the value the sequence approaches.

**Theorem:** If $f$ is continuous, then $x_t \xrightarrow{} x^\star$.

Time complexity: $\log n$

```python
from utils import disp
disp('bisection_method.png')
```

## Newton's method (Newton-Raphson)

### Basic idea

Newton's method approximates a nonlinear function, which near its root by a linear function, can be solved by hand
It is based on Taylor's theorem, which gives the linear approximation of a function $f(x)$ in a neighborhood of some point $x_0$ as

$$f(x) \approx f(x_0) + f^\prime(x_0) (x - x_0)$$

We can set this equal to 0 to solve for $x$

$$x = x_0 - \frac{f(x_0)}{f^\prime(x_0)}$$

### Algorithm - for root finding

Assume the function $f(x)$, its derivative $f^\prime(x)$, and an initial guess are given.

* Set $t=0$

1. Calculate $x_{t+1} = x_t - \frac{f(x_t)}{f^\prime(x_t)}$

2. If the convergence criteria is met, then stop; otherwise, set $t \xleftarrow{} t+1$ and return to step 1.

### Warnings

Convergence depends on choice of $x_0$ and on $f$

Unlike bisection, Newton might not converge.

### Newton method theory

**Theorem**: If $f^{\prime\prime}$ is continuous and $x^\star$ is a root of $f$, with $f^\prime(x_0) \neq 0$, then there exists a neighborhood $N$ of $x^\star$ s.t. Newton's method converges to $x^\star$ for any $x_0 \in N$.

The proof uses taylor's approximation. Proof also shows that the convergence order is quadratic.

If Newton converges, then its way faster than bisection. But, added speed has a cost. It requires differentiability and the derivative $f^\prime$. Also, convergence is sensitive to choice of $x_0$.

```python
disp('newtons_method.png')
```

### Newton algorithm (for optimization)

In the context of unconstrained optimization, Newton's method can be used to find the optimum value. Apply the above method to find a root of $g^\prime(x) = 0$. The algorithm becomes

Assume that the 1st derivative $g^\prime(x)$, its 2nd derivative $g^{\prime\prime}(x)$, and an initial guess $x_0$ are given. Set $t=0$,

i. Calculate $x_{t+1} = x_t - \frac{g^\prime(x_t)}{g^{\prime\prime}(x_t)}$

ii. If the convergence criteria is met, then stop; otherwise, set $t \xleftarrow{} t + 1$ and return to step i.

**Example:** Find $x^\star$ to maximize $g(x) = \frac{\log x}{1 + x}$, $x \in [1,5]$.

**a. Find root via bisection**

**b. Find root via newton's method**

```python
import numpy as np
import sympy as sym
import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)

class RootFinder:

    def __init__(self, sympy_f, x):

        self.sympy_f = sympy_f
        self.x = x
        self._calculate_derivative()
        self._calculate_derivative_2()
        self.f = self._lambify(self.sympy_f)

        self.bisection_x = None
        self.newton_x = None
        self.secant_x = None
        self.gss_x = None

        print('Function:')
        print(self.sympy_f)
        print('First derivative:')
        print(self.sympy_deriv)
        print('Second derivative')
        print(self.sympy_deriv_2)

    def _lambify(self, f):
        """Convert a function to a lambda function"""
        return sym.lambdify(self.x, f, "numpy")

    def _calculate_derivative(self):
        """Calculate the derivative of the function"""
        self.sympy_deriv =  sym.diff(self.sympy_f, self.x)
        self.deriv = self._lambify(self.sympy_deriv)

    def _calculate_derivative_2(self):
        """Calculate the second derivative of the function"""
        self.sympy_deriv_2 =  sym.diff(self.sympy_f, self.x, 2)
        self.deriv_2 = self._lambify(self.sympy_deriv_2)

    def set_problem(self, solution_type='root', # or 'max'
                    epsilon=1e-8, iter=1000):
        self.solution_type = solution_type
        self.epsilon = epsilon
        self.iter = iter

        if solution_type == 'root':
            self.problem_f = self.f
            self.problem_deriv = self.deriv
            self.conditional = lambda xs, iteration : ( abs(self.problem_f(xs[-1])) > self.epsilon ) or ( self.iter > iteration )
        elif solution_type == 'max':
            self.problem_f = self.deriv
            self.problem_deriv = self.deriv_2
            self.conditional = lambda xs, iteration : True if len(xs) < 2 else ( abs(self.problem_f(xs[-1] - xs[-2])) > self.epsilon ) or ( self.iter > iteration )

    def bisection_method(self, a, b):
        """
        Bisection method for root finding.
        """
        iteration = 0
        c = (a + b) / 2
        cs = [c]
        while self.conditional(cs, iteration):
            if self.problem_f(a) * self.problem_f(c) < 0:
                b = c
            else:
                a = c
            c = (a + b) / 2
            cs.append(c)
            iteration += 1
        self.bisection_x = c
        return c

    def newtons_method(self, x0):
        """
        Newton's method for root finding.
        """

        iteration = 0
        x = x0
        xs = [x]
        while self.conditional(xs, iteration):
            x = x - self.problem_f(x) / self.problem_deriv(x)
            iteration += 1
            xs.append(x)
        self.newton_x = x
        return x

    def secant_method(self, a, b):
        '''Approximate solution of f(x)=0 on interval [a,b] by the secant method.

        Parameters
        ----------
        f : function
            The function for which we are trying to approximate a solution f(x)=0.
        a,b : numbers
            The interval in which to search for a solution. The function returns
            None if f(a)*f(b) >= 0 since a solution is not guaranteed.

        Returns
        -------
        m_N : number
            The x intercept of the secant line on the the Nth interval
                m_n = a_n - f(a_n)*(b_n - a_n)/(f(b_n) - f(a_n))
            The initial interval [a_0,b_0] is given by [a,b]. If f(m_n) == 0
            for some intercept m_n then the function returns this solution.

        Examples
        --------
        >>> f = lambda x: x**2 - x - 1
        >>> secant(f,1,2,5)
        1.6180257510729614
        '''
        f = self.problem_f

        iteration = 0
        if f(a)*f(b) >= 0:
            print("Secant method fails.")
            return None
        a_n = a
        b_n = b
        xs = [a_n]
        #while abs(self.f(a_n)) > 0.01:
        while self.conditional(xs, iteration):
            m_n = a_n - f(a_n)*(b_n - a_n)/(f(b_n) - f(a_n))
            xs.append(m_n)
            iteration += 1
            f_m_n = f(m_n)
            if f(a_n)*f_m_n < 0:
                a_n = a_n
                b_n = m_n
            elif f(b_n)*f_m_n < 0:
                a_n = m_n
                b_n = b_n
            elif f_m_n == 0:
                break
        self.secant_x = m_n
        return m_n

    def golden_selection_search_method(self, a, b):

        f = self.problem_f
        c = (np.sqrt(5) - 1) / 2

        x1 = b - c * (b - a)
        x2 = a + c * (b - a)
        while abs(b - a) > self.epsilon:
            if f(x2) > f(x1):
                b = x2
                x2 = x1
                x1 = b - c * (b - a)
            else:
                a = x1
                x1 = x2
                x2 = a + c * (b - a)
        self.gss_x = x1 #could be a,b,x1,x2... they all converge

    def visualize(self, low=0, high=5):
        print(f"\nRoots Found: Bisection: x={self.bisection_x}, Newton's Method: x={self.newton_x}, Secant method: x={self.secant_x}, GSS Method: x={self.gss_x}")
        xx = np.linspace(low, high, 100)
        import matplotlib.pyplot as plt
        plt.plot(xx, self.f(xx))
        try:
            plt.scatter([self.bisection_x], [self.f(self.bisection_x)], marker='o', label=f'Bisection: x={round(self.bisection_x,2)}')
        except:
            pass
        try:
            plt.scatter([self.newton_x], [self.f(self.newton_x)], marker='o', label=f'Newton: x={round(self.newton_x,2)}')
        except:
            pass
        try:
            plt.scatter([self.secant_x], [self.f(self.secant_x)], marker='o', label=f'Secant: x={round(self.secant_x,2)}')
        except:
            pass
        try:
            plt.scatter([self.gss_x], [self.f(self.gss_x)], marker='o', label=f'GSS: x={round(self.gss_x,2)}')
        except:
            pass
        plt.legend()
        plt.show()

```

```python
x = sym.Symbol('x')
rf = RootFinder(sym.log(x) / (x + 1), x)
rf.set_problem(solution_type='root', # or 'max'
               epsilon=1e-8, iter=1000)
b = rf.bisection_method(0, 5)
n = rf.newtons_method(0.5)
s = rf.secant_method(0.5, 5)
rf.visualize()

```

```python
x = sym.Symbol('x')
rf = RootFinder(4 * x**2 * sym.exp(-2 * x), x)
rf.set_problem(solution_type='max', # or 'max'
               epsilon=1e-8, iter=1000)
b = rf.bisection_method(0, 5)
n = rf.newtons_method(0.5)
s = rf.secant_method(0.5, 5)
rf.golden_selection_search_method(0.5, 5)
rf.visualize()
```

```python
x = sym.Symbol('x')
rf = RootFinder(sym.exp(-x) - sym.log(x), x)
rf.set_problem(solution_type='root', # or 'max'
               epsilon=1e-8, iter=1000)
b = rf.bisection_method(0, 5)
n = rf.newtons_method(0.5)
s = rf.secant_method(0.5, 5)
rf.visualize()

```

## Secant method

### Basic idea

Newton's method requires a formula for $f^\prime(x)$. To avoid this, approximate $f^\prime(x)$ at $x_0$ by a difference ratio. That is, recall from calculus that
$$f^\prime(x) \approx \frac{f(x+h) - f(x)}{h}$$
where $h$ is small and positive.

This can also be expressed as
$$f^\prime(x) \approx \frac{f(x) - f(x_0)}{x - x_0}$$

The secant method follows Newton's method except we substitute a differenceratio for $f^\prime(x)$. Name is because the linear approximaiton is a secant not a tangent.

### Algorithm

Suppose $f(x)$ and 2 initial guesses $x_0$ and $x_1$ are given. Set $t=1$.

i. Calculate $x_{t+1} = x_t - \frac{f^\prime(x_t)}{\frac{f^\prime(x_t) - f^\prime(x_{t-1})}{x_t - x_{t-1}}}$

i.e. $x_{t+1} = x_t - f^\prime(x_t) \frac{x_t - x_{t-1}}{f^\prime(x_t) - f^\prime(x_{t-1})}$

If the convergence criteria are satisfied, then stop. Else, set $t\xleftarrow{} t+1$ and return to step i.

### Remarks

Can be unstable at early iterations bc the difference ratio may be a poor approximation of $f^\prime$; reasonable sacrifice if $f^\prime$ is not available.

If secant method converges, order is almost quadratic (similar to newton method).

## Fisher Scoring

In maximum likelihood applications, the goal is to find roots of the log-likelihood function, ie. $l^\prime(\hat{\theta}) = 0$.

In this context, Newton's method looks like

$$\theta_{t+1} = \theta_t - \frac{l^\prime(\theta_t)}{l^{\prime\prime}(\theta_t)}$$

$t \geq 0$.

Recall that $-l^{\prime\prime}(\theta)$ is an approximate of the Fisher information $I_n(\theta)$. So, we can rewrite Newton's method as

$$\theta_{t+1} = \theta_t + \frac{l^\prime(\theta_t)}{I_n(\theta_t)}$$

$t \geq 0$.

This modification is called Fisher's scoring

### Example: $g(x) = \frac{\log x}{1+x}$

#### Newton's Method

$x_{t+1} = x_t + h_t$ where $h_t = -\frac{g^\prime(x_t)}{g^{\prime\prime}(x_t)}$

$g^\prime(x) = -log(x)/(x + 1)^2 + 1/(x*(x + 1))$

The netwon increment is

$h_t = \frac{(x_t + 1)(1 + 1/x_t - \log x_t)}{3 + 4/x_t + 1/(x_t)^2 - 2 \log x_t}$

#### Secant method

$x_{t+1} = x_t - g^\prime(x_t) \frac{x_t - x_{t-1}}{g^\prime(x_t) - g^\prime(x_{t-1})}$

## Golden section search (GSS) method

It is a simple way of finding the minimizer of a function $f$ which has a simple minimum on the interval $[a, b]$. Assume that $f(x)$ is a continuous function.

Note that the method applies as well as to find the maximum.

The ideas of GSS are similar to bisection.

In designing the method, we seek to satisfy 2 goals:

1. An optimal reduction factor for the search interval
2. minimum number of function calls

So we want to perform 1 new function evaluation in each step. Also, we want a constant reduction factor, say c, for the size of the interval.

For $x_1$ and $x_2$ in $[a,b]$, there are 2 cases.

1. if $f(x_1) < f(x_2)$, then $[a,b] \xrightarrow{} [a, x_2]$ with interval size reduction $x_2 - a = c(b-a) \xrightarrow{} x_2 = (1 - c)a + cb$
2. if $f(x_1) > f(x_2)$, then $[a,b] \xrightarrow{} [x_1, b]$ with interval size reduction $b - x_1 = c(b-a) \xrightarrow{} x_1 = ca + (1 - c)b$

Thus, once we know "c", we know the location of $x_1$ and $x_2$

Without loss of generality, we focus on the case $f(x_1) < f(x_2)$. For ease of calculation, take $[a, b] = [0, 1]$.

If $f(x_1) < f(x_2)$, then we recycle $x_1 = 1 - c$ and have to determine where to evaluate next, either a the left, or at the right of $1 - c$.

i. Suppose we place a new function evaluation at the left of $x_1 = 1 - c$. It can be shown that $c$ is obtained by solving $c^2 + c - 1 = 0$. The positive root leads to $c = \frac{-1 + \sqrt{5}}{2}$ which equals approximately $0.6180$.

ii. Suppose we place a new function evaluation at the right of $x_1 = 1 - c$. It can be shown that $c$ is obtained by solving $1 - c^2 = 0$. The (double) root of this equation is 1, which is unhelpful because it does not give any reduction! So, we exclude this possibility.

Thus, we have 2 rules:

1. If $f(x_1) < f(x_2)$, we keep $x_1$ which becomes $x_2$ and we compute a new $x_1$. using $b - x_1 = c(b-a) \xrightarrow{} x_1 = ca + (1 - c)b$.

2. If $f(x_1) > f(x_2)$, we keep $x_2$ which becomes $x_1$ and we compute a new $x_2$. using $x_2 - a = c(b-a) \xrightarrow{} x_2 = (1 - c)a + cb$.

$c = \frac{\sqrt{5} - 1}{2}$ is called the **golden ratio**.

### GSS Algorithm

**Step 1:** Given initial interval $[a_1, b_1]$ and precision $\epsilon$, set $c = \frac{\sqrt{5} - 1}{2} = 0.618$.  Calculate $x^1_1 = b_1 - c(b_1 - a_1)$ and $x_2^1 = a_1 + c(b_1 - a_1)$.

**Step 2:** If $f(x_2^i) > f(x_1^i)$, update $a_{i+1} = a_i$, $b_{i+1} = x_2^i$, $x_2^{i+1} = x_1^i$, $x_1^{i+1} = b_{i+1} - c(b_{i+1} - a_{i+1})$

If $f(x_2^i) \leq f(x_1^i)$, update $a_{i+1} = x_1^i$, $b_{i+1} = b_i$, $x_1^{i+1} = x_2^i$, $x_2^{i+1} = a_{i+1} + c(b_{i+1} - a_{i+1})$

**Step 3:** If $|b_{i+1} - a_{i+1}| \leq \epsilon$, stop; otherwise, set $i \xleftarrow{} i+1$ and go to step 2.

## Multivariate optimization problems

### Newton's method

Suppose that $g(\vec{x})$ is a function of several variables, say $\vec{x} = (x_1, x_2, ..., x_p) \in \mathbf{R}^p$.

Newton's method works exactly the same as before, just the derivatives are more complicated.

$\dot{g}(\vec{x})$ is the gradient-vector of first partial derivatives

$\ddot{g}(\vec{x})$ is the Hessian-matrix of second partial derivatives

Based on Taylor's formula, Newton's method is $\vec{x}^{(t+1)} = \vec{x}^{(t)} - [\ddot{g}(x^{(t)})]^{-1} \dot{g}(x^{(t)})$

**Example:** Let $\vec{x} = (x_1, x_2)$ and $g(\vec{x}) = 0.5 x_1^2 + 2.5 x_2^2$

$g: \mathbf{R}^2 \xrightarrow{} \mathbf{R}$

($\mathbf{R}^p \xrightarrow{} \mathbf{R}$)

Using Newton's update, we have $\vec{x}^{(t+1)} = \vec{x}^{(t)} - [\ddot{g}(x^{(t)})]^{-1} \dot{g}(x^{(t)})$

Where $$\dot{g}(x) = \begin{bmatrix} \frac{\delta g}{\delta x_1}\\ \frac{\delta g}{\delta x_2}\end{bmatrix} = \begin{bmatrix} x_1 \\ 5x_2\\ \end{bmatrix}$$ and $$\ddot{g}(\vec{x}) = \begin{bmatrix}
\frac{\delta^2 g}{\delta x_1^2} & \frac{\delta^2 g}{\delta x_1 x_2}\\
\frac{\delta^2 g}{\delta x_2 x_1} & \frac{\delta^2 g}{\delta x_2^2}\\
\end{bmatrix} = \begin{bmatrix}
1 & 0\\
0 & 5\\
\end{bmatrix}$$

**Example:** (Example 2.4 on pg 35 in Gibbons + H)

Apply Newton's method to a complicated bivariate function

```python
import numpy as np
import sympy as sym
from itertools import permutations, product

class RootFinder:

    def __init__(self, sympy_f):
        self.sympy_f = sympy_f
        self.vars = list(sympy_f.free_symbols)
        self._calculate_derivative()
        self._calculate_derivative_2()
        self._inv_jacobian()
        self.f = self._lambify(self.sympy_f)

        self.newton_x = None

        print('Function:')
        print(self.sympy_f)
        print('First derivative:')
        print(self.deriv_vec)
        print('Second derivative')
        print(self.jacobian)
        print()

    def _lambify(self, f):
        """Convert a function to a lambda function"""
        return lambda x : float(f.evalf(subs=dict(zip(self.vars, x))))

    def _calculate_derivative(self):
        """Calculate the derivative of the function"""

        num_params = len(self.vars)
        self.deriv_vec = np.zeros((num_params,), dtype=object)
        self.deriv_vec_lambdify = np.zeros((num_params,), dtype=object)
        # Add all squared terms
        for a in self.vars:
            idx_a = self.vars.index(a)
            sympy_partial_deriv = sym.diff(self.sympy_f, a)
            self.deriv_vec[idx_a] = sympy_partial_deriv
            self.deriv_vec_lambdify[idx_a] = self._lambify(sympy_partial_deriv)

    def _calculate_derivative_2(self):
        """Calculate the second derivative of the function"""
        # Add all pairs
        combins = list(permutations(self.vars, 2))

        # Establish jacobian
        num_params = len(self.vars)
        jacobian = np.zeros((num_params, num_params), dtype=object)
        jacobian_lambdify = np.zeros((num_params, num_params), dtype=object)

        # Add all squared terms
        for a in self.vars:
            idx_a = self.vars.index(a)
            sympy_partial_deriv = sym.diff(self.sympy_f, a)
            sympy_sec_partial_deriv = sym.diff(sympy_partial_deriv, a)
            jacobian[idx_a, idx_a] = sympy_sec_partial_deriv
            jacobian_lambdify[idx_a, idx_a] = self._lambify(sympy_sec_partial_deriv)

        for (a, b) in combins:
            idx_a = self.vars.index(a)
            idx_b = self.vars.index(b)
            sympy_partial_deriv = sym.diff(self.sympy_f, a)
            sympy_sec_partial_deriv = sym.diff(sympy_partial_deriv, b)
            jacobian[idx_a, idx_b] = sympy_sec_partial_deriv
            jacobian_lambdify[idx_a, idx_b] = self._lambify(sympy_sec_partial_deriv)

        self.jacobian = jacobian
        self.jacobian_lambdify = jacobian_lambdify

    def _inv_jacobian(self):
        jacobian = sym.Matrix(self.jacobian)
        inv_jacobian = jacobian.inv()
        inv_jacobian = np.array(inv_jacobian, dtype=object)
        self.inv_jacobian = inv_jacobian

        inv_jacobian_lambdify = np.zeros((len(inv_jacobian), len(inv_jacobian)), dtype=object)
        for i in range(len(inv_jacobian)):
            for j in range(len(inv_jacobian)):
                inv_jacobian_lambdify[i, j] = self._lambify(inv_jacobian[i, j])
        self.inv_jacobian_lambdify = inv_jacobian_lambdify

    def _evaluate_func_arr(self, f, x):
        # function f is an array (vector or matrix) of lambdas
        # x is a scalar or array of input values
        shape = f.shape
        output = np.zeros(shape)
        for idx in product(*[range(s) for s in shape]):
            output[idx] = f[idx](x)
        return output

    def newtons_method(self, x0, epsilon):
        """
        Newton's method for root finding.

        x0 must be of length = number of variables in expression, organized according to order in `self.vars`
        """
        print("Starting Newton-Raphson with specification:", dict(zip( self.vars, x0 )))
        x = x0
        while abs(self.f(x)) > epsilon:
            inv_jacob_x = self._evaluate_func_arr(self.inv_jacobian_lambdify, x)
            deriv_x = self._evaluate_func_arr(self.deriv_vec_lambdify, x)
            x = x - inv_jacob_x @ deriv_x
        self.newton_x = x
        return x

x = sym.Symbol('x')
y = sym.Symbol('y')
rf = RootFinder(0.5 * x**2 + 2.5 * y**2)

n = rf.newtons_method([1, 1], 1e-6)
print("Found root:", n)
```

    Function:
    0.5*x**2 + 2.5*y**2
    First derivative:
    [1.0*x 5.0*y]
    Second derivative
    [[1.00000000000000 0]
     [0 5.00000000000000]]

    Starting Newton-Raphson with specification: {x: 1, y: 1}
    Found root: [0. 0.]

```python
x = sym.Symbol('x')
y = sym.Symbol('y')
rf = RootFinder(-(5 * x**2 + y**2 + 4 * x * y - 14 * x - 6 * y + 20))

# Add minimum functionality
#n = rf.newtons_method([1, 1], 1e-6)
#print("Found root:", n)
```

    Function:
    -5*x**2 - 4*x*y + 14*x - y**2 + 6*y - 20
    First derivative:
    [-10*x - 4*y + 14 -4*x - 2*y + 6]
    Second derivative
    [[-10 -4]
     [-4 -2]]

```python
x = sym.Symbol('x')
y = sym.Symbol('y')
rf = RootFinder((-1)*((((x**2)+y-11)**2)+(x+(y**2)-7)**2))

# Add minimum functionality
n = rf.newtons_method([1, 1], 1e-6)
print("Found root:", n)
```

    Function:
    -(x + y**2 - 7)**2 - (x**2 + y - 11)**2
    First derivative:
    [-4*x*(x**2 + y - 11) - 2*x - 2*y**2 + 14
     -2*x**2 - 4*y*(x + y**2 - 7) - 2*y + 22]
    Second derivative
    [[-12*x**2 - 4*y + 42 -4*x - 4*y]
     [-4*x - 4*y -4*x - 12*y**2 + 26]]

    Starting Newton-Raphson with specification: {x: 1, y: 1}
    Found root: [-3.77935206 -3.2832109 ]

## Newton-like methods

Newton's methods rely on updating equations $x^{(t+1)} = x^{(t)} + h^{(t)}$ where $h^{(t)} = -[ \ddot{g}(x^{(t)})]^{-1} \dot{g}(x^{(t)})$. The difficulty in the multivariate case is the derivation and/or computation of the Hessian matrix and its inverse.

A Newton-like method uses instead $$x^{(t+1)} = x^{(t)} - [M^{(t)}]^{-1} \dot{g}(x^{(t)})$$ where $M^{(t)}$ approximates the Hessian matrix.

**Question:** Is it possible to use some other matrix, say $M^{(t)}$, in place of the Hessian?

Yes!

This idea inspires a medley of algorithms:

## Steepest ascent/descent method

Fix matrices $M^{(t)}$ and numbers $\alpha^{(t)}, t\geq 0$.

Ascent/decsent method look like $$x^{(t+1)} = x^{(t)} - \alpha^{(t)} [M^{(t)}]^{-1} \dot{g}(x^{(t)})$$

Goal is to choose $M^{(t)}$ and $\alpha^{(t)}$ s.t. the function increases/decreases when $x^{(t)}$ is updated to $x^{(t+1)}$.

We simplify the problem by setting $M^{(t)} = -I$ for ascent and descent takes $M^{(t)} = I$.

Motivation is the basic fact from multivariate calculus that the gradient points in the direction of steepest ascent. Then, the algorithm looks like

$$x^{(t+1)} = x^{(t)} + \alpha^{(t)} \dot{g}(x^{(t)})$$

 Or, for steepest descent,

$$x^{(t+1)} = x^{(t)} - \alpha^{(t)} \dot{g}(x^{(t)})$$

s.t. $t \geq 0$ and where $\alpha^{(t)}$ is the **line search** parameter that determines how far to go in a given direction.

### How do we pick a good $\alpha^{(t)}$?

i. Determine appropriate value for $\alpha^{(t)}$ at each iteration in a one-dimensional minimization problem

$$\min_{\alpha^{(t)}} g( x^{(t)} - \alpha^{(t)} \dot{g}(x^{(t)}) )$$

can be solved by methods already discussed.

Golden Section Search and Bisection can be used for steepest descent.

ii. Another approach is "backtracking"

- Backtracking approach determines $\alpha^{(t)}$ iteratively: (Ascent)

1. Start with $\alpha^{(t)} = 1$

2. Compute update $x^{(t+1)}$ with this $\alpha^{(t)}$

3. If ascent holds, then increment $t$; otherwise, set $\alpha^{(t)} \xleftarrow{} \frac{\alpha^{(t)}}{2}$ and go back to step 2.

### Common choices of the step size

1. Exact line search

$$\alpha^{(t)} \equiv \argmin_{\alpha \geq 0} g(x^{(t)} - \alpha^{(t)} \dot{g}(x^{(t)}))$$

if $g$ is convex, the one-dimensional minimization problem is also convex.

2. Limited minimization

$$\alpha^{(t)} \equiv \argmin_{\alpha \in [0, s]} g(x^{(t)} - \alpha \dot{g}(x^{(t)}))$$

3. Constant stepsize $\alpha^{(t)} = s > 0$ for all $t$

simple rule but may not converge if it is too large or may converge too slow bc it is too small.

4. Diminishing step size

$\alpha^{(t)} \xrightarrow{} 0$ and $\sum_{t=1}^\infty \alpha^{(t)} = \infty$

For example, $\alpha^{(t)} = \frac{1}{t}$

Descent not guaranteed at each step, only later when $\alpha^{(t)}$ becomes small

$\sum_{t=1}^\infty \alpha^{(t)} = \infty$ imposed to guarantee progress, does not become too slow.

Remarks: Good theoretical guarantees, but unless the right sequence is chosen, can also be a slow method.

5. Successive step size reduction

Well known examples are Armijo rule (i.e. backtracking) and Goldstein rule

**Example:** For exact line search, we want to apply the method of steepest descent to the function $f(x,y) = 4x^2 - 4xy + 2y^2$ with initial point $x_0 = (2,3)$

Remember, we are trying to find $\alpha^{(t)}$ st
$$\alpha^{(t)} \equiv \argmin_{\alpha \geq 0} g(x^{(t)} - \alpha^{(t)} \dot{g}(x^{(t)}))$$

Gradient of $f$ is $\nabla f(x,y) = [\frac{\delta f}{\delta x}, \frac{\delta f}{\delta y}]^\prime = [8x-4y , 4y-4x]^\prime$, which is the steepest descent direction.

Starting at $x^{(0)} = (2,3)^\prime$, we ahve $\nabla f(x^{(0)}) = \nabla f(2,3) = (4,4)^\prime$

Choose $- \nabla f(x^{(0)}) = (-4, -4)^\prime$ as the direction of steepest descent.

Perform line search along $(-4,4)^\prime$ direction.

$$\begin{align*}
\phi(\alpha^{(0)}) &= f( (2,3) - \alpha^{(0)} \nabla f(2,3) )\\
&= f( x^{(0)} - \alpha^{(0)} \nabla f(x^{(0)}))\\
&= f(2 - 4 \alpha^{(0)}, 3 - 4 \alpha^{(0)})\\
\end{align*}$$

Next step is to

$$\begin{align*}
\min_{\alpha^{(0)}} \phi(x^{(0)})\\
\min_{\alpha^{(0)}} f(2 - 4 \alpha^{(0)}, 3 - 4 \alpha^{(0)})\\
\min_{\alpha^{(0)}} (32\alpha^{(0)2} - 32 \alpha^{(0)} + 10)\\
\end{align*}$$

This strictly convex function has a strict global minimum when $\phi^\prime(\alpha) = 0$ at $\alpha^{(0)} = \frac{1}{2}$.

Set $x^{(1)} = x^{(0)} - \alpha^{(0)} \nabla f(x^{(0)}) = (2,3) - \frac{1}{2} (4,4) = (0,1)$

Continuing the process, we have $\nabla f(x^{(1)}) = \nabla f(0,1) = (-4, 4)$ and by defining $\phi(x^{(1)}) = f(x^{(1)} - \alpha^{(1)} \nabla f(x^{(1)})) = f((0,1) - \alpha^{(1)} (-4, 4)) = f(-4\alpha^{(1)}, 1 - 4\alpha^{(1)})$.

Proceed in a similar manner as before to get $\alpha^{(1)} = \frac{1}{10}$.

Set $x^{(2)} = x^{(1)} - \alpha^{(1)} \nabla f(x^{(1)}) = (0, 1) - \frac{1}{10} (-4, 4) = (\frac{2}{5}, \frac{3}{5})$.

Repeat this process until solution.

We can see that the method of steepest descent produces a sequence of iterates $x^{(t)}$ that is converging to the strict global minimizer of $f(x,y)$ at $x^\star = (0,0)$

```python
# Code this
```

### Theorems

The following theorems describe some important properties of the method of steepest descent.

**Theorem 1:** Let $f: \mathbf{R}^n \xrightarrow{} \mathbf{R}$ be continuously differentiable on $\mathbf{R}^n$ and let $x_0 \in D$. Let $\alpha^\star > 0$ be the minimizer of $\phi(\alpha) = f(x_0 - \alpha \nabla f(x_0)), \alpha \geq 0$ and let $x_1 = x_0 - \alpha^\star \nabla f(x_0)$. Then, $f(x_1) < f(x_0)$.

In layman's terms: *Therefore, if you are heading in the direction of steepest descent, then you are headed in the right direction.*

**Theorem 2:** Let $f: \mathbf{R}^n \xrightarrow{} \mathbf{R}$ be continuously differentiable in $\mathbf{R}^n$ and let $x_k$ and $x_{k+1}$, for $k \geq 0$ be two consecutive iterates produced by the method of steepest descent. Then, the steepest directions from $x_k$ and $x_{k+1}$ are orthogonal, that is $\nabla f(x_k) \times \nabla f(x_{k+1}) = 0$.

**Theorem 3:** Let $f: \mathbf{R}^n \xrightarrow{} \mathbf{R}$ be a coercive function with continuous partial derivatives on $\mathbf{R}^n$. Then, for any initial guess $x_0$ the sequence of iterates produced by the method of steepest descent from $x_0$ contains a subsequence that converges to a critical point of $f$.

This result can be proved by applying the Bolzano-Weierstranss theorem.


```
%matplotlib inline
from sympy import *
init_printing()
```

```
x = symbols('x')
f = sqrt(1+x) - sqrt(1-x)
f
```

```
f_strich = simplify(diff(f, x))
f_strich
```

```
krel = Abs(expand(f_strich * x / f))
krel
```

    /usr/lib/python3.3/site-packages/IPython/core/formatters.py:239: FormatterWarning: Exception in image/png formatter:
    \left\lvert{\frac{x}{2 x + 2 \sqrt{- x + 1} \sqrt{x + 1} - 2} + \frac{x}{2 x - 2 \sqrt{- x + 1} \sqrt{x + 1} + 2}}\right\rvert
         ^
    Expected a delimiter (at char 5), (line:1, col:6)
      FormatterWarning,

$$\left\lvert{\frac{x}{2 x + 2 \sqrt{- x + 1} \sqrt{x + 1} - 2} + \frac{x}{2 x - 2 \sqrt{- x + 1} \sqrt{x + 1} + 2}}\right\rvert$$

```
krel_daniel = x**2 / 2 * 1 / (sqrt(1-x**2)-(1-x**2))
krel_daniel
```

```
simplify(krel_daniel-krel)
```

    /usr/lib/python3.3/site-packages/IPython/core/formatters.py:239: FormatterWarning: Exception in image/png formatter:
    \frac{1}{x^{2} + \sqrt{- x^{2} + 1} - 1} \left(x^{2} / 2 - \left(x^{2} + \sqrt{- x^{2} + 1} - 1\right) \left\lvert{\frac{x^{2}}{\left(x - \sqrt{- x + 1} \sqrt{x + 1} + 1\right) \left(x + \sqrt{- x + 1} \sqrt{x + 1} - 1\right)}}\right\rvert\right)
                                                                                ^
    Expected a delimiter (at char 108), (line:1, col:109)
      FormatterWarning,

$$\frac{1}{x^{2} + \sqrt{- x^{2} + 1} - 1} \left(\frac{x^{2}}{2} - \left(x^{2} + \sqrt{- x^{2} + 1} - 1\right) \left\lvert{\frac{x^{2}}{\left(x - \sqrt{- x + 1} \sqrt{x + 1} + 1\right) \left(x + \sqrt{- x + 1} \sqrt{x + 1} - 1\right)}}\right\rvert\right)$$

```
plot(krel, (x, -1/2, 1/2))
```

```
plot(krel_daniel, (x, -1/2, 1/2))
```

```

```


# Kernel Design

It's easy to make new kernels in GPflow. To demonstrate, we'll have a look at the Brownian motion kernel, whose function is
\begin{equation}
k(x, x') = \sigma^2 \text{min}(x, x')
\end{equation}
where $\sigma^2$ is a variance parameter.

```python
import gpflow
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from gpflow.utilities import print_summary, positive

plt.style.use("ggplot")
%matplotlib inline
```

To make this new kernel class, we inherit from the base class `gpflow.kernels.Kernel` and implement the three functions below. **NOTE:** Depending on the kernel to be implemented, other classes can be more adequate. For example, if the kernel to be implemented is isotropic stationary, you can immediately subclass `gpflow.kernels.IsotropicStationary` (at which point you
only have to override `K_r` or `K_r2`; see the `IsotropicStationary` class docstring). Stationary but anisotropic kernels should subclass `gpflow.kernels.AnisotropicStationary` and override `K_d`.

#### `__init__`
In this simple example, the constructor takes no argument (though it could, if that was convenient, for example to pass in an initial value for `variance`). It *must* call the constructor of the superclass with appropriate arguments. Brownian motion is only defined in one dimension, and we'll assume that the `active_dims` are `[0]`, for simplicity.

We've added a parameter to the kernel using the `Parameter` class. Using this class lets the parameter be used in computing the kernel function, and it will automatically be recognised for optimization (or MCMC). Here, the variance parameter is initialized at 1, and constrained to be positive.

#### `K`
This is where you implement the kernel function itself. This takes two arguments, `X` and `X2`. By convention, we make the second argument optional (it defaults to `None`).

Inside `K`, all the computation must be done with TensorFlow - here we've used `tf.minimum`. When GPflow executes the `K` function, `X` and `X2` will be TensorFlow tensors, and parameters such as `self.variance` behave like TensorFlow tensors as well.

#### `K_diag`
This convenience function allows GPflow to save memory at predict time. It's simply the diagonal of the `K` function, in the case where `X2` is `None`. It must return a one-dimensional vector, so we use TensorFlow's reshape command.

```python
class Brownian(gpflow.kernels.Kernel):
    def __init__(self):
        super().__init__(active_dims=[0])
        self.variance = gpflow.Parameter(1.0, transform=positive())

    def K(self, X, X2=None):
        if X2 is None:
            X2 = X
        return self.variance * tf.minimum(X, tf.transpose(X2))  # this returns a 2D tensor

    def K_diag(self, X):
        return self.variance * tf.reshape(X, (-1,))  # this returns a 1D tensor

k_brownian = Brownian()
print_summary(k_brownian, fmt="notebook")
```

name             	class    	transform  	prior  	trainable  	shape  	dtype  	  value

Brownian.variance	Parameter	Softplus   	       	True       	()     	float64	      1

We can now evaluate our new kernel function and draw samples from a Gaussian process with this covariance:

```python
np.random.seed(23)  # for reproducibility

def plotkernelsample(k, ax, xmin=0, xmax=3):
    xx = np.linspace(xmin, xmax, 300)[:, None]
    K = k(xx)
    ax.plot(xx, np.random.multivariate_normal(np.zeros(300), K, 5).T)
    ax.set_title("Samples " + k.__class__.__name__)

def plotkernelfunction(k, ax, xmin=0, xmax=3, other=0):
    xx = np.linspace(xmin, xmax, 100)[:, None]
    ax.plot(xx, k(xx, np.zeros((1, 1)) + other))
    ax.set_title(k.__class__.__name__ + " k(x, %.1f)" % other)

f, axes = plt.subplots(1, 2, figsize=(12, 4), sharex=True)
plotkernelfunction(k_brownian, axes[0], other=2.0)
plotkernelsample(k_brownian, axes[1])
```

## Using the kernel in a model

Because we've inherited from the `Kernel` base class, this new kernel has all the properties needed to be used in GPflow. It also has some convenience features such as allowing the user to call

`k(X, X2)`

which computes the kernel matrix.

To show that this kernel works, let's use it inside GP regression. We'll see that Brownian motion has quite interesting properties. To add a little flexibility, we'll add a `Constant` kernel to our `Brownian` kernel, and the `GPR` class will handle the noise.

```python
np.random.seed(42)
X = np.random.rand(5, 1)
Y = np.sin(X * 6) + np.random.randn(*X.shape) * 0.001

k1 = Brownian()
k2 = gpflow.kernels.Constant()
k = k1 + k2

m = gpflow.models.GPR((X, Y), kernel=k)
# m.likelihood.variance.assign(1e-6)

opt = gpflow.optimizers.Scipy()
opt.minimize(m.training_loss, variables=m.trainable_variables)
print_summary(m, fmt="notebook")

xx = np.linspace(0, 1.1, 100).reshape(100, 1)
mean, var = m.predict_y(xx)
plt.plot(X, Y, "kx", mew=2)
(line,) = plt.plot(xx, mean, lw=2)
_ = plt.fill_between(
    xx[:, 0],
    mean[:, 0] - 2 * np.sqrt(var[:, 0]),
    mean[:, 0] + 2 * np.sqrt(var[:, 0]),
    color=line.get_color(),
    alpha=0.2,
)
```

## See also

For more details on how to manipulate existing kernels (or the one you just created!), we refer to the [Manipulating kernels](../advanced/kernels.ipynb) notebook.


# Equation of motion

```python
from sympy import *
import numpy as np
import cloudpickle

from scipy.integrate import odeint, ode
import matplotlib.pyplot as plt
from matplotlib import animation, rc
from IPython.display import HTML
from utils import swing_animation
```

```python
# init_printing()
```

```python
# macro
frac = Rational
```

```python
# unit vector
i = Matrix([1, 0])
j = Matrix([0, 1])
```

```python
join = lambda l: Matrix([x for s in l for x in s])
```

```python
q = Matrix(var('q1:4'))
dq = Matrix(var('dq1:4'))
ddq= Matrix(var('ddq1:4'))

z = q.col_join(dq)
dz= dq.col_join(ddq)

u = Matrix(var('u1:3'))

m = Matrix(var('m1:4'))
I = Matrix(var('I1:4'))
l = Matrix(var('l0:4'))
lc = Matrix(var('lc1:4'))
d = Matrix(var('d1_2:4'))
g = var('g')
constraints = Matrix(var('k c w q_max'))
parameters = join([m, I, l, lc, d, [g]])
# parameters = m.col_join(I).col_join(l).col_join(lc).col_join(d).col_join(Matrix([g]))
```

```python
e0 = cos(q1) * (-j) + sin(q1) * (-i)
e1 = cos(q1) * (-i) + sin(q1) * ( j)
e2 = cos(q1+q2) * (-j) + sin(q1+q2) * (-i)
e3 = cos(q1+q3) * ( j) + sin(q1+q3) * ( i)
```

```python
# position of joints
p1 = l0 * e0
p2 = p1 + d1_2 *  e1
p3 = p1 + d1_3 *(-e1)
p21= p2 + l2 * e2
p31= p3 + l3 * e3
```

```python
# postion of mass center
pc1 = p1 + (d1_2-lc1) * e1
pc2 = p2 + lc2 * e2
pc3 = p3 + lc3 * e3

pc = pc1.row_join(pc2).row_join(pc3)
```

```python
# mass center velocity
dpc1 = pc1.jacobian(z) * dz
dpc2 = pc2.jacobian(z) * dz
dpc3 = pc3.jacobian(z) * dz

dpc = dpc1.row_join(dpc2).row_join(dpc3)
```

```python
# Energy
T1 = simplify(frac(1, 2) * I1 * dq1 * dq1     + frac(1, 2) * m1 * dpc1.dot(dpc1))
T2 = simplify(frac(1, 2) * I2 *(dq1 + dq2)**2 + frac(1, 2) * m2 * dpc2.dot(dpc2))
T3 = simplify(frac(1, 2) * I3 *(dq1 + dq3)**2 + frac(1, 2) * m3 * dpc3.dot(dpc3))
T = Matrix([T1 + T2 + T3])
```

```python
U = Matrix([m1 * g * pc1[1] + m2 * g * pc2[1] + m3 * g * pc3[1]])
```

```python
tmp = T.jacobian(dq)
H = tmp.jacobian(dq)
C = tmp.jacobian(q) * dq - T.jacobian(q).T
G = U.jacobian(q).T
```

```python
# two actuator
B = Matrix(np.vstack([np.zeros(2), np.eye(2)]))
```

```python
ddq = H.LUsolve(-C - G + B * u)
# ddq = H.inv() * (-C - G - B * u)
# Don't do that. The computation expense is much higher by a few level!!!!!!!!!!
ddq_free = H.LUsolve(B * u)
```

```python
ddq_np = lambdify([z, u, parameters], ddq, 'numpy')
```

```python
# parameters.T
```

```python
# kinematics
p1_np = lambdify([z, parameters], p1)
p2_np = lambdify([z, parameters], p2)
p3_np = lambdify([z, parameters], p3)
p21_np = lambdify([z, parameters], p21)
p31_np = lambdify([z, parameters], p31)
```

# Simulation

```python
# parameters.T
```

```python
param0 = np.array([1, 1, 1,
                   1/12, 1/12, 1/12,
                   1.5, 1, 1, 1,
                   0.5, 0.5, 0.5,
                   0.5, 0.5,
                   9.8])
```

```python
def f(y, t, u, param, constraints=None):
    if constraints is not None:
        ddq = ddq_np(y, u, param, constraints).squeeze()
    else:
        ddq = ddq_np(y, u, param).squeeze()
    return np.r_[y[3:], ddq]
```

```python
t = np.arange(0, 10, 0.01)
y0 = np.array([np.deg2rad(30), 0, 0, 0, 0, 0])
u0 = np.array([0, 0])
sol = odeint(f, y0, t, args=(u0, param0))
```

```python
plt.plot(t, sol[:, 0])
plt.show()
```

```python
anim = swing_animation(sol, param0)
HTML(anim.to_html5_video())
```

# Comparing odeint with ode
-------
ODE is slightly faster than odeint. The error between ode45 and odeint is **trivial**.

```python
## Energy
E_np = lambdify([z, parameters], U+T)
```

```python
E1 = np.array([E_np(sol[i, :], param0) for i in range(len(sol))])
E2 = np.array([E_np(sol2[i, :], param0) for i in range(len(sol))])
```

# Constarint

```python
fc = Matrix([0,
                        ( k * (q_max-q2) - c * dq2) / (exp((q_max-q2) / w) + 1) +
                        (-k * (q_max+q2) - c * dq2) / (exp((q_max+q2) / w) + 1),
                        ( k * (q_max-q3) - c * dq3) / (exp((q_max-q3) / w) + 1) +
                        (-k * (q_max+q3) - c * dq3) / (exp((q_max+q3) / w) + 1)
                       ])
```

```python
con0 = np.array([100, 10, 0.01, np.deg2rad(80)])
```

```python
# # test constraint force
fc_np = lambdify([z, parameters, constraints], fc)
# fc_lambda = lambda x: fc_np(np.array([0, x, 0, 0, 0, 0]), param0, con0)
# res = np.array([fc_lambda(x) for x in np.linspace(-2, 2, 100)]).squeeze()
# plt.plot(np.linspace(-2, 2, 100), res[:, 1])
# plt.show()
```

```python
ddq = H.LUsolve(-C - G + B * u + fc)
ddq_np = lambdify([z, u, parameters, constraints], ddq)
```

```python
# constraints.T
```

```python
# con0 = np.array([100, 10, 0.01, np.deg2rad(80)])
y0 = np.array([np.deg2rad(60), 0, 0, 1, 2, 2])
u0 = np.array([0, 0])
sol = odeint(f, y0, t, args=(u0, param0, con0))
```

```python
fc_ = np.array([fc_np(sol[i, :], param0, con0) for i in range(len(sol))])
```

```python
# F_np = lambdify([z, parameters], G + C)
# [F_np(sol[i, :], param0)[2] for i in range(len(sol))]
```

```python
fig = plt.figure()
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)
ax1.plot(sol[:, 2])
ax1.plot([0, 1000], [ np.pi/2, np.pi/2], 'r--')
ax1.plot([0, 1000], [-np.pi/2,-np.pi/2], 'r--')
ax2.plot(fc_[:, 1])
plt.show()
```

```python
# anim = swing_animation(sol, param0)
# HTML(anim.to_html5_video())
```

# Open loop control

```python
ddq1 = -(H[0, 1] * u1 + H[0, 2] * u2 + C[0, 0] + G[0, 0]) / H[0, 0]
```

```python
ddq = Matrix([ddq1, u1, u2])# + H.LUsolve(fc)
```

```python
ddq_np = lambdify([z, u, parameters, constraints], ddq)
```

```python
# u0 = np.array([0, 1])
# y0 = np.array([np.pi/3, 0, 0, 0, 0, 0])
# ddq_np(y0, u0, param0, con0)
```

```python
from swing_config import *
```

```python
amp = np.ones(2) * amp_max * 0.7
omega = np.ones(2) * ome_max * 0.5
phi = np.ones(2) * phi_max / 4
a_max = np.ones(2) * 100
q_max = np.ones(2) * con0[-1]
# amp * np.sin(omega * np.pi/3 + phi)
```

```python
def fv_gen(amp, omega, phi, q_max):
    return lambda t, y: amp * np.sin(omega * t + phi) / (1 + np.exp((np.abs(y[1:3])-q_max) / 0.01) * np.logical_or(np.abs(y[1:3]) < q_max, y[1:3] * y[4:] > 0))
```

```python
fv = fv_gen(amp, omega, phi, q_max)
# fv(0.1, y0)
```

```python
def f_open_loop(y, t_, param, con, a_max, fv, dt=0.01):
    v = y[4:]
    vt= fv(t_, y)
    a = np.clip((vt - v)/dt, -a_max, a_max)
    ddq_ = ddq_np(y, a, param, con).squeeze()
    return np.r_[y[3:], ddq_]
```

```python
t0 = 0
t1 = 10
dt = 0.01
t = np.arange(t0, t1, dt)
q0 = np.array([np.pi/6, 0, 0])
a0 = np.array([0, 0])
v0 = fv(t0, np.r_[q0, np.zeros(3)])
y0 = np.r_[q0, 0, v0]
ddq_np(y0, a0, param0, con0)
sol = odeint(f_open_loop, y0, t, args=(param0, con0, a_max, fv, dt))
```

```python
fig = swing_plot(t, sol)
plt.show(fig)
if np.max(np.abs(sol[:, 0])) > np.pi / 2:
    print('succeed')
else:
    print('fail')
```

```python
anim = swing_animation(sol, param0)
```

```python
HTML(anim.to_html5_video())
```

```python
def open_loop_test(amp, omega, phi):
    amp = np.array([1, 1]) * 2 * np.pi *amp
    omega = np.array([1, 1]) * 10 * omega
    phi = np.array([np.pi*2, np.pi*2]) * phi
    fv = fv_gen(amp, omega, phi, q_max)
    t0 = 0
    t1 = 10
    dt = 0.01
    t = np.arange(t0, t1, dt)
    q0 = np.array([np.pi/6, 0, 0])
    a0 = np.array([0, 0])
    v0 = fv(t0, np.r_[q0, np.zeros(3)])
    y0 = np.r_[q0, 0, v0]
    ddq_np(y0, a0, param0, con0)
    sol = odeint(f, y0, t, args=(param0, con0, a_max, dt))
    return sol
```

# Utils

```python
def kinematic(y, param):
    p1_ = p1_np(y ,param).squeeze()
    p2_ = p2_np(y, param).squeeze()
    p3_ = p3_np(y, param).squeeze()
    p21_= p21_np(y, param).squeeze()
    p31_= p31_np(y, param).squeeze()
    return (p1_, p2_, p3_, p21_, p31_)
```

```python
def swing_plot(t, sol):
    fig, axes = plt.subplots(2, 3, figsize=(9, 7))
    axes = axes.ravel()
    for i, ax in enumerate(axes):
        ax.plot(t, sol[:, i])
        if i < 3:
            ax.plot([t[0], t[-1]], [ np.pi/2, np.pi/2], 'r--')
            ax.plot([t[0], t[-1]], [-np.pi/2,-np.pi/2], 'r--')
            ax.set_title('$q_{}$'.format(i+1))
        else:
            ax.set_title('$\dot{q}_' + '{}$'.format(i-2))
    return fig

```

```python
def swing_animation(sol, param0):
    fig, ax = plt.subplots(figsize=(6, 6))
    plt.axis('equal')
    plt.axis([-3, 3, -3, 3])

    line1, = ax.plot([], [], lw=1, color='k', linestyle='-', marker='o', ms=5)
    line2, = ax.plot([], [], lw=2, color='b', linestyle='-', marker='o', ms=3)

    def init():
        line1.set_data([], [])
        line2.set_data([], [])
        return (line1, line2)

    def animate(i):
        y = sol[i, :]
        p1, p2, p3, p21, p31 = kinematic(sol[i, :], param0)
        line1.set_data([0, p1[0]], [0, p1[1]])
        line2.set_data([p21[0], p2[0], p3[0], p31[0]], [p21[1], p2[1], p3[1], p31[1]])
        return (line1,line2)
    # call the animator. blit=True means only re-draw the parts that have changed.
    anim = animation.FuncAnimation(fig, animate, init_func=init,
                                   frames=1000, interval=10, blit=True)
    return anim
```

# Serialization

```python
#cloudpickle.dump(f, open('swing_dynamic.dll', 'wb'))
cloudpickle.dump(f_open_loop, open('swing_open_loop_dynamic.dll', 'wb'))
cloudpickle.dump(kinematics, open('swing_kinematic.dll', 'wb'))
```

```python
from collections import namedtuple
```

```python
Solution = namedtuple('Solution', ['t', 'response', 'parameters'])
```

```python
sol1 = Solution(t, sol, param0)
```

```python
cloudpickle.dump(sol1, open('./sol_eg.pkl', 'wb'))
```


```python
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import numpy as np
import sympy as sym
```

```python
def f(X, n, q, b, c):
    Z = np.zeros(len(X))
    for i in range(len(X)):
        for j in range(int(n)):
            for k in range(int(n)):
                Z[i] += q[j][k]*X[i][j]*X[i][k]

        for j in range(int(n)):
            Z[i] += b[j]*X[i][j]
    return Z
```

```python
def f_mesh(X, Y, q, b, c):
    Z = np.zeros(len(X))
    Z = q[0][0]*X*X + q[0][1]*X*Y + q[1][0]*Y*X + q[1][1]*Y*Y + b[0]*X + b[1]*Y + c
    return Z
```

```python
def f2(X, Y, n, q, b, c):
    Z = q[0][0]*X*X + q[0][1]*X*Y + q[1][0]*Y*X + q[1][1]*Y*Y + b[0]*X + b[1]*Y + int(c[0])
    return Z
```

```python
def gd_with_momentum(X_new, X1, Y1, Z1, n, q, b, c, alpha=0.10, beta=0.8, precision=0.0001, max_iter=200):
    X_old = np.zeros((1, 2))
    X_new = np.zeros((1, 2))
    dfr = np.zeros((1, 2))

    X_new[0][0] = 4.9
    X_new[0][1] = 4.9
    i = 0
    Xs = np.zeros((max_iter, 2))
    Ys = np.zeros(max_iter)

    V = np.zeros((max_iter + 1, 2))

    x, y = sym.symbols('x y')
    df1 = sym.diff(f2(x, y, n, q, b, c), x)
    df2 = sym.diff(f2(x, y, n, q, b, c), y)

    while np.all(abs(X_new - X_old)) > precision and max_iter > i:
        Xs[i] = X_new
        Ys[i] = f2 (X_new[0][0], X_new[0][1], n, q, b, c)

        X_old = X_new

        dfr[0][0] = df1.evalf(subs={x: X_old[0][0], y: X_old[0][1]})
        dfr[0][1] = df2.evalf(subs={x: X_old[0][0], y: X_old[0][1]})

        i += 1
        V[i] = beta * V[i - 1] + (1 - beta) * dfr
        X_new = X_new - alpha * V[i]
        # print("V: {}, dfr: {}, X_new: {}".format(V[i], dfr, X_new))

        alpha *= 0.99

    print("Finished with {} step".format(i))
    if (i < max_iter):
        Xs[i] = X_new
        Ys[i] = f2(X_new[0][0], X_new[0][1], n, q, b, c)

        for j in range(max_iter - 1, i, -1):
            Xs = np.delete(Xs, j, axis=0)
            Ys = np.delete(Ys, j, axis=0)
    return Xs, Ys

```

```python
X1 = np.arange(-5, 5, 0.1)
Y1 = np.arange(-5, 5, 0.1)
Z1 = np.zeros(len(X1))

X_new = np.zeros((100,2))

for i in range(len(X1)):
    X_new[i][0] = X1[i]
    X_new[i][1] = Y1[i]

Z1 = f(X_new, n, q, b, c)

x_list, y_list = gd_with_momentum(X_new, X1, Y1, Z1, n, q, b, c)
```

    Finished with 200 step

```python
n = input("Enter power of your function: ")
q = []
for i in range(int(n)):
    q.append(input("Enter the function's coefficient matrix q's row q[{}]: ".format(i)).split())
b = input("Enter the function's coefficient b matrix: ").split()
c = input("Enter the function's constant c: ")

for i in range(int(n)):
    q[i] = list(map(float, q[i]))
b = list(map(float, b))
c = list(map(float, c))
```

    Enter power of your function: 2
    Enter the function's coefficient matrix q's row q[0]: 1 0
    Enter the function's coefficient matrix q's row q[1]: 0 2
    Enter the function's coefficient b matrix: 0 0
    Enter the function's constant c: 0

```python
X1, Y1 = np.meshgrid(X1, Y1)
Z1 = f_mesh(X1, Y1, q, b, c)

X, Y = zip(*x_list)
Z = y_list

ax = plt.subplots(nrows=1, ncols=1, figsize=(10,10))
cs = plt.contour(X1, Y1, Z1)
plt.clabel(cs, inline=1, fontsize=10)
colors = ['b', 'g', 'm', 'c', 'orange']
for j in range(1, len(X)):
    ax[1].annotate('', xy=(X[j], Y[j]), xytext=(X[j-1], Y[j-1]),
                arrowprops={'arrowstyle': '->', 'color': 'r', 'lw': 1},
                va='center', ha='center')
ax[1].scatter(X, Y, s=40, lw=0)
ax[1].set_xlabel('X')
ax[1].set_ylabel('Y')
ax[1].set_title('Minimizing function')
```

```python
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(X1, Y1, Z1, rstride=1, cstride=1,
                cmap='viridis', edgecolor='none')
ax.set_title('surface')
plt.show()
```

```python
plt.figure()
cs = plt.contour(X1, Y1, Z1)
plt.clabel(cs, inline=1, fontsize=10)
plt.title('Two-dimensional contour plot')
plt.show()
```

```python

```


```python
import sympy as sp
from sympy import Rational
from sympy import pprint

```

```python
Mc, Mm, Rp, rh_c, rh_m, Vc, Vm, Mp = sp.symbols('Mc, Mm, Rp, rh_c, rh_m, Vc, Vm, Mp')
```

```python
MOI_c = Rational(2,5) * Mc * (Rational(1,2) * Rp) ** 2
MOI_m = Rational(2,5) * Mm * (Rp **2 - (Rational(1,2) * Rp) ** 2)
```

```python
MOI = MOI_c + MOI_m

pprint(MOI) # Verified
```

         2          2
    Mc⋅Rp    3⋅Mm⋅Rp
    ────── + ────────
      10        10

```python
# Substitute mass by rho*V
MOI = MOI.subs([
    (Mm, rh_m * Vm),
    (Mc, rh_c * Vc)
])

# Mantle volume according to V_sphere = (4/3) pi * r ** 3
Vm_val = Rational(4,3) * sp.pi * ( Rp ** 3 - (Rational(1,2) * Rp) ** 3)

# Core volume according to V_sphere = (4/3) pi * r ** 3
Vc_val = Rational(4,3) * sp.pi * (Rational(1,2) * Rp) ** 3

# Substitute the volumes by their symbolic expressions.
MOI = MOI.subs([
    (Vm, Vm_val),
    (Vc, Vc_val)
])

# Substitute rho_c by 4* rho_m for MOI in terms of rho_m
MOI_rhm = MOI.subs([
    (rh_c, 4 * rh_m)
])

# MOI in terms of rh_c
MOI_rhc = MOI.subs([(rh_m, Rational(1,4) * rh_c)])

# Expression for MOI
pprint(MOI_rhm)
pprint(MOI_rhc)
```

        5
    π⋅Rp ⋅rhₘ
    ─────────
        15
        5
    π⋅Rp ⋅rh_c
    ──────────
        60

```python
# Summation of both mass contributions.
M = Mm + Mc

# Replace Mass by rho*V
M = M.subs([
    (Mm, rh_m * Vm),
    (Mc, rh_c * Vc)
])

# Replace volume by expressions.
M = M.subs([
    (Vm, Vm_val),
    (Vc, Vc_val)
])

# Put Mass in terms of only rho_c
M = M.subs([(rh_m, Rational(1,4) * rh_c)])

# Put rho_c in temrs of total mass
rh_c_sub = sp.solve(sp.Eq(Mp, M), rh_c)[0]

# Replace MOI expression with rho_c by rho_c found by mass.
Solved = MOI_rhc.subs([
    (rh_c, rh_c_sub)
])

MOI_p = sp.simplify(Solved)

pprint(MOI_p / (Rational(2,5) * Mp * (Rp**2)))
```

    1/11

## Problem 5

```python
import numpy as np
```

```python
mass = np.array([
    59736,
    893.3,
    0.65,
    214.7
]) * 10 ** 20

radius_surface = np.array([
    6371.0,
    1821.3,
    248.0,
    1352.6
]) * 10 ** 3

G = 6.674 * 10 ** (-11)
```

```python
# print(np)
# V_esc = np.sqrt(np.divide(G*mass,radius_surface))
x = 2* G*mass/ radius_surface
# print(x)
import sympy as sp
vesc = sp.Matrix(np.sqrt(x.astype(float)))

```

```python
g = G * mass / radius_surface / radius_surface

print(g)
```

    [9.822162851846851 1.7972999454236644 0.07053362382934444
     0.7832122663239973]

```python
vesc
```

    Matrix([
    [11187.2248148606],
    [2558.68028116063],
    [187.041913536391],
    [1455.59122794131]])

```python
vesc

```

    Matrix([
    [11187.2248148606],
    [2558.68028116063],
    [187.041913536391],
    [1455.59122794131]])

```python
def v0(h, g):
    inc  = 2 * h * g
    inc = inc.astype(float)
    return np.sqrt(inc)
```

```python
h = np.array([60, 65000, 10E12, 10000])
```

```python
from sympy import pprint
v0 = v0(h, g)
```

```python
vesc = np.array(vesc)
```

```python
vesc = vesc.flatten()
```

```python
v0/vesc
print(v0)
```

    [3.43316114e+01 4.83372520e+02 1.18771734e+06 1.25156883e+02]

```python
alpha = (2 * h / radius_surface + (h**2/radius_surface/radius_surface))
```

```python
print(alpha)
```

    [1.88354364e-05 7.26512774e-02 1.62591059e+15 1.48409964e-02]

```python
np.set_printoptions(precision=4)
```

```python
print(alpha)
```

    [1.8835e-05 7.2651e-02 1.6259e+15 1.4841e-02]

```python

```


## Harmonic Oscillator Basis Set

Solving the Schrödinger equation with a potential energy function defined as a quantum Harmonic Oscillator yields a set of wavefunctions that can be used as basis set for quantum mechanical calculations. The Schrödinger equation with the Harmonic Oscillator is defined as follows:

$$\begin{align}
    \hat{H}\Psi &= E\Psi &&\mbox{Schrödinger Equation}\\
    \\
    \hat{H} &= \hat{T} + \hat{V} &&\mbox{Hamiltonian Operator Expansion}\\
    \\
    \hat{T} &= -\frac{\hbar^2}{2m}\nabla^2 &&\mbox{Kinetic Energy Operator} \\
    \hat{T} &= -\frac{\hbar^2}{2m} \frac{\partial^2}{dr^2} &&\mbox{Simplifying KE Operator to 1D space} \\
    \\
    \hat{V} &= \frac{1}{2}kr^2 &&\mbox{Potential Energy Operator = Harmonic Oscillator Potential} \\
    \hat{H} &= -\frac{\hbar^2}{2m}\frac{\partial^2}{\partial r^2} + \frac{1}{2}kr^2
\end{align}$$

The wavefunctions produced are:

$$\begin{align}
    \psi_0 &= c_0 \cdot e^{-\frac{\alpha x^2}{2}} \\
    \psi_1 &= c_0 \cdot \sqrt{2\alpha}y \cdot e^{-\frac{\alpha x^2}{2}} \\
    \psi_2 &= c_0 \cdot \frac{2y^2 - 1}{\sqrt{2}} \cdot e^{-\frac{\alpha x^2}{2}} \\
    \psi_3 &= c_0 \cdot \frac{2y^3 - 3y}{\sqrt{3}} \cdot e^{-\frac{\alpha x^2}{2}} \\
    \vdots \\
    \psi_n &= c_0 \cdot H_n(y) \cdot e^{-\frac{\alpha x^2}{2}} \\
    \\
    c_0 &= \left[\frac{\alpha}{\pi}\right]^\frac{1}{4} \\
    y &= \sqrt{\alpha}x \\
    x &= r - r_e \\
    r &= \mbox{Bond Distance} \\
    r_e &= \mbox{Optimal Internuclear Bond Distance} \\
    \alpha &= \frac{\mu\omega}{\hbar} \\
    \mu &= \hbox{Reduced Atomic Mass} \\
    \omega &= \hbox{First Term Vibrational Constant} \\
    H_n(y) &= \hbox{The nth Hermite Polynomial} \\
    \end{align}$$

The Hermite polynomials are added into the wavefunction to allow $\psi$ to be orthonormal and satisfy the orthonormality condition stated below:
$$\int_{-\infty}^{\infty}{\psi_i \psi_j} = \delta_{i,j}$$

A sample of Hermite polynomials for selected $n$ values are presented below:
$$\begin{matrix}
    n & H_n \\
    0 & 1 \\
    1 & 2y \\
    2 & 4y^2-2 \\
    3 & 8y^3 - 12y
\end{matrix}$$

All information in this notebook originates from the following website, and the Franck-Condon Calculations Mathematica notebook provided by Dr. Jerry LaRue of Chapman University.

```python
#Write out terms for KE, and PE
#V = 0 Wavefunction
#Move to other notebook and make like a worksheet
```

## Unit Analysis

The question of units used for the wavefunction along with its inputs are of the utmost importantance. Many of the more confusing details of the wavefunction implementation are related towards maintaining the correct units for the wavefunction.

To begin with, the overlap integral of two wavefunctions: $\int_{-\infty}^{\infty}{\psi_i\psi_j}=\delta_{ij}$, should result in a unitless quantity of 1 or 0. Unitless normalization of the function can handle ensuring that the output of this integral can only be 0 or 1, but the units for the parameters of the wavefunction must be chosen carefully to allow the integral to be unitless.

The integral will result in units of $r\cdot\psi^2$, normally $r$ will have units of $\mathring A$. Thus, it would imply that $\psi^2$ must have units of $\frac{1}{\mathring A}$, further implying that $\psi$ will have units of $\frac{1}{\sqrt{\mathring A}}$ as:
$$
\begin{align}
    r\cdot &\psi^2 \\
    \mathring A\cdot &\frac{1}{\sqrt{\mathring A}}^2 \\
    \mathring A\cdot &\frac{1}{\mathring A} \\
    &\frac{\mathring A}{\mathring A} \\
    \mbox{Unitless }&\mbox{Quantity}
\end{align}
$$

One method to ensure that the wavefunction will have units of $\frac{1}{\sqrt{\mathring A}}$ is to have the wavefunction be unitless by creating a term that can be multiplied by the quantities having units in the wavefunction equation to cancel out the units. Then, a unit quantity of $\frac{1}{\sqrt{\mathring A}}$ is multiplied by the unitless output of the wavefunction.

Given this unit scheme, the inputs and variables of the wavefunction should have the following units:
$$
\begin{align}
    r &= \mathring A \\
    r_e &= \mathring A \\
    x &= \mathring A \\
    \mu &= \mbox{AMU} \\
    \\
    \omega &= \frac{1}{s} &&\mbox{Needs to be converted from }\frac{1}{cm} \\
    \\
    \omega &= \frac{1}{cm} \\
    \omega &= \frac{1}{cm} \cdot 100 \\
    \omega &= \frac{1}{m} \cdot c\left[\mbox{Speed of Light}\right] \\
    \omega &= \frac{1}{m} \cdot \frac{m}{s} \\
    \omega &= \frac{1}{s} \\
\end{align}
$$

$\hbar$ normally has units of $\frac{kg m^2}{s}$, but in order to properly calculate $\alpha$, the units of $\hbar$ will be converted to $\frac{AMU \mathring A^2}{s}$.
$$
\begin{align}
    \hbar &= \frac{kg m^2}{s} \\
    \hbar &= \frac{kg m^2}{s} \cdot \left( 6.022 \cdot 10^{26} \frac{\mbox{AMU}}{kg} \right) \\
    \hbar &= \frac{\mbox{AMU} m^2}{s} \cdot \left(10^{20} \frac{\mathring A^2}{m^2} \right)\\
    \hbar &= \frac{\mbox{AMU} \mathring A^2}{s} \\
    \\
    \alpha &= \frac{\omega\mu}{\hbar} \\
    \alpha &= \frac{\frac{1}{s}\cdot \mbox{AMU}}{ \left(\frac{\mbox{AMU} \mathring A^2}{s} \right)} \\
    \alpha &= \frac{1}{s}\cdot \mbox{AMU} \cdot \left(\frac{s}{\mbox{AMU} \mathring A^2} \right) \\
    \alpha &= \frac{\mbox{AMU}}{s} \cdot \frac{s}{\mbox{AMU} \mathring A^2} \\
    \alpha &= \frac{1}{\mathring A^2}
\end{align}
$$

It should be noted that the central purpose of $\alpha$ is to act as a converter value that makes other values unitless, as well as being used to give the wavefunction final units of $\frac{1}{\sqrt{\mathring A}}$.

$$
\begin{align}
    y &= \sqrt{\alpha}x \\
    y &= \sqrt{\frac{1}{\mathring A^2}} \cdot \mathring A \\
    y &= \frac{1}{\mathring A} \cdot \mathring A \\
    y &= \frac{\mathring A}{\mathring A} \\
    y &= 1 &&y\mbox{ becomes a unitless quantity.}
\end{align}
$$

Below follows a unit analysis of the wavefunction itself taking into consideration the above variable units:
$$
\begin{align}
    \psi_n &= H_n(y) \cdot e^{-\frac{\alpha x^2}{2}} \\
    \psi_n &= H_n(\mbox{Unitless Quantity}) \cdot e^{-\frac{\frac{1}{\mathring A^2}}{\mathring A^2}} \\
    \psi_n &= \left(\mbox{Unitless Quantity}\right) \cdot e^{-\frac{\mathring A^2}{\mathring A^2}} \\
    \psi_n &= \left(\mbox{Unitless Quantity}\right) \cdot e^{\left(\mbox{Unitless Quantity}\right)} \\
    \psi_n &= \left(\mbox{Unitless Quantity}\right) \cdot \left(\mbox{Unitless Quantity}\right) \\
    \psi_n &= \mbox{Unitless Quantity}
\end{align}
$$

As can be seen above, the wavefunction now produces a unitless quantity and all that is left to do is multiply in the unit of $\frac{1}{\sqrt{\mathring A}}$. This is done through $c_0$.
$$
\begin{align}
    c_0 &= \left[\frac{\alpha}{\pi}\right]^\frac{1}{4} \\
    c_0 &= \left[\frac{\frac{1}{\mathring A^2}}{\pi}\right]^\frac{1}{4} \\
    c_0 &= \left[\frac{1}{\mathring A^2}\right]^{\frac{1}{4}} \\
    c_0 &= \frac{1}{\mathring A^{\frac{2}{4}}} \\
    c_0 &= \frac{1}{\mathring A^{\frac{1}{2}}} \\
    c_0 &= \frac{1}{\sqrt{\mathring A}} \\
\end{align}
$$

Thus, $c_0$ has units of $\frac{1}{\sqrt{\mathring A}}$, meaning that $c_0$ is multipled into the wavefunction to allow the unitless wavefunction to have the correct unit, resulting in the final harmonic oscillator wavefunction:
$$\psi_n = c_0 \cdot H_n(y) \cdot e^{-\frac{\alpha x^2}{2}}$$

```python
#Allow Notebook to Import from Comp_Chem_Package
import sys
if("win" in sys.platform):
    sys.path.append("..\\..\\Backend")
else:
    sys.path.append("../../Backend")

from compChemGlobal import *

#Enter Input Data

#Include link to NIST and ask students to find values for a certain diatomic

#Re in units of Angstroms/
#Optimal Diatomic Bond Distance
Re = 1.128323

#w in units of 1/cm
#First Vibrational Constant
w = 2169.81358

#u in units of A
#Reduced Atomic Mass for Diatomic Molecule
u = 12*16/(12+16)

#Anticipate ahead of time?
#How would expect certain things to change the wavefunctions
#W, Re, and u
#Predict changes in input parameters and
#appearances of wavefunctions
#Reverse order of notebook, theory first code second
#Very clear for what I am asking
#Scaffold, start with parameters
#Clear in expectations
#Outcomes and predictiosn
#How developing this code, and all the pourposes
#Calculate Franc-Condon, transitions
#All in q
```

```python
#All conversion factors acquired from Google

#convert w from 1/cm to 1/m
w *= 100

#convert from 1/m to 1/s using the speed of light, m/s
#multiply by 2pi to adjust waveform
w *= c * 2 * pi

#convert hbar to units of AMU Angstrom^2 / s from kg m^2 / s
#convert from kg to AMU
hbar2 = hbar
hbar2 *= 6.022 * pow(10, 26)
#convert from m^2 to angstrom^2
hbar2 *= pow(10, 20)

#compute alpha with units of 1/Angstrom^2
alpha = (u*w) / hbar2
```

```python
#Define all functions here
#########################################################################

#converts r to x, where x = r-Re
#r in Angstroms
#Re in Angstroms
#Returns in units of Angstroms
def rToX(r):
    return r - Re

#########################################################################

#Represents the e^x term in the harmonic oscillator
#r in units of angstrom
#unitless return
def eTerm(r):
    x = rToX(r)
    return exp( -alpha * pow(x,2) / 2 )

#########################################################################

#prepares input to the hermite function
#r in units of angstroms
#unitless return
def hermiteInput(r):
    x = rToX(r)
    return sqrt(alpha) * x

#########################################################################

#Function that creates a new Harmonic Oscillator Wavefunction (HOW)
#Each harmonic oscillator function takes a position r in angstroms as input, and returns
#a value in units of 1/sqrt(angstroms)
def newHOW(n):

    normalization = 1 / superSqrt( pow(2,n) * factorial(n) )
    return lambda r : normalization * pow(alpha/pi, 0.25) * eTerm(r) * hermitePolynomials(
        n, hermiteInput(r))
```

```python
#Graphing Code
import numpy as np

fig = plot.go.FigureWidget()
fig.update_layout(
    title_text = "Harmonic Oscillators Wavefunction",
    xaxis_title_text = "r in Angstroms",
    yaxis_title_text = "Wavefunction Output",
)

fig.update_layout(showlegend=True)
out = widgets.Output()

basis = []
def graph_wavefunctions(resolution, startR, endR, precision, basisSize, visibleWavefunctions, probability):

    visibility = ["legendonly"] * basisSize

    try:
        for startEnd in visibleWavefunctions.split(";"):
            if("-" in startEnd):
                startEnd = [int(value) for value in startEnd.split("-")]
                for index in range(startEnd[0], startEnd[1]):
                    visibility[index] = True
            else:
                visibility[int(startEnd)] = True
    except:
        visibility = [False] * basisSize

    #build the basis set of the specified size if needed
    if(len(basis) != basisSize):
        for n in range(basisSize):
            basis.append( newHOW(n) )

    dr = 1/resolution
    precision = "0." + str(precision)

    r = [startR + (dr*step) for step in range( int(abs( (endR - startR)/dr ))) ]
    y = []

    name = []
    yCalculations = []
    if("Standard" in probability):
        name.append("HOW")
        yCalculations.append(lambda basisFunction, rValue: basisFunction(rValue))
    if("Probability" in probability):
        name.append("HOW Squared")
        yCalculations.append(lambda basisFunction, rValue: basisFunction(rValue) ** 2)

    fig.data = []
    for index, basisFunction in enumerate(basis):
        y.append([])

        for yCalculation in yCalculations:
            y[index].append([])
            for rValue in r:
                y[index][-1].append( yCalculation(basisFunction, rValue) )

        for yIndex, yData in enumerate(y[index]):
            fig.add_trace(plot.go.Scatter(
                name = name[yIndex] + " " + str(index),
                mode = "lines",
                line = {"shape":"spline"},
                x = r, y = yData,

                hovertemplate =
                    "<b>r = %{x:" + precision + "f}</b><br>" +
                    "<b>Ψ(r) = %{y:" + precision + "f}</b>",
                hoverlabel = dict(
                    font_size = 16
                ),

                visible = visibility[index]
            ))
```

```python
#Widgets Code

#code to determine the appropriate start and end values
start = round(-sqrt(-2 * log(pow(10, -15)) / alpha) + Re, 2)
end = round(sqrt(-2 * log(pow(10, -15)) / alpha) + Re, 2)

resolution = widgets.BoundedFloatText(
    value = 100,
    min = .001,
    step = .01,
    max = pow(10, 10),
    description = '<p style="font-family:verdana;font-size:15px">Resolution</p>',
)

precision = widgets.BoundedIntText(
    value = 2,
    min = 0,
    max = 20,
    step = 1,
    description = '<p style="font-family:verdana;font-size:15px">Precision</p>'
)

basisSize = widgets.BoundedIntText(
    value = 5,
    min = 1,
    max = pow(10,10),
    step = 1,
    description = '<p style="font-family:verdana;font-size:15px">Basis Size</p>'
)

visibleWavefunctions = widgets.Text(
    value = "0-" + str(basisSize.value),
    description = '<p style="font-family:verdana;font-size:15px">Visible Ψs</p>'
)

startR = widgets.FloatText(
    value = start,
    step = .1,
    description = '<p style="font-family:verdana;font-size:15px">Start r</p>'
)

endR = widgets.FloatText(
    value = end,
    step = .1,
    description = '<p style="font-family:verdana;font-size:15px">End r</p>'
)

probability = widgets.Dropdown(
    options = ["Standard", "Probability Distribution", "Standard & Probability"],
    description = "Graph Mode"
)

def wavefunctionUpdate(change):
    graph_wavefunctions(resolution.value, startR.value, endR.value, precision.value, basisSize.value, visibleWavefunctions.value, probability.value)

resolution.observe(wavefunctionUpdate, "value")
precision.observe(wavefunctionUpdate, "value")
startR.observe(wavefunctionUpdate, "value")
endR.observe(wavefunctionUpdate, "value")
basisSize.observe(wavefunctionUpdate, "value")
visibleWavefunctions.observe(wavefunctionUpdate, "value")
probability.observe(wavefunctionUpdate, "value")

graph_wavefunctions(resolution.value, startR.value, endR.value, precision.value, basisSize.value, visibleWavefunctions.value, probability.value)

display(widgets.VBox([fig,
             widgets.HBox([resolution, precision, basisSize]),
             widgets.HBox([startR, endR, visibleWavefunctions, probability])]))
display(out)
```

    VBox(children=(FigureWidget({
        'data': [{'hoverlabel': {'font': {'size': 16}},
                  'hovertemplate…

    Output()

```python
import sys
sys.path.append("..\\Comp_Chem_Package")

from how import *
from basisSet import *
from nistScraper import *

dc = getDiatomicConstants("H2")

n1 = how(dc, 20)

bset = basisSet(dc, how, 10)

n1.graph()
bset.graph()
```

    VBox(children=(FigureWidget({
        'data': [{'fill': 'none',
                  'hoverlabel': {'font': {'size': 16}}…

    VBox(children=(FigureWidget({
        'data': [{'fill': 'none',
                  'hoverlabel': {'font': {'size': 16}}…

    <basisSet.basisSet at 0x26e3c6c6b50>

```python
#Building the PES for the Harmonic Oscillator
from howPES import howPES

howP = howPES(dc)
howP.graph()
```

```python
# Solving the Schrodinger Equation for the Harmonic Oscillator
#Based on equations from the following references:
#    https://phys.libretexts.org/Bookshelves/University_Physics/Book%3A_University_Physics_(OpenStax)/Map%3A_University_Physics_III_-_Optics_and_Modern_Physics_(OpenStax)/07%3A_Quantum_Mechanics/7.06%3A_The_Quantum_Harmonic_Oscillator#:~:text=Unlike%20a%20classical%20oscillator%2C%20the,%3D%E2%84%8F%CF%89%3Dhf.
#    http://hyperphysics.phy-astr.gsu.edu/hbase/quantum/hosc.html#c1
from operators import *
from basisSet import *
from diatomicConstants import *
from schrodinger import *
from graphable import *
from nistScraper import *

#Declare all variables here
graph = widgets.VBox([])
solution = [0]

#Declare widgets
mass = widgets.FloatText(
    min = pow(10, -8),
    value = 0.5,
    step = .1,
    description = '<p style="font-family:verdana;font-size:15px">AMU</p>'
)
w = widgets.FloatText(
    min = pow(10, -8),
    value = 2,
    step = 100,
    description = '<p style="font-family:verdana;font-size:15px">ω in 1/cm</p>'
)
basisSize = widgets.IntText(
    min = 1,
    value = 5,
    description = '<p style="font-family:verdana;font-size:15px">Basis Size</p>'
)

dc = buildDiatomicConstants(w=w.value, u=mass.value)
basis = basisSet(dc, size=4)

def buildSolution(graph, returnSolution):

    dc["w"] = w.value
    dc["u"] = mass.value

    basis = basisSet(dc, size=basisSize.value)
    pes = howPES(dc)

    T = TOperator(basis, integrationStart = -inf)
    V = VOperator(basis, pes, integrationStart = -inf)

    H = T + V
    solution = schrod(H, basis, pes)

    returnSolution.pop()
    returnSolution.append(solution)

    #update the graph
    graph.children = solution.graph(getGraph=True).children

#Set up widget observation
observeFunction = lambda value : buildSolution(graph, solution)
mass.observe(observeFunction)
w.observe(observeFunction)
basisSize.observe(observeFunction)

#Graph data
buildSolution(graph, solution)
display(graph)
display(widgets.HBox([mass, w, basisSize]))
```

```python
print(solution[0].eigenVectors[0])
```

```python
basis[0]
```

```python

```


# Prova II- Computação Centífica II
> Autor: Gil Miranda<br>
> Contato: gil.neto@ufrj.br; gilsmneto@gmail.com

## Set-up of dependencies and libraries

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate
```

## The equation to solve

$$
\epsilon \frac{\mathrm{d}^2u}{\mathrm{d}x^2} + 2(1-x^2)u + u^2 = 1
$$

$$
\frac{\mathrm{d}^2u}{\mathrm{d}x^2} = \frac{1 - 2(1-x^2)u - u^2}{\epsilon}
$$

Tomando $\frac{\mathrm{d}^2u}{\mathrm{d}x^2} = f(x)$, e uma solução como $u(x) \approx u_h(x) = \displaystyle\sum^N_{j = 1} c_j \varphi_j(x)$, podemos reescrever o problema em forma de um sistema
$$
A\vec{c} = \vec{f}
$$

Onde
$$
f_i = \int^1_{-1}f(x)\varphi_i \mathrm{d}x
$$
$$
A_{ij} = \int^1_{-1}\frac{\mathrm{d}\varphi_j}{\mathrm{d}x}\frac{\mathrm{d}\varphi_i}{\mathrm{d}x} \mathrm{d}x
$$
e $c_j$ podemos descobrir resolvendo o sistema, daí podemos utilizar a solução aproximada $u_h(x)$

### Calculando a matriz A

Vamos calcular primeiro os termos da matriz A, para isso resolvemos a integral acima, temos:
$$
\varphi_j = sen(j\pi x)\\
\frac{\mathrm{d}\varphi_j}{\mathrm{d}x} = j\pi cos(j \pi x)
$$
Então a integral a ser resolvida é:
$$
\int^1_{-1}(j\pi cos(j \pi x))(i\pi cos(i \pi x)) \mathrm{d}x
$$
E utilizando a identidade: $cos(a)cos(b) = \frac{1}{2}(cos(a+b)+cos(a-b))$, podemos resolver a integral
$$
\begin{align}
\int^1_{-1}\frac{\pi^2 ij}{2} cos(\pi x(j+i)) + cos(\pi x(j-i)) \mathrm{d}x &= \frac{\pi^2 ij}{2} \left(\int^1_{-1} cos(\pi x(j+i))\mathrm{d}x + \int^1_{-1}cos(\pi x(j-i)) \mathrm{d}x \right)\\
&= \frac{\pi^2 ij}{2} \left(\frac{sen(\pi x(j+i))}{\pi(i+j)} + \frac{sen(\pi x(j-i))}{\pi(j-i)}\right)\Big|_{-1}^{1}\\
&= \frac{\pi^2 ij}{2} \left(\frac{sen(\pi(j+i))}{\pi(i+j)} + \frac{sen(\pi(j-i))}{\pi(j-i)} - \frac{sen(-\pi(j+i))}{\pi(i+j)} - \frac{sen(-\pi(j-i))}{\pi(j-i)}\right)\\
&\text{Como $j, i$ são números inteiros, temos que sua soma também é, e como $sen(n\pi) = 0, n \in \mathbb{N}$}\\
&= 0
\end{align}
$$
Portanto a integral é $0$, mas temos uma restrição em $i,j$ por conta do termo $(j-i)$ sendo dividido não podemos ter $j = i$ nesta integral, logo só é válida para $i \neq j$, portanto
$$A_{ij} = 0, \, \forall i \neq j$$
Olhando para o caso $i = j$, temos:
$$
\int^1_{-1}(j\pi cos(j \pi x))(i\pi cos(i \pi x)) \mathrm{d}x = \int^1_{-1}j^2\pi^2 cos^2(j \pi x) \mathrm{d}x
$$
E usando a identidade $cos^2(x) =  \frac{1}{2}(1 + cos(2x)$
$$
\begin{align}
\int^1_{-1}j^2\pi^2 cos^2(j \pi x) \mathrm{d}x &= \int^1_{-1}\frac{j^2\pi^2}{2} (1 + cos(2j \pi x) ) \mathrm{d}x\\
&= \frac{j^2\pi^2}{2}\int^1_{-1} (1 + cos(2j \pi x) ) \mathrm{d}x\\
&= \frac{j^2\pi^2}{2}\left(\int^1_{-1} 1 \mathrm{d}x + \int^1_{-1} cos(2j \pi x) ) \mathrm{d}x\right)\\
&= \frac{j^2\pi^2}{2}\left(x\Big|_{-1}^{1} + \frac{sen(2j \pi x)}{2\pi j}\Big|_{-1}^{1}\right)\\
&= j^2\pi^2
\end{align}
$$
Logo, para os elementos de $A$ com $i = j$, temos $A_{ij} = j^2 \pi^2$

### Defining necessaries

```python
N = 5
xs = np.linspace(-1,1,N)
def base(j, x):
    return np.sin(j*np.pi*x)
```

```python
## Defining the Integrals of the basis function

def F1(j,i,x):
    a = (np.pi*i*j/2)
    b = np.sin(np.pi*x*(i+j))/(i+j)
    c = np.sin(np.pi*x*(j-i))/(j-i)
    return a*(b+c)

def F2(j):
    #a = np.sin(2*np.pi*j*x)/np.pi*j
    return (j**2)*(np.pi**2)
```

```python
def F3(i,x):
    return (i**2)*(np.pi**2)
```

```python
A = np.zeros((N,N))
```

```python
for i in range(N):
    for j in range(N):
        if i == j:
            A[i][j] = F2(i+1)
        else:
            A[i][j] = F1(j+1,i+1,1) - F1(j+1,i+1, -1)
```

```python
A
```

```python

```