# Mish Derivatves

```python
import torch
from torch.nn import functional as F
```

```python
inp = torch.randn(100) + (torch.arange(0, 1000, 10, dtype=torch.float)-500.)
inp
```

    tensor([-500.3069, -490.6361, -480.3858, -471.2755, -459.0872, -451.1570,
            -440.2400, -429.6230, -419.9467, -408.3055, -402.3395, -389.1660,
            -380.1614, -369.7649, -359.7261, -348.8759, -338.7170, -329.2680,
            -319.7470, -309.6079, -301.3083, -290.9236, -279.3832, -267.8622,
            -259.3479, -249.7400, -240.8742, -229.2343, -219.3999, -210.0166,
            -199.8259, -191.5603, -178.9595, -171.4488, -160.3362, -150.1327,
            -139.2230, -130.8046, -121.8909, -108.4913, -100.5724,  -88.9087,
             -79.9365,  -70.3478,  -60.1005,  -49.9595,  -37.6322,  -29.9353,
             -18.9407,  -11.9213,   -2.5633,   10.6869,   18.9005,   29.4622,
              41.7188,   49.6080,   59.3583,   71.3071,   80.2604,   91.4908,
             100.2913,  108.7626,  118.8391,  129.8859,  139.5593,  150.6612,
             161.5152,  170.5409,  179.0472,  187.5896,  199.0938,  210.3955,
             221.2551,  229.1151,  240.5497,  250.7286,  260.3474,  268.6524,
             280.6704,  291.0199,  302.0525,  309.1079,  320.3692,  330.5589,
             340.5503,  350.1638,  359.5840,  369.0214,  379.5835,  390.5975,
             398.7851,  408.7201,  420.3418,  430.0718,  440.4431,  449.6514,
             459.1114,  468.1187,  480.6921,  490.0807])

```python
import sympy
from sympy import Symbol, Function, Expr, diff, simplify, exp, log, tanh
x = Symbol('x')
f =  Function('f')
```

## Overall Derivative

```python
diff(x*tanh(log(exp(x)+1)))
```

$\displaystyle \frac{x \left(1 - \tanh^{2}{\left(\log{\left(e^{x} + 1 \right)} \right)}\right) e^{x}}{e^{x} + 1} + \tanh{\left(\log{\left(e^{x} + 1 \right)} \right)}$

```python
simplify(diff(x*tanh(log(exp(x)+1))))
```

$\displaystyle - \frac{x \left(\tanh^{2}{\left(\log{\left(e^{x} + 1 \right)} \right)} - 1\right) e^{x} - \left(e^{x} + 1\right) \tanh{\left(\log{\left(e^{x} + 1 \right)} \right)}}{e^{x} + 1}$

## Softplus

$ \Large \frac{\partial}{\partial x} Softplus(x) = 1 - \frac{1}{e^{x} + 1} $

Or, from PyTorch:

$ \Large \frac{\partial}{\partial x} Softplus(x) = 1 - e^{-Y} $

Where $Y$ is saved output

```python
class SoftPlusTest(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp, threshold=20):
        y = torch.where(inp < threshold, torch.log1p(torch.exp(inp)), inp)
        ctx.save_for_backward(y)
        return y

    @staticmethod
    def backward(ctx, grad_out):
        y, = ctx.saved_tensors
        res = 1 - (-y).exp_()
        return grad_out * res

```

```python
torch.allclose(F.softplus(inp), SoftPlusTest.apply(inp))
```

    True

```python
torch.autograd.gradcheck(SoftPlusTest.apply, inp.to(torch.float64).requires_grad_())
```

    True

## $tanh(Softplus(x))$

```python
diff(tanh(f(x)))
```

$\displaystyle \left(1 - \tanh^{2}{\left(f{\left(x \right)} \right)}\right) \frac{d}{d x} f{\left(x \right)}$

```python
class TanhSPTest(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp, threshold=20):
        ctx.save_for_backward(inp)
        sp = torch.where(inp < threshold, torch.log1p(torch.exp(inp)), inp)
        y = torch.tanh(sp)
        return y

    @staticmethod
    def backward(ctx, grad_out, threshold=20):
        inp, = ctx.saved_tensors
        sp = torch.where(inp < threshold, torch.log1p(torch.exp(inp)), inp)
        grad_sp = 1 - torch.exp(-sp)
        tanhsp = torch.tanh(sp)
        grad = (1 - tanhsp*tanhsp) * grad_sp
        return grad_out * grad

```

```python
torch.allclose(TanhSPTest.apply(inp), torch.tanh(F.softplus(inp)))
```

    True

```python
torch.autograd.gradcheck(TanhSPTest.apply, inp.to(torch.float64).requires_grad_())
```

    True

## Mish

```python
diff(x * f(x))
```

$\displaystyle x \frac{d}{d x} f{\left(x \right)} + f{\left(x \right)}$

```python
diff(x*tanh(f(x)))
```

$\displaystyle x \left(1 - \tanh^{2}{\left(f{\left(x \right)} \right)}\right) \frac{d}{d x} f{\left(x \right)} + \tanh{\left(f{\left(x \right)} \right)}$

```python
simplify(diff(x*tanh(f(x))))
```

$\displaystyle \frac{x \frac{d}{d x} f{\left(x \right)}}{\cosh^{2}{\left(f{\left(x \right)} \right)}} + \tanh{\left(f{\left(x \right)} \right)}$

```python
diff(tanh(f(x)))
```

$\displaystyle \left(1 - \tanh^{2}{\left(f{\left(x \right)} \right)}\right) \frac{d}{d x} f{\left(x \right)}$

```python
class MishTest(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp, threshold=20):
        ctx.save_for_backward(inp)
        sp = torch.where(inp < threshold, torch.log1p(torch.exp(inp)), inp)
        tsp = torch.tanh(sp)
        y = inp.mul(tsp)
        return y

    @staticmethod
    def backward(ctx, grad_out, threshold=20):
        inp, = ctx.saved_tensors
        sp = torch.where(inp < threshold, torch.log1p(torch.exp(inp)), inp)
        grad_sp = 1 - torch.exp(-sp)
        tsp = torch.tanh(sp)
        grad_tsp = (1 - tsp*tsp) * grad_sp
        grad = inp * grad_tsp + tsp
        return grad_out * grad

```

```python
torch.allclose(MishTest.apply(inp), inp.mul(torch.tanh(F.softplus(inp))))
```

    True

```python
torch.autograd.gradcheck(TanhSPTest.apply, inp.to(torch.float64).requires_grad_())
```

    True


## Méthode des Trapèzes

On sait que l'aire d'un trapèze est donnée par : $$Aire = \frac{(B + b)h}{2}$$

On définit : $$W_i = \frac{(f(x_i) + f(x_{i+1}))h}{2}$$

On trouve une formule pour approximer notre intégrale :

$$
\begin{align*}
    I &\approx \sum_{i=0}^{n-1} W_i \\
      &\approx \sum_{i=0}^{n-1} \frac{h}{2}(f(x_i) + f(x_{i+1}) \\
      &\approx \frac{h}{2}(f(x_0) + f(x_1) + f(x_1) + ... + f(x_{n-1}) + f(x_{n-1}) + f(x_n)) \\
      &\approx \frac{h}{2} \left[ f(x_0) + 2 \sum_{i=0}^{n-1}f(x_i) + f(x_n)) \right] \\
      &\approx h \left[ \frac{1}{2}(f(x_0) + f(x_n)) + \sum_{i=0}^{n-1}f(x_i)) \right]
\end{align*}
$$

### Itération
- On diminue progressivement le pas de h de $\frac{1}{2}$ à chaque itération jusqu'à atteindre la précision souhaitée.
- /!\ une itération = une évaluation complète de l'intégrale.

Pour une fonction : $$\int_a^b f(x)dx$$

$n = 2^{k-1}$ => 1, 2, 4, 8,...

$h = \frac{b-a}{n}$ => $h$, $\frac{h}{2}$, $\frac{h}{4}$,...

Inconvénient : convergence lente donc besoin de beaucoup d'itérations pour être précis.

```python
import numpy as np
import math

# precision de l'évaluation numérique de l'intégrale (trapèzes)
TRESHOLD_PRECISION = 1e-9
DIMENSIONS = [1] #[1, 2, 10, 15]
X_0 = -3
X_N = 3

class Trapezes():
  def __init__(self, f, x_0, x_n):
    self.f = f
    self.x_0 = x_0
    self.x_n = x_n

  # borne inférieure : x_0
  # borne supérieure : x_n
  # en pratique on itère sur la méthode
  # et on diminue le pas h progressivement (de 1/2 à chaque itération)
  # k : itération en cours
  def trapezes(self, k):
    # si k = 1, le nombre de trapèzes serait de 1 (en effet, 2^0 = 1)
    if k == 1:
      # Aire d'un trapèze
      h = (self.x_n - self.x_0)
      return (self.f(self.x_0) + self.f(self.x_n)) * h / 2

    # nombre de trapèzes
    n = int(math.pow(2, k-1))
    # intervalle h
    h = (self.x_n - self.x_0) / n

    sum = 1/2 * (self.f(self.x_0) + self.f(self.x_n))
    for i in range(1, n):
      x_i = self.x_0 + (i * h)
      sum += self.f(x_i)

    return h * sum

  # N : nombre de dimensions
  def integrate(self, iterations, precision):
    new_integral = 0
    for k in range(1, iterations):
      old_integral = new_integral
      new_integral = self.trapezes(k)
      if abs(old_integral - new_integral) < precision and k > 1:
        return new_integral, k
        break

def gaussian(x):
  f = math.exp(-(x**2) / 2)
  return f

def gaussian_2_dim(x, y):
  return math.exp(-(x**2 + y**2) / 2)

def gaussian_theo(N):
  return math.sqrt((2*math.pi)**N)

print(f"dimension 1")
print("-----------")
print(f"Valeur théorique approximative: {gaussian_theo(N=1)}")

gaussian_trapezes = Trapezes(gaussian, X_0, X_N)
gaussian_trapezes_value = gaussian_trapezes.integrate(iterations = 20, precision = TRESHOLD_PRECISION)
print(f"Valeur de la gaussienne à 1 dimension (trapèzes): {gaussian_trapezes_value}")

print(f"dimension 2")
print("-------------")
print(f"Valeur théorique approximative: {gaussian_theo(N=2)}")

print("-------------")

def universe_model(x, omega_m, omega_rad, omega_k, omega_lambda):

gaussian_trapezes = Trapezes(gaussian, X_0, X_N)
gaussian_trapezes_value = gaussian_trapezes.integrate(iterations = 20, precision = TRESHOLD_PRECISION)
print(f"Valeur de la gaussienne à 1 dimension (trapèzes): {gaussian_trapezes_value}")

```

    dimension 1
    -----------
    Valeur théorique approximative: 2.5066282746310002
    Valeur de la gaussienne à 1 dimension (trapèzes): (2.4998608892968623, 16)
    dimension 2
    -------------
    Valeur théorique approximative: 6.283185307179586

## Méthode de Romberg

Combine la méthode itérative des trapèzes avec l'extrapolation de Richardson.
**Erreur sur la méthode des Trapèzes**: $E(h) = c_1h^2 + c_2h^4 + ... = \sum_i c_ih^{2i}$

### Extrapolation de Richardson

Pour un pas de h :
$$G = I(h) + E(h)$$

Pour un pas de h/2 :
$$G = I(h/2) + E(h/2)$$

On obtient le système :

$$
\begin{align}
\begin{cases}
    G = I(h) + ch^p \\
    G = I(h/2) + c(h/2)^p \implies 2^pG = 2^pI(h/2) + ch^p
\end{cases}
\end{align}
$$

On soustrait l'équation $(1)$ à l'équation $(2)$ :

$$
\begin{align}
    2^pG - G &= 2^pI(h/2) - I(h) \\
    (2^p - 1)G &= 2^p I(h/2) - I(h) \\
    G &= \frac{2^p I(h/2) - I(h)}{2^p - 1}
\end{align}
$$

Nuos avons donc une formule pour l'extrapolation de Richardson :

$$
G = \frac{2^p I(h/2) - I(h)}{2^p - 1}
$$

**Concrètement**:

On va se constituer une matrice $n x n$ avec _n : le nombre d'itérations_. Elle contiendra dans sa première colonne les approximations de l'intégrale par la méthode des trapèzes. Ensuite on rempli "le reste" en cascade avec la formule de l'extrapolation de Richardson.

**Algorithme**:

On remarque que plus le nombre d'itérations augmente, plus la précision augmente.

```python
import numpy as np
import math

class Romberg():
  def __init__(self, f, x_0, x_n):
    self.f = f
    self.x_0 = x_0
    self.x_n = x_n

  def trapezes(self, k):
    # si k = 1, le nombre de trapèzes serait de 1 (en effet, 2^0 = 1)
    if k == 1:
      # Aire d'un trapèze
      h = (self.x_n - self.x_0)
      return (self.f(self.x_0) + self.f(self.x_n)) * h / 2

    # nombre de trapèzes
    n = int(math.pow(2, k-1))
    # intervalle h
    h = (self.x_n - self.x_0) / n

    sum = 1/2 * (self.f(self.x_0) + self.f(self.x_n))
    for i in range(1, n):
      x_i = self.x_0 + (i * h)
      sum += self.f(x_i)

    return h * sum

  def richardson_extrapolation(self, I_prev, I_current, k):
    return ((2**k * I_current) - I_prev) / (2**k - 1)

  def integrate(self, iterations, precision):
    # R(n, m)
    self.map = np.zeros((iterations, iterations))

    """
    iterations = 3
       j
       __________________
    i | trap   x    x    |
      | trap rich   x    |
      | trap rich approx |
       ------------------
    """
    for i in range(1, iterations):
      i_prime = i - 1
      # Méthode des trapèzes
      self.map[i_prime, 0] = self.trapezes(i)
      for j in range(1, i):
        # Extrapolation de Richardson
        self.map[i_prime, j] = self.richardson_extrapolation(self.map[i_prime-1, j-1], self.map[i_prime, j-1], 2*j)

      if abs(self.map[i_prime - 1, i_prime - 1] - self.map[i_prime, i_prime]) < precision:
        return self.map[i_prime, i_prime], i
        break

# precision de l'évaluation numérique de l'intégrale (trapèzes)
TRESHOLD_PRECISION = 1e-9
DIMENSIONS = [1] #[1, 2, 10, 15]
X_0 = -3
X_N = 3

def gaussian(x):
  f = math.exp(-(x**2) / 2)
  return f

# romberg
gaussian_romberg = Romberg(gaussian, X_0, X_N)
gaussian_romberg_value = gaussian_romberg.integrate(iterations = 10, precision = TRESHOLD_PRECISION)
print(f"Valeur de la gaussienne à 2 dimension (romberg): {gaussian_romberg_value}")
```

    Valeur de la gaussienne à 2 dimension (romberg): (2.4998608894818006, 8)

### Méthode de Monte-Carlo

- Plus efficace pour calculer des intégrales à plusieurs dimensions.
- On tire les points aléatoirement dans le domaine de la fonction.

$$
\begin{align}
    <f> &= \frac{1}{b-a} \int_a^b f(x)dx \\
    \int_a^b f(x)dx &= (b-a)<f> \\
    \int_a^b f(x)dx &\approx (b-a) \frac{1}{N} \sum_{i=0}^{n-1} f(x_i)
\end{align}
$$

```python
import math
import random
import numpy as np

def gaussian_theo(N):
  return math.sqrt((2*math.pi)**N)

def gaussian(x_vec):
    sum = 0
    # x^2 + y^2 + z^2 + ...
    for x_i in x_vec:
        sum += x_i**2

    return math.exp(-(sum) / 2)

class MonteCarlo():
    def __init__(self, f, x_0, x_n):
        self.f = f
        self.x_0 = x_0
        self.x_n = x_n

    def integrate(self, iterations):
        sum = 0
        # e.g.: 3 = intégrale triple
        dimension = len(self.x_0)
        # N * N * N * ... = N^(dimension)
        volume = np.prod([x_n - x_0 for x_0, x_n in zip(self.x_0, self.x_n)])
        # somme des f(x_i)
        sum = 0
        for i in range(iterations):
            x_i = [random.uniform(self.x_0[k], self.x_n[k]) for k in range(dimension)]
            sum += self.f(x_i)

        integral = volume * (1/iterations) * sum
        return integral

print(f"predicted gaussian integral value: {gaussian_theo(3)}")
integral = MonteCarlo(gaussian, [-3, -3, -3], [3, 3, 3]).integrate(1000000)
print(f"approximated gaussian integral value: {integral}")
```

    predicted gaussian integral value: 15.749609945722419
    approximated gaussian integral value: 15.627813816362522

#### Amélioration

- Utiliser une répartition arbitraire selon la forme de la fonction à intégrer.


# Linear Advection Equation

\begin{equation}
\begin{aligned}
    \partial_t u(t,x) + \partial_x (a(x) u(t,x)) &= 0, && t \in (0,T), x \in (x_{min}, x_{max}), \\
    u(0,x) &= u_0(x), && x \in (x_{min}, x_{max}), \\
    \text{boundary conditions}, &&& x \in \partial (x_{min}, x_{max}).
\end{aligned}
\end{equation}

The boundary conditions depend on the sign of the transport velocity $a$ at the boundary. In particular, specifying a Dirichlet type boundary condition is only allowed for inflow boundaries, e.g. $a(x_{min}) > 0$ at $x = x_{min}$.

```julia
using Revise
using SummationByPartsOperators, OrdinaryDiffEq
using Plots, LaTeXStrings, Printf

# general parameters
xmin = -1.
xmax = +1.
tspan = (0., 8.0)
afunc(x) = one(x)
u0func(x) = sinpi(x)
# Dirichlet type boundary conditions; they are used only at inflow boundaries
left_bc(t) = t >= 3 ? sinpi(t) : zero(t)
right_bc(t) = zero(t)

# discretisation parameters
interior_order = 4
N = 101
# whether a split form should be applied or not
split_form = Val(false)

# setup spatial semidiscretization
D = derivative_operator(MattssonSvärdShoeybi2008(), 1, interior_order, xmin, xmax, N)
# whether or not artificial dissipation should be applied: nothing, dissipation_operator(D)
Di = nothing
semi = VariableLinearAdvectionNonperiodicSemidiscretization(D, Di, afunc, split_form, left_bc, right_bc)
ode = semidiscretize(u0func, semi, tspan)

# solve ode
sol = solve(ode, SSPRK104(), dt=D.Δx, adaptive=false,
            saveat=range(first(tspan), stop=last(tspan), length=200))

# visualise the result
plot(xguide=L"x", yguide=L"u")
plot!(evaluate_coefficients(sol[end], semi), label="")
```

```julia
# make a movie
anim = Animation()
idx = 1
x, u = evaluate_coefficients(sol[idx], semi)

fig = plot(x, u, xguide=L"x", yguide=L"u", xlim=extrema(x), ylim=(-1.05, 1.05),
           #size=(1024,768), dpi=250,
           label="", title=@sprintf("\$t = %6.2f \$", sol.t[idx]))
for idx in 1:length(sol.t)
    fig[1] = x, sol.u[idx]
    plot!(title=@sprintf("\$t = %6.2f \$", sol.t[idx]))
    frame(anim)
end
gif(anim)
```


# Nonlinear Systems of Equations

Non-linear systems are extensions of the linear systems cases except the systems involve products and powers of the unknown variables. Non-linear problems are often quite difficult to manage, especially when the systems are large (many rows and many variables).
The solution to non-linear systems, if non-trivial or even possible, are usually iterative. Within the iterative steps is a linearization component – these linear systems which are intermediate computations within the overall solution process are treated by an appropriate linear system method (direct or iterative).
Consider the system below:

\begin{gather}
\begin{matrix}
x^2 & +~y^2  \\
 e^x & +~y  \\
\end{matrix}
\begin{matrix}
= 4\\
= 1\\
\end{matrix}
\end{gather}

Suppose we have a solution guess $x_{k},y_{k}$, which of course could be wrong, but we could linearize about that guess as

\begin{gather}
\mathbf{A} =
\begin{pmatrix}
x_k & + ~y_k  \\
0 & + ~1  \\
\end{pmatrix}
~\mathbf{x} =
\begin{pmatrix}
x_{k+1} \\
y_{k+1} \\
\end{pmatrix}
~ \mathbf{b} =
\begin{pmatrix}
4\\
1 - e^{x_k}\\
\end{pmatrix}
\end{gather}

Now if we assemble the system in the usual fashion, $\mathbf{A} \cdot \mathbf{x_{k+1}} = ~ \mathbf{b}~$ we have a system of linear equations\footnote{Linear in $\mathbf{x_{k+1}}$}, which expanded look like:

\begin{gather}
\begin{pmatrix}
x_k & + ~y_k  \\
0 & + ~1  \\
\end{pmatrix}
\cdot
\begin{pmatrix}
x_{k+1} \\
y_{k+1} \\
\end{pmatrix}
~ =
\begin{pmatrix}
4\\
1 - e^{x_k}\\
\end{pmatrix}
\end{gather}

Now that the system is linear, and we can solve for $\mathbf{x_{k+1}}$ using our linear system solver for the new guess.
If the system is convergent (not all are) then if we update, and repeat  we will eventually find a result.

What one really needs is a way to construct the linear system that has a systematic update method, that is discussed below

## Multiple-variable extension of Newton’s Method

This section presents the Newton-Raphson method as a way to sometimes solve systems of non-linear equations.

Consider an example where the function \textbf{f} is a vector-valued function of a vector argument.

\begin{gather}
\mathbf{f(x)} =
\begin{matrix}
f_1 = & x^2 & +~y^2 & - 4\\
f_2 = & e^x & +~y  & - 1\\
\end{matrix}
\end{gather}

Let's also recall Newtons method for scalar valued function of a single variable.

\begin{equation}
x_{k+1}=x_{k} - \frac{  f(x_{k})  }{   \frac{df}{dx}\rvert_{x_k} }
\label{eqn:NewtonFormula}
\end{equation}

When extending to higher dimensions, the analog for $x$ is the vector \textbf{x} and the analog for the function $f()$ is the vector function \textbf{f()}.
What remains is an analog for the first derivative in the denominator (and the concept of division of a matrix).

The analog to the first derivative is a matrix called the Jacobian which is comprised of the first derivatives of the function \textbf{f} with respect to the arguments \textbf{x}.
For example for a 2-value function of 2 arguments (as our example above)

\begin{equation}
\frac{df}{dx}\rvert_{x_k} =>
\begin{pmatrix}
\frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} \\
~ & ~ \\
\frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} \\
\end{pmatrix}
\label{eqn:Jacobian}
\end{equation}

Next recall that division is replaced by matrix multiplication with the multiplicative inverse, so the analogy continues as

\begin{equation}
\frac{1}{\frac{df}{dx}\rvert_{x_k}} =>
{\begin{pmatrix}
\frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} \\
~ & ~ \\
\frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} \\
\end{pmatrix}}^{-1}
\label{eqn:JacobianInverse}
\end{equation}

Let's name the Jacobian \textbf{J(x)}.

So the multi-variate Newton's method can be written as

\begin{equation}
\mathbf{x}_{k+1}=\mathbf{x}_{k} - \mathbf{J(x)}^{-1}\rvert_{x_k} \cdot \mathbf{f(x)}\rvert_{x_k}
\label{eqn:VectorNewtonFormula}
\end{equation}

In the linear systems lessons we did find a way to solve for an inverse, but it's not necessary, and is computationally expensive to invert in these examples -- a series of rearrangement of the system above yields a nice scheme that does not require inversion of a matrix.

First, move the $\mathbf{x}_{k}$ to the left-hand side.

\begin{equation}
\mathbf{x}_{k+1}-\mathbf{x}_{k} = - \mathbf{J(x)}^{-1}\rvert_{x_k} \cdot \mathbf{f(x)}\rvert_{x_k}
\end{equation}

Next multiply both sides by the Jacobian (The Jacobian must be non-singular otherwise we are dividing by zero)

\begin{equation}
\mathbf{J(x)}\rvert_{x_k} \cdot (\mathbf{x}_{k+1}-\mathbf{x}_{k}) = - \mathbf{J(x)}\rvert_{x_k} \cdot \mathbf{J(x)}^{-1}\rvert_{x_k} \cdot \mathbf{f(x)}\rvert_{x_k}
\end{equation}

Recall a matrix multiplied by its inverse returns the identity matrix (the matrix equivalent of unity)

\begin{equation}
-\mathbf{J(x)}\rvert_{x_k} \cdot (\mathbf{x}_{k+1}-\mathbf{x}_{k}) = \mathbf{f(x)}\rvert_{x_k}
\end{equation}

So we now have an algorithm:

1) Start with an initial guess $\mathbf{x}_{k}$, compute $\mathbf{f(x)}\rvert_{x_k}$, and $\mathbf{J(x)}\rvert_{x_k}$.

2) Test for stopping.  Is $\mathbf{f(x)}\rvert_{x_k}$ close to zero?  If yes, exit and report results, otherwise continue.

3) Solve the linear system $\mathbf{J(x)}\rvert_{x_k} \cdot (\mathbf{x}_{k+1}-\mathbf{x}_{k}) = \mathbf{f(x)}\rvert_{x_k}$.

4) Test for stopping.  Is $ (\mathbf{x}_{k+1}-\mathbf{x}_{k})$ close to zero?  If yes, exit and report results, otherwise continue.

5) Compute the update $\mathbf{x}_{k+1} = \mathbf{x}_{k} - (\mathbf{x}_{k+1}-\mathbf{x}_{k}) $, then

6) Move the update into the guess vector $\mathbf{x}_{k} <=\mathbf{x}_{k+1}$ =and repeat step 1. Stop after too many steps.

## Example using Analytical Derivatives

Now to complete the example we will employ this algorithm.

The function (repeated)

\begin{gather}
\mathbf{f(x)} =
\begin{matrix}
f_1 = & x^2 & +~y^2 & - 4\\
f_2 = & e^x & +~y  & - 1\\
\end{matrix}
\end{gather}

Then the Jacobian, here we will compute it analytically because we can

\begin{equation}
\mathbf{J(x)}=>
{\begin{pmatrix}
2x & 2y \\
~ & ~ \\
e^x & 1 \\
\end{pmatrix}}
\end{equation}

Now for the scripts.

We will start by defining the two equations, and their derivatives, as well a a vector valued function `func` and its Jacobian `jacob` as below.  Here the two modules `LinearSolverPivot` and `vector_matrix_lib` are just python source code files containing prototype functions.

```python
#################################################################
# Newton Solver Example -- Analytical Derivatives               #
#################################################################
import math   # This will import math module from python distribution
from LinearSolverPivot import linearsolver # This will import our solver module
from vector_matrix_lib import writeM,writeV,vdotv,vvsub # This will import our vector functions

def eq1(x,y):
    eq1 = x**2 + y**2 - 4.0
    return(eq1)

def eq2(x,y):
    eq2 = math.exp(x) + y - 1.0
    return(eq2)

def ddxeq1(x,y):
    ddxeq1 = 2.0*x
    return(ddxeq1)

def ddyeq1(x,y):
    ddyeq1 = 2.0*y
    return(ddyeq1)

def ddxeq2(x,y):
    ddxeq2 = math.exp(x)
    return(ddxeq2)

def ddyeq2(x,y):
    ddyeq2 = 1.0
    return(ddyeq2)

def func(x,y):
    func  = [0.0 for i in range(2)] # null list
    # build the function
    func[0] = eq1(x,y)
    func[1] = eq2(x,y)
    return(func)

def jacob(x,y):
    jacob = [[0.0 for j in range(2)] for i in range(2)] # constructed list
    #build the  jacobian
    jacob[0][0]=ddxeq1(x,y)
    jacob[0][1]=ddyeq1(x,y)
    jacob[1][0]=ddxeq2(x,y)
    jacob[1][1]=ddyeq2(x,y)
    return(jacob)
```

Next we create vectors to store values, and supply initial guesses to the system, and echo the inputs.

```python
deltax  = [0.0 for i in range(2)] # null list
xguess  = [0.0 for i in range(2)] # null list
myfunc  = [0.0 for i in range(2)] # null list
myjacob = [[0.0 for j in range(2)] for i in range(2)] # constructed list
# supply initial guess
xguess[0] = float(input("Value for x : "))
xguess[1] = float(input("Value for y : "))
# build the initial function
myfunc = func(xguess[0],xguess[1])
#build the initial jacobian
myjacob=jacob(xguess[0],xguess[1])
#write initial results
writeV(xguess,2,"Initial X vector ")
writeV(myfunc,2,"Initial FUNC vector ")
writeM(myjacob,2,2,"Initial Jacobian ")
# solver parameters
tolerancef = 1.0e-9
tolerancex = 1.0e-9
```

    Value for x :  2
    Value for y :  3

    ------ Initial X vector  ------
    2.0
    3.0
    -----------------------------
    ------ Initial FUNC vector  ------
    9.0
    9.38905609893065
    -----------------------------
    ------ Initial Jacobian  ------
    [4.0, 6.0]
    [7.38905609893065, 1.0]
    -----------------------------

Now we apply the algorithm a few times, here the count is set to 10. So eneter the loop, test for stopping, then update.

```python
# Newton-Raphson
for iteration in range(10):
    myfunc = func(xguess[0],xguess[1])
    testf = vdotv(myfunc,myfunc,2)
    if testf <= tolerancef :
        print("f(x) close to zero\n test value : ", testf)
        break
    myjacob=jacob(xguess[0],xguess[1])
    deltax=linearsolver(myjacob,myfunc)
    testx = vdotv(deltax,deltax,2)
    if testx <= tolerancex :
        print("solution change small\n test value : ", testx)
        break
    xguess=vvsub(xguess,deltax,2)
##    print("iteration : ",iteration)
##    writeV(xguess,2,"Current X vector ")
##    writeV(myfunc,2,"Current FUNC vector ")
print("Exiting Iteration : ",iteration)
writeV(xguess,2,"Exiting X vector ")
writeV(myfunc,2,"Exiting FUNC vector ")
```

    f(x) close to zero
     test value :  4.741631714784361e-10
    Exiting Iteration :  6
    ------ Exiting X vector  ------
    -1.8162690125838175
    0.8373700502918618
    -----------------------------
    ------ Exiting FUNC vector  ------
    2.1727197990095704e-05
    1.446388252723807e-06
    -----------------------------

## Quasi-Newton Method using Finite Difference Approximations for the Derivative
The next variant is to approximate the derivatives -- usually a Finite-Difference approximation is used, either forward, backward, or centered differences -- generally determined based on the actual behavior of the functions themselves or by trial and error.

For really huge systems, we usually make the program itself make the adaptions as it proceeds.

The coding for a finite-difference representation of a Jacobian is shown in Listing that follows
In constructing the Jacobian, we observe that each column of the Jacobian is simply the directional derivative of the function with respect to the variable associated with the column.
For instance, the first column of the Jacobian in the example is first derivative of the first function (all rows) with respect to the first variable, in this case $x$.  The second column is the first derivative of the second function with respect to the second variable, $y$.
This structure is useful to generalize the Jacobian construction method because we could write (yet another) prototype function that can take the directional derivatives for us, and just insert the returns as columns; in the example we simply modified the `ddx` and `ddy` functions from analytical to simple finite differences.

The example listing is specific to the 2X2 function in the example, but the extension to more general cases is evident.

```python
#################################################################
# Newton Solver Example -- Numerical Derivatives               #
#################################################################
import math   # This will import math module from python distribution
from LinearSolverPivot import linearsolver # This will import our solver module
from vector_matrix_lib import writeM,writeV,vdotv,vvsub # This will import our vector functions

def eq1(x,y):
    eq1 = x**2 + y**2 - 4.0
    return(eq1)

def eq2(x,y):
    eq2 = math.exp(x) + y - 1.0
    return(eq2)
##############################################################
# This portion is changed for finite-difference method to evaluate derivatives #
##############################################################
def ddxeq1(x,y):
    delta = 1.0e-6
    ddxeq1 = (eq1(x+delta,y)-eq1(x,y))/delta
    return(ddxeq1)

def ddyeq1(x,y):
    delta = 1.0e-6
    ddyeq1 = (eq1(x,y+delta)-eq1(x,y))/delta
    return(ddyeq1)

def ddxeq2(x,y):
    delta = 1.0e-6
    ddxeq2 = (eq2(x+delta,y)-eq2(x,y))/delta
    return(ddxeq2)

def ddyeq2(x,y):
    delta = 1.0e-6
    ddyeq2 = (eq2(x,y+delta)-eq2(x,y))/delta
    return(ddyeq2)
##############################################################
def func(x,y):
    func  = [0.0 for i in range(2)] # null list
    # build the function
    func[0] = eq1(x,y)
    func[1] = eq2(x,y)
    return(func)

def jacob(x,y):
    jacob = [[0.0 for j in range(2)] for i in range(2)] # constructed list
    #build the  jacobian
    jacob[0][0]=ddxeq1(x,y)
    jacob[0][1]=ddyeq1(x,y)
    jacob[1][0]=ddxeq2(x,y)
    jacob[1][1]=ddyeq2(x,y)
    return(jacob)
deltax  = [0.0 for i in range(2)] # null list
xguess  = [0.0 for i in range(2)] # null list
myfunc  = [0.0 for i in range(2)] # null list
myjacob = [[0.0 for j in range(2)] for i in range(2)] # constructed list
# supply initial guess
xguess[0] = float(input("Value for x : "))
xguess[1] = float(input("Value for y : "))
# build the initial function
myfunc = func(xguess[0],xguess[1])
#build the initial jacobian
myjacob=jacob(xguess[0],xguess[1])
#write initial results
writeV(xguess,2,"Initial X vector ")
writeV(myfunc,2,"Initial FUNC vector ")
writeM(myjacob,2,2,"Initial Jacobian ")
# solver parameters
tolerancef = 1.0e-9
tolerancex = 1.0e-9
# Newton-Raphson
for iteration in range(10):
    myfunc = func(xguess[0],xguess[1])
    testf = vdotv(myfunc,myfunc,2)
    if testf <= tolerancef :
        print("f(x) close to zero\n test value : ", testf)
        break
    myjacob=jacob(xguess[0],xguess[1])
    deltax=linearsolver(myjacob,myfunc)
    testx = vdotv(deltax,deltax,2)
    if testx <= tolerancex :
        print("solution change small\n test value : ", testx)
        break
    xguess=vvsub(xguess,deltax,2)
##    print("iteration : ",iteration)
##    writeV(xguess,2,"Current X vector ")
##    writeV(myfunc,2,"Current FUNC vector ")
print("Exiting Iteration : ",iteration)
writeV(xguess,2,"Exiting X vector ")
writeV(myfunc,2,"Exiting FUNC vector using Finite-Differences")
```

    Value for x :  0
    Value for y :  2

    ------ Initial X vector  ------
    0.0
    2.0
    -----------------------------
    ------ Initial FUNC vector  ------
    0.0
    2.0
    -----------------------------
    ------ Initial Jacobian  ------
    [1.000088900582341e-06, 4.0000010006480125]
    [1.0000005001842283, 1.000000000139778]
    -----------------------------
    f(x) close to zero
     test value :  1.124762923231742e-16
    Exiting Iteration :  5
    ------ Exiting X vector  ------
    -1.816264071231508
    0.8373678009903361
    -----------------------------
    ------ Exiting FUNC vector using Finite-Differences ------
    1.0581842957435583e-08
    7.077372021768724e-10
    -----------------------------

    ()

```python

```

## Exercises

Write a script that forward defines the multi-variate functions and implements the Newton-Raphson technique.
Implement the method, using analytical derivatives, and find a solution to:
\begin{gather}
\begin{matrix}
 x^3 & +~3y^2 & = 21\\
x^2& +~2y  & = -2 \\
\end{matrix}
\end{gather}

Repeat the exercise, except use finite-differences to approximate the derivatives.

```python

```

Write a script that forward defines the multi-variate functions and implements the Newton-Raphson technique.
Implement the method, using analytical derivatives, and find a solution to:
\begin{gather}
\begin{matrix}
x^2 & +~ y^2 & +~z^2 & =~ 9\\
~ & ~ & xyz & =~ 1\\
x & +~ y & -z^2 & =~ 0\\
\end{matrix}
\end{gather}

Repeat the exercise, except use finite-differences to approximate the derivatives.

```python

```

Write a script that forward defines the multi-variate functions and implements the Newton-Raphson technique.
Implement the method, using analytical derivatives, and find a solution to:
\begin{gather}
\begin{matrix}
xyz & -~ x^2 & +~y^2 & =~ 1.34\\
~ & xy &-~z^2 & =~ 0.09\\
e^x & -~ e^y & +z & =~ 0.41\\
\end{matrix}
\end{gather}

Repeat the exercise, except use finite-differences to approximate the derivatives.

```python

```


# Sweep Signals and their Spectra

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the masters module Selected Topics in Audio Signal Processing, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Linear Sweep

A [linear sweep](https://en.wikipedia.org/wiki/Chirp#Linear) is an exponential signal with linear increase in its instantaneous frequency. It is defined as

\begin{equation}
x(t) = e^{j \omega(t) t}
\end{equation}

with

\begin{equation}
\omega(t) = \omega_\text{l} - \frac{\omega_\text{u} - \omega_\text{l}}{2 T} t
\end{equation}

where $\omega_\text{l}$ and $\omega_\text{u}$ denote its lower and upper frequency limit, and $T$ its total duration. The linear sweep is generated in the following by sampling the continuous time.

```python
import numpy as np
import matplotlib.pyplot as plt
import soundfile as sf

fs = 48000  # sampling frequency
om_l = 2*np.pi*200  # lower angluar frequency of sweep
om_u = 2*np.pi*18000  # upper angular frequency of sweep
T = 5  # duration of sweep
k = (om_u - om_l)/(2*T)

t = np.linspace(0, T, fs*T)
x = np.exp(1j*(om_l + k*t)*t)
```

A short section of the linear sweep signal is plotted for illustration

```python
idx = range(5000)  # portion of the signal to show

plt.figure(figsize=(10, 5))
plt.plot(t[idx], np.real(x[idx]))
plt.xlabel(r'$t$ in s')
plt.ylabel(r'$x(t)$')
plt.grid()
```

### Auralization

Lets listen to the linear sweep. Please be careful with the volume of your speakers or headphones. Start with a very low volume and increase if necessary. This holds especially for the low and high frequencies which can damage your speakers at high levels.

```python
sf.write('linear_sweep.wav', np.real(x), fs)
```

<audio src="linear_sweep.wav" controls>Your browser does not support the audio element.</audio>
[linear_sweep.wav](linear_sweep.wav)

### Spectrogram

The spectrogram of the linear sweep is computed and plotted

```python
plt.figure(figsize=(8, 6))
plt.specgram(x, Fs=fs, sides='onesided')
plt.xlabel('$t$ in s')
plt.ylabel('$f$ in Hz');
```

### Spectrum of a Linear Sweep (Analytic Solution)

The analytic solution of the Fourier transform of the linear sweep signal is used to compute and plot its overall magnitude spectrum $|X(j \omega)|$

```python
from scipy.special import fresnel

f = np.linspace(10, 20000, 1000)
om = 2*np.pi*f
om_s = 2*np.pi*200  # lower angluar frequency of sweep
om_e = 2*np.pi*18000  # upper angular frequency of sweep
T = 5  # duration of sweep
k = (om_e - om_s)/(2*T)

a = (om-om_s)/np.sqrt(2*np.pi*k)
b = (2*k*T-(om-om_s))/np.sqrt(2*np.pi*k)
Sa, Ca = fresnel(a)
Sb, Cb = fresnel(b)

X = np.sqrt(np.pi/(2*k)) * np.sqrt((Ca + Cb)**2 + (Sa + Sb)**2)

plt.figure(figsize=(8, 5))
plt.plot(f, X)
plt.xlabel('$f$ in Hz')
plt.ylabel(r'$|X(f)|$')
plt.grid()
```

### Crest Factor

The [Crest factor](https://en.wikipedia.org/wiki/Crest_factor) of the sweep is computed

```python
xrms = np.sqrt(1/T * np.sum(np.real(x)**2) * 1/fs)
C = np.max(np.real(x)) / xrms
print('Crest factor C = {:<1.5f}'.format(C))
```

    Crest factor C = 1.41421

## Exponential Sweep

An [exponential sweep](https://en.wikipedia.org/wiki/Chirp#Exponential) is an exponential signal with an exponential increase in its instantaneous frequency. It is defined as

\begin{equation}
x(t) = e^{j \frac{\omega_l}{\ln(k)} (k^t - 1)}
\end{equation}

with

\begin{equation}
k = \left( \frac{\omega_\text{u}}{\omega_\text{l}} \right)^\frac{1}{T}
\end{equation}

where $\omega_\text{l}$ and $\omega_\text{u}$ denote its lower and upper frequency limit, and $T$ its total duration. The exponential sweep is generated in the following by sampling the continuous time.

```python
om_l = 2*np.pi*100  # lower angluar frequency of sweep
om_u = 2*np.pi*18000  # upper angular frequency of sweep
T = 5  # duration of sweep
k = (om_e / om_s)**(1/T)

t = np.linspace(0, T, fs*T)
x = np.exp(1j*om_s * (k**(t) - 1) / np.log(k))
```

A short section of the linear sweep signal is plotted for illustration

```python
idx = range(5000)  # portion of the signal to show

plt.figure(figsize=(10, 5))
plt.plot(t[idx], np.real(x[idx]))
plt.xlabel(r'$t$ in s')
plt.ylabel(r'$x(t)$')
plt.grid()
```

### Spectrogram

The spectrogram of the logarithmic sweep is computed and plotted

```python
plt.figure(figsize=(8, 6))
plt.specgram(x, Fs=fs, sides='onesided')
plt.xlabel('$t$ in s')
plt.ylabel('$f$ in Hz');
```

### Spectrum

The discrete Fourier transform of the logarithmic sweep is computed and its magnitude spectrum is plotted.

```python
X = np.fft.rfft(np.real(x))
f = np.linspace(0, fs/2, len(X))

plt.figure(figsize=(8, 6))
plt.plot(f, 20*np.log10(np.abs(X)))
plt.xlabel(r'$f$ in Hz')
plt.ylabel(r'$|X(f)|$ in dB')
plt.axis([20, 20000, 40, 70])
plt.grid()
```

### Auralization

Lets listen to the exponential sweep. Please be careful with the volume of your speakers or headphones. Start with a very low volume and increase if necessary. This holds especially for the low and high frequencies which can damage your speakers at high levels.

```python
sf.write('exponential_sweep.wav', np.real(x), fs)
```

<audio src="exponential_sweep.wav" controls>Your browser does not support the audio element.</audio>
[exponential_sweep.wav](exponential_sweep.wav)

**Copyright**

This notebook is provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text/images/data are licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Selected Topics in Audio Signal Processing - Supplementary Material*.


```python
import sympy as sp
```

```python
sp.symbols('x')
```

$\displaystyle x$

```python
x= sp.symbols('x')
```

```python
y=sp.symbols('y')
```

```python
x+x
```

$\displaystyle 2 x$

```python
#2x+1=0
sp.solve(2*x+1)
```

    [-1/2]

```python
#3x+8=2x-3
ecuación = sp.Eq(3*x+8,2*x-3)
```

```python
sp.solve(ecuación)
```

    [-11]

```python
#x^2-1=0
soluciones = sp.solve(x**2-1)
```

```python
soluciones
```

    [-1, 1]

```python
soluciones[0]
```

$\displaystyle -1$

```python
soluciones[1]
```

$\displaystyle 1$

```python
sp.solve(x**2+1)
```

    [-I, I]

```python
sp.solve(x**2-3*x+1)
```

    [3/2 - sqrt(5)/2, sqrt(5)/2 + 3/2]

```python
sp.init_printing()
```

```python
sp.solve(x**2-3*x+1)
```

```python
#x^3-5*x^2=-6*x
```

```python
ecuacion = sp.Eq(x**3-5*x**2,-6*x)  #se usa coma "," no igual
```

```python
ecuacion
```

```python
sp.solve(ecuacion)
```

```python
sp.plot(x**2)
```

```python
sp.plot(x**2,x+1)
```

```python
sp.plot(x**2,ylim=(0,9),xlim=(0,3), line_color='m')
```

```python
f = 1/(x**2-4)
```

```python
sp.solve(x**2-4)
```

```python
#f(-1)=
f.subs(x,-1)
```

```python
f.subs(x,x+y)
```

```python
g = x**2-3*x*y
g
```

```python
g.subs(x,1)
```

```python
g.subs(y,2)
```

```python
g.subs(x,1).subs(y,2)
```

```python
g.subs([[x,1],[y,2]])
```

```python
f.subs(y,1)
```

```python
f.subs(x,g)
```

```python
sp.factor(x**2-2*x+1)
```

```python
sp.expand(x*(x-2)*(3*x-2)**2)   #expand trata de simplificar siempre
```

```python

```


```python
import numpy as np
import numpy.linalg as LA
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt
```

# Logistic Regression

## Maximim Log-likelihood

Here we will use logistic regression to conduct a binary classification.  The logistic regression process will be formulated using Maximum Likelihood estimation.  To begin, consider a random variable $y \in\{0,1\}$.  Let the $p$ be the probability that $y=1$. Given a set of $N$ trials, the liklihood of the sequence $y_{1}, y_{2}, \dots, y_{N}$ is given by:

$\mathcal{L} = \Pi_{i}^{N}p^{y_{i}}(1-p)^{1 - y_{i}}$

Given a set of labeled training data, the goal of maximum liklihood estimation is to determine a probability distribution that best recreates the empirical distribution of the training set.

The log-likelihood is the logarithmic transformation of the likelihood function.  As logarithms are strictly increasing functions, the resulting solution from maximizing the likelihood vs. the log-likelihood are the equivalent.  Given a dataset of cardinality $N$, the log-likelihood (normalized by $N$) is given by:

$l = \frac{1}{N}\sum_{i=1}^{N}\Big(y_{i}\log(p) + (1 - y_{i})\log(1 - p)\Big)$

## Logistic function

Logistic regression performs binary classification based on a probabilistic interpretation of the data.  Essentially, the process seeks to assign a probability to new observations.  If the probability associated with the new instance of data is greater than 0.5, then the new observation is assigned to 1 (for example).  If the probability associated with the new instance of the data is less than 0.5, then it is assigned to 0.  To map the real numerical values into probabilities (which must lie between 0 and 1), logistic regression makes use of the logistic (sigmoid) function, given by:

$\sigma(t) = \frac{1}{1 + e^{-t}}$

Note that by setting $t=0$, $\sigma(0) = 0.5$, which is the decision boundary.  We should also note that the derivative of the logistic function with respect to the parameter $t$ is:

$\frac{d}{dt}\sigma(t) = \sigma(t)(1 - \sigma(t))$

## Logistic Regression and Derivation of the Gradient

Let's assume the training data consists of $N$ observations, where observation $i$ is denoted by the pair $(y_{i},\mathbf{x}_{i})$, where $y \in \{0,1\}$ is the label for the feature vector $\mathbf{x}$.  We wish to compute a linear decision boundary that best seperates the labeled observations.  Let $\mathbf{\theta}$ denote the vector of coefficients to be estimated.  In this problem, the log likelihood can be expressed as:

$l =  \frac{1}{N}\sum_{i=1}^{N}\Big(y_{i}\log\big(\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big) + (1 - y_{i}) \log\big( 1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big)\Big)$

The gradient of the objective with respect to the $j^{th}$ element of $\mathbf{\theta}$ is:
$$
\begin{aligned}
\frac{d}{d\theta^{(j)}} l &=  \frac{1}{N}\sum_{i=1}^{N}\Bigg( \frac{d}{d\theta^{(j)}} y_{i}\log\big(\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big) + \frac{d}{d\theta^{(j)}}(1 - y_{i}) \log\big( 1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big)\Bigg) \\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i}}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} - \frac{1 - y_{i}}{1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} \Bigg)\frac{d}{d\theta^{(j)}}\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i}}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} - \frac{1 - y_{i}}{1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} \Bigg)\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)x_{i}^{(j)}\\
&=  \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i} - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)}\Bigg)\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)x_{i}^{(j)}\\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(y_{i} - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Bigg)x_{i}^{(j)}
\end{aligned}
$$

where the last equation has the familiar form of the product of the prediciton error and the $j^{th}$ feature.  With the gradient of the log likelihood function, the parameter vector $\mathbf{\theta}$ can now be calculated via gradient ascent (as we're maximizing the log likelihood):

$$
\begin{equation}
    \mathbf{\theta}^{(j)}(k+1) = \mathbf{\theta}^{(j)}(k) + \alpha \frac{1}{N}\sum_{i=1}^{N}\Bigg( y_{i} - \sigma(\mathbf{\theta}^{T}(k)\mathbf{x}_{i}))\Bigg)x_{i}^{(j)}
\end{equation}
$$

```python
# Supporting Methods

#logistic function
def sigmoid(a):
    return 1/(1 + np.exp(-a))

#ll function
def log_likelihood(x, y, theta):
    logits = np.dot(x, theta)
    log_like = np.sum(y * logits - np.log(1 + np.exp(logits)))
    return log_like
```

```python
#Load the data
iris = datasets.load_iris()
x = iris.data[:,2:]                    #features will be petal width and petal length
y = (iris.target==2).astype(np.int).reshape(len(x),1)    #1 of iris-virginica, and 0 ow

#Prepare Data for Regression
#pad x with a vector of ones for computation of intercept
x_aug = np.concatenate( (x,np.ones((len(x),1))) , axis=1)
```

```python
#sklearn logistic regression
log_reg = LogisticRegression(penalty='none')
log_reg.fit(x,y)
log_reg.get_params()
coefs = log_reg.coef_.reshape(-1,1)
intercept = log_reg.intercept_
theta_sklearn = np.concatenate((coefs, intercept.reshape(-1,1)), axis=0)
print("sklearn coefficients:")
print(theta_sklearn)
print("sklearn log likelihood: ", log_likelihood(x_aug, y, theta_sklearn))
```

    sklearn coefficients:
    [[  5.75452053]
     [ 10.44681116]
     [-45.27248307]]
    sklearn log likelihood:  -10.281754052558687

    C:\Users\danie\Anaconda3\Lib\site-packages\sklearn\utils\validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      return f(*args, **kwargs)

```python
#Perform Logistic Regression
num_iterations = 40000
alpha = 1e-2

theta0 = np.ones((3,1))
theta = []
theta.append(theta0)

k=0
while k < num_iterations:

    #compute prediction error
    e = y - sigmoid(np.dot(x_aug, theta[k]))

    #compute the gradient of the log-likelihood
    grad_ll = np.dot(x_aug.T, e)

    #gradient ascent
    theta.append(theta[k] + alpha * grad_ll)

    #update iteration step
    k += 1

    if k % 4000 == 0:
        #print("iteration: ", k, " delta: ", delta)
        print("iteration: ", k, " log_likelihood:", log_likelihood(x_aug, y, theta[k]))

theta_final = theta[k]
print("scratch coefficients:")
print(theta_final)
```

    iteration:  4000  log_likelihood: -11.529302688612685
    iteration:  8000  log_likelihood: -10.800986140073512
    iteration:  12000  log_likelihood: -10.543197464480874
    iteration:  16000  log_likelihood: -10.425775111214602
    iteration:  20000  log_likelihood: -10.36535749825992
    iteration:  24000  log_likelihood: -10.331961877189842
    iteration:  28000  log_likelihood: -10.312622172168293
    iteration:  32000  log_likelihood: -10.301055609225223
    iteration:  36000  log_likelihood: -10.293975480174714
    iteration:  40000  log_likelihood: -10.289566343598294
    scratch coefficients:
    [[  5.5044475 ]
     [ 10.17562424]
     [-43.60242548]]

```python
#Plot the data and the decision boundary

#create feature data for plotting
x_dec_bnd = np.linspace(0,7,100).reshape(-1,1)
#classification boundary from sklearn
y_sklearn = (theta_sklearn[2] * np.ones((100,1)) + theta_sklearn[0] * x_dec_bnd) / -theta_sklearn[1]
#classification boundary from scratch
y_scratch = (theta_final[2] * np.ones((100,1)) + theta_final[0] * x_dec_bnd) / -theta_final[1]

y_1 = np.where(y==1)[0] #training data, iris-virginica
y_0 = np.where(y==0)[0] #training data, not iris-virginica
plt.plot(x[y_0,0],x[y_0,1],'bo',label="not iris-virginica")
plt.plot(x[y_1,0],x[y_1,1],'k+',label="iris-virginica")
plt.plot(x_dec_bnd,y_sklearn,'r',label="sklearn dec. boundary")
plt.plot(x_dec_bnd,y_scratch,'g',label="scratch dec. boundary")
plt.xlabel('petal length')
plt.ylabel('petal width')
plt.title('Logistic Regression Classification')
plt.xlim((3,7))
plt.ylim((0,3.5))
plt.legend()
plt.show()
```


# Solving ODEs with scipy.integrate.solve_ivp

## Solving ordinary differential equations (ODEs)

Here we will revisit the differential equations solved in 5300_Jupyter_Python_intro_01.ipynb with `odeint`, only now we'll use `solve_ivp` from Scipy.  We'll compare the new and old solutions as we go.

### First-order ODE

```python
# Import the required modules
import numpy as np
import matplotlib.pyplot as plt

from scipy.integrate import solve_ivp   # Now preferred to odeint
```

Let's try a one-dimensional first-order ODE, say:

$\begin{align}
\quad
\frac{dv}{dt} = -g, \quad \mbox{with} \quad v(0) = 10
\end{align}$

in some appropriate units (we'll use MKS units by default).  This ODE can be separated and directly integrated:

$\begin{align}
  \int_{v_0=10}^{v} dv' = - g \int_{0}^{t} dt'
  \quad\Longrightarrow\quad
    v - v_0 = - g (t - 0)
  \quad\Longrightarrow\quad
   v(t) = 10 - gt
\end{align}$

The goal is to find the solution $v(t)$ as an array `v_pts` at the times in the array `t_pts`.

```python
# Define a function which calculates the derivative
def dv_dt_new(t, v, g=9.8):
    """Returns the right side of a simple first-order ODE with default g."""
    return -g

t_start = 0.
t_end = 10.
t_pts = np.linspace(t_start, t_end, 20)  # 20 points between t=0 and t=10.

v_0 = np.array([10.0])  # initial condition, in form of a list or numpy array

abserr = 1.e-8
relerr = 1.e-8

solution = solve_ivp(dv_dt_new, (t_start, t_end), v_0, t_eval=t_pts,
                     rtol=relerr, atol=abserr)
    # solve_ivp( function for rhs with (t, v) argument (cf. (v,t) for odeint),
    #            tspan=(starting t value, ending t value),
    #            initial value of v(t), array of points we want to know v(t),
    #            method='RK45' is the default method,
    #            rtol=1.e-3, atol=1.e-6 are default tolerances
    #          )
v_pts = solution.y  # array of results at t_pts
```

```python
v_pts.shape   # 1 x 100 matrix (row vector)
```

    (1, 20)

Here's how we did it before with odeint:

```python
from scipy.integrate import odeint

# Define a function which calculates the derivative
def dv_dt(v, t, g=9.8):
    """Returns the right side of a simple first-order ODE with default g."""
    return -g

t_pts = np.linspace(0., 10., 20)     # 20 points between t=0 and t=10.
v_0 = 10.  # the initial condition
v_pts_odeint = odeint(dv_dt, v_0, t_pts)  # odeint( function for rhs,
                                          #         initial value of v(t),
                                          #         array of t values )

```

```python
v_pts_odeint.shape   # 100 x 1 matrix (column vector)
```

    (20, 1)

Make a table comparing results (using `flatten()` to make the matrices into arrays):

```python
print('    t     v(t) [solve_ivp]    v(t) [odeint]')
for t, v_solve_ivp, v_odeint in zip(t_pts,
                                    v_pts.flatten(),
                                    v_pts_odeint.flatten()):
    print(f' {t:6.3f}   {v_solve_ivp:12.7f}       {v_odeint:12.7f}')
```

        t     v(t) [solve_ivp]    v(t) [odeint]
      0.000     10.0000000         10.0000000
      0.526      4.8421053          4.8421053
      1.053     -0.3157895         -0.3157895
      1.579     -5.4736842         -5.4736842
      2.105    -10.6315789        -10.6315789
      2.632    -15.7894737        -15.7894737
      3.158    -20.9473684        -20.9473684
      3.684    -26.1052632        -26.1052632
      4.211    -31.2631579        -31.2631579
      4.737    -36.4210526        -36.4210526
      5.263    -41.5789474        -41.5789474
      5.789    -46.7368421        -46.7368421
      6.316    -51.8947368        -51.8947368
      6.842    -57.0526316        -57.0526316
      7.368    -62.2105263        -62.2105263
      7.895    -67.3684211        -67.3684211
      8.421    -72.5263158        -72.5263158
      8.947    -77.6842105        -77.6842105
      9.474    -82.8421053        -82.8421053
     10.000    -88.0000000        -88.0000000

Differences between `solve_ivp` and `odeint`:
* `dv_dt(t, v)`  vs.  `dv_dt(v, t)`, i.e., the function definitions have the arguments reversed.
* With `odeint`, you only specify the full array of $t$ points you want to know $v(t)$ at.  With `solve_ivp`, you first specify the starting $t$ and ending $t$ as a tuple: `(t_start, t_end)` and then (optionally) specify `t_eval=t_pts` to evaluate $v$ at the points in the `t_pts` array.
* `solve_ivp` returns an object from which $v(t)$ (and other results) can be found, while `ode_int` returns $v(t)$.
* For this single first-order equation, $v(t)$ is returned for the $N$ requested $t$ points as a $1 \times N$ two-dimensional array by `solve_ivp` and as a $N \times 1$ array by `odeint`.
* `odeint` has no choice of solver while the `solve_ivp` solver can be set by `method`.  The default is `method='RK45'`, which is good, general-purpose Runge-Kutta solver.

### Second-order ODE

Suppose we have a second-order ODE such as:

$$
\quad y'' + 2 y' + 2 y = \cos(2x), \quad \quad y(0) = 0, \; y'(0) = 0
$$

We can turn this into two first-order equations by defining a new dependent variable. For example,

$$
\quad z \equiv y' \quad \Rightarrow \quad z' + 2 z + 2y = \cos(2x), \quad z(0)=y(0) = 0.
$$

Now introduce the vector

$$
  \mathbf{U}(x) = \left(\begin{array}{c}
                         y(x) \\
                         z(x)
                        \end{array}
                  \right)
        \quad\Longrightarrow\quad
    \frac{d\mathbf{U}}{dx} = \left(\begin{array}{c}
                                    z \\
                                    -2 y' - 2 y + \cos(2x)
                                   \end{array}
                             \right)
$$

We can solve this system of ODEs using `solve_ivp` with lists, as follows.  We will try it first without specifying the relative and absolute error tolerances rtol and atol.

```python
# Define a function for the right side
def dU_dx_new(x, U):
    """Right side of the differential equation to be solved.
    U is a two-component vector with y=U[0] and z=U[1].
    Thus this function should return [y', z']
    """
    return [U[1], -2*U[1] - 2*U[0] + np.cos(2*x)]

# initial condition U_0 = [y(0)=0, z(0)=y'(0)=0]
U_0 = [0., 0.]

x_pts = np.linspace(0, 15, 20)  # Set up the mesh of x points
result = solve_ivp(dU_dx_new, (0, 15), U_0, t_eval=x_pts)
y_pts = result.y[0,:]   # Ok, this is tricky.  For each x, result.y has two
                        #  components.  We want the first component for all
                        #  x, which is y(x).  The 0 means the first index and
                        #  the : means all of the x values.

```

Here's how we did it before with `odeint`:

```python
# Define a function for the right side
def dU_dx(U, x):
    """Right side of the differential equation to be solved.
    U is a two-component vector with y=U[0] and z=U[1].
    Thus this function should return [y', z']
    """
    return [U[1], -2*U[1] - 2*U[0] + np.cos(2*x)]

# initial condition U_0 = [y(0)=0, z(0)=y'(0)=0]
U_0 = [0., 0.]

x_pts = np.linspace(0, 15, 20)  # Set up the mesh of x points
U_pts = odeint(dU_dx, U_0, x_pts)  # U_pts is a 2-dimensional array
y_pts_odeint = U_pts[:,0]  # Ok, this is tricky.  For each x, U_pts has two
                           #  components.  We want the upper component for all
                           #  x, which is y(x).  The : means all of the first
                           #  index, which is x, and the 0 means the first
                           #  component in the other dimension.
```

Make a table comparing results (using `flatten()` to make the matrices into arrays):

```python
print('    x     y(x) [solve_ivp]    y(x) [odeint]')
for x, y_solve_ivp, y_odeint in zip(x_pts,
                                    y_pts.flatten(),
                                    y_pts_odeint.flatten()):
    print(f' {x:6.3f}   {y_solve_ivp:12.7f}       {y_odeint:12.7f}')
```

        x     y(x) [solve_ivp]    y(x) [odeint]
      0.000      0.0000000          0.0000000
      0.789      0.1360331          0.1360684
      1.579      0.0346990          0.0347028
      2.368     -0.2285865         -0.2287035
      3.158     -0.0975122         -0.0974702
      3.947      0.2065834          0.2067492
      4.737      0.0927010          0.0927536
      5.526     -0.2041225         -0.2042677
      6.316     -0.0865181         -0.0865921
      7.105      0.2064689          0.2066669
      7.895      0.0832986          0.0832707
      8.684     -0.2080277         -0.2081975
      9.474     -0.0799116         -0.0799972
     10.263      0.2092969          0.2094602
     11.053      0.0765611          0.0765810
     11.842     -0.2104970         -0.2107011
     12.632     -0.0731547         -0.0731411
     13.421      0.2117489          0.2118952
     14.211      0.0695941          0.0696868
     15.000     -0.2129369         -0.2130316

Not very close agreement by the end.  Run both again with greater accuracy.

```python
relerr = 1.e-10
abserr = 1.e-10

result = solve_ivp(dU_dx_new, (0, 15), U_0, t_eval=x_pts,
                   rtol=relerr, atol=abserr)
y_pts = result.y[0,:]

U_pts = odeint(dU_dx, U_0, x_pts,
               rtol=relerr, atol=abserr)
y_pts_odeint = U_pts[:,0]

print('    x     y(x) [solve_ivp]    y(x) [odeint]')
for x, y_solve_ivp, y_odeint in zip(x_pts,
                                    y_pts.flatten(),
                                    y_pts_odeint.flatten()):
    print(f' {x:6.3f}   {y_solve_ivp:12.7f}       {y_odeint:12.7f}')
```

        x     y(x) [solve_ivp]    y(x) [odeint]
      0.000      0.0000000          0.0000000
      0.789      0.1360684          0.1360684
      1.579      0.0347028          0.0347028
      2.368     -0.2287035         -0.2287035
      3.158     -0.0974702         -0.0974702
      3.947      0.2067492          0.2067492
      4.737      0.0927536          0.0927536
      5.526     -0.2042678         -0.2042678
      6.316     -0.0865921         -0.0865921
      7.105      0.2066669          0.2066669
      7.895      0.0832707          0.0832707
      8.684     -0.2081975         -0.2081975
      9.474     -0.0799972         -0.0799972
     10.263      0.2094602          0.2094602
     11.053      0.0765810          0.0765810
     11.842     -0.2107011         -0.2107011
     12.632     -0.0731411         -0.0731411
     13.421      0.2118952          0.2118952
     14.211      0.0696868          0.0696868
     15.000     -0.2130316         -0.2130316

Comparing the results from when we didn't specify the errors we see that the default error tolerances for solve_ivp were insufficient.  Moral: specify them explicitly.


# Introduction to Quantum Physics

### A complex Number

$c = a + ib$

Acircle of radius 1:  $e^{-i\theta}$

### Single Qubit System ($\mathcal{C}^{2}$ -space)

$|\psi \rangle = \alpha |0 \rangle + \beta | 1 \rangle $

$ \langle \psi | \psi \rangle = 1 \implies \alpha^{2} + \beta^{2} = 1 $

- Operators are 2 by 2 matrices, vectors are 2 by 1 column vectors.

#### General form of single qubit Unitary Operation

A single qubit quantum state can be written as

$$\left|\psi\right\rangle = \alpha\left|0\right\rangle + \beta \left|1\right\rangle$$

where $\alpha$ and $\beta$ are complex numbers. In a measurement the probability of the bit being in $\left|0\right\rangle$ is $|\alpha|^2$ and $\left|1\right\rangle$ is $|\beta|^2$. As a vector this is

$$
\left|\psi\right\rangle =
\begin{pmatrix}
\alpha \\
\beta
\end{pmatrix}
$$
Where
$$\left| 0 \right\rangle =
\begin{pmatrix}
1 \\
0
\end{pmatrix}; \left|1\right\rangle =
\begin{pmatrix}
0 \\
1
\end{pmatrix}.
$$

Note due to conservation probability $|\alpha|^2+ |\beta|^2 = 1$ and since global phase is undetectable $\left|\psi\right\rangle := e^{i\delta} \left|\psi\right\rangle$ we only requires two real numbers to describe a single qubit quantum state.

A convenient representation is

$$\left|\psi\right\rangle = \cos(\theta/2)\left|0\right\rangle + \sin(\theta/2)e^{i\phi}\left|1\right\rangle$$

where $0\leq \phi < 2\pi$, and $0\leq \theta \leq \pi$.  From this it is clear that there is a one-to-one correspondence between qubit states ($\mathbb{C}^2$) and the points on the surface of a unit sphere ($\mathbb{R}^3$). This is called the Bloch sphere representation of a qubit state.

Quantum gates/operations are usually represented as matrices. A gate which acts on a qubit is represented by a $2\times 2$ unitary matrix $U$. The action of the quantum gate is found by multiplying the matrix representing the gate with the vector which represents the quantum state.

$$\left|\psi'\right\rangle = U\left|\psi\right\rangle$$

A general unitary must be able to take the $\left|0\right\rangle$ to the above state. That is

$$
U = \begin{pmatrix}
\cos(\theta/2) & a \\
e^{i\phi}\sin(\theta/2) & b
\end{pmatrix}
$$

where $a$ and $b$ are complex numbers constrained such that $U^\dagger U = I$ for all $0\leq\theta\leq\pi$ and $0\leq \phi<2\pi$. This gives 3 constraints and as such $a\rightarrow -e^{i\lambda}\sin(\theta/2)$ and $b\rightarrow e^{i\lambda+i\phi}\cos(\theta/2)$ where $0\leq \lambda<2\pi$ giving

$$
U = \begin{pmatrix}
\cos(\theta/2) & -e^{i\lambda}\sin(\theta/2) \\
e^{i\phi}\sin(\theta/2) & e^{i\lambda+i\phi}\cos(\theta/2)
\end{pmatrix}.
$$

This is the most general form of a single qubit unitary.

### Quantum Gates:
Quanum gates are Unitary Transformation. There exist a universal set of quantum gates. Hadamard gate, X,Y,Z, paili gates, Cnot gate are few examples.

### Multiqubit System ($\mathcal{C}^{4}$ -space)

$|\psi \rangle = \alpha |00 \rangle + \beta | 01 \rangle + \gamma |10 \rangle + \delta | 11 \rangle $

$ \langle \psi | \psi \rangle = 1 \implies \alpha^{2} + \beta^{2} + \gamma^{2} + \delta^{2} = 1 $

- Operators are 4 by 4 matrices, vectors are 4 by 1 column vectors.

#### Realization of multi-qubit through single qubit

The space of quantum computer grows exponential with the number of qubits. For $n$ qubits the complex vector space has dimensions $d=2^n$. To describe states of a multi-qubit system, the tensor product is used to "glue together" operators and basis vectors.

Let's start by considering a 2-qubit system. Given two operators $A$ and $B$ that each act on one qubit, the joint operator $A \otimes B$ acting on two qubits is

$$\begin{equation}
	A\otimes B =
	\begin{pmatrix}
		A_{00} \begin{pmatrix}
			B_{00} & B_{01} \\
			B_{10} & B_{11}
		\end{pmatrix} & A_{01} 	\begin{pmatrix}
				B_{00} & B_{01} \\
				B_{10} & B_{11}
			\end{pmatrix} \\
		A_{10} 	\begin{pmatrix}
					B_{00} & B_{01} \\
					B_{10} & B_{11}
				\end{pmatrix} & A_{11} 	\begin{pmatrix}
							B_{00} & B_{01} \\
							B_{10} & B_{11}
						\end{pmatrix}
	\end{pmatrix},
\end{equation}$$

where $A_{jk}$ and $B_{lm}$ are the matrix elements of $A$ and $B$, respectively.

Analogously, the basis vectors for the 2-qubit system are formed using the tensor product of basis vectors for a single qubit:
$$\begin{equation}\begin{split}
	\left|{00}\right\rangle &= \begin{pmatrix}
		1 \begin{pmatrix}
			1  \\
			0
		\end{pmatrix} \\
		0 \begin{pmatrix}
			1  \\
			0
		\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 1 \\ 0 \\ 0 \\0 \end{pmatrix}~~~\left|{01}\right\rangle = \begin{pmatrix}
	1 \begin{pmatrix}
	0 \\
	1
	\end{pmatrix} \\
	0 \begin{pmatrix}
	0  \\
	1
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix}0 \\ 1 \\ 0 \\ 0 \end{pmatrix}\end{split}
\end{equation}$$

$$\begin{equation}\begin{split}\left|{10}\right\rangle = \begin{pmatrix}
	0\begin{pmatrix}
	1  \\
	0
	\end{pmatrix} \\
	1\begin{pmatrix}
	1 \\
	0
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 1 \\ 0 \end{pmatrix}~~~ 	\left|{11}\right\rangle = \begin{pmatrix}
	0 \begin{pmatrix}
	0  \\
	1
	\end{pmatrix} \\
	1\begin{pmatrix}
	0  \\
	1
	\end{pmatrix}
	\end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \\1 \end{pmatrix}\end{split}
\end{equation}.$$

Note we've introduced a shorthand for the tensor product of basis vectors, wherein $\left|0\right\rangle \otimes \left|0\right\rangle$ is written as $\left|00\right\rangle$. The state of an $n$-qubit system can described using the $n$-fold tensor product of single-qubit basis vectors. Notice that the basis vectors for a 2-qubit system are 4-dimensional; in general, the basis vectors of an $n$-qubit sytsem are $2^{n}$-dimensional.

### Superposition and Entanglement

Superposition in 2 qubit system : $|\psi \rangle = \alpha |00 \rangle + \beta | 01 \rangle + \gamma |10 \rangle + \delta | 11 \rangle $

- It can be written in direct product of lower qubit states.

Entanglement in two qubit system:

An entanglement circuit:

- It can not be written in direct product of lower qubit states

$\begin{bmatrix}
    p   \\
    q
\end{bmatrix} \otimes \begin{bmatrix}
    r   \\
    s
\end{bmatrix} = c \begin{bmatrix}
    m   \\
    0 \\
    0 \\
    n
\end{bmatrix}$

### Quantum Circuits and Quantum Algorithms

Execution of Quantum Algorithm (A unitary matrix), is challangeing task. This is because one need to find out product of single or multi-qubit gates to represent that algorithm.

$$
QFT: F_{N} =  \frac{1}{\sqrt{N}} \left( \begin{array}{cccccc}
    1  &         1   &        1       &     1          &   \cdots   &  1   \\
    1  &  \omega_{n} & \omega_{n}^{2} & \omega_{n}^{3} &   \cdots   & \omega_{n} ^{N-1}\\
    1  &  \omega_{n}^{2} & \omega_{n}^{4} & \omega_{n}^{6} &   \cdots   & \omega_{n} ^{2(N-1)}\\
    1  &  \omega_{n}^{3} & \omega_{n}^{6} & \omega_{n}^{9} &   \cdots   & \omega_{n} ^{3(N-1)}\\
   \vdots  & \vdots  & \vdots         & \vdots         &    \dots   & \vdots \\
   1  &  \omega_{n}^{(N-1)} & \omega_{n}^{2(N-1)} & \omega_{n}^{3(N-1)} &   \cdots   & \omega_{n} ^{(N-1((N-1)}\\
\end{array}\right )
$$

Figure: Execution of QFT algorithm

### Measurement

- Only observable can be measured.
- What is being measured? E, P or X? Non of these are being measured. We measure probability through $<\psi | \psi>$.
- After measurement, system collapse to one of the eigen state. Due to time evolution, it will superposed to many states in future.

### Noise and Error

- Current time Quantum Computers are **Noisy Intermediate-Scale Quantum (NISQ)**
- Decoherence : Quantum decoherence is the loss of quantum coherence. Decoherence can be viewed as the loss of information from a system into the environment (often modeled as a heat bath).
- Circuit depth: The depth of a circuit is the longest path in the circuit. The path length is always an integer number, representing the number of gates it has to execute in that path.
- Fidelity: fidelity is the measure of the distance between two quantum states. Fidelity equal to 1, means that two states are equal. In the case of a density matrix, fidelity represents the overlap with a reference pure state.

```python

```


```python
import numpy as np
import sympy as sy
```

```python
s,z = sy.symbols('s,z', real=False)
l_1, l_2 = sy.symbols('l1, l2')
```

```python
Phi = sy.Matrix([[1.13, 0.52], [0.52, 1.13]])
Gamma = sy.Matrix([[0.13],[0.52]])
L = sy.Matrix([[l_1, l_2]])

M = z*sy.Matrix.eye(2) - (Phi - Gamma*L)
M
```

    Matrix([
    [0.13*l1 + z - 1.13,     0.13*l2 - 0.52],
    [    0.52*l1 - 0.52, 0.52*l2 + z - 1.13]])

```python
chPoly = sy.poly(M.det(), z)
chPoly
```

    Poly(1.0*z**2 + (0.13*l1 + 0.52*l2 - 2.26)*z + 0.1235*l1 - 0.52*l2 + 1.0065, z, domain='RR[l1,l2]')

```python
chDesired = sy.simplify(sy.expand((z-np.exp(-0.5))**2))

sol = sy.solve((chPoly - chDesired).coeffs(), [l_1, l_2])
```

```python
sol
```

    {l1: 1.61072237375216, l2: 1.61066302305182}

```python

```


# Weyl Tensor calculations using Symbolic module

```python
import sympy
from sympy import cos, sin, sinh
from einsteinpy.symbolic import MetricTensor, WeylTensor

sympy.init_printing()
```

### Defining the Anti-de Sitter spacetime Metric

```python
syms = sympy.symbols("t chi theta phi")
t, ch, th, ph = syms
m = sympy.diag(-1, cos(t) ** 2, cos(t) ** 2 * sinh(ch) ** 2, cos(t) ** 2 * sinh(ch) ** 2 * sin(th) ** 2).tolist()
metric = MetricTensor(m, syms)
```

### Calculating the Weyl Tensor (with all indices covariant)

```python
weyl = WeylTensor.from_metric(metric)
weyl.tensor()
```

```python
weyl.config
```

    'llll'

docs/source/examples/Weyl Tensor symbolic calculation.ipynb


##Ejercicio 4 Practica 1
Para cada uno de los siguientes sistemas encontrar todos los puntos de equilibrio y determinar el tipo de cada punto de equilibio aislado.
 * b)

 $$\left\{ \begin{array}{lcc}
            \dot{x}_{1}=-x_{1}+x_{2}\\
             \\ \dot{x}_{2}=\frac{x_{1}}{10}-2x_{1}-x_{1}^{2}-\frac{x_{1}^{3}}{10}
             \end{array}
\right.$$

```python
import sympy as sym
```

```python
#Con esto las salidas van a ser en LaTeX
sym.init_printing(use_latex=True)
```

```python
x_1, x_2 = sym.symbols('x_1 x_2')
```

```python
X = sym.Matrix([x_1, x_2])
X
```

```python
f_1 = -x_1 + x_2
```

```python
f_2 = sym.Rational(1,10) * x_1 - 2 * x_2 - x_1 ** 2 - sym.Rational(1,10) * x_1 ** 3
```

```python
F = sym.Matrix([f_1,f_2])
```

```python
F
```

```python
A = F.jacobian(X)
A
```

```python
# puntos de equilibrio del sistema
pes = sym.solve([f_1,f_2])
pes
```

```python
type(pes[0])
```

    dict

```python
A_1 = A.subs(pes[0])
A_1
```

```python
A_1.eigenvals()
```

```python
A_2 = A.subs(pes[1])
A_2
```

```python
lambda_2 = A_2.eigenvals()
lambda_2
```

```python
sym.N(lambda_2.keys()[0])
```

```python
sym.N(lambda_2.keys()[1])
```

```python
A_3 = A.subs(pes[2])
A_3
```

```python
lambda_3 = A_3.eigenvals()
lambda_3
```

```python
sym.N(lambda_3.keys()[0])
```

```python
sym.N(lambda_3.keys()[1])
```

```python

```


# Solutions

## Question 1

> `1`. Simplify the following expressions:

> $\frac{3}{\sqrt{3}}$:

```python
import sympy as sym

expression = sym.S(3) / sym.sqrt(3)
sym.simplify(expression)
```

$\displaystyle \sqrt{3}$

> $\frac{2 ^ {78}}{2 ^ {12}2^{-32}}$:

```python
sym.S(2) ** 78 / (sym.S(2) ** 12 * sym.S(2) ** (-32))
```

$\displaystyle 316912650057057350374175801344$

> $8^0$:

```python
sym.S(8) ** 0
```

$\displaystyle 1$

> $a^4b^{-2}+a^{3}b^{2}+a^{4}b^0$:

```python
a = sym.Symbol("a")
b = sym.Symbol("b")
sym.factor(a ** 4 * b ** (-2) + a ** 3 * b ** 2 + a ** 4 * b ** 0)
```

$\displaystyle \frac{a^{3} \left(a b^{2} + a + b^{4}\right)}{b^{2}}$

## Question 2

> `2`. Solve the following equations:

> $x + 3 = -1$:

```python
x = sym.Symbol("x")
equation = sym.Eq(x + 3, -1)
sym.solveset(equation, x)
```

$\displaystyle \left\{-4\right\}$

> $3 x ^ 2 - 2 x = 5$:

```python
equation = sym.Eq(3 * x ** 2 - 2 * x, 5)
sym.solveset(equation, x)
```

$\displaystyle \left\{-1, \frac{5}{3}\right\}$

> $x (x - 1) (x + 3) = 0$:

```python
equation = sym.Eq(x * (x - 1) * (x + 3), 0)
sym.solveset(equation, x)
```

$\displaystyle \left\{-3, 0, 1\right\}$

> $4 x ^3 + 7x - 24 = 1$:

```python
equation = sym.Eq(4 * x ** 3 + 7 * x - 24, 1)
sym.solveset(equation, x)
```

$\displaystyle \left\{- \frac{7}{12 \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}} + \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}, - \frac{\sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}}{2} + \frac{7}{24 \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}} + i \left(\frac{7 \sqrt{3}}{24 \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}} + \frac{\sqrt{3} \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}}{2}\right), - \frac{\sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}}{2} + \frac{7}{24 \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}} + i \left(- \frac{\sqrt{3} \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}}{2} - \frac{7 \sqrt{3}}{24 \sqrt[3]{\frac{25}{8} + \frac{\sqrt{51654}}{72}}}\right)\right\}$

## Question 3

> `3`. Consider the equation: $x ^ 2 + 4 - y = \frac{1}{y}$:

> Find the solution to this equation for $x$.

```python
y = sym.Symbol("y")
equation = sym.Eq(x ** 2 + 4 - y, 1 / y)
solution = sym.solveset(equation, x)
solution
```

$\displaystyle \left\{- \sqrt{\frac{y^{2} - 4 y + 1}{y}}, \sqrt{\frac{y^{2} - 4 y + 1}{y}}\right\}$

> Obtain the specific solution when $y = 5$. Do this in two ways:
> substitute the value in to your equation and substitute the value in to
> your solution.

```python
solution.subs({y: 5})
```

$\displaystyle \left\{- \frac{\sqrt{30}}{5}, \frac{\sqrt{30}}{5}\right\}$

```python
solution = sym.solveset(equation.subs({y: 5}), x)
solution
```

$\displaystyle \left\{- \frac{\sqrt{30}}{5}, \frac{\sqrt{30}}{5}\right\}$

## Question 4

> `4`. Consider the quadratic: $f(x)=4x ^ 2 + 16x + 25$:

> Calculate the discriminant of the quadratic equation $4x ^ 2 + 16x + 25 =
> 0$. What does this tell us about the solutions to the equation? What
> does this tell us about the graph of $f(x)$?

```python
quadratic = 4 * x ** 2 + 16 * x + 25
sym.discriminant(quadratic)
```

$\displaystyle -144$

This is negative so we know that the equation does not have any real solutions and
hence the graph does not cross the x-axis.
Since the coefficient of $x^2$ is positive it means that the graph is above
the $y=0$ line.

> By completing the square, show that the minimum point of $f(x)$ is
> $\left(-2, 9\right)$

```python
a, b, c = sym.Symbol("a"), sym.Symbol("b"), sym.Symbol("c")
completed_square = a * (x - b) ** 2 + c
sym.expand(completed_square)
```

$\displaystyle a b^{2} - 2 a b x + a x^{2} + c$

This gives $a=4$.

```python
completed_square = completed_square.subs({a: 4})
sym.expand(completed_square)
```

$\displaystyle 4 b^{2} - 8 b x + c + 4 x^{2}$

Comparing the coefficients of $x$ we have the equation:

$$
  - 8 b = 16
$$

```python
equation = sym.Eq(-8 * b, 16)
sym.solveset(equation, b)
```

$\displaystyle \left\{-2\right\}$

Substituting:

```python
completed_square = completed_square.subs({b: -2})
sym.expand(completed_square)
```

$\displaystyle c + 4 x^{2} + 16 x + 16$

Comparing the coefficients of $x^0$ this gives:

$$c+16=25$$

```python
equation = sym.Eq(c + 16, 25)
sym.solveset(equation, c)
```

$\displaystyle \left\{9\right\}$

```python
completed_square = completed_square.subs({c: 9})
completed_square
```

$\displaystyle 4 \left(x + 2\right)^{2} + 9$

The lowest value of $f(x)$ is for $x=-2$ which gives: $f(-2)=9$ as expected.

## Question 5

> `5`. Consider the quadratic: $f(x)=-3x ^ 2 + 24x - 97$:

> Calculate the discriminant of the quadratic equation $-3x ^ 2 + 24x - 97 =
> 0$. What does this tell us about the solutions to the equation? What
> does this tell us about the graph of $f(x)$?

```python
quadratic = -3 * x ** 2 + 24 * x - 97
sym.discriminant(quadratic)
```

$\displaystyle -588$

This is negative so we know that the equation does not have any real solutions and
hence the graph does not cross the x-axis.
Since the coefficient of $x^2$ is negative it means that the graph is below
the $y=0$ line.

> By completing the square, show that the maximum point of $f(x)$ is
> $\left(4, -49\right)$

```python
a, b, c = sym.Symbol("a"), sym.Symbol("b"), sym.Symbol("c")
completed_square = a * (x - b) ** 2 + c
sym.expand(completed_square)
```

$\displaystyle a b^{2} - 2 a b x + a x^{2} + c$

This gives $a=-3$.

```python
completed_square = completed_square.subs({a: -3})
sym.expand(completed_square)
```

$\displaystyle - 3 b^{2} + 6 b x + c - 3 x^{2}$

Comparing the coefficients of $x$ we have the equation:

$$
  6 b = 24
$$

```python
equation = sym.Eq(6 * b, 24)
sym.solveset(equation, b)
```

$\displaystyle \left\{4\right\}$

Substituting:

```python
completed_square = completed_square.subs({b: 4})
sym.expand(completed_square)
```

$\displaystyle c - 3 x^{2} + 24 x - 48$

Comparing the coefficients of $x^0$ this gives:

$$c-48=-97$$

```python
equation = sym.Eq(c - 48, -97)
sym.solveset(equation, c)
```

$\displaystyle \left\{-49\right\}$

```python
completed_square = completed_square.subs({c: -49})
completed_square
```

$\displaystyle - 3 \left(x - 4\right)^{2} - 49$

The highest value of $f(x)$ is for $x=4$ which gives: $f(4)=-49$ as expected.

`6`. Consider the function $f(x) = x^ 2 + a x + b$.

> Given that $f(0) = 0$ and $f(3) = 0$ obtain the values of $a$ and $b$.

Substituting 0 in to $f$ gives:

```python
expression = x ** 2 + a * x + b
expression.subs({x: 0})
```

$\displaystyle b$

This implies that $b=0$. Substituting back in to the expression:

```python
expression = expression.subs({b: 0})
expression
```

$\displaystyle a x + x^{2}$

Substituting $x=3$ in to this expression gives:

```python
expression.subs({x: 3})
```

$\displaystyle 3 a + 9$

This gives the equation:

$$
    3 a + 9 = 0
$$

```python
sym.solveset(expression.subs({x: 3}), a)
```

$\displaystyle \left\{-3\right\}$

Our expression is thus:

```python
expression = expression.subs({a: -3})
expression
```

$\displaystyle x^{2} - 3 x$

> By completing the square confirm that graph of $f(x)$ has a line of symmetry
> at $x=\frac{3}{2}$

```python
completed_square = a * (x - b) ** 2 + c
sym.expand(completed_square)
```

$\displaystyle a b^{2} - 2 a b x + a x^{2} + c$

We see that $a=1$ and. Substituting:

```python
completed_square = completed_square.subs({a: 1})
sym.expand(completed_square)
```

$\displaystyle b^{2} - 2 b x + c + x^{2}$

This gives:

$$
    -2b=-3
$$

```python
equation = sym.Eq(-2 * b, -3)
sym.solveset(equation, b)
```

$\displaystyle \left\{\frac{3}{2}\right\}$

Substituting:

```python
completed_square = completed_square.subs({b: sym.S(3) / 2})
sym.expand(completed_square)
```

$\displaystyle c + x^{2} - 3 x + \frac{9}{4}$

Which gives:

$$
    c + 9 / 4 = 0
$$

```python
equation = sym.Eq(c + sym.S(9) / 4, 0)
sym.solveset(equation, c)
```

$\displaystyle \left\{- \frac{9}{4}\right\}$

Substituting:

```python
completed_square = completed_square.subs({c: -sym.S(9) / 4})
completed_square
```

$\displaystyle \left(x - \frac{3}{2}\right)^{2} - \frac{9}{4}$

Thus $x=3/2$ is a line of symmetry.