```python
import jax
import jax.numpy as jnp
from IPython.display import display, Latex
```

## If else condition with `lax`

$$
f(\mathbf{x}) = \sum_{x \in \mathbf{x}} \begin{cases}
    x^2,& \text{if } x \gt 5\\
    x^3,             & \text{otherwise}
\end{cases}
$$

```python
x = [jnp.array(10.0), jnp.array(2.0)]

@jax.jit
@jax.value_and_grad
def f(x):
  bool_val = jax.tree_map(lambda val: val > 5.0, x)
  ans = jax.tree_map(lambda val, bool: jax.lax.cond(bool, lambda: val**2, lambda: val**3), x, bool_val)
  return jax.tree_util.tree_reduce(lambda a, b: a + b, ans)

value, grad = f(x)

display(Latex(f"$f(\mathbf{{x}}) = {value}$"))
for idx in range(len(x)):
  display(Latex(f"$\\frac{{df}}{{dx_{idx}}} = {grad[idx]}$"))
```

    WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

$f(\mathbf{x}) = 108.0$

$\frac{df}{dx_0} = 20.0$

$\frac{df}{dx_1} = 12.0$

## Pair-wise distance with `vmap`

```python
# create vour pairwise function
def distance(a, b):
    return jnp.linalg.norm(a - b)

# map based combinator to operate on all pairs
def all_pairs(f):
    f = jax.vmap(f, in_axes=(None, 0))
    f = jax.vmap(f, in_axes=(0, None))
    return f

# transform to operate over sets
distances = all_pairs(distance)

# Example
x = jnp.array([1.0, 2.0, 3.0])
y = jnp.array([3.0, 4.0, 5.0])
distances(x, y)
```

    DeviceArray([[2., 3., 4.],
                 [1., 2., 3.],
                 [0., 1., 2.]], dtype=float32)

## Compute Hessian with `jax`

Let us consider Linear regression loss function

\begin{align}
\mathcal{L}(\boldsymbol{\theta}) &= (\boldsymbol{y} - X\boldsymbol{\theta})^T(\boldsymbol{y} - X\boldsymbol{\theta})\\
\frac{d\mathcal{L}}{d\boldsymbol{\theta}} &= -2X^T\boldsymbol{y} + 2X^TX\boldsymbol{\theta}\\
H_{\mathcal{L}}(\boldsymbol{\theta}) &= 2X^TX
\end{align}

```python
def loss_function_per_point(theta, x, y):
  y_pred = x.T@theta
  return jnp.square(y_pred - y)

def loss_function(theta, x, y):
  loss_per_point = jax.vmap(loss_function_per_point, in_axes=(None, 0, 0))(theta, x, y)
  return jnp.sum(loss_per_point)

def gt_loss(theta, x, y):
  return jnp.sum(jnp.square(x@theta - y))

def gt_grad(theta, x, y):
  return 2 * (x.T@x@theta - x.T@y)

def gt_hess(theta, x, y):
  return 2 * x.T@x
```

### Simulate dataset

```python
key = jax.random.PRNGKey(0)
key, subkey1, subkey2 = jax.random.split(key, num=3)
N = 100
D = 11
x = jax.random.uniform(key, shape=(N, D))
y = jax.random.uniform(subkey1, shape=(N,))
theta = jax.random.uniform(subkey2, shape=(D,))
```

### Verify loss and gradient values

```python
loss_and_grad_function = jax.value_and_grad(loss_function)

loss_val, grad = loss_and_grad_function(theta, x, y)

assert jnp.allclose(loss_val, gt_loss(theta, x, y))
assert jnp.allclose(grad, gt_grad(theta, x, y))
```

### Verify hessian matrix

#### Way-1

```python
hess = jax.hessian(loss_function)(theta, x, y)

assert jnp.allclose(hess, gt_hess(theta, x, y))
```

#### Way-2

```python
hess = jax.jacfwd(jax.jacrev(loss_function))(theta, x, y)

assert jnp.allclose(hess, gt_hess(theta, x, y))
```


# What's up with polynomial regression?

Why do we have to use this `PolynomialFeatures` thing from scikit? What does it do?

Let's imagine we have some data from which we know the true function we want our model to learn. This function is:
\begin{align}
y = 3x + 1x^2 -2
\end{align}

```python
import numpy
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

x_values = numpy.array([[-2], [-1], [0], [1], [2]])
y_values = numpy.array([[-4], [-4], [-2], [2], [8]])
```

```python
x_values
```

    array([[-2],
           [-1],
           [ 0],
           [ 1],
           [ 2]])

Polynomial regression extends the linear model by adding extra predictors, obtained by raising each of the original predictors to a power. Our original predictors were:
```
[-2, -1, 0, 1, 2]
```

```python
transformer = PolynomialFeatures(degree=2)
x_values_transformed = transformer.fit_transform(x_values)
```

```python
x_values_transformed
```

    array([[ 1., -2.,  4.],
           [ 1., -1.,  1.],
           [ 1.,  0.,  0.],
           [ 1.,  1.,  1.],
           [ 1.,  2.,  4.]])

What the transformer has done for each value of x is to expand it from a single number into an array for three numbers:

1. The bias (always 1.0, the feature in which all polynomial powers are zero )
2. The original value
3. The value, squared

It has *extended the linear model* by adding extra predictors.

We can now hand off the transformed values off to the linear regression model's ``fit`` method and it will understand that it needs to fit a second-degree polynomial instead of a straight line.

```python
model = LinearRegression()
model.fit(x_values_transformed,y_values)
```

    LinearRegression()

Now we can predict. In the same way we transformed out `x` inputs with the `PolynomialFeatures` class before training the model, we will need to transform any `x` values for which we want to predict a `y`:

```python
# Values of x for which we want to predict a y
x_pred = [[3], [-3]]

x_pred_transformed = transformer.fit_transform(x_pred)

model.predict(x_pred_transformed)
```

    array([[16.],
           [-2.]])

Are these correct?

\begin{align}
y = (3 * 3) + 3^2 -2 \\
y = (3 * -3) + -3^2 -2
\end{align}

Yes, they are! Our model correctly learned the function! If you still aren't convinced from just two examples that the model has correctly learned the true function:

```python
intercept = model.intercept_[0]
slope = model.coef_[0]
print(f"Intercept is: {intercept}")
print(f"Slope is: {slope}")
```

    Intercept is: -2.000000000000004
    Slope is: [0. 3. 1.]

And our true function, again:
\begin{align}
y = 3x + 1x^2 -2
\end{align}

```python

```


# 2022-02-14 QR Factorization

* Sahitya's office hours: Friday 11-12:30

## Last time

* Revisit projections, rotations, and reflections
* Constructing orthogonal bases

## Today

* Gram-Schmidt process
* QR factorization
* Stability and ill conditioning
* Intro to performance modeling

```julia
using LinearAlgebra
using Plots
using Polynomials
default(linewidth=4, legendfontsize=12)

function vander(x, k=nothing)
    if isnothing(k)
        k = length(x)
    end
    m = length(x)
    V = ones(m, k)
    for j in 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end
```

    vander (generic function with 2 methods)

# Gram-Schmidt orthogonalization

Suppose we're given some vectors and want to find an orthogonal basis for their span.

$$ \Bigg[ a_1 \Bigg| a_2 \Bigg] = \Bigg[ q_1 \Bigg| q_2 \Bigg] \begin{bmatrix} r_{11} & r_{12} \\ 0 & r_{22} \end{bmatrix} $$

# A naive algorithm

```julia
function gram_schmidt_naive(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        for k in 1:j-1
            r = Q[:,k]' * v
            v -= Q[:,k] * r
            R[k,j] = r
        end
        R[j,j] = norm(v)
        Q[:,j] = v / R[j,j]
    end
    Q, R
end
```

    gram_schmidt_naive (generic function with 1 method)

```julia
x = LinRange(-1, 1, 10)
A = vander(x, 4)
Q, R = gram_schmidt_naive(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A);
```

    norm(Q' * Q - I) = 3.684346652564232e-16
    norm(Q * R - A) = 1.6779947319649215e-16

# What do orthogonal polynomials look like?

```julia
x = LinRange(-1, 1, 200)
A = vander(x, 6)
Q, R = gram_schmidt_naive(A)
plot(x, Q)
```

What happens if we use more than 50 values of $x$? Is there a continuous limit?

# Theorem
## Every full-rank $m\times n$ matrix ($m \ge n$) has a unique reduced $Q R$ factorization with $R_{j,j} > 0$

The algorithm we're using generates this matrix due to the line:
```julia
        R[j,j] = norm(v)
```

# Solving equations using $QR = A$

If $A x = b$ then $Rx = Q^T b$.

```julia
x1 = [-0.9, 0.1, 0.5, 0.8] # points where we know values
y1 = [1, 2.4, -0.2, 1.3]
scatter(x1, y1)
A = vander(x1, 3)
Q, R = gram_schmidt_naive(A)
p = R \ (Q' * y1)
p = A \ y1
plot!(x, vander(x, 3) * p)
```

# How accurate is it?

```julia
m = 30
x = LinRange(-1, 1, m)
A = vander(x, m)
Q, R = gram_schmidt_naive(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 0.00043830040698919693
    norm(Q * R - A) = 1.25091691161899e-15

    1.25091691161899e-15

# A variant with more parallelism

\begin{align}
(I - q_2 q_2^T) (I - q_1 q_1^T) v &= (I - q_1 q_1^T - q_2 q_2^T + q_2 q_2^T q_1 q_1^T) v \\
&= \Bigg( I - \Big[ q_1 \Big| q_2 \Big] \begin{bmatrix} q_1^T \\ q_2^T \end{bmatrix} \Bigg) v
\end{align}

```julia
function gram_schmidt_classical(A)
    m, n = size(A)
    Q = zeros(m, n)
    R = zeros(n, n)
    for j in 1:n
        v = A[:,j]
        R[1:j-1,j] = Q[:,1:j-1]' * v
        v -= Q[:,1:j-1] * R[1:j-1,j]
        R[j,j] = norm(v)
        Q[:,j] = v / norm(v)
    end
    Q, R
end
```

    gram_schmidt_classical (generic function with 1 method)

```julia
m = 10
x = LinRange(-1, 1, m)
A = vander(x, m)
Q, R = gram_schmidt_classical(A)
@show norm(Q' * Q - I)
@show norm(Q * R - A)
```

    norm(Q' * Q - I) = 6.339875256299394e-11
    norm(Q * R - A) = 1.217027619812654e-16

    1.217027619812654e-16

# Cost of Gram-Schmidt?

* We'll count flops (addition, multiplication, division*)
* Inner product $\sum_{i=1}^m x_i y_i$?
* Vector "axpy": $y_i = a x_i + y_i$, $i \in [1, 2, \dotsc, m]$.
* Look at the inner loop:
```julia
for k in 1:j-1
    r = Q[:,k]' * v
    v -= Q[:,k] * r
    R[k,j] = r
end
```

# Counting flops is a bad model

* We load a single entry (8 bytes) and do 2 flops (add + multiply). That's an **arithmetic intensity** of 0.25 flops/byte.
* Current hardware can do about 10 flops per byte, so our best algorithms will run at about 2% efficiency.
* Need to focus on memory bandwidth, not flops.

## Dense matrix-matrix mulitply

* [BLIS project](https://github.com/flame/blis/)
* [Analytic modeling](https://www.cs.utexas.edu/users/flame/pubs/TOMS-BLIS-Analytical.pdf)


# Hopf Bifurcation: The Emergence of Limit-cycle Dynamics

*Cem Özen*, May 2017.

A *Hopf bifurcation* is a critical point in which a periodic orbit appears or disappears through a local change in the stability of a fixed point in a dynamical system as one of the system parameters is varied. Hopf bifurcations occur in many of the well-known dynamical systems such as the Lotka-Volterra model, the Lorenz model, the Selkov model of glycolysis, the Belousov-Zhabotinsky reaction model, and the Hodgkin-Huxley model for nerve membrane.

In this notebook, I will consider a system of chemical reactions known by the name *Brusselator* in literature (see: https://en.wikipedia.org/wiki/Brusselator for more information) as a model for Hopf bifurcations. The Brusselator reactions are given by

$A \longrightarrow X$

$2X + Y\longrightarrow 3X$

$B + X \longrightarrow Y + D$

$X \longrightarrow E$

For the sake of simplicity, we will assume that the reaction constants of all these reactions are unity (i.e. in all the reactions, $k=1$ ). Furthermore let's assume that the reactant concentrations $A$ and $B$ are so large that  they remain constant. Therefore, only $X$ and $Y$ concentrations will be dynamical.

The rate equations for $X$ and $Y$ are then given by

$$
\begin{eqnarray}
\dot{X} & = & A + X^2Y - BX - X, \\
\dot{Y} & = & BX - X^2Y
\end{eqnarray}
$$

The X-nullcline and the Y-nullcline are given by the conditions of  $0 =  A + X^2Y - BX - X$ and $0 =  BX - X^2Y$ respectively. From these equations, we obtain:

$$
\begin{eqnarray}
Y(X) & = & \frac{-A + X(B+1)}{X^2},  & \quad \textrm{(X-nullcline)}  \\
Y(X) & = & \frac{B}{X}, & \quad   \textrm{(Y-nullcline)}
\end{eqnarray}
$$

In this notebook, I will also demonstrate how one can perform symbolical computations using Python's `SymPy` library. We also need extra Jupyter Notebook functionality to render nice display of the resulting equations. (Notice that we are using LaTex in typesetting this document particularly for the purpose of producing nice looking equations).

```python
import numpy as np
from numpy.linalg import eig
from scipy import integrate
import sympy
from IPython.display import display, Math, Latex
import matplotlib.pyplot as plt
sympy.init_printing(use_latex='mathjax')
%matplotlib inline
```

Let's obtain the nullcline equations using `SymPy`:

```python
X, Y, A, B = sympy.symbols('X Y A B')   # you need to introduce the sysmbols first

# let's get the X-nullcline as a function of X:
sympy.solve(sympy.Eq(A + X**2 * Y - B * X - X, 0), Y)
```

$$\left [ \frac{1}{X^{2}} \left(- A + X \left(B + 1\right)\right)\right ]$$

```python
# let's get the Y-nullcline as a function of X:
sympy.solve(sympy.Eq(B * X - X**2 * Y, 0), Y)
```

$$\left [ \frac{B}{X}\right ]$$

Now let's find the fixed points ($X^*, Y^*$) of this 2-D system (there is only one, actually). The fixed point is given by the simultaneous solution of the X-nullcline and Y-nullcline equations, therefore

$$ (X^*, Y^*)  =  (A, \frac{B}{A}) $$

For the sake of using `SymPy`, let's obtain this solution once again:

```python
# Solve the system of equations defined by the X-nullcline and Y-nullcline with respect to X and Y:
sympy.solve([A + X**2 * Y - B * X - X,  B * X - X**2 * Y], [X, Y])
```

$$\left [ \left ( A, \quad \frac{B}{A}\right )\right ]$$

Now, a bifurcation analysis of the Brusselator model requires us to keep track of the local stability of its fixed point. This can be done, according to the *linearized stability analysis* by evaluating the Jacobian  matrix at the fixed point.

The Jacobian matrix at the fixed point is given by:

$$
\begin{eqnarray}
J & = &  \left\vert\matrix{{\partial f \over \partial x} & {\partial f\over \partial y}  \cr
                           {\partial g \over \partial x} & {\partial g\over \partial y}
                           }\right\vert_{(X^*, Y^*)} \\
  & = & \left\vert\matrix{ -B + 2XY - 1 & X^2  \cr
                            B - 2XY & -X^2
                          }\right\vert_{(X^*, Y^*)}  \\
  & = & \left\vert\matrix{ B - 1 & A^2  \cr
                           -B & -A^2
                          }\right\vert
\end{eqnarray}
$$

This result can also be obtained very easily using `SymPy`:

```python
# define the Brusselator dynamical system as a SymPy matrix
brusselator = sympy.Matrix([A + X**2 * Y - B * X - X, B * X - X**2 * Y])
```

```python
# Jacobian matrix with respect to X and Y
J = brusselator.jacobian([X, Y])
J
```

$$\left[\begin{matrix}- B + 2 X Y - 1 & X^{2}\\B - 2 X Y & - X^{2}\end{matrix}\right]$$

```python
# Jacobian matrix evaluated at the coordinates of the fixed point
J_at_fp = J.subs({X:A, Y:B/A})   # substitute X with A and Y with B/A
J_at_fp
```

$$\left[\begin{matrix}B - 1 & A^{2}\\- B & - A^{2}\end{matrix}\right]$$

A limit-cycle can emerge in a 2-dimensional, attractive dynamical system if the fixed point of the system  goes unstable. In this case, trajectories must be pulled by a limit cycle. (According to the Poincare-Bendixson theorem, a 2-dimensional system cannot have strange attractors). In this case, the Hopf bifurcation is called a *supercritical Hopf bifurcation*, because limit cycle is stable.

In the following, we will see how the stable fixed point (spiral) of the Brusselator goes unstable, giving rise to a limit cycle in turn. Conditions for the stability are determined by the trace and the determinant of the Jacobian. So let's evaluate them:

```python
Delta = J_at_fp.det()  # determinant of the Jacobian
Delta.simplify()
```

$$A^{2}$$

```python
tau = J_at_fp.trace() # trace of the Jacobian
tau
```

$$- A^{2} + B - 1$$

To have an unstable spiral we need:

$$
\begin{eqnarray}
\tau & > & 0  \quad \Rightarrow \quad & B > A^2 + 1  \quad  \textrm{required} \\
\Delta & > & 0  \quad {} \quad & \textrm{automatically satisfied} \\
\tau^2 & < & 4 \Delta \quad {} \quad & \textrm{automatically satisfied}
\end{eqnarray}
$$

The second and third conditions were satisfied because of the first condition, automatically. Therefore we need to have:

$$ B > A^2 + 1 $$ for limit cycles.

## Birth of A Limit Cycle: Hopf Bifurcation
### Numerical Simulation of the Brusselator System
In the following, I perform a numerical simulation of the (supercritical) Hopf bifurcation in the Brusselator system by varying the parameter $B$ while the value of $A=1$.

```python
# Brusselator System:
def dX_dt(A, B, X, t):
    x, y = X[0], X[1]
    return np.array([A + x**2 * y - B * x -x,
                     B * x - x**2 * y])

T = 50 * np.pi                      # simulation time
dt = 0.01                           # integration time step

# time steps to be used in integration of the Brusselator system
t=np.arange(0, T, dt)

# create a canvas and 3 subplots..we will use each one for different choice of A and B paraeters
fig, ax = plt.subplots(1, 3)
fig.set_size_inches(13,5)

def plotter(A, B, ax):
    """
    This function draws a phase portrait by assigning a vector characterizing how the concentrations
    change at a given value of X and Y. It also draws a couple of example trajectories.
    """
    # Drow direction fields using matplotlib 's quiver function, similar to what we did
    # in class but qualitatively
    xmin, xmax  = 0,  5                           # min and max values of x axis in the plot
    ymin, ymax  = 0,  5                           # min and max values of y axis in the plot
    x = np.linspace(xmin, xmax, 10)               # divide x axis to intervals
    y = np.linspace(ymin, ymax, 10)               # divide y axis to intervals
    X1 , Y1  = np.meshgrid(x, y)            # from these intervals create a grid
    DX1, DY1 = dX_dt(A, B, [X1, Y1], t)     # compute rate of change of the concentrations on grid points
    M = (np.hypot(DX1, DY1))                # norm of the rate of changes
    M[ M == 0] = 1.                         # prevention against divisions by zero
    DX1 /= M                                # we normalize the direction field vector (each has unit length now)
    DY1 /= M                                # we normalize the direction field vector (each has unit length now)
    Q = ax.quiver(X1, Y1, DX1, DY1, M, pivot='mid', cmap=plt.cm.jet)

    num_traj = 5  # number of trajectories

    # choose several initial points (x_i, y_j),  for i and j chosen as in linspace calls below
    X0  = np.asarray(list(zip(np.linspace(xmin, xmax, num_traj), np.linspace(ymin, ymax, num_traj))))
    vcolors = plt.cm.jet_r(np.linspace(0., 1., num_traj))  # colors for each trajectory

    # integrate the Brusellator ODE's using all initial points to produce corresponding trajectories
    X = np.asarray([integrate.odeint(lambda x, t: dX_dt(A, B, x, t), X0i,t)
                          for X0i in X0])
    # plot the trajectories we obtained above
    for i in range(num_traj):
        x, y = X[i, :, :].T          # x and y histories for trajectory i
        ax.plot(x, y, '-', c=vcolors[i], lw=2)
    # set limits, put labels etc..
    ax.set_xlim(xmin=xmin, xmax=xmax)
    ax.set_ylim(ymin=ymin, ymax=ymax)
    ax.set_xlabel("X", fontsize = 20)
    ax.set_ylabel("Y", fontsize = 20)
    ax.annotate("A={}, B={}".format(A, B), xy = (0.4, 0.9), xycoords = 'axes fraction', fontsize = 20, color = "k")

# Now let's prepare plots for the following choices of A and B:
plotter(A=1, B=1, ax=ax[0])
plotter(A=1, B=2, ax=ax[1])
plotter(A=1, B=3, ax=ax[2])
```

Above you see how a limit cycle can be created in a dynamical system, as one of the system parameters is varied.
Here we have kept $A=1$ but varied $B$ from 1 to 2.5. Note that $B=2$ is the borderline value, marking the change in the stability of the fixed point. For $B<1$ the fixed point is stable but as we cross the value $B=2$, it changes its character to unstable and a limit cycle is born. This phenomenon is an example of *Hopf Bifurcation*.

On the leftmost panel we have a stable spiral. Technically, this means that the Jacobian at the fixed point has two complex eigenvalues (a complex conjugate pair). The fact that the eigenvalues are complex is responsible for the spiralling effect. In stable spirals, the real part of the eigenvalues are negative, which is why these spiralling solutions decay, that is, trajectories nearby fall on to the fixed point. As the bifurcation parameter (here $B$) varies, the real part of the complex eigenvalues increase, reach zero at certain value of $B$, and keep growing now on the positive side. If the real part of the eigenvalues are positive, the fixed point is  an unstable spiral; trajectories nearby are pushed out of the fixed point (see the rightmost panel and plot below). Since this 2-D dynamical system is attractive, by the Poincare-Bendixson theorem, the emergence of the unstable spiral accompanies the birth of a limit-cycle. Notice that the panel in the middle is the borderline case between the stable and unstable spirals: in this case the real part of the eigenvalues is exactly zero (see plots below); the linear stabilization theory falsely predicts a neutral oscillation (i.e a center) at $B=2$---due to purely imaginary eigenvalues. However, the fixed point is still a stable spiral then.

### Eigenvalues of the Jacobian

```python
# Eigenvalues of the Jacobian at A=1, B=1  (fixed point is stable spiral)
J_numeric = np.asarray(J_at_fp.evalf(subs={A:1, B:1})).astype(np.float64)
w, _ = eig(J_numeric)
w
```

    array([-0.5+0.8660254j, -0.5-0.8660254j])

```python
# Eigenvalues of the Jacobian at A=1, B=3  (fixed point is unstable spiral)
J_numeric = np.asarray(J_at_fp.evalf(subs={A:1, B:3})).astype(np.float64)
w, _ = eig(J_numeric)
w
```

    array([ 0.5+0.8660254j,  0.5-0.8660254j])

Let's prepare plots showing how the real and imaginary parts of the eigenvalues change as $B$ is varied.

```python
from numpy.linalg import eig

a = 1
eigen_real, eigen_imag = [], []
B_vals = np.linspace(1, 3, 20)
for b in B_vals:
    J_numeric = np.asarray(J_at_fp.evalf(subs={A:a, B:b})).astype(np.float64)
    w, _ = eig(J_numeric)
    eigen_real.append(w[0].real)
    eigen_imag.append(abs(w[0].imag))
eigen_real = np.asanyarray(eigen_real)
eigen_imag = np.asarray(eigen_imag)
```

```python
fig, ax = plt.subplots(1, 2)
fig.set_size_inches(10,5)
fig.subplots_adjust(wspace=0.5)
ax[0].axhline(y=0, c="k", ls="dashed")
ax[0].plot(B_vals, eigen_real)
ax[0].set_ylabel(r"$\mathfrak{Re}(\lambda)$", fontsize = 20)
ax[0].set_xlabel(r"$B$", fontsize = 20)
ax[1].set_ylabel(r"$|\mathfrak{Im}(\lambda)|$", fontsize = 20)
ax[1].set_xlabel(r"$B$", fontsize = 20)
ax[1].plot(B_vals, eigen_imag)
```

Hopf bifurcation, is only one type of bifurcation, albeit a very important one for physical and biological systems. There are other types of bifurcation in which one can create or destroy fixed points or alter their properties in different ways than a Hopf bifurcations does. If you are curious, I suggest you to perform your own numerical experiements by playing with the values of $A$, $B$
or both.

### An Animation of the Hopf Bifurcation

```python
from matplotlib import animation, rc
from IPython.display import HTML

# Brusselator System:
def dX_dt(A, B, X, t):
    x, y = X[0], X[1]
    return np.array([A + x**2 * y - B * x -x, B * x - x**2 * y])

T = 50 * np.pi                      # simulation time
dt = 0.01                           # integration time step

# time steps to be used in integration of the Brusselator system
t=np.arange(0, T, dt)

num_traj = 5                          # number of trajectories
xmin, xmax  = 0,  5                   # min and max values of x axis in the plot
ymin, ymax  = 0,  5                   # min and max values of y axis in the plot
A = 1.                                # we will keep A parameter constant

# vary B parameter
Bmin, Bmax, numB = 1., 3., 100        # min, max, number of steps for varying B
Bvals = np.linspace(Bmin, Bmax, numB)

# set up the figure, the axis, and the plot element we want to animate
fig = plt.figure()
fig.set_size_inches(8,8)
ax = plt.axes(xlim=(xmin, xmax), ylim=(ymin, ymax))
ax.set_ylabel("Y", fontsize = 20)
ax.set_xlabel("X", fontsize = 20)

# choose a set of initial points for our trajectories (in each frame we will use the same set)
X0  = list(zip(np.linspace(xmin, xmax, num_traj), np.linspace(ymin, ymax, num_traj)))

# choose a color set for our trajectories
vcolors = plt.cm.jet_r(np.linspace(0., 1., num_traj))

# prepare the mesh grid
x = np.linspace(xmin, xmax, 15)             # divide x axis to intervals
y = np.linspace(ymin, ymax, 15)             # divide y axis to intervals
X1 , Y1  = np.meshgrid(x, y)                # from these intervals create a grid

# set up the lines, the quiver and the text object
lines = [ax.plot([], [], [], '-', c=c, lw=2)[0] for c in vcolors]
Q = ax.quiver(X1, Y1, [], [], [], pivot='mid', cmap=plt.cm.jet)
text = ax.text(0.02, 0.95, '', fontsize=20, transform=ax.transAxes)

# initialization function: plot the background of each frame. Needs to return each object to be updated
def init():
    for line in lines:
        line.set_data([], [])
    Q.set_UVC([], [], [])
    text.set_text("")
    return Q, lines, text

# animation function.  This is called sequentially
def animate(i):
    B = Bvals[i]
    DX1, DY1 = dX_dt(A, B, [X1, Y1], t)     # compute rate of change of the concentrations on grid points
    M = (np.hypot(DX1, DY1))                # norm of the rate of changes
    M[ M == 0] = 1.                         # prevention against divisions by zero
    DX1 /= M                                # we normalize the direction field vector (each has unit length now)
    DY1 /= M                                # we normalize the direction field vector (each has unit length now)
    Q.set_UVC(DX1, DY1, M)
    # integrate the Brusellator ODE's for the set of trajectories, store them in X
    for line, X0i in zip(lines, X0):
        X = integrate.odeint(lambda x, t: dX_dt(A, B, x, t), X0i,t)
        x, y = X.T        # get x and y for current trajectory
        line.set_data(x, y)
    text.set_text("A={:.2f}, B={:.2f}".format(A, B))
    return Q, lines, text

# call the animator.  blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig, animate, init_func=init, frames=100, interval=30, blit=False)
# instantiate the animator.
#anim = animation.FuncAnimation(fig, animate, init_func=init, frames=1000, interval=200, blit=True)
#HTML(anim.to_html5_video())
rc('animation', html='html5')
plt.close()
anim
```

In the animation above, we see how the direction field gets modified as $B$ is varied. Also shown several trajectories that are initalized at various points (I have chosen them on the $Y=X$ line here).

## Notes:

Should you encounter difficulty in running the embedded animation, try to launch Jupter Notebook using the command:

`jupyter notebook --NotebookApp.iopub_data_rate_limit=10000000000`

## References:

1) Strogatz, S.H (2015). *Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering, Second Edition*, Boulder, USA: Westview Press.

2) https://en.wikipedia.org/wiki/Brusselator


# 11 ODE integrators: Verlet

```python
from importlib import reload

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators2
```

```python
reload(integrators2)
```

    <module 'integrators2' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2017/PHY494-resources/11_ODEs/integrators2.py'>

## Velocity Verlet

Use expansion *forward* and *backward* (!) in time (Hamiltons (i.e. Newton without friction) equations are time symmetric)

\begin{align}
r(t + \Delta r) &\approx r(t) + \Delta t\, v(t) + \frac{1}{2m} \Delta t^2 F(t)\\
r(t) &\approx r(t + \Delta t) - \Delta t\, v(t + \Delta t) + \frac{1}{2m} \Delta t^2 F(t+\Delta t)
\end{align}

Solve for $v$:
\begin{align}
v(t+\Delta t) &\approx v(t) + \frac{1}{2m} \Delta t \big(F(t) + F(t+\Delta t)\big)
\end{align}

Complete **Velocity Verlet** integrator consists of the first and third equation.

In practice, split into three steps (calculate the velocity at the half time step):
\begin{align}
v(t+\frac{\Delta t}{2}) &= v(t) + \frac{\Delta t}{2} \frac{F(t)}{m} \\
r(t + \Delta r) &= r(t) + \Delta t\, v(t+\frac{\Delta t}{2})\\
v(t+\Delta t) &= v(t+\frac{\Delta t}{2}) + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m}
\end{align}

When writing production-level code, remember to re-use $F(t+\Delta t)$ als the "new" starting $F(t)$ in the next iteration (and don't recompute).

### Integration of planetary motion
Gravitational potential energy:
$$
U(r) = -\frac{GMm}{r}
$$
with $r$ the distance between the two masses $m$ and $M$.

#### Central forces
$$
U(\mathbf{r}) = f(r) = f(\sqrt{\mathbf{r}\cdot\mathbf{r}})\\
\mathbf{F} = -\nabla U(\mathbf{r}) = -\frac{\partial f(r)}{\partial r} \, \frac{\mathbf{r}}{r}
$$

#### Force of gravity
\begin{align}
\mathbf{F} &= -\frac{G m M}{r^2} \hat{\mathbf{r}}\\
\hat{\mathbf{r}} &= \frac{1}{\sqrt{x^2 + y^2}} \left(\begin{array}{c} x \\ y \end{array}\right)
\end{align}

#### Integrate simple planetary orbits
Set $M = 1$ (one solar mass) and $m = 3.003467×10^{-6}$ (one Earth mass in solar masses) and try initial conditions

\begin{alignat}{1}
x(0) &= 1,\quad  y(0) &= 0\\
v_x(0) &= 0,\quad  v_y(0) &= 6.179
\end{alignat}

Note that we use the following units:
* length in astronomical units (1 AU = 149,597,870,700 m )
* mass in solar masses (1 $M_☉ = 1.988435×10^{30}$ kg)
* time in years (1 year = 365.25 days, 1 day = 86400 seconds)

In these units, the gravitational constant is $G = 4\pi^2$ (in SI units $G = 6.674×10^{-11}\, \text{N}\cdot\text{m}^2\cdot\text{kg}^{-2}$).

```python
M_earth = 3.003467e-6
M_sun = 1.0
```

```python
G_grav = 4*np.pi**2

def F_gravity(r, m=M_earth, M=M_sun):
    rr = np.sum(r*r)
    rhat = r/np.sqrt(rr)
    return -G_grav*m*M/rr * rhat

def U_gravity(r, m=M_earth, M=M_sun):
    return -G_grav*m*M/np.sqrt(np.sum(r*r))
```

Let's now integrate the equations of motions under gravity with the **Velocity Verlet** algorithm:

```python
def planet_orbit(r0=np.array([1, 0]), v0=np.array([0, 6.179]), mass=M_earth, dt=0.001, t_max=1):
    """2D planetary motion with velocity verlet"""
    dim = len(r0)
    assert len(v0) == dim

    nsteps = int(t_max/dt)

    r = np.zeros((nsteps, dim))
    v = np.zeros_like(r)

    r[0, :] = r0
    v[0, :] = v0

    # start force evaluation for first step
    Ft = F_gravity(r[0], m=mass)
    for i in range(nsteps-1):
        vhalf = v[i] + 0.5*dt * Ft/mass
        r[i+1, :] = r[i] + dt * vhalf
        Ftdt = F_gravity(r[i+1], m=mass)
        v[i+1] = vhalf + 0.5*dt * Ftdt/mass
        # new force becomes old force
        Ft = Ftdt

    return r, v
```

```python
r, v = planet_orbit(dt=0.1, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

These are not closed orbits (as we would expect from a $1/r$ potential). But it gets much besser when stepsize is reduced to 0.01 (just rerun the code with `dt = 0.01` and replot):

```python
r, v = planet_orbit(dt=0.01, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `rk4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

The file `integrators2.py` contains almost all code that you will need.

### Implement gravity force in `integrators2.py`
Add `F_gravity` to the `integrators2.py` module. Use the new function `unitvector()`.

### Planetary orbits with `integrators2.py`

```python
r0 = np.array([1, 0])
v0 = np.array([0, 6.179])
```

```python
import integrators2
from importlib import reload
reload(integrators2)
```

    <module 'integrators2' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2017/PHY494-resources/11_ODEs/integrators2.py'>

Use the new function `integrators2.integrate_newton_2d()` to integrate 2d coordinates.

#### RK4

```python
trk4, yrk4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=integrators2.F_gravity,
                                       integrator=integrators2.rk4)
```

```python
rxrk4, ryrk4 = yrk4[:, 0, 0], yrk4[:, 0, 1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxrk4, ryrk4)
```

```python
integrators2.analyze_energies(trk4, yrk4, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(trk4),
        integrators2.energy_conservation(trk4, yrk4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 3000 steps: 3.5366039779498594e-06

#### Euler

```python
te, ye = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                         h=0.01,
                            force=F_gravity,
                            integrator=integrators2.euler)
rex, rey = ye[:, 0].T
```

```python
ax = plt.subplot(1,1,1)
ax.plot(rx, ry, label="RK4")
ax.plot(rex, rey, label="Euler")
ax.legend(loc="best")
ax.set_aspect(1)
```

```python
integrators2.analyze_energies(te, ye, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Euler for {} steps: {}".format(
        len(te),
        integrators2.energy_conservation(te, ye, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Euler for 3000 steps: 0.6763156457080265

*Euler* is just awful... but we knew that already.

#### Velocity Verlet

```python
tv, yv = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
rxv, ryv = yv[:, 0].T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxv, ryv, label="velocity Verlet")
ax.plot(rxrk4, ryrk4, label="RK4")
ax.legend(loc="best")
```

```python
integrators2.analyze_energies(tv, yv, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv),
        integrators2.energy_conservation(tv, yv, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 3000 steps: 6.37822713158452e-05

*Velocity Verlet* only has moderate accuracy, especially when compared to *RK4*.

However, let's look at energy conservation over longer times:

#### Longer time scale stability
Run RK4 and Velocity Verlet for longer.

```python
tv2, yv2 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv2),
        integrators2.energy_conservation(tv2, yv2, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 100000 steps: 6.379584728912368e-05

```python
t4, y4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.rk4)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(t4),
        integrators2.energy_conservation(t4, y4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 100000 steps: 0.00011860288209883538

Velocity Verlet shows **good long-term stability** but relative low precision. On the other hand, RK4 has high precision but the accuracy decreases over time.

* Use a *Verlet* integrator when energy conservation is important and long term stability is required (e.g. molecular dynamics simulations). It is generally recommended to use an integrator that conserves some of the inherent symmetries and structures of the governing physical equations (e.g. for Hamilton's equations of motion, time reversal symmetry and the symplectic and area-preserving structure).
* Use *RK4* for high short-term accuracy (but may be difficult to know what "short term" should mean) or when solving general differential equations.

```python

```


```python
from sympy import symbols, Integral, integrate

x = symbols('x')

f = x**4*(1 - x)**4/(1 + x**2)
f
```

$\displaystyle \frac{x^{4} \left(1 - x\right)^{4}}{x^{2} + 1}$

```python
F = Integral(f, x)
F
```

$\displaystyle \int \frac{x^{4} \left(1 - x\right)^{4}}{x^{2} + 1}\, dx$

```python
F.doit()
```

$\displaystyle \frac{x^{7}}{7} - \frac{2 x^{6}}{3} + x^{5} - \frac{4 x^{3}}{3} + 4 x - 4 \operatorname{atan}{\left(x \right)}$

```python
F = integrate(f, x)
F
```

$\displaystyle \frac{x^{7}}{7} - \frac{2 x^{6}}{3} + x^{5} - \frac{4 x^{3}}{3} + 4 x - 4 \operatorname{atan}{\left(x \right)}$

```python
F = integrate(f, (x, 0, 1))
F
```

$\displaystyle \frac{22}{7} - \pi$


## Numerical integration of Ordinary Differential Equations
This notebook serves as a quick refresher on ordinary differential equations. If you are familiar with the topic: feel free to skim this notebook.

We will first consider the decay of tritium as an example:

$$
\mathrm{^3H \overset{\lambda}\rightarrow\ ^3He + e^- + \bar{\nu_e}}
$$

We will not concern ourselves with the products, instead we will only take interest in the number density of $\mathrm{^3H}$ as function of time, let's call it $y(t)$. The rate of change of $y(t)$ is proportional to itself and the decay constant ($\lambda$):

$$
\frac{dy(t)}{dt} = -\lambda y(t)
$$

you probably know the solution to this class of differential equations (either from experience or by guessing an appropriate ansatz). SymPy can of course also solve this equation:

```python
import sympy as sym
sym.init_printing()
t, l = sym.symbols('t lambda')
y = sym.Function('y')(t)
dydt = y.diff(t)
expr = sym.Eq(dydt, -l*y)
expr
```

```python
sym.dsolve(expr)
```

Now, pretend for a while that this function lacked an analytic solution. We could then integrate this equation *numerically* from an initial state for a predetermined amount of time by discretizing the time into a seriers of small steps.

### Explicit methods
For each step taken we would update $y$ by multiplying the derivative with the step size (assuming that the derivate is approximately constant on the scale of the step-size), formally this method is known as "forward Euler":

$$
y_{n+1} = y_n + y'(t_n)\cdot \Delta h
$$

this is known as an *explicit* method, i.e. the derivative at the current time step is used to calculate the next step *forward*.

For demonstration purposes only, we implement this in Python:

```python
import numpy as np
def euler_fw(rhs, y0, tout, params):
    y0 = np.atleast_1d(np.asarray(y0, dtype=np.float64))
    dydt = np.empty_like(y0)
    yout = np.zeros((len(tout), len(y0)))
    yout[0] = y0
    t_old = tout[0]
    for i, t in enumerate(tout[1:], 1):
        dydt[:] = rhs(yout[i-1], t, *params)
        h = t - t_old
        yout[i] = yout[i-1] + dydt*h
        t_old = t
    return yout
```

applying this function on our model problem:

```python
def rhs(y, t, decay_constant):
    return -decay_constant*y  # the rate does not depend on time ("t")
```

```python
tout = np.linspace(0, 2e9, 100)
y0 = 3
params = (1.78e-9,)  # 1 parameter, decay constant of tritium
yout = euler_fw(rhs, y0, tout, params)
```

and plotting the solution & the numerical error using matplotlib:

```python
import matplotlib.pyplot as plt
%matplotlib inline

def my_plot(tout, yout, params, xlbl='time / a.u.', ylabel=None, analytic=None):
    fig, axes = plt.subplots(1, 2 if analytic else 1, figsize=(14, 4))
    axes = np.atleast_1d(axes)
    for i in range(yout.shape[1]):
        axes[0].plot(tout, yout[:, i], label='y%d' % i)
    if ylabel:
        axes[0].set_ylabel(ylabel)
    for ax in axes:
        ax.set_xlabel(xlbl)
    if analytic:
        axes[0].plot(tout, analytic(tout, yout, params), '--')
        axes[1].plot(tout, yout[:, 0] - yout[0]*np.exp(-params[0]*(tout-tout[0])))
        if ylabel:
            axes[1].set_ylabel('Error in ' + ylabel)

```

```python
def analytic(tout, yout, params):
    return yout[0, 0]*np.exp(-params[0]*tout)
my_plot(tout, yout, params, analytic=analytic, ylabel='number density / a.u.')
```

We see that 100 points gave us almost plotting accuracy.

Unfortunately, Euler forward is not practical for most real world problems. Usually we want a higher order formula (the error in Euler forward scales only as $n^{-1}$), and we want to use an adaptive step size (larger steps when the function is smooth). So we use the well tested LSODA algorithm (provided in scipy as ``odeint``):

```python
from scipy.integrate import odeint
yout, info = odeint(rhs, y0, tout, params, full_output=True)
my_plot(tout, yout, params, analytic=analytic)
print("Number of function evaluations: %d" % info['nfe'][-1])
```

We can see that ``odeint`` was able to achieve a much higher precision using fewer number of function evaluations.

### Implicit methods
For a large class of problems we need to base the step not on the derivative at the current time point, but rather at the next one (giving rise to an implicit expression). The simplest implicit stepper is "backward euler":

$$
y_{n+1} = y_n + y'(t_{n+1})\cdot \Delta h
$$

Problems requiring this type of steppers are known as "stiff". We will not go into the details of this (LSODA actually uses something more refined and switches between explicit and implicit steppers).

In the upcoming notebooks we will use ``odeint`` to solve systems of ODEs (and not only linear equations as in this notebook). The emphasis is not on the numerical methods, but rather on how we, from symbolic expressions, can generate fast functions for the solver.

### Systems of differential equations
In order to show how we would formulate a system of differential equations we will here briefly look at the [van der Pol osciallator](https://en.wikipedia.org/wiki/Van_der_Pol_oscillator). It is a second order differential equation:

$$
{d^2y_0 \over dx^2}-\mu(1-y_0^2){dy_0 \over dx}+y_0= 0
$$

One way to reduce the order of our second order differential equation is to formulate it as a system of first order ODEs, using:

$$ y_1 = \dot y_0 $$

which gives us:

$$
\begin{cases}
\dot y_0 = y_1 \\
\dot y_1 = \mu(1-y_0^2) y_1-y_0
\end{cases}
$$

Let's call the function for this system of ordinary differential equations ``vdp``:

```python
def vdp(y, t, mu):
    return [
        y[1],
        mu*(1-y[0]**2)*y[1] - y[0]
    ]
```

using "Euler forward":

```python
tout = np.linspace(0, 200, 1024)
y_init, params = [1, 0], (17,)
y_euler = euler_fw(vdp, y_init, tout, params)  # never mind the warnings emitted here...
```

```python
my_plot(tout, y_euler, params)
```

That does not look like an oscillator. (we see that Euler forward has deviated to values with enormous magnitude), here the advanced treatment by the ``odeint`` solver is far superior:

```python
y_odeint, info = odeint(vdp, y_init, tout, params, full_output=True)
print("Number of function evaluations: %d, number of Jacobian evaluations: %d" % (info['nfe'][-1], info['nje'][-1]))
my_plot(tout, y_odeint, params)
```

We see that LSODA has evaluated the Jacobian. But we never gave it an explicit representation of it―so how could it?

It estimated the Jacobian matrix by using finite differences. Let's see if we can do better if we provide a function to calculate the (analytic) Jacobian.

## Exercise: manually write a function evaluating a Jacobian
First we need to know what signature ``odeint`` expects, we look at the documentation by using the ``help`` command: (or using ``?`` in IPython)

```python
help(odeint)  # just skip to "Dfun"
```

so the signature needs to be: ``(state-vector, time, parameters) -> matrix``

```python
%load_ext scipy2017codegen.exercise
```

Use either the * ``%exercise`` * or * ``%load`` * magic to get the exercise / solution respecitvely (*i.e.* delete the whole contents of the cell except for the uncommented magic command). Replace **???** with the correct expression.

Remember that our system is defined as:
$$
\begin{cases}
\dot y_0 = y_1 \\
\dot y_1 = \mu(1-y_0^2) y_1-y_0
\end{cases}
$$

```python
%exercise exercise_jac_func.py
```

```python
J_func(y_init, tout[0], params[0])
```

```python
y_odeint, info = odeint(vdp, y_init, tout, params, full_output=True, Dfun=J_func)
```

```python
my_plot(tout, y_odeint, params)
print("Number of function evaluations: %d, number of Jacobian evaluations: %d" % (info['nfe'][-1], info['nje'][-1]))
```

So this time the integration needed to evaluate both the ODE system function and its Jacobian fewer times than when using finite difference approximations. The reason for this is that the more accurate the Jacobian is, the better is the convergence in the iterative (Newton's) method solving the implicit system of equations.

For larger systems of ODEs the importance of providing a (correct) analytic Jacobian can be much bigger.

### SymPy to the rescue
Instead of writing the jacobian function by hand we could have used SymPy's ``lambdify`` which we will introduce next. Here is a sneak peak on how it could be achieved:

```python
y = y0, y1 = sym.symbols('y0 y1')
mu = sym.symbols('mu')
J = sym.Matrix(vdp(y, None, mu)).jacobian(y)
J_func = sym.lambdify((y, t, mu), J)
J
```


A random variable is a numerical description of the outcome of a statistical experiment. A random variable that may assume only a finite number or an infinite sequence of values is said to be discrete; one that may assume any value in some interval on the real number line is said to be continuous.

# Expected value

The expected value for a variable X is the value of X we would **expect** to get after performing the experiment once. ***It is also called the expectation, average, and mean value***. Mathematically speaking, for a random variable X that can take values x1,x2,x3,...........,xn, the expected value (EV) is given by:

\begin{align}
EV(X) = {x_1*P(X=x_1)+x_2*P(X=x_2)+x_3*P(X=x_3)+......+x_n*P(X=x_n)}
\end{align}

Sample question:

Hugo is considering buying a scratch-off lottery ticket that costs 3 dollars. If the ticket wins, he can redeem the ticket for 50 dollars. If the ticket loses, the ticket is worthless. According to the lottery's website, 2%, percent of all tickets are winners.

Find the expected value of buying a scratch-off lottery ticket.

1. There is a 2%, percent chance of the ticket winning. If this happens, the value is 50−3=47 (the redemption value minus the cost of the ticket).
2. There is a 98%, percent chance of the ticket losing. In this case, the value is −3 (just the cost of the ticket).
3. Let's multiply each value by its probability because expected value is the sum of all possible values, each value multiplied by its probability.

| Type           | Value | Probability | Value.Probability |
| -------------- | ----- | ----------- | ----------------- |
| Winning ticket | 47    | 0.020       | .94               |
| Losing ticket  | −3    | 0.980       | −2.94             |

The expected value is 0.94+(−2.94)=−2

# Binomial Probability

The formula for finding binomial probability is given by -

\begin{align}
P(X=r) = {C^n_r \cdot p^r(1-p)^{n-r}}
\end{align}

Sample question:

Steph's little brother Luke only has a 20%, percent chance of making a free-throw. He is going to shoot 4444 free-throws.
What is the probability that he makes exactly 2 of the 4 free throws?

1. n=4, equals, 4 trials (shots)
2. each shot is either a make or miss
3. probability that he makes a shot is p=0.20
4. shots are independent

P(exactly 2 makes) =4C2*(0.20)2*(0.80)2
```
= (4−2)!/2!*4!*0.04*0.64

= (4*3*2*1/(2*2))*0.04*0.64

= 6*0.04*0.64

= 0.1536
```

# Cumulative probability

A cumulative probability refers to the probability that the value of a random variable falls within a specified range. Frequently, cumulative probabilities refer to the probability that a random variable is less than or equal to a specified value.
Cumulative probability of X, denoted by F(x), is defined as the probability of the variable being less than or equal to x.

 Consider a coin flip experiment. If we flip a coin two times, we might ask: What is the probability that the coin flips would result in one or fewer heads? The answer would be a cumulative probability. It would be the probability that the coin flip results in zero heads plus the probability that the coin flip results in one head. Thus, the cumulative probability would equal:

P(X < 1) = P(X = 0) + P(X = 1) = 0.25 + 0.50 = 0.75

| Number of heads | Probability | Cumulative Probability |
| --------------- | ----------- | ---------------------- |
| 0               | 0.25        | 0.25                   |
| 1               | 0.50        | 0.75                   |
| 2               | 0.25        | 1.00                   |

A CDF, or a cumulative distribution function, is a distribution which plots the cumulative probability of X against X.

A PDF, or Probability Density Function, however, is a function in which the area under the curve, gives you the cumulative probability.

The main difference between the cumulative probability distribution of a continuous random variable and a discrete one, is the way we plot them. While the continuous variables’ cumulative distribution is a curve, the distribution for discrete variables looks more like a bar chart.

A commonly observed type of distribution among continuous variables is the uniform distribution. For a continuous random variable following a uniform distribution, the value of probability density is equal for all possible values.

PDFs are more commonly used in real life. The reason is that it is much easier to see patterns in PDFs as compared to CDFs. For example, here are the PDF and the CDF of a uniformly distributed continuous random variable.

The PDF clearly shows uniformity, as the probability density’s value remains constant for all possible values. However, the CDF does not show any trends that help you identify quickly that the variable is uniformly distributed.

it is clear that the symmetrical nature of the variable is much more apparent in the PDF than in the CDF.

# Normal Distribution

All data that is normally distributed follows the **1-2-3** rule. This rule states that there is:

1. 68% probability of the variable lying within 1 standard deviation of the mean

2. 95% probability of the variable lying within 2 standard deviations of the mean

3. 99.7% probability of the variable lying within 3 standard deviations of the mean

***Standardised random variable (Z-Score)*** tells us how many standard deviations away from the mean the random variable is.

\begin{align}
Z = {X - \mu \over \sigma}
\end{align}

Cumulative probability from Z-Score is calculated using Z Table.

If we need to find the cumulative probability for Z = 0.68

The intersection of row **0.6** and column **0.08**, i.e. **0.7517**, is the answer.

# Central Limit Theorem

So, there are two important properties for a sampling distribution of the mean:

1. Sampling distribution’s mean (μX̄) = Population mean (μ)

2. Sampling distribution’s standard deviation (Standard error) = σ/√n, where σ is the population’s standard deviation and n is the sample size

### The central limit theorem says that, for any kind of data, provided a high number of samples has been taken, the following properties hold true:

1. Sampling distribution’s mean (μX̄) = Population mean (μ)

2. Sampling distribution’s standard deviation (Standard error) = σ/√n

3. For n > 30, the sampling distribution becomes a normal distribution

## Confidence Interval

\begin{align}
Confidence Interval = {(X-\frac {Z^*S}{\sqrt(n)},X+\frac {Z^*S}{\sqrt(n)})}
\end{align}

\begin{align}
Margin of error = {\frac {Z^*S}{\sqrt(n)}}
\end{align}

#### Z* is the Z-score associated

Some commonly used Z* values are:

| Confidence Level| Z* |
| --------------- | ----------- |
| 90%              | ±1.65        |
| 95%             | ±1.96       |
| 99%             | ±2.58        |

```python

```


# Base rate fallacy example

In this notenook we work an example of the base rate fallacy using Bayes Theorem.

Assume that we have two random variables $HasDisease$ and $FailsTest$.  $HasDisease=y$ indicates that a person has the disease while $HasDisease=n$ indicates that the person in disease free.  In addition, we have a test which attempts to detect the disease.  $FailsTest=y$ indicates that our test says a person hasthe disease while $FailsTest=n$ indicates that our test says a person does not have the disease.

In this notebook you can play around with the probabilities of interest and see now likely it is that, given you fail the test, that you actually have  the disease.

Suppose we know the following probabilities:

\begin{align}
Pr(FailsTest=y | HasDisease=y) &= FailAndHasDisease \\
Pr(FailsTest=n | HasDisease=y) &= NotFailAndHasDisease \\
Pr(FailsTest=y | HasDisease=n) &= FailAndNotHasDisease \\
Pr(FailsTest=n | HasDisease=n) &= NotFailAndNotHasDisease \\
\end{align}

And we know the prior probability of the disease in the population

$$
Pr(HasDisease=y).
$$

Note, the point of the base rate fallacy is that you need all 5 probabilities to compute what you are interested in, namely the probability you have the disease given you fail the test, denoted

$$
Pr(HasDisease=y | FailsTest=y).
$$

Without, $Pr(HasDisease=y)$ you cannot truly understand  $Pr(HasDisease=y | FailsTest=y)$.

You can play aroun with the numbers in the next cell to see how things work out.

```python
FailAndHasDisease = 1.0
NotFailAndHasDisease = 0.0
FailAndNotHasDisease = 0.01
NotFailAndNotHasDisease = 0.99
HasDisease = 1./1000
```

Bayes theorem says that

$$
Pr(HasDisease=y | FailsTest=y) = \frac{Pr(FailsTest=y | HasDisease=y) Pr(HasDisease=y)}{Pr(FailsTest=y)}
$$

Our table gives us the two terms in the numerator, we get the demoninator from the Law of total probability.

\begin{align}
Pr(FailsTest=y) & = Pr(FailsTest=y | HasDisease=y) Pr(HasDisease=y) + Pr(FailsTest=y | HasDisease=n) Pr(HasDisease=n) \\
                & = Pr(FailsTest=y | HasDisease=y) Pr(HasDisease=y) + Pr(FailsTest=y | HasDisease=n) (1- Pr(HasDisease=y))
\end{align}

So, the whole thing is

$$
Pr(HasDisease=y | FailsTest=y) = \frac{Pr(FailsTest=y | HasDisease=y) Pr(HasDisease=y)}{(Pr(FailsTest=y | HasDisease=y) Pr(HasDisease=y) + Pr(FailsTest=y | HasDisease=n) (1- Pr(HasDisease=y)))}
$$

```python
FailAndHasDisease*HasDisease/(FailAndHasDisease*HasDisease + FailAndNotHasDisease*(1-HasDisease))
```

    0.09099181073703368

This matches the result we did by hand in class.  Play around with the probabilities and see what you discover.

```python

```

lectures/05 Basic Statistics, Probability, and Linear Algebra/BaseRateFallacy.ipynb


# Logistic Regression

A binary classifier.

## Logistic Regression Prediction

To perform a prediction using logistic regression we use the following formula:

\begin{equation}
\hat{y} = \sigma (w^Tx + b)
\end{equation}

\begin{equation}
\sigma(z) = \frac{1}{1 + e^{-z}}
\end{equation}

$\sigma(z)$: Also called the sigmoid function. The output will never be greater than one

$\hat{y}^{(i)}$: The predicted label for example $i$

$w$: Vector of weights

$b$: bias (a real number)

$x^{(i)}$: The $i$th input or training example

${y}^{(i)}$: Ground truth label for example $i$

We want $\hat{y}$ to be as close as possible to $y$, or $\hat{y} \approx y$

## Loss (error) function $\mathcal{L}$

\begin{equation*}
\mathcal{L}(\hat{y}^{(i)},y^{(i)}) = -(y^{(i)}\log{\hat{y}^{(i)}} + (1 - y^{(i)}) \log({1 - \hat{y}^{(i)}}))
\end{equation*}

This function will measure how close our output $\hat{y}$ is to the true label $y$.

### Intuition
We want $\mathcal{L}$ to be as small as possible.

If $y = 1$ then $\mathcal{L}(\hat{y},y) = - \log \hat{y}$.

The second expression cancels out because $(1-1) = 0$. So now we want $- \log \hat{y}$ to be as large as possible. That means we want $\hat{y}$ to be large. The sigmoid function above, $\sigma (z)$ can never be greater than one.

If $y = 0$ then $\mathcal{L}(\hat{y},y) = - \log (1 - \hat{y})$

Similar reasoning, now we want $\hat{y}$ as small as possible because we still want $\log 1 -\hat{y}$ large

Another option is to use the squared error function: $\mathcal{L}(\hat{y},y) = \frac{1}{2}(\hat{y} - y)^2$  But this will produce a non-convex surface which is not good for gradient descent because it may not find the global optimum, but rather only a local optimum.

## Cost function $J$ used in Logistic Regression

\begin{equation*}
J(w,b) = - \frac{1}{m}\sum_{i=1}^m \mathcal{L}(\hat{y}^{(i)},y^{(i)})
\end{equation*}

The cost function is the average, the sum, over the loss functions, divided by the number of examples $m$. This measures how well your parameters, $w$ and $b$ are performing on the training set. So the optimization problem is to minimize $J(w,b)$. We want to find $w$ and $b$ that make $J$ as small as possible.

## Gradient Descent

_Repeat until convergence:_
\begin{equation}
w = w - \alpha \frac{\partial J(w, b)}{\partial w}
\end{equation}

\begin{equation}
b = b - \alpha \frac{\partial J(w, b)}{\partial b}
\end{equation}

$\alpha$: Learning rate. How big of a step we take towards the minimum with each iteration

### Initializing $w$
Typically this is initialized to zero. Since the error surface is convex, it should arrive at the same minimum given any initialization.


# Solution {-}

PVA model

```python
from sympy import Matrix, symbols, sqrt, eye, integrate
from lib.vanloan import numeval

q, dt = symbols('q dt')

F = Matrix([[0, 1, 0],
            [0, 0, 1],
            [0, 0, 0]])

G = Matrix([[0],
            [0],
            [sqrt(q)]])

phi = eye(3) + F*dt + (F*dt)**2/2
phi
```

$\displaystyle \left[\begin{matrix}1 & dt & \frac{dt^{2}}{2}\\0 & 1 & dt\\0 & 0 & 1\end{matrix}\right]$

```python
# Process noise
Q = integrate(phi@G@G.T@phi.T, dt)
Q
```

$\displaystyle \left[\begin{matrix}\frac{dt^{5} q}{20} & \frac{dt^{4} q}{8} & \frac{dt^{3} q}{6}\\\frac{dt^{4} q}{8} & \frac{dt^{3} q}{3} & \frac{dt^{2} q}{2}\\\frac{dt^{3} q}{6} & \frac{dt^{2} q}{2} & dt q\end{matrix}\right]$

Numerical example

```python
from numpy import array, zeros, arange
import matplotlib.pyplot as plt

# System parameters
dt = 0.1
q = 1

F = array([[0, 1, 0],
           [0, 0, 1],
           [0, 0, 0]])

G = array([[0],
           [0],
           [sqrt(q)]])

# Numerical evaluation
[phi, Q_true] = numeval(F, G, dt)

# Process noise approximation
Q_approx = array([[0, 0, 0],
                  [0, 0, 0],
                  [0, 0, q*dt]])

# Plot vectors
var_true = []
var_approx = []

# Initial covariance
P = zeros([3, 3])
for i in range(0, 200):
    P = phi@P@phi.T + Q_true
    var_true.append(P)

# Initial covariance
P = zeros([3, 3])
for i in range(0, 200):
    P = phi@P@phi.T + Q_approx
    var_approx.append(P)

# Difference at step 200
print(var_true[199] - var_approx[199])
```

    [[1.99333335e+03 1.99500000e+02 9.98333333e+00]
     [1.99500000e+02 1.99666667e+01 1.00000000e+00]
     [9.98333333e+00 1.00000000e+00 0.00000000e+00]]


```python
from sympy.physics.mechanics import *
import sympy as sp
mechanics_printing(pretty_print=True)
```

```python
m1, m2, m3, m4, m5, l1, l2, l3, l4, l5 = sp.symbols(r'm_1 m_2 m_3 m_4 m_5 l_1 l_2 l_3 l_4 l_5')
t, g, h = sp.symbols('t g h')
v1, v2, v3, v4, v5 = dynamicsymbols(r'\theta_1 \theta_2 \theta_3 \theta_4 \theta_5')
dv1, dv2, dv3, dv4, dv5 = dynamicsymbols(r'\theta_1 \theta_2 \theta_3 \theta_4 \theta_5', 1)
```

```python
x1 = l1*sp.sin(v1)
y1 = -l1*sp.cos(v1)
x2 = x1+l2*sp.sin(v2)
y2 = y1+-l2*sp.cos(v2)
x3 = x2+l3*sp.sin(v3)
y3 = y2+-l3*sp.cos(v3)
x4 = x3+l4*sp.sin(v4)
y4 = y3+-l4*sp.cos(v4)
x5 = x4+l5*sp.sin(v5)
y5 = y4+-l5*sp.cos(v5)

dx1 = x1.diff(t)
dy1 = y1.diff(t)
dx2 = x2.diff(t)
dy2 = y2.diff(t)
dx3 = x3.diff(t)
dy3 = y3.diff(t)
dx4 = x4.diff(t)
dy4 = y4.diff(t)
dx5 = x5.diff(t)
dy5 = y5.diff(t)
```

```python
V = (m1*g*y1)+(m2*g*y2)+(m3*g*y3)+(m4*g*y4)+(m5*g*y5)
T = (sp.Rational(1,2)*m1*(dx1**2+dy1**2))+(sp.Rational(1,2)*m2*(dx2**2+dy2**2))+(sp.Rational(1,2)*m3*(dx3**2+dy3**2))+(sp.Rational(1,2)*m4*(dx4**2+dy4**2))+(sp.Rational(1,2)*m5*(dx5**2+dy5**2))
L = T-V
```

```python
LM = LagrangesMethod(L,[v1,v2,v3,v4,v5])
```

```python
soln = LM.form_lagranges_equations()
```

```python
soln
```

```python
# solvedsoln = sp.solve((sp.Eq(soln[0]),sp.Eq(soln[1]),sp.Eq(soln[2]),sp.Eq(soln[3]),sp.Eq(soln[4])),(v1.diff(t,t),v2.diff(t,t),v3.diff(t,t),v4.diff(t,t),v5.diff(t,t)))
```

```python
# solvedsoln
```


# Probit Maximum Likelihood

In this homework you should implement the maximum likelihood estimator for the probit model. To remind you, this model is defined as follows:
    $$
    \begin{align}
    y_i  &\in \{0,1\} \\
    \Pr\{y_i=1\} &= \Phi(x_i \beta) \\
    L(\beta)   & = \Pi_{i=1}^N  \Phi(x_i \beta)^{y_i} (1-\Phi(x_i \beta))^{1-y_i} \\
    \beta  & \in \mathbb{R}^k \\
    x_i  & \sim N\left([0,0,0],\left[ \begin{array}{ccc} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1\end{array} \right] \right) \\
    k & = 3
    \end{align}
    $$

Where $\Phi$ is the standard Normal cdf. Think of $x_i$ as a row-vector. You should proceed as follows:

1. define a data generating function with default argument `N=10000`, generating `N` simulated data points from this model. Generate the data using $\beta=[1,1.5,-0.5]$. The function should return a `Dict` as outlined in the code.
1. Define the log likelihood function, $l(\beta) = \log(L)$
1. Write a function `plotLike` to plot the log likelihood function for different parameter values. Follow the outline of that function.
1. Fill in the similare function `plotGrad`, plotting the gradient.
1. Define the function `maximize_like`. this should optimize your log likelihood function.
1. (Optional) Define the gradient of the log likelihood function and use it in another optimization `maximize_like_grad`.
1. (Optional) Define the hessian of the log likelihood function and use it in another optimization `maximize_like_grad_hess`.
1. (Optional) Use the hessian of the log likelihood function to compute the standard errors of your estimates and use it in `maximize_like_grad_se`

## Tests (NOT optional)

* The code comes with a test suite that you should fill out.
* There are some example tests, you should make those work and maybe add other ones.
* Please do not change anything in the file structure.

```julia

```


# Conditioning of evaluating tan()

```python
import numpy as np
import matplotlib.pyplot as pt
```

Let us estimate the sensitivity of evaluating the $\tan$ function:

```python
x = np.linspace(-5, 5, 1000)
pt.ylim([-10, 10])
pt.plot(x, np.tan(x))
```

```python
x = np.pi/2 - 0.0001
#x = 0.1
x
```

    1.5706963267948966

```python
np.tan(x)
```

    9999.9999666616441

```python
dx = 0.00005
np.tan(x+dx)
```

    19999.99998335545

## Condition number estimates

### From evaluation data

```python
#clear

np.abs(np.tan(x+dx) - np.tan(x))/np.abs(np.tan(x)) / (np.abs(dx) / np.abs(x))
```

    31413.926693068603

### Using the derivative estimate

```python
import sympy as sp

xsym = sp.Symbol("x")

f = sp.tan(xsym)
df = f.diff(xsym)
df
```

    tan(x)**2 + 1

Evaluate the derivative estimate. Use `.subs(xsym, x)` to substitute in the value of `x`.

```python
#clear
(xsym*df/f).subs(xsym, x)
```

    15706.9633726542

```python

```

demos/error_and_fp/Conditioning of Evaluating tan.ipynb


# Week 1
### Section 0.3, problem 7

[link to relevant section of the book](http://scidiv.bellevuecollege.edu/dh/Calculus_all/CC_0_3_functions)

$f(x) = x^2 + 3$ , $g(x) = \sqrt{x - 5}$ , and $h(x) = \frac{x}{x - 2}$

### (a) evaluate $f(1)$, $g(1)$, and $h(1)$

```python
import math
import numpy as np

def f(x):
    return x**2 + 3

def g(x):
    if x-5 >= 0:
        return math.sqrt(x-5)
    else:
        return str(abs(x-5))+'i'

def h(x):
    try:
        return x/(x-2)
    except ZeroDivisionError:
        return np.NaN

print(
    "f of 1:", f(1),
    "\ng of 1:", g(1),
    "\nh of 1:", h(1)
)
```

    f of 1: 4
    g of 1: 4i
    h of 1: -1.0

###  (b) graph f(x), g(x) and h(x) for $-5 \le x \le 10$

```python
from bokeh.plotting import figure, show
from bokeh.io import output_notebook
from bokeh.layouts import row

output_notebook()
```

    Loading BokehJS ...

```python
# plot of f(x)
p = figure(plot_width=300, plot_height=300,
           tools='pan,wheel_zoom,reset', title="f(x) plot")

x = [x/100 for x in range(-500, 1100)]
y = [f(num) for num in x]

p.line(x, y, line_width=2)

#plot of g(x)
p2 = figure(plot_width=300, plot_height=300,
            tools='pan,wheel_zoom,reset', title="g(x) plot")

x = [x/100 for x in range(-500, 1100)]
y = [g(num) for num in x]

p2.line(x, y, line_width=2)

#plot of h(x)
p3 = figure(plot_width=300, plot_height=300,
            tools='pan,wheel_zoom,reset', title="h(x) plot")

x = [x/100 for x in range(-500, 1100)]
y = [h(num) for num in x]

p3.line(x, y, line_width=2)

show(row(p, p2, p3))
```

### (c) evaluate f(3x), g(3x), and h(3x)

```python
import sympy

# set printing to unicode/latex
sympy.init_printing(use_unicode=True)
```

```python
# define our symbols
x = sympy.symbols('x')

# make our functions using sympy
f = x**2 + 3
g = sympy.sqrt(x-5)
h = x / (x-2)
```

```python
# evaluate functions
f.subs(x, 3*x)
```

```python
g.subs(x, 3*x)
```

```python
h.subs(x, 3*x)
```

### evaluate $f(x+h)$, $g(x+h)$, and $h(x+h)$
(assuming the $h$ in $h(x+h)$ is a separate number)

```python
# I'll assign the symbol 'h' to the variable y
y = sympy.symbols("h")

f.subs(x, x+y)
```

```python
g.subs(x, x+y)
```

```python
h.subs(x, x+y)
```


```python
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.gridspec import GridSpec
from matplotlib.ticker import ScalarFormatter
import math
```

This notebook assumes you have completed the notebook [Introduction of sine waves](TDS_Introduction-sine_waves.ipynb). This notebook follows the same pattern of time domain waveform generation: instantaneous frequency -> phase step -> total phase -> time domain waveform.

Our goal is to track features of different acoustic impedance in material using a low power time domain waveform. Time delay spectrometry (TDS) is one implementation of this goal. To understand TDS we need to understand the waveform which is used by TDS called a chirp. A chirp is a sinusoid that is constantly varying in frequency. The chirp is generated by integrating a varying phase step which is derived from an instantaneous frequency profile. We will generate a chirp in this notebook.
The phase of the chirp can be found by integrating the instantaneous frequency:

\begin{equation}
	f(t)=\frac{f_{end}-f_{start}}{T_c}t + f_{start}
\end{equation}

\begin{equation}
\Delta\phi(t) = 2\pi f(t)\Delta t
\end{equation}

\begin{equation}
\phi (t)=\int_{}^{} \Delta\phi(t) = \int_{}^{} 2\pi f(t) dt = \int_{}^{}\frac{f_{end}-f_{start}}{T_c}tdt + \int_{}^{}f_{start}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\int_{}^{}tdt + f_{start}\int_{}^{}dt
\end{equation}

\begin{equation}
 \phi (t)= \frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t
\end{equation}

This gives the time series value of

\begin{equation}
x(t) = e^{j\phi (t)} = e^{j(\frac{f_{end}-f_{start}}{T_c}\frac{t^2}{2} + f_{start}t)}
\end{equation}

But the formula for phase requires squaring time which will cause numeric errors as the time increases. Another approach is to implement the formula for phase as a cummulative summation.

\begin{equation}
\phi_{sum} (N)=\sum_{k=1}^{N} \Delta\phi(k) = \sum_{k=1}^{N} 2\pi f(k) t_s = \sum_{k=1}^{N}(\frac{f_{end}-f_{start}}{T_c}k + f_{start})t_s
\end{equation}

This allow for the phase always stay between 0 and two pi by subtracting two phi whenever the phase exceeds the value. We will work with the cummlative sum of phase, but then compare it to the integral to determine how accurate the cummulative sum is.

```python
#max free 8 points per sample

#Tc is the max depth we are interested in
Tc_sec=5

f_start_Hz=3000
#talk about difference and similarity of sine wave example, answer why not 32 samples
f_stop_Hz=20000

#sample rate
fs=44100
samplesPerCycle=fs/f_stop_Hz
ts=1/fs

total_samples= math.ceil(fs*Tc_sec)
n = np.arange(0,total_samples, step=1, dtype=np.float64)
t_sec=n*ts
t_usec = t_sec *1e6

#This is the frequency of the chirp over time. We assume linear change in frequency
chirp_freq_slope_HzPerSec=(f_stop_Hz-f_start_Hz)/Tc_sec

#Compute the instantaneous frequency which is a linear function
chirp_instantaneous_freq_Hz=chirp_freq_slope_HzPerSec*t_sec+f_start_Hz
chirp_instantaneous_angular_freq_radPerSec=2*np.pi*chirp_instantaneous_freq_Hz

#Since frequency is a change in phase the we can plot it as a phase step
chirp_phase_step_rad=chirp_instantaneous_angular_freq_radPerSec*ts

#The phase step can be summed (or integrated) to produce the total phase which is the phase value
#for each point in time for the chirp function
chirp_phase_rad=np.cumsum(chirp_phase_step_rad)
#The time domain chirp function
chirp = np.exp(1j*chirp_phase_rad)
```

```python
def calc_chirp_phase_step(f_start_Hz, f_stop_Hz, Tc_sec,fs):
    ts=1/fs
    total_samples= math.ceil(fs*Tc_sec)
    n = np.arange(0,total_samples, step=(total_samples-1), dtype=np.float64)
    t_sec=n*ts

    #This is the frequency of the chirp over time. We assume linear change in frequency
    chirp_freq_slope_HzPerSec=(f_stop_Hz-f_start_Hz)/Tc_sec

    #Compute the instantaneous frequency which is a linear function
    chirp_instantaneous_freq_Hz=chirp_freq_slope_HzPerSec*t_sec+f_start_Hz
    chirp_instantaneous_angular_freq_radPerSec=2*np.pi*chirp_instantaneous_freq_Hz

    #Since frequency is a change in phase the we can plot it as a phase step
    chirp_phase_step_rad=chirp_instantaneous_angular_freq_radPerSec*ts
    return chirp_phase_step_rad

chirp_phase_step_rad=calc_chirp_phase_step(f_start_Hz, f_stop_Hz, Tc_sec,fs)
print(f'chirp_phase_step_rad[0] = {chirp_phase_step_rad[0]} rad and the end is chirp_phase_step_rad[-1] = {chirp_phase_step_rad[-1]}')
print(f'amplitude = {chirp_phase_step_rad[-1] - chirp_phase_step_rad[0]} rad')
print(f'offset = {(chirp_phase_step_rad[-1] - chirp_phase_step_rad[0])/2+chirp_phase_step_rad[0]} rad')

```

    chirp_phase_step_rad[0] = 0.42742757191697867 rad and the end is chirp_phase_step_rad[-1] = 2.8495061615799755
    amplitude = 2.4220785896629966 rad
    offset = 1.638466866748477 rad

```python
#We can see, unlike the complex exponential, the chirp's instantaneous frequency is linearly increasing.
#This corresponds with the linearly increasing phase step.
fig, ax = plt.subplots(2, 1, sharex=True,figsize = [8, 8])
lns1=ax[0].plot(t_usec,chirp_instantaneous_freq_Hz,linewidth=4, label='instantanous frequency');
ax[0].set_title('Comparing the instantaneous frequency and phase step')
ax[0].set_ylabel('instantaneous frequency (Hz)')

axt = ax[0].twinx()
lns2=axt.plot(t_usec,chirp_phase_step_rad,linewidth=2,color='black', linestyle=':', label='phase step');
axt.set_ylabel('phase step (rad)')

#ref: https://stackoverflow.com/questions/5484922/secondary-axis-with-twinx-how-to-add-to-legend
lns = lns1+lns2
labs = [l.get_label() for l in lns]
ax[0].legend(lns, labs, loc=0)

#We see that summing or integrating the linearly increasing phase step gives a quadratic function of total phase.
ax[1].plot(t_usec,chirp_phase_rad,linewidth=4,label='chirp');
ax[1].plot([t_usec[0], t_usec[-1]],[chirp_phase_rad[0], chirp_phase_rad[-1]],linewidth=1, linestyle=':',label='linear (x=y)');
ax[1].set_title('Cumulative quandratic phase function of chirp')
ax[1].set_xlabel('time ($\mu$sec)')
ax[1].set_ylabel('total phase (rad)')
ax[1].legend();

```

    <IPython.core.display.Javascript object>

```python
print(f'chirp_phase_step_rad[0] = {chirp_phase_step_rad[0]} rad and the end is chirp_phase_step_rad[-1] = {chirp_phase_step_rad[-1]}')
print(f'amplitude = {chirp_phase_step_rad[-1] - chirp_phase_step_rad[0]} rad')
print(f'offset = {(chirp_phase_step_rad[-1] - chirp_phase_step_rad[0])/2+chirp_phase_step_rad[0]} rad')
```

    chirp_phase_step_rad[0] = 0.42742757191697867 rad and the end is chirp_phase_step_rad[-1] = 2.8495061615799755
    amplitude = 2.4220785896629966 rad
    offset = 1.638466866748477 rad


# COURSE: Master math by coding in Python
## SECTION: Introduction to Sympy and LaTeX

#### https://www.udemy.com/course/math-with-python/?couponCode=MXC-DISC4ALL
#### INSTRUCTOR: sincxpress.com

Note about this code: Each video in this section of the course corresponds to a section of code below. Please note that this code roughly matches the code shown in the live recording, but is not exactly the same -- the variable names, order of lines, and parameters may be slightly different.

```python

```

# VIDEO: Intro to sympy, part 1

```python
# import the sympy package
import sympy as sym

# optional setup for "fancy" printing (used later)
# sym.init_printing()

# import additional functions for nice printing
from IPython.display import display, Math
```

```python
# create symbolic variables
x,y,z = sym.symbols('x,y,z')

x

```

```python
# x doesn't have a value; it's just a symbol
x + 4
```

```python
# for "fancy" printing (note: you need to run this line only once)
sym.init_printing()
x + 4
```

```python
# more fun...
display( x**y )
display( y/z )
```

```python
# let's compare with numpy
import numpy as np

display(sym.sqrt(2)) # square root in symbolic math
display(np.sqrt(2)) # square root in numeric/precision math

```

### Exercises

```python
# 1)
display(y*x**2)

# 2)
display( sym.sqrt(4)*x )

# 3)
display( sym.sqrt(x)*sym.sqrt(x) )
```

# VIDEO: Intro to Latex

```python
# basic latex coding is easy:
display(Math('4+5=7'))
```

```python
# special characters are indicated using \\
display(Math('\\sigma = \\mu \\times \\sqrt{5}'))

# outside Python, use one \
display(Math('\\sigma = \\mu \times \\sqrt{5}'))

# subscripts and superscripts
display(Math('x_n + y^m - z^{m+n}'))

# fractions
display(Math('\\frac{1+x}{2e^{\pi}}'))

# right-click to change properties
```

```python
# regular text requires a special tag
f = 4
display(Math('Set x equal to %g'%f))

display(Math('\\text{Set x equal to %g}'%f))
```

### Latex code in a markdown cell

Note: this is not for evaluating variables or other numerical code!

$$ \frac{1+\sqrt{2x}}{e^{i\pi}} $$

### Exercises!

```python
# 1)
display(Math('4x+5y-8z=17'))

# 2)
display(Math('\\sin(2\\pi f t+\\theta)'))

# 3)
display(Math('e=mc^2'))

# 4)
display(Math('\\frac{4+5x^2}{(1+x)(1-x)}'))
```

# VIDEO: Intro to sympy, part 2

```python
# using Greek characters as variable names

mu,alpha,sigma = sym.symbols('mu,alpha,sigma')

expr = sym.exp( (mu-alpha)**2/ (2*sigma**2) )

display(expr)

```

```python
# can also use longer and more informative variable names
hello = sym.symbols('hello')

hello/3

```

```python
# substituting numbers for variables

# don't forget to define variables before using them!
display(x+4)

(x+4).subs(x,3)
```

```python
# substituting multiple variables

x,y = sym.symbols('x,y') # can create multiple variables in one line
expr = x+y+5

# substituting one variable
print( expr.subs(x,5) )

# substituting multiple variables
print( expr.subs({x:5,y:4}) )

```

```python
# using sympy to print latex code
expr = 3/x

print( sym.latex(expr) )

# notice:
print( sym.latex(3/4) )
print( sym.latex('3/4') )
# but
print( sym.latex(sym.sympify('3/4')) )

```

### Exercise!

```python
for i in range(-2,3):
    ans = (x+4).subs(x,i**2)
    display(Math('\\text{With }x=%g:\; x^2+4 \\quad \\Rightarrow \\quad %g^2+4 =%g' %(i,i,ans)))
```

# VIDEO: Example: Use sympy to understand the law of exponents

```python
x,y,z = sym.symbols('x,y,z')

ex = x**y * x**z

display(ex)
display( sym.simplify(ex) )

```

```python
expr1 = x**y * x**z
expr2 = x**y / x**z
expr3 = x**y * y**z

display(Math('%s = %s' %( sym.latex(expr1),sym.latex(sym.simplify(expr1)) ) ))
display(Math('%s = %s' %( sym.latex(expr2),sym.latex(sym.simplify(expr2)) ) ))
display(Math('%s = %s' %( sym.latex(expr3),sym.latex(sym.simplify(expr3)) ) ))

```

```python
# using sym.Eq

sym.Eq(4,2+2)
```

```python
display( sym.Eq(x-3,4) )

# using variables
lhs = x-3
rhs = x
display( sym.Eq(lhs,rhs) )
```

```python
lhs = x-3
rhs = x+4#-7
display( sym.Eq(lhs,rhs) )

# or
sym.Eq( lhs-rhs )
```

#### NOTE ABOUT `SYM.EQ`
Sympy changed the behavior of `sym.Eq` since I made this course. To test against zero, you need to write `sym.Eq(expr,0)` instead of just `sym.Eq(expr)`.

If you are using a sympy version earlier than 1.5, you don't need to change anything. If you get a warning or error message, then simply add `,0` as the second input to `sym.Eq`.

To test which version of sympy you have, type
`sym.__version__`

```python
display( sym.Eq(expr1,expr1) )

# but...
display( sym.Eq(expr1 - sym.simplify(expr1)) )

display( sym.Eq(sym.expand(  expr1-sym.simplify(expr1)  )) )

```

```python
# btw, there's also a powsimp function:
display( sym.powsimp(expr1) )

# and its converse
res = sym.powsimp(expr1)
display( sym.expand_power_exp(res) )
```

# VIDEO: printing with f-strings

```python
# basic intro to f-strings

svar = 'Mike'
nvar = 7

print(f'Hi my name is {svar} and I eat {nvar} chocolates every day.')

```

```python
# now with latex integration

x,y = sym.symbols('x,y')
expr = 3/x

# trying to print using symbolic variables
display(Math(f'\\frac{x}{y}'))
display(Math(f'\\frac{{sym.latex(expr)}}{{y}}'))
display(Math(f'\\frac{{{x}}}{{{y}}}'))
display(Math(f'\\frac{{{sym.latex(expr)}}}{{{y}}}'))

# my preference for mixing replacements with latex
display(Math('\\frac{%s}{%s}'%(sym.latex(expr),y)))
```

```python
# print using numeric variables
u,w = 504,438

display(Math(f'\\frac{u}{w}'))
display(Math(f'\\frac{{u}}{{w}}'))
display(Math(f'\\frac{{{u}}}{{{w}}}'))

# my preference for mixing replacements with latex
display(Math('\\frac{%g}{%g}'%(u,w)))
```

```python

```

# VIDEO: Sympy and LaTex: Bug hunt!

```python
mu,alpha = sym.symbols('mu,alpha')

expr = 2*sym.exp(mu**2/alpha)

display(Math( sym.latex(expr) ))
```

```python
Math('1234 + \\frac{3x}{\sin(2\pi t+\\theta)}')
```

```python
a = '3'
b = '4'

# answer should be 7
print(sym.sympify(a)+sym.sympify(b))

```

```python
x = sym.symbols('x')
sym.solve( 4*x - 2 )
```

```python
# part 1 of 2

q = x**2
r = x**2

display(q)
display(r)
```

```python
# part 2 of 2

q,r = sym.symbols('q,r')

q = sym.sympify('x^2')
r = sym.sympify('x**2')

display(Math(sym.latex(q)))
display(r)

sym.Eq(q,r)
```

```python
x = sym.symbols('x')

equation = (4*x**2 - 5*x + 10)**(1/2)
display(equation)
equation.subs(x,3)
```

```python
x,y = sym.symbols('x,y')

equation = 1/4*x*y**2 - x*(5*x + 10*y**2)**(3)
display(equation)
```


# Mandatory exercises

1. Consider the ODE
\begin{equation}
    \begin{split}
        y'-y-t=0,\\
        y(0)=0
    \end{split}
\end{equation}

    1. Use forward Euler method to compute an approximation of the solution at $t=0.2$ with step size $h=0.1$.
    2. Use backward Euler method to compute an approximation of the solution at $t=0.2$ with step size $h=0.1$.
    3. Use classical Runge-Kutta method to compute an approximation of the solution at $t=0.1$ with step size $h=0.1$.

2. Consider the ODE
\begin{equation}
    \begin{split}
        y'-cos(yt)=0,\\
        y(0)=0
    \end{split}
\end{equation}
Write down the backward Euler formula to solve the ODE. Comparing with problem 1.2, what is the difficulty here?

3. Ordinary differential equations can be used to describe the movement of a satellite around a planet. Let $x(t)$ and $y(t)$ denote the $x$ and $y$ coordinate, respectively, for the satellite position at time $t$. The planet’s position is at the origin. Then the following model describes how the position of the satellite changes with time:
\begin{equation}
    \begin{split}
        x''(t)= -k\frac{x(t)}{\left(\sqrt{x(t)^2+y(t)^2}\right)^3}\\
        y''(t)= -k\frac{y(t)}{\left(\sqrt{x(t)^2+y(t)^2}\right)^3}
    \end{split}
\end{equation}
For simplicity, assume that $k = 1$. The initial time is $t = 0$, the initial satellite position $(x(0), y(0)) = (a, b)$ and the initial velocity in the $x$-and $y$-directions $(x'(0), y'(0)) = (c, d)$.

    1. Formulate the model above as a system of first order ordinary differential equations. Remember initial conditions.
    2. (*) Describe how you can use the model to simulate the satellite orbit around the planet for $0\leq t\leq100$ with the help of Matlab’s built-in function `ode45` or Python function `scipy.integrate.solve_ivp`. You do not have to write a complete program. It is enough to write the code for representing the mathematical model and to show how `ode45` or `scipy.integrate.solve_ivp` can be called to solve the ODE system.
    3. (*) You wrote a code for the SIR model in the lab session. Can you use that code to simulate the movement of a satellite around a planet?

# Non-mandatory exercises

4. Implement the forward Euler method in Matlab/Python. For simplicity we consider scalar ODEs. The function header should be:
```Fortran
function [t, y] = euler(f,a,b,y0,n)
```
or
```python
def euler(f,a,b,y0,n):
    ...
    return [t, y]
```
The input parameter `f` should be a function handle to a Matlab/Python function evaluating the right-hand side of the ODE. The values `a` and `b` represent
initial time and final time, respectively. The last input parameter, `n`, is the number of time steps.
The output parameter `y` is a vector, where $y_k$ is the approximate solution at time step `k`, $k = 0,\dots, n$.

5. What modifications would be required in your function `euler`, to make it able to handle *a system* of ODEs?

6. Your next task is to simulate how an object is winched from the ground and into a helicopter. As the object is winched, it will make a pendular movement under the helicopter. Consequently, this process can be modelled as a variable–length pendulum:
\begin{equation}
    \theta''(t)+2\frac{r'(t)}{r(t)}\theta'(t)+\frac{g}{r(t)}\sin(\theta(t))=0
\end{equation}
where $\theta(t)$ is the angle of the pendulum, $r(t)$ is the length of the pendulum, and $g$ is the gravity. The winching is represented by a ”winching
strategy” in the form of a model for how $r'(t)$ varies with time. This gives an additional differential equation:
\begin{equation}
    r'(t)=f(t,\theta(t),\theta'(t)).
\end{equation}
The purpose of the simulation is to investigate the effect of different winching strategies.
Suppose that you are to use Heun’s method to simulate how the object is winched to the helicopter. Your present task is to give a detailed
description of how to apply Heun’s method to this problem. (Note: this is a purely theoretical exercise. You are not required to carry out the
computations, only to describe the approach in principle.)


This is my attempt at trying to resolve the issue of different results from $P_{n}\prime cos \theta$ in eqn. 12.98.

For this solution App II, eqn. 65 is referred to.

Applying the appendix II eqn. 65, to 12.98, I'd done the following.

```python

from sympy import symbols, diff, legendre, cos, sin, simplify, pi
alpha, m,n,z,theta = symbols('alpha m n z theta')

```

```python
Pnz = legendre(n,z) # the 'generic term'
```

## 1) The 'naive' substitution where $z = cos \theta$
This is what I'd done before, and shown Gaurav.

```python
generic_subs = simplify(diff(Pnz, z).subs(z, cos(theta)))
generic_subs

```

$\displaystyle - \frac{n \left(\cos{\left(\theta \right)} P_{n}\left(\cos{\left(\theta \right)}\right) - P_{n - 1}\left(\cos{\left(\theta \right)}\right)\right)}{\sin^{2}{\left(\theta \right)}}$

## 2) Do the partial derivative ( $\frac{\partial}{\partial \theta}P_{n}\prime cos \theta$ ) anew with sympy

```python
Pncostheta = legendre(n, cos(theta))
specific_partialdiff = simplify(diff(Pncostheta, theta))
specific_partialdiff
```

$\displaystyle \frac{n \left(\cos{\left(\theta \right)} P_{n}\left(\cos{\left(\theta \right)}\right) - P_{n - 1}\left(\cos{\left(\theta \right)}\right)\right)}{\sin{\left(\theta \right)}}$

This already gives something different!!

## 3) eqn. 12.98 as written in textbook

```python
eqn1298 = - (n*(n+1)/((2*n+1)*sin(theta)))*(legendre(n-1, cos(theta))-legendre(n+1, cos(theta)))
eqn1298
```

$\displaystyle - \frac{n \left(n + 1\right) \left(P_{n - 1}\left(\cos{\left(\theta \right)}\right) - P_{n + 1}\left(\cos{\left(\theta \right)}\right)\right)}{\left(2 n + 1\right) \sin{\left(\theta \right)}}$

```python
n_values = [2, 10, 21]
theta_values = [pi/4, pi/15, 1.98*pi]

for nval, thetaval in zip(n_values, theta_values):
    values = {'theta': thetaval,
            'n': nval}
    # substitute numeric values and see what happens
    subst_values = [ eqn1298.subs(values).evalf(), specific_partialdiff.subs(values).evalf(), generic_subs.subs(values).evalf()]
    print(subst_values)
```

    [-1.50000000000000, -1.50000000000000, 2.12132034355964]
    [-5.85744699177188, -5.85744699177188, 28.1727639688434]
    [11.4510073170455, 11.4510073170455, 182.368411710620]

The order of numeric values is from the equations defined by cases 3),2) and 1) in that order.

## The textbook eqn. 12.98 is CORRECT!
It was my naive substitution that led to different results somewhere.

## Matter 2: *m,n* index ordering.
This is me verifying stuff for the [Mathematics Exchange post](https://math.stackexchange.com/q/4156607/933933) I opened.

In the textbook, the solution for $K_{mn}$ (eqn. 12.107), when $m \neq n$ is given by:

$\frac{sin\:\alpha( P_{m}(cos\:\alpha)P^{\prime}_{n}(cos\:\alpha) - P_{n}(cos\:\alpha)P^{\prime}_{m}(cos\:\alpha))}{m(m+1) - n(n+1)}$

However, the Mathematica code implementation has the equivalent of:

$\frac{sin\:\alpha( P_{n}(cos\:\alpha)P^{\prime}_{m}(cos\:\alpha) - P_{m}(cos\:\alpha)P^{\prime}_{n}(cos\:\alpha))}{m(m+1) - n(n+1)}$

```python
Pncosalpha = legendre(n, cos(alpha))
Pmcosalpha = Pncosalpha.subs(n,m)

Pnpr_cosalpha = diff(Pncosalpha, alpha)
Pmpr_cosalpha = diff(Pmcosalpha, alpha)

```

```python
intextbook = (sin(alpha)/(m*(m+1)-n*(n+1)))*(Pmcosalpha*Pnpr_cosalpha - Pncosalpha*Pmpr_cosalpha)
intextbook
```

$\displaystyle \frac{\left(\frac{m \left(\cos{\left(\alpha \right)} P_{m}\left(\cos{\left(\alpha \right)}\right) - P_{m - 1}\left(\cos{\left(\alpha \right)}\right)\right) \sin{\left(\alpha \right)} P_{n}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1} - \frac{n \left(\cos{\left(\alpha \right)} P_{n}\left(\cos{\left(\alpha \right)}\right) - P_{n - 1}\left(\cos{\left(\alpha \right)}\right)\right) \sin{\left(\alpha \right)} P_{m}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1}\right) \sin{\left(\alpha \right)}}{m \left(m + 1\right) - n \left(n + 1\right)}$

```python
incode = (sin(alpha)/(m*(m+1)-n*(n+1)))*(Pncosalpha*Pmpr_cosalpha - Pmcosalpha*Pnpr_cosalpha)
incode
```

$\displaystyle \frac{\left(- \frac{m \left(\cos{\left(\alpha \right)} P_{m}\left(\cos{\left(\alpha \right)}\right) - P_{m - 1}\left(\cos{\left(\alpha \right)}\right)\right) \sin{\left(\alpha \right)} P_{n}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1} + \frac{n \left(\cos{\left(\alpha \right)} P_{n}\left(\cos{\left(\alpha \right)}\right) - P_{n - 1}\left(\cos{\left(\alpha \right)}\right)\right) \sin{\left(\alpha \right)} P_{m}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1}\right) \sin{\left(\alpha \right)}}{m \left(m + 1\right) - n \left(n + 1\right)}$

```python
subs_vals = {'alpha':pi/3, 'm':5,'n':2}
incode.subs(subs_vals)
```

$\displaystyle - \frac{147}{32768}$

```python
intextbook.subs(subs_vals)
```

$\displaystyle \frac{147}{32768}$

```python

```


# Functions
So far in this course we've explored equations that perform algebraic operations to produce one or more results. A *function* is a way of encapsulating an operation that takes an input and produces exactly one ouput.

For example, consider the following function definition:

\begin{equation}f(x) = x^{2} + 2\end{equation}

This defines a function named ***f*** that accepts one input (***x***) and returns a single value that is the result calculated by the expression *x2 + 2*.

Having defined the function, we can use it for any input value. For example:

\begin{equation}f(3) = 11\end{equation}

You've already seen a few examples of Python functions, which are defined using the **def** keyword. However, the strict definition of an algebraic function is that it must return a single value. Here's an example of defining and using a Python function that meets this criteria:

```python
# define a function to return x^2 + 2
def f(x):
    return x**2 + 2

# call the function
f(3)
```

    11

You can use functions in equations, just like any other term. For example, consider the following equation:

\begin{equation}y = f(x) - 1\end{equation}

To calculate a value for ***y***, we take the ***f*** of ***x*** and subtract 1. So assuming that ***f*** is defined as previously, given an ***x*** value of 4, this equation returns a ***y*** value of **17** (*f*(4) returns 42 + 2, so 16 + 2 = 18; and then the equation subtracts 1 to give us 17). Here it is in Python:

```python
x = 4
y = f(x) - 1
print(y)
```

    17

Of course, the value returned by a function depends on the input; and you can graph this with the iput (let's call it ***x***) on one axis and the output (***f(x)***) on the other.

```python
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,f(x), color='purple')

plt.show()
```

As you can see (if you hadn't already figured it out), our function is a *quadratic function* - it returns a squared value that results in a parabolic graph when the output for multiple input values are plotted.

## Bounds of a Function
Some functions will work for any input and may return any output. For example, consider the function ***u*** defined here:

\begin{equation}u(x) = x + 1\end{equation}

This function simply adds 1 to whatever input is passed to it, so it will produce a defined output for any value of ***x*** that is a *real* number; in other words, any "regular" number - but not an *imaginary* number like √-1, or ∞ (infinity). You can specify the set of real numbers using the symbol ${\rm I\!R}$ (note the double stroke). The values that can be used for ***x*** can be expressed as a *set*, which we indicate by enclosing all of the members of the set in "{...}" braces; so to indicate the set of all possible values for x such that x is a member of the set of all real numbers, we can use the following expression:

\begin{equation}\{x \in \rm I\!R\}\end{equation}

### Domain of a Function
We call the set of numbers for which a function can return value it's *domain*, and in this case, the domain of ***u*** is the set of all real numbers; which is actually the default assumption for most functions.

Now consider the following function ***g***:

\begin{equation}g(x) = (\frac{12}{2x})^{2}\end{equation}

If we use this function with an ***x*** value of **2**, we would get the output **9**; because (12 ÷ (2•2))2 is 9. Similarly, if we use the value **-3** for ***x***, the output will be **4**. However, what happens when we apply this function to an ***x*** value of **0**? Anything divided by 0 is undefined, so the function ***g*** doesn't work for an ***x*** value of 0.

So we need a way to denote the domain of the function ***g*** by indicating the input values for which a defined output can be returned. Specifically, we need to restrict ***x*** to a specific list of values - specifically any real number that is not 0. To indicate this, we can use the following notation:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is not equal to 0*, and we can incorporate this into the function's definition like this:

\begin{equation}g(x) = (\frac{12}{2x})^{2}, \{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

Or more simply:

\begin{equation}g(x) = (\frac{12}{2x})^{2},\;\; x \ne 0\end{equation}

When you plot the output of a function, you can indicate the gaps caused by input values that are not in the function's domain by plotting an empty circle to show that the function is not defined at this point:

```python
%matplotlib inline

# Define function g
def g(x):
    if x != 0:
        return (12/(2*x))**2

# Plot output from function g
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [g(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

# plot an empty circle to show the undefined point
plt.plot(0,g(0.0000001), color='purple', marker='o', markerfacecolor='w', markersize=8)

plt.show()
```

Note that the function works for every value other than 0; so the function is defined for x = 0.000000001, and for x = -0.000000001; it only fails to return a defined value for exactly 0.

OK, let's take another example. Consider this function:

\begin{equation}h(x) = 2\sqrt{x}\end{equation}

Applying this function to a non-negative ***x*** value returns a meaningful output; but for any value where ***x*** is negative, the output is undefined.

We can indicate the domain of this function in its definition like this:

\begin{equation}h(x) = 2\sqrt{x}, \{x \in \rm I\!R\;\;|\;\; x \ge 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is greater than or equal to 0*.

Or, you might see this in a simpler format:

\begin{equation}h(x) = 2\sqrt{x},\;\; x \ge 0\end{equation}

Note that the symbol ≥ is used to indicate that the value must be *greater than **or equal to*** 0; and this means that **0** is included in the set of valid values. To indicate that the value must be *greater than 0, **not including 0***, use the > symbol. You can also use the equivalent symbols for *less than or equal to* (≤) and *less than* (<).

When plotting a function line that marks the end of a continuous range, the end of the line is shown as a circle, which is filled if the function includes the value at that point, and unfilled if it does not.

Here's the Python to plot function ***h***:

```python
%matplotlib inline

def h(x):
    if x >= 0:
        import numpy as np
        return 2 * np.sqrt(x)

# Plot output from function h
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [h(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='purple')

# plot a filled circle at the end to indicate a closed interval
plt.plot(0, h(0), color='purple', marker='o', markerfacecolor='purple', markersize=8)

plt.show()
```

Sometimes, a function may be defined for a specific *interval*; for example, for all values between 0 and 5:

\begin{equation}j(x) = x + 2,\;\; x \ge 0 \text{ and } x \le 5\end{equation}

In this case, the function is defined for ***x*** values between 0 and 5 *inclusive*; in other words, **0** and **5** are included in the set of defined values. This is known as a *closed* interval and can be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \le x \le 5 \}\end{equation}

It could also be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; [0,5] \}\end{equation}

If the condition in the function was **x > 0 and x < 5**, then the interval would be described as *open* and 0 and 5 would *not* be included in the set of defined values. This would be indicated using one of the following expressions:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \lt x \lt 5 \}\end{equation}
\begin{equation}\{x \in \rm I\!R\;\;|\;\; (0,5) \}\end{equation}

Here's function ***j*** in Python:

```python
%matplotlib inline

def j(x):
    if x >= 0 and x <= 5:
        return x + 2

# Plot output from function j
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)
y = [j(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('j(x)')
plt.grid()

# Plot x against k(x)
plt.plot(x, y, color='purple')

# plot a filled circle at the ends to indicate an open interval
plt.plot(0, j(0), color='purple', marker='o', markerfacecolor='purple', markersize=8)
plt.plot(5, j(5), color='purple', marker='o', markerfacecolor='purple', markersize=8)

plt.show()
```

Now, suppose we have a function like this:

\begin{equation}
k(x) = \begin{cases}
  0, & \text{if } x = 0, \\
  1, & \text{if } x = 100
\end{cases}
\end{equation}

In this case, the function has highly restricted domain; it only returns a defined output for 0 and 100. No output for any other ***x*** value is defined. In this case, the set of the domain is:

\begin{equation}\{0,100\}\end{equation}

Note that this does not include all real numbers, it only includes 0 and 100.

When we use Python to plot this function, note that it only makes sense to plot a scatter plot showing the individual values returned, there is no line in between because the function is not continuous between the values within the domain.

```python
%matplotlib inline

def k(x):
    if x == 0:
        return 0
    elif x == 100:
        return 1

# Plot output from function k
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)
# Get the k(x) values for every value in x
y = [k(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('k(x)')
plt.grid()

# Plot x against k(x)
plt.scatter(x, y, color='purple')

plt.show()
```

### Range of a Function
Just as the domain of a function defines the set of values for which the function is defined, the *range* of a function defines the set of possible outputs from the function.

For example, consider the following function:

\begin{equation}p(x) = x^{2} + 1\end{equation}

The domain of this function is all real numbers. However, this is a quadratic function, so the output values will form a parabola; and since the function has no negative coefficient or constant, it will be an upward opening parabola with a vertex that has a y value of 1.

So what does that tell us? Well, the minimum value that will be returned by this function is 1, so it's range is:

\begin{equation}\{p(x) \in \rm I\!R\;\;|\;\; p(x) \ge 1 \}\end{equation}

Let's create and plot the function for a range of ***x*** values in Python:

```python
%matplotlib inline

# define a function to return x^2 + 1
def p(x):
    return x**2 + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('p(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,p(x), color='purple')

plt.show()
```

Note that the ***p(x)*** values in the plot drop exponentially for ***x*** values that are negative, and then rise exponentially for positive ***x*** values; but the minimum value returned by the function (for an *x* value of 0) is **1**.

```python

```

Basics Of Algebra by Hiren/01-08-Functions.ipynb