```
# Authors:
## Antonio del Rio Chanona      https://www.imperial.ac.uk/people/a.del-rio-chanona
## Edgar Ivan Sanchez Medina    https://www.mpi-magdeburg.mpg.de/person/103552/2316

import numpy as np
import matplotlib.pyplot as plt
from IPython.display import HTML
from matplotlib.animation import FuncAnimation
```

Along this notebook you will find text in blue that describe the **coding tasks** that you have to implement. Within the code cell, your implementation needs to be between the line:

`#-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#-#`

and the line:

`#-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#-#`

# **1. Optimization basics**

An optimization problem has the form

\begin{equation}
\begin{aligned}
\min_{x \in X} \quad & f(x)\\
\textrm{s.t.} \quad & g_i(x) \leq 0, i= 1, ..., m\\
\end{aligned}
\end{equation}

The vector $x = (x_1,..., x_n)$ is the optimization variable of the problem, the function $f : \mathbb{R}^n \to \mathbb{R} $ is the objective function, the functions $g_i : \mathbb{R}^n \to \mathbb{R} $,
$i = 1, ...,m$, are the (inequality) constraint functions. A vector $x^⋆$ is called optimal, if it has the smallest objective value among all vectors
that satisfy the constraints.

**Convex optimization**

A convex optimization problem is one in which the objective and the
constraint functions are convex, which means they satisfy the inequality:

\begin{equation}
f(\alpha x + \beta y) \leq \alpha f(x) + \beta f(y)
\end{equation}

for all $x,y \in \mathbb{R}^n$ and all $\alpha, \beta \in \mathbb{R}$ with $\alpha + \beta = 1$, $\alpha  \geq 0$ and $\beta  \geq 0$

It is worth nothing that any equality constraint can be represented as two inequality constraints. This enforce that, in a convex problem, the equality constraints must be linear.

Therefore, an optimization problem is convex whenever the following requirements are met:

* The objective function is convex.
* The inequality constraints are convex.
* The equality constraints are linear.

**Global and local optima**

An optimal solution $x^*$ is said to be the global optimum when the constraints are met at this point and $f(x^*) \leq f(x),  \forall x \in X$. When this condition is met only whitin a certain neightborhood, the solution is called local optimum.

# **2. Gradient descent**

The optimization methods known as "descent methods" minimize a function by applying the following update rule at each iteration:

\begin{equation}
x^{(k+1)} = x^{(k)} +  \alpha^{(k)} \Delta x^{(k)}
\end{equation}

in this rule, $\Delta^{(k)}$ denotes the **direction** at the iteration $k$, and $\alpha^{(k)} \geq 0$ is a scalar value called **step size**. These methods are called descent methods, because at each iteration $f \left(x^{(k+1)} \right) \leq  f \left(x^{(k)} \right)$.

In the method called *Gradient Descent* the direction is chosen to be the negative of the gradient: $ \Delta x := - \nabla f(x) $. therefore, the algorithm is:

**Algorithm**

1. Given a starting point $x \in X$.
2. Repeat
3. $~~~~~~$ $ \Delta x := - \nabla f(x) $
2. $~~~~~~$ Choose $\alpha$.
3. $~~~~~~$ Update: $ x = x +  \alpha \Delta x$
4. until stopping criterion is met.

Code step 3 and 5 within the following code.

```
############################
# --- Gradient Descent --- #
############################

def gradient_descent(f, x0, grad_f, lr, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    Gradient Descent
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        lr       : Learning rate
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:

        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#-#
        grad_i  =      # compute gradient
        x       =      # compute step
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#

        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using Gradient Descent \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

One way to approximate the (numerically) gradient of a function  is the **finite diffirences method**. There exist mainly three type of finite difference approximations:

* Backward difference  $f'(x) \approx \frac{f(x_k) - f(x_k - \epsilon)}{\epsilon}$
* Forward difference  $f'(x) \approx \frac{f(x_k + \epsilon) + f(x_k)}{\epsilon}$
* Central difference $f'(x) \approx \frac{f(x_k + \frac{\epsilon}{2}) - f(x_k - \frac{\epsilon}{2})}{\epsilon}$

However, the central difference approximation gives the most accurate one among these three. Therefore, let's implement that one here.

Implement the **central finite differences**.

```
######################################
# --- Central finite differences --- #
######################################

def central_finite_diff(f, x):
      '''
      Central finite differences approximation.
      INPUTS:
          f  : Function
          x  : Position where to approximate the gradient
      OUTPUTS:
          grad: Approximation of the gradient of f at x
      '''
      dim = x.shape[0]
      eps  = np.sqrt(np.finfo(float).eps)  # Step-size is the square root of the machine precision
      grad = np.zeros((1,dim))

      for i in range(dim):
          e           = np.zeros((1,dim))
          e[0,i]      = eps
          #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#-#
          grad_approx =       # compute central finite diff.
          #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#-#
          grad[0,i]     = grad_approx

      return grad
```

The next cell contains the test function that we are going to use here, but this can be replace by any function. In order to approximate the gradients we are going to use the central finite differences method with five-points.

```
def Rosenbrock_f(x):
    '''
    Rosenbrock function
    '''
    n = np.shape(x)[1]
    z = np.sum(100*(x[:,1:] - x[:,:n-1]**2)**2 + (x[:,:n-1] - 1)**2, axis=1)
    return z

###############################################
# --- Central finite differences 5 points --- #
###############################################

def central_finite_diff5(f, x):
      '''
      Five-points method for central finite differences.
      INPUTS:
          f  : Function
          x  : Position where to approximate the gradient
      OUTPUTS:
          grad: Approximation of the gradient of f at x
      '''
      dim = x.shape[1]
      # Step-size is taken as the square root of the machine precision
      eps  = np.sqrt(np.finfo(float).eps)
      grad = np.zeros((1,dim))

      for i in range(dim):
          e           = np.zeros((1,dim))
          e[0,i]      = eps
          grad_approx = (f(x - 2*e) - 8*f(x - e) + 8*f(x + e) - f(x + 2*e) )/(12*eps)

          grad[0,i]     = grad_approx

      return grad
```

```
# --- Gradient Descent --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = gradient_descent(Rosenbrock_f, x0, central_finite_diff5, 0.001, traj=True)
```

```
# Plot function
x_1 = np.linspace(0,1)
x_2 = np.linspace(0,1)
X, Y = np.meshgrid(x_1, x_2)
Z = Rosenbrock_f(np.append(X.reshape(-1,1), Y.reshape(-1,1), axis=1))
Z = Z.reshape(X.shape)

x_list    = np.array(x_list).reshape(-1,x0.shape[1])

x_summary = []
f_summary = []
for i in range(x_list.shape[0]):
  if i % 100 == 0:
    x_summary.append(x_list[i])
    f_summary.append(f_list[i])
x_summary = np.array(x_summary).reshape(-1,x0.shape[1])

fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
display_value = ax.text(0.05, 0.2, '', transform=ax.transAxes)
plt.close()

def animate(i):
    ax.plot(x_summary[:i, 0], x_summary[:i, 1], 'k.', alpha=0.6)    # Animate points
    display_value.set_text('Min = ' + str(f_summary[i]))          # Animate display value
    ax.set_title('Rosenbrock function, Iteration: ' + str(i*100))  # Animate title
    return display_value

anim = FuncAnimation(fig, animate, frames=len(f_summary), interval=100, repeat_delay=800)

HTML(anim.to_jshtml())
```

# **3. Steepest descent**

By contrast, the steepest descent method uses the steepest descent direction (as opposed to the negative gradient in Gradient Descent) as the direction to move to at each iteration.

In order to find the steepest direction, we can approximate
the function via a first-order Taylor expansion:

\begin{equation}
f(x + \Delta x) \approx f(x) + \nabla f(x)^T \Delta x
\end{equation}

The second term of this approximation, $\nabla f(x)^T \Delta x$,  gives the approximate change in $f$ for a small step $\Delta x$. Therefore, to minimize the function $f$ we have to make this second term, $\nabla f(x)^T \Delta x$, as negative as possible. To do this, we have to choose the step $\Delta x$ such that $\Delta x = \text{argmin} \{ \nabla f(x)^T \Delta x | \| \Delta x \| =1 \}$. The constraint, $\| \Delta x \| = 1$, ensures that the solution to this problem is sensible, othewise a very large $\Delta x$ would always be chosen.

**Algorithm**

1. Given a starting point $x \in X$.
2. Repeat
3. $~~~~~~$ $ \Delta x := \|f(x)\| \ast \text{argmin} \{ \nabla f(x)^T \Delta x | \| \Delta x \| =1 \} $
2. $~~~~~~$ Choose $\alpha$.
3. $~~~~~~$ Update: $ x = x +  \alpha \Delta x$
4. until stopping criterion is met.

In general, we may consider various norms for the minimization problem. And the interpretation of steepest descent with different norms leads to different algorithms. For example, by using the $l_2$ norm:

\begin{equation}
\Delta x = \text{argmin} \{ \nabla f(x)^T \Delta x | \| \Delta x \|_2 =1 \}
\end{equation}

From the Cauchy-Schwarz inequality we know that

\begin{equation}
\nabla f(x)^T \Delta x \leq \| \nabla f(x) \| \| \Delta x \|
\end{equation}

with equality when $\Delta x = \lambda \nabla f(x)$, $\lambda \in \mathbb{R}$. Since $ \| \Delta x \| = 1$ and we want to minimize:

\begin{equation}
\Delta x^* = - \frac{\nabla f(x)}{\| \nabla f(x) \|}
\end{equation}

and the update rule becomes exactly equal to the Gradient Descent one (this is not the case if we take a different norm). For this reason we say that Gradient Descent is just a special case of Steepest Descent.

#### **Exact line search**

Ideally we would need to take the step size $\alpha$ (aka learning rate in the Machine Learning field) that minimizes the funtion at the next point. In other words we would need to solve the following in order to get the optimal step size at each iteration:

\begin{equation}
\alpha^* \in \text{argmin}_{\alpha \geq 0} ~~~ f \left( x^{(k)} - \alpha \nabla f \left(x^{(k)} \right) \right)
\end{equation}

Solving the above problem is known as *exact line search*, but this would require to solve an adittional optimization problem that can be computationaly expensive in most applications. Therefore, the learning rate $\alpha$ is most of the times chosen heuristically or using other approximations to the exact line search (e.g. backtracking).

#### **Backtracking line search**

As mentioned before, backtracking line search is one of the approximation methods for the exact line search, and it relies on two constants $A$ and $B$, such that $0 < A < 0.5$ and $0 < B < 1$. The role of the constant $A$ is to reduce the slope of the line in which the search will be performed (see following plot), and the constant $B$ weights the previous learning rate $\alpha$.

Therefore, at each iteration we would have:

* Set $\alpha = 1$
* While the following condition holds: $f \left( x - \alpha \nabla f(x) \right) > f(x) - A \alpha \nabla f(x)^T \nabla f(x)$, reduce $\alpha$ according to the following: $\alpha := B \alpha$. The inequality condition used here is known as the Armijo–Goldstein condition.

Typical values for these two constants are $A = [0.01, 0.3]$ and $B = [0.1, 0.8]$.

# **4. Gradient descent with momentum**

The idea behind this extension is to avoid most of the (unnecesary) zig-zag movements of Gradient descent by accumulating momentum along the direction towards the optimum while we iterate.

Therefore, the update rule of Gradient Descent is modified like this:

\begin{equation}
x^{(k+1)} = x^{(k)} + v^{(k)}
\end{equation}

where $v^{(k)}$ is the velocity term defined by:

\begin{equation}
v^{(k)} = \beta v^{(k-1)} - \alpha \nabla f(x^{(k)})
\end{equation}

where $\beta \in [0,1]$ is the momentum hyperparameter commonly set to 0.9.

Code a function for the **velocity** term calculation and a function for the **line search** method.

```
####################
# --- Momentum --- #
####################

def momentum(grad_i, v_prev, lr, beta=0.9):
    '''
    Momentum function
    INPUTS:
        grad_i  : Gradient of function at current position
        v_prev  : velocity value at the previous position
        beta    : Momentum hyperparameter
    OUTPUTS:
        v       : Velocity term
    '''
    #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#-#
    v =               # compute velocity term
    #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#

    return v

#######################
# --- Line search --- #
#######################

def ls(grad_i, x, f):
    '''
    Line search for determining learning rate
    INPUTS:
        grad_i  : Gradient of function at current position
        x       : Current position
        f       : Objective function
    OUTPUTS:
        lr    : Optimal learning rate
        iter  : Number of iterations needed in line search
    '''
    iter = 0
    lr   = 1
    #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#-#
          # compute line search loop

    #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#
        iter += 1

    return lr, iter

#############################################
# --- Line search with Armijo-Goldstein --- #
#############################################

def line_search(grad_i, x, f, A=0.1, B=0.8):
    '''
    Line search for determining learning rate
    INPUTS:
        grad_i  : Gradient of function at current position
        x       : Current position
        f       : Objective function
    OUTPUTS:
        lr    : Optimal learning rate
        iter  : Number of iterations needed in line search
    '''
    iter = 0
    lr   = 1
    # Armijo-Goldstein condition loop
    while f(x - lr*grad_i) > f(x) - A*lr*np.dot(grad_i,grad_i.T) and iter<100:
        lr  = B*lr
        iter += 1

    return lr, iter
```

```
##########################################################
# --- Gradient Descent with line search and momentum --- #
##########################################################

def GD_ls_momentum(f, x0, grad_f, beta=0.9, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    Gradient Descent with line search and momentum
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        beta     : Parameter beta for the momentum calculation
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    v_prev = 0      # initialize at zero to get normal GD at first step
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                               # compute gradient
        lr      = line_search(grad_i, x, f)[0]              # compute learning rate using line search
        v       = momentum(grad_i, v_prev, lr, beta=beta)   # compute momentum
        x       = x + v                                     # compute step
        v_prev  = v                                         # update previous momentum term
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using Gradient Descent with momentum \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- Gradient Descent with line search and momentum --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = GD_ls_momentum(Rosenbrock_f, x0, central_finite_diff5, beta=0.95, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with line search and momentum')
plt.show()
```

# **5. Nesterov Accelerated Gradient Descent**

The Nesterov Accelerated Gradient Descent (NAG) is a further improvement to the Gradient Descent with momentum algorithm. The step direction in NAG is calculated based on the gradient on an approximated future position instead of the current position, in this way, more gradient information is included into the update step compared to the traditional momentum approach.

Therefore, the velocity term in NAG is determined by:

$v^{(k)} = \beta v^{(k-1)} - \alpha \nabla f(\tilde{x}^{(k)})$

where $\tilde{x}^{(k)}$ is the approximated future position calculated as:

$\tilde{x}^{(k)} = x^{(k)} + \beta v^{(k-1)}$

Code a function for the **nesterov** calculation of velocity.

```
####################
# --- Nesterov --- #
####################

def nesterov(grad_tilde, v_prev, lr, beta=0.9):
    '''
    Momentum function
    INPUTS:
        grad_tilde  : Gradient of function at nesterov modified position
        v_prev      : velocity value at the previous position
        beta        : Momentum hyperparameter
    OUTPUTS:
        v           : Velocity term
    '''
    #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#
    v =               # compute the Nesterov velocity term
    #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#

    return v
```

Implement your function **`nesterov` into the `NAG`** function bellow.

```
#################################
# --- NAG with line search  --- #
#################################

def NAG(f, x0, grad_f, beta=0.9, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    Nesterov Accelerated Gradient Descent with line search
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        beta     : Parameter beta for the momentum calculation
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    v_prev = 0      # initialize at zero to get normal GD-momentum at first step
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                               # compute gradient at current position

        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#
        x_tilde =             # nesterov modified position
        g_tilde =             # compute gradient of function at x_tilde
        lr      =             # compute learning rate using line search
        v       =             # compute momentum
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#

        x       = x + v                                     # compute step
        v_prev  = v                                         # update previous momentum term
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using NAG \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- NAG --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = NAG(Rosenbrock_f, x0, central_finite_diff5, beta=0.95, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with NAG')
plt.show()
```

# **6. AdaGrad**

We have been using line search as a way to determine a reasonable value for the learning rate. However, in such approach, the learning rate $\alpha$ is a unique value that is applied in all dimensions of the gradient. Can you see the problem of this? Imagine you have different ranges on each dimension of the gradient, then the learning rate will favour each dimension differently. One straightforward approach would be to have a learning rate for each dimension, i.e. the $\alpha$ would be a vector instead of a scalar. The problem with this is that in many applications you can easily end-up with thousand or millions of dimensions (think about a Deep Neural Network). Therefore, one of the first approaches to this problem was AdaGrad, which adaptively scale the learning rate for each dimension. Another advantage is that we do not have to adjust the learning rate manually, as this is adaptively adjusted.

The way in which AdaGrad does what we explained above is to update the position according to:

$x^{(k)} = x^{(k)} - \alpha \frac{\nabla f(x^{(k)})}{V^{(k)}}$

where $V^{(k)}$ is the accumulate historical gradient that is calculated as:

$V^{(k)} = \sqrt{\sum_{i=1}^{k} \left(\nabla f(x^{(k)})\right)^2 + \epsilon}$

where $\epsilon$ is just a small number that prevents the division by zero.

Code a function for the **gradient historical accumulation** term.

```
############################################
# --- Gradient historical accumulation --- #
############################################

def grad_accumulation(g_traj, eps=1e-8):
    '''
    Gradient historical accumulation
    Note: A vectorized form might be more efficient, but we will keep it like
          this, for educational purposes.
    INPUTS:
        g_traj  : List of historical gradients
        eps     : small number to prevent division by zero
    OUTPUTS:
        V       : Gradient accumulation
    '''
    dim    = g_traj[0].shape[1]
    g_traj = np.array(g_traj).reshape(-1, dim)
    #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#-#
    V      =          # compute accumulation historical gradient
    #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#-#

    return V
```

```
###################
# --- AdaGrad --- #
###################

def adagrad(f, x0, grad_f, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    AdaGrad optimization algorithm
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10
    g_traj = []             # To store gradient trajectory

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                  # compute gradient
        g_traj.append(grad_i)                  # save gradient
        lr      = 0.01                        # learning rate
        if iter_i == 0:
          x  = x - lr*grad_i                # compute step at first iteration
        else:
          V  = grad_accumulation(g_traj)
          x  = x - lr*grad_i/V              # compute step
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

        # print optimization status
        if iter_i % 2000 == 0:
          print('Current iteration: ', iter_i)

    print(' Optimization using AdaGrad \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- AdaGrad --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = adagrad(Rosenbrock_f, x0, central_finite_diff5, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with AdaGrad')
plt.show()
```

As you can see, the problem becomes evident: **for this problem AdaGrad is super slow!**
This is because, of course, by reducing the gradient at each iteration the steps that we take are very small. And, since we are accumulating the gradients, we are reducing the gradient by a larger amount at each iteration. Therefore, the nice feature of AdaGrad of taking care of the different ranges in the dimensions of the gradient comes at the expense of very slow convergence.

This takes us to an important point in optimization: **Always scale your features!**

You can use [standarization or min-max sclaing](https://en.wikipedia.org/wiki/Feature_scaling) for example, but the idea of scaling your features is that you overcome this difficulty of having enormous differences in the ranges in your gradient.

# **7. AdaDelta**

AdaDelta is an extension of AdaGrad that tries to overcome the slow convergence by not considering all the accumulated historical gradients, but only the gradients in a certain window over a given period. This is accomplished by calculating $V^{(k)}$ as follows:

$V^{(k)} = \sqrt{\rho V^{(k-1)} + (1-\rho) \left(\nabla f(x^{(k)})\right)^2 + \epsilon}$

where $\rho$ is a exponential decay parameter, usually set to 0.95. The reason for this specific form is that, instead of inneficiently store $w$ number of gradients, the same calculation can be obtained as the exponentially decaying average of the squared gradients that we see in the above expression.

In the [original paper](https://arxiv.org/pdf/1212.5701.pdf), the authors noted that the hypothetical units in the update are not consistent. Hence, they defined an accumulation term also for the parameters updates. For this reason, they also definied the accumulation of the parameter updates as follows:

$E^{(k)} = \sqrt{\rho E^{(k-1)} + (1-\rho) \Delta {x^{(k)}}^2 + \epsilon}$

and since  $\Delta x^{(k)}$ is unknown at the current iteration, it is approximated using the previous position as:

$ \Delta x^{(k)} = - \frac{E^{(k-1)}}{V^{(k)}} \nabla f(x^{(k)}) $

```
####################
# --- AdaDelta --- #
####################

def adadelta(f, x0, grad_f, rho=0.95, eps=1e-8, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    AdaDelta optimization algorithm
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        rho      : Exponential decay parameter
        eps      : Small constant to avoid division over zero
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    E_g = 0; E_x = 0
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                                           # compute gradient
        E_g     = rho*E_g + (1-rho)*grad_i*grad_i                       # exponential decay average on gradients
        x_delta = - np.sqrt(E_x + eps)*grad_i/np.sqrt(E_g + eps)        # compute x_delta
        E_x     = rho*E_x + (1-rho)*x_delta*x_delta                     # exponential decay average on parameters
        x       = x + x_delta                                           # compute step
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using AdaDelta \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- AdaDelta --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = adadelta(Rosenbrock_f, x0, central_finite_diff5, rho=0.9, eps=1e-8, max_iter=3000, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with AdaDelta')
plt.show()
```

As you can see, for this specific problem, AdaDelta does not do a good job. However, if you set the maximum number of iterations to some number, e.g. 5000, for both AdaGrad and AdaDelta and compare the results, you will see that AdaDelta jumps faster into a zone close to the optimum.

The problems on the convergence in AdaDelta has been reported in the literature (e.g. Section 4.4 of this [book](https://books.google.de/books?id=IbnEDwAAQBAJ&pg=PA189&lpg=PA189&dq=adadelta+not+converging&source=bl&ots=f2i8liEovl&sig=ACfU3U2nzVAPCLLtC3Os_cxmHmh7acOBww&hl=en&sa=X&ved=2ahUKEwjl4YLJ7MTqAhWNs4sKHQUfCrQQ6AEwA3oECAgQAQ#v=onepage&q=adadelta%20not%20converging&f=false) and [this paper](https://openreview.net/pdf?id=ryQu7f-RZ)). And even the original paper make the comparison of AdaDelta saying:

* *Setting the hyperparameters to $\epsilon=1^{-6}$ and $\rho=0.95$ we
achieve 2.00% test set error compared to the 2.10% of Schaul
et al. While this is nowhere near convergence it gives a sense
of how quickly the algorithms can optimize the classification
objective.*

You might be asking why then AdaDelta is used, even if it fails to optimize our (relatively simple) Rosenbrock function. A posible answer is: in many machine learning applications, specially in Deep learning, being close to the optimum is enough for the task we want to accomplish. And if AdaDelta takes us relatively fast to this zone, then maybe is worth considering.

Another interesting thing to note here is that even though you do not have a learning rate that has to be tunned, the hyperparameter $\epsilon$ influences the behaviour of AdaDelta quite significantly for certain problems (e.g. ours here).

Try different **combinations of the hyperparameters** $\epsilon$ and $\rho$ and see their influence in convergence. Also, **compare AdaGrad and AdaDelta** in terms of how far one goes towards the optimum in a fixed number of iterations.

# **8. RMSProp**

RMSProp was proposed by Geoffrey Hinton in his [Coursera lecture](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf) and it is pretty much the same as the idea 1 presented at the [AdaDelta paper](https://arxiv.org/pdf/1212.5701.pdf). This idea is: adapt the learning rate using the gradiet information of the previous $w$ steps by computing the average exponential decay. Hence, RMSprop update rule is:

$x^{(k)} = x^{(k)} - \alpha \frac{\nabla f(x^{(k)})}{V^{(k)}}$

where

$V^{(k)} = \sqrt{\rho V^{(k-1)} + (1-\rho) \left(\nabla f(x^{(k)})\right)^2 + \epsilon}$

For the sake of completeness let's also implement it in such form.

Code the **accumulation of squared gradients** of RMSProp.

```
####################
# --- RMSProp --- #
####################

def rmsprop(f, x0, grad_f, lr=0.001, rho=0.95, eps=1e-8, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    RMSProp optimization algorithm
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        rho      : Exponential decay parameter
        eps      : Small constant to avoid division over zero
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    V = 0
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                                           # compute gradient
        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#-#
        V       =           # exponential decay average on gradients
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#-#
        x       = x - lr*grad_i/V                                       # compute step
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using RMSProp \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- RMSProp --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = rmsprop(Rosenbrock_f, x0, central_finite_diff5, rho=0.9, eps=1e-8, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with RMSProp')
plt.show()
```

Well, this is a surprise, the algorithm converged to the optimum if we use the RMSProp form!

Similar result was encounter in the [book](https://books.google.de/books?id=IbnEDwAAQBAJ&pg=PA189&lpg=PA189&dq=adadelta+not+converging&source=bl&ots=f2i8liEovl&sig=ACfU3U2nzVAPCLLtC3Os_cxmHmh7acOBww&hl=en&sa=X&ved=2ahUKEwjl4YLJ7MTqAhWNs4sKHQUfCrQQ6AEwA3oECAgQAQ#v=onepage&q=rmsprop&f=false) we mentioned aboved. The reason is that the accumulation of the parameters update ($E^{(k-1)}$ in AdaDelta) can act as an accelerator term at the fist iterations. However, when approximating the optimum, this same "kind of momentum" prevents the algorithm from convergence.

# **9. Adam**

Adam combines the nice key property of the momentum methods with the adaptive learning rate methods. In addition to keep the accumulation term of the squared gradients $V^{(k)}$, Adam also has an accumulation term for past gradients (like the momentum methods).

The strategy of Adam is to calculate two moments for the gradients:

*   First moment (mean):

$~~~~~~~~~~~~~~~~~ m^{(k)}=\beta_1 m^{(k-1)} + (1-\beta_1) \nabla f(x^{(k)})$

*   Second moment (uncentered variance):

$~~~~~~~~~~~~~~~~~ V^{(k)}= \beta_2 V^{(k-1)} + (1-\beta_2) \left( \nabla f(x^{(k)}) \right)^2$

where $\beta_1$ and $\beta_2$ are exponential decay rates. Recommended values for $\beta_1$, $\beta_2$ and $\epsilon$ are 0.9, 0.999 and $10^{-8}$ respectively.

However, the [authors noted](https://arxiv.org/pdf/1412.6980.pdf) that during the first iterations the method is biased towards zero. Terefore, they used bias-corrected moments defined as:

*   First bias-corrected moment (mean):

$~~~~~~~~~~~~~~~~~ \hat{m}^{(k)}=\frac{m^{(k)}}{1- \beta_1^{k}}$

*   Second moment (uncentered variance):

$~~~~~~~~~~~~~~~~~ \hat{V}^{(k)}= \frac{V^{(k)}}{1- \beta_2^{k}}$

Note the terms $\beta_1^{k}$ and $\beta_2^{k}$ are the beta values to the power of the iteration number. Therefore, the update rule for Adam is:

$x^{(k+1)} = x^{(k)} - \alpha \frac{\hat{m}^{(k)}}{\sqrt{\hat{V}^{(k)}} + \epsilon}$

Implement the **moments equations** and the **update rule** of Adam.

```
################
# --- Adam --- #
################

def adam(f, x0, grad_f, lr=0.1, beta_1=0.9, beta_2=0.999, eps=1e-8, max_iter=1e5, grad_tol=1e-4, traj=False):
    '''
    Adam optimization algorithm
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        beta_1   : Exponential decay parameter 1
        beta_2   : Exponential decay parameter 2
        eps      : Small constant to avoid division over zero
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        plot     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # plotting
    if traj == True:
        x_list = []
        f_list = []

    # optimization loop
    m = 0;  V = 0
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                                           # compute gradient
        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
        m       =                    # Moment 1
        V       =                    # Moment 2
        m_hat   =                    # Biased-corrected moment 1
        V_hat   =                    # Biased-corrected moment 2
        x       =                    # compute step
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#-#
        iter_i += 1

        # plotting
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(' Optimization using Adam \n')
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list,

    return x, iter_i
```

```
# --- Adam --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = adam(Rosenbrock_f, x0, central_finite_diff5, lr=0.05, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title('Rosenbrock with Adam')
plt.show()
```

# **10. Newton's method**

This optimization method calculates the inverse of the Hessian matrix to obtain faster convergence than the first-order gradient descent methods.

The update rule of the Newton's method is:

$ x^{(k+1)} = x^{(k)} - \left(\nabla^2 f(x^{(k)}) \right)^{-1}\nabla f(x^{(k)}) $

The second term from the right hand side of the above equation is then the Newton step, and it requires, from the inverse of the Hessian, for the Hessian matrix to be positive definite. However, for some non-convex problems the Hessian might not be invertible.

Similarly as what we did for approximating the gradient using central finite differences, we can approximate the Hessian. For the details and formulas of the Hessian approximation look [here](https://en.wikipedia.org/wiki/Finite_difference).

```
###################################################
# --- Central second order finite differences --- #
###################################################

def Second_diff_fxx(f, x):
    '''
      Central finite differences approximation of Hessian
      INPUTS:
          f  : Function
          x  : Position where to approximate the Hessian
      OUTPUTS:
          Hxx: Approximation of the Hessian of f at x
      '''

    dim   = np.shape(x)[1]
    # Step-size is taken as the square root of the machine precision
    eps  = np.sqrt(np.finfo(float).eps)
    Hxx   = np.zeros((dim,dim))

    for j in range(dim):
        # compute Fxx (diagonal elements)
        x_d_f       = np.copy(x)             # forward step
        x_d_b       = np.copy(x)             # backward step
        x_d_f[0,j]  = x_d_f[0,j] + eps
        x_d_b[0,j]  = x_d_b[0,j] - eps
        Hxx[j,j]    = (f(x_d_f) -2*f(x) + f(x_d_b))/eps**2

        for i in range(j+1,dim):
            # compute Fxy (off-diagonal elements)
            # Fxy
            x_d_fxfy    = np.copy(x_d_f)
            x_d_fxfy[0,i] = x_d_fxfy[0,i] + eps
            x_d_fxby    = np.copy(x_d_f)
            x_d_fxby[0,i] = x_d_fxby[0,i] - eps
            x_d_bxfy    = np.copy(x_d_b)
            x_d_bxfy[0,i] = x_d_bxfy[0,i] + eps
            x_d_bxby    = np.copy(x_d_b)
            x_d_bxby[0,i] = x_d_bxby[0,i] - eps
            Hxx[j,i]    = (f(x_d_fxfy) - f(x_d_fxby) - f(x_d_bxfy) + f(x_d_bxby))/(4*eps**2)
            Hxx[i,j]    = Hxx[j,i]

    return Hxx

x1       = np.array([2.,2.]).reshape(1,-1)
Hxx = Second_diff_fxx(Rosenbrock_f, x1)
print('Hxx = ',Hxx)
```

```
###########################
# --- Newton's Method --- #
###########################

def newton(f, x0, grad_f, H_f, max_iter=1e3, grad_tol=1e-4, traj=False):
    '''
    Newton's method

    Note: the Hessian can become ill-conditioned
    INPUTS:
        f        : Function
        x0       : Initial guess
        grad_f   : Gradient function
        H_f      : Hessian function
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        traj     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    n      = np.shape(x0)[0]
    x      = np.copy(x0)
    iter_i = 0
    grad_i = grad_tol*10

    # trajectory
    if traj == True:
            x_list = []
            f_list = []

    # optimization loop
    while np.sum(np.abs(grad_i)) > grad_tol and iter_i < max_iter:
        grad_i  = grad_f(f,x)                         # compute gradient
        Hxx     = H_f(f,x)                            # compute Hessian
        x       = x - (np.linalg.inv(Hxx)@grad_i.T).T # update
        iter_i += 1

        # trajectory
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(" Optimization using Newton's method \n")
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)
    print('Final grad: ', grad_i)

    # trajectory
    if traj == True:
        return x, x_list, f_list

    return x, iter_i
```

```
# --- Newton's method --- #
x0 = np.array([0.,0.]).reshape(1,-1)

xf, x_list, f_list = newton(Rosenbrock_f, x0, central_finite_diff5, Second_diff_fxx, traj=True)
```

```
x_array = np.array(x_list).reshape(-1,2)
fig  = plt.figure()
cs   = plt.scatter(x_array[:,0], x_array[:,1], marker=".", c=f_list , cmap="seismic")
cbar = fig.colorbar(cs)
plt.xlabel(r'$x_1$'); plt.ylabel(r'$x_2$'); plt.title("Rosenbrock with Newton's method")
plt.show()
```

Of course, using the Hessian information accelerates the optimization significantly! But, this comes at the expense of not only obtaining the Hessian, but also inverting it! This is the reason why second-order methods are not used in practice with large datasets. But, if the dataset is relatively small, go ahead and use second-order methods (whenever you have access to the second derivatives, of course).

# **11. Constrained Newton's method**

If we recall the Newton's method, the step (called Newton step) that we take at each iteration is:

\begin{equation}
\Delta x = - \left(\nabla^2 f(x^{(k)}) \right)^{-1}\nabla f(x^{(k)})
\end{equation}

A nice interpretation of the Newton’s step is to set
it by minimizing the second-order approximation of $f$ at $x$.
In case of (equality) constrained problems, the Newton's method is slightly modified in two main aspects:

* The initial point must be feasible.
* The definition of Newton step is modified to make sure that the newton step is a feasible direction of the problem.

To derive the Constrained Newton's method the objective function in the original problem:

\begin{equation}
\begin{aligned}
\min_{x \in X} \quad & f(x)\\
\textrm{s.t.} \quad & h_i(x) = 0, i= 1, ..., m\\
\end{aligned}
\end{equation}

is approximated by the second-order Taylor expansion:

\begin{equation}
\begin{aligned}
\min_{\Delta x \in X} \quad & \hat{f}(x + \Delta x) = f(x) + \nabla f(x)^T \Delta x + \frac{1}{2} \Delta x ^T \nabla^2 f(x) \Delta x\\
\textrm{s.t.} \quad & h_i(x + \Delta x) = 0, i= 1, ..., m\\
\end{aligned}
\end{equation}

The Newton step $\Delta x$ now can be obtained by solving the following system of equations:

\begin{gather}
 \begin{bmatrix}
 \nabla^2 f(x) & \nabla h(x)^T \\
 \nabla h(x) & 0
 \end{bmatrix}
 \begin{bmatrix}
 \Delta x \\
 \lambda
 \end{bmatrix}
 =
 -
 \begin{bmatrix}
 \nabla f(x) \\
 h(x)
 \end{bmatrix}
\end{gather}

The stopping criterion is regularly defined by the Newton decrement as follows:

\begin{equation}
 \frac{\Delta x^T \nabla^2 f(x) \Delta x}{2} \leq \epsilon
\end{equation}

In this example, we are going to work with the following simple problem:

\begin{equation}
\begin{aligned}
\min_{x_1, x_2 \in X} \quad & x_1^2 + 5x_2^2\\
\textrm{s.t.} \quad & -x_1-x_2-2 = 0
\end{aligned}
\end{equation}

Also, we are going to use the exact gradient and Hessian.

```
def Obj(x):
  '''
  Objective function
  '''
  return x[:,0]**2 + 5*x[:,1]**2

def gradient(x):
  '''
  Gradient of objective function
  '''
  g1 = 2*x[:,0]
  g2 = 10*x[:,1]
  return  np.array([g1,g2])

exact_hess = np.array([[2,0],[0, 10]])  # Exact Hessian of objective function

def Const(x):
  '''
  Equality constrained
  '''
  return -x[:,0] - x[:,1] - 2

J_constraint = np.array([-1, -1]).reshape(1,-1)
```

```
# Plot
x_1 = np.linspace(-4,4)
x_2 = np.linspace(-2,2)
x = np.vstack((x_1,x_2)).reshape(-1,2)
X, Y = np.meshgrid(x_1, x_2)
Z = Obj(np.append(X.reshape(-1,1), Y.reshape(-1,1), axis=1))
Z = Z.reshape(X.shape)

const_val = Const(x)

fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
plt.xlim(-4, 4)
plt.ylim(-2, 2)
plt.plot(x_1, -x_1-2, 'b-',label='h(x)=0')
plt.legend()
```

First, let's solve it with a Scipy solver just for comparison and to make sure we obtain the same. Note we are using a different algorithm (SLSQP), since constrained optimization using Scipy is restricted to only a few methods.

```
# Using Scipy SLSQP

from scipy.optimize import minimize

def obj_f(x):
  ''' Re-write objective function in the scipy form'''
  return x[0]**2 + 5*x[1]**2

initial = np.array([-2, 0])
cons = ({'type': 'eq', 'fun': lambda x:  -x[0] -x[1] - 2})

res = minimize(obj_f, initial, method='SLSQP', tol=1e-15, constraints=cons)
print(res.fun)
print(res.x)
```

Implement the **constrained Newton step**.

```
###################################
# --- Constrained Newton step --- #
###################################

def Const_Newton_step(H, J, g, h):
  '''
  Determine the constrained Newton step

    INPUTS:
      H: Hessian matrix of the objective function at x
      J: Jacobian of constraints at x
      g: Gradient of objective function at x
      h: Constraints value at x

    OUTPUTS:
      step: Constrained Newton step
      lambda: Lagrange multipliers
  '''
  dim = H.shape[0]
  num_const = J.shape[0]
  dim_lag = dim + num_const
  #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
  A                 =    # Create matrix of zeros
  A[:dim,:dim]      =    # Substitute the Hessian of objective function
  A[dim:,:dim][0]   =    # Substitute the Jacobian of constraints
  A[:dim,dim:][:,0] =    # Substitute the Jacobian of constraints
  B     =                # Append the gradient of objective function and constraint values
  sol   =                # Solve the matrix system
  #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#
  step  = sol[:dim]
  lamda = sol[dim:]

  return step, lamda
```

Code the steps of the **constrained Newton's method**.

```
#######################################
# --- Constrained Newton's Method --- #
#######################################

def const_newton(f, const, x0, grad_f, H_f, max_iter=1e3, tol=1e-4, traj=False):
    '''
    Constrained Newton's method

    INPUTS:
        f        : Function
        const    : Constraint
        x0       : Initial guess
        grad_f   : Gradient function
        H_f      : Hessian function
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        traj     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''
    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    decr   = 10

    # trajectory
    if traj == True:
            x_list = []
            f_list = []

    # optimization loop
    while decr/2 > tol and iter_i < max_iter:

        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
        H       =                # compute Hessian of Obj function
        J       =                # compute Jacobian of constraint
        g       =                # compute gradient of Obj function
        h       =                # compute constraint value
        delta_x, lamb =          # compute step and lambda
        x       =                # update x
        decr    =                # decrement
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#

        iter_i += 1

        # trajectory
        if traj == True:
            x_list.append(x.flatten().tolist())
            f_list.append(f(x))

    print(" Optimization using constrained Newton's method \n")
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)

    # trajectory
    if traj == True:
        return x, x_list, f_list

    return x, iter_i
```

```
# --- Constrained Newton's method --- #
x0 = np.array([3,1.5]).reshape(1,-1)

xf, x_list, f_list = const_newton(Obj, Const, x0, gradient, Second_diff_fxx, traj=True)
```

```
# Plot solution
fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
plt.xlim(-4, 4)
plt.ylim(-2, 2)
plt.plot(x_1, -x_1-2, 'b-',label='h(x)=0')
plt.plot(x0[:,0], x0[:,1], 'ro', label='initial point')
plt.plot(x_list[1][0], x_list[1][1], 'r*', label='Optimum', ms=10)
plt.legend()
```

# **12. Barrier method (interior-point methods)**

With the Constrained Newton's method we accomplished to solve optimization problem with equality constraints. With interior-point methods we can solve convex problems that include inequality constraints. Interior-point methods solve this type of problems by solving a sequence of equality constrained problems using Newton's method. The barrier method is one of these interior-point algorithms, and is the one we are going to focus here.

The goal is to approximately formulate the inequality constrained problem

\begin{equation}
\begin{aligned}
\min_{x \in X} \quad & f(x)\\
\textrm{s.t.} \quad & h_i(x) = 0, i= 1, ..., m\\
\quad & g_j(x) \leq 0, j=1,...,n
\end{aligned}
\end{equation}

as an equality constrained problem to which Newton’s method can be applied. This is accomplished by making the inequality constraints $g_j(x)$ implicit in the objective function $f(x)$.

\begin{equation}
\begin{aligned}
\min_{x \in X} \quad & f(x) + \sum_{j=1}^{n}I_-(g_j(x))\\
\textrm{s.t.} \quad & h_i(x) = 0, i= 1, ..., m\\
\end{aligned}
\end{equation}

where $I_-$ is the indicator function for the nonpositive reals:

\begin{equation}
\begin{aligned}
    I_-(u)=
\begin{cases}
    0, &  u \leq 0\\
    \infty,    & u > 0
\end{cases}
\end{aligned}
\end{equation}

The problem with this formulation is that, even when we accomplished to make the inequality constraints implicit in the objective function, this objective function is not (in general) differentiable, so Newton's method cannot be applied. The basic idea of the barrier method is to approximate this indicator function as:

\begin{equation}
    \hat{I}_-(u) = - \frac{1}{t} \log (-u)
\end{equation}

Therefore, the approximated problem that is going to be solved is:

\begin{equation}
\begin{aligned}
\min_{x \in X} \quad & f(x) + \sum_{j=1}^{n}- \frac{1}{t} \log (-g_j(x))\\
\textrm{s.t.} \quad & h_i(x) = 0, i= 1, ..., m\\
\end{aligned}
\end{equation}

where the function $\phi(x) = - \sum_{j=1}^{n} \log (-g_j(x))$ is called the *logarithmic barrier* for the problem.

The accuracy of this approximation improves as the parameter $t$ increases. However, when the parameter $t$ is large, the objective function
is difficult to minimize by Newton’s method, since its Hessian varies rapidly near the boundary of the feasible set. This problem can be overcomed by solving a sequence of problems of the same form, increasing the parameter $t$ (and therefore the accuracy of the approximation) at each step, and starting each Newton minimization at the solution of the problem for the previous value of t.

The gradient and the Hessian of the logarithmic barrier function are given by:

\begin{equation}
    \nabla \phi (x) = \sum_{j=1}^{n} - \frac{1}{g_j(x)} \nabla g_j(x)
\end{equation}

\begin{equation}
    \nabla^2 \phi (x) = \sum_{j=1}^{n} \frac{1}{g_j(x)^2} \nabla g_j(x) \nabla g_j(x)^T - \sum_{j=1}^{n} \frac{1}{g_j(x)} \nabla^2 g_j(x)
\end{equation}

**Algorithm**

1. Given a strictly feasible starting point $x \in X$, $t := t^{(0)}$ $\alpha > 0$, $\epsilon >0$.
2. Repeat until $n/t \leq \epsilon$
3. $~~~~~~$ Solve approximated problem with $t:= t^{(k)}$ starting from $x^{k}$ using Newton's method
2. $~~~~~~$ Update $x^{k+1} := x^{*(k)}$
3. $~~~~~~$ Increase $t^{k+1} := \alpha t^{(k)}$

The example problem to solve is:

\begin{equation}
\begin{aligned}
\min_{x_1, x_2 \in X} \quad & x_1^2 + 5x_2^2\\
\textrm{s.t.} \quad & -x_1-x_2-2 = 0\\
& -x_1-x_2 \leq 0
\end{aligned}
\end{equation}

Here, we will use exact gradients and Hessians.

```
def Ineq_const(x):
  '''
  Inequality constraint
  '''
  return -x[:,0] + x[:,1]

grad_ineq = np.array([-1, 1])           # Gradient of inequality constraint
hess_ineq = np.array([[0,0], [0,0]])    # Hessian of inequality constraint

```

```
# Plot
x = np.vstack((x_1,x_2)).reshape(-1,2)
ineq_const_val = Ineq_const(x)
fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
plt.xlim(-4, 4)
plt.ylim(-2, 2)
plt.plot(x_1, -x_1-2, 'b-',label='h(x)=0')
plt.plot(x_1, x_1, 'b--',label='g(x)=0')
plt.fill_between(x_1, x_1, np.repeat(4, len(x_1)), color='k', alpha=0.3)
plt.legend()
```

Observe the impact of the parameter $t$ on the approximation of the objective function of the barrier method. Play, by increasing its value to see how this approximation changes.

```
##########################################
# --- Approximation with log barrier --- #
##########################################

def Obj_barrier(x):
  '''
  Objective function with log barrier
  '''
  t = 0.1
  phi = - np.log(-Ineq_const(x))
  return x[:,0]**2 + (5*x[:,1])**2 + 1/t*phi

# Plot approximation of log barrier
Z2 = Obj_barrier(np.append(X.reshape(-1,1), Y.reshape(-1,1), axis=1))
Z2 = Z2.reshape(X.shape)
const_val = Const(x)
x = np.vstack((x_1,x_2)).reshape(-1,2)
ineq_const_val = Ineq_const(x)

fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
plt.contour(x_1, x_2, Z2, colors='green', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
plt.xlim(-4, 4)
plt.ylim(-2, 2)
plt.plot(x_1, -x_1-2, 'b-',label='h(x)=0')
plt.plot(x_1, x_1, 'b--',label='g(x)=0')
plt.fill_between(x_1, x_1, np.repeat(4, len(x_1)), color='k', alpha=0.3)
plt.legend()
```

```
# Using Scipy SLSQP to see what to expect

initial = np.array([-2, 0])
cons = ({'type': 'eq', 'fun': lambda x:  -x[0] -x[1] - 2},
        {'type': 'ineq', 'fun': lambda x:  x[0] -x[1]})

res = minimize(obj_f, initial, method='SLSQP', tol=1e-15, constraints=cons)
print(res.fun)
print(res.x)
```

Code the **Hessian and the gradient of the logarithmic barrier**.

```
#######################################
# --- Hessian logarithmic barrier --- #
#######################################

def hess_logbarrier(x):
  '''
  Hessian of the logarithmic barrier function
  '''
  val_ineq = Ineq_const(x)[0]         # Value of the inequality constraint at x
  #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
  h1 =          # First term
  h2 =          # Second term
  #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#
  return h1 + h2

########################################
# --- Gradient logarithmic barrier --- #
########################################

def grad_logbarrier(x):
  '''
  Gradient of the logarithmic barrier function
  '''
  val_ineq = Ineq_const(x)[0]         # Value of the inequality constraint at x
  #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
  g1 =                                # Gradient of log barrier
  #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#
  return g1

```

Implement the **barrier method steps**.

```
##########################
# --- Barrier method --- #
##########################

def barrier(f, eq_const, x0, grad_f, t, alpha=1.01, max_iter=1e4, tol=1e-4, traj=False):
    '''
    Barrier method (interior-point methods)

    INPUTS:
        f        : Function
        eq_const : Equality constraint
        x0       : Initial guess
        grad_f   : Gradient function
        t        : Barrier parameter
        alpha    : Parameter to increase t
        max_iter : Maximum number of iterations
        grad_tol : Tolerance for gradient approximation
        traj     : Boolean for plotting
    OUTPUTS:
        x        : Optimal point
        iter_i   : Number of iterations needed
    '''

    # initialize problem
    x      = np.copy(x0)
    iter_i = 0
    n_iq   = 1             # number of inequality constraints

    # trajectory
    if traj == True:
            x_list = []
            f_list = []

    # optimization loop
    while n_iq/t >= tol*0.001 and iter_i < max_iter:
      decr = 10
      while decr/2 > tol and iter_i < max_iter:

        #-#-#-#-#-#-#-#-#-# Start of your code #-#-#-#-#-#-#
        H =                # Hessian of objective function
        J =                # compute Jacobian of constraint
        g =                # Gradient of objective function
        h =                # compute (equality) constraint value
        delta_x, la =      # compute step and lambda
        x       =          # update x
        decr    =          # decrement
        #-#-#-#-#-#-#-#-#-# End of your code #-#-#-#-#-#-#

        iter_i += 1
      t       = t * alpha                         # update t

      # trajectory
      if traj == True:
          x_list.append(x.flatten().tolist())
          f_list.append(f(x))

    print(" Optimization using barrier method \n")
    print('Iterations: ', iter_i)
    print('Optimal x : ', x)

    # trajectory
    if traj == True:
        return x, x_list, f_list

    return x, iter_i
```

```
# --- Barrier method --- #
x0 = np.array([3.5,1.5]).reshape(1,-1)
t0 = 0.001

xf, x_list, f_list = barrier(Obj, Const, x0, gradient, t0, traj=True)
```

```
# Plot solution
fig = plt.figure(figsize=(6,5))
left, bottom, width, height = 0.15, 0.1, 0.8, 0.8
ax = fig.add_axes([left, bottom, width, height])
contours = ax.contour(x_1, x_2, Z, colors='black', alpha=0.8)
ax.clabel(contours, inline=True, fontsize=8)
ax.set_xlabel(r'$x_1$')
ax.set_ylabel(r'$x_2$')
plt.xlim(-4, 4)
plt.ylim(-2, 2)
plt.plot(x0[:,0], x0[:,1], 'ro', label='initial point')
plt.plot(x_list[-1][0], x_list[-1][1], 'r*', label='Optimum', ms=10)
plt.plot(x_1, -x_1-2, 'b-',label='h(x)=0')
plt.plot(x_1, x_1, 'b--',label='g(x)=0')
plt.fill_between(x_1, x_1, np.repeat(4, len(x_1)), color='k', alpha=0.3)
plt.legend()
```


# Dynamic Models

### Preliminaries

- Goal
  - Introduction to dynamic (=temporal) Latent Variable Models, including the Hidden Markov Model and Kalman filter.
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp.605-615 on Hidden Markov Models
    - Bishop pp.635-641 on Kalman filters
    - Faragher (2012), [Understanding the Basis of the Kalman Filter](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Faragher-2012-Understanding-the-Basis-of-the-Kalman-Filter.pdf)
    - Minka (1999), [From Hidden Markov Models to Linear Dynamical Systems](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Minka-1999-from-HMM-to-LDS.pdf)

### Example Problem

- We consider a one-dimensional cart position tracking problem, see [Faragher 2012](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Faragher-2012-Understanding-the-Basis-of-the-Kalman-Filter.pdf).

- The hidden states are the position $z_t$ and velocity $\dot z_t$. We can apply an external acceleration/breaking force $u_t$. (Noisy) observations are represented by $x_t$.

- The equations of motions are given by

$$\begin{align}
\begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} &=  \begin{bmatrix} 1 & \Delta t \\ 0 & 1\end{bmatrix} \begin{bmatrix} z_{t-1} \\ \dot z_{t-1}\end{bmatrix} + \begin{bmatrix} (\Delta t)^2/2 \\ \Delta t\end{bmatrix} u_t + \mathcal{N}(0,\Sigma_z) \\
x_t &= \begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} + \mathcal{N}(0,\Sigma_x)
\end{align}$$

- Task: Infer the position $z_t$ after 10 time steps. (Solution later in this lesson).

### Dynamical Models

- In this lesson, we consider models where the sequence order of observations matters.

- Consider the _ordered_ observation sequence $x^T \triangleq \left(x_1,x_2,\ldots,x_T\right)$.
  - (For brevity, in this lesson we use the notation $x_t^T$ to denote $(x_t,x_{t+1},\ldots,x_T)$ and drop the subscript if $t=1$, so $x^T = x_1^T = \left(x_1,x_2,\ldots,x_T\right)$).

- We wish to develop a generative model
    $$ p( x^T )$$
that 'explains' the time series $x^T$.

- We cannot use the IID assumption $p( x^T ) = \prod_t p(x_t )$. In general, we _can_ use the **chain rule** (a.k.a. **the general product rule**)

$$\begin{align*}
p(x^T) &= p(x_T|x^{T-1}) \,p(x^{T-1}) \\
  &=  p(x_T|x^{T-1}) \,p(x_{T-1}|x^{T-2}) \cdots p(x_2|x_1)\,p(x_1) \\
  &= p(x_1)\prod_{t=2}^T p(x_t\,|\,x^{t-1})
\end{align*}$$

- Generally, we will want to limit the depth of dependencies on previous observations. For example, a $K$th-order linear **Auto-Regressive** (AR) model is given by
$$\begin{align*}
  p(x_t\,|\,x^{t-1}) = \mathcal{N}\left(x_t \,\middle|\,  \sum_{k=1}^K a_k x_{t-k}\,,\sigma^2\,\right)
\end{align*}$$
limits the dependencies to the past $K$ samples.

### State-space Models

- A limitation of AR models is that they need a lot of parameters in order to create a flexible model. E.g., if $x_t$ is an $M$-dimensional discrete variable, then a $K$th-order AR model will have about $M^{K}$ parameters.

- Similar to our work on Gaussian Mixture models, we can create a flexible dynamic system by introducing _latent_ (unobserved) variables  $z^T \triangleq \left(z_1,z_2,\dots,z_T\right)$ (one $z_t$ for each observation $x_t$). In dynamic systems, $z_t$ are called _state variables_.

- A **state space model** is a particular latent variable dynamical model defined by
$$\begin{align*}
 p(x^T,z^T) &= \underbrace{p(z_1)}_{\text{initial state}} \prod_{t=2}^T \underbrace{p(z_t\,|\,z_{t-1})}_{\text{state transitions}}\,\prod_{t=1}^T \underbrace{p(x_t\,|\,z_t)}_{\text{observations}}
\end{align*}$$
  - The condition $p(z_t\,|\,z^{t-1}) = p(z_t\,|\,z_{t-1})$ is called a $1$st-order Markov condition.

- The Forney-style factor graph for a state-space model:

### Hidden Markov Models and Linear Dynamical Systems

- A **Hidden Markov Model** (HMM) is a specific state-space model with discrete-valued state variables $z_t$.

- Typically, $z_t$ is a $K$-dimensional one-hot coded latent 'class indicator' with transition probabilities $a_{jk} \triangleq p(z_{tk}=1\,|\,z_{t-1,j}=1)$, or equivalently,
  $$p(z_t|z_{t-1}) = \prod_{k=1}^K \prod_{j=1}^K a_{jk}^{z_{t-1,j}\cdot z_{tk}}$$
which is usually accompanied by an initial state distribution $p(z_{1k}=1) = \pi_k$.

- The classical HMM has also discrete-valued observations but in pratice any (probabilistic) observation model $p(x_t|z_t)$ may be coupled to the hidden Markov chain.

- Another well-known state-space model with continuous-valued state variables $z_t$ is the **(Linear) Gaussian Dynamical System** (LGDS), which is defined as

$$\begin{align*}
p(z_t\,|\,z_{t-1}) &= \mathcal{N}\left(\, A z_{t-1}\,,\,\Sigma_z\,\right) \\
p(x_t\,|\,z_t) &= \mathcal{N}\left(\, C z_t\,,\,\Sigma_x\,\right) \\
p(z_1) &= \mathcal{N}\left(\, \mu_1\,,\,\Sigma_1\,\right)
\end{align*}$$

- Note that the joint distribution over all states and observations $\{(x_1,z_1),\ldots,(x_t,z_t)\}$ is a (large-dimensional) Gaussian distribution. This means that, in principle, every inference problem on the LGDS model also leads to a Gaussian distribution.

- HMM's and LGDS's (and variants thereof) are at the basis of a wide range of complex information processing systems, such as speech and language recognition, robotics and automatic car navigation, and even processing of DNA sequences.

### Common Signal Processing Tasks as Message Passing-based Inference

- As we have seen, inference tasks in linear Gaussian state space models can be analytically solved.

- However, these derivations quickly become cumbersome and prone to errors.

- Alternatively, we could specify the generative model in a (Forney-style) factor graph and use automated message passing to infer the posterior over the hidden variables. Here follows some examples.

- **Filtering**, a.k.a. state estimation: estimation of a state (at time step $t$), based on past and current (at $t$) observations.

- **Smoothing**: estimation of a state based on both past and future observations. Needs backward messages from the future.

- **Prediction**: estimation of future state or observation based only on observations of the past.

### Kalman Filtering

- Technically, a [**Kalman filter**](https://en.wikipedia.org/wiki/Kalman_filter) is the solution to the recursive estimation (inference) of the hidden state $z_t$ based on past observations in an LGDS, i.e., Kalman filtering solves the problem $p(z_t\,|\,x^t)$ based on the previous estimate $p(z_{t-1}\,|\,x^{t-1})$ and a new observation $x_t$ (in the context of the given model specification of course).

- Let's infer the Kalman filter for a scalar linear Gaussian dynamical system:
$$\begin{align*}
    p(z_t\,|\,z_{t-1}) &= \mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2) \tag{state transition}   \\
    p(x_t\,|\,z_t) &= \mathcal{N}(x_t\,|\,c z_t,\sigma_x^2) \tag{observation}
\end{align*}$$

- Kalman filtering comprises inferring $p(z_t\,|\,x^t)$ from a given prior estimate $p(z_{t-1}\,|\,x^{t-1})$ (available after the previous time step) and a new observation $x_t$. Let us assume that
$$\begin{align}
p(z_{t-1}\,|\,x^{t-1}) = \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) \tag{prior}
\end{align}$$

- Note that everything is Gaussian, so it is _in principle_ possible to execute inference problems analytically and the result will be a Gaussian posterior:
  - (In the following derivation we make use of the renormalization equality $\mathcal{N}(x\,|\,cz,\sigma^2) = \frac{1}{c}\mathcal{N}\left(z \,\big|\,\frac{x}{c},\left(\frac{\sigma}{c}\right)^2\right)$).

$$\begin{align*}
\underbrace{p(z_t\,|\,x^t)}_{\text{posterior}} &= p(z_t\,|\,x_t,x^{t-1}) \propto p(x_t,z_t\,|\,x^{t-1}) \\
  &\propto p(x_t\,|\,z_t) \,p(z_t\,|\,x^{t-1}) \\
  &= p(x_t\,|\,z_t) \, \sum_{z_{t-1}} p(z_t,z_{t-1}\,|\,x^{t-1}) \\
  &= \underbrace{p(x_t\,|\,z_t)}_{\text{observation}} \, \sum_{z_{t-1}} \underbrace{p(z_t\,|\,z_{t-1})}_{\text{state transition}} \, \underbrace{p(z_{t-1}\,|\,x^{t-1})}_{\text{prior}} \\
    &= \mathcal{N}(x_t\,|\,c z_t,\sigma_x^2) \sum_{z_{t-1}} \mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2) \, \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) \\
  &= \frac{1}{c}\mathcal{N}\left(z_t\bigm| \frac{x_t}{c} ,\left(\frac{\sigma_x}{c}\right)^2\right)  \sum_{z_{t-1}} \frac{1}{a}\underbrace{\mathcal{N}\left(z_{t-1}\bigm|  \frac{z_t}{a},\left(\frac{\sigma_z}{a}\right)^2 \right) \, \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2)}_{\text{use Gaussian multiplication formula SRG-6}} \\
  &\propto \underbrace{\mathcal{N}\left(z_t\,\bigm| \,\frac{x_t}{c} ,\left(\frac{\sigma_x}{c}\right)^2\right) \cdot \mathcal{N}\left(z_t\, \bigm|\,a \mu_{t-1},\sigma_z^2 + \left(a \sigma_{t-1}\right)^2 \right)}_{\text{use SRG-6 again}} \\
  &\propto \mathcal{N}\left( z_t \,|\, \mu_t, \sigma_t^2\right)
\end{align*}$$
with
$$\begin{align*}
  \rho_t^2 &= a^2 \sigma_{t-1}^2 + \sigma_z^2 \tag{predicted variance}\\
  K_t &= \frac{c \rho_t^2}{c^2 \rho_t^2 + \sigma_x^2} \tag{Kalman gain} \\
  \mu_t &= \underbrace{a \mu_{t-1}}_{\text{prior prediction}} + K_t \cdot \underbrace{\left( x_t - c a \mu_{t-1}\right)}_{\text{prediction error}} \tag{posterior mean}\\
  \sigma_t^2 &= \left( 1 - c\cdot K_t \right) \rho_t^2 \tag{posterior variance}
\end{align*}$$

- Kalman filtering consists of computing/updating these last four equations for each new observation ($x_t$). This is a very efficient recursive algorithm to estimate the state $z_t$ from all observations (until $t$).

- It turns out that it's also possible to get an analytical result for $p(x_t|x^{t-1})$, which is the **model evidence** in a filtering context. See [optional slides](#kalman-proof) for details.

### Multi-dimensional Kalman Filtering

- The Kalman filter equations can also be derived for multidimensional state-space models. In particular, for the model
$$\begin{align*}
z_t &= A z_{t-1} + \mathcal{N}(0,\Gamma) \\
x_t &= C z_t + \mathcal{N}(0,\Sigma)
\end{align*}$$
the Kalman filter update equations for the posterior $p(z_t |x^t) = \mathcal{N}\left(z_t \bigm| \mu_t, V_t \right)$ are given by (see Bishop, pg.639)
$$\begin{align*}
P_t &= A V_{t-1} A^T + \Gamma \tag{predicted variance}\\
K_t &= P_t C^T \cdot \left(C P_t C^T  + \Sigma \right)^{-1} \tag{Kalman gain} \\
\mu_t &= A \mu_{t-1} + K_t\cdot\left(x_t - C A \mu_{t-1} \right) \tag{posterior state mean}\\
V_t &= \left(I-K_t C \right) P_{t}  \tag{posterior state variance}
\end{align*}$$

### Kalman Filtering and the Cart Position Tracking Example Revisited

#### CODE EXAMPLE

- We can now solve the cart tracking problem of the introductory example by implementing the Kalman filter.

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate()
IJulia.clear_output();
```

```julia
using LinearAlgebra, PyPlot
include("scripts/cart_tracking_helpers.jl")

# Specify the model parameters
Δt = 1.0                     # assume the time steps to be equal in size
A = [1.0 Δt;
     0.0 1.0]
b = [0.5*Δt^2; Δt]
Σz = convert(Matrix,Diagonal([0.2*Δt; 0.1*Δt])) # process noise covariance
Σx = convert(Matrix,Diagonal([1.0; 2.0]))     # observation noise covariance;

# Generate noisy observations
n = 10                # perform 10 timesteps
z_start = [10.0; 2.0] # initial state
u = 0.2 * ones(n)     # constant input u
noisy_x = generateNoisyMeasurements(z_start, u, A, b, Σz, Σx);

m_z = noisy_x[1]                                    # initial predictive mean
V_z = A * (1e8*Diagonal(I,2) * A') + Σz             # initial predictive covariance

for t = 2:n
    global m_z, V_z, m_pred_z, V_pred_z
    #predict
    m_pred_z = A * m_z + b * u[t]                   # predictive mean
    V_pred_z = A * V_z * A' + Σz                    # predictive covariance
    #update
    gain = V_pred_z * inv(V_pred_z + Σx)            # Kalman gain
    m_z = m_pred_z + gain * (noisy_x[t] - m_pred_z) # posterior mean update
    V_z = (Diagonal(I,2)-gain)*V_pred_z             # posterior covariance update
end
println("Prediction: ",ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=m_pred_z,v=V_pred_z))
println("Measurement: ", ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=noisy_x[n],v=Σx))
println("Posterior: ", ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=m_z,v=V_z))
plotCartPrediction2(m_pred_z[1], V_pred_z[1], m_z[1], V_z[1], noisy_x[n][1], Σx[1][1]);
```

### The Cart Tracking Problem Revisited: Inference by Message Passing

- Let's solve the cart tracking problem by sum-product message passing in a factor graph like the one depicted above. All we have to do is create factor nodes for the state-transition model $p(z_t|z_{t-1})$ and the observation model $p(x_t|z_t)$. Then we just build the factor graph and let [ForneyLab](http://forneylab.org) execute the message passing schedule.

- Since the factor graph is just a concatination of $n$ identical "sections" (one for each time step), we only have to specify a single section. When running the message passing algorithm we will explictly use the posterior of the previous timestep as prior in the next one. Let's build a section of the factor graph:

```julia
fg = FactorGraph()
z_prev_m = Variable(id=:z_prev_m); placeholder(z_prev_m, :z_prev_m, dims=(2,))
z_prev_v = Variable(id=:z_prev_v); placeholder(z_prev_v, :z_prev_v, dims=(2,2))
bu = Variable(id=:bu); placeholder(bu, :bu, dims=(2,))

@RV z_prev ~ GaussianMeanVariance(z_prev_m, z_prev_v, id=:z_prev) # p(z_prev)
@RV noise_z ~ GaussianMeanVariance(constant(zeros(2), id=:noise_z_m), constant(Σz, id=:noise_z_v)) # process noise
@RV z = constant(A, id=:A) * z_prev + bu + noise_z; z.id = :z # p(z|z_prev) (state transition model)
@RV x ~ GaussianMeanVariance(z, constant(Σx, id=:Σx)) # p(x|z) (observation model)
placeholder(x, :x, dims=(2,));
ForneyLab.draw(fg)
```

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

<svg width="614pt" height="746pt"
 viewBox="0.00 0.00 614.00 746.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 742)">
<title>G</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-742 610,-742 610,4 -4,4"/>

<g id="node1" class="node">
<title>16853265665615086174</title>
<polygon fill="none" stroke="black" points="244.5,-405 172.5,-405 172.5,-333 244.5,-333 244.5,-405"/>
<text text-anchor="middle" x="208.5" y="-371.8" font-family="Times,serif" font-size="9.00">+</text>
<text text-anchor="middle" x="208.5" y="-361.8" font-family="Times,serif" font-size="9.00">addition_1</text>
</g>

<g id="node8" class="node">
<title>392773488757477144</title>
<polygon fill="lightgrey" stroke="black" points="71,-279 0,-279 0,-225 71,-225 71,-279"/>
<text text-anchor="middle" x="35.5" y="-249.8" font-family="Times,serif" font-size="9.00">placeholder_bu</text>
</g>

<g id="edge8" class="edge">
<title>16853265665615086174--392773488757477144</title>
<path fill="none" stroke="black" d="M172.23,-343.89C142.51,-324.13 100.67,-296.32 71.13,-276.68"/>
<text text-anchor="start" x="127.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">bu</text>
<text text-anchor="start" x="71.13" y="-279.28" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="153.23" y="-346.49" font-family="Times,serif" font-size="8.00">3 in2 </text>
</g>

<g id="node12" class="node">
<title>8279433045920090609</title>
<polygon fill="none" stroke="black" points="260.5,-288 186.5,-288 186.5,-216 260.5,-216 260.5,-288"/>
<text text-anchor="middle" x="223.5" y="-254.8" font-family="Times,serif" font-size="9.00">×</text>
<text text-anchor="middle" x="223.5" y="-244.8" font-family="Times,serif" font-size="9.00">multiplication_1</text>
</g>

<g id="edge13" class="edge">
<title>16853265665615086174--8279433045920090609</title>
<path fill="none" stroke="black" d="M213.08,-332.87C214.93,-318.74 217.05,-302.46 218.89,-288.31"/>
<text text-anchor="start" x="216.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">variable_1</text>
<text text-anchor="start" x="199.89" y="-290.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="194.08" y="-326.47" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="node2" class="node">
<title>16971855759571518693</title>
<polygon fill="lightgrey" stroke="black" points="268,-54 171,-54 171,0 268,0 268,-54"/>
<text text-anchor="middle" x="219.5" y="-24.8" font-family="Times,serif" font-size="9.00">placeholder_z_prev_m</text>
</g>

<g id="node3" class="node">
<title>16152393752916881261</title>
<polygon fill="none" stroke="black" points="463.5,-405 359.5,-405 359.5,-333 463.5,-333 463.5,-405"/>
<text text-anchor="middle" x="411.5" y="-371.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="411.5" y="-361.8" font-family="Times,serif" font-size="9.00">gaussianmeanvariance_1</text>
</g>

<g id="node4" class="node">
<title>11889488514055484480</title>
<polygon fill="lightgrey" stroke="black" points="435.5,-279 381.5,-279 381.5,-225 435.5,-225 435.5,-279"/>
<text text-anchor="middle" x="408.5" y="-249.8" font-family="Times,serif" font-size="9.00">noise_z_v</text>
</g>

<g id="edge2" class="edge">
<title>16152393752916881261--11889488514055484480</title>
<path fill="none" stroke="black" d="M405.42,-332.73C404.62,-326.83 403.93,-320.75 403.5,-315 402.62,-303.26 403.39,-290.28 404.58,-279.24"/>
<text text-anchor="start" x="403.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z_v</text>
<text text-anchor="start" x="385.58" y="-281.84" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="392.42" y="-326.33" font-family="Times,serif" font-size="8.00">3 v </text>
</g>

<g id="node14" class="node">
<title>1120904904009419172</title>
<polygon fill="lightgrey" stroke="black" points="606,-279 551,-279 551,-225 606,-225 606,-279"/>
<text text-anchor="middle" x="578.5" y="-249.8" font-family="Times,serif" font-size="9.00">noise_z_m</text>
</g>

<g id="edge12" class="edge">
<title>16152393752916881261--1120904904009419172</title>
<path fill="none" stroke="black" d="M462.51,-332.87C491.48,-312.92 526.63,-288.72 550.59,-272.22"/>
<text text-anchor="start" x="500.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z_m</text>
<text text-anchor="start" x="531.59" y="-274.82" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="447.51" y="-326.47" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="node5" class="node">
<title>296474236434058327</title>
<polygon fill="none" stroke="black" points="450.5,-639 346.5,-639 346.5,-567 450.5,-567 450.5,-639"/>
<text text-anchor="middle" x="398.5" y="-605.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="398.5" y="-595.8" font-family="Times,serif" font-size="9.00">gaussianmeanvariance_2</text>
</g>

<g id="node10" class="node">
<title>5366754812594295938</title>
<polygon fill="none" stroke="black" points="345.5,-522 273.5,-522 273.5,-450 345.5,-450 345.5,-522"/>
<text text-anchor="middle" x="309.5" y="-488.8" font-family="Times,serif" font-size="9.00">+</text>
<text text-anchor="middle" x="309.5" y="-478.8" font-family="Times,serif" font-size="9.00">addition_2</text>
</g>

<g id="edge4" class="edge">
<title>296474236434058327--5366754812594295938</title>
<path fill="none" stroke="black" d="M357.97,-566.72C352.46,-561.06 347.12,-555.07 342.5,-549 336.29,-540.85 330.61,-531.38 325.77,-522.32"/>
<text text-anchor="start" x="342.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="red">z</text>
<text text-anchor="start" x="306.77" y="-524.92" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="342.97" y="-560.32" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="node11" class="node">
<title>16811625003687350813</title>
<polygon fill="lightgrey" stroke="black" points="514.5,-513 460.5,-513 460.5,-459 514.5,-459 514.5,-513"/>
<text text-anchor="middle" x="487.5" y="-483.8" font-family="Times,serif" font-size="9.00">Σx</text>
</g>

<g id="edge5" class="edge">
<title>296474236434058327--16811625003687350813</title>
<path fill="none" stroke="black" d="M425.69,-566.87C439.08,-549.57 454.95,-529.06 467.26,-513.15"/>
<text text-anchor="start" x="446.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="red">Σx</text>
<text text-anchor="start" x="448.26" y="-515.75" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="412.69" y="-560.47" font-family="Times,serif" font-size="8.00">3 v </text>
</g>

<g id="node6" class="node">
<title>927500160059107682</title>
<polygon fill="lightgrey" stroke="black" points="478,-54 383,-54 383,0 478,0 478,-54"/>
<text text-anchor="middle" x="430.5" y="-24.8" font-family="Times,serif" font-size="9.00">placeholder_z_prev_v</text>
</g>

<g id="node7" class="node">
<title>6196693582439278653</title>
<polygon fill="lightgrey" stroke="black" points="431.5,-738 365.5,-738 365.5,-684 431.5,-684 431.5,-738"/>
<text text-anchor="middle" x="398.5" y="-708.8" font-family="Times,serif" font-size="9.00">placeholder_x</text>
</g>

<g id="edge3" class="edge">
<title>6196693582439278653--296474236434058327</title>
<path fill="none" stroke="black" d="M398.5,-683.8C398.5,-670.31 398.5,-653.68 398.5,-639.1"/>
<text text-anchor="start" x="398.5" y="-659.6" font-family="Times,serif" font-size="8.00" fill="red">x</text>
<text text-anchor="start" x="379.5" y="-641.7" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="379.5" y="-677.4" font-family="Times,serif" font-size="8.00">1 out </text>
</g>

<g id="node9" class="node">
<title>12367512816635229008</title>
<polygon fill="lightgrey" stroke="black" points="174.5,-162 120.5,-162 120.5,-108 174.5,-108 174.5,-162"/>
<text text-anchor="middle" x="147.5" y="-132.8" font-family="Times,serif" font-size="9.00">A</text>
</g>

<g id="edge6" class="edge">
<title>5366754812594295938--16853265665615086174</title>
<path fill="none" stroke="black" d="M273.21,-471.43C255.02,-462.82 234.2,-449.88 221.5,-432 215.96,-424.2 212.68,-414.54 210.77,-405.19"/>
<text text-anchor="start" x="221.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="red">variable_2</text>
<text text-anchor="start" x="191.77" y="-407.79" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="254.21" y="-474.03" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="edge10" class="edge">
<title>5366754812594295938--16152393752916881261</title>
<path fill="none" stroke="black" d="M340.66,-449.87C353.2,-435.74 367.63,-419.46 380.18,-405.31"/>
<text text-anchor="start" x="364.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z</text>
<text text-anchor="start" x="361.18" y="-407.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="321.66" y="-443.47" font-family="Times,serif" font-size="8.00">3 in2 </text>
</g>

<g id="edge7" class="edge">
<title>8279433045920090609--12367512816635229008</title>
<path fill="none" stroke="black" d="M186.2,-233.02C172.53,-224.42 158.47,-212.74 150.5,-198 144.66,-187.19 143.42,-173.72 143.85,-162.09"/>
<text text-anchor="start" x="150.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="red">A</text>
<text text-anchor="start" x="124.85" y="-164.69" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="173.2" y="-235.62" font-family="Times,serif" font-size="8.00">3 a </text>
</g>

<g id="node13" class="node">
<title>2946848904195685870</title>
<polygon fill="none" stroke="black" points="361.5,-171 289.5,-171 289.5,-99 361.5,-99 361.5,-171"/>
<text text-anchor="middle" x="325.5" y="-137.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="325.5" y="-127.8" font-family="Times,serif" font-size="9.00">z_prev</text>
</g>

<g id="edge1" class="edge">
<title>8279433045920090609--2946848904195685870</title>
<path fill="none" stroke="black" d="M254.66,-215.87C267.2,-201.74 281.63,-185.46 294.18,-171.31"/>
<text text-anchor="start" x="278.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev</text>
<text text-anchor="start" x="275.18" y="-173.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="235.66" y="-209.47" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="edge9" class="edge">
<title>2946848904195685870--16971855759571518693</title>
<path fill="none" stroke="black" d="M290.5,-99C275.9,-84.4 259.25,-67.75 245.73,-54.23"/>
<text text-anchor="start" x="272.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev_m</text>
<text text-anchor="start" x="226.73" y="-56.83" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="275.5" y="-92.6" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="edge11" class="edge">
<title>2946848904195685870--927500160059107682</title>
<path fill="none" stroke="black" d="M360.17,-99C374.63,-84.4 391.13,-67.75 404.51,-54.23"/>
<text text-anchor="start" x="384.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev_v</text>
<text text-anchor="start" x="385.51" y="-56.83" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="347.17" y="-92.6" font-family="Times,serif" font-size="8.00">3 v </text>
</g>
</g>
</svg>

Now that we've built the factor graph, we can perform Kalman filtering by inserting measurement data into the factor graph and performing message passing.

```julia
include("scripts/cart_tracking_helpers.jl")
algo = messagePassingAlgorithm(z)
source_code = algorithmSourceCode(algo)
eval(Meta.parse(source_code))
marginals = Dict()
messages = Array{Message}(undef,6)
z_prev_m_0 = noisy_x[1]
z_prev_v_0 = A * (1e8*Diagonal(I,2) * A') + Σz
for t=2:n
    data = Dict(:x => noisy_x[t], :bu => b*u[t],:z_prev_m => z_prev_m_0, :z_prev_v => z_prev_v_0)
    step!(data, marginals, messages) # perform msg passing (single timestep)
    # Posterior of z becomes prior of z in the next timestep:
    z_prev_m_0 = ForneyLab.unsafeMean(marginals[:z])
    z_prev_v_0 = ForneyLab.unsafeCov(marginals[:z])
end
# Collect prediction p(z[n]|z[n-1]), measurement p(z[n]|x[n]), corrected prediction p(z[n]|z[n-1],x[n])
prediction      = messages[5].dist # the message index is found by manual inspection of the schedule
measurement     = messages[6].dist
corr_prediction = convert(ProbabilityDistribution{Multivariate, GaussianMeanVariance}, marginals[:z])
println("Prediction: ",prediction)
println("Measurement: ",measurement)
println("Posterior: ", corr_prediction)

# Make a fancy plot of the prediction, noisy measurement, and corrected prediction after n timesteps
plotCartPrediction(prediction, measurement, corr_prediction);
```

- Note that both the analytical Kalman filtering solution and the message passing solution lead to the same results. The advantage of message passing-based inference with ForneyLab is that we did not need to derive any inference equations. ForneyLab took care of all that.

### Recap Dynamical Models

- Dynamical systems do not obey the sample-by-sample independence assumption, but still can be specified, and state and parameter estimation equations can be solved by similar tools as for static models.

- Two of the more famous and powerful models with latent states include the hidden Markov model (with discrete states) and the Linear Gaussian dynamical system (with continuous states).

- For the LGDS, the Kalman filter is a well-known recursive state estimation procedure. The Kalman filter can be derived through Bayesian update rules on Gaussian distributions.

- If anything changes in the model, e.g., the state noise is not Gaussian, then you have to re-derive the inference equations again from scratch and it may not lead to an analytically pleasing answer.

- $\Rightarrow$ Generally, we will want to automate the inference processes. As we discussed, message passing in a factor graph is a visually appealing method to automate inference processes. We showed how Kalman filtering emerged naturally by automated message passing.

##  OPTIONAL SLIDES

### Proof of Kalman filtering equations including evidence updating

- Now let's proof the Kalman filtering equations including evidence updating by probabilistic calculus:
$$\begin{align*}
\overbrace{p(z_t\,|\,x^t)}^{\text{posterior}} &= p(z_t\,|\,x_t,x^{t-1}) \\
 &= \frac{p(x_t,z_t\,|\,x^{t-1})}{p(x_t\,|\,x^{t-1})} \\
 &= \frac{p(x_t\,|\,z_t) \,p(z_t\,|\,x^{t-1})}{p(x_t\,|\,x^{t-1})} \\
 &= \frac{p(x_t\,|\,z_t) \, \sum_{z_{t-1}} p(z_t,z_{t-1}\,|\,x^{t-1}) }{p(x_t\,|\,x^{t-1})} \\
 &= \frac{p(x_t\,|\,z_t) \, \sum_{z_{t-1}} p(z_t\,|\,z_{t-1}) \, p(z_{t-1}\,|\,x^{t-1})}{p(x_t\,|\,x^{t-1})} \\
 &= \frac{\overbrace{\mathcal{N}(x_t|c z_t, \sigma_x^2)}^{\text{likelihood}} \, \sum_{z_{t-1}} \overbrace{\mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2)}^{\text{state transition}} \, \overbrace{\mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) }^{\text{prior}} }{\underbrace{p(x_t\,|\,x^{t-1})}_{\text{evidence}}}
\end{align*}$$

- The posterior  $p(z_t\,|\,x^t)$ is proportional to the numerator, which by making use of the renormalization equality
$$\begin{align}
\mathcal{N}(x\,|\,cz,\sigma^2) = \frac{1}{c}\cdot \mathcal{N}\left(z \,\big|\,\frac{x}{c},\left(\frac{\sigma}{c}\right)^2\right) \tag{renormalization}\,,
\end{align}$$
can be evaluated with Gaussian multiplication rules:

$$\begin{align*}
\mathcal{N}(x_t|c z_t, &\sigma_x^2) \, \sum_{z_{t-1}} \underbrace{\mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2)}_{\text{use renormalization}} \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) \\
&= \mathcal{N}(x_t|c z_t, \sigma_x^2) \, \sum_{z_{t-1}} \frac{1}{a}\underbrace{\mathcal{N}\left(z_{t-1}\bigm|  \frac{z_t}{a},\left(\frac{\sigma_z}{a}\right)^2 \right) \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2)}_{\text{use Gaussian multiplication formula SRG-6}}  \\
&= \frac{1}{a} \mathcal{N}(x_t|c z_t, \sigma_x^2) \, \sum_{z_{t-1}} \underbrace{\mathcal{N}\left(\mu_{t-1}\bigm|  \frac{z_t}{a},\left(\frac{\sigma_z}{a}\right)^2 + \sigma_{t-1}^2 \right)}_{\text{not a function of }z_{t-1}} \underbrace{\mathcal{N}(z_{t-1} \,|\, \cdot, \cdot)}_{\text{sums to }1} \\
&= \frac{1}{a} \underbrace{\mathcal{N}(x_t|c z_t, \sigma_x^2)}_{\text{use renormalization rule}} \, \underbrace{\mathcal{N}\left(\mu_{t-1}\bigm|  \frac{z_t}{a},\left(\frac{\sigma_z}{a}\right)^2 + \sigma_{t-1}^2 \right)}_{\text{use renormalization rule}} \\
&= \frac{1}{c} \underbrace{\mathcal{N}\left(z_t \bigm|  \frac{x_t}{c}, \left( \frac{\sigma_x}{c}\right)^2 \right) \mathcal{N}\left(z_t\, \bigm|\,a \mu_{t-1},\sigma_z^2 + \left(a \sigma_{t-1}\right)^2 \right)}_{\text{use SRG-6 again}} \\
&= \underbrace{\frac{1}{c} \mathcal{N}\left( \frac{x_t}{c} \bigm| a \mu_{t-1}, \left( \frac{\sigma_x}{c}\right)^2+ \sigma_z^2 + \left(a \sigma_{t-1}\right)^2\right)}_{\text{use renormalization}} \, \mathcal{N}\left( z_t \,|\, \mu_t, \sigma_t^2\right)\\
  &= \underbrace{\mathcal{N}\left(x_t \,|\, ca \mu_{t-1}, \sigma_x^2 + c^2(\sigma_z^2+a^2\sigma_{t-1}^2) \right)}_{\text{evidence } p(x_t|x^{t-1})} \underbrace{\mathcal{N}\left( z_t \,|\, \mu_t, \sigma_t^2\right)}_{\text{posterior }p(z_t|x^t) }
\end{align*}$$
with
$$\begin{align*}
  \rho_t^2 &= a^2 \sigma_{t-1}^2 + \sigma_z^2 \tag{predicted variance}\\
  K_t &= \frac{c \rho_t^2}{c^2 \rho_t^2 + \sigma_x^2} \tag{Kalman gain} \\
  \mu_t &= \underbrace{a \mu_{t-1}}_{\text{prior prediction}} + K_t \cdot \underbrace{\left( x_t - c a \mu_{t-1}\right)}_{\text{prediction error}} \tag{posterior mean}\\
  \sigma_t^2 &= \left( 1 - c\cdot K_t \right) \rho_t^2 \tag{posterior variance}
\end{align*}$$

### Extensions of Generative Gaussian Models

- Using the methods of the previous lessons, it is possible to create your own new models based on stacking Gaussian and categorical distributions in new ways:

```julia
open("../../styles/aipstyle.html") do f display("text/html", read(f, String)) end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```