# Neuromatch Academy: Week 2, Day 3, Tutorial 3
# Linear Dynamical Systems & The Kalman Filter
__Content creators:__ Caroline Haimerl and Byron Galbraith

__Content reviewers:__ Jesse Livezey, Matt Krause, and Michael Waskom

**Useful reference:**
- Roweis, Ghahramani (1998): A unifying review of linear Gaussian Models
- Bishop (2006): Pattern Recognition and Machine Learning

**Acknowledgement**

This tutorial is in part based on code originally created by Caroline Haimerl for Dr. Cristina Savin's *Probabilistic Time Series* class at the Center for Data Science, New York University

```python
#@title Video 1: Introduction
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="6f_51L3i5aQ", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

---
# Tutorial Objectives

In the previous tutorials we looked at inferring discrete latent states that give rise to our measurements. In this tutorial, we will learn how to infer a latent model when our states are continuous. Particular attention is paid to the Kalman filter and it's mathematical foundation.

In this tutorial, you will:
* Review linear dynamical systems
* Learn about and implement the Kalman filter
* Explore how the Kalman filter can be used to smooth data from an eye-tracking experiment

```python
# Install PyKalman (https://pykalman.github.io/)
!pip install pykalman --quiet

# Imports
import numpy as np
import matplotlib.pyplot as plt
import pykalman
from scipy import stats
```

```python
#@title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Data retrieval and loading
import io
import os
import hashlib
import requests

fname = "W2D3_mit_eyetracking_2009.npz"
url = "https://osf.io/jfk8w/download"
expected_md5 = "20c7bc4a6f61f49450997e381cf5e0dd"

if not os.path.isfile(fname):
  try:
    r = requests.get(url)
  except requests.ConnectionError:
    print("!!! Failed to download data !!!")
  else:
    if r.status_code != requests.codes.ok:
      print("!!! Failed to download data !!!")
    elif hashlib.md5(r.content).hexdigest() != expected_md5:
      print("!!! Data download appears corrupted !!!")
    else:
      with open(fname, "wb") as fid:
        fid.write(r.content)

def load_eyetracking_data(data_fname=fname):

  with np.load(data_fname, allow_pickle=True) as dobj:
    data = dict(**dobj)

  images = [plt.imread(io.BytesIO(stim), format='JPG')
            for stim in data['stimuli']]
  subjects = data['subjects']

  return subjects, images
```

```python
#@title Helper functions
np.set_printoptions(precision=3)

def plot_kalman(state, observation, estimate=None, label='filter', color='r-',
                title='LDS', axes=None):
    if axes is None:
      fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(16, 6))
      ax1.plot(state[:, 0], state[:, 1], 'g-', label='true latent')
      ax1.plot(observation[:, 0], observation[:, 1], 'k.', label='data')
    else:
      ax1, ax2 = axes

    if estimate is not None:
      ax1.plot(estimate[:, 0], estimate[:, 1], color=color, label=label)
    ax1.set(title=title, xlabel='X position', ylabel='Y position')
    ax1.legend()

    if estimate is None:
      ax2.plot(state[:, 0], observation[:, 0], '.k', label='dim 1')
      ax2.plot(state[:, 1], observation[:, 1], '.', color='grey', label='dim 2')
      ax2.set(title='correlation', xlabel='latent', ylabel='observed')
    else:
      ax2.plot(state[:, 0], estimate[:, 0], '.', color=color,
               label='latent dim 1')
      ax2.plot(state[:, 1], estimate[:, 1], 'x', color=color,
               label='latent dim 2')
      ax2.set(title='correlation',
              xlabel='real latent',
              ylabel='estimated latent')
    ax2.legend()

    return ax1, ax2

def plot_gaze_data(data, img=None, ax=None):
    # overlay gaze on stimulus
    if ax is None:
        fig, ax = plt.subplots(figsize=(8, 6))

    xlim = None
    ylim = None
    if img is not None:
        ax.imshow(img, aspect='auto')
        ylim = (img.shape[0], 0)
        xlim = (0, img.shape[1])

    ax.scatter(data[:, 0], data[:, 1], c='m', s=100, alpha=0.7)
    ax.set(xlim=xlim, ylim=ylim)

    return ax

def plot_kf_state(kf, data, ax):
    mu_0 = np.ones(kf.n_dim_state)
    mu_0[:data.shape[1]] = data[0]
    kf.initial_state_mean = mu_0

    mu, sigma = kf.smooth(data)
    ax.plot(mu[:, 0], mu[:, 1], 'limegreen', linewidth=3, zorder=1)
    ax.scatter(mu[0, 0], mu[0, 1], c='orange', marker='>', s=200, zorder=2)
    ax.scatter(mu[-1, 0], mu[-1, 1], c='orange', marker='s', s=200, zorder=2)
```

---
# Section 1: Linear Dynamical System (LDS)

```python
#@title Video 2: Linear Dynamical Systems
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="2SWh639YgEg", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

Latent state variable: $$s_t = Fs_{t-1}+\zeta_t$$

Measured/observed variable: $$y_t = Hs_{t}+\eta_t$$

The latent state variable has dimension $D$ and the measured variable dimension $N$, dimensionality reduction here means that $D<N$.

Both latent and measured variable have Gaussian noise terms:

\begin{eqnarray}
\zeta_t & \sim & N(0, Q) \\
\eta_t & \sim & N(0, R) \\
s_0 & \sim & N(\mu_0, \Sigma_0)
\end{eqnarray}

As a consequence, $s_t$, $y_t$ and their joint distributions are Gaussian so we can easily compute the marginals and conditionals.

Just as in the HMM, the structure is that of a Markov chain where the state at time point $t$ is conditionally independent of previous states given the state at time point $t-1$.

## Section 1.1: Sampling

The first thing we will investigate is how to generate timecourse samples from a linear dynamical system given its parameters. We will start by defining the following system:

```python
# task dimensions
n_dim_state = 2
n_dim_obs = 2

# initialize model parameters
params = {
  'F': 0.5 * np.eye(n_dim_state),  # state transition matrix
  'Q': np.eye(n_dim_obs),  # state noise covariance
  'H': np.eye(n_dim_state),  # observation matrix
  'R': 0.1 * np.eye(n_dim_obs),  # observation noise covariance
  'mu_0': np.zeros(n_dim_state),  # initial state mean
  'sigma_0': 0.1 * np.eye(n_dim_state),  # initial state noise covariance
}
```

**Note**: We used a parameter dictionary `params` above. As the number of parameters we need to provide to our functions increases, it can be beneficial to condense them into a data structure like this to clean up the number of inputs we pass in. The trade-off is that we have to know what is in our data structure to use those values, rather than looking at the function signature directly.

### Exercise 1: Sampling from a linear dynamical system

In this exercise you will implement the dynamics functions of a linear dynamical system to sample both a latent space trajectory (given parameters set above) and noisy measurements.

```python
def sample_lds(n_timesteps, params, seed=0):
  """ Generate samples from a Linear Dynamical System specified by the provided
  parameters.

  Args:
  n_timesteps (int): the number of time steps to simulate
  params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)
  seed (int): a random seed to use for reproducibility checks

  Returns:
  ndarray, ndarray: the generated state and observation data
  """
  n_dim_state = params['F'].shape[0]
  n_dim_obs = params['H'].shape[0]

  # set seed
  np.random.seed(seed)

  # precompute random samples from the provided covariance matrices
  # mean defaults to 0
  zi = stats.multivariate_normal(cov=params['Q']).rvs(n_timesteps)
  eta = stats.multivariate_normal(cov=params['R']).rvs(n_timesteps)

  # initialize state and observation arrays
  state = np.zeros((n_timesteps, n_dim_state))
  obs = np.zeros((n_timesteps, n_dim_obs))

  ###################################################################
  ## TODO for students: compute the next state and observation values
  # Fill out function and remove
  raise NotImplementedError("Student excercise: compute the next state and observation values")
  ###################################################################

  # simulate the system
  for t in range(n_timesteps):
    # write the expressions for computing state values given the time step
    if t == 0:
      state[t] = ...
    else:
      state[t] = ...

    # write the expression for computing the observation
    obs[t] = ...

  return state, obs

# Uncomment below to test your function
# state, obs = sample_lds(100, params)
# print('sample at t=3 ', state[3])
# plot_kalman(state, obs, title='sample')
```

```python
# to_remove solution
def sample_lds(n_timesteps, params, seed=0):
  """ Generate samples from a Linear Dynamical System specified by the provided
  parameters.

  Args:
  n_timesteps (int): the number of time steps to simulate
  params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)
  seed (int): a random seed to use for reproducibility checks

  Returns:
  ndarray, ndarray: the generated state and observation data
  """
  n_dim_state = params['F'].shape[0]
  n_dim_obs = params['H'].shape[0]

  # set seed
  np.random.seed(seed)

  # precompute random samples from the provided covariance matrices
  # mean defaults to 0
  zi = stats.multivariate_normal(cov=params['Q']).rvs(n_timesteps)
  eta = stats.multivariate_normal(cov=params['R']).rvs(n_timesteps)

  # initialize state and observation arrays
  state = np.zeros((n_timesteps, n_dim_state))
  obs = np.zeros((n_timesteps, n_dim_obs))

  # simulate the system
  for t in range(n_timesteps):
    # write the expressions for computing state values given the time step
    if t == 0:
      state[t] = stats.multivariate_normal(mean=params['mu_0'],
                                           cov=params['sigma_0']).rvs(1)
    else:
      state[t] = params['F'] @ state[t-1] + zi[t]

    # write the expression for computing the observation
    obs[t] = params['H'] @ state[t] + eta[t]

  return state, obs

state, obs = sample_lds(100, params)
print('sample at t=3 ', state[3])
with plt.xkcd():
  plot_kalman(state, obs, title='sample')
```

### Interactive Demo: Adjusting System Dynamics
To test your understanding of the parameters of a linear dynamical system, think about what you would expect if you made the following changes:
1. Reduce observation noise $R$
2. Increase respective temporal dynamics $F$

Use the interactive widget below to vary the values of $R$ and $F$.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(R=widgets.FloatLogSlider(0.1, min=-3, max=1),
                  F=widgets.FloatSlider(0.5, min=0.0, max=1.0))
def explore_dynamics(R=0.1, F=0.5):
    params = {
    'F': F * np.eye(n_dim_state),  # state transition matrix
    'Q': np.eye(n_dim_obs),  # state noise covariance
    'H': np.eye(n_dim_state),  # observation matrix
    'R': R * np.eye(n_dim_obs),  # observation noise covariance
    'mu_0': np.zeros(n_dim_state),  # initial state mean,
    'sigma_0': 0.1 * np.eye(n_dim_state),  # initial state noise covariance
    }

    state, obs = sample_lds(100, params)
    plot_kalman(state, obs, title='sample')
```

---
# Section 2: Kalman Filtering

```python
#@title Video 3: Kalman Filtering
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="VboZOV9QMOI", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

We want to infer the latent state variable $s_t$ given the measured (observed) variable $y_t$.

$$P(s_t|y_1, ..., y_t, y_{t+1}, ..., y_T)\sim N(\hat{\mu_t}, \hat{\Sigma_t})$$

First we obtain estimates of the latent state by running the filtering from $n=0,....N$.

$$s_t^{pred}\sim N(\hat{\mu}_t^{pred},\hat{\Sigma}_t^{pred})$$

Where $\hat{\mu}_t^{pred}$ and $\hat{\Sigma}_t^{pred}$ are derived as follows:

\begin{eqnarray}
\hat{\mu}_1^{pred} & = & F\hat{\mu}_{0} \\
\hat{\mu}_t^{pred} & = & F\hat{\mu}_{t-1}
\end{eqnarray}

*this is the prediction for $s_t$ obtained simply by taking the expected value of $s_{t-1}$ and projecting it forward one step using the transition probability matrix $A$*

\begin{eqnarray}
\hat{\Sigma}_0^{pred} & = & F\hat{\Sigma}_{0}F^T+Q \\
\hat{\Sigma}_t^{pred} & = & F\hat{\Sigma}_{t-1}F^T+Q
\end{eqnarray}

*same for the covariance taking into account the noise covariance $Q$*

update from observation to obtain $\hat{\mu}_t^{filter}$ and $\hat{\Sigma}_t^{filter}$

project to observational space:
$$y_t^{pred}\sim N(H\hat{\mu}_t^{pred}, H\hat{\Sigma}_t^{pred}H^T+R)$$

update prediction by actual data:

\begin{eqnarray}
s_t^{filter} & \sim & N(\hat{\mu}_t^{filter}, \hat{\Sigma}_t^{filter}) \\
\hat{\mu}_t^{filter} & = & \hat{\mu}_t^{pred}+K_t(y_t-H\hat{\mu}_t^{pred}) \\
\hat{\Sigma}_t^{filter} & = & (I-K_tH)\hat{\Sigma}_t^{pred}
\end{eqnarray}

Kalman gain matrix:
$$K_t=\hat{\Sigma}_t^{pred}H^T(H\hat{\Sigma}_t^{pred}H^T+R)^{-1}$$

*we use the latent-only prediction to project it to the observational space and compute a correction proportional to the error $y_t-HFz_{t-1}$ between prediction and data, coefficient of this correction is the Kalman gain matrix*

*if measurement noise is small and dynamics are fast -> estimation will depend mostly on observed data*

In order to explore the impact of filtering, we will use the following noisy periodic system:

```python
# task dimensions
n_dim_state = 2
n_dim_obs = 2

# initialize model parameters
params = {
  'F': np.array([[1., 1.], [-(2*np.pi/20.)**2., .9]]),  # state transition matrix
  'Q': np.eye(n_dim_obs),  # state noise covariance
  'H': np.eye(n_dim_state),  # observation matrix
  'R': 1.0 * np.eye(n_dim_obs),  # observation noise covariance
  'mu_0': np.zeros(n_dim_state),  # initial state mean
  'sigma_0': 0.1 * np.eye(n_dim_state),  # initial state noise covariance
}

state, obs = sample_lds(100, params)
plot_kalman(state, obs, title='sample')
```

## Exercise 2: Implement Kalman filtering
In this exercise you will implement the Kalman filter (forward) process. Your focus will be on writing the expressions for the Kalman gain, filter mean, and filter covariance at each time step (refer to the equations above).

```python
def kalman_filter(data, params):
  """ Perform Kalman filtering (forward pass) on the data given the provided
  system parameters.

  Args:
    data (ndarray): a sequence of osbervations of shape(n_timesteps, n_dim_obs)
    params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)

  Returns:
    ndarray, ndarray: the filtered system means and noise covariance values
  """
  # pulled out of the params dict for convenience
  F = params['F']
  Q = params['Q']
  H = params['H']
  R = params['R']

  n_dim_state = F.shape[0]
  n_dim_obs = H.shape[0]
  I = np.eye(n_dim_state)  # identity matrix

  # state tracking arrays
  mu = np.zeros((len(data), n_dim_state))
  sigma = np.zeros((len(data), n_dim_state, n_dim_state))

  # filter the data
  for t, y in enumerate(data):
    if t == 0:
      mu_pred = params['mu_0']
      sigma_pred = params['sigma_0']
    else:
      mu_pred = F @ mu[t-1]
      sigma_pred = F @ sigma[t-1] @ F.T + Q

    ###########################################################################
    ## TODO for students: compute the filtered state mean and covariance values
    # Fill out function and remove
    raise NotImplementedError("Student excercise: compute the filtered state mean and covariance values")
    ###########################################################################
    # write the expression for computing the Kalman gain
    K = ...
    # write the expression for computing the filtered state mean
    mu[t] = ...
    # write the expression for computing the filtered state noise covariance
    sigma[t] = ...

  return mu, sigma

# Uncomment below to test your function
# filtered_state_means, filtered_state_covariances = kalman_filter(obs, params)
# plot_kalman(state, obs, filtered_state_means, title="my kf-filter",
#             color='r', label='my kf-filter')
```

```python
# to_remove solution
def kalman_filter(data, params):
  """ Perform Kalman filtering (forward pass) on the data given the provided
  system parameters.

  Args:
    data (ndarray): a sequence of osbervations of shape(n_timesteps, n_dim_obs)
    params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)

  Returns:
    ndarray, ndarray: the filtered system means and noise covariance values
  """
  # pulled out of the params dict for convenience
  F = params['F']
  Q = params['Q']
  H = params['H']
  R = params['R']

  n_dim_state = F.shape[0]
  n_dim_obs = H.shape[0]
  I = np.eye(n_dim_state)  # identity matrix

  # state tracking arrays
  mu = np.zeros((len(data), n_dim_state))
  sigma = np.zeros((len(data), n_dim_state, n_dim_state))

  # filter the data
  for t, y in enumerate(data):
    if t == 0:
      mu_pred = params['mu_0']
      sigma_pred = params['sigma_0']
    else:
      mu_pred = F @ mu[t-1]
      sigma_pred = F @ sigma[t-1] @ F.T + Q

    # write the expression for computing the Kalman gain
    K = sigma_pred @ H.T @ np.linalg.inv(H @ sigma_pred @ H.T + R)
    # write the expression for computing the filtered state mean
    mu[t] = mu_pred + K @ (y - H @ mu_pred)
    # write the expression for computing the filtered state noise covariance
    sigma[t] = (I - K @ H) @ sigma_pred

  return mu, sigma

filtered_state_means, filtered_state_covariances = kalman_filter(obs, params)
with plt.xkcd():
  plot_kalman(state, obs, filtered_state_means, title="my kf-filter",
              color='r', label='my kf-filter')
```

---
# Section 3: Fitting Eye Gaze Data

```python
#@title Video 4: Fitting Eye Gaze Data
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="M7OuXmVWHGI", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

Tracking eye gaze is used in both experimental and user interface applications. Getting an accurate estimation of where someone is looking on a screen in pixel coordinates can be challenging, however, due to the various sources of noise inherent in obtaining these measurements. A main source of noise is the general accuracy of the eye tracker device itself and how well it maintains calibration over time. Changes in ambient light or subject position can further reduce accuracy of the sensor. Eye blinks introduce a different form of noise as interruptions in the data stream which also need to be addressed.

Fortunately we have a candidate solution for handling noisy eye gaze data in the Kalman filter we just learned about. Let's look at how we can apply these methods to a small subset of data taken from the [MIT Eyetracking Database](http://people.csail.mit.edu/tjudd/WherePeopleLook/index.html) [[Judd et al. 2009](http://people.csail.mit.edu/tjudd/WherePeopleLook/Docs/wherepeoplelook.pdf)]. This data was collected as part of an effort to model [visual saliency](http://www.scholarpedia.org/article/Visual_salience) -- given an image, can we predict where a person is most likely going to look.

```python
# load eyetracking data
subjects, images = load_eyetracking_data()
```

## Interactive Demo: Tracking Eye Gaze

We have three stimulus images and five different subjects' gaze data. Each subject fixated in the center of the screen before the image appeared, then had a few seconds to freely look around. You can use the widget below to see how different subjects visually scanned the presented image. A subject ID of -1 will show the stimulus images without any overlayed gaze trace.

Note that the images are rescaled below for display purposes, they were in their original aspect ratio during the task itself.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(subject_id=widgets.IntSlider(-1, min=-1, max=4),
                  image_id=widgets.IntSlider(0, min=0, max=2))
def plot_subject_trace(subject_id=-1, image_id=0):
  if subject_id == -1:
    subject = np.zeros((3, 0, 2))
  else:
    subject = subjects[subject_id]
  data = subject[image_id]
  img = images[image_id]

  fig, ax = plt.subplots()
  ax.imshow(img, aspect='auto')
  ax.scatter(data[:, 0], data[:, 1], c='m', s=100, alpha=0.7)
  ax.set(xlim=(0, img.shape[1]), ylim=(img.shape[0], 0))
```

## Section 3.1: Fitting data with `pykalman`

Now that we have data, we'd like to use Kalman filtering to give us a better estimate of the true gaze. Up until this point we've known the parameters of our LDS, but here we need to estimate them from data directly. We will use the `pykalman` package to handle this estimation using the EM algorithm.

Before exploring fitting models with `pykalman` it's worth pointing out some naming conventions used by the library:

$$
\begin{align}
F &: \texttt{transition_matrices} &
Q &: \texttt{transition_covariance}\\
H &:\texttt{observation_matrices} &
R &:\texttt{observation_covariance}\\
\mu_0 &: \texttt{initial_state_mean} & \Sigma_0 &: \texttt{initial_state_covariance}
\end{align}
$$

The first thing we need to do is provide a guess at the dimensionality of the latent state. Let's start by assuming the dynamics line-up directly with the observation data (pixel x,y-coordinates), and so we have a state dimension of 2.

We also need to decide which parameters we want the EM algorithm to fit. In this case, we will let the EM algorithm discover the dynamics parameters i.e. the $F$, $Q$, $H$, and $R$ matrices.

We set up our `pykalman` `KalmanFilter` object with these settings using the code below.

```python
# set up our KalmanFilter object and tell it which parameters we want to
# estimate
np.random.seed(1)

n_dim_obs = 2
n_dim_state = 2

kf = pykalman.KalmanFilter(
  n_dim_state=n_dim_state,
  n_dim_obs=n_dim_obs,
  em_vars=['transition_matrices', 'transition_covariance',
           'observation_matrices', 'observation_covariance']
)
```

Because we know from the reported experimental design that subjects fixated in the center of the screen right before the image appears, we can set the initial starting state estimate $\mu_0$ as being the center pixel of the stimulus image (the first data point in this sample dataset) with a correspondingly low initial noise covariance $\Sigma_0$. Once we have everything set, it's time to fit some data.

```python
# Choose a subject and stimulus image
subject_id = 1
image_id = 2
data = subjects[subject_id][image_id]

# Provide the initial states
kf.initial_state_mean = data[0]
kf.initial_state_covariance = 0.1*np.eye(n_dim_state)

# Estimate the parameters from data using the EM algorithm
kf.em(data)

print(f'F =\n{kf.transition_matrices}')
print(f'Q =\n{kf.transition_covariance}')
print(f'H =\n{kf.observation_matrices}')
print(f'R =\n{kf.observation_covariance}')
```

We see that the EM algorithm has found fits for the various dynamics parameters. One thing you will note is that both the state and observation matrices are close to the identity matrix, which means the x- and y-coordinate dynamics are independent of each other and primarily impacted by the noise covariances.

We can now use this model to smooth the observed data from the subject. In addition to the source image, we can also see how this model will work with the gaze recorded by the same subject on the other images as well, or even with different subjects.

Below are the three stimulus images overlayed with recorded gaze in magenta and smoothed state from the filter in green, with gaze begin (orange triangle) and gaze end (orange square) markers.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(subject_id=widgets.IntSlider(1, min=0, max=4))
def plot_smoothed_traces(subject_id=0):
  subject = subjects[subject_id]
  fig, axes = plt.subplots(ncols=3, figsize=(18, 4))
  for data, img, ax in zip(subject, images, axes):
    ax = plot_gaze_data(data, img=img, ax=ax)
    plot_kf_state(kf, data, ax)
```

Why do you think one trace from one subject was sufficient to provide a decent fit across all subjects? If you were to go back and change the subject_id and/or image_id for when we fit the data using EM, do you think the fits would be different?

Finally, recall that the orignial task was to use this data to help devlop models of visual salience. While our Kalman filter is able to provide smooth estimates of observed gaze data, it's not telling us anything about *why* the gaze is going in a certain direction. In fact, if we sample data from our parameters and plot them, we get what amounts to a random walk.

```python
kf_state, kf_data = kf.sample(len(data))
ax = plot_gaze_data(kf_data, img=images[2])
plot_kf_state(kf, kf_data, ax)
```

This should not be surprising, as we have given the model no other observed data beyond the pixels at which gaze was detected. We expect there is some other aspect driving the latent state of where to look next other than just the previous fixation location.

In summary, while the Kalman filter is a good option for smoothing the gaze trajectory itself, especially if using a lower-quality eye tracker or in noisy environmental conditions, a linear dynamical system may not be the right way to approach the much more challenging task of modeling visual saliency.

# Bonus

## Review on Gaussian joint, marginal and conditional distributions

Assume

\begin{eqnarray}
z & = & [x^Ty^T]^T \\
z & = & \begin{bmatrix}x \\y\end{bmatrix}\sim N\left(\begin{bmatrix}a \\b\end{bmatrix}, \begin{bmatrix}A & C \\C^T & B\end{bmatrix}\right)
\end{eqnarray}

then the marginal distributions are

\begin{eqnarray}
x & \sim & N(a, A) \\
y & \sim & N(b,B)
\end{eqnarray}

and the conditional distributions are

\begin{eqnarray}
x|y & \sim & N(a+CB^{-1}(y-b), A-CB^{-1}C^T) \\
y|x & \sim & N(b+C^TA^{-1}(x-a), B-C^TA^{-1}C)
\end{eqnarray}

*important take away: given the joint Gaussian distribution we can derive the conditionals*

## Kalman Smoothing

```python
#@title Video 5: Kalman Smoothing and the EM Algorithm
# Insert the ID of the corresponding youtube video
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="4Ar2mYz1Nms", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

Obtain estimates by propagating from $y_T$ back to $y_0$ using results of forward pass ($\hat{\mu}_t^{filter}, \hat{\Sigma}_t^{filter}, P_t=\hat{\Sigma}_{t+1}^{pred}$)

\begin{eqnarray}
s_t & \sim & N(\hat{\mu}_t^{smooth}, \hat{\Sigma}_t^{smooth}) \\
\hat{\mu}_t^{smooth} & = & \hat{\mu}_t^{filter}+J_t(\hat{\mu}_{t+1}^{smooth}-F\hat{\mu}_t^{filter}) \\
\hat{\Sigma}_t^{smooth} & = & \hat{\Sigma}_t^{filter}+J_t(\hat{\Sigma}_{t+1}^{smooth}-P_t)J_t^T \\
J_t & = & \hat{\Sigma}_t^{filter}F^T P_t^{-1}
\end{eqnarray}

This gives us the final estimate for $z_t$.

\begin{eqnarray}
\hat{\mu}_t & = & \hat{\mu}_t^{smooth} \\
\hat{\Sigma}_t & = & \hat{\Sigma}_t^{smooth}
\end{eqnarray}

### Exercise 3: Implement Kalman smoothing

In this exercise you will implement the Kalman smoothing (backward) process. Again you will focus on writing the expressions for computing the smoothed mean, smoothed covariance, and $J_t$ values.

```python
def kalman_smooth(data, params):
  """ Perform Kalman smoothing (backward pass) on the data given the provided
  system parameters.

  Args:
    data (ndarray): a sequence of osbervations of shape(n_timesteps, n_dim_obs)
    params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)

  Returns:
    ndarray, ndarray: the smoothed system means and noise covariance values
  """
  # pulled out of the params dict for convenience
  F = params['F']
  Q = params['Q']
  H = params['H']
  R = params['R']

  n_dim_state = F.shape[0]
  n_dim_obs = H.shape[0]

  # first run the forward pass to get the filtered means and covariances
  mu, sigma = kalman_filter(data, params)

  # initialize state mean and covariance estimates
  mu_hat = np.zeros_like(mu)
  sigma_hat = np.zeros_like(sigma)
  mu_hat[-1] = mu[-1]
  sigma_hat[-1] = sigma[-1]

  # smooth the data
  for t in reversed(range(len(data)-1)):
    sigma_pred = F @ sigma[t] @ F.T + Q  # sigma_pred at t+1
    ###########################################################################
    ## TODO for students: compute the smoothed state mean and covariance values
    # Fill out function and remove
    raise NotImplementedError("Student excercise: compute the smoothed state mean and covariance values")
    ###########################################################################

    # write the expression to compute the Kalman gain for the backward process
    J = ...
    # write the expression to compute the smoothed state mean estimate
    mu_hat[t] = ...
    # write the expression to compute the smoothed state noise covariance estimate
    sigma_hat[t] = ...

  return mu_hat, sigma_hat

# Uncomment once the kalman_smooth function is complete
# smoothed_state_means, smoothed_state_covariances = kalman_smooth(obs, params)
# axes = plot_kalman(state, obs, filtered_state_means, color="r",
#                    label="my kf-filter")
# plot_kalman(state, obs, smoothed_state_means, color="b",
#             label="my kf-smoothed", axes=axes)
```

```python
# to_remove solution
def kalman_smooth(data, params):
  """ Perform Kalman smoothing (backward pass) on the data given the provided
  system parameters.

  Args:
    data (ndarray): a sequence of osbervations of shape(n_timesteps, n_dim_obs)
    params (dict): a dictionary of model paramters: (F, Q, H, R, mu_0, sigma_0)

  Returns:
    ndarray, ndarray: the smoothed system means and noise covariance values
  """
  # pulled out of the params dict for convenience
  F = params['F']
  Q = params['Q']
  H = params['H']
  R = params['R']

  n_dim_state = F.shape[0]
  n_dim_obs = H.shape[0]

  # first run the forward pass to get the filtered means and covariances
  mu, sigma = kalman_filter(data, params)

  # initialize state mean and covariance estimates
  mu_hat = np.zeros_like(mu)
  sigma_hat = np.zeros_like(sigma)
  mu_hat[-1] = mu[-1]
  sigma_hat[-1] = sigma[-1]

  # smooth the data
  for t in reversed(range(len(data)-1)):
    sigma_pred = F @ sigma[t] @ F.T + Q  # sigma_pred at t+1

    # write the expression to compute the Kalman gain for the backward process
    J = sigma[t] @ F.T @ np.linalg.inv(sigma_pred)
    # write the expression to compute the smoothed state mean estimate
    mu_hat[t] = mu[t] + J @ (mu_hat[t+1] - F @ mu[t])
    # write the expression to compute the smoothed state noise covariance estimate
    sigma_hat[t] = sigma[t] + J @ (sigma_hat[t+1] - sigma_pred) @ J.T

  return mu_hat, sigma_hat

smoothed_state_means, smoothed_state_covariances = kalman_smooth(obs, params)
with plt.xkcd():
  axes = plot_kalman(state, obs, filtered_state_means, color="r",
                     label="my kf-filter")
  plot_kalman(state, obs, smoothed_state_means, color="b",
              label="my kf-smoothed", axes=axes)
```

**Forward vs Backward**

Now that we have implementations for both, let's compare their peformance by computing the MSE between the filtered (forward) and smoothed (backward) estimated states and the true latent state.

```python
print(f"Filtered MSE: {np.mean((state - filtered_state_means)**2):.3f}")
print(f"Smoothed MSE: {np.mean((state - smoothed_state_means)**2):.3f}")
```

In this example, the smoothed estimate is clearly superior to the filtered one. This makes sense as the backward pass is able to use the forward pass estimates and correct them given all the data we've collected.

So why would you ever use Kalman filtering alone, without smoothing? As Kalman filtering only depends on already observed data (i.e. the past) it can be run in a streaming, or on-line, setting. Kalman smoothing relies on future data as it were, and as such can only be applied in a batch, or off-line, setting. So use Kalman filtering if you need real-time corrections and Kalman smoothing if you are considering already-collected data.

## The Expectation-Maximization (EM) Algorithm

- want to maximize $log p(y|\theta)$

- need to marginalize out latent state *(which is not tractable)*

$$p(y|\theta)=\int p(y,s|\theta)dz$$

- add a probability distribution $q(s)$ which will approximate the latent state distribution

$$log p(y|\theta)\int_s q(s)dz$$

- can be rewritten as

$$\mathcal{L}(q,\theta)+KL\left(q(s)||p(s|y),\theta\right)$$

- $\mathcal{L}(q,\theta)$ contains the joint distribution of $y$ and $s$

- $KL(q||p)$ contains the conditional distribution of $s|y$

#### Expectation step
- parameters are kept fixed
- find a good approximation $q(s)$: maximize lower bound $\mathcal{L}(q,\theta)$ with respect to $q(s)$
- (already implemented Kalman filter+smoother)

#### Maximization step
- keep distribution $q(s)$ fixed
- change parameters to maximize the lower bound $\mathcal{L}(q,\theta)$

As mentioned, we have already effectively solved for the E-Step with our Kalman filter and smoother. The M-step requires further derivation, which is covered in the Appendix. Rather than having you implement the M-Step yourselves, let's instead turn to using a library that has already implemented EM for exploring some experimental data from cognitive neuroscience.

### The M-step for a LDS
*(see Bishop, chapter 13.3.2 Learning in LDS)*
Update parameters of the probability distribution

*For the updates in the M-step we will need the following posterior marginals obtained from the Kalman smoothing results* $\hat{\mu}_t^{smooth}, \hat{\Sigma}_t^{smooth}$

$$
\begin{eqnarray}
E(s_t) &=& \hat{\mu}_t \\
E(s_ts_{t-1}^T) &=& J_{t-1}\hat{\Sigma}_t+\hat{\mu}_t\hat{\mu}_{t-1}^T\\
E(s_ts_{t}^T) &=& \hat{\Sigma}_t+\hat{\mu}_t\hat{\mu}_{t}^T
\end{eqnarray}
$$

**Update parameters**

Initial parameters
$$
\begin{eqnarray}
\mu_0^{new}&=& E(s_0)\\
Q_0^{new} &=& E(s_0s_0^T)-E(s_0)E(s_0^T) \\
\end{eqnarray}
$$

Hidden (latent) state parameters
$$
\begin{eqnarray}
F^{new} &=& \left(\sum_{t=2}^N E(s_ts_{t-1}^T)\right)\left(\sum_{t=2}^N E(s_{t-1}s_{t-1}^T)\right)^{-1} \\
Q^{new} &=& \frac{1}{T-1} \sum_{t=2}^N E\big(s_ts_t^T\big) - F^{new}E\big(s_{t-1}s_{t}^T\big) - E\big(s_ts_{t-1}^T\big)F^{new}+F^{new}E\big(s_{t-1}s_{t-1}^T\big)\big(F^{new}\big)^{T}\\
\end{eqnarray}
$$

Observable (measured) space parameters
$$H^{new}=\left(\sum_{t=1}^N y_t E(s_t^T)\right)\left(\sum_{t=1}^N E(s_t s_t^T)\right)^{-1}$$
$$R^{new}=\frac{1}{T}\sum_{t=1}^Ny_ty_t^T-H^{new}E(s_t)y_t^T-y_tE(s_t^T)H^{new}+H^{new}E(s_ts_t^T)H_{new}$$

## Handling Eye Blinks

In the MIT Eyetracking Database, raw tracking data includes times when the subject blinked. The way this is represented in the data stream is via negative pixel coordinate values.

We could try to mitigate these samples by simply deleting them from the stream, though this introduces other issues. For instance, if each sample corresponds to a fixed time step, and you arbitrarily remove some samples, the integrity of that consistent timestep between samples is lost. It's sometimes better to flag data as missing rather than to pretend it was never there at all, especially with time series data.

Another solution is to used masked arrays. In `numpy`, a [masked array](https://numpy.org/doc/stable/reference/maskedarray.generic.html#what-is-a-masked-array) is an `ndarray` with an additional embedded boolean masking array that indicates which elements should be masked. When computation is performed on the array, the masked elements are ignored. Both `matplotlib` and `pykalman` work with masked arrays, and, in fact, this is the approach taken with the data we explore in this notebook.

In preparing the dataset for this noteook, the original dataset was preprocessed to set all gaze data as masked arrays, with the mask enabled for any pixel with a negative x or y coordinate.


PyLMI-SDP
=========

*Symbolic linear matrix inequalities (LMI) and semi-definite programming (SDP) tools for Python*

This package includes a set of classes to represent and manipulate LMIs symbolically using [SymPy](http://sympy.org).
It also includes tools to export LMIs to [CVXOPT](http://abel.ee.ucla.edu/cvxopt/userguide/coneprog.html#semidefinite-programming) SDP input and to the [SDPA](http://sdpa.sourceforge.net/) format.

Depends on [SymPy](http://sympy.org) and [NumPy](http://www.numpy.org/); and optionally on [CVXOPT](http://cvxopt.org) and on [SciPy](http://www.scipy.org) (for sparse matrices).
Single codebase supporting both Python 2.7 and Python 3.x.
PyLMI-SDP is tested for various combinations of Python and sympy. See [here](https://github.com/cdsousa/PyLMI-SDP/actions/workflows/ci.yaml).

PyLMI-SDP is at [GitHub](https://github.com/cdsousa/PyLMI-SDP).

[](https://github.com/cdsousa/PyLMI-SDP/actions/workflows/ci.yaml)
[](https://coveralls.io/r/cdsousa/PyLMI-SDP?branch=master)

LMI Definition
--------------

### Examples

```python
from sympy import symbols, Matrix
from lmi_sdp import LMI_PD, LMI_NSD
variables = symbols('x y z')
x, y, z = variables
lmi = LMI_PD(Matrix([[x+1, y+2], [y+2, z+x]]))
lmi
```

$\displaystyle \left[\begin{matrix}x + 1 & y + 2\\y + 2 & x + z\end{matrix}\right] > 0$

```python
from lmi_sdp import init_lmi_latex_printing
from sympy import latex
init_lmi_latex_printing()
print(latex(lmi))
```

    \left[\begin{matrix}x + 1 & y + 2\\y + 2 & x + z\end{matrix}\right] \succ 0

```python
lmi.expanded(variables)
```

$\displaystyle \left[\begin{matrix}1.0 & 0\\0 & 1.0\end{matrix}\right] x + \left[\begin{matrix}0 & 1.0\\1.0 & 0\end{matrix}\right] y + \left[\begin{matrix}0 & 0\\0 & 1.0\end{matrix}\right] z + \left[\begin{matrix}1.0 & 2.0\\2.0 & 0\end{matrix}\right] \succ 0$

```python
lmi_2 = LMI_NSD( Matrix([[-x, -y], [-y, -z-x]]), Matrix([[1, 2], [2, 0]]))
lmi_2
```

$\displaystyle \left[\begin{matrix}- x & - y\\- y & - x - z\end{matrix}\right] \preceq \left[\begin{matrix}1 & 2\\2 & 0\end{matrix}\right]$

```python
lmi_2.canonical()
```

$\displaystyle \left[\begin{matrix}x + 1 & y + 2\\y + 2 & x + z\end{matrix}\right] \succeq 0$

Convertion to CVXOPT SDP
------------------------

### Example

(from CVXOPT [SDP example](http://abel.ee.ucla.edu/cvxopt/userguide/coneprog.html#semidefinite-programming))

```python
from sympy import symbols, Matrix
from lmi_sdp import LMI_NSD, init_lmi_latex_printing

init_lmi_latex_printing()

variables = symbols('x1 x2 x3')
x1, x2, x3 = variables

min_obj = x1 - x2 + x3

LMI_1 = LMI_NSD(
        x1*Matrix([[-7, -11], [-11, 3]]) +
        x2*Matrix([[7, -18], [-18, 8]]) +
        x3*Matrix([[-2, -8], [-8, 1]]),
        Matrix([[33, -9], [-9, 26]]))

LMI_2 = LMI_NSD(
        x1*Matrix([[-21, -11, 0], [-11, 10, 8], [0, 8, 5]]) +
        x2*Matrix([[0, 10, 16], [10, -10, -10], [16, -10, 3]]) +
        x3*Matrix([[-5, 2, -17], [2, -6, 8], [-17, 8, 6]]),
        Matrix([[14, 9, 40], [9, 91, 10], [40, 10, 15]]))

min_obj
```

$\displaystyle x_{1} - x_{2} + x_{3}$

```python
LMI_1.expanded(variables)
```

$\displaystyle \left[\begin{matrix}-7.0 & -11.0\\-11.0 & 3.0\end{matrix}\right] x_{1} + \left[\begin{matrix}7.0 & -18.0\\-18.0 & 8.0\end{matrix}\right] x_{2} + \left[\begin{matrix}-2.0 & -8.0\\-8.0 & 1.0\end{matrix}\right] x_{3} \preceq \left[\begin{matrix}33 & -9\\-9 & 26\end{matrix}\right]$

```python
LMI_2.expanded(variables)
```

$\displaystyle \left[\begin{matrix}-21.0 & -11.0 & 0\\-11.0 & 10.0 & 8.0\\0 & 8.0 & 5.0\end{matrix}\right] x_{1} + \left[\begin{matrix}0 & 10.0 & 16.0\\10.0 & -10.0 & -10.0\\16.0 & -10.0 & 3.0\end{matrix}\right] x_{2} + \left[\begin{matrix}-5.0 & 2.0 & -17.0\\2.0 & -6.0 & 8.0\\-17.0 & 8.0 & 6.0\end{matrix}\right] x_{3} \preceq \left[\begin{matrix}14 & 9 & 40\\9 & 91 & 10\\40 & 10 & 15\end{matrix}\right]$

```python
from cvxopt import solvers
from lmi_sdp import to_cvxopt

solvers.options['show_progress'] = False

c, Gs, hs = to_cvxopt(min_obj, [LMI_1, LMI_2], variables)

sol = solvers.sdp(c, Gs=Gs, hs=hs)
print(sol['x'])
```

    [-3.68e-01]
    [ 1.90e+00]
    [-8.88e-01]

Export to SDPA Format
---------------------

### Example

```python
from sympy import symbols, Matrix
from lmi_sdp import LMI_PSD, to_sdpa_sparse

variables = x1, x2 = symbols('x1 x2')

min_obj = 10*x1 + 20*x2
lmi_1 = LMI_PSD(
        -Matrix([[1, 0, 0, 0], [0, 2, 0, 0], [0, 0, 3, 0], [0, 0, 0, 4]]) +
        Matrix([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]])*x1 +
        Matrix([[0, 0, 0, 0], [0, 1, 0, 0], [0, 0, 5, 2], [0, 0, 2, 6]])*x2)
lmi_1
```

$\displaystyle \left[\begin{matrix}x_{1} - 1 & 0 & 0 & 0\\0 & x_{1} + x_{2} - 2 & 0 & 0\\0 & 0 & 5 x_{2} - 3 & 2 x_{2}\\0 & 0 & 2 x_{2} & 6 x_{2} - 4\end{matrix}\right] \succeq 0$

```python
dat = to_sdpa_sparse(min_obj, lmi_1, variables, comment='test sparse')
print(dat)
```

    "test sparse"
    2 = ndim
    3 = nblocks
    1 1 2 = blockstruct
    10.0, 20.0 = objcoeffs
    0 1 1 1 1.0
    0 2 1 1 2.0
    0 3 1 1 3.0
    0 3 2 2 4.0
    1 1 1 1 1.0
    1 2 1 1 1.0
    2 2 1 1 1.0
    2 3 1 1 5.0
    2 3 1 2 2.0
    2 3 2 2 6.0

Author
------

[Cristóvão Duarte Sousa](https://github.com/cdsousa)

Install
-------

From PyPi:

    pip install PyLMI-SDP

From git source:

    git clone https://github.com/cdsousa/PyLMI-SDP.git
    cd PyLMI-SDP
    python setup.py install

License
-------

Simplified BSD License. See [License File](LICENSE.txt)

```python

```


```python
import holoviews as hv
hv.extension('bokeh')
hv.opts.defaults(hv.opts.Curve(width=500),
                 hv.opts.Points(width=500),
                 hv.opts.Image(width=500, colorbar=True, cmap='Viridis'))
```

```python
import numpy as np
import scipy.signal
import scipy.linalg
```

# Estimadores adaptivos parte I

Hasta ahora hemos estudiando sistemas lineales donde:

- sus coeficientes quedan fijos luego del diseño y son constantes en el tiempo
- hacen supuestos sobre los estadísticos de la señal/ruido

¿Qué podemos hacer si

- no podemos hacer supuestos sobre los estadísticos?
- los estadísticos de la señal/ruido cambian en el tiempo?
- estamos en un escenario donde los datos llegan continuamente (data streaming)?

Cuando los estadísticos cambian en el tiempo decimos que la señal es **no estacionaria**.

En estos casos necesitamos un estimador de tipo **adaptivo**, es decir sistemas y filtros cuyos coeficientes se pueden **adaptar** a medida que llegan nuevos datos. Estos estimadores se diseñan de acuerdo a un método de optimización que es *online*

La siguiente figura muestra algunos ejemplos de aplicaciones de sistemas adaptivos

El método de optimización online que utilizaremos principalmente en este curso es el gradiente descendente estocástico. Revisemos a continuación los fundamentos.

## Gradiente descendente

Sea un vector de pesos $w$ de largo $L+1$ que guarda los coeficientes de un estimador

Sea ahora una función de costo que mapea el vector de pesos a un número real

$$
J(w): \mathbb{R}^{L+1} \to \mathbb{R}
$$

La función de costo debe ser tal que a menor $J$ menor sea el error del estimador

Para entrenar un estimador o filtro adaptivo se tienen los siguientes pasos conceptuales

1. Partimos de una solución inicial $w_0$
1. Modificamos iterativamente $w$ tal que $J(w_{t+1}) < J(w_t)$
1. Nos detenemos al cumplir un cierto criterio

Para modificar iterativamete y eficientemente los pesos utilizaremos la regla del **gradiente descendente** (GD)

$$
w_{t+1} = w_t - \mu \frac{dJ(w)}{dw},
$$

donde $\mu$ se conoce como tasa de aprendizaje o "paso"

- Imaginemos $J$ como una superficie de $L+1$ dimensiones
- En cada punto el gradiente negativo de $J$ nos indica hacia donde está el descenso más abrupto
- La tasa $\mu$ nos da el largo del salto entre $w_t$ y $w_{t+1}$

Observando la **expansión de Taylor de primer orden** de $J$ en $w_{t}$

$$
\begin{align}
J(w_{t+1}) &= J(w_t) + \frac{dJ(w_t)}{dw} (w_{t+1} - w_{t})  \nonumber \\
&= J(w_t) -\mu \left \| \frac{dJ(w_t)}{dw} \right \|^2 \leq J(w_t) \nonumber
\end{align}
$$

es decir que usando la regla GD con $\mu>0$ y asumiendo que $J$ es convexo entonces se cumple que $J$ siempre decrece monotónicamente.

La siguiente gráficas interactivas muestran una superficie de costo no convexa para un parámetro unidimensional. Cada punto representa una solución que parte desde una posición inicial distinta. Las flechas corresponden a la derivada multiplicada por la tasa de aprendizaje.

Estudie la evolución de las tres soluciones en cada caso. En primer lugar se utiliza $\mu=0.05$

```python
J = lambda w : (w-1)**2 + 0.2*np.sin(2*np.pi*w) #  Función de costo
gradJ = lambda w : 2*(w-1) + 0.2*2*np.pi*np.cos(2*np.pi*w) # Gradiente
mu = 0.05 # Tasa de aprendizaje
iteraciones = 15
wt = np.zeros(shape=(iteraciones, 3))
wt[0, :] = np.array([0.05, 0.4, 1.9]) # Solución inicial
w_plot = np.linspace(0, 2, num=100)

for k in range(1, iteraciones):
    wt[k, :] = wt[k-1, :] - mu*gradJ(wt[k-1, :])
```

```python
loss_surface = hv.Curve((w_plot, J(w_plot)), 'w', 'J')
hMap = hv.HoloMap(kdims='Iteración')
for k in range(iteraciones):
    dots = hv.Points((wt[k, :], J(wt[k, :]))).opts(size=10, color='k')
    mag = mu*gradJ(wt[k, :])
    angle = np.pi/2 - np.sign(-mag)*np.pi/2
    mag = np.abs(mag)
    arrows = hv.VectorField((wt[k, :], J(wt[k, :]), angle, mag)).opts(pivot='tail',
                                                                      magnitude=hv.dim('Magnitude'),
                                                                      rescale_lengths=False)
    hMap[k] =  dots * arrows

loss_surface * hMap
```

:::{warning}

Dependiendo de donde partimos la solución final es distinta. El gradiente descedente puede quedarse "atorado" en un mínimo local o en un punto silla

:::

Ahora observe como evolucionan las tres soluciones con $\mu=0.5$, es decir 10 veces más grande que el caso anterior

```python
J = lambda w : (w-1)**2 + 0.2*np.sin(2*np.pi*w) #  Función de costo
gradJ = lambda w : 2*(w-1) + 0.2*2*np.pi*np.cos(2*np.pi*w) # Gradiente
mu = 0.5 # Tasa de aprendizaje
iteraciones = 15
wt = np.zeros(shape=(iteraciones, 3))
wt[0, :] = np.array([0.05, 0.4, 1.9]) # Solución inicial
w_plot = np.linspace(0, 2, num=100)

for k in range(1, iteraciones):
    wt[k, :] = wt[k-1, :] - mu*gradJ(wt[k-1, :])
```

```python
loss_surface = hv.Curve((w_plot, J(w_plot)), 'w', 'J')
hMap = hv.HoloMap(kdims='Iteración')
for k in range(iteraciones):
    dots = hv.Points((wt[k, :], J(wt[k, :]))).opts(size=10, color='k')
    mag = mu*gradJ(wt[k, :])
    angle = np.pi/2 - np.sign(-mag)*np.pi/2
    mag = np.abs(mag)
    arrows = hv.VectorField((wt[k, :], J(wt[k, :]), angle, mag)).opts(pivot='tail',
                                                                      magnitude=hv.dim('Magnitude'),
                                                                      rescale_lengths=False)
    hMap[k] =  dots * arrows

loss_surface * hMap
```

:::{warning}

Si la tasa de aprendizaje es muy alta, los pasos son muy largos y podríamos no converger a un punto estacionario

:::

Los ejemplos anteriores nos han mostrado algunas de las limitaciones del algoritmo de gradiente descendente. Es importante tenerlas en cuenta cuando lo utilicemos en nuestras aplicaciones

## Gradiente descendente en el filtro de Wiener

Para el filtro de Wiener teníamos que

$$
J(h) = \sigma_d^2 - 2 \textbf{h}^T R_{ud} + \textbf{h}^T R_{uu} \textbf{h},
$$

por ende

$$
\frac{dJ(h)}{dh} = -2 R_{ud} + 2 R_{uu} \textbf{h}
$$

y finalmente

$$
\textbf{h}_{t+1} = \textbf{h}_{t} (I - 2 \mu R_{uu}) + 2\mu R_{ud}
$$

En este caso la condición para una convergencia estable es

$$
0 < \mu < \frac{1}{\lambda_{\text{max}}},
$$

donde $\lambda_{\text{max}}$ es valor propio más grande de $R_{uu}$

(La prueba de esto puede encontrarse en *Haykin, "Adaptive filter theory", Sección 4.3*)

## Gradiente descendente estocástico (SGD)

El filtro de Wiener es óptimo pero no adaptivo:

- Requiere de $N$ muestras de $u$ y $d$ para estimar $R_{ud}$ y $R_{uu}$
- Los pesos se adaptan luego de haber presentado las $N$ muestras: Es una estrategia de tipo **batch**
- Asume que la señal es estacionaria

Si nuestros son no estacionarios significa que debemos adaptar el filtro a medida que nuevas muestras son observadas
. Para lograr esto podemos usar la versión estocástica del GD: SGD

En SGD:
- los pesos se adaptan luego de haber presentado una sola muestra o un conjunto pequeño de muestras (mini-batch)
- no hay garantía de llegar al óptimo en un problema convexo, pero es más eficiente computacionalmente que GD

El siguiente esquema muestra una comparación entre la trayectoria de $w$ cuando se usa GD (negro) y SGD (rojo). En general la trayectoria de SGD será más ruidosa y también podría requerir más pasos, pero cada paso es mucho más económico

## Algoritmo Least Mean Square (LMS)

Podemos extender el filtro de Wiener al caso no-estacionario usando SGD, el resultado es un algoritmo simple que además es robusto: **El algoritmo LMS**

- Fue fue inventado en 1960 por [Bernard Widrow](https://en.wikipedia.org/wiki/Bernard_Widrow) y Ted Hoff
- A diferencia del filtro de Wiener no se requiere conocimiento estadístico del proceso. Tampoco se requiere calcular e invertir la matriz de correlación
- El algoritmo LMS se ajusta o entrena de manera recursiva y online

Consideremos la función de costo **estocástica**  para la arquitectura FIR que utilizamos para el filtro de Wiener

$$
\begin{align}
J^s_n(\textbf{w}) &= e_n^2 \nonumber \\
&= (d_n - y_n)^2 \nonumber \\
&= (d_n - \textbf{w}^T \textbf{u}_n )^2 \nonumber \\
&= (d_n - \sum_{k=0}^{L} w_{n, k} u_{n-k} )^2 \nonumber
\end{align}
$$

donde definimos $\textbf{u}_n = [u_n, u_{n-1}, \ldots, u_{n-L}]$.

:::{note}

A diferencia del filtro de Wiener no aplicamos el valor esperado al error cuadrático. Se usa el error cuadrático instantaneo

:::

Para continuar calculamos el gradiente en función del peso $w_{n, k}$

$$
\frac{d J^s_n (\textbf{w})}{d w_{n, k}} = - 2 e_n u_{n-k}
$$

Luego, usando la regla SGD llegamos a

$$
w_{n+1, k} = w_{n, k} + 2 \mu e_n u_{n-k}, k=0, 1, \ldots, L
$$

y que en forma matricial es

$$
\begin{align}
\textbf{w}_{n+1} &= \textbf{w}_{n} + 2 \mu e_n \textbf{u}_{n}\nonumber \\
&= \textbf{w}_{n} + 2 \mu (d_n -  \textbf{w}_{n}^T \textbf{u}_{n}) \textbf{u}_{n}, \nonumber
\end{align}
$$

que se conoce como la regla de **Widrow-Hoff**

:::{important}

El algoritmo LMS estima el error instantaneo y actualiza los pesos recursivamente

:::

La complejidad de este algoritmo es $L+1$.

### Convergencia del algoritmo LMS (Haykin 6.5)

El algoritmo LMS tiende en la media al valor óptimo

$$
\mathbb{E}[\textbf{w}_n] \to \textbf{w}^*
$$

para $n\to \infty$

Además convergence en la media cuadrada: La varianza de $\textbf{w}_n - \textbf{w}^*$ tiene al valor mínimo de $J$ para $n\to \infty$

Esto se cumple si

$$
0 < \mu < \frac{2}{\text{Tr}[R_{uu}]}
$$

donde $R_{uu} = \mathbb{E}[\textbf{u}_n \textbf{u}_n^T ]$ es la matriz de autocorrelación y $\text{Tr}[]$ el operador que calcula la traza de una matriz

### Algoritmo Normalized LMS (NLMS)

Tenemos la siguiente regla iterativa

$$
\begin{align}
\textbf{w}_{n+1} &= \textbf{w}_{n} + 2 \mu (d_n -  \textbf{w}_{n}^T \textbf{u}_{n}) \textbf{u}_{n} \nonumber \\
& = \textbf{w}_{n} + \Delta \textbf{w}_n \nonumber
\end{align}
$$

que se puede interpretar graficamente como

(donde $\textbf{x}(k)$ y $\textbf{w}(k)$ corresponden a $\textbf{u}_n$ y $\textbf{w}_n$ en nuestra notación, respectivamente)

:::{note}

Los cambios en el vector de peso $\Delta \textbf{w}_n$ son paralelos a $\textbf{u}_{n}$. Además estos cambios podrían estar dominados por

$$
\max \textbf{u}_{n} = [u_n, u_{n-1}, \ldots, u_{n-L}]
$$

:::

El algoritmo **Normalized LMS** (NLMS) corrige este problema ponderando por la varianza de $\textbf{u}_{n}$

$$
\textbf{w}_{n+1} = \textbf{w}_{n} + 2 \mu (d_n -  \textbf{w}_{n}^T \textbf{u}_{n}) \frac{\textbf{u}_{n}}{\left(\|\textbf{u}_{n}\|^2 + \delta\right)}
$$

donde la constante $\delta$ es un valor pequeño que se usa para evitar divisiones por cero. En lo que sigue usaremos NLMS para revisar algunas aplicaciones

## Implementación del filtro NLMS en Python

Podemos implementar las ecuaciones del filtro NLMS como se muestra a continuación

```python
class Filtro_NLMS:

    def __init__(self, L, mu, delta=1e-6, winit=None):
        self.L = L
        self.w = np.zeros(shape=(L+1, ))
        self.mu = mu
        self.delta = delta

    def update(self, un, dn):
        # Asumiendo que un = [u[n], u[n-1], ..., u[n-L]]
        unorm = np.dot(un, un) + self.delta
        yn = np.dot(self.w, un)
        self.w += 2*self.mu*(dn - yn)*(un/unorm)
        return yn
```

- El filtro recibe como entrada el orden $L$ y la tasa de aprendizaje $\mu$
- Se asume un vector cero para los pesos iniciales, pero también en la práctica podemos partir de una solución anterior si esta existiera
- Para actualizar el vector de pesos es necesario entregar el vector $\textbf{u}_n \in \mathbb{R}^{L+1}$ y la salida deseada $d_n \in \mathbb{R}$. La función `update` retorna la salida predicha por el filtro $y_n = w_n^T \textbf{u}_n
$

A continuación probaremos este filtro con una aplicación conocida como **Adaptive line enhancement** (ALE). ALE se refiere a un sistema adaptivo para eliminar ruido blanco aditivo de una señal. El sistema aprende un filtro pasabanda en torno a la frecuencia de interés

En ALE usamos como señal deseada

$$
d_n = u_n = \textbf{u}_n[0]
$$

El valor predicho por el filtro será la señal $u$ pero libre de ruido blanco. Esto se debe a que el ruido blanco no tiene correlación y por ende el filtro adaptivo no lo puede predecir

```python
# Digamos que u = s + n
# El objetivo es limpiar u para obtener s
# s es una señal determínista y n es ruido blanco

Fs, f0 =  100, 5
t = np.arange(0, 4, 1/Fs)
s = np.sin(2.0*np.pi*t*f0)
n = 0.5*np.random.randn(len(t))
s[t>2.0] += 5  # Simulemos un cambio abrupto en la media de la señal
#s += s*(0.5 + 0.5*np.cos(2.0*np.pi*t/2))  # Tremolo (AM)
u = s + n
```

A diferencia de un filtro estático (como el filtro de Wiener) es posible filtrar incluso ante cambios bruscos en la señal.

Estudie como cambia el resultado del filtro con distintos valores de $\mu$

```python
L = 20
u_preds = {}

for mu in np.logspace(-2, 0, num=10):
    myfilter = Filtro_NLMS(L=L, mu=mu)
    u_preds[mu] = np.zeros(shape=(len(u),))
    for k in range(L+1, len(u)):
        u_preds[mu][k] = myfilter.update(u[k-L-1:k][::-1], u[k])
```

```python
hMap = hv.HoloMap(kdims='mu')
for mu, u_pred in u_preds.items():
    s1 = hv.Curve((t, s), 'Tiempo', 'Señal', label='Limpia')
    s2 = hv.Scatter((t, u), 'Tiempo', 'Señal', label='Contaminada')
    s3 = hv.Curve((t, u_pred), 'Tiempo', 'Señal', label='Filtrada')
    hMap[mu] = hv.Overlay([s1, s2, s3]).opts(hv.opts.Overlay(legend_position='top'),
                                             hv.opts.Curve(ylim=(-5, 10), height=350))
hMap
```

:::{important}

La tasa de aprendizaje $\mu$ controla la velocidad de adaptación. Pero una tasa demasiado grande provoca que el filtro sea inestable. En general el valor óptimo de $\mu$ depende del problema y del valor de $L$

:::

La siguiente figura muestra la respuesta en frecuencia del filtro en función del tiempo para $\mu=0.02$

Observe como a medida que se adapta el filtro se concentra en la frecuencia fundamental de la señal, que en este caso es 5 Hz

```python
L = 20
u_preds = {}
myfilter = Filtro_NLMS(L=L, mu=0.02)
H_history = np.zeros(shape=(512, len(u)))
for k in range(L+1, len(u)):
    myfilter.update(u[k-L-1:k][::-1], u[k])
    fk, Hk = scipy.signal.freqz(b=myfilter.w, a=1, fs=Fs)
    H_history[:, k] = np.abs(Hk)
```

```python
hv.Image((t, fk, H_history), kdims=['Tiempo [s]', 'Frecuencia [Hz]']).opts(cmap='Blues')
```

## Comparación entre Filtro de Wiener/GD y algoritmo LMS/SGD

- **Supuestos**:  Wiener requiere un ambiente estacionario lo cual nos permite calcular $R_{uu}$ y $R_{ud}$. En LMS la señal puede ser no estacionaria.
- **Aprendizaje:** En el filtro de Wiener el aprendizaje es determinista. En LMS el aprendizaje viene **promediando** a nivel de los estimadores de $w$. En LMS el aprendizaje es estadístico.
- **Optimalidad:** Wiener es óptimo en cambio LMS es sub-óptimo (localmente óptimo). LMS tiende a la solución de Wiener
- **Costo:** LMS se actualiza online y tiene costo $L$. Wiener se entrena offline y tiene costo $L^2$

A continuación se muestra un diagrama que compara el filtro de Wiener y el algoritmo LMS

```python

```


# +
# **First Notebook: Virtual machine test and assignment submission**
#### This notebook will test that the virtual machine (VM) is functioning properly and will show you how to submit an assignment to the autograder.  To move through the notebook just run each of the cells.  You will not need to solve any problems to complete this lab.  You can run a cell by pressing "shift-enter", which will compute the current cell and advance to the next cell, or by clicking in a cell and pressing "control-enter", which will compute the current cell and remain in that cell.  At the end of the notebook you will export / download the notebook and submit it to the autograder.
#### ** This notebook covers: **
#### *Part 1:* Test Spark functionality
#### *Part 2:* Check class testing library
#### *Part 3:* Check plotting
#### *Part 4:* Check MathJax formulas
#### *Part 5:* Export / download and submit

### ** Part 1: Test Spark functionality **

#### ** (1a) Parallelize, filter, and reduce **

```
# Check that Spark is working
largeRange = sc.parallelize(xrange(100000))
reduceTest = largeRange.reduce(lambda a, b: a + b)
filterReduceTest = largeRange.filter(lambda x: x % 7 == 0).sum()

print reduceTest
print filterReduceTest

# If the Spark jobs don't work properly these will raise an AssertionError
assert reduceTest == 4999950000
assert filterReduceTest == 714264285
```

#### ** (1b) Loading a text file **

```
# Check loading data with sc.textFile
import os.path
baseDir = os.path.join('data')
inputPath = os.path.join('cs100', 'lab1', 'shakespeare.txt')
fileName = os.path.join(baseDir, inputPath)

rawData = sc.textFile(fileName)
shakespeareCount = rawData.count()

print shakespeareCount

# If the text file didn't load properly an AssertionError will be raised
assert shakespeareCount == 122395
```

### ** Part 2: Check class testing library **

#### ** (2a) Compare with hash **

```
# TEST Compare with hash (2a)
# Check our testing library/package
# This should print '1 test passed.' on two lines
from test_helper import Test

twelve = 12
Test.assertEquals(twelve, 12, 'twelve should equal 12')
Test.assertEqualsHashed(twelve, '7b52009b64fd0a2a49e6d8a939753077792b0554',
                        'twelve, once hashed, should equal the hashed value of 12')
```

#### ** (2b) Compare lists **

```
# TEST Compare lists (2b)
# This should print '1 test passed.'
unsortedList = [(5, 'b'), (5, 'a'), (4, 'c'), (3, 'a')]
Test.assertEquals(sorted(unsortedList), [(3, 'a'), (4, 'c'), (5, 'a'), (5, 'b')],
                  'unsortedList does not sort properly')
```

### ** Part 3: Check plotting **

#### ** (3a) Our first plot **
#### After executing the code cell below, you should see a plot with 50 blue circles.  The circles should start at the bottom left and end at the top right.

```
# Check matplotlib plotting
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from math import log

# function for generating plot layout
def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999', gridWidth=1.0):
    plt.close()
    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')
    ax.axes.tick_params(labelcolor='#999999', labelsize='10')
    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:
        axis.set_ticks_position('none')
        axis.set_ticks(ticks)
        axis.label.set_color('#999999')
        if hideLabels: axis.set_ticklabels([])
    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')
    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])
    return fig, ax

# generate layout and plot data
x = range(1, 50)
y = [log(x1 ** 2) for x1 in x]
fig, ax = preparePlot(range(5, 60, 10), range(0, 12, 1))
plt.scatter(x, y, s=14**2, c='#d6ebf2', edgecolors='#8cbfd0', alpha=0.75)
ax.set_xlabel(r'$range(1, 50)$'), ax.set_ylabel(r'$\log_e(x^2)$')
pass
```

### ** Part 4: Check MathJax Formulas **

#### ** (4a) Gradient descent formula **
#### You should see a formula on the line below this one: $$ \scriptsize \mathbf{w}_{i+1} = \mathbf{w}_i - \alpha_i \sum_j (\mathbf{w}_i^\top\mathbf{x}_j  - y_j) \mathbf{x}_j \,.$$

#### This formula is included inline with the text and is $ \scriptsize (\mathbf{w}^\top \mathbf{x} - y) \mathbf{x} $.

#### ** (4b) Log loss formula **
#### This formula shows log loss for single point. Log loss is defined as: $$  \begin{align} \scriptsize \ell_{log}(p, y) = \begin{cases} -\log (p) & \text{if } y = 1 \\\ -\log(1-p) & \text{if } y = 0 \end{cases} \end{align} $$

### ** Part 5: Export / download and submit **

#### ** (5a) Time to submit **

#### You have completed the lab.  To submit the lab for grading you will need to download it from your IPython Notebook environment.  You can do this by clicking on "File", then hovering your mouse over "Download as", and then clicking on "Python (.py)".  This will export your IPython Notebook as a .py file to your computer.
#### To upload this file to the course autograder, go to the edX website and find the page for submitting this assignment.  Click "Choose file", then navigate to and click on the downloaded .py file.  Now click the "Open" button and then the "Check" button.  Your submission will be graded shortly and will be available on the page where you submitted.  Note that when submission volumes are high, it may take as long as an hour to receive results.


# Latex
## or should I say

### Prof. Robert Quimby
© 2019 Robert Quimby

## In this tutorial you will...

- Be introduced to document preparation with $\LaTeX$
- "Markup" a plain text document with $\LaTeX$ "tags"
- Create beautiful equations using $\LaTeX$'s math mode
- Start writing a research paper on Overleaf.com using the AASTex macros

## Introduction to $\LaTeX$

* $\LaTeX$ is a ["markup" language](https://en.wikipedia.org/wiki/Markup_language) (other markup languages you might know include [HTML](https://www.w3schools.com/html/) and [XML](https://en.wikipedia.org/wiki/XML))
* $\LaTeX$ can be used to "typeset" a document (define which words should be in $\textbf{bold}$, which should have a $\huge{larger}$ or $\tiny{smaller}$ font, etc.)
* After you write a document using $\LaTeX$ commands, you can send the file to a program that will "compile" the commands and produce a typeset, PDF document.
* Although there are many markup languages, $\LaTeX$ is especially good for producing mathematical equations. It is thus the standard tool for scientific writing.

## Overleaf.com documentation

https://www.overleaf.com/learn/latex/Main_Page

## $\LaTeX$ Tag Basics

$\LaTeX$ tags usually start with a backslash, `\`, followed by the tag name.

Examples:
- `\newline` -- ends the current line (of text) and starts a new one
- `\LaTeX` -- produces the fancy $\LaTeX$ name
- `\copyright` -- makes the copyright symbol

You can use curly brackets, `{}`, to mark text that applies to a tag.

Examples:
- `\textbf{be bold}` -- indicates the text, "be bold," should be written in a (**b**old **f**ace) font
- `\title{My amazing title}` -- sets the title of the document to "My amazing title"

You can also use `\begin` and `\end` to apply a tag to an extended section of text:

`\begin{some_command}
  [what ever text some_command should be applied to]
\end{some_command}`

### Comments
The percent symbol, `%`, is used for comments. Anything on a line after a `%` will be considered ignored by the compiler.

### Escape Character

The ["escape character"](https://en.wikipedia.org/wiki/Escape_character) in $\LaTeX$ is the backslash, `\`. So if you want a percent symbol to be interpreted as a percent symbol (and not the start of a comment), you would type `\%`.

For example, 100% would be written `100\%` in your text file (note the backslash will **not** appear in the output file).

## Create a $\LaTeX$ document on Overleaf.com

https://www.overleaf.com/

### Quotations

In $\LaTeX$ you **never** use the double quote symbol (the symbol you get when you hold the shift key and press the key to the left of the return key). Instead, if you want two quotes, you type the single quote character twice.

In fancy documents quotation marks placed at the start and end of the quotation should angle toward the text. That is the quotation should begin with quotes that slope down (left to right) toward the quotation and then slope up at the end of the quotation. To achieve this look, you should start your quotation with back quotes and then end with regular quotes.

For example, you should make quotes \`\`like this'' **and** "not like this".

## Math mode

To tell the interpreter that you want to use mathematical symbols, you need to switch to math mode.

You can enter math mode in-line with other text by enclosing the math commands in dollar signs, like this:

`$ax^2 + bx + c = 0$`

Note that markdown cells in Jupyter Notebooks can also accept some $\LaTeX$ commands. For example, the equation above can be displayed as $ax^2 + bx + c = 0$

Or you can set an equation on its own line (with an equation number!) like this:

`\begin{equation}
ax^2 + bx + c = 0
\end{equation}`

## Including Figures

#### I recommend the `graphicx` package

* you must tell $\LaTeX$ to load the package in the preamble (before the `\begin{document}` statement)
* the command to do this is:

\usepackage{graphicx}

* Then use the `\includegraphics` command inside the `figure` environment:

\begin{figure}[!h]
\begin{center}
  \includegraphics[width=1.0\textwidth]{some_filename.pdf}
\end{center}
\caption{Put text here to describe the figure.}\label{some_label}
\end{figure}

## Quick start guide

Check Overleaf.com's guides:
- [Learn LaTeX in 30 minutes](https://www.overleaf.com/learn/latex/Learn_LaTeX_in_30_minutes)
- [Creating a document in LaTeX](https://www.overleaf.com/learn/latex/Creating_a_document_in_LaTeX)
- [Cheat Sheet](https://www.overleaf.com/latex/templates/a-quick-guide-to-latex/fghqpfgnxggz#.W6p8LpNKjdQ) of $\LaTeX$ commands


```python
!pip install qiskit
from qiskit import QuantumCircuit, assemble, Aer
from qiskit.visualization import plot_histogram, plot_bloch_vector
```

    Collecting qiskit
      Downloading qiskit-0.36.1.tar.gz (13 kB)
    Collecting qiskit-terra==0.20.1
      Downloading qiskit_terra-0.20.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
         |████████████████████████████████| 6.5 MB 4.1 MB/s
    Collecting qiskit-aer==0.10.4
      Downloading qiskit_aer-0.10.4-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (18.0 MB)
         |████████████████████████████████| 18.0 MB 42.1 MB/s
    Collecting qiskit-ibmq-provider==0.19.1
      Downloading qiskit_ibmq_provider-0.19.1-py3-none-any.whl (240 kB)
         |████████████████████████████████| 240 kB 51.1 MB/s
    Collecting qiskit-ignis==0.7.0
      Downloading qiskit_ignis-0.7.0-py3-none-any.whl (200 kB)
         |████████████████████████████████| 200 kB 53.2 MB/s
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.4->qiskit) (1.21.6)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.4->qiskit) (1.4.1)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (1.24.3)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (2.8.2)
    Collecting requests-ntlm>=1.1.0
      Downloading requests_ntlm-1.1.0-py2.py3-none-any.whl (5.7 kB)
    Collecting websocket-client>=1.0.1
      Downloading websocket_client-1.3.2-py3-none-any.whl (54 kB)
         |████████████████████████████████| 54 kB 3.0 MB/s
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (2.23.0)
    Collecting websockets>=10.0
      Downloading websockets-10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (112 kB)
         |████████████████████████████████| 112 kB 65.1 MB/s
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Collecting retworkx>=0.8.0
      Downloading retworkx-0.11.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 44.0 MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (5.4.8)
    Collecting ply>=3.10
      Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.9 MB/s
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (1.7.1)
    Collecting python-constraint>=1.4
      Downloading python-constraint-1.4.0.tar.bz2 (18 kB)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (0.3.4)
    Collecting tweedledum<2.0,>=1.1
      Downloading tweedledum-1.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (943 kB)
         |████████████████████████████████| 943 kB 43.9 MB/s
    Collecting scipy>=1.0
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 1.2 MB/s
    Collecting stevedore>=3.0.0
      Downloading stevedore-3.5.0-py3-none-any.whl (49 kB)
         |████████████████████████████████| 49 kB 6.8 MB/s
    Collecting symengine>=0.9
      Downloading symengine-0.9.2-cp37-cp37m-manylinux2010_x86_64.whl (37.5 MB)
         |████████████████████████████████| 37.5 MB 1.2 MB/s
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.19.1->qiskit) (1.15.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (3.0.4)
    Collecting cryptography>=1.3
      Downloading cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
         |████████████████████████████████| 4.0 MB 47.8 MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading ntlm_auth-1.5.0-py2.py3-none-any.whl (29 kB)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (2.21)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (4.11.3)
    Collecting pbr!=2.1.0,>=2.0.0
      Downloading pbr-5.9.0-py2.py3-none-any.whl (112 kB)
         |████████████████████████████████| 112 kB 42.3 MB/s
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (4.2.0)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (3.8.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.20.1->qiskit) (1.2.1)
    Building wheels for collected packages: qiskit, python-constraint
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.36.1-py3-none-any.whl size=11887 sha256=d0a6d0dd82886993426ce16fc853c6124f1e0bc0ac45c7e14c5924b3222dbf98
      Stored in directory: /root/.cache/pip/wheels/16/d2/58/340c0b0c7701a58b596126e5eb31c347c8dbfda8b949a555bd
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24081 sha256=9cc971537cc592f661686ad5c526da971738d773363f1204006593758d4aad42
      Stored in directory: /root/.cache/pip/wheels/07/27/db/1222c80eb1e431f3d2199c12569cb1cac60f562a451fe30479
    Successfully built qiskit python-constraint
    Installing collected packages: pbr, tweedledum, symengine, stevedore, scipy, retworkx, python-constraint, ply, ntlm-auth, cryptography, websockets, websocket-client, requests-ntlm, qiskit-terra, qiskit-ignis, qiskit-ibmq-provider, qiskit-aer, qiskit
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed cryptography-37.0.2 ntlm-auth-1.5.0 pbr-5.9.0 ply-3.11 python-constraint-1.4.0 qiskit-0.36.1 qiskit-aer-0.10.4 qiskit-ibmq-provider-0.19.1 qiskit-ignis-0.7.0 qiskit-terra-0.20.1 requests-ntlm-1.1.0 retworkx-0.11.0 scipy-1.7.3 stevedore-3.5.0 symengine-0.9.2 tweedledum-1.1.1 websocket-client-1.3.2 websockets-10.3

```python
from qiskit.visualization import plot_bloch_multivector
#from matplotlib import style
import math
import numpy as np
```

```python
qasm_sim = Aer.get_backend('qasm_simulator')
##statevec_sim = Aer.get_backend('statevector_simulator')
```

```python
##setting the length of input string
n = 3
```

```python
constant_black_box = QuantumCircuit(n+1)
output = np.random.randint(2) ##generating random integer
if output == 1:
  constant_black_box.x(n)

constant_black_box.draw()

```

q_0:

q_1:

q_2:

q_3:

```python
balanced_black_box = QuantumCircuit(n+1)
balanced_string = "101"

for qubit in range(len(balanced_string)):
  if balanced_string[qubit] == '1':
    balanced_black_box.x(qubit)

balanced_black_box.barrier()

for qubits in range(n):
  balanced_black_box.cx(qubit,n)

balanced_black_box.barrier()

for qubits in range(len(balanced_string)):
  if balanced_string[qubit] == '1':
    balanced_black_box.x(qubit)

balanced_black_box.draw()
```

     ┌───┐ ░                 ░
q_0: ┤ X ├─░─────────────────░────────────────
     └───┘ ░                 ░
q_1: ──────░─────────────────░────────────────
     ┌───┐ ░                 ░ ┌───┐┌───┐┌───┐
q_2: ┤ X ├─░───■────■────■───░─┤ X ├┤ X ├┤ X ├
     └───┘ ░ ┌─┴─┐┌─┴─┐┌─┴─┐ ░ └───┘└───┘└───┘
q_3: ──────░─┤ X ├┤ X ├┤ X ├─░────────────────
           ░ └───┘└───┘└───┘ ░

```python
deu = QuantumCircuit((n+1),n)

for qubit in range(n):
  deu.h(qubit)

deu.x(n)
deu.h(n)
deu.draw()
```

     ┌───┐
q_0: ┤ H ├─────
     ├───┤
q_1: ┤ H ├─────
     ├───┤
q_2: ┤ H ├─────
     ├───┤┌───┐
q_3: ┤ X ├┤ H ├
     └───┘└───┘
c: 3/══════════

```python
deu += balanced_black_box
deu.draw()
```

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: The QuantumCircuit.__iadd__() method is being deprecated. Use the compose() (potentially with the inplace=True argument) and tensor() methods which are more flexible w.r.t circuit register compatibility.
      """Entry point for launching an IPython kernel.

     ┌───┐┌───┐ ░                 ░                 ░ ┌─┐                     »
q_0: ┤ H ├┤ X ├─░─────────────────░─────────────────░─┤M├─────────────────────»
     ├───┤└───┘ ░                 ░                 ░ └╥┘┌─┐                  »
q_1: ┤ H ├──────░─────────────────░─────────────────░──╫─┤M├──────────────────»
     ├───┤┌───┐ ░                 ░ ┌───┐┌───┐┌───┐ ░  ║ └╥┘┌─┐               »
q_2: ┤ H ├┤ X ├─░───■────■────■───░─┤ X ├┤ X ├┤ X ├─░──╫──╫─┤M├───────────────»
     ├───┤├───┤ ░ ┌─┴─┐┌─┴─┐┌─┴─┐ ░ ├───┤├───┤├───┤ ░  ║  ║ └╥┘┌───┐┌───┐┌───┐»
q_3: ┤ X ├┤ H ├─░─┤ X ├┤ X ├┤ X ├─░─┤ H ├┤ H ├┤ H ├─░──╫──╫──╫─┤ H ├┤ H ├┤ H ├»
     └───┘└───┘ ░ └───┘└───┘└───┘ ░ └───┘└───┘└───┘ ░  ║  ║  ║ └───┘└───┘└───┘»
c: 3/══════════════════════════════════════════════════╩══╩══╩════════════════»
                                                       0  1  2                »
«      ░ ┌─┐┌───┐         ░                 ░
«q_0: ─░─┤M├┤ X ├─────────░─────────────────░────────────────
«      ░ └╥┘└┬─┬┘         ░                 ░
«q_1: ─░──╫──┤M├──────────░─────────────────░────────────────
«      ░  ║  └╥┘ ┌─┐┌───┐ ░                 ░ ┌───┐┌───┐┌───┐
«q_2: ─░──╫───╫──┤M├┤ X ├─░───■────■────■───░─┤ X ├┤ X ├┤ X ├
«      ░  ║   ║  └╥┘└───┘ ░ ┌─┴─┐┌─┴─┐┌─┴─┐ ░ └───┘└───┘└───┘
«q_3: ─░──╫───╫───╫───────░─┤ X ├┤ X ├┤ X ├─░────────────────
«      ░  ║   ║   ║       ░ └───┘└───┘└───┘ ░
«c: 3/════╩═══╩═══╩══════════════════════════════════════════
«         0   1   2

```python
## Adding constant black box in dj circuit

# deu += constant_black_box
# deu.draw()

```

```python
for qubit in range(n):
  deu.h(n)
deu.barrier()

for i in range(n):
  deu.measure(i,i)

deu.draw()
```

     ┌───┐┌───┐ ░                 ░                 ░ ┌─┐                     »
q_0: ┤ H ├┤ X ├─░─────────────────░─────────────────░─┤M├─────────────────────»
     ├───┤└───┘ ░                 ░                 ░ └╥┘┌─┐                  »
q_1: ┤ H ├──────░─────────────────░─────────────────░──╫─┤M├──────────────────»
     ├───┤┌───┐ ░                 ░ ┌───┐┌───┐┌───┐ ░  ║ └╥┘┌─┐               »
q_2: ┤ H ├┤ X ├─░───■────■────■───░─┤ X ├┤ X ├┤ X ├─░──╫──╫─┤M├───────────────»
     ├───┤├───┤ ░ ┌─┴─┐┌─┴─┐┌─┴─┐ ░ ├───┤├───┤├───┤ ░  ║  ║ └╥┘┌───┐┌───┐┌───┐»
q_3: ┤ X ├┤ H ├─░─┤ X ├┤ X ├┤ X ├─░─┤ H ├┤ H ├┤ H ├─░──╫──╫──╫─┤ H ├┤ H ├┤ H ├»
     └───┘└───┘ ░ └───┘└───┘└───┘ ░ └───┘└───┘└───┘ ░  ║  ║  ║ └───┘└───┘└───┘»
c: 3/══════════════════════════════════════════════════╩══╩══╩════════════════»
                                                       0  1  2                »
«      ░ ┌─┐┌───┐         ░                 ░                 ░ ┌─┐
«q_0: ─░─┤M├┤ X ├─────────░─────────────────░─────────────────░─┤M├──────
«      ░ └╥┘└┬─┬┘         ░                 ░                 ░ └╥┘┌─┐
«q_1: ─░──╫──┤M├──────────░─────────────────░─────────────────░──╫─┤M├───
«      ░  ║  └╥┘ ┌─┐┌───┐ ░                 ░ ┌───┐┌───┐┌───┐ ░  ║ └╥┘┌─┐
«q_2: ─░──╫───╫──┤M├┤ X ├─░───■────■────■───░─┤ X ├┤ X ├┤ X ├─░──╫──╫─┤M├
«      ░  ║   ║  └╥┘└───┘ ░ ┌─┴─┐┌─┴─┐┌─┴─┐ ░ ├───┤├───┤├───┤ ░  ║  ║ └╥┘
«q_3: ─░──╫───╫───╫───────░─┤ X ├┤ X ├┤ X ├─░─┤ H ├┤ H ├┤ H ├─░──╫──╫──╫─
«      ░  ║   ║   ║       ░ └───┘└───┘└───┘ ░ └───┘└───┘└───┘ ░  ║  ║  ║
«c: 3/════╩═══╩═══╩══════════════════════════════════════════════╩══╩══╩═
«         0   1   2                                              0  1  2

```python
from qiskit import execute
```

```python
shots = 1024
results = execute(deu,backend = qasm_sim,shots=shots).result()
answer = results.get_counts()
plot_histogram(answer)
```

```python

```