# Tutorial 1: Linear regression with MLE
**Module 4: How do we know how certain we should be?**

**Originally By Neuromatch Academy**

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil  with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Madineh Sarvestani, Patrick Mineault, Ella Batty, Michael Waskom

**Content Modifiers**: Konrad Kording, Ilenna Jones

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

```python
# @title Due Dates Calendar

from ipywidgets import widgets
from IPython.display import display, IFrame, YouTubeVideo

out1 = widgets.Output()
with out1:
  calendar = IFrame(src="https://calendar.google.com/calendar/embed?src=356b9d2nspjttvgbb3tvgk2f58%40group.calendar.google.com&ctz=America%2FNew_York", width=600, height=480)
  display(calendar)

out = widgets.Tab([out1])
out.set_title(0, 'Calendar')

display(out)
```

    Tab(children=(Output(),), _titles={'0': 'Calendar'})

---
# Tutorial Objectives

*Estimated timing of tutorial: 30 minutes*

This is Tutorial 2 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will use a different approach to fit linear models that incorporates the random 'noise' in our data.
- Learn about probability distributions and probabilistic models
- Learn how to calculate the likelihood of our model parameters
- Learn how to implement the maximum likelihood estimator, to find the model parameter with the maximum likelihood

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/2mkq4/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Plotting Functions
def plot_density_image(x, y, theta, sigma=1, ax=None):
  """ Plots probability distribution of y given x, theta, and sigma

  Args:

    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta (float): Slope parameter
    sigma (float): standard deviation of Gaussian noise

  """

  # plot the probability density of p(y|x,theta)
  if ax is None:
    fig, ax = plt.subplots()

  xmin, xmax = np.floor(np.min(x)), np.ceil(np.max(x))
  ymin, ymax = np.floor(np.min(y)), np.ceil(np.max(y))
  xx = np.linspace(xmin, xmax, 50)
  yy = np.linspace(ymin, ymax, 50)

  surface = np.zeros((len(yy), len(xx)))
  for i, x_i in enumerate(xx):
    surface[:, i] = stats.norm(theta * x_i, sigma).pdf(yy)

  ax.set(xlabel='x', ylabel='y')

  return ax.imshow(surface, origin='lower', aspect='auto', vmin=0, vmax=None,
            cmap=plt.get_cmap('Wistia'),
            extent=[xmin, xmax, ymin, ymax])
```

---
# Section 1: Maximum Likelihood Estimation (MLE)

```python
# @title Video 1: Maximum Likelihood Estimation
from ipywidgets import widgets

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="liTs_2PhiTU", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1])
out.set_title(0, 'Youtube')

display(out)
```

    Tab(children=(Output(),), _titles={'0': 'Youtube'})

This video covers Maximum Likelihood Estimation (MLE) in the context of a 1D linear regression.

## Section 1.1: Gaussian noise

 Click here for text recap of relevant part of video

In the previous tutorial we made the assumption that the data was drawn from a linear relationship with noise added, and found an effective approach for estimating model parameters based on minimizing the mean squared error.

In that case we treated the noise as simply a nuisance, but what if we factored it directly into our model?

Recall our linear model:

\begin{align}
y = \theta x + \epsilon.
\end{align}

The noise component $\epsilon$ is often modeled as a random variable drawn from a Gaussian distribution (also called the normal distribution).

The Gaussian distribution is described by its [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (pdf)
\begin{align}
\mathcal{N}(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{align}

and is dependent on two parameters: the mean $\mu$ and the variance $\sigma^2$. We often consider the noise signal to be Gaussian "white noise", with zero mean and unit variance:

\begin{align}
\epsilon \sim \mathcal{N}(0, 1).
\end{align}

### Interactive Demo 1.1: Gaussian Distribution Explorer

Use the explorer widget below to see how varying the $\mu$ and $\sigma$ parameters change the location and shape of the samples.

1.   What effect does varying $\mu$ have on the pdf?
2.   What effect does varying $\sigma$ have on the pdf?

```python
# @markdown Make sure you execute this cell to enable the widget!

@widgets.interact(mu=widgets.FloatSlider(0.0, min=-2.0, max=2.0),
                  sigma=widgets.FloatSlider(1.0, min=0.5, max=2.0))
def plot_normal_dist(mu=0, sigma=1):

  # Generate pdf & samples from normal distribution with mu/sigma
  rv = stats.norm(mu, sigma)
  x = np.linspace(-5, 5, 100)
  y = rv.pdf(x)
  samples = rv.rvs(1000)

  # Plot
  fig, ax = plt.subplots()
  ax.hist(samples, 20, density=True, color='g', histtype='stepfilled', alpha=0.8,
          label='histogram')
  ax.plot(x, y, color='orange', linewidth=3, label='pdf')
  ax.vlines(mu, 0, rv.pdf(mu), color='y', linewidth=3, label='$\mu$')
  ax.vlines([mu-sigma, mu+sigma], 0, rv.pdf([mu-sigma, mu+sigma]), colors='red',
            color='b', linewidth=3, label='$\sigma$')
  ax.set(xlabel='x', ylabel='probability density', xlim=[-5, 5], ylim=[0, 1.0])
  ax.legend()
```

```python
# to_remove explanation

"""
1. Mu controls the mean of the Gaussian distribution - it shifts it along the x-axis.
2. Sigma controls the width of the Gaussian distribution - small sigma results in a narrow
distribution, large sigma results in a very wide distribution
"""
```

## Section 1.2: Probabilistic Models

*Estimated timing to here from start of tutorial: 11 min*

Now that we have a model of our noise component $\epsilon$ as random variable, how do we incorporate this back into our original linear model from before? Consider again our simplified model $y = \theta x + \epsilon$ where the noise has zero mean and unit variance $\epsilon \sim \mathcal{N}(0, 1)$. We can now also treat $y$ as a random variable drawn from a Gaussian distribution where $\mu = \theta x$ and $\sigma^2 = 1$:

\begin{align}
y \sim \mathcal{N}(\theta x, 1)
\end{align}

which is to say that the probability of observing $y$ given $x$ and parameter $\theta$ is
\begin{align}
p(y|x,\theta) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(y-\theta x)^2}
\end{align}

Note that in this and the following sections, we will focus on a single data point (a single pairing of $x$ and $y$). We have dropped the subscript $i$ just for simplicity (that is, we use $x$ for a single data point, instead of $x_i$).

Now, let's revisit our original sample dataset where the true underlying model has $\theta = 1.2$.

```python
# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence

np.random.seed(121)
theta = 1.2
n_samples = 30
x = 10 * np.random.rand(n_samples) # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples) # sample from a standard normal distribution
y = theta * x + noise
```

This time we can plot the density of $p(y|x,\theta=1.2)$ and see how $p(y)$ changes for different values of $x$.

```python
# @markdown Execute this cell to visualize p(y|x, theta=1.2)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))

# Invokes helper function to generate density image plots from data and parameters
im = plot_density_image(x, y, 1.2, ax=ax1)
plt.colorbar(im, ax=ax1)
ax1.axvline(8, color='k')
ax1.set(title=r'p(y | x, $\theta$=1.2)')

# Plot pdf for given x
ylim = ax1.get_ylim()
yy = np.linspace(ylim[0], ylim[1], 50)
ax2.plot(yy, stats.norm(theta * 8, 1).pdf(yy), color='orange', linewidth=2)
ax2.set(
    title=r'p(y|x=8, $\theta$=1.2)',
    xlabel='y',
    ylabel='probability density');
```

## Section 1.3: Likelihood Estimation

*Estimated timing to here from start of tutorial: 15 min*

Now that we have our probabilistic model, we turn back to our original challenge of finding a good estimate for $\theta$ that fits our data. Given the inherent uncertainty when dealing in probabilities, we talk about the [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) that some estimate $\hat{\theta}$ fits our data. The likelihood function $\mathcal{L}(\theta)$ is equal to the probability density function parameterized by that $\theta$:

\begin{align}
\mathcal{L}(\theta|x,y) = p(y|x,\theta) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(y-\theta x)^2}
\end{align}

### Coding Exercise 1.3: Likelihood Function

In this exercise you will implement the likelihood function $\mathcal{L}(\theta|x, y)$ for our linear model where $\sigma = 1$.

After implementing this function, we can produce probabilities that our estimate $\hat{\theta}$ generated the provided observations. We will try with one of the samples from our dataset.

TIP: Use `np.exp` and `np.sqrt` for the exponential and square root functions, respectively.

```python
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    ndarray: the likelihood values for the theta_hat estimate
  """
  sigma = 1
  ##############################################################################
  ## TODO for students: implement the likelihood function
  # Fill out function and remove
  raise NotImplementedError("Student exercise: implement the likelihood function")
  ##############################################################################

  # Compute Gaussian likelihood
  pdf = ...

  return pdf

print(likelihood(1.0, x[1], y[1]))
```

```python
# to_remove solution
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the likelihood value for the theta_hat estimate
  """
  sigma = 1

  # Compute Gaussian likelihood
  pdf = 1 / np.sqrt(2 * np.pi * sigma**2) * np.exp(-(y - theta_hat * x)**2 / (2 * sigma**2))

  return pdf

print(likelihood(1.0, x[1], y[1]))
```

We should see that $\mathcal{L}(\theta=1.0|x=2.1,y=3.7) \approx 0.11$. So far so good, but how does this tell us how this estimate is better than any others?

When dealing with a set of data points, as we are with our dataset, we are concerned with their joint probability -- the likelihood that all data points are explained by our parameterization. Since we have assumed that the noise affects each output independently, we can factorize the likelihood, and write:

\begin{align}
\mathcal{L}(\theta|\mathbf{x}, \mathbf{y}) = \prod_{i=1}^N \mathcal{L}(\theta|x_i,y_i),
\end{align}

where we have $N$ data points $\mathbf{x} = [x_1,...,x_N]$ and $\mathbf{y} = [y_1,...,y_N]$.

In practice, such a product can be numerically unstable. Indeed multiplying small values together can lead to [underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow), the situation in which the digital representation of floating point number reaches its limit. This problem can be circumvented by taking the logarithm of the likelihood because the logarithm transforms products into sums:

\begin{align}
\log\mathcal{L}(\theta|\mathbf{x}, \mathbf{y}) = \sum_{i=1}^N \log\mathcal{L}(\theta|x_i,y_i)
\end{align}

We can take the sum of the log of the output of our `likelihood` method applied to the full dataset to get a better idea of how different $\hat{\theta}$ compare. We can also plot the different distribution densities over our dataset and see how they line up qualitatively.

```python
# @markdown Execute this cell to visualize different distribution densities
theta_hats = [0.5, 1.0, 2.2]
fig, axes = plt.subplots(ncols=3, figsize=(16, 4))
for theta_hat, ax in zip(theta_hats, axes):
  ll = np.sum(np.log(likelihood(theta_hat, x, y)))  # log likelihood
  im = plot_density_image(x, y, theta_hat, ax=ax)
  ax.scatter(x, y)
  ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat}, log likelihood: {ll:.2f}')
plt.colorbar(im, ax=ax);
```

Using the log likelihood calculation, we see that $\mathcal{L}(\theta=1.0) > \mathcal{L}(\theta=0.5) > \mathcal{L}(\theta=2.2)$.

This is great: now we have a way to compare estimators based on likelihood. But like with the MSE approach, we want an analytic solution to find the best estimator. In this case, we want to find the estimator that maximizes the likelihood.

## Section 1.4: Finding the Maximum Likelihood Estimator

*Estimated timing to here from start of tutorial: 23 min*

 Click here for text recap of relevant part of video

We want to find the parameter value $\hat\theta$ that makes our data set most likely:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \mathcal{L}(\theta|X,Y)
\end{align}

We discussed how taking the logarithm of the likelihood helps with numerical stability, the good thing is that it does so without changing the parameter value that maximizes the likelihood. Indeed, the $\log()$ function is *monotonically increasing*, which means that it preserves the order of its inputs. So we have:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \sum_{i=1}^m \textrm{log} \mathcal{L}(\theta|x_i,y_i)
\end{align}

Now substituting our specific likelihood function and taking its logarithm, we get:
\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} [-\frac{N}{2} \operatorname{log} 2\pi\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^N (y_i-\theta x_i)^2].
\end{align}

Note that maximizing the log likelihood is the same as minimizing the negative log likelihood (in practice optimization routines are developed to solve minimization not maximization problems). Because of the convexity of this objective function, we can take the derivative of our negative log likelihhood, set it to 0, and solve - just like our solution to minimizing MSE.

\begin{align}
\frac{\partial\operatorname{log}\mathcal{L}(\theta|x,y)}{\partial\theta}=\frac{1}{\sigma^2}\sum_{i=1}^N(y_i-\theta x_i)x_i = 0
\end{align}

This looks remarkably like the equation we had to solve for the optimal MSE estimator, and, in fact, we arrive to the exact same solution!

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \hat{\theta}_{\textrm{MSE}} = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

```python
# Compute theta_hat_MLE
theta_hat_mle = (x @ y) / (x @ x)
```

```python
#@title
#@markdown Execute this cell to visualize density with theta_hat_mle

# Plot the resulting distribution density
fig, ax = plt.subplots()
ll = np.sum(np.log(likelihood(theta_hat_mle, x, y))) # log likelihood
im = plot_density_image(x, y, theta_hat_mle, ax=ax)
plt.colorbar(im, ax=ax);
ax.scatter(x, y)
ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat_mle:.2f}, log likelihood: {ll:.2f}');
```

---
# Summary

*Estimated timing of tutorial: 30 minutes*

Likelihood vs probability:

  - $\mathcal{L}(\theta|x, y) = p(y|x, \theta)$
  - $p(y|x, \theta)$ -> "probability of observing the response $y$ given parameter $\theta$ and input $x$"
  - $\mathcal{L}(\theta|x, y)$ -> "likelihood model that parameters $\theta$ produced response $y$ from input $x$"

Log-likelihood maximization:

  - We take the $\log$ of the likelihood function for computational convenience
  - The parameters $\theta$ that maximize $\log\mathcal{L}(\theta|x, y)$ are the model parameters that maximize the probability of observing the data.

**Key point**: the log-likelihood is a flexible cost function, and is often used to find model parameters that best fit the data.

---
# Notation

\begin{align}
x &\quad \text{input, independent variable}\\
y &\quad \text{response measurement, dependent variable}\\
\mathbf{x} &\quad \text{vector of input values}\\
\mathbf{y} &\quad \text{vector of measurements}\\
\epsilon &\quad \text{measurement error, noise contribution}\\
\epsilon \sim \mathcal{N}(\mu, \sigma^2) &\quad \text{random variable } \epsilon \text{ is distributed according to a Gaussian distribution, with mean } \mu \text{ and variance } \sigma^2\\
\mu &\quad \text{mean}\\
\sigma^2 &\quad \text{variance}\\
\sigma &\quad \text{standard deviation}\\
\theta &\quad \text{parameter}\\
\hat{\theta} &\quad \text{estimate of parameter}\\
\mathcal{L}(\theta|x, y) &\quad \text{likelihood of that parameter } \theta \text{ producing response } y \text{ from input } x \\
p(y|x, \theta) &\quad \text{probability of observing the response } y \text{ given input } x \text{ and parameter } \theta \\
\end{align}

---
# Bonus

We can also see $\mathrm{p}(\mathrm{y} | \mathrm{x}, \theta)$ as a function of $x$. This is the stimulus likelihood function, and it is useful in case we want to decode the input $x$ from observed responses $y$. This is what is relevant from the point of view of a neuron that does not have access to the outside world and tries to infer what's out there from the responses of other neurons!


# Experiment 4: Air Track II---Newton’s Second Law
### Objectives
- To learn to take good measurements
- To learn how to estimate and propagate errors
- To learn how to take measurements to verify a theory

### Equipment
- One Airtrack, Blower, and Cart
- One accessory kit containing: one pulley, one mass hanger, and masses for cart and hanger.
- Two (2) photogates
- One 30 cm ruler
- One vernier caliper
- One two-meter stick
- One scale

### Safety
- Be careful placing the carts on the track. Do not damage the track.
- Please place paper underneath the carts when they are resting on the track without the air turned on. **Do not slide the carts on the track without the air turned on.**

# Introduction
In 1686, Isaac Newton published his three laws of motion using information he gained from experiments performed by both Galileo Galilei and Johannes Kepler:
1. If there is no net force, there is no acceleration
2. If there is a net force, then *F = ma*
3. If object A places a force on object B, then object B places an equal but opposite force on object A.

Today you will be verifying Newton’s second law by using a known force to accelerate an airtrack glider, measuring that acceleration, and comparing it to the predicted acceleration.

# Theory
### Kinematics
First and foremost, you will need to compute the acceleration of the aitrack glider. The information you will have, however, is velocity computed by a photogate. The photogate measures the velocity of the glider as it passes through by timing how long the infrared LED is blocked by the object (sometimes called a “flag”) passing through and uses the definition of velocity:

\begin{equation}
v \equiv \frac{\Delta x}{\Delta t}
\tag{1}
\end{equation}

where $\Delta x$ is the length of the flag and $\Delta t$ is the time the LED was blocked.

To determine the acceleration of the glider, consider the definition of acceleration:
\begin{equation}
a_{x} \equiv \frac{\Delta v_{x}}{\Delta t}
\tag{2}
\end{equation}

To use this, you will need to know the velocity of the cart at two different times. Think carefully about how you might measure those times.

Keep in mind that you will need to determine the errors in your measurements and propagate those errors into your computed acceleration.

### Force
A proper prediction of the acceleration of the glider from Newton’s second law requires great care. Recall that the force of gravity on an object is given by $F = mg$ where $g \equiv 9.81 \pm 0.005 m/s^2$ so that $ \delta g = 0.005 m/s^2$. You should find that the acceleration of the system is some fraction of g involving the masses of both the glider and the hanging mass.

### Uncertainty Analysis
You will need to compute the uncertainty in both your predicted and your measured accelerations. The uncertainty in your predicted acceleration will depend on the uncertainty in “g” as well as the uncertainties in the masses. The uncertainty in your “measured” acceleration will depend on the uncertainties in your measured velocities and times.

The Capstone software can measure time to within $ \delta t = 0.1 ms$.

Your Lab Manual and the previous labs can guide you in estimating the uncertainties in your measurements and propagating those uncertainties into your computed accelerations. The uncertainty in the masses of the glider and hanging mass will be determined by the accuracy of the scale. For your velocities, it might be easier to use the standard deviation of a large number of measurements as your measurement uncertainty. The uncertainty in your acceleration, however, will have to be computed using the three equations in your Lab Manual.

In order to minimize random errors, it is extremely important that each of your measurements be performed several times.

# Experimental Procedure

### Setting up the Air Track
Ensure that the pulley is securely inserted into the top hole in the bracket at the far end of the track and that it spins freely. Make sure that the blower is functioning properly (you should be able to feel the air streaming out of the pin-holes). Make sure that the hanging mass will not strike the table as it falls.

You will need to level the airtrack so that the only force comes from the hanging mass. Look back to the second lab for instructions if you’ve forgotten how to do this.

### Setting up a Photogate
1. Turn on the PASCO 850 Interface and start the PASCO Capstone software.
2. Plug a photogate into a Digital input.
3. Click the “Hardware Setup” tab in the left “Tools” palette, left-click the jack on the diagram where you inserted the plug, and select “Photogate” from the drop-down menu.
4. You should see a tab labeled “Timer Setup.” Open that tab and set up a pre-configured timer:
    1. Select the photogate you just installed.
    2. You will be using this photogate with a single flag.
    3. The computer need only keep track of the speed through the gate.
    4. A text box requesting the length of the flag in meters will appear. Measure the flag as best you can and enter that information in the box.
    5. Give this sensor a name such as “Photogate 1” or similar.

Repeat steps 2—4e for the second photogate.

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from P201_Functions import *

# Flag length measurement (m) and its uncertainty
flag_len = 0.1
delta_x = 0.005

# Acceleration Due to Gravity
g = 9.81
g_un = 0.005
```

When you adjust the height of the photogate, make sure that it is triggered only by the flag. The string should not trigger the photogate. You can tell when the photogate is triggered by looking for the red LED to light when the infrared LED is blocked. Ensure that the red LED only lights when the proper portion of the cart passes through.

Something to think about when choosing the positions for your photogates: Where should the mass hanger be while the cart goes through a photogate?

### Data Collection
**To release the cart: Hold the cart in position and turn on the blower. The cart must be released when the blower is running at full speed.**

Capstone will display the photogate data any way you like. The best tool for displaying the data from a photogate, however, might be a table that plots the time that the photogate was triggered in one column and the velocity through that photogate in the other.

Your goal is to confirm or deny Newton’s Second Law: $F = ma$. Notice that, if you had data points for the force (tension), *F*, as a function of the acceleration, *a*, you could create a graph. If that graph had the right shape, you might say that you had confirmed Newton’s Second Law. How should you plot this graph so that the slope has meaning?

You will need to plot a few different points of tension versus acceleration for each cart mass---*at least five* points, but the more data you have, the better your graph will be. Therefore, pick a cart mass and measure its acceleration for each of five different hanging masses. Once you’ve done that, change the cart mass and repeat the experiment. Please make sure that your cart does not bounce back through the second photogate after striking the bumper at the end of the track.

Your final result should be a graph of the force on the glider vs. the acceleration of the glider taken at a number of different masses for the hanging mass and the glider mass.

**Remember: Time is short---taking data is your first priority.** However, if the data you are taking is poor, then you might as well not be taking any data at all. Always dedicate one team member to compute the results of your experiments (in this case, they should be computing your measured accelerations) to see if your data matches your theoretical prediction. If your data is *not* matching your predictions, you should stop *and* diagnose the problem (check both your experiment and your prediction) before continuing.

You must take data with at least two different cart masses. If there is time, you can create a third graph with a third cart mass if you like. But you might make a preliminary plot of the data you have and try and fill in any gaps in that data, first.

```python
# Raw Data
# Cart 1

#### Enter Raw Data Here!!!!!!!!!!!!!! ####

# Mass of the cart (kg) and its uncertainty
m_cart1 = 0.19025
delta_m_cart = 0.005

# Mass of hanging mass (kg) and its uncertainty
m_hang = 0.00683
delta_m_hang = 0.005

# Theoretical Force of tension (N)
theo_FT = 0.06468

# Predicted Acceleration (m/s^2) and its uncertainty
a_p = 0.340
delta_a_p = 0.2492

###########################################

# Reads the name of the csv file and gets the data
df = pd.read_csv("./ExampleFiles/Lab4 cart 1.csv")

# Calculates the change in velocity and time
Delta_v = df['v2 (m/s)'] - df['v1 (m/s)']
Delta_t = df['t2 (s)'] - df['t1 (s)']

# Calculates the uncertainty of the velocity based on Equation 1
uncertainty_v = np.sqrt(delta_x**2 + 0.1**2)

# Finds the uncertainty of acceleration and adds it to the dataframe
unc_am = df['a (m/s^2)']*np.sqrt((uncertainty_v/Delta_v)**2 + (0.1/Delta_t)**2)
df.insert(5, "𝛿am (m/s^2)", unc_am)

# Finds the average measured acceleration
av_am = sum(df['a (m/s^2)'])/len(df['a (m/s^2)'])

# prints out the cart data
print("Data for: ")
print("m_cart = %0.5f kg              𝛿m_cart = %0.3f" % (m_cart1, delta_m_cart))
print("m_hang = %0.5f kg              𝛿m_hang = %0.3f" % (m_hang, delta_m_hang))
print("Predicted a_p = %0.3f m/s^2      𝛿a_p = %0.4f" % (a_p, delta_a_p))
print("Theoretical F_T = %0.5f N" % (theo_FT))
print()
print('-'*70)

# Prints information about the file
#df.info()
print(df)
print()
# Print statements for uncertainty + average
print("𝛿v = %0.5f m/s" % (uncertainty_v))
print("Average am = %0.5f m/s^2" % (av_am))
print('-'*70)
print()

# Inserts force into dataframe
force1 = df['a (m/s^2)']*m_cart1
df.insert(6, "Force (N)", force1)

# Defines the x and y values
acceleration = df.filter(['a (m/s^2)'], axis=1).dropna()
force = df.filter(['Force (N)'], axis=1).dropna()

# Create a figure of reasonable size and resolution, white background, black edge color
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')

# Gets the data values for x and y
x_data = force.values.reshape(-1, 1)
y_data = acceleration.values.reshape(-1, 1)

xi = df['Force (N)'].to_numpy()
yi = df['a (m/s^2)'].to_numpy()

# Creates the base plot with titles
#plt.plot(x_data,y_data,'b.')
plt.errorbar(x_data,y_data,a_p,theo_FT,'b.',label='Raw Data')
plt.ylabel('Acceleration (m/s^2)')
plt.xlabel('Force (N)')
plt.title('Force on the Glider vs. the Acceleration of the Glider')

# Takes the x and y values to make a trendline
intercept, slope, dintercept, dslope = linear_fit_plot_errors(xi,yi,0.0277,0.0291)

# Adds the legend to the plot
plt.legend()

# Displays the plot
plt.show()

print()
```

***

```python
# Raw Data
# Cart 2

#### Enter Raw Data Here!!!!!!!!!!!!!! ####

# Mass of the cart (kg) and its uncertainty
m_cart2 = 0.19025
delta_m_cart = 0.005

# Mass of hanging mass (kg) and its uncertainty
m_hang = 0.01668
delta_m_hang = 0.005

# Theoretical Force of tension (N)
theo_FT = 0.08205

# Predicted Acceleration (m/s^2) and its uncertainty
a_p = 0.43125
delta_a_p = 0.2396

###########################################

# Reads the name of the csv file and gets the data
df = pd.read_csv("./ExampleFiles/Lab4 cart 2.csv")

# Calculates the change in velocity and time
Delta_v = df['v2 (m/s)'] - df['v1 (m/s)']
Delta_t = df['t2 (s)'] - df['t1 (s)']

# Calculates the uncertainty of the velocity based on Equation 1
uncertainty_v = np.sqrt(delta_x**2 + 0.1**2)

# Finds the uncertainty of acceleration and adds it to the dataframe
unc_am = df['a (m/s^2)']*np.sqrt((uncertainty_v/Delta_v)**2 + (0.1/Delta_t)**2)
df.insert(5, "𝛿am (m/s^2)", unc_am)

# Finds the average measured acceleration
av_am = sum(df['a (m/s^2)'])/len(df['a (m/s^2)'])

# prints out the cart data
print("Data for: ")
print("m_cart = %0.5f kg              𝛿m_cart = %0.3f" % (m_cart2, delta_m_cart))
print("m_hang = %0.5f kg              𝛿m_hang = %0.3f" % (m_hang, delta_m_hang))
print("Predicted a_p = %0.3f m/s^2      𝛿a_p = %0.4f" % (a_p, delta_a_p))
print("Theoretical F_T = %0.5f N" % (theo_FT))
print()
print('-'*70)

# Prints information about the file
#df.info()
print(df)
print()
# Print statements for uncertainty + average
print("𝛿v = %0.5f m/s" % (uncertainty_v))
print("Average am = %0.5f m/s^2" % (av_am))
print('-'*70)
print()

# Inserts force into dataframe
force1 = df['a (m/s^2)']*m_cart2
df.insert(6, "Force (N)", force1)

#av_f = sum(df['Force (N)'])/len(df['Force (N)'])
#print("Average f = %0.5f" % (av_f))

# Defines the x and y values
acceleration = df.filter(['a (m/s^2)'], axis=1).dropna()
force = df.filter(['Force (N)'], axis=1).dropna()

# Create a figure of reasonable size and resolution, white background, black edge color
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')

# Gets the data values for x and y
x_data = force.values.reshape(-1, 1)
y_data = acceleration.values.reshape(-1, 1)

xi = df['Force (N)'].to_numpy()
yi = df['a (m/s^2)'].to_numpy()

# Creates the base plot with titles
#plt.plot(x_data,y_data,'b.')
plt.errorbar(x_data,y_data,a_p,theo_FT,'b.',label='Raw Data')
plt.ylabel('Acceleration (m/s^2)')
plt.xlabel('Force (N)')
plt.title('Force on the Glider vs. the Acceleration of the Glider')

# Takes the x and y values to make a trendline
#print(xi)
#print(yi)
intercept, slope, dintercept, dslope = linear_fit_plot_errors(xi,yi,0.05155827,0.09257908)

# Adds the legend to the plot
plt.legend()

# Displays the plot
plt.show()

print()
```

***

```python
# Raw Data
# Cart 3

#### Enter Raw Data Here!!!!!!!!!!!!!! ####

# Mass of the cart (kg) and its uncertainty
m_cart3 = 0.30021
delta_m_cart = 0.005

# Mass of hanging mass (kg) and its uncertainty
m_hang = 0.00686
delta_m_hang = 0.005

# Theoretical Force of tension (N)
theo_FT = 0.06579318164

# Predicted Acceleration (m/s^2) and its uncertainty
a_p = 0.2191571954
delta_a_p = 0.1603

###########################################

# Reads the name of the csv file and gets the data
df = pd.read_csv("./ExampleFiles/Lab4 cart 3.csv")

# Calculates the change in velocity and time
Delta_v = df['v2 (m/s)'] - df['v1 (m/s)']
Delta_t = df['t2 (s)'] - df['t1 (s)']

# Calculates the uncertainty of the velocity based on Equation 1
uncertainty_v = np.sqrt(delta_x**2 + 0.1**2)

# Finds the uncertainty of acceleration and adds it to the dataframe
unc_am = df['a (m/s^2)']*np.sqrt((uncertainty_v/Delta_v)**2 + (0.1/Delta_t)**2)
df.insert(5, "𝛿am (m/s^2)", unc_am)

# Finds the average measured acceleration
av_am = sum(df['a (m/s^2)'])/len(df['a (m/s^2)'])

# prints out the cart data
print("Data for: ")
print("m_cart = %0.5f kg              𝛿m_cart = %0.3f" % (m_cart3, delta_m_cart))
print("m_hang = %0.5f kg              𝛿m_hang = %0.3f" % (m_hang, delta_m_hang))
print("Predicted a_p = %0.3f m/s^2      𝛿a_p = %0.4f" % (a_p, delta_a_p))
print("Theoretical F_T = %0.5f N" % (theo_FT))
print()
print('-'*70)

# Prints information about the file
#df.info()
print(df)
print()
# Print statements for uncertainty + average
print("𝛿v = %0.5f m/s" % (uncertainty_v))
print("Average am = %0.5f m/s^2" % (av_am))
print('-'*70)
print()

# Inserts force into dataframe
force1 = df['a (m/s^2)']*m_cart3
df.insert(6, "Force (N)", force1)

# Defines the x and y values
acceleration = df.filter(['a (m/s^2)'], axis=1).dropna()
force = df.filter(['Force (N)'], axis=1).dropna()

# Create a figure of reasonable size and resolution, white background, black edge color
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')

# Gets the data values for x and y
x_data = force.values.reshape(-1, 1)
y_data = acceleration.values.reshape(-1, 1)

xi = df['Force (N)'].to_numpy()
yi = df['a (m/s^2)'].to_numpy()

# Creates the base plot with titles
#plt.plot(x_data,y_data,'b.')
plt.errorbar(x_data,y_data,a_p,theo_FT,'b.',label='Raw Data')
plt.ylabel('Acceleration (m/s^2)')
plt.xlabel('Force (N)')
plt.title('Force on the Glider vs. the Acceleration of the Glider')

# Takes the x and y values to make a trendline
#print(df['Force (N)'])
#print(xi)
intercept, slope, dintercept, dslope = linear_fit_plot_errors(xi,yi,0.038837,0.039346)

# Adds the legend to the plot
plt.legend()

# Displays the plot
plt.show()

print()
```

***

```python
# Raw Data
# Cart 4

#### Enter Raw Data Here!!!!!!!!!!!!!! ####

# Mass of the cart (kg) and its uncertainty
m_cart4 = 0
delta_m_cart = 0

# Mass of hanging mass (kg) and its uncertainty
m_hang = 0
delta_m_hang = 0

# Theoretical Force of tension (N)
theo_FT = 0

# Predicted Acceleration (m/s^2) and its uncertainty
a_p = 0
delta_a_p = 0

###########################################

# Reads the name of the csv file and gets the data
df = pd.read_csv("./ExampleFiles/Lab4 cart 00.csv")

# Calculates the change in velocity and time
Delta_v = df['v2 (m/s)'] - df['v1 (m/s)']
Delta_t = df['t2 (s)'] - df['t1 (s)']

# Calculates the uncertainty of the velocity based on Equation 1
uncertainty_v = np.sqrt(delta_x**2 + 0.1**2)

# Finds the uncertainty of acceleration and adds it to the dataframe
unc_am = df['a (m/s^2)']*np.sqrt((uncertainty_v/Delta_v)**2 + (0.1/Delta_t)**2)
df.insert(5, "𝛿am (m/s^2)", unc_am)

# Finds the average measured acceleration
av_am = sum(df['a (m/s^2)'])/len(df['a (m/s^2)'])

# prints out the cart data
print("Data for: ")
print("m_cart = %0.5f kg              𝛿m_cart = %0.3f" % (m_cart4, delta_m_cart))
print("m_hang = %0.5f kg              𝛿m_hang = %0.3f" % (m_hang, delta_m_hang))
print("Predicted a_p = %0.3f m/s^2      𝛿a_p = %0.4f" % (a_p, delta_a_p))
print("Theoretical F_T = %0.5f N" % (theo_FT))
print()
print('-'*70)

# Prints information about the file
#df.info()
print(df)
print()
# Print statements for uncertainty + average
print("𝛿v = %0.5f m/s" % (uncertainty_v))
print("Average am = %0.5f m/s^2" % (av_am))
print('-'*70)
print()

# Inserts force into dataframe
force1 = df['a (m/s^2)']*m_cart4
df.insert(6, "Force (N)", force1)

# Defines the x and y values
acceleration = df.filter(['a (m/s^2)'], axis=1).dropna()
force = df.filter(['Force (N)'], axis=1).dropna()

# Create a figure of reasonable size and resolution, white background, black edge color
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')

# Gets the data values for x and y
x_data = force.values.reshape(-1, 1)
y_data = acceleration.values.reshape(-1, 1)

xi = df['Force (N)'].to_numpy()
yi = df['a (m/s^2)'].to_numpy()

# Creates the base plot with titles
#plt.plot(x_data,y_data,'b.')
plt.errorbar(x_data,y_data,a_p,theo_FT,'b.',label='Raw Data')
plt.ylabel('Acceleration (m/s^2)')
plt.xlabel('Force (N)')
plt.title('Force on the Glider vs. the Acceleration of the Glider')

# Takes the x and y values to make a trendline
#print(force)
intercept, slope, dintercept, dslope = linear_fit_plot_errors(xi,yi,0)

# Adds the legend to the plot
plt.legend()

# Displays the plot
plt.show()

print()
```

***

```python
# Raw Data
# Cart 5

#### Enter Raw Data Here!!!!!!!!!!!!!! ####

# Mass of the cart (kg) and its uncertainty
m_cart5 = 0
delta_m_cart = 0

# Mass of hanging mass (kg) and its uncertainty
m_hang = 0
delta_m_hang = 0

# Theoretical Force of tension (N)
theo_FT = 0

# Predicted Acceleration (m/s^2) and its uncertainty
a_p = 0
delta_a_p = 0

###########################################

# Reads the name of the csv file and gets the data
df = pd.read_csv("./ExampleFiles/Lab4 cart 00.csv")

# Calculates the change in velocity and time
Delta_v = df['v2 (m/s)'] - df['v1 (m/s)']
Delta_t = df['t2 (s)'] - df['t1 (s)']

# Calculates the uncertainty of the velocity based on Equation 1
uncertainty_v = np.sqrt(delta_x**2 + 0.1**2)

# Finds the uncertainty of acceleration and adds it to the dataframe
unc_am = df['a (m/s^2)']*np.sqrt((uncertainty_v/Delta_v)**2 + (0.1/Delta_t)**2)
df.insert(5, "𝛿am (m/s^2)", unc_am)

# Finds the average measured acceleration
av_am = sum(df['a (m/s^2)'])/len(df['a (m/s^2)'])

# prints out the cart data
print("Data for: ")
print("m_cart = %0.5f kg              𝛿m_cart = %0.3f" % (m_cart5, delta_m_cart))
print("m_hang = %0.5f kg              𝛿m_hang = %0.3f" % (m_hang, delta_m_hang))
print("Predicted a_p = %0.3f m/s^2      𝛿a_p = %0.4f" % (a_p, delta_a_p))
print("Theoretical F_T = %0.5f N" % (theo_FT))
print()
print('-'*70)

# Prints information about the file
#df.info()
print(df)
print()
# Print statements for uncertainty + average
print("𝛿v = %0.5f m/s" % (uncertainty_v))
print("Average am = %0.5f m/s^2" % (av_am))
print('-'*70)
print()

# Inserts force into dataframe
force1 = df['a (m/s^2)']*m_cart5
df.insert(6, "Force (N)", force1)

# Defines the x and y values
acceleration = df.filter(['a (m/s^2)'], axis=1).dropna()
force = df.filter(['Force (N)'], axis=1).dropna()

# Create a figure of reasonable size and resolution, white background, black edge color
fig=plt.figure(figsize=(7,5), dpi= 100, facecolor='w', edgecolor='k')

# Gets the data values for x and y
x_data = force.values.reshape(-1, 1)
y_data = acceleration.values.reshape(-1, 1)

xi = df['Force (N)'].to_numpy()
yi = df['a (m/s^2)'].to_numpy()

# Creates the base plot with titles
#plt.plot(x_data,y_data,'b.')
plt.errorbar(x_data,y_data,a_p,theo_FT,'b.',label='Raw Data')
plt.ylabel('Acceleration (m/s^2)')
plt.xlabel('Force (N)')
plt.title('Force on the Glider vs. the Acceleration of the Glider')

# Takes the x and y values to make a trendline
#print(force)
intercept, slope, dintercept, dslope = linear_fit_plot_errors(xi,yi,0)

# Adds the legend to the plot
plt.legend()

# Displays the plot
plt.show()

print()
```

***

# Your Report
### Experimental Methods
You will need to write a few paragraphs to carefully describe how you set up and executed your
experiment. You must include the following items:
- Explain your procedure to ensure that the acceleration of the cart was caused only by the gravitational force due to the hanging mass.
- What did you use for your “flag” for the photogate?
- How did you take your data to reduce random errors?

Write your procedure section so that someone else could perform the experiment by just reading it (but do not write instructions). Talk about any problems you encountered and how you handled them.

### Results and Conclusion
This will be your longest section. Remember that your goal is to compare your theoretical prediction with your experimental results. Use good paragraph structure to link the results together:
- How do the slopes of your Tension vs. Acceleration graphs compare to the mass of the cart? Use the % error formula and discuss.
- Did you have any random errors? Any systematic errors? How do you know?
- Does your data support or refute Newton’s Second Law? *Why?*
- How might you reduce your uncertainties if you were to run the experiment again?

### Questions
1. Calculate the acceleration of a 0.5 kg cart with a 1.0 kg hanging mass.
2. Calculate the acceleration of a 1.0 kg cart with a 0.5 kg hanging mass.
3. Calculate the uncertainty of the acceleration for each of these. Which is lower?
4. Does the photogate measure the true instantaneous velocity of the cart? *Why or why not?*

### Appendix
*Include your signed raw data sheet(s)*
*Include a sample calculation for every computation you need to make.*


# Seminar 12

# Introduction to numerical optimization (Y. E. Nesterov Introduction to convex optimization, ch. 1 $\S$ 1.1)

 1. Review of the spring term topics
 2. Problem statement
 3. General scheme of any optimization method
 4. Comparison of optimization methods
 5. Methods to solve one-dimansional minimization problem

## Syllabus of the spring term

Also, see on the [GitHub course page](https://github.com/amkatrutsa/MIPT-Opt#spring-term).

1. Methods to solve **unconstrained** optimization problem
    - One-dimensional optimization problem (**already today!**)
    - Gradient descent
    - Newton method
    - Quasi-Newton methods
    - Conjugate gradient method
    - Optional:
        - Lerast squares problem
        - Optimal methods and lower bounds
2. Methods to solve **constrained** optimization problem
    - Linear programming
    - Projected gradient method and conditional gradient method
    - Barrier method
    - Penalty function methods
    - Augmented Lagrangian method

## Organizational

1. One seminar and one lecture per week
2. Two problem sets
3. Midterm in the middle of the term
4. Final test at the end of the term
5. Oral exam at the end of the term (grading for a semester is similar to the fall term)
6. Minitests in the deginning of every class
7. Homework assignment almost every week: $\TeX$ or Jupyter Notebook

## Problem statement

\begin{equation}
\begin{split}
& \min_{x \in S} \; f_0(x)\\
\text{s.t. } & f_j(x) = 0, \; j = 1,\ldots,m\\
& g_k(x) \leq 0, \; k = 1,\ldots,p
\end{split}
\end{equation}
where $S \subseteq \mathbb{R}^n$, $f_j: S \rightarrow \mathbb{R}, \; j = 0,\ldots,m$, $g_k: S \rightarrow \mathbb{R}, \; k=1,\ldots,p$

All functions are at least continuous.

Important fact: **nonlinear** optimization problem in its general form is

**numerically intractable**!

## Analytical results
- First order necessary condition:

if $x^*$ is a local minimum point of the differentiable function $f(x)$, then
$$
f'(x^*) = 0
$$
- Second order necessary condition:

if $x^*$ is a local minimum point of the twice differentiable function $f(x)$, then

$$
f'(x^*) = 0 \quad \text{и} \quad f''(x^*) \succeq 0
$$
- Sufficient condition:

Assume $f(x)$ is twice differentiable function and $x^*$ satisfies the following condition

$$
f'(x^*) = 0 \quad f''(x^*) \succ 0,
$$
then $x^*$ is a strict local minimum point of function $f(x)$.

**Remark**: check that you can prove these claims!

## Features of numerical solutions

1. Exact solution of the given problem is impossible due to precision of machine arithmetic
2. It is necessary to define the way to check if current point is a solution or not
3. It is necessary to define what information about the problem is stored

## General iterative scheme

Given: initial guess $x$, required tolerance $\varepsilon$.

```python
def GeneralScheme(x, epsilon):

    while StopCriterion(x) > epsilon:

        OracleResponse = RequestOracle(x)

        UpdateInformation(I, x, OracleResponse)

        x = NextPoint(I, x)

    return x
```

### Questions
1. What are the possible stopping criteria?
2. What is an oracle and what is it for?
3. What is information model?
4. How to get next point?

#### Stopping criteria
1. Convergence in $x$:
$$
\| x_k - x^*  \|_2 < \varepsilon
$$
2. Convergence in $f$:
$$
\| f_k - f^* \|_2 < \varepsilon
$$
3. Necessary condition
$$
\| f'(x_k) \|_2 < \varepsilon
$$

But we don't know $x^*$!

Then
$$
\|x_{k+1} - x_k \| = \|x_{k+1} - x_k + x^* - x^* \| \leq \|x_{k+1} - x^* \| + \| x_k - x^* \| \leq 2\varepsilon
$$

The same is true for the convergence in $f$, but sometimes $f^*$ can be estimated!

**Remark**: better practise is to use relative difference in argument and functional, for example $\dfrac{\|x_{k+1} - x_k \|_2}{\| x_k \|_2}$

#### What is oracle?
**Definition**: oracle is some abstact machine that responses on the sequaential method requests

OOP analogy:

- oracle is a virtual method of the base class
- every problem is derived class
- oracle is defined for every particular problem according to the declaration in the base class

**Black box concept**
1. Iterative method can use only oracle responses
2. Oracle responses are *local*

#### Information about the problem
1. Every oracle response gives **local** information about function behaviour in the given point
2. After aggregation of the oracle responses, we update **global** information about objective function:
    - curvature
    - descent direction
    - etc

#### Compute next point

$$
x_{k+1} = x_{k} + \alpha_k h_k
$$

- **Line search**: fix direction $h_k$ and search for this direction the optimal value of $\alpha_k$
- **Trust region method**: fix appropriate size of *region* in some norm $\| \cdot  \| \leq \alpha$ and *model* of the objective function, which is a good approximation in the considered region.
    Next, we search direction $h_k$, that minimizes the chosen model of the objective function and does not lead to the point $x_k + h_k$ lying outside of the considered region

Questions:
1. How to choose $\alpha_k$?
2. How to choose $h_k$?
3. How to choose model?
4. How to choose region?
5. How to choose region size?

    In this course we consider only line search methods!

However someiemes the concept of trust region methods will be helpful.

## How to compare optimization methods?
For given class of problems one can compare the following quantities:
1. Complexity
    - analytical: number of the oracle requests to solve the problem with accuracy $\varepsilon$
    - arithmetic: total number of computations to solve the problem with accuracy $\varepsilon$
2. Convergence speed
3. Experiments

### Convergence speed
1. Sublinear
$$
\| x_{k+1} - x^* \|_2 \leq C k^{\alpha},
$$
where $\alpha < 0$ и $ 0 < C < \infty$
2. Linear (geometric progression)
$$
\| x_{k+1} - x^* \|_2 \leq Cq^k,
$$
where $q \in (0, 1)$ and $ 0 < C < \infty$

3. Superlinear
$$
\| x_{k+1} - x^* \|_2 \leq Cq^{k^p},
$$
where $q \in (0, 1)$, $ 0 < C < \infty$ and $p > 1$
4. Quadratic
$$
\| x_{k+1} - x^* \|_2 \leq C\| x_k - x^* \|^2_2, \qquad \text{or} \qquad \| x_{k+1} - x^* \|_2 \leq C q^{2^k}
$$
where $q \in (0, 1)$ and $ 0 < C < \infty$

### Optimal methods: can we do better?
- Authors prove lower bounds of the convergence speed for given set of problems and methods of given order
- Next, the methods, for which these lower bounds are tight, were proposed $\Rightarrow$ optimality is proved
- Later more about convergence theorem

```python
%matplotlib inline
import matplotlib.pyplot as plt

USE_COLAB = False
if not USE_COLAB:
    plt.rc("text", usetex=True)

import numpy as np
C = 10
alpha = -0.5
q = 0.9
num_iter = 7
sublinear = np.array([C * k**alpha for k in range(1, num_iter + 1)])
linear = np.array([C * q**k for k in range(1, num_iter + 1)])
superlinear = np.array([C * q**(k**2) for k in range(1, num_iter + 1)])
quadratic = np.array([C * q**(2**k) for k in range(1, num_iter + 1)])
plt.figure(figsize=(12,8))
plt.semilogy(np.arange(1, num_iter+1), sublinear,
             label=r"Sublinear, $\alpha = -0.5$")
plt.semilogy(np.arange(1, num_iter+1), superlinear,
             label=r"Superlinear, $q = 0.5, p=2$")
plt.semilogy(np.arange(1, num_iter+1), linear,
             label=r"Linear, $q = 0.5$")
plt.semilogy(np.arange(1, num_iter+1), quadratic,
             label=r"Quadratic, $q = 0.5$")
plt.xlabel("Number of iteration, $k$", fontsize=24)
plt.ylabel("Error rate upper bound", fontsize=24)
plt.legend(loc="best", fontsize=28)
plt.xticks(fontsize = 28)
_ = plt.yticks(fontsize = 28)
```

### On interpretations of the convergence theorems (B.T. Polyak Introduction to optimization, ch. 1, $\S$ 6)
1. What answers give convergence theorems?
     - class of problems for which the method is applicable (it is important to minimize the number of restrictions!)
         - convexity
         - smoothness
     - qualitative behaviour of the method
         - is initial guess significant for convergence of method?
         - what convergence can be observed?
     - estimate of convergence speed
         - theoretical estimate of the method behavior without experiments
         - detection of properties that affect convergence (condition number, dimension, etc)
         - sometimes one can set number of iterations to achieve required accuracy in advance

2. What answers do not give convergence theorem?
    - convergence of the method **does not mean** that it has to be used
    - convergence estimates depend on unknown constants - only theoretical interest
    - theorems do not take into account rounding errors and accuracy of solving auxiliary problems

**Main point**: it is necessary to use common sense and be careful in usage of convergence theorem!

## Classes of problems
1. Uncontrained optimization
    - Lipschitz objective function
    - Lipschitz gradient of the objective function
2. Contrained optimization
    - polytope
    - sets with simple structure
    - general form

## Classes of methods
1. Zero order method: oracle returns only objective function $f(x)$

2. Fist order method : oracle returns objective function $f(x)$ and its gradient $f'(x)$

3. Second order method: oracle returns objective function $f(x)$, its gradient $f'(x)$ and its hessian $f''(x)$.

**Q**: do methods of higher order exist?

1. One-step methods
$$
x_{k+1} = \Phi(x_k)
$$
2. Multi-step methods
$$
x_{k+1} = \Phi(x_k, x_{k-1}, ...)
$$

## One-dimensional optimization
**Definition**. Funtion $f(x)$ is unimodal in interval $[a, b]$, if there exists such point $x^* \in [a, b]$, that
- $f(x_1) > f(x_2)$ for any $a \leq x_1 < x_2 < x^*$,

and

- $f(x_1) < f(x_2)$ for any $x^* < x_1 < x_2 \leq b$.

**Q**: what geometry of unimodal function?

### Bisection method

Idea from the first term CS course: divide given interval $[a,b]$ on two equal parts till minimum of the unimodal function is not found

Denite by $N$ the number of computations of function $f$, then one can perform $K = \frac{N - 1}{2}$ iterations and the following estimate holds:
$$
|x_{K+1} - x^*| \leq \frac{b_{K+1} - a_{K+1}}{2} = \left( \frac{1}{2} \right)^{\frac{N-1}{2}} (b - a) \approx 0.5^{K} (b - a)
$$

```python
def binary_search(f, a, b, epsilon, callback=None):
    c = (a + b) / 2.0
    while abs(b - a) > epsilon:
#         Check left subsegment
        y = (a + c) / 2.0
        if f(y) <= f(c):
            b = c
            c = y
        else:
#         Check right subsegment
            z = (b + c) / 2.0
            if f(c) <= f(z):
                a = y
                b = z
            else:
                a = c
                c = z
        if callback is not None:
            callback(a, b)
    return c
```

```python
def my_callback(a, b, left_bound, right_bound, approximation):
    left_bound.append(a)
    right_bound.append(b)
    approximation.append((a + b) / 2.0)
```

```python
# %matplotlib inline
import numpy as np
# import matplotlib.pyplot as plt

left_boud_bs = []
right_bound_bs = []
approximation_bs = []

callback_bs = lambda a, b: my_callback(a, b,
            left_boud_bs, right_bound_bs, approximation_bs)

# Target unimodal function on given segment
f = lambda x: (x - 2) * x * (x + 2)**2 # np.power(x+2, 2)
# f = lambda x: -np.sin(x)
x_true = -2
# x_true = np.pi / 2.0
a = -3
b = -1.5
epsilon = 1e-8
x_opt = binary_search(f, a, b, epsilon, callback_bs)
print(np.abs(x_opt - x_true))
plt.figure(figsize=(10,6))
plt.plot(np.linspace(a,b), f(np.linspace(a,b)))
plt.title("Objective function", fontsize=20)
plt.xticks(fontsize=20)
_ = plt.yticks(fontsize=20)
```

### Golden search method
Idea: divide interval $[a,b]$ not on two eqwual parts, but in the golden ratio.

Estimate convergence speed like in bisection method:
$$
|x_{K+1} - x^*| \leq b_{K+1} - a_{K+1} = \left( \frac{1}{\tau} \right)^{N-1} (b - a) \approx 0.618^K(b-a),
$$
where $\tau = \frac{\sqrt{5} + 1}{2}$.

- Constant of linear convergence is **higher**, than corresponding constant in bisection method
- Number of function calls is **less** than for the bisection method

```python
def golden_search(f, a, b, tol=1e-5, callback=None):
    tau = (np.sqrt(5) + 1) / 2.0
    y = a + (b - a) / tau**2
    z = a + (b - a) / tau
    while b - a > tol:
        if f(y) <= f(z):
            b = z
            z = y
            y = a + (b - a) / tau**2
        else:
            a = y
            y = z
            z = a + (b - a) / tau
        if callback is not None:
            callback(a, b)
    return (a + b) / 2.0
```

```python
left_boud_gs = []
right_bound_gs = []
approximation_gs = []

cb_gs = lambda a, b: my_callback(a, b, left_boud_gs, right_bound_gs, approximation_gs)
x_gs = golden_search(f, a, b, epsilon, cb_gs)

print(f(x_opt))
print(f(x_gs))
print(np.abs(x_opt - x_true))
```

    6.93889390875399e-18
    9.549014390504221e-18
    9.313225746154785e-10

### Comparison of the methods for one-dimensional problem

```python
plt.figure(figsize=(10,6))
plt.semilogy(np.arange(1, len(approximation_bs) + 1), np.abs(x_true - np.array(approximation_bs, dtype=np.float64)), label="Binary search")
plt.semilogy(np.arange(1, len(approximation_gs) + 1), np.abs(x_true - np.array(approximation_gs, dtype=np.float64)), label="Golden search")
plt.xlabel(r"Number of iterations, $k$", fontsize=24)
plt.ylabel("Error rate upper bound", fontsize=24)
plt.legend(loc="best", fontsize=24)
plt.xticks(fontsize = 24)
_ = plt.yticks(fontsize = 24)
```

```python
%timeit binary_search(f, a, b, epsilon)
%timeit golden_search(f, a, b, epsilon)
```

    21.1 µs ± 280 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
    84.1 µs ± 345 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

## Example of other behaviour of the considered methods

$$
f(x) = \sin(\sin(\sin(\sqrt{x}))), \; x \in [2, 60]
$$

```python
f = lambda x: np.sin(np.sin(np.sin(np.sqrt(x))))
x_true = (3 * np.pi / 2)**2
a = 2
b = 60
epsilon = 1e-8
plt.plot(np.linspace(a,b), f(np.linspace(a,b)))
plt.xticks(fontsize = 24)
_ = plt.yticks(fontsize = 24)
plt.title("Objective function", fontsize=20)
```

## Comparison of convergence speed and execution time

### Bisection method

```python
left_boud_bs = []
right_bound_bs = []
approximation_bs = []

callback_bs = lambda a, b: my_callback(a, b,
            left_boud_bs, right_bound_bs, approximation_bs)

x_opt = binary_search(f, a, b, epsilon, callback_bs)
print(np.abs(x_opt - x_true))
```

    2.1968899233115735e-07

### Golden section method

```python
left_boud_gs = []
right_bound_gs = []
approximation_gs = []

cb_gs = lambda a, b: my_callback(a, b, left_boud_gs, right_bound_gs, approximation_gs)
x_gs = golden_search(f, a, b, epsilon, cb_gs)

print(np.abs(x_opt - x_true))
```

    2.1968899233115735e-07

### Convergence

```python
plt.figure(figsize=(8,6))
plt.semilogy(np.abs(x_true - np.array(approximation_bs, dtype=np.float64)), label="Binary")
plt.semilogy(np.abs(x_true - np.array(approximation_gs, dtype=np.float64)), label="Golden")
plt.legend(fontsize=24)
plt.xticks(fontsize=24)
_ = plt.yticks(fontsize=24)
plt.xlabel(r"Number of iterations, $k$", fontsize=24)
plt.ylabel("Error rate upper bound", fontsize=24)
```

### Execution time

```python
%timeit binary_search(f, a, b, epsilon)
%timeit golden_search(f, a, b, epsilon)
```

    461 µs ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    436 µs ± 3.71 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

## Recap
1. Introduction to numerical optimization
2. General scheme of any optimization method
3. How to compare optimization methods
4. Zoo of the optimization methods and problems
5. One-dimensional minimization