# Fitting a straight line to data

_Inspired by [Hogg et al. 2010](https://arxiv.org/abs/1008.4686) and [@jakevdp's notes](https://github.com/jakevdp/ESAC-stats-2014)_.

Python imports we'll need later...

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize

%matplotlib inline
rnd = np.random.RandomState(seed=42)
```

---

# Intro and choice of objective function

I want to start with a problem that everyone is probably familiar with or has at least seen before. The problem is this: we observe $N$ independent data points $\boldsymbol{y}=\{y_1,y_2,...y_N\}$ with uncertainties $\boldsymbol{\sigma}=\{\sigma_1,\sigma_2,...\sigma_N\}$ at perfectly-measured values $\boldsymbol{x}=\{x_1,x_2,...x_N\}$. We have reason to believe that the these data were generated by a process that is well-represented by a straight-line, and the only reason that the data deviate from this straight line is because of uncorrelated, Gaussian measurement noise in the $y$-direction. Let's first generate some data that meet these qualifications:

```python
n_data = 16 # number of data points
a_true = 1.255 # randomly chosen truth
b_true = 4.507
```

---

### Exercise 1:

1. Randomly generate an array of uniformly-distributed `x` values from the domain `(0,2)`.
2. Sort the values in ascending order.

```python
# Fill in your solution here
x = rnd.rand(n_data) * 2
x = np.sort(x)
```

Execute the code below and verify that it executes:

```python
# evaluate the true model at the given x values
y = a_true*x + b_true

# Heteroscedastic Gaussian uncertainties only in y direction
y_err = rnd.uniform(0.1, 0.2, size=n_data) # randomly generate uncertainty for each datum
y = y + rnd.normal(0, y_err) # add noise to y data
```

```python
plt.errorbar(x, y, y_err, marker='o', linestyle='none')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.tight_layout()
```

---

Now let's forget that we did that -- we know nothing about the model parameters, except that we think the true values of the data are well-described by a linear relation! We would like to measure the "best-fit" parameters of this model (for a straight line, the slope and intercept $(a,b)$) given the data above. In math, our model for the data $y$ is:
$$
\begin{align}
y &= f(x \,;\, a, b) + {\rm noise}\\
f(x \,;\, a, b) &= a\,x + b
\end{align}
$$

For a given set of parameters, $(a,b)$, we can evaluate our model $f(x \,;\, a, b)$ at a given $x$ location to compute the value of $y$ that we would expect in the absence of noise. For example, for the $n$th datum and for a given set of parameter values $(a,b)$:

$$
\tilde{y}_n = f(x_n \,;\, a, b)
$$

Now, we somehow want to search through all possible values of $a,b$ to find the "best" values, given the data, with some definition of "best." When we say this word, we are implying that we want to _optimize_ (find the maximum or minimum) some _objective function_ (a function that takes our data, our model, and returns a quantification of "best", usually as a scalar). Numerically, this scalar objective function can be any function (though you probably want it to be convex) and you will see different choices in practice. You have some leeway in this choice depending on whether your goal is _prediction_, _discovery_, or _data compression_.

However, for _inference_—the typical use-case for us as scientists—you don't have this freedom: one of the conclusions of this talk is going to be that __you have no choice about what "best" means__! Before we get there, though, let's explore what seem like reasonable choices.

Here are a few desirable features we'd like any objective function to have:

1. For a given set of parameters, we should compare our predicted values to the measured values and base our objective function on the differences
2. The scalar value should be dimensionless (the value of the objective function shouldn't care if we use kilometers vs. parsecs)
3. Data points that have larger errors should contribute less to the objective function (if a datum has a large offset from the predicted value, it shouldn't matter _if_ the datum has a large uncertainty)
4. Convexity

To meet these three criteria, whatever objective function we choose should operate on the (dimensionless) quantities:

$$
\chi_n = \frac{y_n - \tilde{y}_n(x_n; a,b)}{\sigma_n}
$$

i.e. the difference between our predicted values $\tilde{y}$ and the observed $y$ values, weighted by the inverse uncertainties $\sigma$. The uncertainties have the same units as the data, so this is a dimensionless quantity. It also has the nice property that, as we wanted, points with large uncertainties are _downweighted_ relative to points with small uncertainties. Here are some ideas for objective functions based on this scalar:

- __Weighted absolute deviation__: the sum of the absolute values

    $\sum_n^N \, \left|\chi_n\right|$

- __Weighted squared deviation__: the sum of the squares

    $\sum_n^N \, \chi_n^2$

- __Weighted absolute deviation to some power__ $p$:

    $\sum_n^N \, \left|\chi_n\right|^p $

_(Note: don't show this to statisticians or they will get me fired. To a statistician, $\chi^2$ is a distribution not a statistic...but astronomers seem to use this terminology.)_

For simplicity, let's just compare two of these: the absolute deviation and the squared deviation.

---

### Exercise 2:

Implement the functions to compute the weighted deviations below

```python
# FILL IN THESE FUNCTIONS:

def line_model(pars, x):
    return pars[0] * x + pars[1]

def weighted_absolute_deviation(pars, x, y, y_err):
    chi = np.divide(y - line_model(pars,x), y_err)
    return sum(np.abs(chi))

def weighted_squared_deviation(pars, x, y, y_err):
    chi_squared = np.divide(y - line_model(pars,x), y_err)**2
    return sum(chi_squared)

```

Verify that you've correctly implemented your functions by executing the following cell:

```python
_pars = [1., -10.]
_x = np.arange(16)
_y = _x
_yerr = np.ones_like(_x)

truth = np.array([-10.,  -9.,  -8.,  -7.,  -6.,  -5.,  -4.,  -3.,  -2.,  -1.,   0., 1.,   2.,   3.,   4.,   5.])
assert np.allclose(line_model(_pars, _x), truth), 'Error in line_model() function!'
assert weighted_absolute_deviation(_pars, _x, _y, _yerr) == 160., 'Error in weighted_absolute_deviation() function!'
assert weighted_squared_deviation(_pars, _x, _y, _yerr) == 1600., 'Error in weighted_squared_deviation() function!'
```

---

We can demonstrate that these are convex (over some domain) by computing the objective function values over a grid of parameter values (a grid in $a, b$):

```python
# make a 256x256 grid of parameter values centered on the true values
a_grid = np.linspace(a_true-2., a_true+2, 256)
b_grid = np.linspace(b_true-2., b_true+2, 256)
a_grid,b_grid = np.meshgrid(a_grid, b_grid)
ab_grid = np.vstack((a_grid.ravel(), b_grid.ravel())).T
```

```python
fig,axes = plt.subplots(1, 2, figsize=(9,5.1), sharex=True, sharey=True)

for i,func in enumerate([weighted_absolute_deviation, weighted_squared_deviation]):
    func_vals = np.zeros(ab_grid.shape[0])
    for j,pars in enumerate(ab_grid):
        func_vals[j] = func(pars, x, y, y_err)

    axes[i].pcolormesh(a_grid, b_grid, func_vals.reshape(a_grid.shape),
                       cmap='Blues', vmin=func_vals.min(), vmax=func_vals.min()+256) # arbitrary scale

    axes[i].set_xlabel('$a$')

    # plot the truth
    axes[i].plot(a_true, b_true, marker='o', zorder=10, color='#de2d26')
    axes[i].axis('tight')
    axes[i].set_title(func.__name__, fontsize=14)

axes[0].set_ylabel('$b$')

fig.tight_layout()
```

There are minima in both cases near the true values of the parameters (good), but the functions clearly look different. Which one should we choose for finding the best parameters?

In order to pick between these two, or any of the arbitrary objective functions we could have chosen, we have to _justify_ using one function over the others. In what follows, we'll justify optimizing the sum of the squared deviations (so-called "least-squares fitting") by thinking about the problem _probabilistically_, rather than procedurally.

### Least-squares fitting

Let's review the assumptions we made above in generating our data:

1. The data were generated by a straight line
2. Uncorrelated, _known_ Gaussian uncertainties in $y$ cause deviations between the data and predictions
3. The data points are independent
4. The $x$ data are known perfectly, or at least their uncertainties are _far smaller_ than the uncertainties in $y$

First off, these assumptions tell us that for each datum $(x_n, y_n)$ there is some true $y_{n,{\rm true}}$, and because of limitations in our observing process we can't observe the truth, but we know that the values we do observe will be Gaussian (Normal) distributed around the true value. _(Note: This assumption tends to be a good or at least a conservative approximation in practice, but there are certainly more complex situations when, e.g., you have asymmetric uncertainties, or error distributions with large tails!)_. In math:

$$
\begin{align}
p(y \,|\, y_{\rm true}) &= \mathcal{N}(y \,|\, y_{\rm true}, \sigma^2) \\
\mathcal{N}(y \,|\, y_{\rm true}, \sigma^2) &= (2\pi \sigma^2)^{-1/2} \, \exp\left(-\frac{1}{2} \frac{(y-y_{\rm true})^2}{\sigma^2} \right)
\end{align}
$$

This is the likelihood of observing a particular $y$ given the true $y_{\rm true}$. Note that in our model, all of the $y_{\rm true}$'s must lie on a line. It is also interesting that the argument of the normal distribution looks a lot like $\chi^2$!

What about considering two data points, $y_1$ and $y_2$? Now we need to write down the _joint_ probability

$$
p(y_1, y_2 \,|\, y_{1,{\rm true}}, \sigma_1, y_{2,{\rm true}}, \sigma_2)
$$

But, note that in assumption 3 above, we are assuming the data are independent. In that case, the random error in one point does not affect the random error in any other point, so the joint probability can be turned into a product:

$$
p(\{y_n\} \,|\, \{y_{n,{\rm true}}\}, \{\sigma_n\}) = \prod_n^N \, p(y_n \,|\, y_{n,{\rm true}}, \sigma_n)
$$

This is the full expression for the likelihood of the observed data given the true $y$ values. Recall that these true values, according to our assumptions, must lie on a line with some parameters, and we're trying to infer those parameters! We can compute a particular $y_{n,{\rm true}}$ using $x_n$ and a given set of model parameters $a, b$. With that in mind, we can write the likelihood instead as:

$$
p(\{y_n\} \,|\, a, b, \{x_n\}, \{\sigma_n\}) = \prod_n^N \, p(y_n \,|\, a, b, x_n, \sigma_n)
$$

So what are the "best" values of the parameters $a, b$? They are the ones that _maximize_ this likelihood!

The product on the right of the likelihood is a product over exponentials (well, Gaussians), which can be annoying to deal with. But, maximizing the likelihood is equivalent to maximizing the _log_-likelihood -- so we can get rid of the product and all of those exponentials by taking the log of both sides:

$$
\begin{align}
\ln p(\{y_n\} \,|\, a, b, \{x_n\}, \{\sigma_n\}) &= \sum_n^N \, \ln\left[p(y_n \,|\, a, b, x_n, \sigma_n)\right] \\
&= \sum_n^N \ln \left[(2\pi \sigma_n^2)^{-1/2} \,
    \exp\left(-\frac{1}{2} \frac{(y_n-(a\,x_n+b))^2}{\sigma_n^2} \right) \right] \\
&= -\frac{N}{2}\ln(2\pi)
    - \frac{1}{2} \sum_n^N \left[\frac{(y_n-(a\,x_n+b))^2}{\sigma_n^2} + \ln{\sigma_n^2} \right]
\end{align}
$$

In this case, the uncertainties are known and constant, so to maximize this expression we only care that (abbreviating the likelihood as $\mathcal{L}$):

$$
\begin{align}
\ln \mathcal{L} &= - \frac{1}{2} \sum_n^N \left[\frac{(y_n-(a\,x_n+b))^2}{\sigma_n^2}\right] + {\rm const.} \\
&= - \frac{1}{2} \sum_n^N \, \chi_n^2 + {\rm const.} \\
\end{align}
$$

Apparently, _minimizing_ the sum of the weighted squared deviations is equivalent to _maximizing_ the (log) likelihood derived from thinking about the probability of the data! That is great because (a) it directly gives us the uncertainties on the inferred model parameters, and (b) it's an analytic way to solve this problem using linear algebra which is _really_ fast!

### Least-squares / maximum likelihood with matrix calculus

Using linear algebra, we can simplify and generalize a lot of the expressions above. In what follows, all vectors are column vectors and are represented by lower-case bold symbols. Matrices are upper-case bold symbols.

We'll start by writing our model as a matrix equation. To do that, we need a way to, for a given set of parameters, compute the set of predicted $y$'s. This is done by defining the parameter vector, $\boldsymbol{\theta}$, and a matrix typically called the _design matrix_, $\boldsymbol{X}$:

$$
\boldsymbol{\theta} = \begin{bmatrix} b \\ a \end{bmatrix} \quad
\boldsymbol{X} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_N \end{bmatrix}
$$

(note the order of the parameters!). With these definitions, the vector of predicted $y$ values is just

$$
\boldsymbol{y}_{\rm pred} = \boldsymbol{X} \, \boldsymbol{\theta}
$$

so the deviation vector between the prediction and the data is just $(\boldsymbol{y}-\boldsymbol{X} \, \boldsymbol{\theta})$ where

$$
\boldsymbol{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{bmatrix}
$$

But how do we include the uncertainties? We'll pack the list of uncertainties (variances) into the trace of a 2D, $N \times N$ matrix called the _covariance matrix_. Because we are assuming the uncertainties are independent, the off-diagonal terms are all zero:

$$
\boldsymbol{\Sigma} = \begin{bmatrix}
\sigma_1^2 & 0 & \dots & 0 \\
0 & \sigma_2^2 & \dots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & 0 & \sigma_N^2
\end{bmatrix}
$$

With these matrices, we can write the expression for $\chi^2$ (and therefore the log-likelihood) very concisely:

$$
\begin{align}
\chi^2 &= \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right)^\mathsf{T} \,
    \boldsymbol{\Sigma}^{-1} \,
    \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right) \\
\ln\mathcal{L} &= -\frac{1}{2}\left[N\,\ln(2\pi)
    + \ln|\boldsymbol{\Sigma}|
    + \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right)^\mathsf{T} \,
      \boldsymbol{\Sigma}^{-1} \,
      \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right)
\right]
\end{align}
$$

In this form, the terms in the $\chi^2$ have a nice geometric interpretation: This looks like a distance between the data and the model computed with the metric $\boldsymbol{\Sigma}$.

If you solve for the optimum of the log-likelihood function (take the derivative with respect to $\boldsymbol{\theta}$ and set equal to 0), you find that:

$$
\newcommand{\trpo}[1]{{#1}^{\mathsf{T}}}
\newcommand{\bs}[1]{\boldsymbol{#1}}
\bs{\theta}_{\rm best} = \left[\trpo{\bs{X}} \, \bs{\Sigma}^{-1} \, \bs{X}\right]^{-1} \,
    \trpo{\bs{X}} \, \bs{\Sigma}^{-1} \, \bs{y}
$$

Getting the best-fit parameters just requires a few simple linear algebra operations! As an added bonus, we also get the _uncertainties_ on the parameters. The $2\times2$ covariance matrix for the best-fit parameters is given by the matrix:

$$
\newcommand{\trpo}[1]{{#1}^{\mathsf{T}}}
\newcommand{\bs}[1]{\boldsymbol{#1}}
C = \left[\trpo{\bs{X}} \, \bs{\Sigma}^{-1} \, \bs{X}\right]^{-1}
$$

That means we can just write out the linear algebra explicitly and use `numpy.linalg` to solve it for us!

### Exercise 3:

Implement the necessary linear algebra to solve for the best-fit parameters and the parameter covariance matrix, defined above. Call these `best_pars` and `pars_Cov`, respectively.

```python
X = np.vstack((np.ones(n_data), x)).T
sigma = np.diag(y_err)
sigma_inv = np.linalg.inv(sigma)

pars_Cov = np.linalg.inv(np.matmul(np.matmul(X.T, sigma_inv), X))
best_pars = np.matmul(np.matmul(np.matmul(pars_Cov, X.T),sigma_inv), y)

print(best_pars)
```

    [4.52632121 1.22928402]

```python
np.polyfit(x,y,deg=1)
```

    array([1.22042911, 4.54541   ])

Now let's look at the covariance matrix of the parameters (the uncertainty in the parameters) and plot the 1 and 2-sigma error ellipses:

```python
# some tricks to get info we need to plot an ellipse, aligned with
#    the eigenvectors of the covariance matrix
eigval,eigvec = np.linalg.eig(pars_Cov)
angle = np.degrees(np.arctan2(eigvec[1,0], eigvec[0,0]))
w,h = 2*np.sqrt(eigval)
```

```python
from matplotlib.patches import Ellipse

fig,ax = plt.subplots(1, 1, figsize=(5,5))

for n in [1,2]:
    ax.add_patch(Ellipse(best_pars, width=n*w, height=n*h, angle=angle,
                         fill=False, linewidth=3-n, edgecolor='#555555',
                         label=r'{}$\sigma$'.format(n)))

ax.plot(b_true, a_true, marker='o', zorder=10, label='truth')
ax.plot(best_pars[0], best_pars[1], marker='o', zorder=9, label='estimate')

ax.set_xlabel('$b$')
ax.set_ylabel('$a$')
ax.legend(loc='best')

fig.tight_layout()
```

There we have it! The best-fit parameters and their errors for the straight-line fit, optimized with the only justifyable objective function, directly from a few linear algebra calculations.

This approach can be generalized somewhat (e.g. to account for correlated errors as off-diagonal elements in the covariance matrix $\Sigma$), but it only works **for models with linear parameters**, meaning that parameters enter linearly in the model function $f$.

---
# Bonus material (peruse at your leisure)

## The Bayesian approach

Let's review what we did so far. We found that standard weighted least squares fitting is a justified approach to estimating the best-fit parameters because it optimizes the likelihood of the data under the assumptions of our model; it optimizes a _justified scalar objective function_. We then fit our straight-line model to the data using and got back a point-estimate of the best parameters along with a covariance matrix describing the uncertainties in the parameters. This is the way of the _frequentist_. What we're going to do now is see what happens if we switch to a Bayesian methodology instead. While the two methods end up looking mathematically identical, there are fundamental philosophical differences that can lead to very different interpretations and implementations when models are more complex than the toy example we use above.

As Bayesians, we aren't interested in a point-estimate of the best parameters, but rather we're interested in the inferred distribution of possible parameter values (the _posterior probability distribution function_ over parameters). So how do we write down or solve for this posterior pdf? Before we get to that, let's take a look at a fundamental equation of Bayesian statistics, [Bayes' theorem](https://en.wikipedia.org/wiki/Bayes'_theorem), which we'll derive using the joint probability of $A$ and $B$ which are conditional on some other information $I$ that, right now, we don't care about. For example, $A$ could be the time it takes to get from here to NYC, $B$ could be the amount of traffic on the road, and $I$ could include the information that we're driving a car and not walking. Bayes' theorem as expressed below is not controversial -- Bayesians and Frequentists agree that this is just how joint and conditional probabilities work. We start by writing down the joint probability of $A$ and $B$, then factor it in two ways into conditional proabilities:

$$
p(A,B \,|\, I) = p(A\,|\,B, I)\,p(B\,|\, I) = p(B\,|\,A, I)\,p(A \,|\, I)
$$

Now we look at the right two expressions, and divide by one of the marginal probabilities to get:

$$
p(A\,|\,B, I) = \frac{p(B\,|\,A, I)\,p(A \,|\, I)}{p(B\,|\, I)}
$$

Ok, so that's all fine. Now let's replace $A$ and $B$ with variables that represent, from our example above, our data $D=(\{x_n\},\{y_n\},\{\sigma_n\})$ and our model parameters $\boldsymbol{\theta}$:

$$
p(\boldsymbol{\theta}\,|\,D, I) = \frac{p(D\,|\,\boldsymbol{\theta}, I)\,p(\boldsymbol{\theta} \,|\, I)}{p(D\,|\, I)}
$$

In just switching the meaning of the variables, this expression becomes controversial! Frequentists would object to the above for two main reasons:

1. The term on the left hand side is a probability over parameters given the data (the _posterior_ pdf) $p(\boldsymbol{\theta}\,|\,D, I)$. This is something that a frequentist would say cannot exist - there is only one true vector of parameters that we are trying to learn, not a distribution!
2. The right-most term in the numerator is a probability over parameters _with no dependence on the data_ (the _prior_ pdf). This encapsulates all of our prior knowledge about the parameters before we did the experiment and observed some data. This is perhaps the aspect of Bayesian inference that frequentists most disagree with.

The differences above result from the fact that probability means something different to Frequentists and Bayesians. Bayesians think of probability as representing a _degree of belief_ about something, whereas a frequentist thinks of a probability as related to _limiting frequencies of occurrence_ in repeated trials or observations. This is a rich topic and I highly recommend reading [this series of blogposts](http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/) by Jake Vanderplas to learn more. For now, let's put on Bayesian hats and take a look at the implications of the expression above.

(_It's good to rememeber that we're all friends. The differences are based on philosophy and so can lead to some heated discussions and debates, but remember we're all trying to do science -- we're on the same team!_)

## Bayes' theorem and Bayesian inference

Let's decompose Bayes' theorem (as applied to modeling and inference). The four terms in Bayes' theorem above have names that are good to be familiar with:

- $p(\boldsymbol{\theta}\,|\,D, I)$ - __posterior probability__:
    This is the thing we are after when we do Bayesian inference or model fitting. We want to know what the distribution of possible parameter values is, given the data we observe and any prior information or assumptions $I$.

- $p(D\,|\,\boldsymbol{\theta}, I)$ - __likelihood__:
    This is the likelihood of the data given a particular set of model parameters. We've already seen this object and used it above to find the best-fit model parameters by maximizing this function. In a Bayesian context, it can also be thought of as a distribution -- it's a distribution that generates new datasets given a model instance. For that reason, we typically refer to models that produce a likelihood as _generative models_ because they specify how to  generate new data sets that look like the one you observe. As we saw above when we wrote the likelihood function for a straight line model and data with Gaussian errors, the likelihood usually contains a component that can be interpreted as the _noise model_.

- $p(\boldsymbol{\theta} \,|\, I)$ - __prior probability__
    This contains any relevant information about our parameters that we know before observing the data. This can include physical constraints, previous measurements, or anything, really. This flexibility is what makes the prior a somewhat controversial object. In practice, the prior only really matters if it is much narrower than the likelihood function. If the prior is broad with respect to the likelihood, the information in the likelihood makes the prior almost irrelevant. However, there are several subtleties to choosing priors that need to be considered. As an example, one subtlety comes from the choice of coordinates for the model parameters: a prior that is broad and flat in a parameter $\alpha$ won't be broad and flat if you change variables to $\beta = \alpha^2$.

- $p(D\,|\, I)$ - __evidence__ or __fully marginalized likelihood__ (FML)
    In many cases the evidence is simply a normalization constant and, for some of the most relevant algorithms used in inference, can be ignored. This term involves an integral over all of parameter space that can be very difficult to compute:

    $$
    p(D\,|\, I) = \int \,\mathrm{d}\boldsymbol{\theta} \, p(D\,|\,\boldsymbol{\theta}, I) \, p(\boldsymbol{\theta} \,|\, I)
    $$

    If you need to do Bayesian model selection (e.g., decide between models with different parameters), you unfortunately need to compute this quantity. But if you only _think_ you need the FML, beware!

So how do we make use of all of this, in practice?

Let's return to our example of fitting a line to data with the same data as above. In some sense, we are almost done once we write down an expression for the posterior pdf. If we ignore the FML, this amounts to multiplying a likelihood by a prior pdf. Well, we've already done the most important part: we already wrote down the likelihood function! This is often the hardest part and what we spend the most time doing as scientists (well, assuming you're not building the instrument to observe the data!). We now need to define a prior pdf over the model parameters. Here we have some flexibility. Two possibilities you can always consider:

1. A completely uninformative prior, based on dimensionality, symmetry, or entropy arguments (sometimes, this will mean using a _flat prior_ or _uniform prior_)
2. An empirical prior, based on previous _independent data_ that constrains this model (e.g., a previous measurement of the model parameters from an earlier dataset)

For simplicity, we're going to assume a flat prior over both slope and intercept. Note that for this problem, this is [_not_ an uninformative prior](http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4-bayesian-in-python/). For now, we'll assume that the data are informative enough that the small bias we introduce by using this prior is negligible. Let's now define the functions we'll need, and recall that

$$
\ln\mathcal{L} = -\frac{1}{2}\left[N\,\ln(2\pi)
    + \ln|\boldsymbol{\Sigma}|
    + \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right)^\mathsf{T} \,
      \boldsymbol{\Sigma}^{-1} \,
      \left(\boldsymbol{y} - \boldsymbol{X}\,\boldsymbol{\theta}\right)
\right]
$$

### Exercise 4:

Implement the log-prior method (`ln_prior`) on the model class below.

#### Solution:

```python
class StraightLineModel(object):

    def __init__(self, x, y, y_err):
        """
        We store the data as attributes of the object so we don't have to
        keep passing it in to the methods that compute the probabilities.
        """
        self.x = np.asarray(x)
        self.y = np.asarray(y)
        self.y_err = np.asarray(y_err)

    def ln_likelihood(self, pars):
        """
        We don't need to pass in the data because we can access it from the
        attributes. This is basically the same as the weighted squared
        deviation function, but includes the constant normalizations for the
        Gaussian likelihood.
        """
        N = len(self.y)
        dy = self.y - line_model(pars, self.x)
        ivar = 1 / self.y_err**2 # inverse-variance
        return -0.5 * (N*np.log(2*np.pi) + np.sum(2*np.log(self.y_err)) + np.sum(dy**2 * ivar))

    def ln_prior(self, pars):
        """
        The prior only depends on the parameters, so we don't need to touch
        the data at all. We're going to implement a flat (uniform) prior
        over the ranges:
            a : [0, 100]
            b : [-50, 50]

        """

        a, b = pars # unpack parameters
        ln_prior_val = 0. # we'll add to this

        if a < 0 or a > 100.:
            return -np.inf
        else:
            ln_prior_val += np.log(1E-2) # normalization, log(1/100)

        if b < -50 or b > 50.:
            return -np.inf
        else:
            ln_prior_val += np.log(1E-2) # normalization, log(1/100)

        return ln_prior_val

    def ln_posterior(self, pars):
        """
        Up to a normalization constant, the log of the posterior pdf is just
        the sum of the log likelihood plus the log prior.
        """
        lnp = self.ln_prior(pars)
        if np.isinf(lnp): # short-circuit if the prior is infinite (don't bother computing likelihood)
            return lnp

        lnL = self.ln_likelihood(pars)
        lnprob = lnp + lnL

        if np.isnan(lnprob):
            return -np.inf

        return lnprob

    def __call__(self, pars):
        return self.ln_posterior(pars)
```

```python
model = StraightLineModel(x, y, y_err)
```

Now we'll repeat what we did above to map out the value of the log-posterior over a 2D grid of parameter values. Because we used a flat prior, you'll notice it looks identical to the visualization of the `weighted_squared_deviation` -- only the likelihood has any slope to it!

```python
def evaluate_on_grid(func, a_grid, b_grid, args=()):
    a_grid,b_grid = np.meshgrid(a_grid, b_grid)
    ab_grid = np.vstack((a_grid.ravel(), b_grid.ravel())).T

    func_vals = np.zeros(ab_grid.shape[0])
    for j,pars in enumerate(ab_grid):
        func_vals[j] = func(pars, *args)

    return func_vals.reshape(a_grid.shape)
```

```python
fig,axes = plt.subplots(1, 3, figsize=(14,5.1), sharex=True, sharey=True)

# make a 256x256 grid of parameter values centered on the true values
a_grid = np.linspace(a_true-5., a_true+5, 256)
b_grid = np.linspace(b_true-5., b_true+5, 256)

ln_prior_vals = evaluate_on_grid(model.ln_prior, a_grid, b_grid)
ln_like_vals = evaluate_on_grid(model.ln_likelihood, a_grid, b_grid)
ln_post_vals = evaluate_on_grid(model.ln_posterior, a_grid, b_grid)

for i,vals in enumerate([ln_prior_vals, ln_like_vals, ln_post_vals]):
    axes[i].pcolormesh(a_grid, b_grid, vals,
                       cmap='Blues', vmin=vals.max()-1024, vmax=vals.max()) # arbitrary scale

axes[0].set_title('log-prior', fontsize=20)
axes[1].set_title('log-likelihood', fontsize=20)
axes[2].set_title('log-posterior', fontsize=20)

for ax in axes:
    ax.set_xlabel('$a$')

    # plot the truth
    ax.plot(a_true, b_true, marker='o', zorder=10, color='#de2d26')
    ax.axis('tight')

axes[0].set_ylabel('$b$')

fig.tight_layout()
```

### Exercise 5:

Subclass the `StraightLineModel` class and implement a new prior. Replace the flat prior above with an uncorrelated 2D Gaussian centered on $(\mu_a,\mu_b) = (3., 5.5)$ with root-variances $(\sigma_a,\sigma_b) = (0.05, 0.05)$. Compare the 2D grid plot with the flat prior to the one with a Gaussian prior

#### Solution:

```python
class StraightLineModelGaussianPrior(StraightLineModel): # verbose names are a good thing!

    def ln_prior(self, pars):
        a, b = pars # unpack parameters
        ln_prior_val = 0. # we'll add to this

        # prior on a is a Gaussian with mean, stddev = (3, 0.05)
        ln_prior_val += -0.5*(a - 3.)**2/0.05**2 # this is not normalized properly, but that's ok

        # prior on b is a Gaussian with mean, stddev = (5.5, 0.05)
        ln_prior_val += -0.5*(b - 5.5)**2/0.05**2 # this is not normalized properly, but that's ok

        return ln_prior_val
```

```python
model_Gprior = StraightLineModelGaussianPrior(x, y, y_err)
```

```python
fig,axes = plt.subplots(1, 3, figsize=(14,5.1), sharex=True, sharey=True)

ln_prior_vals2 = evaluate_on_grid(model_Gprior.ln_prior, a_grid, b_grid)
ln_like_vals2 = evaluate_on_grid(model_Gprior.ln_likelihood, a_grid, b_grid)
ln_post_vals2 = evaluate_on_grid(model_Gprior.ln_posterior, a_grid, b_grid)

for i,vals in enumerate([ln_prior_vals2, ln_like_vals2, ln_post_vals2]):
    axes[i].pcolormesh(a_grid, b_grid, vals,
                       cmap='Blues', vmin=vals.max()-1024, vmax=vals.max()) # arbitrary scale

axes[0].set_title('log-prior', fontsize=20)
axes[1].set_title('log-likelihood', fontsize=20)
axes[2].set_title('log-posterior', fontsize=20)

for ax in axes:
    ax.set_xlabel('$a$')

    # plot the truth
    ax.plot(a_true, b_true, marker='o', zorder=10, color='#de2d26')
    ax.axis('tight')

axes[0].set_ylabel('$b$')

fig.tight_layout()
```

---

Now what do we do? The answer depends a bit on your intentions. If you'd like to propagate the posterior pdf (as in, pass on to other scientists to use your results), what do you do if the posterior pdf isn't analytic? And what numbers do you put in your abstract? One option is to draw samples from your posterior pdf and compute summary statistics (e.g., median and quantils) using the samples. That's the approach we're going to take.

## MCMC

One of the most common and powerful class of methods people use for generating these samples is Markov Chain Monte Carlo (MCMC), but there are other options (e.g., brute-force or monte carlo rejection sampling). MCMC methods are useful because they scale reasonably to higher dimensions (well, at least better than brute-force). A disadvantage to these methods comes from the "Markov Chain" part of the name: there is always some correlation between nearby steps in a chain of samples, so you have to compute second-order statistics on the samples to try to verify whether your samples are truly random or fair samples from the target distribution (your posterior pdf).

The simplest MCMC algorithm is known as Metropolis-Hastings. I'm not going to explain it in detail, but in pseudocode, it is:

- Start from some position in parameter space, $\theta_0$ with posterior probability $\pi_0$
- Iterate from 1 to $N_{\rm steps}$:
    - Sample an offset from $\delta\theta_0$ from some proposal distribution
    - Compute a new parameter value using this offset, $\theta_{\rm new} = \theta_0 + \delta\theta_0$
    - Evaluate the posterior probability at the new new parameter vector, $\pi_{\rm new}$
    - Sample a uniform random number, $r \sim \mathcal{U}(0,1)$
    - if $\pi_{\rm new}/\pi_0 > 1$ or $\pi_{\rm new}/\pi_0 > r$:
        - store $\theta_{\rm new}$
        - replace $\theta_0,\pi_0$ with $\theta_{\rm new},\pi_{\rm new}$
    - else:
        - store $\theta_0$ again

The proposal distribution has to be chosen and tuned by hand. We'll use a spherical / uncorrelated Gaussian distribution with root-variances set by hand:

```python
def sample_proposal(*sigmas):
    return np.random.normal(0., sigmas)

def run_metropolis_hastings(p0, n_steps, model, proposal_sigmas):
    """
    Run a Metropolis-Hastings MCMC sampler to generate samples from the input
    log-posterior function, starting from some initial parameter vector.

    Parameters
    ----------
    p0 : iterable
        Initial parameter vector.
    n_steps : int
        Number of steps to run the sampler for.
    model : StraightLineModel instance (or subclass)
        A callable object that takes a parameter vector and computes
        the log of the posterior pdf.
    proposal_sigmas : list, array
        A list of standard-deviations passed to the sample_proposal
        function. These are like step sizes in each of the parameters.
    """
    p0 = np.array(p0)
    if len(proposal_sigmas) != len(p0):
        raise ValueError("Proposal distribution should have same shape as parameter vector.")

    # the objects we'll fill and return:
    chain = np.zeros((n_steps, len(p0))) # parameter values at each step
    ln_probs = np.zeros(n_steps) # log-probability values at each step

    # we'll keep track of how many steps we accept to compute the acceptance fraction
    n_accept = 0

    # evaluate the log-posterior at the initial position and store starting position in chain
    ln_probs[0] = model(p0)
    chain[0] = p0

    # loop through the number of steps requested and run MCMC
    for i in range(1,n_steps):
        # proposed new parameters
        step = sample_proposal(*proposal_sigmas)
        new_p = chain[i-1] + step

        # compute log-posterior at new parameter values
        new_ln_prob = model(new_p)

        # log of the ratio of the new log-posterior to the previous log-posterior value
        ln_prob_ratio = new_ln_prob - ln_probs[i-1]

        if (ln_prob_ratio > 0) or (ln_prob_ratio > np.log(np.random.uniform())):
            chain[i] = new_p
            ln_probs[i] = new_ln_prob
            n_accept += 1

        else:
            chain[i] = chain[i-1]
            ln_probs[i] = ln_probs[i-1]

    acc_frac = n_accept / n_steps
    return chain, ln_probs, acc_frac
```

Now we'll run the sampler! Let's start from some arbitrary position allowed by our prior.

### Exercise 6:

Choose a starting position, values for `a` and `b` to start the MCMC from. In general, a good way to do this is to sample from the prior pdf. Generate values for `a` and `b` by sampling from a uniform distribution over the domain we defined above. Then, run the MCMC sampler from this initial position for 8192 steps. Play around with ("tune" as they say) the `proposal_sigmas` until you get an acceptance fraction around ~40%.

#### Solution:

```python
p0 = [6.,6.]
chain,_,acc_frac = run_metropolis_hastings(p0, n_steps=8192, model=model,
                                           proposal_sigmas=[0.058,0.05])
print("Acceptance fraction: {:.1%}".format(acc_frac))
```

    Acceptance fraction: 40.1%

Let's look at the chain returned, the parameter value positions throughout the sampler run:

```python
fig,ax = plt.subplots(1, 1, figsize=(5,5))

ax.pcolormesh(a_grid, b_grid, ln_post_vals,
              cmap='Blues', vmin=ln_post_vals.max()-128, vmax=ln_post_vals.max()) # arbitrary scale
ax.axis('tight')

fig.tight_layout()

ax.plot(a_true, b_true, marker='o', zorder=10, color='#de2d26')
ax.plot(chain[:512,0], chain[:512,1], marker='', color='k', linewidth=1.)

ax.set_xlabel('$a$')
ax.set_ylabel('$b$')
```

We can also look at the individual parameter traces, i.e. the 1D functions of parameter value vs. step number for each parameter separately:

```python
fig,axes = plt.subplots(len(p0), 1, figsize=(5,7), sharex=True)

for i in range(len(p0)):
    axes[i].plot(chain[:,i], marker='', drawstyle='steps')

axes[0].axhline(a_true, color='r', label='true')
axes[0].legend(loc='best')
axes[0].set_ylabel('$a$')

axes[1].axhline(b_true, color='r')
axes[1].set_ylabel('$b$')

fig.tight_layout()
```

From these trace plots, we can see by eye that it takes the sampler about a few hundred steps to converge. When we look at the samples returned or when we compute our summary statistics, we don't want to include these parameter values! In addition, there is likely some correlation between nearby steps. We can attempt to remove some of the correlated steps by _thinning_ the chain, i.e. by downsampling. We can do both simultaneously using Python indexing tricks. Certainly by step 2000 the chains look converged, so from there on we'll keep only every 8th step:

```python
good_samples = chain[2000::8]
good_samples.shape
```

    (774, 2)

We're left with 774 samples; we hope these are approximately uncorrelated, converged samples from the posterior pdf (there are other ways we can check, but these are out of scope for this workshop). Now you have to choose what summary statistics to report. You have some options, but a reasonable choice is to report the median, 16th, and 84th percentiles:

```python
low,med,hi = np.percentile(good_samples, [16, 50, 84], axis=0)
upper, lower = hi-med, med-low

disp_str = ""
for i,name in enumerate(['a', 'b']):
    fmt_str = '{name}={val:.2f}^{{+{plus:.2f}}}_{{-{minus:.2f}}}'
    disp_str += fmt_str.format(name=name, val=med[i], plus=upper[i], minus=lower[i])
    disp_str += r'\quad '

from IPython import display
disp_str = "${}$".format(disp_str)
display.Latex(data=disp_str)
```

$a=1.24^{+0.06}_{-0.06}\quad b=4.51^{+0.06}_{-0.07}\quad $

Recall that the true values are:

```python
a_true, b_true
```

    (1.255, 4.507)

We've now done this problem the Bayesian way as well! Now, instead of drawing the "best-fit" line over the data, we can take a handful of samples and plot a line for each of the samples, as a way to visualize the uncertainty we have in the model parameters:

```python
plt.figure(figsize=(6,5))
plt.errorbar(x, y, y_err, linestyle='none', marker='o', ecolor='#666666')

x_grid = np.linspace(x.min()-0.1, x.max()+0.1, 128)
for pars in good_samples[:128]: # only plot 128 samples
    plt.plot(x_grid, line_model(pars, x_grid),
             marker='', linestyle='-', color='#3182bd', alpha=0.1, zorder=-10)

plt.xlabel('$x$')
plt.ylabel('$y$')
plt.tight_layout()
```

Or, we can plot the samples using a _corner plot_ to visualize the structure of the 2D and 1D (marginal) posteriors:

```python
# uncomment and run this line if the import fails:
# !source activate statsseminar; pip install corner
import corner
```

```python
fig = corner.corner(chain[2000:], bins=32, labels=['$a$', '$b$'], truths=[a_true, b_true])
```

---

## Finally, the problem you came here for: fitting a straight line to data with intrinsic scatter

We made it! We're now ready to do the problem we set out to do. In the initial model, we assumed that we knew the uncertainties in our measurements exactly and that the data were drawn from a one-dimensional line. We're now going to relax that assumption and assume that either (a) the data uncertainties have been underestimated or (b) there is intrinsic scatter in the true model (in the absence of other information, these two ideas are degenerate). Let's first generate some data. We'll assume the latter of the two ideas, and we'll further assume that the model line is convolved with an additional Gaussian in the $y$ direction, with the new parameter being the intrinsic width of the relation expressed as a variance $V$:

```python
V_true = 0.5**2
n_data = 42

# we'll keep the same parameters for the line as we used above
```

```python
x = rnd.uniform(0, 2., n_data)
x.sort() # sort the values in place

y = a_true*x + b_true

# Heteroscedastic Gaussian uncertainties only in y direction
y_err = rnd.uniform(0.1, 0.2, size=n_data) # randomly generate uncertainty for each datum

# add Gaussian intrinsic width
y = rnd.normal(y, np.sqrt(y_err**2 + V_true)) # re-sample y data with noise and intrinsic scatter
```

```python
plt.errorbar(x, y, y_err, linestyle='none', marker='o', ecolor='#666666')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.tight_layout()
```

Let's first naively fit the data assuming no intrinsic scatter using least-squares:

```python
X = np.vander(x, N=2, increasing=True)
Cov = np.diag(y_err**2)
Cinv = np.linalg.inv(Cov)
```

```python
best_pars = np.linalg.inv(X.T @ Cinv @ X) @ (X.T @ Cinv @ y)
pars_Cov = np.linalg.inv(X.T @ Cinv @ X)
```

```python
plt.errorbar(x, y, y_err, linestyle='none', marker='o', ecolor='#666666')

x_grid = np.linspace(x.min()-0.1, x.max()+0.1, 128)
plt.plot(x_grid, line_model(best_pars[::-1], x_grid), marker='', linestyle='-', label='best-fit line')
plt.plot(x_grid, line_model([a_true, b_true], x_grid), marker='', linestyle='-', label='true line')

plt.xlabel('$x$')
plt.ylabel('$y$')

plt.legend(loc='best')
plt.tight_layout()
```

The covariance matrix for the parameters is:

```python
pars_Cov
```

    array([[ 0.00199108, -0.0014514 ],
           [-0.0014514 ,  0.00142195]])

We clearly get a biased result and yet _very_ precise measurements of the parameters when we don't take in to account the intrinsic scatter. What we need to do now is modify out model to include the scatter as a free parameter. Unfortunately, it enters the model non-linearly so there is no solution using linear algebra or least-squares. Instead, we just write a new likelihood function and optimize it numerically. One choice we'll make is to use the parameter $\ln{V}$ instead of $V$ for reasons I'll explain later. To implement the new model, we'll subclass our `StraightLineModel` class and define new likelihood and prior functions.

### Exercise 7:

Subclass the `StraightLineModel` class and implement new prior and likelihood functions (`ln_prior` and `ln_likelihood`). The our model will now have 3 parameters: `a`, `b`, and `lnV` the log of the intrinsic scatter variance. Use flat priors on all of these parameters. In fact, we'll be even lazier and forget the constant normalization terms: if a parameter vector is within the ranges below, return 0. (log(1.)) otherwise return -infinity:

#### Solution:

```python
class StraightLineIntrinsicScatterModel(StraightLineModel):

    def ln_prior(self, pars):
        """ The prior only depends on the parameters """

        a, b, lnV = pars

        # flat priors on a, b, lnV
        if a < -10 or a > 10 or b < -100. or b > 100. or lnV < -10. or lnV > 10.:
            return -np.inf

        # this is only valid up to a numerical constant
        return 0.

    def ln_likelihood(self, pars):
        """ The likelihood function evaluation requires a particular set of model parameters and the data """
        a,b,lnV = pars
        V = np.exp(lnV)

        N = len(y)
        dy = y - line_model([a,b], self.x)
        ivar = 1 / (self.y_err**2 + V) # inverse-variance now includes intrinsic scatter
        return -0.5 * (N*np.log(2*np.pi) - np.sum(np.log(ivar)) + np.sum(dy**2 * ivar))
```

```python
scatter_model = StraightLineIntrinsicScatterModel(x, y, y_err)
```

```python
x0 = [5., 5., 0.] # starting guess for the optimizer

# we have to minimize the negative log-likelihood to maximize the likelihood
result_ml_scatter = minimize(lambda *args: -scatter_model.ln_likelihood(*args),
                             x0=x0, method='BFGS')
result_ml_scatter
```

          fun: 36.77010525530865
     hess_inv: array([[ 0.02207535, -0.02292126,  0.00034518],
           [-0.02292126,  0.0317851 , -0.00038581],
           [ 0.00034518, -0.00038581,  0.05651076]])
          jac: array([0., 0., 0.])
      message: 'Optimization terminated successfully.'
         nfev: 130
          nit: 18
         njev: 26
       status: 0
      success: True
            x: array([ 1.10279177,  4.74635002, -1.17285508])

```python
plt.errorbar(x, y, y_err, linestyle='none', marker='o', ecolor='#666666')

x_grid = np.linspace(x.min()-0.1, x.max()+0.1, 128)
plt.plot(x_grid, line_model(result_ml_scatter.x[:2], x_grid), marker='', linestyle='-', label='best-fit line')
plt.plot(x_grid, line_model([a_true, b_true], x_grid), marker='', linestyle='-', label='true line')

plt.xlabel('$x$')
plt.ylabel('$y$')

plt.legend(loc='best')
plt.tight_layout()
```

```python
V_true, np.exp(result_ml_scatter.x[2])
```

    (0.25, 0.3094820831577057)

It looks like the maximum likelihood estimate is a little bit better, and we get a reasonable measurement of the intrinsic scatter, but none of this gives us a handle on the uncertainty. How do we quantify the uncertainty in the now 3 parameters? We'll just run MCMC.

### Exercise 8:

To quantify our uncertainty in the parameters, we'll run MCMC using the new model. Run MCMC for 65536 steps and visualize the resulting chain. Make sure the acceptance fraction is between ~25-50%.

#### Solution:

```python
p0 = [6., 6., -1.]
chain,_,acc_frac = run_metropolis_hastings(p0, n_steps=2**16, model=scatter_model,
                                           proposal_sigmas=[0.15,0.15,0.2])
acc_frac
```

    0.3381500244140625

```python
fig,axes = plt.subplots(len(p0), 1, figsize=(5,7), sharex=True)

for i in range(len(p0)):
    axes[i].plot(chain[:,i], marker='', drawstyle='steps')

axes[0].axhline(a_true, color='r', label='true')
axes[0].legend(loc='best')
axes[0].set_ylabel('$a$')

axes[1].axhline(b_true, color='r')
axes[1].set_ylabel('$b$')

axes[2].axhline(np.log(V_true), color='r')
axes[2].set_ylabel(r'$\ln V$')

fig.tight_layout()
```

```python
fig = corner.corner(chain[2000:], bins=32, labels=['$a$', '$b$', r'$\ln V$'],
                    truths=[a_true, b_true, np.log(V_true)])
```

Now we'll again compute the percentiles for the 1D, marginal distributions:

```python
good_samples = chain[2000::8]
good_samples.shape
```

    (7942, 3)

```python
low,med,hi = np.percentile(good_samples, [16, 50, 84], axis=0)
upper, lower = hi-med, med-low

disp_str = ""
for i,name in enumerate(['a', 'b', r'\ln V']):
    fmt_str = '{name}={val:.2f}^{{+{plus:.2f}}}_{{-{minus:.2f}}}'
    disp_str += fmt_str.format(name=name, val=med[i], plus=upper[i], minus=lower[i])
    disp_str += r'\quad '

disp_str = "${}$".format(disp_str)
display.Latex(data=disp_str)
```

$a=1.11^{+0.15}_{-0.15}\quad b=4.74^{+0.18}_{-0.18}\quad \ln V=-1.11^{+0.25}_{-0.24}\quad $

Compare this to the diagonal elements of the covariance matrix we got from ignoring the intrinsic scatter and doing least-squares fitting:

```python
disp_str = ""
for i,name in zip([1,0], ['a', 'b']):
    fmt_str = r'{name}={val:.2f} \pm {err:.2f}'
    disp_str += fmt_str.format(name=name, val=best_pars[i], err=np.sqrt(pars_Cov[i,i]))
    disp_str += r'\quad '

disp_str = "${}$".format(disp_str)
display.Latex(data=disp_str)
```

$a=1.06 \pm 0.04\quad b=4.81 \pm 0.04\quad $

The parameter uncertainties estimated from the MCMC samples are much larger -- this reflects our uncertainty about the intrinsic scatter of the points. Precision is highly model dependent.


# Redondeo

$\newcommand{\RR}{\mathbb{R}}$
$\newcommand{\FF}{\mathbb{F}}$

Tenemos una representación del conjunto no-numerable $\RR$ a través del conjunto finito $\FF$. ¿Cómo podemos pasar de un número real a un número flotante, si el número real no es exactamente representable (lo cual es el caso para ¡*casi todos* los números reales!)? Extenderemos para este fin los números reales y los flotantes con $\pm \infty$, y hablaremos de los reales extendidos $\RR^* := \RR \cup \{-\infty, +\infty \}$  y $\FF^* := \FF \cup \{-\infty, +\infty \}$.

Un mapeo $\bigcirc: \RR^* \to \FF^*$ es una *operación de redondeo* si

- Para toda $x \in \FF^*$, $\bigcirc(x) = x$.
- Si $x, y \in \RR^*$, y $x \le y$, entonces $\bigcirc(x) \le \bigcirc(y)$.

Es decir, el redondeo deja invariantes los números representables en el sistema de punto flotante, y preserva el orden.

Utilizaremos dos modos de redondeo: $\bigtriangleup$, que redondea para arriba (hacia $+\infty$), y $\bigtriangledown$, que redondea para abajo (hacia $-\infty$). Se definen como sigue:

- $\bigtriangleup \! (x) := \min \{y \in \FF^*: y \ge x \}$
- $\bigtriangledown(x) := \max \{y \in \FF^*: y \le x \}$

Hablamos de $x$ redondeado para arriba y $x$ redondeado para abajo, respectivamente; estos dos modos de redondeo se llaman modos de redondeo *dirigidos*.

Nota que no es evidente cómo implementar estas operaciones en la computadora, ya que ¡*no podemos representar los números reales originales*!

[1] (i) Si tuviéramos un número real positivo $x$ (es decir, con  precisión infinita), ¿cómo podríamos encontrar $\bigtriangleup(x)$ y $\bigtriangledown(x)$?

(ii) Encuentra $\bigtriangleup(0.1)$ y $\bigtriangledown(0.1)$ para aritmética flotante de IEEE. ¿En cuánto difieren? ¿Qué podemos decir sobre el *error de redondeo*?

[2] Haz de nuevo el ejercicio anterior para $x=1.1$ y para $x=10.1$.

[3] ¿Qué pasa con $\bigtriangleup(x)$ y $\bigtriangledown(x)$ si $x \in \FF^*$?

[4] ¿Cuál es la relación entre $\bigtriangleup(-x)$ y $\bigtriangledown(x)$?

Existen otros modos de redondeo además de los redondeos arriba mencionados:

- Redondeo a cero (truncamiento): $\square_z(x) = {\rm sign}(x) \max\{y \in \FF^*: y\leq |x| \}$

- Redondeo al más cercano (*round to nearest*): como su nombre lo indica, se redondea al número de punto flotante más cercano, usando $\bigtriangleup(x)$ o $\bigtriangledown(x)$ según sea el caso. Definiendo $\mu = (\bigtriangleup(x) + \bigtriangledown(x))/2$, entonces:

\begin{equation}
\square_n(x) = \left\{
\begin{array}{1 1}
\bigtriangledown(x), &x\in[\bigtriangledown(x),\mu)\\
\bigtriangleup(x),   &x\in[\mu,\bigtriangleup(x)].\\
\end{array}
\right.
\end{equation}

- Redondeo al más cercano *parejo* (*round to nearest even*): es parecido al modo anterior *logrando* que el redondeo hacia arriba y hacia abajo ocurran con la misma probabilidad. (La sutileza en la definición de $\square_n(x)$ está en que la definición involucra un intervalo cerrado y uno semicerrado.) Este redondeo involucra en la definición la paridad del último dígito de la representación de punto flotante de $\bigtriangleup(x)$ y $\bigtriangledown(x)$. Este modo de redondeo es el modo más común.

## Aritmética de punto flotante

¿Cómo podemos hacer aritmética en el mundo de punto flotante?

[5] Encuentra unos ejemplos de pares de  números $x , y \in \FF$ tal que $x \oplus y \notin \FF$. (Aquí, $\FF$ denota a los flotantes de doble precisión de IEEE, y $\oplus$ es alguna operación aritmética entre $x$ y $y$.)

[6] ¿Qué podemos hacer al respecto?

[7] En los reales tenemos que, si se cumple $x+y = x+y'$, entonces $y = y'$. ¿Se cumple esto entre los números de punto flotante? Si tu respuesta es **no**, da un ejemplo.

[8] Analiza el caso de iterar el mapeo $f:[0,1] \to [0,1]$ dado por $f(x) = 3x \mathrm{\ mod\ } 1$, con la condición inicial $x_0 = \frac{1}{10}$:

1. ¿Qué pasa analíticamente?
2. ¿Qué pasa numéricamente?
3. ¿Qué pasa si consideras una condición inicial $x_0$ arbitraria?

[Nota: $\mathrm{mod\ } 1$ quiere decir que sólo consideramos la parte fraccionaria entre $0$ y $1$ de la respuesta en cada paso.]

## Aplicando redondeo para obtener resultados garantizados

Ya estamos en condiciones para empezar a hacer cálculos útiles.

Siguiendo al libro de Tucker, consideremos la suma infinita

$$S = \sum_{n=1}^\infty \frac{1}{n^2}.$$

Se sabe que $S = \frac{\pi^2}{6}$.

[9] Calcula $S$ numéricamente de manera ingenua.

Para calcular $S$ de forma numérica pero *garantizada*, tenemos dos tareas: debemos lidiar con la suma infinita, y luego garantizar que el resultado realmente contenga el valor verdadero.

[10] Sea la cola de la suma $T_N := \sum_{n=N+1}^\infty \frac{1}{n^2}$. Utiliza un argumento geométrico para mostrar que

$$\int_{N+1}^\infty \frac{1}{x^2} dx < T_N < \int_{N+1}^\infty \frac{1}{(x-1)^2} dx,$$

y así encuentra cotas para $T_N$.

[11] Usa redondeo para abajo y arriba para calcular cotas para la parte inicial $S_N := \sum_{n=1}^N n^{-2}$.

[Para cambiar el modo del redondeo en Julia, usamos

    set_rounding(Float64, RoundUp)

]

[12] Utiliza tus dos últimos resultados para dar cotas *rigurosas* (es decir, garantizadas) para $S$.

Verifica que el valor verdadero sí esté contenido adentro de tus cotas.

[13] Repite el cálculo con `BigFloat` para obtener más precisión.

[En Julia, para cambiar la precisión de los `BigFloat`, usamos

    set_bigfloat_precision(100).

]


# Decision Tree

Classification And Regression Trees (CART for short) is a term introduced by [Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman) to refer to Decision Tree algorithms that can be used for classification or regression predictive modeling problems.

In this lab assignment, you will implement various ways to calculate impurity which is used to split data in constructing the decision trees and apply the Decision Tree algorithm to solve two real-world problems: a classification one and a regression one.

```python
# import packages
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from matplotlib.legend_handler import HandlerLine2D
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV

# make this notebook's output stable across runs
np.random.seed(0)
```

## Gini impurity and Entropy

**Coding Part 1: Implement the functions to calculate Gini impurity and entropy.**

#### Gini impurity

The CART algorithm recursively splits the training set into two subsets using a single feature k and a threshold $t_k$. The best feature and threshold are chosen to produce the purest subsets weighted by their size. **Gini impurity** measures the impurity of the data points in a set and is used to evaluate how good a split is when the CART algorithm searches for the best pair of feature and the threshold.

To compute Gini impurity for a set of items with J classes, suppose $i \in \{1, 2, \dots, J\}$ and let $p_i$ be the fraction of items labeled with class i in the set.
\begin{align}
I(p) = 1 - \sum_{i=1}^J p_i^2
\end{align}

In this exercise, you will implement the function to calculate gini impurity.

```python
def gini_impurity(x):
    """
    TODO: This function calculate the Gini impurity of an array.

    Args:
    x: a numpy ndarray
    """
    gini = 0
    return gini = 0
```

```python
np.testing.assert_equal(0, gini_impurity(np.array([1, 1, 1])))
np.testing.assert_equal(0.5, gini_impurity(np.array([1, 0, 1, 0])))
np.testing.assert_equal(3/4, gini_impurity(np.array(['a', 'b', 'c', 'd'])))
np.testing.assert_almost_equal(2.0/3, gini_impurity(np.array([1, 2, 3, 1, 2, 3])))
```

#### Entropy

Another popular measure of impurity is called **entropy**, which measures the average information content of a message. Entropy is zero when all messages are identical. When it applied to CART, a set's entropy is zero when it contains instances of only one class. Entropy is calculated as follows:
\begin{align}
I(p) = - \sum_{i=1}^J p_i log_2{p_i}
\end{align}

Here, you will implement entropy impurity function.

```python
def entropy(x):
    """
    TODO: This function calculate the entropy of an array.

    Args:
    x: a numpy ndarray
    """
    e = 0

    return e
```

```python
np.testing.assert_equal(0, entropy(np.array([1, 1, 1])))
np.testing.assert_equal(1.0, entropy(np.array([1, 0, 1, 0])))
np.testing.assert_equal(2.0, entropy(np.array(['a', 'b', 'c', 'd'])))
np.testing.assert_almost_equal(1.58496, entropy(np.array([1, 2, 3, 1, 2, 3])), 4)
```

---

## Decision Tree Classifier

**Coding Part 2: In this exercise, we will apply the Decision Tree classifier to classify the Iris flower data.**

### Iris dataset

The Iris data set contains the morphologic variation of Iris flowers of three related species (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each observation (see image below):
- Sepal.Length: sepal length in centimeters.
- Sepal.Width: sepal width in centimeters.
- Petal.Length: petal length in centimeters.
- Petal.Width: petal width in centimeters.

    Iris setosa
    Iris versicolor
    Iris virginica

```python
# load the iris train and test data from CSV files
train = pd.read_csv("iris_train.csv")
test = pd.read_csv("iris_test.csv")

train_x = train.iloc[:,0:4]
train_y = train.iloc[:,4]

test_x = test.iloc[:,0:4]
test_y = test.iloc[:,4]

# print the number of instances in each class
print(train_y.value_counts().sort_index())
print(test_y.value_counts().sort_index())
```

### Train and visualize a simple Decision Tree

```python
# TODO: read the scikit-learn doc on DecisionTreeClassifier and train a Decision Tree with max depth of 2
dtc = None
```

Now let's visualize the decision tree we just trained on the iris dataset and see how it makes predictions.

```python
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

dot_data = StringIO()
feature_names = train_x.columns
class_names = train_y.unique()
class_names.sort()
export_graphviz(dtc, out_file=dot_data, feature_names=feature_names, class_names=class_names, filled=True, rounded=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
```

Decision trees are easy to inteprete and is often referred to as *whitebox* machine learning algorithm. Let's see how this decision tree represented above makes predictions. Suppose you find an iris flower and want to classify it into setosa, versicolor or virginica. You start at the root node (the very top node in the tree). In this node, we check if the flower's patel length is smaller than or equal to 2.35 cm. If it is, we move to the left child and predict setosa to be its class. Otherwise, we move to the right child node. Then similarly we check if the petal length is smaller than or equal to 4.95 cm. If it is, we move to its left child node and predict versicolor to be its class. Otherwise, we move to its right child and predict virginica to be its class.

### Prediction with Decision tree

With this simple decision tree above, we can apply it to make predictions on the test dataset and evaluate its performance.

```python
# TODO: use the trained decision tree model to make predictions on the test data and evaluate the model performance
#       using evaluate(y, z) above.

print("model accuracy: {}".format(accuracy_score(test_y, test_z)))
print("model confusion matrix:\n {}".format(confusion_matrix(test_y, test_z, labels=['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'])))
```

### Hyper-parameters

Hyper-parameter controls the complexity of the decision tree model. For example, the deeper the tree is, the more complex patterns the model will be able to capture. In this exercise, we train the decision trees with increasing number of maximum depth and plot its performance. We should see the accuracy of the training data increase as the tree grows deeper, but the accuracy on the test data might not as the model will eventually start to overfit and does not generalize well on the unseen test data.

```python
max_depths = np.linspace(1, 8, 8, endpoint=True)
train_results = []
test_results = []

for max_depth in max_depths:
    # TODO: train the decision tree model with various max_depth, make predictions and evaluate on both train and test data.
    dt = DecisionTreeClassifier(max_depth=max_depth)
    dt.fit(train_x, train_y)

    train_z = dt.predict(train_x)
    train_accuracy = accuracy_score(train_y, train_z)
    train_results.append(train_accuracy)

    test_z = dt.predict(test_x)
    test_accuracy = accuracy_score(test_y, test_z)
    test_results.append(test_accuracy)

# plot the accuracy on train and test data
line1, = plt.plot(max_depths, train_results, 'b', label="Train Accuracy")
line2, = plt.plot(max_depths, test_results, 'r', label="Test Accuracy")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("Accuracy")
plt.xlabel("Max Depth")
plt.show()
```

### Fine-tune the decision tree classifier
Decision tree is a very powerful model with very few assumptions about the incoming training data (unlike linear models, which assume the data linear), however, it is more likely to overfit the data and won't generalize well to unseen data. To void overfitting, we need to restrict the decision tree's freedom during training via regularization (e.g. max_depth, min_sample_split, max_leaf_nodes and etc.).

To fine-tune the model and combat overfitting, use grid search with cross-validation (with the help of the GridSearchCV class) to find the best hyper-parameter settings for the DecisionTreeClassifier. In particular, we would like to fine-tune the following hyper-parameters:
- **criteria**: this defines how we measure the quality of a split. we can choose either "gini" for the Gini impurity or "entropy" for the information gain.
- **max_depth**: the maximum depth of the tree. This indicates how deep the tree can be. The deeper the tree, the more splits it has and it captures more information about the data. But meanwhile, deeper trees are more likely to overfit the data. For this practice, we will choose from {1, 2, 3} given there are only 4 features in the iris dataset.
- **min_samples_split**: This value represents the minimum number of samples required to split an internal node. The smaller this value is, the deeper the tree will grow, thus more likely to overfit. On the other hand, if the value is really large (the size of the training data in the extreme case), the tree will be very shallow and could suffer from underfit. In this practice, we choose from {0.05, 0.1, 0.2}

```python
# TODO: fine-tune the model, use grid search with cross-validation.
parameters = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [1, 2, 4, 8],
    'min_samples_split': [0.01, 0.05, 0.1, 0.2, 0.4, 0.8]
}

dt = DecisionTreeClassifier()
grid = # TODO
grid.fit(train_x, train_y)

# summarize the results of the grid search
print("The best score is {}".format(grid.best_score_))
print("The best hyper parameter setting is {}".format(grid.best_params_))
```

### Prediction and Evaluation

Now we have a fine-tuned decision tree classifier based on the training data, let's apply this model to make predictions on the test data and evaluate its performance.

```python
grid.predict(test_x)

print("model accuracy: {}".format(accuracy_score(test_y, test_z)))
print("model confusion matrix:\n {}".format(confusion_matrix(test_y, test_z, labels=['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'])))
```

---

## Decision Tree Regressor

**Coding Part 3: In this exercise, we will apply the Decision Tree classifier to predict the California housing prices.**

### California Housing Dataset

The California Housing dataset appeared in a 1997 paper titled Sparse Spatial Autoregressions by Pace, R. Kelley and Ronald Barry, published in the Statistics and Probability Letters journal. They built it using the 1990 California census data. It contains one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).

```python
# Load train and test data from CSV files.
train = pd.read_csv("housing_train.csv")
test = pd.read_csv("housing_test.csv")

train_x = train.iloc[:,0:8]
train_y = train.iloc[:,8]

test_x = test.iloc[:,0:8]
test_y = test.iloc[:,8]
```

```python
# TODO: fine-tune the model, use grid search with cross-validation.
parameters = {
    'max_depth': [4, 8, 16, 32],
    'min_samples_split': [0.001, 0.01, 0.05, 0.1, 0.2],
    'max_features': [2, 4, 8]
}

dtr = # TODO
grid = # TODO
grid.fit(train_x, train_y)

# summarize the results of the grid search
print("The best score is {}".format(grid.best_score_))
print("The best hyper parameter setting is {}".format(grid.best_params_))
```

```python
# Use the fine-tuned model to make predictions on the test data
test_z = grid.predict(test_x)

print("model Root Mean Squared Error: {}".format(np.sqrt(mean_squared_error(test_y, test_z))))
print("model Mean Absolute Error: {}".format(mean_absolute_error(test_y, test_z)))
```

### End of ML 310 Lab Assignment 2
---


# The Harmonic Oscillator Strikes Back

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week we continue our adventures with the harmonic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2} m \omega_0^2 {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

If the equilibrium value of the harmonic oscillator is not zero, then

$$ F=- \omega_0^2 m (x-x_{eq}), \text{ }  V(x) = {1 \over 2} m \omega_0^2 (x-x_{eq})^2$$

## 1. Harmonic oscillator from last time (with some better defined conditions)

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= -m \omega_0^2 (x-x_{eq}) $$

$$ a = - \omega_0^2 (x-x_{eq}) $$

$$ x(t)'' = - \omega_0^2 (x-x_{eq}) $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

This is already solved to remind you how we found these values

```python
import sympy as sym
sym.init_printing()
```

**Note** that this time we define some of the properties of the symbols. Namely, that the frequency is always positive and real and that the positions are always real

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
x(t),omega0
```

```python
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

```python
sol,sol.args[0],sol.args[1]
```

**Note** this time we define the initial positions and velocities as real

```python
x0,v0=sym.symbols("x_0,v_0",real=True)
ics=[sym.Eq(sol.args[1].subs(t, 0), x0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

### 1.1 Equation of motion for $x(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

### 1.2 Equation of motion for $p(t)$

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
```

## 2. Time average values for a harmonic oscillator

If we want to understand the average value of a time dependent observable, we need to solve the following integral

$${\left\langle {A(t)} \right\rangle}_t  = \begin{array}{*{20}{c}}
{\lim }\\
{\tau  \to 0}
\end{array}\frac{1}{\tau }\int\limits_0^\tau  {A(t)dt} $$

### 2.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
xfunc=full_sol.args[1]
xavet=(xfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
xavet
```

The computer does not always make the best choices the first time. If you treat each sum individually this is not a hard limit to do by hand. The computer is not smart. We can help it by inseting an `expand()` function in the statement

```python
xavet=(xfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
xavet
```

### 2.2 Excercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

```python
import sympy as sym
sym.init_printing()

m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
p=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))

tau=sym.symbols("tau",nonnegative=True,real=True)
pfunc=p.args[1]
pavet=(pfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
pavet

pavet=(pfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
pavet
```

### 2.3 Exercise: Calculate the average kinetic energy of a harmonic oscillator

```python
import sympy as sym
sym.init_printing()

m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
p=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))

tau=sym.symbols("tau",nonnegative=True,real=True)
pfunc=p.args[1]

ke=pfunc**2/(2*m)
ke

keavet=(ke.integrate((t,0,tau))/tau).limit(tau,sym.oo)
keavet

keavet=(ke.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
keavet
```

## 3.  Ensemble (Thermodynamic) Average values for a harmonic oscillator

If we want to understand the thermodynamics ensemble average value of an observable, we need to solve the following integral.

$${\left\langle {A(t)} \right\rangle}_{T}  = \frac{\int{A e^{-\beta H}dqdp}}{\int{e^{-\beta H}dqdp} } $$

You can think of this as a Temperature average instead of a time average.

Here $\beta=\frac{1}{k_B T}$ and the classical Hamiltonian, $H$ is

$$ H = \frac{p^2}{2 m} + V(q)$$

**Note** that the factors of $1/h$ found in the classical partition function cancel out when calculating average values

### 3.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

For a harmonic oscillator with equilibrium value $x_{eq}$, the Hamiltonian is
$$ H = \frac{p^2}{2 m} + \frac{1}{2} m \omega_0 (x-x_{eq})^2 $$

First we will calculate the partition function $\int{e^{-\beta H}dqdp}$

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z
```

Then we can calculate the numerator $\int{A e^{-\beta H}dqdp}$

```python
numx=sym.integrate(xT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numx
```

And now the average value

```python
xaveT=numx/z
xaveT
```

### 3.2 Exercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

After calculating the value, explain why you think you got this number

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z

nump=sym.integrate(pT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
nump

paveT=nump/z
paveT
```

### 3.3 Exercise: Calculate the average kinetic energy

The answer you get here is a well known result related to the energy equipartition theorem

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z

numke=sym.integrate(pT**2/(2*m)*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numke

keaveT=nump/z
keaveT
```

# Back to the lecture

## 4. Exercise Verlet integrators

In this exercise we will write a routine to solve for the equations of motion for a hamonic oscillator.

Plot the positions and momenta (seprate plots) of the harmonic oscillator as a functions of time.

Calculaate trajectories using the following methods:
1. Exact solution
2. Simple taylor series expansion
3. Predictor-corrector method
4. Verlet algorithm
5. Leapfrog algorithm
6. Velocity Verlet algorithm

```python
#exact solution

import sympy as sym
sym.init_printing()

#defining variables and variable conditions of differential
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)

#solving ordinary differential eq
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
solve=sym.dsolve(dfeq)

#defining variables and variable conditions of pos and mom
x0,v0=sym.symbols("x_0,v_0",real=True)
#initial conditions of pos and mom
inicon=[sym.Eq(solve.args[1].subs(t, 0), x0),
        sym.Eq(solve.args[1].diff(t).subs(t, 0), v0)]
#solving for constants
iniconsolved=sym.solve(inicon)

#eq of pos using solved constants
pos=solve.subs(iniconsolved[0])
pos
#eq of vel using solved constants
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
v=sym.Function("v")
vel=sym.Eq(v(t),solve.args[1].subs(iniconsolved[0]).diff(t))

#plotting pos and mom by choosing initial conditions
posplot=pos.subs({x0:5,xeq:0,v0:5,omega0:1})
velplot=vel.subs({x0:5,xeq:0,v0:5,omega0:1})
sym.plot(posplot.rhs,(t,0,10)),sym.plot(velplot.rhs,(t,0,10))

pos,vel
```

```python
#simple taylor series expansion
#xt=xt0+(pt0/m)*deltat
#pt=pt0+(mat0)*deltat

import sympy as sym
sym.init_printing()

#naming variables and setting recursion
xt0=0
t=0.5
vt0=1
pos=[]
for i in range(0,100):
    vt=vt0+1/2*t**2*-xt0
    vt0=vt
    xt=xt0+vt0*t
    xt0=xt
    pos.append(xt)

pt0=0
t=0.5
at0=1
vel=[]
for i in range(0,100):
    at=at0+1/2*t**2-vt0
    at0=at
    vt=vt0+at0*t
    vt0=vt
    vel.append(vt)

#plotting
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
#predictor corrector method
#don't
```

```python
#verlet algorithm
#r(t+dt)=2r(t)-r(t-dt)+dt**2*a
#v(t)=(r(t+dt)-r(t-dt))/2dt

import sympy as sym
sym.init_printing()

#naming variables and setting recursion
xt1=1
xt2=0
t=0.5
pos=[]

#t0=t+dt
#t1=t
#t2=t-dt
for i in range(0,100):
    xt0=2*xt1-xt2+t**2*-xt1
    xt2=xt1
    xt1=xt0
    pos.append(xt2)

xt0=0
xt2=1
t=2
vel=[]
for i in range(0,100):
    vt=(xt2-xt0)/2*t
    xt0=xt2
    xt2=vt
    vel.append(vt)

#plotting
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
#leapfrog
#r(t+dt)=r(t)+dtv(t+1/2dt)
#v(t+1/2dt)=v(t-1/2dt)+dta(t)
#v(t)=1/2(v(t+1/2dt)+v(t-1/2dt))

import sympy as sym
sym.init_printing()

#naming variables and setting recursion
xt0=1
vtminushalfdt=0
t=0.2
pos=[]
for i in range(0,100):
    vtplushalfdt=vtminushalfdt+t*-xt0
    vtminushalfdt=vtplushalfdt
    xt1=xt0+t*vtplushalfdt
    xt0=xt1
    pos.append(xt1)

xt0=1
t=0.3
vtminushalfdt=1
vtplushalfdt=2
vel=[]
for i in range(0,100):
    vt=1/2*(vtplushalfdt+vtminushalfdt)
    vtminushalfdt=vt
    vtplushalfdt=vtminushalfdt+t*-xt0
    vtminushalfdt=vtplushalfdt
    xt1=xt0+t*vtplushalfdt
    xt0=xt1
    vel.append(vt)

#plotting
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
#velocity verlet
#r(t+dt)=r(t)+dtv(t)+1/2dt^2a(t)
#v(t+dt)=v(t)+1/2dt(a(t)+a(t+dt))
#1st step: v(t+1/2dt)=v(t)+1/2dta(t)
#2nd step: v(t+dt)=v(t+1/2dt)+1/2dta(t+dt)

import sympy as sym
sym.init_printing()

#naming variables and setting recursion

dt=0.01
x0=0
v0=1
pos=[]
for i in range(0,1000):
    x1=x0+v0*dt+1/2*-x0*dt**2
    x0=x1
    v1=v0+1/2*(-x1-x0)*dt
    v0=v1
    pos.append(x1)

xt0=1
dt=2
vt0=1
vel=[]
for i in range(0,10):
    vtplushalfdt=vt0+1/2*dt*-xt0
    xt0=vtplushalfdt
    vel.append(vtplushalfdt)

#plotting
import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```


# Messtechnik HS2021 - Tutorial 8

## Aufgabe 1: Fredholmintegral und Tikhonov-Regularisierung
---------
Abstandsmessungen zwischen zwei paramagnetischen Zentren (Elektronenspins mit $S = 1/2$) werden in der EPR-Spektroskopie mittels der DEER-Pulssequenz gemacht. Ein Fredholmintegral erster Ordnung beschreibt das experimentell gemessene Signal

$$ S(t) = \int_0^{\rm t_{max}} K(t,r) P(r) dr $$

wobei $K(t,r)$ das dipolare Kernel ist, welches das Zeitsignal mit der Abstandsverteilung $P(r)$ verbindet. In der Realität, sind das exprimentelle Signal $\mathbf{S}$ und die Abstandsverteilung $\mathbf{P}$ Vektoren der Länge $n$ und $m$. Das dipolare Kernel $\mathrm{K}$ ist folglich eine Matrix mit der Dimension $ m \times n $. Das Zeitsignal wird somit als lineare Matrixoperation beschrieben:

$$ \mathbf{S} = \mathrm{K} \mathbf{P} $$

---------

### 1a)
Importieren Sie das package `deerlab` und laden Sie die experimentellen Zeitachse und Daten, indem Sie die Datei `DEER_signal.npz` mit der Numpyfunktion `load()` verwenden.

```python
import numpy as np
import deerlab as dl
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")

data = np.load('DEER_signal.npz')
t    = data['t']
S    = data['S']
```

### 1b)
Definieren Sie eine linear Distanzachse $\mathrm{r}$ im Bereich $[1.5,10]\, {\rm nm}$ mit gleicher Anzahl Punkte wie $t$ und berechnen Sie mit der Deerlab-Funktion `dipolarkernel()` das passende dipolare Kernel $\mathbf{K}$ für die Zeitachse $\mathbf{t}$ und Distanzachse $\mathbf{r}$.

```python
r = np.linspace(1.5,10,251)
K = dl.dipolarkernel(t,r)
```

### 1c)
Berechnen Sie die Konditionszahl der dipolaren Kernelmatrix $\mathbf{K}$ und kommentieren Sie das Resultat in Bezug auf das weitere Fittingvorgehen.

```python
cond = np.linalg.cond(K)
print(cond)
```

    8.533358561109903e+19

Lösung:
Für ein well-conditioned Matrix ist die Konditionszahl nahe $1$. In diesem Fall kann das Inversionsproblem eines Fredholmintegrals erster Ordnung über die Gleichung

$$ \mathbf{P} = \mathbf{K}^{-1} \mathbf{S} $$
gelöst werden.

Für eine ill-conditioned Matrix ( $\text{cond(} \mathrm{K}) \neq 1$ ) ist die Lösung des Inversionsproblems nicht direkt möglich, sondern nur durch alternative Methoden, wie z.B. Tikhonov-Regularisierung, die die Lösung stabilisieren.

### 1d)
Berechnen Sie die Abstandsverteilung $\mathbf{P}$ durch Inversion der Kernelmatrix anhand der Gleichung:

$$ \mathbf{P} = \mathbf{K}^{-1} \mathbf{S} $$

Stellen Sie das Signal und die erhaltene Abstandsverteilung graphisch dar. Kommentieren Sie das Resultat der Abstandsverteilung und vergleichen Sie es mit der realen Abstandsverteilung ($\mathbf{r}_\text{truth}$, $\mathbf{P}_\text{truth}$), welche Sie auch in `DEER_signal.npz` als Variablen r und P finden.

```python
# Matrixinversion
Kinv = np.linalg.inv(K)
Pinv = S@Kinv

# Normalisierung (siehe Paper: 10.1039/C9CP06111H)
Pinv = Pinv/(sum(Pinv))/(r[1]-r[0])

# Rückwärtsfit Zeitsignal
Sinv = K@Pinv

rtruth = data['r']
Ptruth = data['P']

# Plot
plt.figure(figsize=(16,6))
plt.subplot(121)
plt.plot(t,Sinv,color='dodgerblue')
plt.plot(t,S,'k')
plt.xlabel('t [$\\mu$s]',fontsize=13)
plt.ylabel('S(t)',fontsize=13)
plt.legend(['$S_{inversion}$','$S_{exp}$'],fontsize=13)

plt.subplot(122)
plt.plot(r,Pinv,color='dodgerblue',zorder=1)
plt.fill_between(rtruth,np.zeros_like(Ptruth),Ptruth,color='grey',zorder=2)
plt.xlabel('r [nm]',fontsize=13)
plt.ylabel('P(r)',fontsize=13)
plt.legend(['$P_{inversion}$','$P_{truth}$'],fontsize=13);
plt.ylim((-1,5));

```

### 1e)
Finden Sie nun mit Hilfe der Tikhonov-Regularisierung

$$ \mathbf{P}_\text{opt} = \text{argmin}\left\{ \frac{1}{2} ||\mathbf{K}\mathbf{P} - \mathbf{S}||_2^2 + \frac{\alpha^2}{2} || \mathbf{LP} ||_2^2 \right\} $$

eine optimale Lösung für den Fit des experimentelle Signal, um die richtige Abstandsverteilung herauszufinden.
Der Regularisierungsparameter $\alpha$ wägt die Übereinstimmung der Daten mit dem Fit mit dem Glätte-Penalty

($||\mathbf{LP}||_2^2$) ab.
Je grösser $\alpha$ desto mehr wird das Glättekriterium berücksichtigt und je kleiner $\alpha$ desto grösser die Datenübereinstimmung mit dem Fit.

Verwenden Sie für das Datenfitting und Herausfinden der Abstandsverteilung die Funktion `fitregmodel()` von `deerlab`. Probieren Sie unterschiedliche Regularisierungsparameter $\alpha$ zwischen $10^{-5}$ und $10$ aus und kommentieren Sie den Einfluss auf den Signalfit wie auch auf die resultierene Abstandsverteilung.

*Hinweis*:
Verwenden Sie `help()` um die nötigen Inputs für die `fitregmodel`-Funktion herauszufinden. Verwenden Sie `regparam='tikhonov'` um die Tikhonov-Regularisierung zu wählen und geben Sie der Funktion $\alpha$ über `regparam=` weiter.

```python
# Regularisierungsparameter
alpha = [10**-5,10**-4,10**-3,10**-2,10**-1,10**0,10**1]

# Plotparameter
legtext = ['$\\alpha = 10$','$\\alpha = 1$','$\\alpha = 10^{-1}$','$\\alpha = 10^{-2}$','$\\alpha = 10^{-3}$','$\\alpha = 10^{-4}$','$\\alpha = 10^{-5}$']
colors  = ['indigo','darkorchid','blue','dodgerblue','forestgreen','gold','orange']

plt.figure(figsize=(16,6))
plt.subplot(121)
plt.plot(t,S,'k')
plt.xlabel('t [$\\mu$s]',fontsize=13)
plt.ylabel('S(t)',fontsize=13)

plt.subplot(122)
plt.fill_between(rtruth,np.zeros_like(Ptruth),Ptruth,color='grey')
plt.xlabel('r [nm]',fontsize=13)
plt.ylabel('P(r)',fontsize=13)

for k in range(len(alpha)):

    fit  = dl.fitregmodel(S,K,r,regtype='tikhonov',regparam=alpha[k])
    Pfit = fit.P
    Sfit = K@Pfit

    plt.subplot(121)
    plt.plot(t,Sfit,color=colors[k])

    plt.subplot(122)
    plt.plot(r,Pfit,color=colors[k])
    plt.legend(legtext,fontsize=13)
```

### Zusatz:
Um eine gute Wahl für den Regularisierungsparameter $\alpha$ zu treffen gibt es unterschiedliche Auswahlkriterien wie AIC, BIC, etc.
Hier zeigen wir Ihnen die Unterschiede vom AIC, LR und srGCV Auswahlkriterium

Mehr Informationen zu Auswahlskriterien für die Auswertung von DEER-Messungen in der EPR finden Sie in den folgenden Papers:
- [Journal of Magnetic Resonance 288 (2018) 58–68](https://www.sciencedirect.com/science/article/pii/S1090780718300442?via%3Dihub)
- [Journal of Magnetic Resonance 300 (2019) 28–40](https://www.sciencedirect.com/science/article/pii/S1090780719300084?via%3Dihub)

```python
# Auswahlkriterium für Regularisierungsparameter
regparam = ['aic','srgcv','lr']

# Plotparameter
legtext = ['exp.data','AIC (Akaike Information Criterion)','srGCV (Strong Robust Generalized Cross Validation)','LR (L-curve maximum-curvature method)']
colors  = ['limegreen','dodgerblue','darkorchid']

plt.figure(figsize=(16,6))
plt.subplot(121)
plt.plot(t,S,'k')
plt.xlabel('t [$\\mu$s]',fontsize=13)
plt.ylabel('S(t)',fontsize=13)

plt.subplot(122)
plt.fill_between(rtruth,np.zeros_like(Ptruth),Ptruth,color='grey')
plt.xlabel('r [nm]',fontsize=13)
plt.ylabel('P(r)',fontsize=13)

for k in range(len(regparam)):

    fit  = dl.fitregmodel(S,K,r,regtype='tikhonov',regparam=regparam[k])
    Pfit = fit.P
    Sfit = K@Pfit

    plt.subplot(121)
    plt.plot(t,Sfit,color=colors[k])
    plt.legend(legtext,fontsize=11,loc=1)

    plt.subplot(122)
    plt.plot(r,Pfit,color=colors[k])

```

## Aufgabe 2: Lorentzverteilter stochastischer Prozess
---------

Bei einem stochastischen Prozess ist das Ergebnis einer Messung nicht vorhersehbar. Trotzdem ist es wichtig Informationen über diesen stochastischen Prozess zu gewinnen, indem man charakteristische Grössen wie die Wahrscheinlichkeitsdichtefunktion und die spektrale Leistungsdichte betrachtet. Auch Autokorrelation und Kreuzkorrelation sind wichtige Hilfsmittel um das Signal zu charakterisieren und am Ende besser zu verstehen.

---------

### 2a)
Nehmen Sie an ein stochastischer Prozess $Y$ mit den Werten $y$ sei unkorreliert und lorentzverteilt mit der Wahrscheinlichkeitsdichtefunktion:
$$ q(y) = \frac{\beta}{\pi \left(\beta^2 + y^2 \right) } $$

Generieren Sie aus $N = 10^5$ gleichverteilten Zufallszahlen $x$ (siehe `numpy.random.rand`), mit der Wahrscheinlichkeitdichtefunktion-Eigenschaft

$$ p(x)dx = dx \,\,\,\,\,\,\,\,{\rm wenn} \,\,\,0 < x < 1 ,$$

lorenztverteilte Zufallszahlen $y$.

Stellen Sie die Zufallszahlen $x$ und $y$ in Abhängigkeit von $N$ graphisch dar.

*Hinweis*: Verwenden Sie, um die Zufallsvariabel $y(x)$ aus $p(x)$ und $q(y)$ zu berechnen, die folgende mathematische Formel:
$$ p(x) \frac{dx}{dy} = q(y) $$

Lösung:
Die mathematische Formel vereinfacht sich nach einsetzen von $p(x)$ und $q(y)$ zu:
$\begin{align}
\frac{dx}{dy} &= \frac{\beta}{\pi \left(\beta^2 + y^2 \right) } \\
dx &= \frac{\beta}{\pi \left(\beta^2 + y^2 \right) } dy \\
\int 1\, dx &= \int \frac{\beta}{\pi \left(\beta^2 + y^2 \right) } dy \\
\end{align}$

```python
import sympy as sp
from sympy.solvers.solvers import solve
from IPython.display import display, Math

beta = sp.Symbol('\\beta')
pi   = sp.Symbol('\\pi')
x = sp.Symbol('x')
y = sp.Symbol('y')

left  = sp.Integral(1,x)
right = sp.Integral(beta/(pi*(beta**2 + y**2)),y)
right = sp.simplify(right)
yofx  = solve(left-right,y)
display(Math(str(yofx)))
```

$\displaystyle [-\beta/tan(\pi*x)]$

```python
import numpy as np
import math as m
import matplotlib.pyplot as plt

# Lorentzverteilung Parameter
N = 10**5
β = 1

# Generiere gleichverteilte Random Zahlen zwischen 0-1
uniform = np.random.rand(N)

# Generiere lorentzverteilte Random Zahlen aus x (=gleichverteilt)
lorentz = -β/np.tan(m.pi*uniform)

plt.figure(figsize=(16,6))
plt.subplot(121)
plt.plot(np.log10(uniform))
plt.xlabel('N',fontsize=13)
plt.ylabel('log10(Wert)',fontsize=13)
plt.title('Gleichverteilte Wahrscheinlichkeitsdichte',fontsize=13)
plt.subplot(122)
plt.plot(np.log10(np.abs(lorentz)))
plt.xlabel('N',fontsize=13)
plt.ylabel('log10(Wert)',fontsize=13)
plt.title('Lorentzverteilte Wahrscheinlichkeitsdichte',fontsize=13);
```

### 2b)
Zeigen Sie anhand von Histogrammen (siehe `matlibplot.pyplot.hist()`), dass die Verteilung der Zufallszahlen $x$ und $y$ wirklich den Wahrscheinlichkeitsdichtefunktionen $p(x)$ und $q(y)$ folgt.

*Hinweis*: Die gleichverteilte Wahrscheinlichkeitsdichtefunktion $p(x)$ kann beschrieben werden als:
$$ p(x) = \frac{1}{B-A} \,\,\,\,\,\,\,\,{\rm wenn} \,\,\,A < x < B .$$

```python
# Definiere Werte für Histogram-Bins
binsuni = np.linspace(0,1,500)
binslor = np.linspace(-50,50,500)

# Generiere die theoretischen Wahrscheinlichkeitsdichtefunktionen im bin-Bereich
pdfuni  = np.ones_like(binsuni)
pdflor  = β/(m.pi*(β**2 + binslor**2))

plt.figure(figsize=(16,6))
plt.subplot(121)
plt.hist(uniform,binsuni,density=True)
plt.plot(binsuni,pdfuni,'r--')
plt.xlabel('Wert',fontsize=13)
plt.ylabel('Wahrscheinlichkeitsdichte',fontsize=13)
plt.title('Gleichverteilte Wahrscheinlichkeitsdichte',fontsize=13)
plt.subplot(122)
plt.hist(lorentz,binslor,density=True)
plt.plot(binslor,pdflor,'r--')
plt.xlabel('Wert',fontsize=13)
plt.ylabel('Wahrscheinlichkeitsdichte',fontsize=13)
plt.title('Lorentzverteilte Wahrscheinlichkeitsdichte',fontsize=13);
```

### 2c)
Die Autokorrelation kann als Faltung eines Signals mit sich selbst betrachtet werden. Schreiben Sie eine Funktion, die über das Faltungstheorem die Autokorrelation eines Zeitsignals berechnet.

```python
from numpy.fft import fft,ifft,fftshift,fftfreq

def autocorr_fcn(x):

    X     = fft(x)
    Xstar = np.ndarray.conjugate(X)
    Sxx   = X*Xstar
    Rxx   = ifft(Sxx)

    return Rxx,Sxx

```

### 2d)
Importieren Sie die Zeitachse und das zu analysierende Signal aus dem File `timesignal.npz`, indem Sie die `numpy.load()` verwenden.
Probieren Sie mit Hilfe der Fourier Transformation und der Autokorrelationsfunktion die 2 Frequenzen des Signals herauszufinden.

```python
# Generieren timesignal.npz
frq1 = 2.25
frq2 = 3.34
sig  = 0.13
frac = 0.05
T2   = 8
t    = np.linspace(0,10,2048)
dt   = t[1]-t[0]

np.random.seed(2)
out = frac*(np.exp(1j*2*m.pi*frq1*t)+np.exp(1j*2*m.pi*frq2*t))*np.exp(-t/T2) + np.random.normal(0,sig,len(t))
out = out/np.amax(out)
np.savez('timesignal.npz',t=t,signal=out)
```

```python
# Daten importieren
data = np.load('timesignal.npz')
t    = data['t']
signal = data['signal']

# Frequenzachse und Spektrum
frqax = fftshift(fftfreq(len(t),d=dt))
Sraw  = fftshift(fft(signal))

# Plot
plt.figure(figsize=(24,6))
plt.subplot(131)
plt.plot(t,np.real(out))
plt.xlabel('t [s]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.xlim((-0.1,max(t)))
plt.ylim((-1.1,1.1))
plt.title('Inputsignal',fontsize=13)

plt.subplot(132)
plt.plot(frqax,np.real(Sraw))
plt.xlim((0,max(frqax)))
plt.xlabel('$\\nu$ [Hz]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.title('Outputsignal',fontsize=13)

plt.subplot(133)
plt.plot(frqax,np.real(Sraw))
plt.xlim((0,20))
plt.xlabel('$\\nu$ [Hz]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.title('Outputsignal',fontsize=13);

```

```python
# Berechnung von Autokorrelation
Rxx,Sxx = autocorr_fcn(out)
Sxx     = fftshift(Sxx)

# Plot Input Autokorrelation
plt.figure(figsize=(24,6))
plt.subplot(131)
plt.plot(t,np.real(Rxx))
plt.xlim((-0.1,max(t)))
plt.xlabel('t [s]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.title('Zeitbereich',fontsize=13)

plt.subplot(132)
plt.plot(frqax,np.real(Sxx))
plt.xlabel('$\\nu$ [Hz]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.xlim((0,max(frqax)))
plt.title('Frequenzbereich',fontsize=13)
plt.suptitle('Autokorrelation',fontsize=17)

plt.subplot(133)
plt.plot(frqax,np.real(Sxx))
plt.xlabel('$\\nu$ [Hz]',fontsize=13)
plt.ylabel('Intensität',fontsize=13)
plt.xlim((0,20))
plt.title('Frequenzbereich',fontsize=13)
plt.suptitle('Autokorrelation',fontsize=17);

part1 = Sxx[frqax <=2.8]
frqax1  = frqax[frqax <=2.8]
part2 = Sxx[frqax > 2.8]
frqax2  = frqax[frqax > 2.8]
frq1 = frqax1[part1 == max(part1)]
frq2 = frqax2[part2 == max(part2)]

print([frq1[0],frq2[0]])
```