```python
from nltk.tokenize import word_tokenize
import nltk
import re
import numpy as np
from collections import defaultdict
nltk.download('punkt')
```

    [nltk_data] Downloading package punkt to
    [nltk_data]     C:\Users\furka\AppData\Roaming\nltk_data...
    [nltk_data]   Package punkt is already up-to-date!

    True

```python
def get_data():
    with open("data/shakespeare.txt") as f:
        text = f.read()
    text = re.sub(r"[,!?;-]", ".", text)
    data = word_tokenize(text)
    data = [w.lower() for w in data if w.isalpha() or w == "."]
    return data
```

```python
def get_dict(data):
    vocab = sorted(list(set(data)))
    word2Ind = {}
    Ind2word = {}
    idx = 0
    for word in vocab:
        word2Ind[word] = idx
        Ind2word[idx] = word
        idx += 1
    return word2Ind, Ind2word
```

```python
def initialize_model(N,V):
    '''
    Inputs:
        N:  dimension of hidden vector
        V:  dimension of vocabulary
        random_seed: random seed for consistent results in the unit tests
     Outputs:
        W1, W2, b1, b2: initialized weights and biases
    '''
    W1 = np.random.rand(N, V)
    b1 = np.random.rand(N, 1)
    W2 = np.random.rand(V, N)
    b2 = np.random.rand(V, 1)

    return W1, W2, b1, b2
```

```python
def softmax(z):
    '''
    Inputs:
        z: output scores from the hidden layer
    Outputs:
        yhat: prediction (estimate of y)
    '''
    yhat = np.exp(z) / np.sum(np.exp(z), axis = 0, keepdims = True)
    return yhat
```

```python
def forward_prop(x, W1, W2, b1, b2):
    '''
    Inputs:
        x:  average one hot vector for the context
        W1, W2, b1, b2:  matrices and biases to be learned
     Outputs:
        z:  output score vector
    '''
    h = np.dot(W1, x) + b1
    h = np.maximum(0, h)
    z = np.dot(W2, h) + b2
    return z, h
```

```python
def compute_cost(y, yhat, batch_size):
    logprobs = np.multiply(np.log(yhat), y)
    cost = - (1/batch_size) * np.sum(logprobs)
    cost = np.squeeze(cost)
    return cost
```

```python
def get_idx(context_words, word2Ind):
    """
    Inputs:
        context_word:
        word2Ind
    Outputs:
        idxs : kelimelerin idx karsiligi liste
    """
    idxs = []
    for word in context_words:
        idxs.append(word2Ind[word])
    return idxs

```

```python
def pack_idx_with_frequency(context_words, word2Ind):
    """
    Inputs:
        context_words : center word oncesi ve sonraki kelimler
        word2Ind : kelimlerin karsiliginda indexlere sahip dict
    Outputs:

    """

    freq_dict = defaultdict(int)
    for word in context_words:
        freq_dict[word] += 1
    idxs = get_idx(context_words, word2Ind)
    packed = []
    for i in range(len(idxs)):
        idx = idxs[i]
        freq = freq_dict[context_words[i]]
        packed.append((idx, freq))
    return packed
```

```python
def get_vectors(data, word2Ind, V, C):
    """
    Inputs:
        data : corpus
        word2Ind : kelimlerin karsiliginda indexlere sahip dict
        V : vocablary
        C : context half size
    Outputs:
        x : context'de bulunan kelimlerin one-hot toplamlari / context_size
        y : center word one-hot vektor
    """
    i = C # center word i ile takip edilir
    while True:
        y = np.zeros(V)
        x = np.zeros(V)
        center_word = data[i]
        y[word2Ind[center_word]] = 1
        context_words = data[i - 2: i] + data[i + 1 : i + 1 + C]
        context_len = len(context_words)
        for idx, freq in pack_idx_with_frequency(context_words, word2Ind):
            x[idx] = freq / context_len
        yield x, y
        i += 1
        if i >= len(data):
            i = 0
            print("i'ye 0 atandi")
```

```python
def get_batches(data, word2Ind, V, C, batch_size):
    """
    data : corpus
    word2Ind : Kelimlerin indexlerine sahip bir dict
    V : Vocablary
    C : context half size
    batch_size : Modele verilecek girdi sayisi
    """
    batch_x = []
    batch_y = []
    for x, y in get_vectors(data, word2Ind, V, C):
        while len(batch_x) < batch_size:
            batch_x.append(x)
            batch_y.append(y)
        else:
            yield np.array(batch_x).T, np.array(batch_y).T
            batch_x = []
            batch_y = []

```

### Test All Functions

```python
data = get_data()
word2Ind, Ind2word = get_dict(data)
tmp_V = len(word2Ind)
tmp_C = 2
tmp_N = 50
tmp_batch_size = 4

tmp_x, tmp_y = next(get_batches(data, word2Ind, tmp_V, tmp_C, tmp_batch_size))
print(tmp_x.shape, tmp_y.shape)

W1, W2, b1, b2 = initialize_model(tmp_N, tmp_V)
tmp_z, tmp_h = forward_prop(tmp_x, W1, W2, b1, b2)
tmp_yhat = softmax(tmp_z)
tmp_cost = compute_cost(tmp_y, tmp_yhat, tmp_batch_size)
print(tmp_cost)
```

    (5775, 4) (5775, 4)
    12.812297426050804

## Backpropagation

\begin{align}
 \frac{\partial J}{\partial \mathbf{W_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right )\mathbf{x}^\top \tag{1}\\
 \frac{\partial J}{\partial \mathbf{W_2}} &= (\mathbf{\hat{y}} - \mathbf{y})\mathbf{h^\top} \tag{1}\\
 \frac{\partial J}{\partial \mathbf{b_1}} &= \rm{ReLU}\left ( \mathbf{W_2^\top} (\mathbf{\hat{y}} - \mathbf{y})\right ) \tag{3}\\
 \frac{\partial J}{\partial \mathbf{b_2}} &= \mathbf{\hat{y}} - \mathbf{y} \tag{4}
\end{align}

### Ustteki backprop islemi tek bir ifade icin batch icin alinirsa belli degisiklikler olacaktir

```python
def back_prop(x, yhat, y, h, W1, W2, b1, b2, batch_size):
    '''
    Inputs:
        x:  average one hot vector for the context
        yhat: prediction (estimate of y)
        y:  target vector
        h:  hidden vector (see eq. 1)
        W1, W2, b1, b2:  matrices and biases
        batch_size: batch size
     Outputs:
        grad_W1, grad_W2, grad_b1, grad_b2:  gradients of matrices and biases
    '''
    grad_W2 = (1 / batch_size) * np.dot(yhat - y, h.T)
    grad_b2 = (1 / batch_size) * np.sum(yhat - y, axis = 1, keepdims=True)
    grad_W1 = (1 / batch_size) * np.dot(np.maximum(0, np.dot(W2.T, yhat - y)), x.T)
    grad_b1 = (1 / batch_size) * np.sum(np.maximum(0, np.dot(W2.T, yhat - y)), axis = 1, keepdims=True)
    return grad_W1, grad_W2, grad_b1, grad_b2
```

```python
def update_parameters(parameters, grads, alpha):
    W1, W2, b1, b2 = parameters
    grad_W1, grad_W2, grad_b1, grad_b2 = grads
    W1 = W1 - alpha * grad_W1
    W2 = W2 - alpha * grad_W2
    b1 = b1 - alpha * grad_b1
    b2 = b2 - alpha * grad_b2
    return W1, W2, b1, b2
```

```python
def gradient_descent(data, word2Ind, N, V, num_iters, alpha=0.03, batch_size = 128, C = 2):

    '''
    This is the gradient_descent function

      Inputs:
        data:      text
        word2Ind:  words to Indices
        N:         dimension of hidden vector
        V:         dimension of vocabulary
        num_iters: number of iterations
        alpha : learning rate
        batch_size : number of examples in each batch
        C : context half size
     Outputs:
        W1, W2, b1, b2:  updated matrices and biases after num_iters iterations

    '''
    parameters = initialize_model(N, V)

    i = 0
    for x, y in get_batches(data, word2Ind, V, C, batch_size):
        z, h = forward_prop(x, *parameters)
        yhat = softmax(z)
        cost = compute_cost(y, yhat, batch_size)
        grads = back_prop(x, yhat, y, h, *parameters, batch_size)
        parameters = update_parameters(parameters, grads, alpha)
        i += 1
        if i == num_iters:
            break
        if (i + 1) % 10 == 0:
            print(f"{i+1}.iter Cost : {cost:.6f}")
        if i % 100 == 0:
            alpha *= 0.66
    return W1, W2, b1, b2
```

```python
data = get_data()
word2Ind, Ind2word = get_dict(data)
V = len(word2Ind)
N = 50
C = 2
batch_size = 128
alpha = 0.03
num_iters = 150

```

```python
gradient_descent(data, word2Ind, N, V, num_iters, alpha, batch_size, C)
```

    10.iter Cost : 7.919861
    20.iter Cost : 9.987556
    30.iter Cost : 9.420588
    40.iter Cost : 9.730634
    50.iter Cost : 7.951681
    60.iter Cost : 1.908529
    70.iter Cost : 1.691551
    80.iter Cost : 10.574863
    90.iter Cost : 8.802042
    100.iter Cost : 7.780031
    110.iter Cost : 12.238310
    120.iter Cost : 11.485120
    130.iter Cost : 9.196244
    140.iter Cost : 9.154715
    150.iter Cost : 7.964914

    (array([[0.6219698 , 0.12671832, 0.64021677, ..., 0.14819813, 0.80576001,
             0.81387778],
            [0.30409119, 0.87743403, 0.27619172, ..., 0.36376133, 0.06986089,
             0.4669891 ],
            [0.03019791, 0.50882708, 0.44058658, ..., 0.30806755, 0.88930183,
             0.28626545],
            ...,
            [0.41081758, 0.43521317, 0.87382505, ..., 0.34322489, 0.04860274,
             0.46534924],
            [0.23565185, 0.81324709, 0.28863926, ..., 0.30923743, 0.84514841,
             0.22144138],
            [0.39902241, 0.30826413, 0.54928167, ..., 0.12998885, 0.07113925,
             0.23175899]]),
     array([[0.85062585, 0.19433653, 0.04235671, ..., 0.84236114, 0.80016593,
             0.29392048],
            [0.02274909, 0.99639508, 0.02554441, ..., 0.94838042, 0.81382462,
             0.15721227],
            [0.00897219, 0.82024191, 0.2140257 , ..., 0.62165875, 0.90084365,
             0.09101711],
            ...,
            [0.03522637, 0.64034248, 0.64632307, ..., 0.3075528 , 0.67987709,
             0.59643766],
            [0.62448305, 0.29742823, 0.55683552, ..., 0.08225942, 0.04195118,
             0.25003154],
            [0.02967369, 0.40398106, 0.44137213, ..., 0.90648067, 0.73090467,
             0.99526432]]),
     array([[0.61032945],
            [0.5358972 ],
            [0.88322514],
            [0.71856769],
            [0.73295778],
            [0.40554469],
            [0.51909259],
            [0.38541865],
            [0.29916493],
            [0.98148469],
            [0.66443857],
            [0.03374742],
            [0.39923968],
            [0.6352524 ],
            [0.52397655],
            [0.89299542],
            [0.93250723],
            [0.03426866],
            [0.24584283],
            [0.69623195],
            [0.33589813],
            [0.01775922],
            [0.1969199 ],
            [0.96315124],
            [0.00166809],
            [0.41191473],
            [0.72388129],
            [0.16446475],
            [0.21121686],
            [0.67647767],
            [0.95408039],
            [0.35058037],
            [0.80821358],
            [0.54117581],
            [0.4906112 ],
            [0.35974325],
            [0.40722071],
            [0.9422693 ],
            [0.80615255],
            [0.41648366],
            [0.40882981],
            [0.04963889],
            [0.12799035],
            [0.95953588],
            [0.34496246],
            [0.33763607],
            [0.619731  ],
            [0.72167394],
            [0.94588638],
            [0.02769827]]),
     array([[0.14180444],
            [0.2678395 ],
            [0.95445799],
            ...,
            [0.21701193],
            [0.70085438],
            [0.62933685]]))

```python
num_iters = 150
alpha = 0.03
gradient_descent(data, word2Ind, N, V, num_iters, alpha, batch_size, C)
```

    10.iter Cost : 10.814292
    20.iter Cost : 8.810384
    30.iter Cost : 11.351997
    40.iter Cost : 9.166524
    50.iter Cost : 12.098641
    60.iter Cost : 2.335376
    70.iter Cost : 1.993430
    80.iter Cost : 11.902983
    90.iter Cost : 9.226281
    100.iter Cost : 6.141662
    110.iter Cost : 8.760855
    120.iter Cost : 9.700995
    130.iter Cost : 6.349505
    140.iter Cost : 9.719155
    150.iter Cost : 8.535599

    (array([[0.6219698 , 0.12671832, 0.64021677, ..., 0.14819813, 0.80576001,
             0.81387778],
            [0.30409119, 0.87743403, 0.27619172, ..., 0.36376133, 0.06986089,
             0.4669891 ],
            [0.03019791, 0.50882708, 0.44058658, ..., 0.30806755, 0.88930183,
             0.28626545],
            ...,
            [0.41081758, 0.43521317, 0.87382505, ..., 0.34322489, 0.04860274,
             0.46534924],
            [0.23565185, 0.81324709, 0.28863926, ..., 0.30923743, 0.84514841,
             0.22144138],
            [0.39902241, 0.30826413, 0.54928167, ..., 0.12998885, 0.07113925,
             0.23175899]]),
     array([[0.85062585, 0.19433653, 0.04235671, ..., 0.84236114, 0.80016593,
             0.29392048],
            [0.02274909, 0.99639508, 0.02554441, ..., 0.94838042, 0.81382462,
             0.15721227],
            [0.00897219, 0.82024191, 0.2140257 , ..., 0.62165875, 0.90084365,
             0.09101711],
            ...,
            [0.03522637, 0.64034248, 0.64632307, ..., 0.3075528 , 0.67987709,
             0.59643766],
            [0.62448305, 0.29742823, 0.55683552, ..., 0.08225942, 0.04195118,
             0.25003154],
            [0.02967369, 0.40398106, 0.44137213, ..., 0.90648067, 0.73090467,
             0.99526432]]),
     array([[0.61032945],
            [0.5358972 ],
            [0.88322514],
            [0.71856769],
            [0.73295778],
            [0.40554469],
            [0.51909259],
            [0.38541865],
            [0.29916493],
            [0.98148469],
            [0.66443857],
            [0.03374742],
            [0.39923968],
            [0.6352524 ],
            [0.52397655],
            [0.89299542],
            [0.93250723],
            [0.03426866],
            [0.24584283],
            [0.69623195],
            [0.33589813],
            [0.01775922],
            [0.1969199 ],
            [0.96315124],
            [0.00166809],
            [0.41191473],
            [0.72388129],
            [0.16446475],
            [0.21121686],
            [0.67647767],
            [0.95408039],
            [0.35058037],
            [0.80821358],
            [0.54117581],
            [0.4906112 ],
            [0.35974325],
            [0.40722071],
            [0.9422693 ],
            [0.80615255],
            [0.41648366],
            [0.40882981],
            [0.04963889],
            [0.12799035],
            [0.95953588],
            [0.34496246],
            [0.33763607],
            [0.619731  ],
            [0.72167394],
            [0.94588638],
            [0.02769827]]),
     array([[0.14180444],
            [0.2678395 ],
            [0.95445799],
            ...,
            [0.21701193],
            [0.70085438],
            [0.62933685]]))

```python
# visualizing the word vectors here
from matplotlib import pyplot
%config InlineBackend.figure_format = 'svg'
words = ['king', 'queen','lord','man', 'woman','dog','wolf',
         'rich','happy','sad']

embs = (W1.T + W2)/2.0 # bu satira dikkat burada embedding vektoru aliniyor.

# given a list of words and the embeddings, it returns a matrix with all the embeddings
idx = [word2Ind[word] for word in words]
X = embs[idx, :]
print(X.shape, idx)  # X.shape:  Number of words of dimension N each
```

    (10, 50) [2744, 3949, 2960, 3022, 5672, 1452, 5671, 4189, 2315, 4276]

```python

```


# Function Approximation

Sciences Po, Spring 2019

## Outline

1. Overview of Approximation Methods
	1. Interpolation
	1. Regression
1. Polynomial Interpolation
1. Spline Interpolation
1. Multidimensional Approximation

## Approximation Methods

* Confronted with a non-analytic function $f$ (i.e. something not like $log(x)$), we need a way to numerically represent $f$ in a computer.
	* If your problem is to compute a value function in a dynamic problem, you don't have an *analytic representation* of $V$.
	* If you need to compute an equilibrium distribution for your model, you probably can't tell it's from one parametric family or another. Hence you need to approximate it.
* Approximations use *data* of some kind which informs us about $f$. Most commonly, we know the function values $f(x_i)$ at a corresponding finite set of points $X = \{x_i\}_{i=1}^N$.
* The task of approximation is to take that data and tell us what the function value is at $f(y),y\not \in X$.
* To an economist this should sound very familiar: take a dataset, learn it's structure, and make predictions.
* The only difference is that we can do much better here, because we have more degree's of freedom (we can choose our $X$ in $Y=\beta X + \epsilon$)

## Some Taxonomy

* Local Approximations: approximate function and it's derivative $f,f'$ at a *single* point $x_0$. Taylor Series:
	$$ f(x) = f(x_0) + (x-x_0)f'(x_0) + \frac{(x-x_0)^2}{2}f''(x_0) + \dots + \frac{(x-x_0)^n}{n!}f^{n}(x_0) $$
* Interpolation or *Colocation*: find a function $\hat{f}$ that is a good fit to $f$, and require that $\hat{f}$ *passes through* the points. If we think of there being a *residual* $\epsilon_i = f(x_i) - \hat{f}(x_i)$ at each grid point $i$, this methods succeeds in setting $\epsilon_i=0,\forall i$.
* Regression: Minimize some notion of distance (or squared distance) between $\hat{f}$ and $f$, without the requirement of pass through.

## Doing Interpolation in Julia

* In practice, you will make heavy use of high-quality interpolation packages in julia.
* List in the end.
* Nevertheless, function approximation is *extremely* problem-specific, so sometimes a certain approach does not work for your problem.
* This is why we will go through the mechanics of some common methods.
* I would like you to know where to start drilling if you need to go and hack somebody elses code.

## Basis Functions

1. What is a basis?
2. What is a basis function?

### *Basis* in Linear Algebra

* We call a basis $B$ of a vector space $V$ over a field $F$ (such as the real numbers $\mathbb{R}$) a *linearly independent subset* of $V$ that *spans* $V$.
* In other words $B \subset V$ is called a *basis* if it satisfies
    1. linear independence: for all subsets $b_1,b_2,\dots,b_n$, and for all scalars $a_1,a_2,\dots,a_n\in F$, we have that if $b_1 a_1 + b_2 a_2 +\dots+b_n a_n = 0$, then $a_1 = a_2 =\dots= a_n = 0$
    1. spanning of $V$: for all vectors $\mathbf{v} \in V$, one can choose $b_1,b_2,\dots,b_n \in B$ and $v_1,v_2,\dots,v_n \in V$ s.t. $\mathbf{v} = b_1 v_1 + b_2 v_2 +\dots+b_n v_n$
* From those definitions you can see that the $b$ are like *weights*, and that one can represent vectors are *linear combinations* of other vectors.

### Basis Example in R2

* The set $\mathbb{R}^2$ of ordered pairs of real numbers is a vector space $V$ for
    1. component-wise addition: $(a,b) + (c,d) = (a+c, b+d)$
    2. scalar multiplication: $\lambda (a,b) = (\lambda a, \lambda b), \lambda \in \mathbb{R}$.
* One basis called the *standard basis* of $V$ consists of two vectors:
    1. $e_1 = (1,0)$
    2. $e_2 = (0,1)$
* Any vector $v=(a,b) \in \mathbb{R}^2$ can be uniquely written as $v = ae_1 + be_2$
* Any other pair of linearly independent vectors like $(1,1)$ and $(-1,2)$ is also a basis of $\mathbb{R}^2$.

### Higher Dimensions

* If the field $F$ has more than 2 dimensions, $n$ say, we work with $n$-tuples.
* For example, $e_i = (0,\dots,0,1,0\dots,0)$ with the $i$th component $1$.
* The set of all *monomials* forms a particular basis: the *monomial basis*: ${1,X,X^2,X^3,\dots}$
* **Any** set of polynomials with a unique polynomial for each degree is also a basis, called a *polynomial sequence*. A prominent example here are the *Chebyshev polynomials*, which we will encounter again.

## What is a Basis *function*?

* Now we know that every vector $\mathbf{v}$ in a vector space $V$ can be represented by a linear combination of *basis vectors*.
* Similarly, we can represent every *continuous* function in a particular *function space* $F$ by a linear combination of *basis functions*.
* Another good name for basis functions is *blending functions*. We often use a mixture of several basis functions to find an approximation.

## Approximating Functions

* Let $F$ by the space of continuous real-valued functions with domain $\mathbf{x}\in \mathbb{R}$. $F$ is a vector space.
* Next, we define an inner-product operation on that space:
    $$
    <g,h> = \int_\mathbf{x} g(x) h(x) w(x) dx
    $$
    where functions $g,h,w \in F$ and $w$ is a weighting function.
* the pair $\{F, <.,.>\}$ form an *inner-product vector space*.
* Now we want to approximate a **known** function $f:\mathbf{x}\mapsto\mathbb{R}$ in $\{F, <.,.>\}$

* Let's define $\hat{f}(\cdot;c)$ to be our parametric approximation function. We generically define this as

$$
    \hat{f}(x;c) = \sum_{j=0}^{J-1} c_j \phi_j(x)
$$

where
* $\phi_j : \mathbb{R}^d \mapsto \mathbb{R}$ is called a **basis function**, and $\Phi_J = \{\phi_j\}_{j=0}^{J-1}$.
* $c={c_0,c_1,\dots,c_{J-1}}$ is a coefficient vector
* The integer $J$ is the *order* of the interpolation.
* Our problem is to choose $(\phi_i,c)$ in some way.
* We want to minimize the residual function, i.e. $\gamma(x,c) \equiv f(x) - \sum_{j=0}^{J-1} c_j \phi_j(x)$

* To find coefficients $c$, one could for example minimize the squared errors, OLS:
    $$
    c^* = \arg \min \int_\mathbf{x} \gamma(x,c) w(x) dx
    $$
* In standard OLS, we'd set the weighting function to $w(x) = 1$

We will construct a *grid* of $M\geq J$ points ${x_1,\dots,x_M}$ within the domain $\mathbb{R}^d$, and we will denote the *residuals* at each grid point by $\epsilon = {\epsilon_1,\dots,\epsilon_M}$:

$$
    \left[\begin{array}{c}
        \epsilon_1 \\
         \vdots \\
        \epsilon_M \\ \end{array} \right]  = \left[\begin{array}{c} f(x_1) \\ \vdots \\ f(x_M)  \end{array} \right] - \left[\begin{array}{ccc}
        \phi_1(x_1) & \dots & \phi_J(x_1) \\
        \vdots      & \ddots & \vdots \\
        \phi_1(x_M) & \dots & \phi_J(x_M)
        \end{array} \right]  \cdot
        \left[\begin{array}{c} c_1 \\ \vdots \\ c_J  \end{array} \right] \\
        \mathbf{\epsilon} = \mathbf{y} - \mathbf{\Phi c}
$$

* *Interpolation* or *colocation* occurs when $J=M$, i.e. we have a square matrix of basis functions, and can exactly solve this.
* We basically need to solve the system

$$
\begin{aligned}
\sum_{j=1}^n c_j \phi_j(x_i) &= f(x_i),\forall i=1,2,\dots,n \\
                  \mathbf{\Phi c}&= \mathbf{y}
\end{aligned}
$$

where the second line uses vector notation, and $\mathbf{y}$ has all values of $f$.
* Solution: $\mathbf{c}= \mathbf{\Phi}^{-1}y$.

## Regression Basics

* If we have more evaluation points than basis functions, $M>J$ say, interpolation nodes than basis functions, we cannot do that. Instead we can define a loss function, and minimize it.
* In the case of squared loss, of course, this leads to the least squares solution:

$$
\begin{aligned} e_i &= f(x_i) - \sum_{j=1}^n c_j \phi_j(x_i) \\
        \min_c e_i^2 & \implies \\
        c            &= (\Phi'\Phi)^{-1} \Phi'y
\end{aligned}
 $$

## Spectral and Finite Element Methods

* **Spectral Methods** are such that the basis functions are non-zero over the entire domain of $f$.
	* Polynomial interpolation
	* Chebychev interpolation
* **Finite Element** methods are such that basis functions are non-zero only on a subset of the domain.
	* Splines
		* Linear splines, i.e. splines of degree 1, a.k.a. *linear approximation*
		* Higher order splines, mainly the *cubic spline*.

## What makes a **good** Approximation?

* Should be arbitrarily accurate as we increase $n$.
* $\Phi$ Should be efficiently (fast) computable. If $\Phi$ were differentiable, we could easily get e.g. $\hat{f}'(x) = \sum_{j=1}^J c_j \phi_j'(x_i)$
* $c$ Should be efficiently (fast) computable.

## What Polynomial to use? What form for $\Phi$?

* In principle the *monomial basis* could be used. It is just the power functions of $x$: $1,x,x^2,x^3,\dots$
* Stacking this up for each evaluation node gives the *Vandermonde Matrix*:

$$
V = \left[\begin{matrix}
        1 & x_1 & \dots & x_1^{n-2} & x_1^{n-1} \\
        1 & x_2 & \dots & x_2^{n-2} & x_2^{n-1} \\
        \vdots & \vdots & \ddots &  & \vdots \\
        1 & x_m & \dots & x_m^{n-2} & x_m^{n-1}
        \end{matrix} \right]
$$

for the case with $m$ evaluation nodes for $x$, and $n$ basis functions for each $x_i$.

### Choosing Basis Matrices

* If we choose $\Phi$ so that it's elements share a lot of information, it will be hard to invert it.
* So, if the elements share very little information, it's easier to invert, hence good for us
* For example, if we have $\Phi$ close to diagonal (i.e. different basis $\phi_j(x)$ are non-zero at different $x$).
* Such $\Phi$ are generated by using *orthogonal Polynomials* (under our inner product definition).
* An orthogonal Polynomial has 2 sequences of polynomials orthogonal under an inner product: their inner product is zero under a certain weighting function $w(x)$.
* For example, the Chebyshev polynomials are orthogonal wrt. to weight $\frac{1}{\sqrt{1-x^2}}$
* for degrees $n,m$ of that polynomial, $T_n,T_m$, we have
    $$
    \int_{-1}^1 T_n(x)T_m(x)\,\frac{dx}{\sqrt{1-x^2}}=
    \begin{cases}
        0 & n\ne m \\ \pi & n=m=0 \\ \frac{\pi}{2} & n=m\ne 0
    \end{cases}
    $$

### Chebyshev Polynomials *of the first kind*

* There is a nice recursion to get $T_n(x)$:
    \begin{align}
        T_0(x) & = 1 \\
        T_1(x) & = x \\
        T_{n+1}(x) & = 2xT_n(x) - T_{n-1}(x).
    \end{align}
* Code that up and compute $T_{3}(0.5)$!
* Check you get the same as with the alternative closed form $T_n(x) = \cos(n \arccos(x))$

```julia
function T(x,n)
    @assert (x >= -1) & (x <= 1)
    if n == 0
        return 1.0
    elseif n==1
        return x
    else
        2*x*T(x,n-1) - T(x,n-2)
    end
end
T2(x,n) = cos(n* acos(x))
@show T(0.5,3)
using Test
@assert T2(0.5,3) == T(0.5,3)
```

    T(0.5, 3) = -1.0

```julia
# weighting function
w(x) = 1.0 / sqrt(1-x^2)
using Statistics
using Test

# simple integration works for n not equal m
@assert isapprox(mean(T(x,0) * T(x,1) * w(x) for x in range(-0.99,stop = 0.99, length=100)), 0.0, atol = 1e-16)
@assert isapprox(mean(T(x,4) * T(x,5) * w(x) for x in range(-0.99,stop = 0.99, length=100)), 0.0, atol = 1e-16)
```

## Chebyshev Nodes

* *Chebyshev Nodes* are the roots of the chebyshev polynomial (i.e. where $P(x) = 0$).
* They are defined in the interval $[-1,1]$ as
    $$ x_i = \cos\left(\frac{2k-1}{2n} \pi\right), k=1,\dots,n $$
* Which maps to general interval $[a,b]$ as
	$$ x_i = \frac{1}{2} (a+b) + \frac{1}{2} (b-a) \cos\left(\frac{2k-1}{2n} \pi\right) , k=1,\dots,n $$
* Chebyshev nodes are **not** evenly spaced: there are more points towards the boundaries. You might remember that from Gaussian Integration nodes.

```julia
using Plots
using FastGaussQuadrature
gcnodes = gausschebyshev(21)  # generate 11 Chebyshev Nodes
gr()
scatter(gcnodes,ones(21),ylims=(0.9,1.1),m=(:red,:+),legend=false,size=(600,100),yticks=nothing)
```

### Constructing $\Phi$ as $T$ evaluated *at* the Chebyshev Nodes
* Combining Chebyshev nodes evaluated at the roots $T$ of the Cheby polynomial to construct $\Phi$ is a particularly good idea.
* Doing so, we obtain an interpolation matrix $\Phi$ with typical element
	$$ \phi_{ij} = \cos\left( \frac{(n-i+0.5)(j-1)\pi}{n}\right)  $$
* And we obtain that $\Phi$ is indeed orthogonal
	$$ \Phi^T \Phi = \text{diag}\{n,n/2,n/2,\dots,n/2\}  $$

```julia
using BasisMatrices
x = range(-4, stop = 4 ,length = 10)
ϕ = Basis(ChebParams(length(x),-4,4))
ϕ
```

    ┌ Info: Recompiling stale cache file /Users/florian.oswald/.julia/compiled/v1.1/BasisMatrices/65PSC.ji for BasisMatrices [08854c51-b66b-5062-a90d-8e7ae4547a49]
    └ @ Base loading.jl:1184

    1 dimensional Basis on the hypercube formed by (-4.0,) × (4.0,).
    Basis families are Cheb

```julia
S, y = nodes(ϕ)
```

    ([-3.95075, -3.56403, -2.82843, -1.81596, -0.625738, 0.625738, 1.81596, 2.82843, 3.56403, 3.95075], ([-3.95075, -3.56403, -2.82843, -1.81596, -0.625738, 0.625738, 1.81596, 2.82843, 3.56403, 3.95075],))

```julia
Φ = BasisMatrix(ϕ, Expanded(), S, 0)
```

    BasisMatrix{Expanded} of order [0]

```julia
Φ.vals[1]' * Φ.vals[1]
```

    10×10 Array{Float64,2}:
     10.0          -1.11022e-16  -2.22045e-16  …  -2.10942e-15  -5.32907e-15
     -1.11022e-16   5.0          -1.41698e-15     -4.571e-15    -2.46571e-15
     -2.22045e-16  -1.41698e-15   5.0             -2.33515e-15  -5.35038e-15
     -1.88738e-15  -5.05353e-16  -1.91117e-15     -4.6033e-15   -2.2613e-15
     -1.22125e-15  -2.38199e-15  -1.46093e-15     -2.66302e-15  -4.90537e-15
     -2.77556e-15  -1.43304e-15  -2.71577e-15  …  -4.67867e-15  -2.25549e-15
     -1.66533e-15  -3.38498e-15  -2.09657e-15     -2.23671e-15  -4.98201e-15
     -3.88578e-15  -1.76844e-15  -3.9379e-15      -4.87009e-15  -2.42363e-15
     -2.10942e-15  -4.571e-15    -2.33515e-15      5.0          -4.90931e-15
     -5.32907e-15  -2.46571e-15  -5.35038e-15     -4.90931e-15   5.0

## (Chebyshev) Interpolation Proceedure

* Let's summarize this proceedure.
* Instead of Chebyshev polynomials we could be using any other suitable family of polynomials.
* To obtain a Polynomial interpolant $\hat{f}$, we need:
	1. a function to $f$ interpolate. We need to be able to get the function values somehow.
	1. A set of (Chebyshev) interpolation nodes at which to compute $f$
	1. An interpolation matrix $\Phi$ that corresponds to the nodes we have chosen.
	1. A resulting coefficient vector $c$
* To obtain the value of the interpolation at $x'$ off our grid, we also need a way to evaluate $\Phi(x')$.
	1. Evaluate the Basis function $\Phi$ at $x'$: get $\Phi(x')$
	2. obtain new values as $y = \Phi(x') c$.

## Polynomial Interpolation with `Julia`: `ApproxFun.jl`

* [`ApproxFun.jl`](https://github.com/ApproxFun/ApproxFun.jl) is a Julia package based on the Matlab package [`chebfun`](http://www.chebfun.org). It is quite amazing.
* More than just *function approximation*. This is a toolbox to actually *work* with functions.
* given 2 functions $f,g$, we can do algebra with them, i.e. $h(x) = f(x) + g(x)^2$
* We can differentiate and integrate
* Solve ODE's and PDE's
* represent periodic functions
* Head over to the [website and look at the readme](https://github.com/JuliaApproximation/ApproxFun.jl).

```julia
using LinearAlgebra, SpecialFunctions, Plots, ApproxFun
x = Fun(identity,0..10)
f = sin(x^2)
g = cos(x)

h = f + g^2
r = roots(h)
rp = roots(h')

using Plots
plot(h,labels = "h")
scatter!(r,h.(r),labels="roots h")
scatter!(rp,h.(rp),labels = "roots h'")
```

```julia
# works even with discontinuities!
ff = x->sign(x-0.1)/2 + cos(4*x);  # sign introduces a jump at 0.1
x  = Fun(identity)	# set up a function space
space(x)
f  = ff(x)	# define ff on that space
plot(f)	# plot
```

```julia
# whats the first deriv of that function at at 0.785?
println(f'(0.785))
# and close to the jump, at 0.100001?
println(f'(0.1000001))
# integral of f?
g1 = cumsum(f)
g = g1 + f(-1)
integral = norm(f-g)
integral
```

    -0.006370611665950006
    -1.5576748429320297

    2.357186284506263

* The main purpose of this package is to manipulate analytic functions, i.e. function with an algebraic representation.
* There is the possibility to supply a set of data points and fit a polynomial:

```julia
S = Chebyshev(1..2);
n = 100; m = 50;
p = range(1,stop=2,length=n);   # a non-default grid
v = exp.(p);           # values at the non-default grid
V = Array{Float64}(undef,n,m); # Create a Vandermonde matrix by evaluating the basis at the grid
for k = 1:m
    V[:,k] = Fun(S,[zeros(k-1);1]).(p)
end
f = Fun(S,V\v);
@show f(1.1)
@show exp(1.1)

```

    f(1.1) = 3.004166023946432
    exp(1.1) = 3.0041660239464334

    3.0041660239464334

## Splines: Piecewise Polynomial Approximation

* Splines are a finite element method, i.e. there are regions of the function domain where some basis functions are zero.
* As such, they provide a very flexible framework for approximation instead of high-order polynomials.
	* Keep in mind that Polynomials basis functions are non-zero on the entire domain. Remember the Vandermonde matrix.
* They bring some element of local approximation back into our framework. What happens at one end of the domain to the function is not important to what happens at the other end.
* Looking back at the previous plot of random data: we are searching for **one** polynomial to fit **all** those wiggles. A spline will allow us to design **different** polynomials in different parts of the domain.

### Splines: Basic Setup

* The fundamental building block is the *knot vector*, or the *breakpoints vector* $\mathbf{z}$ of length $p$. An element of $\mathbf{z}$ is $z_i$.
* $\mathbf{z}$ is ordered in ascending order.
* $\mathbf{z}$ spans the domain $[a,b]$ of our function, and we have that $a=z_1,b=z_p$
* A spline is of *order k* if the polynomial segments are k-th order polynomials.
* Literature: [@deboor] <cite data-cite=deboo></cite> is the definitive reference for splines.

### Splines: Characterization

* Given $p$ knots, there are $p-1$ polynomial segments of order $k$, each characterized by $k+1$ coefficients, i.e. a total of $(p-1)(k+1)$ parameters.
* However, we also require the spline to be continuous and differentiable of degree $k-1$ at the $p-2$ interior breakpoints.
* Imposing that uses up an additional $k(p-2)$ conditions.
* We are left with $n = (p-1)(k+1) - k(p-2) = p+k-1$ free parameters.
* A Spline of order $k$ and $p$ knots can thus be written as a linear combination of it's $n = p+k-1$ basis functions.

## B-Splines: Definition

* We mostly use Basis Splines, or **B-Splines**.
* Here is a recursive definition of a B-Spline (and what is used in `ApproXD`):
* Denote the $j$-th basis function of degree $k$ with knot vector $\mathbf{z}$  at $x$ as $B_j^{k,\mathbf{z}} (x)$
* Again, there are $n = k + p - 1$ $B$'s (where $p$` = length(z)`)
* We can define $B_j^{k,\mathbf{z}} (x)$ recursively like this:

	$$
    B_j^{k,\mathbf{z}} (x) = \frac{x-z_{j-k}}{z_j - z_{j-k}} B_{j-1}^{k-1,\mathbf{z}} (x)  + \frac{z_{j+1}-x}{z_{j+1} - z_{j+1-k}} B_{j}^{k-1,\mathbf{z}} (x), j=1,\dots,n
    $$

* The recursion starts with

$$ B_j^{0,\mathbf{z}} (x) = \begin{cases}
    1 & \text{if }z_j \leq x <	z_{j+1}\\
    0 & \text{otherwise.}
    \end{cases}
$$

* For this formulation to work, we need to extend the knot vector for $j<1,j>p$:

$$ z_j = \begin{cases}
    a & \text{if }j \leq 1\\
    b & \text{if }j \geq p
    \end{cases}
$$

* And we need to set the endpoints
	$$ B_0^{k-1,\mathbf{z}} = B_n^{k-1,\mathbf{z}} =0 $$
* You may see that this gives rise to a triangular computation strategy, as pointed out [here](http://www.cs.mtu.edu/~shene/COURSES/cs3621/NOTES/spline/B-spline/bspline-basis.html).

## B-Splines: Derivatives and Integrals

* This is another very nice thing about B-Splines.
* The derivative wrt to it's argument $x$ is

	$$ \frac{d B_j^{k,\mathbf{z}} (x)}{dx} = \frac{k}{z_j - z_{j-k}} B_{j-1}^{k-1,\mathbf{z}} (x)  + \frac{k}{z_{j+1} - z_{j+1-k}} B_{j}^{k-1,\mathbf{z}} (x), j=1,\dots,n
    $$

* Similarly, the Integral is just the sum over the basis functions:
	$$ \int_a^x B_j^{k,\mathbf{z}} (y) dy = \sum_{i=j}^n \frac{z_i - z_{i-k}}{k} B_{i+1}^{k+1,\mathbf{z}} (x)  $$

```julia
using ApproXD
# ] dev https://github.com/floswald/ApproXD.jl
bs = BSpline(7,3,0,1) #7 knots, degree 3 in [0,1]
# how many basis functions? (go back 1 slide.)
# getNumCoefs(bs)
x = range(0,stop =1.0, length = 500)
B = Array(getBasis(collect(x),bs))
plot(x,B,layout=(3,3),grid=false,ylim=(-0.1,1.1),legend=false,linewidth=2,linecolor=:black)
```

```julia
# Notice that placing each of those panels on top of each other generates a sparse matrix!
plot(x,B,grid=false,ylim=(-0.1,1.1),legend=false)
```

## Linear B-Spline: A useful special case

* This is *connecting the dots with a straight line*
* This may incur some approximation error if the underlying function is very curved between the dots.
* However, it has some benefits:
	* it is shape-preserving,
	* it is fast,
	* it is easy to build.
* For a linear spline with evenly spaced breakpoints, this becomes almost trivial.
	* Let's define $h = \frac{b-a}{n-1}$ as the distance between breakpoints.
	* Our basis function becomes very simple, giving us a measure of how far $x$ is from the next knot:

$$ \phi_j (x) = \begin{cases}
    1 - \frac{|x-z_j|}{h} & \text{if } |x-z_j| \leq h \\
    0                    & \text{otherwise}
    \end{cases}
$$

* Notice that each interior basis function (i.e. not 0 and not $n$) has witdth $2h$.

```julia
bs = BSpline(9,1,0,1) #9 knots, degree 1 in [0,1]
x = range(0,stop = 1.0,length = 500)
B = Array(getBasis(collect(x),bs))
plot(x,B,layout=(3,3),grid=false,ylim=(-0.1,1.1),legend=false,linewidth=2,linecolor=:black)
```

## Linear B-Spline: Evaluation

* In order to evaluate the linear interpolator, we need to know only one thing: Which knot span is active, i.e. what is $j$ s.t. $x\in [z_j, z_{j+1}]$?
* This is a classic problem in computer science. Binary search.
* `julia` implements [`searchsortedlast`](http://docs.julialang.org/en/release-0.4/stdlib/sort/#Base.searchsortedlast).
* Once we know $j$, it's easy to get the interpolated value as
	$$ \hat{f}(x) = \frac{x-z_j}{h}  f(z_{j+1}) + \frac{z_{j+1}-x}{h}  f(z_{j}) $$

## The Importance of Knot Placement

* We just talked about *equally spaced knots*. This is just a special case.
* B-Splines give us the flexibility to place the knots where we want.
* Contrary to Polynomial interpolations (where we cannot choose the evaluation nodes), this is very helpful in cases where we know that a function is very curved in a particular region.
* Canonical Example: Runge's function: $f(x) = (1+25x^2)^{-1}$.
* Also: If you know that your function has a kink (i.e. a discontinuous first derivative) at $\hat{x}$, then you can stack breakpoints on top of each other *at* $\hat{x}$
* That's going to be part of your homework!

```julia
gg(x) = (1+25x^2)^(-1)
plot(gg)
```

## Interpolation with [`Interpolations.jl`](https://github.com/JuliaMath/Interpolations.jl)

* Interpolations.jl assumes that data is uniformly spaced on grid `1:N`
* Same for multiple dimensions
* However, we can scale `1:N` to different domains
* finally, we can also supply our own, non-uniform grids.
* Can get gradients of interpolations right away.

```julia
# interpolation
# restart kernel!
using Interpolations
A = rand(10,5)
itp = interpolate(A, Interpolations.BSpline(Quadratic(Reflect(OnCell()))))
itp(1.9,4.9)
```

    0.2609001368014953

```julia
A

```

    10×5 Array{Float64,2}:
     0.0440558  0.0759254  0.424806  0.0733616  0.0104042
     0.131006   0.692091   0.435421  0.363613   0.789312
     0.423814   0.765224   0.962783  0.48646    0.768033
     0.35892    0.394449   0.938537  0.729178   0.706436
     0.54336    0.148386   0.236388  0.580917   0.192891
     0.507772   0.517813   0.207813  0.930954   0.707769
     0.868053   0.959416   0.726591  0.741459   0.283866
     0.293944   0.547819   0.851113  0.550485   0.270864
     0.134517   0.998558   0.975837  0.704406   0.862646
     0.452014   0.307677   0.469785  0.365602   0.178347

```julia
A[2,5]
```

    0.7893123641918425

```julia
A_x = 1.:2.:40.   # x in [1,40]
A = [log(x) for x in A_x] # f(x)
itp = interpolate(A, Interpolations.BSpline(Cubic(Interpolations.Line(OnGrid()))))
sitp = scale(itp, A_x)  # scale x-axis of interpolator
@show sitp(3.) # exactly log(3.)
@show sitp(3.5) # approximately log(3.5)
@show itp(3);  # is the 3rd index(!) in A, not the *value* 3
```

    sitp(3.0) = 1.0986122886681098
    sitp(3.5) = 1.2748716241925298
    itp(3) = 1.6094379124341

```julia
# Same for 2D
A_x1 = 1:.1:10
A_x2 = 1:.5:20
fff(x1, x2) = log(x1+x2)
A = [fff(x1,x2) for x1 in A_x1, x2 in A_x2]
itp = interpolate(A, Interpolations.BSpline(Cubic(Interpolations.Line(OnGrid()))))
sitp = scale(itp, A_x1, A_x2)
sitp(5., 10.) # exactly log(5 + 10)
sitp(5.6, 7.1) # approximately log(5.6 + 7.1)
```

    2.541602006923902

* Very often we need a **gridded** interpolation. I.e. we supply the function values on an irregular grid.
* For this occasion, the `GriddedInterpolation` type is useful.
* For now this only works in 3 modes:
    * Gridded(Linear())
    * Gridded(Constant())  nearest neighbor
    * NoInterp (you must supply index ON grid)

```julia
A = rand(20)
A_x = collect(1.0:2.0:40.0)
knots = (A_x,)
itp = interpolate(knots, A, Gridded(Linear()))
itp(2.0)
```

    0.5342315736445034

```julia
# 2D
A = rand(8,20)
knots = ([x^2 for x = 1:8], [0.2y for y = 1:20])
itp = interpolate(knots, A, Gridded(Linear()))
itp(4,1.2)  # approximately A[2,6]
```

    0.6334118989442268

```julia
# we can mix modes across dimensions!
itp = interpolate(knots, A, (Gridded(Linear()),Gridded(Constant())))
itp(4,1.2)
```

    0.633411898944227

* What about vector valued interpolations?
* Suppose we have a function $f : \mathbb{R} \mapsto \mathbb{R}^2$
* Economics example:

$$
f(x) = \left(\begin{array}{c} \text{savings}(x) \\ \text{consumption}(x) \end{array} \right)
$$

* $x$ is cash on hand.
* We often have situations where several functions are defined on a common support $x$.
* what is $f(1.75)$?

```julia
using StaticArrays
x = range(1,stop = 3, length = 200)  # cash on hand
a = Float64[log(1+j)*i for j in x, i in 1:2]   # cons and save function
b = reinterpret(SVector{2,Float64}, a')[:]
itp = interpolate(b, Interpolations.BSpline(Quadratic(Reflect(OnCell()))))
@show itp(3)
sitp = scale(itp,x)
@show sitp(3);
```

    itp(3) = [0.703147, 1.40629]
    sitp(3) = [1.38629, 2.77259]

```julia
v = sitp(1.75)  # get interpolated values for both function
plot(x,a,labels=["save","cons"],yticks=convert(Array{Float64},v),xlabel="current cash")
vline!([1.75],lab="")
hline!([v[1],v[2]],lab="")
```

## The CompEcon Toolbox of Miranda and Fackler

* another good alternative:
* [CompEcon.jl](https://github.com/spencerlyon2/CompEcon.jl)

## Multidimensional Approximation

* Up to now, most of what we did was in one dimesion.
* Economic problems *often* have more dimension than that.
	* The number of state variables in your value functions are the number of dimensions.
* We can readily extend what we learned into more dimensions.
* However, we will quickly run into feasibility problems: hello *curse of dimensionality*.

## Tensor Product of univariate Basis Functions: Product Rule

* One possibility is to approximate e.g. the 2D function $f(x,y)$ by
	$$ \hat{f}(x,y) = \sum_{i=1}^n \sum_{j=1}^m c_{i,j} \phi_i^x(x) \phi_j^y(y)  $$
	* here $\phi_i^x$ is the basis function in $x$ space,
	* you can see that the coefficient vector $c_{i,j}$ is indexed in two dimensions now.
	* Notice that our initial notation was general enough to encompass this case, as we defined the basis functions as $\mathbb{R}^d \mapsto \mathbb{R}$. So with the product rule, this mapping is just given by $\phi_i^x(x) \phi_j^y(y)$.
* This formulation requires that we take the product of $\phi_i^x(x), \phi_j^y(y)$ at *all* combinations of their indices, as is clear from the summations.
* This is equivalent to the tensor product between $\phi_i^x$ and $\phi_j^y$.

## Computing Coefficients from Tensor Product Spaces

* Extending this into $D$ dimensions, where in each dim $i$ we have $n_i$ basis functions, we get
	$$ \hat{f}(x_1,x_2,\dots,x_D) = \sum_{i_1=1}^{n_1} \sum_{i_2=1}^{n_2} \dots  \sum_{i_D=1}^{n_D} c_{i_1,i_2,\dots,i_D} \phi_{i_1}(x_1) \phi_{i_2}(x_2) \dots \phi_{i_D}(x_D)  $$
* In Vector notation
	$$
	\hat{f}(x_1,x_2,\dots,x_D) =  \left[ \phi_{D}(x_D) \otimes \phi_{D-1}(x_{D-1})  \otimes \dots  \otimes  \phi_{1}(x_1) \right]  c $$
	where $c$ is is an $n=\Pi_{i=1}^D n_i$ column vector
* The solution is the interpolation equation as before,
	$$ \begin{aligned}\Phi c =& y \\
					\Phi   =& \Phi_D \otimes \Phi_{D-1} \otimes \dots \otimes \Phi_{1} \end{aligned} $$

### The Problem with Tensor Product of univariate Basis Functions

* What's the problem?
* Well, solving $\Phi c = y$ is hard.
* If we have as many evaluation points as basis functions in each dimension, i.e. if each single $\Phi_i$ is a square matrix, $\Phi$ is of size (n,n).
* Inverting this is *extremely* hard even for moderately sized problems.
* Sometimes it's not even possible to allocate $\Phi$ in memory.
* Here it's important to remember the sparsity structure of a spline basis function.

```julia
using ApproXD
gr()
bs = ApproXD.BSpline(7,3,0,1) #7 knots, degree 3 in [0,1]
n = 500
eval_points = collect(range(0,stop = 1.0,length = n))
B = Array(ApproXD.getBasis(eval_points,bs))
ys = [string("Basis",i) for i = size(B)[2]-1:-1:0]
xs = reverse(eval_points)
heatmap(xs,ys,reverse(B',dims=1))
```

* This is a cubic spline basis. at most $k+1=4$ basis are non-zero for any $x$.

```julia
heatmap(xs,ys,reverse(B',dims=1) .> 0, cbar = false)
```

## Using Sparsity of Splines

* It may be better to store the splines in sparse format.
* Look at object `B` by typing `B` and `typeof(B)`
* There are sparse system solvers available.
* Creating and storing the inverse of $\Phi$ destroys the sparsity structure (inverse of a sparse matrix is not sparse), and may not be a good idea.
* Look back at [Computing coefficients form the tensor product](#compute-c)
* We only have to sum over the non-zero entries! Every other operation is pure cost.
* This is implemented in `ApproXD.jl` for example via

```julia
    function evalTensor2{T}(mat1::SparseMatrixCSC{T,Int64},
                            mat2::SparseMatrixCSC{T,Int64},
                            c::Vector{T})
```

## High Dimensional Functions: Introducing the Smolyak Grid

* This is a modification of the Tensor product rule.
* It elemininates points from the full tensor product according to their *importance* for the quality of approximation.
* The user controls this quality parameter, thereby increasing/decreasing the size of the grid.
* [@jmmv] <cite data-cite=jmmv></cite>is a complete technical reference for this method.
* [@maliar-maliar] <cite data-cite=maliarmaliar></cite>chapter 4 is very good overview of this topic, and the basis of this part of the lecture.

## The Smolyak Grid in 2 Dimensions

* Approximation level $\mu \in \mathbb{N}$ governs the quality of the approximation.
* Start with a unidimensional grid of points $x$:
	$$ x = \left\{-1,\frac{-1}{\sqrt{2}},0,\frac{1}{\sqrt{2}},1\right\} $$
	which are 5 Chebyshev nodes (it's not important that those are Chebyshev nodes, any grid will work).
* A 2D tensor product $x\otimes x$ gives 25 grid points
	$$ x\otimes x=\left\{(-1,-1),(-1,\frac{-1}{\sqrt{2}}),\dots,(1,1)\right\} $$

* The Smolyak method proceeds differently.
* We construct three nested sets:

$$ \begin{array}{l}
		i=1 : S_1 = \{0\} \\
		i=2 : S_2 = \{0,-1,1\} \\
		i=3 : S_3 = \left\{-1,\frac{-1}{\sqrt{2}},0,\frac{1}{\sqrt{2}},1\right\}  \end{array} $$

* Then, we construct all possible 2D tensor products using elements from these nested sets in a table (next slide).
* Finally, we select only those elements of the table, that satisfy the Smolyak rule:
	$$ i_1 + i_2 \leq d + \mu $$
	where $i_1,i_2$ are column and row index, respectively, and $d,\mu$ are the number of dimensions and the quality of approximation.

# The Smolyak Grid in 2D: Tensor Table

![[@maliar-maliar] table 3: All Tensor Products](../assets/figs/smolyak-tensortab.png)

## Selecting Elements

* Denote the Smolyak grid for $d$ dimensions at level $\mu$ by $\mathcal{H}^{d,\mu}$.
* if $\mu=0$ we have $i_1+i_2\leq 2$. Only one point satisfies this, and
	$$ \mathcal{H}^{2,0} = \{(0,0)\} $$
* if $\mu=1$ we have $i_1+i_2\leq 3$. Three cases satisfy this:
	1. $i_1 = 1, i_2=1 \rightarrow (0,0)$
	1. $i_1 = 1, i_2=2 \rightarrow (0,0),(0,-1),(0,1)$
	1. $i_1 = 2, i_2=1 \rightarrow (0,0),(-1,0),(1,0)$
	* Therefore, the unique elements from the union of all of those is
		$$ \mathcal{H}^{2,1} = \{(0,0),(-1,0),(1,0),(0,-1),(0,1)\} $$

* if $\mu=2$ we have $i_1+i_2\leq 4$. Six cases satisfy this:
	1. $i_1 = 1, i_2=1$
	1. $i_1 = 1, i_2=2$
	1. $i_1 = 2, i_2=1$
	1. $i_1 = 1, i_2=3$
	1. $i_1 = 2, i_2=2$
	1. $i_1 = 3, i_2=1$
	* Therefore, the unique elements from the union of all of those is
		$$ \mathcal{H}^{2,2} = \left\{(-1,1),(0,1),(1,1),(-1,0),(0,0),(1,0),(-1,-1),(0,-1),(1,-1),\left(\frac{-1}{\sqrt{2}},0\right),\left(\frac{1}{\sqrt{2}},0\right),\left(0,\frac{-1}{\sqrt{2}}\right),\left(0,\frac{1}{\sqrt{2}}\right)\right\} $$
* Note that those elements are on the diagonal from top left to bottom right expanding through all the tensor products on table 3.

## Size of Smolyak Grids

* The Smolyak grid grows much slower (at order $d$ to a power of $\mu$) than the Tensor grid (exponential growth)

![[@maliar-maliar] figure 2: Tensor vs Smolyak in 2D](../assets/figs/smolyak-vs-tensor.png)

![[@maliar-maliar] figure 4: Tensor vs Smolyak in 2D, number of grid points](../assets/figs/smolyak-tensor-points.png)

## Smolyak Polynomials

* Corresponding to the construction of grid points, there is the Smolyak way of constructing polynomials.
* This works exactly as before. We start with a one-dimensional set of basis functions (again Chebyshev here, again irrelevant):
	$$ \left\{1,x,2x^2-1,4x^3-3x,8x^4-8x^2+1\right\} $$
* Three nested sets:

$$ \begin{array}{l}
    i=1 : S_1 = \{1\} \\
    i=2 : S_2 = \{1,x,2x^2-1\} \\
    i=3 : S_3 = \left\{1,x,2x^2-1,4x^3-3x,8x^4-8x^2+1\right\}  \end{array}
$$

* Denoting $\mathcal{P}^{d,\mu}$ the Smolyak polynomial, we follow exactly the same steps as for the grids to select elements of the full tensor product table 5:

![[@maliar-maliar] figure 5: All Smolyak Polynomials in 2D](../assets/figs/smolyak-polynomial.png)

## Smolyak Interpolation

This proceeds as in the previouses cases:

1. Evaluate $f$ at all grid points $\mathcal{H}^{d,\mu}$.
1. Evaluate the set of basis functions given by $\mathcal{P}^{d,\mu}$ at all grid points $\mathcal{H}^{d,\mu}$.
1. Solve for the interpolating coefficients by inverting the Basis function matrix.

## Extensions

* There is a lot of redundancy in computing the grids the way we did it.
* More sophisticated approaches take care not to compute repeated elements.

## Smolyak Grids in Julia

* There are at least 2 julia packages that implement this idea:
	* [https://github.com/QuantEcon/BasisMatrices.jl](BasisMatrices.jl)
	* [https://github.com/RJDennis/SmolyakApprox.jl](SmolyakApprox.jl)

# Using (Smolyak from) `BasisMatrices`

* `BasisMatrices` provides general support for all kinds of basis matrices.
* `Cheb` for chebyshev basis matrix
* `Spline` for spline
* `Lin` for linear

### Outline

* We usually start with defining `Params` for each type: bounds, grid points, degrees, etc
* Then we construct a Basis type `basis`
* Then we can get evaluation nodes, i.e. points at which to evaluate our function with `nodes(basis)`
* Then get an evaluted basis matrix with `BasisMatrix(basis)`

```julia
using BasisMatrices
grid1 = range(1,stop = 3,length = 10)
lp = LinParams(grid1)
sp = SplineParams(collect(grid1),0,3)  # nodes, whether only 2 nodes, degree of spline
sm = SmolyakParams(2,1,[3],[2])   # dims,mu,lower bound(s), upper bound(s)
sm2 = SmolyakParams(2,[1,2],[0,-0.5],[3,4.4])   # dims,mu,lower bound(s), upper bound(s)
sm3 = SmolyakParams(3,[1,2,2],[0,-0.5,1],[3,4.4,5])   # dims,mu,lower bound(s), upper bound(s)
```

    Smolyak interpolation parameters in 3 dimensions from [0.0, -0.5, 1.0] × [3.0, 4.4, 5.0]

```julia
# simple interpolation
fi(x) = x.^2 - 0.5*x.^3

sp = SplineParams(collect(grid1),0,3)  # nodes, whether only 2 nodes, degree of spline
sb = Basis(sp)
s,n = BasisMatrices.nodes(sb)
sv =fi(s)
coef, bs_direct = funfitxy(sb, s, sv)   # compute coefficients
plot(fi,1,3,label="truth")
newx = 1 .+ rand(10)*2
newy = funeval(coef,sb,newx)   # evaluate basis at new points
scatter!(newx,newy,label="approx")
```

```julia
# multiple dimensions
basis = Basis(SplineParams(25, -1, 1, 2),SplineParams(20, -1, 2, 1))   # quadratic and linear spline
# get nodes
X, x12 = BasisMatrices.nodes(basis)

# function to interpolate
f2(x1, x2) = cos.(x2) ./ exp.(x1)
f2(X::Matrix) = f2.(X[:, 1], X[:, 2])

# f at nodes
y = f2(X)
Ymat = [f2(i,j) for i in x12[1], j in x12[2]]

coefs, bbs = funfitxy(basis, X, y)
ynew = funeval(coefs, basis, X[5:5, :])[1]
using Test
@test maximum(abs, ynew -  y[5]) <= 1e-12

# plotlyjs()
surface(x12[1],x12[2],Ymat',alpha=0.7,cbar=false)

xnew=hcat(-1 .+ rand(10)*2 , -1 .+ rand(10)*3)
ynew = [funeval(coefs, basis, xnew[i,:]) for i in 1:size(xnew,1)]
scatter!(xnew[:,1],xnew[:,2],ynew,markersize=2,markershape=:rect)

```

```julia
# same with the smolyak grid
# multiple dimensions
using LinearAlgebra
sp = SmolyakParams(2,3, [-1,-1],[1,2])
basis = Basis(sp)
# get nodes
X, x12 = BasisMatrices.nodes(basis)

# # f at nodes
y = f2(X)
Ymat = [f2(i,j) for i in unique(X[:,1]), j in unique(X[:,2])]

# map domain into unit hypercube
cube = BasisMatrices.dom2cube(X, basis.params[1])
# evaluate basis matrix
eb   = BasisMatrices.build_B(sp.d, sp.mu, cube, sp.pinds)
coef = pinv(eb) * y

xnew = hcat(-1 .+ rand(30)*2 , -1 .+ rand(30)*3)
cube = BasisMatrices.dom2cube(xnew, sp)
eb   = BasisMatrices.build_B(sp.d, sp.mu, cube, sp.pinds)
ynew = eb * coef

scatter(X[:,1],X[:,2],y,markersize=2)
scatter!(xnew[:,1],xnew[:,2],ynew,markersize=2,markershape=:rect)
```

## More on Sparse Grids

* Sparse Grids are widely used.
* The [Tasmanian](https://tasmanian.ornl.gov) library is excellent.
* The [Tasmanian.jl](https://github.com/floswald/Tasmanian.jl) is a simple wrapper to that library.
* [Simon Scheidegger](https://sites.google.com/site/simonscheidegger/home) has many useful resources on his website, in particular their joint paper with [Johannes Brumm](http://johannesbrumm.com) features an application of sparse grids for high dimensional economics problems.

```julia
using Tasmanian
Tasmanian.ex2()
```

    ┌ Info: error on initial grid:    0.06798, with 13 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:88
    ┌ Info: refinement level 1 error: 0.02096, with 16 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 2 error: 0.00896, with 36 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 3 error: 0.00314, with 80 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 4 error: 0.00097, with 176 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 5 error: 0.00031, with 384 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 6 error: 9.0e-5, with 824 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102
    ┌ Info: refinement level 7 error: 2.0e-5, with 1688 points
    └ @ Tasmanian /Users/florian.oswald/.julia/dev/Tasmanian/examples/examples.jl:102

    0

```julia

```


calculate  the scattering phase shifts and binding energy for N Omega (5S2) system
using fitted parameters

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
import seaborn as sns
sns.set_style('ticks', {'axes.grid': True})
sns.set_context('poster', font_scale=2.0)
%config InlineBackend.figure_format = 'retina'

plt.rcParams['figure.figsize'] = (12.8, 9.6)
plt.rcParams['figure.facecolor'] = 'white'
```

```python
import pickle
import scipy.optimize
import scipy.constants
```

```python
# use uncertainties library (pip install uncertaintes)
from uncertainties import ufloat
```

```python
from tqdm import tqdm_notebook as tqdm
```

```python
hbarc = (scipy.constants.hbar * scipy.constants.speed_of_light
         / scipy.constants.mega / scipy.constants.electron_volt / scipy.constants.femto)

ainv = 2.333e3
lat_unit = hbarc/ainv
momega = 1.71153e3/ainv
mn = 0.95469e3/ainv
mpi = 0.146e3/ainv
L = 96
bin_num = 20 # number of the jackknife sample
spin = 2
```

```python
m_red = mn * momega/(mn + momega)
```

```python
print(f'N = {mn*ainv:3.2f} [MeV], Omega = {momega*ainv:3.2f} [MeV]')
print(f'redueced mass = {m_red*ainv:3.2f} [MeV]')
```

    N = 954.69 [MeV], Omega = 1711.53 [MeV]
    redueced mass = 612.85 [MeV]

```python
V_fit = lambda p, x: p[0] * np.exp(-p[1]*x**2) + p[2] *  (
                1 - np.exp(-p[3]*x**2)) * (np.exp(-mpi*x)/(x+1.0e-5))**2
```

load fit paremeter (Run "N Omega potential and fit.ipynb" before using this notebook)

```python
with open('pkls/FF_yukawa2_fit_result.pkl', 'rb') as fin:
    FF_yukawa2_prm_av, FF_yukawa2_prm_jk = pickle.load(fin)
```

check parameter

```python
it = 12
xls = np.linspace(0.01, 3.10, 100)/lat_unit
fig, ax = plt.subplots()
ax.plot(xls*lat_unit, V_fit(FF_yukawa2_prm_av[it], xls)*ainv, lw=5)
ax.axhline(0, color='black')
ax.set_xlabel(r'$r$ [fm]', size=48)
ax.set_ylabel(r'$V_\mathrm{C}(r)$ [MeV]', size=48)
ax.set_xlim(-0.10, 3.10)
ax.set_title(rf'$N\Omega$($^{{{spin*2+1:d}}}$S$_{{{spin:d}}}$), $t/a = {it:d}$', size=58, y=1.02)

```

Schroedinger eq.

```python
class Schroedinger_Eq(object):
    """Calculate the scattering phase shifts by solving Schroedinger Eq.

    All parameters are in lattice unit.
    """
    def __init__(self, pot, red_mass, r_min = 1.0e-5, r_max = 10/lat_unit, r_diff = 1.0e-3):
        self.pot = pot
        self.rs = np.arange(r_min, r_max, r_diff)
        self.red_mass = red_mass

    def sch_eq(self, uv, r, k2):
        """
        u = psi(r)/r with u(0) = 0
        solve
        u'' = - 2 mu (E - V(r)) u
        -->
        u' = v
        v' = - (k^2 - 2 mu V(r)) u
        """
        du = uv[1]
        dv = - (k2 - 2 * self.red_mass * self.pot(r)) * uv[0]
        return np.array([du, dv])

    def conv_delta(self, r, u, u_prime, k):
        return np.mod(np.arctan(k*u/u_prime) - k*r, np.pi)

    def plot_asymptotic_behavior(self, k2, u_prime_ini=1.0e-3):
        """
        This method checks convergence of the phase shift.
        """
        trj = scipy.integrate.odeint(self.sch_eq, (0, u_prime_ini), self.rs,
                            args=(k2,))
        fig, ax = plt.subplots()
        ax.plot(self.rs, self.conv_delta(self.rs, trj[:,0], trj[:,1], np.sqrt(k2)),
               label=r'$k^2 = {}$'.format(k2))
        ax.set_xlabel(r'$r$', size=48)
        ax.set_ylabel(r'$\delta$ [rad.]', size=48)
        leg = ax.legend(frameon=True)
        leg.get_frame().set_edgecolor('black')
        leg.get_frame().set_linewidth(2.0)
        return ax

    def calc_phase_shift(self, k2, u_prime_ini=1.0e-3):
        trj = scipy.integrate.odeint(self.sch_eq, (0, u_prime_ini), self.rs,
                            args=(k2,))

        return self.conv_delta(self.rs[-1], trj[-1,0], trj[-1,1], np.sqrt(k2))
```

```python
def eval_deltas(k2s, pot):
    scheq = Schroedinger_Eq(pot, m_red)
    deltas =  []
    for k2 in k2s:
        deltas.append(scheq.calc_phase_shift(k2))

    return np.array(deltas)
```

```python
# calculate the scattering phase shifts
deltas_ff_yukawa2_jk = {}
deltas_ff_yukawa2_av = {}
k2s = np.linspace(1.0e-5, 1.25, 100)*mpi**2
k2s_ff_yukawa2 = k2s
for it in tqdm([11, 12, 13, 14]):
    pot = lambda x: V_fit(FF_yukawa2_prm_av[it], x)
    deltas_ff_yukawa2_av[it] = eval_deltas(k2s, pot)
    _delta = []
    for ibin in range(bin_num):
        pot = lambda x: V_fit(FF_yukawa2_prm_jk[it][ibin,:], x)
        _delta.append(eval_deltas(k2s, pot))
    deltas_ff_yukawa2_jk[it] = np.array(_delta)
```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

```python
mkdir -p ../figs/phase_shifts
```

```python
fig, ax = plt.subplots()
for it, col, fmt, ith in zip([14, 13, 12, 11], ['black', 'green', 'red', 'blue'], ['v', 'o', 's', '^'], [0,1,2,3]):
    ys_jk = np.sqrt(k2s)/np.tan(deltas_ff_yukawa2_jk[it])/mpi
    ax.errorbar(k2s[::4]/mpi**2, ys_jk.mean(axis=0)[::4],
                ys_jk.std(axis=0)[::4] * np.sqrt(ys_jk.shape[0] - 1),
                fmt=fmt, color=col, ms=15, mfc='none', mew=2.0, mec=col, capsize=10, capthick=2,
                zorder=3, label=r'$t/a = {}$'.format(it))
ax.set_xlim(-0.05, 1.05)
ax.axhline(0, color='black')
ax.axvline(0, color='black', linestyle='--')
ax.set_xlabel(r'$(k/m_\pi)^2$', size=48)
ax.set_ylabel(r'$k\cot\delta_0/m_\pi$', size=48)
ax.set_title(r'$N\Omega$($^5$S$_2$)', size=52, y=1.02)

leg = ax.legend(frameon=True)
leg.get_frame().set_edgecolor('black')
leg.get_frame().set_linewidth(2.0)
ax.set_ylim(-0.45, 0.45)
fig.savefig('../figs/phase_shifts/kcot_delta_nomega_t11_14.pdf', bbox_inches='tight')
```

```python
fig, ax = plt.subplots()
for it, col, fmt in zip([14, 13, 12, 11], ['black', 'green', 'red', 'blue'], ['v', 'o', 's', '^']):
    ys_jk = deltas_ff_yukawa2_jk[it]*180/np.pi
    mask = ys_jk < 0
    ys_jk[mask] += 180
    ax.errorbar(k2s[:]/(2.0*m_red)*ainv, ys_jk.mean(axis=0)[:],
                ys_jk.std(axis=0)[:] * np.sqrt(ys_jk.shape[0] - 1),
                fmt=fmt, color=col, ms=15, mfc='none', mew=2.0, mec=col, capsize=10, capthick=2,
                zorder=3, label=r'$t/a = {}$'.format(it))
ax.set_xlim(-0.5, 20.5)
ax.set_ylim(-5, 185)
ax.set_yticks([0,30,60,90,120,150,180])
ax.axhline(0, color='black')
ax.set_xlabel(r'$k^2/2m$ [MeV]', size=48)
ax.set_ylabel(r'phase shift $\delta_0$ [deg]', size=48)
leg = ax.legend(frameon=True)
leg.get_frame().set_edgecolor('black')
leg.get_frame().set_linewidth(2.0)
ax.set_title(r'$N\Omega$($^5$S$_2$)', size=58, y=1.02)
fig.savefig('../figs/phase_shifts/delta_nomega_t11_14.pdf', bbox_inches='tight')
```

# effective range expansion

\begin{equation}
k\cot\delta = - \frac{1}{a_0} + \frac{1}{2}r_\mathrm{eff} k^2
\end{equation}

```python
ere_fit_ff_yukawa2 = {}
for it in [11, 12, 13, 14]:
    ere = lambda p, x: p[0] + 0.5*p[1]*x
    errf = lambda p, y, x, err: (ere(p,x)-y)/err

    ys_jk = np.sqrt(k2s_ff_yukawa2)/np.tan(deltas_ff_yukawa2_jk[it])/mpi
    xs = k2s_ff_yukawa2/mpi**2
    ys = ys_jk.mean(axis=0)
    yerrs = ys_jk.std(axis=0)*np.sqrt(ys_jk.shape[0] - 1)
    ere_fit_prm = []
    for ibin in range(bin_num):
        mask = xs < 0.4
        pfit = scipy.optimize.leastsq(errf, [-5, 0.1],
                                 args=(ys_jk[ibin,:][mask], xs[mask], yerrs[mask]), full_output=True)
        ere_fit_prm.append(pfit[0])

    ere_fit_ff_yukawa2[it] = np.array(ere_fit_prm)
```

check ERE paramters

```python
fig, ax = plt.subplots()

print('t  a0     reff')
for it, col, fmt in zip([11, 12, 13, 14], ['red', 'blue', 'green', 'black'], ['^', 's', 'o', 'v']):
    xs = -1.0/(ere_fit_ff_yukawa2[it][:,0]*mpi)*lat_unit
    ys = ere_fit_ff_yukawa2[it][:,1]/mpi*lat_unit
    ax.errorbar(xs.mean(), ys.mean(),
               ys.std()*np.sqrt(bin_num-1), xerr=xs.std()*np.sqrt(bin_num-1),
               fmt=fmt, color=col, label=r'$t = {}$'.format(it),
                ms=25,  capthick=2, capsize=10)
    print('{} {:4.3f} {:4.3f} {:4.3f} {:4.3f}'.format(it,
            xs.mean(), xs.std()*np.sqrt(bin_num-1),
    ys.mean(), ys.std()*np.sqrt(bin_num-1)))

ax.set_xlabel(r'$a_0$ [fm]', size=48)
ax.set_ylabel(r'$r_\mathrm{eff}$ [fm]', size=48)
ax.set_xlim(-0.5, 10.5)
ax.set_ylim(1.2, 1.4)
ax.set_yticks([1.2, 1.25, 1.3, 1.35, 1.4])
leg = ax.legend(frameon=True)
leg.get_frame().set_edgecolor('black')
leg.get_frame().set_linewidth(2.0)

```

summarize ERE parameter

```python
def plot_ere_prm_nomega_5s2(ax):
    xs = -1.0/(ere_fit_ff_yukawa2[12][:,0]*mpi)*lat_unit
    ys = ere_fit_ff_yukawa2[12][:,1]/mpi*lat_unit

    xs_max = np.max([(-1.0/(ere_fit_ff_yukawa2[it][:,0]*mpi)*lat_unit).mean() for it in [11, 12, 13, 14]])
    xs_min = np.min([(-1.0/(ere_fit_ff_yukawa2[it][:,0]*mpi)*lat_unit).mean() for it in [11, 12, 13, 14]])

    ys_max = np.max([(ere_fit_ff_yukawa2[it][:,1]/mpi*lat_unit).mean() for it in [11, 12, 13, 14]])
    ys_min = np.min([(ere_fit_ff_yukawa2[it][:,1]/mpi*lat_unit).mean() for it in [11, 12, 13, 14]])

    yerrs = ys.std()*np.sqrt(bin_num-1)
    xerrs = xs.std()*np.sqrt(bin_num-1)

    print('{:3.2f}({:3.2f})({:+3.2f})({:+3.2f})'.format(xs.mean(), xerrs, (xs_max-xs.mean()), (xs_min - xs.mean())))
    print('{:3.2f}({:3.2f})({:+3.2f})({:+3.2f})'.format(ys.mean(), yerrs, (ys_max-ys.mean()), (ys_min - ys.mean())))
    xvals = xs.mean(), xerrs, (xs_max-xs.mean()), (xs_min - xs.mean())
    yvals = ys.mean(), yerrs, (ys_max-ys.mean()), (ys_min - ys.mean())
    result = r'$a_0 = {:3.2f}({:3.2f})\left(^{{{:+3.2f}}}_{{{:+3.2f}}}\right)$ fm'.format(xvals[0], xvals[1], xvals[2], xvals[3]) \
          + '\n' + r'$r_\mathrm{{eff}} = {:3.2f}({:3.2f})\left(^{{{:+3.2f}}}_{{{:+3.2f}}}\right)$ fm'.format(yvals[0], yvals[1], yvals[2], yvals[3])

    # outer errorbar (with systematic)
    eb = ax.errorbar(xs.mean(), ys.mean(), [[np.sqrt(yerrs**2 + (ys_min-ys.mean())**2)],
                                       [np.sqrt(yerrs**2 + (ys_max-ys.mean())**2)]],
               xerr=[[np.sqrt(xerrs**2 + (xs_min-xs.mean())**2)],
                    [np.sqrt(xerrs**2 + (xs_max-xs.mean())**2)]],
               fmt='o', color='red', ms=25, capthick=2.0, capsize=10,
                    label=result)
    eb[-1][0].set_linestyle('--')
    eb[-1][1].set_linestyle('--')

    ax.errorbar(xs.mean(), ys.mean(), ys.std()*np.sqrt(bin_num-1),
               xerr=xs.std()*np.sqrt(bin_num-1),
               fmt='o', color='red', ms=25, capthick=2.0, capsize=10)

```

```python
fig, ax = plt.subplots()
plot_ere_prm_nomega_5s2(ax)

ax.set_xlabel(r'$a_0$ [fm]', size=48)
ax.set_ylabel(r'$r_\mathrm{eff}$ [fm]', size=48)
leg = ax.legend(frameon=True)
leg.get_frame().set_edgecolor('black')
leg.get_frame().set_linewidth(2.0)
ax.set_xlim(-0.5, 10.5)
ax.set_ylim(1.18, 1.42)
ax.set_title(r'$N\Omega$($^5$S$_2$)', size=58, y=1.02)

ax.set_yticks(np.arange(1.2, 1.41, 0.05));
```

compare other dibarions

```python
omega_ys = [0.216, 0.278, 0.328]
omega_xs = [1.231, 1.272, 1.339]
```

```python
fig, ax = plt.subplots()

ax.errorbar(omega_xs[1], omega_ys[1],
           yerr=[[omega_ys[1]-omega_ys[0]], [-omega_ys[1]+omega_ys[2]]],
           xerr=[[omega_xs[1]-omega_xs[0]], [-omega_xs[1]+omega_xs[2]]],
           ms=20, fmt='D', color='blue', capthick=2.0, capsize=10,
            mfc='none', mew=2.0, mec='blue',
           label=r'$\Omega\Omega$($^1$S$_0$)')

print('OmegaOmega reff = {:3.2f}({:+3.2f})({:+3.2f}) fm'.format(
    omega_xs[1], omega_xs[2] - omega_xs[1], omega_xs[0] - omega_xs[1]))
print('OmegaOmega ratio = {:3.2f}({:+3.2f})({:+3.2f})'.format(
    omega_ys[1], omega_ys[2] - omega_ys[1], omega_ys[0] - omega_ys[1]))

# n omega (5S2)
ys = -ere_fit_ff_yukawa2[12][:,0] * ere_fit_ff_yukawa2[12][:,1]
xs = ere_fit_ff_yukawa2[12][:,1]/mpi*lat_unit

ys_max = (-ere_fit_ff_yukawa2[13][:,0]* ere_fit_ff_yukawa2[13][:,1]).mean()
ys_min = (-ere_fit_ff_yukawa2[11][:,0]* ere_fit_ff_yukawa2[11][:,1]).mean()

xs_max = (ere_fit_ff_yukawa2[11][:,1]/mpi*lat_unit).mean()
xs_min = (ere_fit_ff_yukawa2[13][:,1]/mpi*lat_unit).mean()
yerrs = ys.std()*np.sqrt(bin_num-1)
xerrs = xs.std()*np.sqrt(bin_num-1)

eb = ax.errorbar(xs.mean(), ys.mean(),
                 [[np.sqrt(yerrs**2 + (ys_min-ys.mean())**2)], [np.sqrt(yerrs**2 + (ys_max-ys.mean())**2)]],
           xerr=[[np.sqrt(xerrs**2 + (xs_min-xs.mean())**2)], [np.sqrt(xerrs**2 + (xs_max-xs.mean())**2)]],
           fmt='o', color='red', ms=20, capthick=2.0, capsize=10, mec='red', mew=2.0, mfc='none')
eb[-1][0].set_linestyle('--')
eb[-1][1].set_linestyle('--')

ax.errorbar(xs.mean(), ys.mean(), ys.std()*np.sqrt(bin_num-1),
           xerr=xs.std()*np.sqrt(bin_num-1),
           fmt='o', color='red', ms=20, capthick=2.0, capsize=10,
           mfc='none', mec='red', mew=2.0, label=r'$N\Omega$($^5$S$_2$)')

print('NOmega reff = {:3.2f}({:3.2f})({:3.2f})({:3.2f}) fm'.format(
        xs.mean(), xerrs, xs_max - xs.mean(), xs_min - xs.mean()))
print('NOmega ratio = {:3.2f}({:3.2f})({:3.2f})({:3.2f})'.format(
        ys.mean(), yerrs, ys_min - ys.mean(), ys_max - ys.mean()))

ax.errorbar([1.7436], [1.7436/5.4112], [0], xerr=[0], fmt='^', ms=25, color='purple',
            mfc='none', mec='purple', mew=2.0,
       label=r'$NN$($^3$S$_1$)', capthick=2.0, capsize=10)
print('1S0 reff = {:3.2f} fm, ratio = {:3.2f}'.format(1.7436, 1.7436/5.4112))

ax.errorbar([2.750], [-2.750/23.7148], [0], xerr=[0], fmt='v', ms=25, color='green',
            mfc='none', mec='green', mew=2.0,
       label=r'$NN$($^1$S$_0$)', capthick=2.0, capsize=10)
print('3S1 reff = {:3.2f} fm, ratio = {:3.2f}'.format(2.750, -2.750/23.7148))

ax.set_ylabel(r'$r_\mathrm{eff}/a_0$', size=48)
ax.set_xlabel(r'$r_\mathrm{eff}$ [fm]', size=48)

leg = ax.legend(ncol=2, frameon=True)
leg.get_frame().set_edgecolor('black')
leg.get_frame().set_linewidth(2.0)
ax.set_xlim(1-0.1, 3+0.1)
#    ax.set_ylim(-0.25, 0.45)
ax.set_ylim(-0.45,1.05)
ax.axhline(0, color='black')

```

```python

```


# Wasserstein GAN for Image Generation

[](https://colab.research.google.com/github/EmoryMLIP/DeepGenerativeModelingIntro/blob/main/examples/WGAN.ipynb)

## Some References

- Original paper on by [Arjovsky et al. (2017)](https://arxiv.org/abs/1701.07875)
- Section 5.2 of our paper  [Introduction to Deep Generative Modeling](https://arxiv.org/abs/2103.05180)

## Short Mathematical Description

In training we approximately solve the saddle point problem

\begin{equation}
\newcommand{\E}{\mathbb{E}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\bftheta}{\boldsymbol{\theta}}
\newcommand{\bfphi}{\boldsymbol{\phi}}
\newcommand{\calX}{\mathcal{X}}
\newcommand{\calZ}{\mathcal{Z}}
\newcommand{\bfx}{\mathbf{x}}
\newcommand{\bfz}{\mathbf{z}}
\newcommand{\bfy}{\mathbf{y}}
\min_{\bftheta} \max_{\bfphi} \E_{\bfz \sim \calZ}\left[f_{\phi}\left(g_{\bftheta}(\bfz)\right)\right] - \E_{\bfx \sim \calX}\left[f_{\phi}(\bfx)\right] \quad \text{ subject to } \quad  f_{\bfphi} \in {\rm Lip}(f)\leq 1
\end{equation}

where

- $g_{\bftheta} : \R^2 \to \R^{784}$ is the generator (here a convolutional neural network)
- $f_{\phi} : \R^{784} \to \R^2$ is the discriminator (here a convolutional neural network)

```python
# install requirements  (Colab only)
import sys,os
if 'google.colab' in sys.modules:
    dgm_dir = '/content/DeepGenerativeModelingIntro/'
    if not os.path.exists(dgm_dir):
        !git clone git://github.com/EmoryMLIP/DeepGenerativeModelingIntro.git

    sys.path.append(os.path.dirname(dgm_dir))#%%
```

```python
import torch.nn.functional as F
import torch.nn as nn
import torch
import torchvision
import numpy as np
import matplotlib.pyplot as plt
from os import path

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
plt.rcParams.update({'font.size': 16})
plt.rcParams.update({'image.interpolation' : None})
# plt.rcParams['figure.figsize'] = [10, 6]
# plt.rcParams['figure.dpi'] = 200
```

## Prepare Image Data

```python
batch_size = 64

import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST

img_transform = transforms.Compose([
    transforms.ToTensor()
])

train_dataset = MNIST(root='./data/MNIST', download=True, train=True, transform=img_transform)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = MNIST(root='./data/MNIST', download=True, train=False, transform=img_transform)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

x,_ = next(iter(train_dataloader))
plt.Figure()
plt.imshow(torchvision.utils.make_grid(x,16,padding=5,pad_value=1.0).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.show()
```

## Select GAN Architecture

```python
class Generator(nn.Module):
    def __init__(self,w,q):
        """
        :param w: number of channels on the finest level
        :param q: latent space dimension
        """
        super(Generator, self).__init__()
        self.w = w
        self.fc = nn.Linear(q, w * 2 * 7 * 7)
        self.conv2 = nn.ConvTranspose2d(w * 2, w, kernel_size=4, stride=2, padding=1)
        self.conv1 = nn.ConvTranspose2d(w, 1, kernel_size=4, stride=2, padding=1)

        self.bn1 = nn.BatchNorm2d(w)
        self.bn2 = nn.BatchNorm2d(2*w)

    def forward(self, z):
        """
        :param z: latent space sample
        :return: g(z)
        """
        gz = self.fc(z)
        gz = gz.view(gz.size(0), self.w * 2, 7, 7)
        gz = self.bn2(gz)
        gz = F.relu(gz)
        gz = self.conv2(gz)
        gz = self.bn1(gz)

        gz = F.relu(gz)
        gz = torch.sigmoid(self.conv1(gz))
        return gz

class Discriminator(nn.Module):
    def __init__(self, w,useSigmoid=True):
        """
        Discriminator for GANs
        :param w: number of channels on finest level
        :param useSigmoid: true --> DCGAN, false --> WGAN
        """
        super(Discriminator, self).__init__()
        self.w = w
        self.useSigmoid = useSigmoid
        self.conv1 = nn.Conv2d(1, w, kernel_size=4, stride=2, padding=1)
        self.conv2 = nn.Conv2d(w, w * 2, kernel_size=4, stride=2, padding=1)
        self.fc = nn.Linear(w * 2 * 7 * 7, 1)
        self.bn1 = nn.BatchNorm2d(w)
        self.bn2 = nn.BatchNorm2d(2*w)

    def forward(self,x):
        """
        :param x: MNIST image or generated image
        :return: d(x), value of discriminator
        """
        x = (x-0.5)/0.5 #
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.leaky_relu(x,0.2)
        x = self.conv2(x)
        x = self.bn2(x)

        x = F.leaky_relu(x,0.2)
        x = x.view(x.shape[0],-1)
        x = self.fc(x)
        if self.useSigmoid:
            x = torch.sigmoid(x)
        return x

q = 2 # latent space dimension
width_disc = 32 # width of discriminator
width_dec = 32 # width of decoder

g = Generator(width_dec,q).to(device)
d = Discriminator(width_disc,useSigmoid=True).to(device)
```

## Train the Generator and Discriminator

We alternate between RMSprop steps of the generator and the discriminator but here it is often suggested
to use more than one step for the discriminator.

```python
optimizer_g = torch.optim.RMSprop(g.parameters(), lr=0.00005)
optimizer_d = torch.optim.RMSprop(d.parameters(), lr=0.00005)
his = np.zeros((0,3))

num_steps = 10000 # number of training steps
plot_interval = 1000  # plot solution every so many steps

init_g = "./results/VAEmnist-q-2-batch_size-%d-w_enc-%d-w_dec-%d-g.pt" % (batch_size,width_disc,width_dec) # path to .pt file that contains weights of a trained generator
out_file = "./results/WGANmnist-q-2-batch_size-%d-w_disc-%d-w_dec-%d" % (batch_size,width_disc,width_dec)
  # base filename saving trained model (extension .pt), history (extension .mat), and intermediate plots (extension .png)
iter_disc = 1 # no. of iterations for training the discriminator
clip_limit = 1e-2 # limit for weights of discriminator
retrain = False

if init_g is not None and path.exists(init_g):
    print("initialize g with weights in %s" % init_g)
    g.load_state_dict(torch.load(init_g ,map_location=torch.device(device)))

if retrain == False and path.exists(out_file + "-g.pt") and path.exists(out_file + "-d.pt") and path.exists(out_file + ".mat"):
    g.load_state_dict(torch.load("%s-g.pt" % (out_file),map_location=torch.device(device)))
    d.load_state_dict(torch.load("%s-d.pt" % (out_file),map_location=torch.device(device)))
    from scipy.io import loadmat
    his_file = loadmat("%s.mat" % (out_file))
    his = his_file['his']
    print("loaded results from %s" % out_file)

else:
    print((3*"--" + "device=%s, q=%d, batch_size=%d, num_steps=%d, w_disc=%d, w_dec=%d" + 3*"--") % (device, q, batch_size, num_steps, width_disc, width_dec))
    if out_file is not None:
        import os
        out_dir, fname = os.path.split(out_file)
        if not os.path.exists(out_dir):
            os.makedirs(out_dir)
        print((3*"--" + "out_file: %s" + 3*"--") % (out_file))

    print((4*"%7s    ") % ("step","J_GAN","J_Gen","ProbDist"))

    from epsTest import epsTest

    train_JGAN = 0.0
    train_JGen = 0.0
    train_epsTest = 0.0
    num_ex = 0

    def inf_train_gen():
        while True:
            for images, targets in enumerate(train_dataloader):
                yield images,targets

    get_true_images = inf_train_gen()

    for step in range(num_steps):
        g.train()
        d.train()
        # update discriminator using ascent on J_GAN =  E_x [d(x)] - E_z[d(g(z))]
        for iter_critic in range(iter_disc):
            x = get_true_images.__next__()[1][0]
            x = x.to(device)

            for p in d.parameters():
                p.data.clamp_(-0.01, 0.01)

            dx = d(x)
            z = torch.randn((x.shape[0],q),device=device)
            gz = g(z)
            dgz = d(gz)
            J_GAN = -(torch.mean(dx) - torch.mean(dgz))
            optimizer_d.zero_grad()
            J_GAN.backward()
            optimizer_d.step()
            train_JGAN -= J_GAN.item() * x.shape[0]

        # update the generator using descent on J_Gen = - E_z[d(g(z))]
        optimizer_g.zero_grad()
        z = torch.randn((x.shape[0], q), device=device)
        gz = g(z)
        dgz = d(gz)
        J_Gen = -torch.mean(dgz)
        J_Gen.backward()
        optimizer_g.step()

        # update history
        train_JGen += J_Gen.item()*x.shape[0]
        train_epsTest += epsTest(gz.detach(),x)

        num_ex += x.shape[0]

        if (step+1) % plot_interval==0:
            train_JGAN /= iter_disc * num_ex
            train_JGen /= num_ex

            print(("%06d   " + 3 * "%1.4e  ") %
                  (step + 1, train_JGAN, train_JGen, train_epsTest))
            his = np.vstack([his, [train_JGAN, train_JGen, train_epsTest]])

            plt.Figure()
            img = gz.detach().cpu()
            img -= torch.min(img)
            img /= torch.max(img)
            plt.imshow(torchvision.utils.make_grid(img, 8, 5).permute((1, 2, 0)))
            plt.title("trainWGANmnist: step=%d" % (step+1))
            if out_file is not None:
                plt.savefig(("%s-step-%d.png") % (out_file,step+1))
            plt.show()

            train_JGAN = 0.0
            train_JGen = 0.0
            train_epsTest = 0.0

            num_ex = 0

    if out_file is not None:
        torch.save(g.state_dict(), ("%s-g.pt") % (out_file))
        torch.save(d.state_dict(), ("%s-d.pt") % (out_file))

        from scipy.io import savemat
        savemat(("%s.mat") % (out_file), {"his":his})
```

    initialize g with weights in ./results/VAEmnist-q-2-batch_size-64-w_enc-32-w_dec-32-g.pt
    loaded results from ./results/WGANmnist-q-2-batch_size-64-w_disc-32-w_dec-32

## Convergence

We plot the GAN loss and the loss of the generator on the left.
Since we are solving a saddle point problem this plot can be difficult to read.
Therefore, we also plot the results of an $\epsilon$ test for equal distribution as an alternative way to meausre the quality of the generator (see [Szekely and Rizzo (2004)](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.226.377))

```python
plt.Figure()
plt.subplot(1,2,1)
plt.plot(his[:,0:2])
plt.legend(("JGAN","JGen"))
plt.title("GAN Objectives")

plt.subplot(1,2,2)
plt.plot(his[:,2])
plt.title("eps test")

```

## Show Samples

```python
ncol = 8
x, _ = next(iter(test_dataloader))
plt.Figure()
plt.subplot(1,2,1)
plt.imshow(torchvision.utils.make_grid(x,ncol,1,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.title("original images")

z = torch.randn((64,q),device=device)
gz = g(z)
plt.subplot(1,2,2)
plt.imshow(torchvision.utils.make_grid(gz.cpu().detach(),ncol,1,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.title("generated images")
plt.show()
```

## Interpolate in Latent Space

```python
z1 = -torch.ones(1,2,device=device)
z2 = torch.ones(1,2,device=device)
with torch.no_grad():

    lam = torch.linspace(0,1,20,device=device).reshape(-1,1)
    zinter = z1*(1-lam) + z2*lam

    gz = g(zinter)

plt.Figure()
plt.imshow(torchvision.utils.make_grid(gz.cpu(),len(lam),2,pad_value=1).permute((1,2,0)),cmap='gray')
plt.axis("off")
plt.margins(0, 0)
plt.show()
```


Refresher Course on Matrix Analysis and Optimization

Franck Iutzeler Jerome Malick
 Fall. 2018

Chap. 2 - Refresher Course

# ``2. Numerical Optimization``

---

Package check and Styling

Outline

    a)  the Gradient Method
    b)  Application to Regression
    c)  an Adaptive Gradient Algorithm
    d)  Application to Classification
    e)  Newton's Algorithm
    f)  To go Further

##  a) the Gradient Method

Go to top

We consider two $\mathbb{R}^2 \to \mathbb{R}$ convex functions with the same global minimizer $(3,1)$ but quite different *shapes* and see how this impacts the performance of gradient-based algorithms. The considered functions $f$ and $g$ and their 3D are:

\begin{array}{rrcll}
f: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & 4 (x_1-3)^2 + 2(x_2-1)^2
\end{array}
 \begin{array}{rrcll}
g: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & \log( 1 + \exp(4 (x_1-3)^2 ) + \exp( 2(x_2-1)^2 ) ) - \log(3) .
\end{array}

**Note:** For convenience, we provide custom plotting functions in <tt>lib/custom_plot_lib.py</tt>

```python
from lib.custom_plot_lib import *
```

 Question 1: Observe how is written the function <tt>f</tt> that return $f(x)$ from input vector $x$. Observe the 3D plot and level plot with <tt>custom_3dplot...</tt> and <tt>level_plot...</tt>. Do the same for  function <tt>g</tt>.

```python
# f
def f(x):
    """
    Function f
    """
    x1 = x[0]
    x2 = x[1]
    return 4*(x1-3)**2+2*(x2-1)**2

f_plot_param  = {'x1_min' : -0.5, 'x1_max' : 5.5,
                 'x2_min' : -0.5, 'x2_max' : 5.5,
                 'nb_points' : 200,
                 'v_min' : 0, 'v_max' : 80, 'levels' : [0.5,1,2,5,10,15],
                 'title' : 'f: a simple function' }

```

```python
custom_3dplot( f, f_plot_param )
level_plot( f, f_plot_param )
```

```python
def g(x):
    return 0 ## To complete

g_plot_param  = {'x1_min' : -0.5, 'x1_max' : 5.5,
                 'x2_min' : -0.5, 'x2_max' : 5.5,
                 'nb_points' : 500,
                 'v_min' : 0, 'v_max' : 100, 'levels' : [0.5,1,2,5,10,15],
                 'title' : 'g: a harder function' }

```

```python
custom_3dplot( g, g_plot_param )
#level_plot( g, g_plot_param )
```

 Question 2:  Fill the function <tt>f_grad</tt> that return $\nabla f(x)$ from input vector $x$. Do the same for  function <tt>g_grad</tt>.

```python
def f_grad(x): # ...................................
    x1 = x[0]
    x2 = x[1]
    g = np.array( [ 0.0  ,  0.0  ] )
    return g
####
```

```python
def g_grad(x): # ...................................
    g = np.array( [ 0.0  ,  0.0  ] )
    return g
####
```

 Question 3:  implement a constant stepsize gradient method <tt>gradient_algorithm(f , f_grad , x0 , step , PREC , ITE_MAX )</tt> that takes:

 <tt>f</tt> and <tt>f_grad</tt>: respectively functions and gradient simulators;

 <tt>x0</tt>: starting point;

 <tt>step</tt>: a stepsize;

 <tt>PREC</tt> and <tt>ITE_MAX</tt>: stopping criteria for sought precision and maximum number of iterations;

and return <tt>x</tt>, the final value, and <tt>x_tab</tt>, the matrix of all vectors stacked vertically.

```python
def gradient_algorithm(f , f_grad , x0 , step , PREC , ITE_MAX ): # .......................................
    x = np.copy(x0)
    x_tab = np.copy(x)

    return x,x_tab
```

 Question 4: Test your gradient descent function on $f$ and $g$: i) Verify that the final point is close to the sought minimizer $(3,1)$; ii) observe the behavior of the iterates with <tt>level_points_plot</tt>. Change the stepsize and give the values for which the algorithm (i) diverges and (ii) oscillates. Compare with theoretical limits by computing the Lipschitz constant of the gradients.

```python
step =  0 # .........................
PREC = 1e-5 # .........................
ITE_MAX = 10 # .........................
x0 = np.array([0,0]) # .........................

x,x_tab = gradient_algorithm(f , f_grad , x0 , step , PREC , ITE_MAX )
level_points_plot( f , x_tab , f_plot_param )
```

```python
step =  0 # .........................
PREC = 1e-5 # .........................
ITE_MAX = 10 # .........................
x0 = np.array([0,0]) # .........................

x,x_tab = gradient_algorithm(g , g_grad , x0 , step , PREC , ITE_MAX )
level_points_plot( g , x_tab , g_plot_param )
```

##  b) Application to Regression

Go to top

We now get back to the problem of predicting the final grade of a student from various features treated in the Matrix part of the course.

We remind that mathematically, from the $m_{learn}  \times (n+1)$  *learning matrix* $A_{learn}$
($m_{learn} = 300$, $n=27$)  comprising of the features values of each training student in line, and the vector of the values of the target features $b_{learn}$;  we seek a size-$(n+1)$ *regression vector* that minimizes the squared error between  $A_{learn} x$ and $b_{learn}$. This problem boils down to the following least square problem:
$$ \min_{x\in\mathbb{R}^{n+1}} s(x) =  \frac{1}{2} \|  A_{learn} x - b_{learn} \|_2^2 . $$

```python
import numpy as np

# File reading
dat_file = np.load('data/student.npz')
A_learn = dat_file['A_learn']
b_learn = dat_file['b_learn']
A_test = dat_file['A_test']
b_test = dat_file['b_test']

m = 395 # number of read examples (total:395)
n = 27 # features
m_learn = 300
```

 Question 1: Construct the suitable function $s$ and gradient simulator as in the previous section.

```python

```

 Question 2: Compute the Lipschitz constant of the gradient of $s$. Find a solution to the minimization of $s$ using your gradient algorithm. Compare with Numpy's Least Square routine.

```python

```

 Question 3: Generate a random Gaussian matrix/vector couple $A,b$ with increasing size. Create simulators to compare the execution time of constant stepsize gradien and pseudo-inverse computation \emph{via} SVD on the least squares problem $\min_x \|Ax-b\|_2^2$. Notably change the *shape* of $A$ from *tall* (nb. of rows $>\!>$ nb. of cols.) to *fat* (nb. of rows $<\!<$ nb. of cols.).

```python

```

##  c) an Adaptive Gradient Algorithm

Go to top

The *Rosenbrock* function $r$ writes

\begin{array}{rrcll}
r: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  &  (1-x_1)^2 + 100(x_2-x_1^2)^2 .
\end{array}

 Question 1: Fill the functions <tt>r</tt> that return $r(x)$ from input vector $x$; and <tt>r_grad</tt> that return $\nabla r(x)$ from input vector $x$. Observe the 3D plot and level plot of the function.

```python
##### Definition of function r
def r(x):
    """ Rosenbrock."""
    x1 = x[0]
    x2 = x[1]
    return (1-x1)**2+100*(x2-x1**2)**2

r_plot_param  = {'x1_min' : -1.5, 'x1_max' : 1.55,
                 'x2_min' : -0.2, 'x2_max' : 1.5,
                 'nb_points' : 200,
                 'v_min' : 0, 'v_max' : 120, 'levels' : [0.05,1,5,15,50,200],
                 'title' : 'r: Rosenbrock function' }
```

```python
custom_3dplot( r,r_plot_param)
level_plot( r, r_plot_param)
```

```python
def r_grad(x): # .........................
    g = np.array( [ 0.0  ,  0.0  ] )
    return g
```

 Question 2: Try to minimize $r$ using your constant stepsize gradient function <tt>gradient_algorithm</tt>. Can you find a stepsize for which the algorithm converges?

```python

```

 Question 3: Implement an *adaptive* stepsize gradient method <tt>gradient_adaptive_algorithm(f , f_grad , x0 , step , PREC , ITE_MAX )</tt> that takes the same inputs and returns the same as the gradient method but implements a *stepsize adaptation method*. For instance, one can use this rule:
\begin{align*}
\mathbf{if} f(x_{k+1})>f(x_k)&:\\
x_{k+1} &= x_k\\
step &= step/2
\end{align*}
which halves the stepsize if a gradient step makes the functional value increase.

Test your method on $r$: i) Verify that the final point is close to the sought minimizer $(1,1)$; ii) observe the behavior of the iterates with <tt>level_points_plot</tt>.

```python
step =  0 # .........................
PREC = 1e-5 # .........................
ITE_MAX = 10 # .........................
x0 = np.array([0,0]) # .........................

x,x_tab = gradient_adaptive_algorithm(r , r_grad , x0 , step , PREC , ITE_MAX )
#level_points_plot( r , x_tab , r_plot_param)
```

##  d) Application to Classification

Go to top

Binary classification is another popular problem in machine learning. Instead of predicting a numerical value, the goal is now to classify the student into two classes: $+1$ -- *pass* i.e. final grade $\geq 10$; and $-1$ -- *fail*. To this purpose, we create a class vector $c_{learn}$  from the observation vector $b_{learn}$ by simply setting $c_{learn}(i) = +1 $ if  $b_{learn}(i)\geq10$ and $-1$ otherwise. Then, the most common approach is to minimize the logistic loss regularized by a squared norm:
\begin{equation}
\min_{x\in\mathbb{R}^{n+1}}  \ell(x) = \sum_{i=1}^{m_{learn} } \log\left( 1 + \exp\left( -c_{learn}(i) a_i^{\mathrm{T}}  x   \right) \right)  + \frac{1}{m}\|x\|^2
\end{equation}
where $a^\mathrm{T}_i$ is the $i$-th row of $A_{learn}$.

Then, from a solution $x^\star$ of this problem, one can classify a new example, represented by its feature vector $a$, as such: the quantity $p(a) = \frac{1}{1+\exp(- a^\mathrm{T} x^\star)}$ estimates the probability of belonging to class $1$; thus, one can decide class $+1$ if for instance $ p(a) \geq 0.5$; otherwise, decide class $-1$.

 Question 1: Compute the gradient of $q(t)  = \log(1+\exp(t))$. Is the function is convex? Deduce that $\ell$ is convex and its gradient.

 Question 2: Construct the suitable function and gradient simulators in order to use your <tt>gradient_adaptive_algorithm</tt> to minimize $\ell$.

```python

```

```python

```

```python

```

```python

```

 Question 3: From a final point of the optimization algorithm above, generate a decision vector corresponding to the testing set $A_{test}$. Evaluate the classification error.

```python

```

```python

```

##  e) Newton's Algorithm

Go to top

 Question 1: Fill the function <tt>f_grad_hessian</tt> that return $\nabla f(x)$ and $H_f(x)$ from input vector $x$. Same thing for $g$.

```python
def f_grad_hessian(x): # ...................................
    g = np.array( [ 0.0  ,  0.0  ] )
    H = np.array(  [ ( 0.0 , 0.0 )  ,  ( 0.0 , 0.0 )  ]  )
    return g,H
####
```

```python
def g_grad_hessian(x): # ...................................
    g = np.array( [ 0.0  ,  0.0  ] )
    H = np.array(  [ ( 0.0 , 0.0 )  ,  ( 0.0 , 0.0 )  ]  )
    return g,H
####
```

 Question 2: Implement Newton's method in <tt>newton_algorithm(f , f_grad_hessian , x0 , PREC , ITE\_MAX )</tt> that takes as an input:

 <tt>f</tt> and <tt>f_grad_hessian</tt>: respectively functions and gradient + Hessian simulators;

 <tt>x0</tt>: starting point;

 <tt>PREC</tt> and <tt>ITE_MAX</tt>: stopping criteria for sought precision and maximum number of iterations;

<tt>x</tt> the final value, and <tt>x_tab</tt>, the matrix of all vectors stacked vertically.

```python
def newton_algorithm(f , f_grad_hessian , x0 , PREC , ITE_MAX ): # .......................................
    x = np.copy(x0)
    x_tab = np.copy(x)

    return x,x_tab
```

 Question 3: Test your method on $f$ and $g$: i) Verify that the final point is close to the sought minimizer $(3,1)$; ii) observe the behavior of the iterates with <tt>level_points_plot</tt>.

Compare graphically constant stepsize gradient and Newton's algorithms with <tt>level_2points_plot</tt>.

Newton's algorithm should take exactly one iteration to converge for function $f$. Why so? Is it the case for function $g$?

```python

```

```python

```

```python

```

 Question 4: Compare Newton method with the adaptive gradient in the case of the classification problem

```python

```

```python

```

```python

```

##  f) To go Further

Go to top

We introduce two functions:

\begin{array}{rrcll}
t: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & (0.6 x_1 + 0.2 x_2)^2 \left((0.6 x_1 + 0.2 x_2)^2 - 4 (0.6 x_1 + 0.2 x_2)+4\right) \\
&            &         & + (-0.2 x_1 + 0.6 x_2)^2
\end{array}
 \begin{array}{rrcll}
p: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  &  \left| x_1-3 \right|  + 2\left| x_2-1\right|
\end{array}

 Question: Test adaptive gradient methods on these functions from different starting points. What do you observe?

```python

```

---

### Package Check and Styling

Go to top

```python
import lib.notebook_setting as nbs

packageList = ['IPython', 'numpy', 'scipy', 'matplotlib', 'cvxopt']
nbs.packageCheck(packageList)

nbs.cssStyling()
```


# Bayesian Statistics for Physicists: 03 Choosing Priors

## Contents of the BSFP series

01 Overview and Setup

02 Bayesian basics

03 Choosing priors

04 Examples of Bayesian updating

05 Sampling

06 Model selection, mixing, ...

07 Gaussian processes

08 Machine learning

A01 Appendices: reference, vocabulary, notation

## Overview of Choosing priors

A key feature of Bayesian statistics is the choice of the prior probability distribution.  We should incorporate all of the information we have, but no more. What are the options for choosing priors?  What are the subtleties?  Places to use an informative prior:

your dataset is small, but there is related information available from other systems;

your model is very flexible but you want to prevent overfitting (so use priors that prefer
      values close to zero);

you want to stay away from particular regions of parameter space: maybe strictly unphysical such as
      negative cross sections or acausal, or maybe
      counter to well-motivated assumptions such as parameters are of order unity;

the posterior from a previous experiment can become a prior ==> Bayesian updating;

other examples??

Last revised: 16-Apr-2019 by Dick Furnstahl [furnstahl.1@osu.edu].

## Contents of this notebook on Choosing priors

 Maximum entropy

 Conjugate priors

## Python/Jupyter set up

See Part 01 for overall installation and setup.

```python
# set up for plots in this notebook using matplotlib (there are other plotting choices)
%matplotlib inline
```

```python
import numpy as np

import scipy.stats as stats
from scipy.stats import norm, uniform

import matplotlib.pyplot as plt
plt.style.use('seaborn') # pretty matplotlib plots

import corner
import pymc3 as pm

```

```python
# make font adjustments
#plt.rcParams['font.size'] = 12
#plt.rcParams['legend.fontsize'] = 'medium'
#plt.rcParams['figure.titlesize'] = 'medium'
plt.rcdefaults()  # revert to defaults for now
```

```python
%%html
<!-- Use html cell magic to add css styling -->
<style>
  em {
      color: red;
  }
  dd {
      margin-left: 15px;
  }
  .red{color: red}
  .blue{color: blue}
</style>
```

### Subtlety with uniform prior

An example from the 2016 workshop on [Bayesian Methods in Astronomy](https://github.com/jakevdp/BayesianAstronomy/blob/master/Index.ipynb) considers fitting a straight line $y = mx+b$ to noisy data. If you take the prior on the slope $m$ to be uniform (flat), thinking that this is non-informative, then look at this plot, which samples lines with uniformly distributed slopes:

```python
%matplotlib inline

x_pts = np.linspace(-1, 1, 11)
fig = plt.figure(figsize=(5,5))
ax = fig.add_subplot(1, 1, 1)
ax.axis([-1., 1., -1., 1.])
for slope in np.linspace(0, 20, 100):   # 20 evenly spaced slopes m
    ax.plot(x_pts, slope * x_pts, '-k', linewidth=1)
ax.set_xlabel(r'$x$')
ax.set_ylabel(r'$y$')
ax.set_title(r'Uniform sampling of slopes $m$ in $y= mx + b$')
ax.set_aspect(1)

```

The density of the lines indicates the relative probability of different slopes. Summary point: flat priors are not necessarily minimally informative.  For the slope we probably want a prior that does not artificially over-weight large slopes; see http://arxiv.org/abs/1411.5018 for some discussion. For example, we might use a flat prior on the *angle* the line makes with the x-axis, which implies
$$
P(m) \propto (1 + m^2)^{-3/2}
$$

### Maximum entropy for priors (appealing to physicists!)

 Basic idea: identify least informative $p(x)$ from maximizing entropy:
   $$
     S[p(x)] = -\int\!dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
   $$
   subject to constraints from the prior information.

 $m(x)$ is an appropriate measure (often uniform, but see Sivia)

 Use Lagrange multiplier(s) to maximize

 One constraint is always normalization: $\int\!dx\, p(x) = 1$
     $\Longrightarrow$ alone it leads to uniform $p(x)$ (actually to $m(x)$)

   If the mean $\mu$ and variance $\sigma^2$ are known, then maximize (may be clearer to do with discretized variables)

   \begin{align}
    Q[p(x)] &= -\int\! dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
    + \lambda_0 \left[ 1 - \int\! dx\, p(x) \right] \\
     & \quad + \lambda_1 \left[\sigma^2 -  \int\! dx\, (x-\mu)^2 p(x)  \right]
   \end{align}

   Then
   $$
    \frac{\delta Q}{\delta p(x)} = 0   \quad\mbox{and}\quad
     m(x) = \mbox{const.}
     \ \Longrightarrow\
     p(x \mid \mu,\sigma) = \frac{1}{\sigma\sqrt{2 \pi}}
     e^{-(x-\mu)^2/2\sigma^2}
   $$

    For $\textbf{x} = \{x_1, \cdots, x_N\}$, we find $p(\textbf{x})$
    is the familiar least-squares likelihood function.  See Sivia for many more details.

If you want a "scale invariant" prior, this often means a flat prior on the *logarithm* of the parameter.  This is an example of a [Jeffreys Prior](https://en.wikipedia.org/wiki/Jeffreys_prior).

[Return to Contents]

### Conjugate priors

Bayes theorem tells us that the posterior follows from multiplying the prior by the likelihood and normalizing:

$$
  p(\theta\,|\,x) = \frac{p(x\,|\,\theta)\, p(\theta)}
                          {\int\!d\theta'\, p(x\,|\,\theta')\, p(\theta')}
$$

If the resulting $p(\theta\,|\,x)$ is in the same family of pdfs as $p(\theta)$, then $p(\theta)$ is said to be a conjugate prior for the likelihood $p(x\,|\,\theta)$.

The likelihood is often a fixed form. If there is freedom to choose a conjugate prior, the Bayesian updating of the posterior is given in closed form.

Comments:

Suppose we are flipping a coin and seek the posterior for the probability of heads $\theta \in [0,1]$.  Let $x$ be the number of successes (heads) in $N$ trials. The likelihood is binomial:
   $$ p(x\mid\theta) = \binom{N}{x}
       \theta^{x} (1-\theta)^{N-x} \;.
   $$
If we choose the prior to be a beta distribution with hyperparameters $a$ and $b$ (which are parameters of the prior as opposed to the model),
   $$ p(\theta\mid a,b) = \textrm{Beta}(\theta\mid a,b)
      = \frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)}
   $$
with beta function $B(a,b)$, then the posterior is of the same form ($\bar x \equiv N - x$):
   $$ p(\theta\mid x,a,b) = \frac{\theta^{x+a-1}(1-\theta)^{\bar x+b-1}}{B(x+a,\bar x+b)} = \textrm{Beta}(\theta\mid x+a,\bar x +b)
   $$
This posterior can be used as the prior for more samples; the hyperparameters just add additional information (check that it doesn't matter if you analyze the data all at once or sequentially because the tosses are independent).

See the [Wikipedia article](https://en.wikipedia.org/wiki/Conjugate_prior) for a big table of conjugate prior pairs.

If the likelihood is a normal distribution with known variance, the conjugate prior is also normal.  If it is a normal distribution with known mean, the inverse gamma distribution is a conjugate prior.

[Return to Contents]