Run the below cell to import all relevant notebooks.

```python
import io, os, sys, types
from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell

def find_notebook(fullname, path=None):
    """find a notebook, given its fully qualified name and an optional path

    This turns "foo.bar" into "foo/bar.ipynb"
    and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
    does not exist.
    """
    name = fullname.rsplit('.', 1)[-1]
    if not path:
        path = ['']
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        #print('searching: %s'%nb_path)
        if os.path.isfile(nb_path):
            return nb_path
        # let import Notebook_Name find "Notebook Name.ipynb"
        nb_path = nb_path.replace("_", " ")
        #print('searching: %s' % nb_path)
        if os.path.isfile(nb_path):
            return nb_path

class NotebookLoader(object):
    """Module Loader for Jupyter Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print ("importing Jupyter notebook from %s" % path)

        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = read(f, 4)

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        #print('Found %d cells'%len(nb.cells))
        try:
          for cell in nb.cells:
            if cell.cell_type == 'code':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.source)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod

class NotebookFinder(object):
    """Module finder that locates Jupyter Notebooks"""
    def __init__(self):
        self.loaders = {}

    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return

        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)

        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

#  register the NotebookFinder with sys.meta_path
print('running importer')
sys.meta_path.append(NotebookFinder())
```

    running importer

```python
pip install -U gif
```

    Requirement already up-to-date: gif in /home/avani/anaconda3/lib/python3.8/site-packages (3.0.0)
    Requirement already satisfied, skipping upgrade: Pillow>=7.1.2 in /home/avani/anaconda3/lib/python3.8/site-packages (from gif) (7.2.0)
    Note: you may need to restart the kernel to use updated packages.

# Logistic Regression

Logistic Regression is a discriminative learning algorithm where the posterior probabilities of $K$ classes can be learnt directly from the data. We would like to ensure that the probabilities sum to one and remain in $[0,1]$.

An initial intuition would be to use our standard linear regression algorithm to predict the $K$ classes given our input data. However, one can find specific examples where this method won't make sense since our prediction from regression model can have values greater than 1 and less than 0.

Instead, we can use the logistic/sigmoid activation function and define our hypothesis as follows:

\begin{equation}
\mathbf{h_w(x) = \sigma(w^{\top}x) = \sigma(z) = \frac{1}{1+e^{-z}}}
\end{equation}

Given this logistic regression model, we can choose values for weights $\mathbf{w}$ by deriving it as the maximum likelihood estimator.

Let's assume the following:

\begin{align}
\mathbf{p(y = 1 \; | \; x;w)} &= \mathbf{h_w(x)}\\
\mathbf{p(y = 0 \; | \; x;w)} &= \mathbf{1 - h_w(x)}
\end{align}

A more precise form would be:

\begin{equation}
\mathbf{P(Y \; | \; X;w)} = \mathbf{(h_w({x}))^y \; (1 - h_w({x}))^{1-y}}
\end{equation}

Assuming that our $\mathbf{m}$ training examples are independently generated, our MLE estimate is as follows:

\begin{align}
L(\mathbf{w}) &= \mathbf{P(Y \; | \; X;w)}\\
&=\prod_{i=1}^{\mathbf{m}} \; \mathbf{P(y^{(i)} \; | \; x^{(i)};w)}
\end{align}

As before, we would like to maximize the log likelihood or minimize the negative log likelihood.

\begin{align}
l(\mathbf{w}) &= \operatorname*{argmax}_\mathbf{w} \; \log L(\mathbf{w}) = \operatorname*{argmin}_\mathbf{w} \; - \log L(\mathbf{w})\\
&= \operatorname*{argmax}_\mathbf{w} \; \sum_{i=1}^\mathbf{m} \; \mathbf{y^{(i)}} \log \mathbf{h(x^{(i)}) \; + \; (1-y^{(i)})} \log \mathbf{(1-h(x^{(i)}))}\\
&= \operatorname*{argmin}_\mathbf{w} \; - \sum_{i=1}^\mathbf{m} \; \mathbf{y^{(i)}} \log \mathbf{h(x^{(i)}) \; + \; (1-y^{(i)})} \log \mathbf{(1-h(x^{(i)}))}
\end{align}

The above equation is nothing but the loss function for logistic regression. Note that there is no closed-form solution to minimize this likelihood or loss function. Hence, we would resolve to our gradient descent for making parameter update. The gradient to minimize the above equation is:

\begin{equation}
\nabla_{\mathbf{w}_j} = \mathbf{(y - h_w(x)) \; x}_j
\end{equation}

Therefore, our gradient update rule is:

\begin{equation}
\mathbf{w}_j \leftarrow \mathbf{w}_j + \alpha \; \nabla_{\mathbf{w}_j}
\end{equation}

The gradient update can be made on a single sample, mini-batch SGD, or batch gradient descent. In both the examples, we will look only at batch gradient descent.

```python
from sklearn import datasets
from sklearn.datasets import fetch_openml
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns; sns.set();
import pandas as pd
from utils import plot_decision_boundary, get_accuracy, get_prediction
from utils import plot_2D_input_datapoints, generate_gifs, sigmoid, normalize
import math
import gif
from tqdm import tqdm
import warnings
warnings.filterwarnings('ignore')
```

```python
# Let's initialize our weights using uniform distribution
def weight_init_uniform_dist(X, y):

    np.random.seed(312)
    n_samples, n_features = np.shape(X)
    _, n_outputs = np.shape(y)

    limit = 1 / math.sqrt(n_features)
    weights = np.random.uniform(-limit, limit, (n_features, n_outputs))
    weights[-1] = 0

    return weights
```

## Example 1 - Binary classifier on a toy dataset

We will now implement logistic regression on a toy dataset that consists of data points in 2D from two different classes.

```python
fig = plt.figure(figsize=(8,8));

np.random.seed(16)

# Generating a dataset
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0, n_informative=1, n_clusters_per_class=1, flip_y=-1)
y = y.reshape((-1, 1))

# Absorbing weight b of the hyperplane
b_ones = np.ones((len(X), 1))
X = np.hstack((X, b_ones))
```

```python
plot_2D_input_datapoints(X, y)
```

```python
# Train-Val-Test split
X_train_val, X_test, Y_train_val, Y_test = train_test_split(X, y, shuffle=True, test_size = 0.2)
X_train, X_val, Y_train, Y_val = train_test_split(X_train_val, Y_train_val, test_size = 0.125)

print("Training dataset: ", X_train.shape)
print("Validation dataset: ", X_val.shape)
print("Test dataset: ", X_test.shape)
```

    Training dataset:  (700, 3)
    Validation dataset:  (100, 3)
    Test dataset:  (200, 3)

```python
# Normalizing X_train and absorbing weight b of the hyperplane
X_normalized_train = normalize(X_train[:, :2])

b_ones = np.ones((len(X_normalized_train), 1))
X_normalized_train = np.hstack((X_normalized_train, b_ones))
```

Before diving into the training methodology, let us look at the pseudo code for logistic regression training.

1.   Initialize the weights $w$ from a uniform distribution.
2.   Normalize the input $X$
3.   Compute the weighted inputs $z$ and pass it through sigmoid activation, $sigmoid(z)$.
4.   Calculate the gradient based on the cross entropy loss function.
5.   Update the weights.
6.   Repeat steps 1-5 until convergence.

```python
# Train on the dataset
def train(X_train, Y_train, weights, num_epochs=1000, learning_rate=0.1):

    """ Method to train a logistic regression model.

    Parameters
    ------------

    X_train: ndarray (num_examples(rows) vs num_columns(columns))
    Input data on which the logistic regression model will be trained
    to acquire optimal weights

    Y_train: ndarray (num_examples(rows) vs class_labels(columns))
    Class labels of training set

    weights: ndarray (num_features vs n_output)
    Weights used to train the network and predict on test set

    num_epochs: int
    Number of epochs you want to train the model

    learning_rate: int
    rate with which weights will be update every epoch.
    """

    n_samples, n_features = np.shape(X_train)
    _, n_outputs = np.shape(y)

    history_weights = []
    epoch = 1

    # Training using Batch GD
    while epoch <= num_epochs:

        # Appending previous history weights/first initialized weights
        history_weights.append(weights)

        # Computing weighted inputs and predicting output
        w_transpose_x = np.dot(X_train, weights)
        y_pred = sigmoid(w_transpose_x)

        # Calculating gradient and updating weights
        gradient = 1 * np.dot(X_train.T, (Y_train - y_pred))
        weights += (learning_rate/n_samples) * gradient
        weights = np.round(weights, decimals=7)
        epoch += 1

    print("Training complete")

    return history_weights
```

```python
# Initializing weights from uniform distribution
weights = weight_init_uniform_dist(X_normalized_train, Y_train)
```

```python
trained_weights = train(X_normalized_train, Y_train, weights, num_epochs=50, learning_rate=0.1)
```

    Training complete

```python
best_weights = trained_weights[-1]

print("Evaluation results")
train_acc, _ = get_prediction(X_train, Y_train, best_weights, get_acc=True, model_type='logreg', predict='yes')
val_acc, _ = get_prediction(X_val, Y_val, best_weights, get_acc=True, model_type='logreg', predict='yes')
test_acc, _ = get_prediction(X_test, Y_test, best_weights, get_acc=True, model_type='logreg', predict='yes')

print("Training accuracy: {:.3f}" .format(train_acc))
print("Validation accuracy: {:.3f}" .format(val_acc))
print("Test accuracy: {:.3f}" .format(test_acc))
```

    Evaluation results
    Training accuracy: 96.857
    Validation accuracy: 95.000
    Test accuracy: 95.500

```python
generate_gifs(X_train, Y_train, trained_weights, 'train', path='content/logistic_reg_training_decision_boundary.gif', model_type='logreg', predict='yes')
```

    Gif/image generated

```python
plot_decision_boundary(X_test, Y_test, best_weights, dataset_type='test', model_type='logreg', predict='yes')
```

## Example 2 - From binary classification to multiclass classification - One vs All classifier

In the previous section, we saw how to build a binary classifier using logistic regression. Next, we will make a multiclass classifier using the binary classifier we built above by using them in one-vs-all and one-vs-one way. In this section we will see how to build a one-vs-all classifier. If we have $K$ classes to classify, we will $K$ binary classifiers. Say, for instance, we have 3 classes: $A, B, $ and $C$. In one-vs-all, we will train a binary classifier where true labels $(+1)$ will correspond to class $A$ and false labels $(0)$ will correspond to be class $B$ and $C$.

```python
fig = plt.figure(figsize=(8,8));

np.random.seed(4)

# Getting only linearly separable dataset
separable = False
while not separable:
    samples = make_classification(n_samples=450, n_features=2, n_redundant=0, n_informative=2, n_clusters_per_class=1, n_classes=3)
    red = samples[0][samples[1] == 0]
    blue = samples[0][samples[1] == 1]
    green = samples[0][samples[1] == 2]
    separable = any([red[:, k].max() < blue[:, k].min() or red[:, k].min() > blue[:, k].max()
                  or green[:, k].min() > blue[:, k].max() or green[:, k].min() > red[:, k].max()
                  or blue[:, k].min() > green[:, k].max() or red[:, k].min() > green[:, k].max() for k in range(2)])

X, y = samples

y = y.reshape((-1, 1))

# Absorbing weight b of the hyperplane
b_ones = np.ones((len(X), 1))
X = np.hstack((X, b_ones))
```

```python
plot_2D_input_datapoints(X[:, :2], y)
```

```python
# Train-Val-Test split
X_train_val, X_test, Y_train_val, Y_test = train_test_split(X, y, shuffle=True, test_size = 0.2)
X_train, X_val, Y_train, Y_val = train_test_split(X_train_val, Y_train_val, test_size = 0.125)

print("Training dataset: ", X_train.shape)
print("Validation dataset: ", X_val.shape)
print("Test dataset: ", X_test.shape)
```

    Training dataset:  (315, 3)
    Validation dataset:  (45, 3)
    Test dataset:  (90, 3)

```python
def train_one_vs_all(X_train, Y_train, base_weights, num_epochs=1000, learning_rate=0.1):

    """Method to train a multiclass classifier using one vs all technique.

    Parameters
    ------------

    X_train: ndarray (num_examples(rows) vs num_columns(columns))
    Input data on which the logistic regression model will be trained
    to acquire optimal weights

    Y_train: ndarray (num_examples(rows) vs class_labels(columns))
    Class labels of training set

    base_weights: ndarray (num_features vs n_outputs)
    Weights used to train the network and predict on test set

    num_epochs: int
    Number of epochs you want to train the model

    learning_rate: int
    rate with which weights will be update every epoch.
    """

    n_samples, n_features = np.shape(X_train)
    _, n_outputs = np.shape(y)

    history_weights = []
    weights_k_classes = []
    epoch = 1

    classes = np.unique(Y_train)

    # Training using Batch GD
    for k in classes:
        # one-vs-all binary classifier
        binary_y_train = np.where(Y_train == k, 1, 0)

        weights = base_weights
        # Appending previous history weights/first initialized weights
        #weights_k_classes.append(weights)

        for epoch in range(1, num_epochs+1):

            # Computing weighted inputs and predicting output
            w_transpose_x = np.dot(X_train, weights)
            y_pred = sigmoid(w_transpose_x)

            # Calculating gradient and updating weights
            gradient = 1 * np.dot(X_train.T, (binary_y_train - y_pred))
            weights += (learning_rate/n_samples) * gradient
            weights = np.round(weights, decimals=7)

            #weights_k_classes.append(weights)
            epoch += 1

        #history_weights.append(weights_k_classes)
        history_weights.append(weights)
        #weights_k_classes = []

    print("Training complete")
    return history_weights
```

```python
# Initializing weights from uniform distribution
weights = weight_init_uniform_dist(X_train, Y_train)
```

```python
trained_weights = train_one_vs_all(X_train, Y_train, weights, num_epochs=50, learning_rate=0.1)
```

    Training complete

```python
# Compute the accuracy of training data and validation data
def predict_one_vs_all(trained_weights, X_input, Y_input):

    num_classes = len(np.unique(Y_input))
    scores = np.zeros((num_classes, (X_input.shape)[0]))

    for k in range(num_classes):

        binary_y_input = np.where(Y_input == k, 1, 0)

        w_transpose_x = np.dot(X_input, trained_weights[k])
        y_pred = sigmoid(w_transpose_x)
        y_pred = y_pred.reshape((-1,))
        scores[k, :] = y_pred

    pred_X = np.argmax(scores, axis=0)
    return pred_X
```

```python
pred_train_one_vs_all = predict_one_vs_all(trained_weights, X_train, Y_train)
pred_val_one_vs_all = predict_one_vs_all(trained_weights, X_val, Y_val)
pred_test_one_vs_all = predict_one_vs_all(trained_weights, X_test, Y_test)

pred_train_one_vs_all = pred_train_one_vs_all.reshape((-1, 1))
pred_val_one_vs_all = pred_val_one_vs_all.reshape((-1, 1))
pred_test_one_vs_all = pred_test_one_vs_all.reshape((-1, 1))

print('Training dataset accuracy: %f' % (np.mean(Y_train == pred_train_one_vs_all)))
print('Validation dataset accuracy: %f' % (np.mean(Y_val == pred_val_one_vs_all)))
print('Test datast accuracy: %f' % (np.mean(Y_test == pred_test_one_vs_all)))
```

    Training dataset accuracy: 0.926984
    Validation dataset accuracy: 0.844444
    Test datast accuracy: 0.977778

```python
plot_2D_input_datapoints(X_train[:, :2], Y_train)
plt.xlabel("$X_0$"); plt.ylabel("$X_1$")

for weight in [trained_weights[0], trained_weights[2]]:
    j = np.array([X_train[:, 0].min(), X_train[:, 0].max()])
    k = -(j * weight[0] + weight[2]) / weight[1]
    plt.plot(j, k, color='k', linestyle="--")
```

## (Optional Reading) Multiclass Logistic Regression

The multi-class scenario for logistic regression is quite similar to the binary case, except that the label $y$ is now an integer in {1, ...., K} where $K$ is the number of classes. Instead of using sigmoid activation function, we know use softmax activation which calculates scores for all classes, instead of just one class in the case sigmoid activation. The softmax function takes weighted inputs and normalizes them into a probability distribution, thereby forcing the prediction score to be between in the interval $[0, 1]$. The class scores for each examples will add up to 1. The class score with the highest probability will be the predicted class. The softmax function is defined as follows:

\begin{align}
\mathbf{\sigma(z_i)} &= \mathbf{\frac{e^{z_{(i)}}}{\sum_{j=1}^K e^{z_{(j)}}}} \ \ \ \text{for}\ i=1,.,.,.,K\
\end{align}

where, $\mathbf{z_i}$ are the weighted inputs for each class.

For instance, for a given vector:

$ z = [0.6, 1.1, -1.5, 1.2, 3.2, -1.1]$

the resultant softmax $(z)$ is:

$ softmax(z) = [0.055, 0.09, 0.0067, 0.1, 0.74, 0.01]$

The loss function for multiclass logistic regression slightly changes since we now have K output classes.

\begin{align}
loss &= L(y, \hat y)\\
&=  - \sum_i y_i \log \hat y_i\\
&= - \log \hat y_c
\end{align}

We can learn the weights by performing gradient descent on the loss function w.r.t to a certain number of hyperparameters. As in the case of binary logistic regression, the loss function is convex and hence, gradient will converge to a global minimum.

We can also apply $L_2$ regularization to the loss function.

You can try to implement this variant for the multiclass synthetic dataset above.


## Wave equation
#### Parameter estimation for the Wave Equation using Gaussian processes (Without temporal discretization since we'd need a second order time scheme)

#### Problem Setup

$u_{tt} - c u_{xx} = 0$

General solution:
$u(x,t) = F(x-ct) + G(x+ct)$ with F, G some functions.

Take $F(x) = x^2$ and $G(x) = \sin(x)$ and $c=1$.

Thus: $u(x,t) = (x-t)^2 + \sin(x + t)$.

$x \in [0, 1], t \in [0,1]$

Set $f = 0$.

Consider $u$ to be a Gaussian process.

$u \sim \mathcal{GP}(0, k_{uu}(x_i, x_j, \theta))$

And the linear operator:

$\mathcal{L}_x^c = \frac{d^2}{dt^2} \cdot - c \frac{d^2}{dx^2} \cdot$

so that

$\mathcal{L}_x^c u = f$

Problem at hand: estimate $c$ (should be $c = 1$ in the end).

#### step 1: Simulate data

```python
import time
import numpy as np
import sympy as sp
import math
import warnings
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
np.random.seed(int(time.time()))

def get_simulated_data(n=20):
    t = np.random.rand(n)
    x = np.random.rand(n)
    y_u = np.multiply(x-t, x-t) + np.sin(x+t)
    y_f = 0*x
    return (x, t, y_u, y_f)

(x,t,y_u,y_f) = get_simulated_data()
```

#### Step 2:Evaluate kernels

$k_{uu}(x_i, x_j; \theta) = \theta exp(-\frac{1}{2l_x}(x_i-x_j)^2 - \frac{1}{2l_t}(t_i-t_j)^2)$

```python
x_i, x_j, t_i, t_j, theta, l_x, l_t, c = sp.symbols('x_i x_j t_i t_j theta l_x l_t c')
kuu_sym = theta*sp.exp(-1/(2*l_x)*((x_i - x_j)**2) - 1/(2*l_t)*((t_i - t_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, t_i, t_j, theta, l_x, l_t), kuu_sym, "numpy")
def kuu(x, t, theta, l_x, l_t):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], t[i], t[j], theta, l_x, l_t)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{\tilde{x}_i}^c \mathcal{L}_{\tilde{x}_j}^c k_{uu}(x_i, x_j; \theta) \\
= \frac{d^4}{dt_i^2 dt_j^2}k_{uu} - c\frac{d^4}{dt_i^2 dx_j^2}k_{uu} - c\frac{d^4}{dx_i^2 dt_j^2}k_{uu} + c^2\frac{d^4}{dx_i^2 dx_j^2}k_{uu}$

```python
kff_sym = sp.diff(kuu_sym, t_i, t_i, t_j, t_j) \
        - c*sp.diff(kuu_sym, t_i, t_i, x_j, x_j) \
        - c*sp.diff(kuu_sym, x_i, x_i, t_j, t_j) \
        + c**2*sp.diff(kuu_sym, x_i, x_i, x_j, x_j)
kff_fn = sp.lambdify((x_i, x_j, t_i, t_j, theta, l_x, l_t, c), kff_sym, "numpy")
def kff(x, t, theta, l_x, l_t, c):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], t[i], t[j], theta, l_x, l_t, c)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{\tilde{x}_i}^c k_{uu}(x_i, x_j; \theta) \\
= \frac{d^2}{dt_i^2}k_{uu} - c\frac{d^2}{dx_i^2}k_{uu}$

```python
kfu_sym = sp.diff(kuu_sym, t_i, t_i) - c*sp.diff(kuu_sym, x_i, x_i)
kfu_fn = sp.lambdify((x_i, x_j, t_i, t_j, theta, l_x, l_t, c), kfu_sym, "numpy")
def kfu(x, t, theta, l_x, l_t, c):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], t[i], t[j], theta, l_x, l_t, c)
    return k
```

```python
def kuf(x, t, theta, l_x, l_t, c):
    return kfu(x, t, theta, l_x, l_t, c).T
```

#### Step 3: Compute NLML

```python
def nlml(params, x, t, y1, y2, s):
    theta_exp = np.exp(params[0])
    l_x_exp = np.exp(params[1])
    l_t_exp = np.exp(params[2]) # params[3] = c
    K = np.block([
        [kuu(x, t, theta_exp, l_x_exp, l_t_exp) + s*np.identity(x.size), kuf(x, t, theta_exp, l_x_exp, l_t_exp, params[3])],
        [kfu(x, t, theta_exp, l_x_exp, l_t_exp, params[3]), kff(x, t, theta_exp, l_x_exp, l_t_exp, params[3]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
def nlml(params, x, t, y1, y2, s):
    p0 = np.exp(params[0])
    p1 = np.exp(params[1])
    p2 = np.exp(params[2])
    K = np.block([
        [kuu(x, t, p0, p1, p2) + s*np.identity(x.size), kuf(x, t, p0, p1, p2, params[3])],
        [kfu(x, t, p0, p1, p2, params[3]), kff(x, t, p0, p1, p2, params[3]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

#### Step 4: Optimise hyperparameters

In this step, we want to sequentially increase the domain for the intial hyperparameters and see, how this influences the success in the optimization.

Our output will be a plot of the results and the minimizer.

```python
# Want to define a nice sequence of numbers, that won't increase too quickly.
def fib_formula(n):
    golden_ratio = (1+math.sqrt(5))/2
    val = (golden_ratio**n - (1-golden_ratio)**n) / math.sqrt(5)
    return int(round(val))
```

```python
def minimize_restarts(x,t,y_u,y_f,n=10):
    nlml_wp = lambda params: nlml(params, x, t, y_u, y_f, 1e-7)
    all_results = []
    for it in range(0,n):
        try:
            all_results.append(minimize(nlml_wp, fib_formula(it+1)*np.random.rand(4), method="Nelder-Mead", options={'maxiter':5000, 'fatol':0.001}))
        except np.linalg.LinAlgError:
            pass
    filtered_results = [m for m in all_results if 0==m.status]
    function_values = [m.fun for m in filtered_results]
    return (min(filtered_results, key = lambda x: x.fun), function_values)
```

```python
repeats = 10
warnings.filterwarnings("ignore") # Suppressing warnings
(m, function_values) = minimize_restarts(x, t, y_u, y_f, repeats)
print(m)

# This plot will show us how frail our model is w.r.t. the initial values
lin = np.linspace(0, repeats, len(function_values))
plt.plot(lin, function_values)
plt.show()
```

```python
m.x[3] # This is our best predicted value for c
```

#### Step 5: Plots with varied hyperparameters

Plotting by varying theta:

```python
lin = np.linspace(2, 10, 50)
res = [nlml((q, m.x[1], m.x[2], m.x[3]), x, t, y_u, y_f, 1e-7) for q in lin]
plt.plot(lin, res)
```

Plotting by varying l_x:

```python
lin = np.linspace(0, 4, 50)
res = [nlml((m.x[0], q, m.x[2], m.x[3]), x, t, y_u, y_f, 1e-7) for q in lin]
plt.plot(lin, res)
```

Plotting by varying l_t:

```python
lin = np.linspace(0, 4, 50)
res = [nlml((m.x[0], m.x[1], q, m.x[3]), x, t, y_u, y_f, 1e-7) for q in lin]
plt.plot(lin, res)
```

Plotting by varying c:

```python
lin = np.linspace(-1, 3, 50)
res = [nlml((m.x[0], m.x[1], m.x[2], q), x, t, y_u, y_f, 1e-7) for q in lin]
plt.plot(lin, res)
# nlml((4.8159, 0.4641, 4.4104, 0.006), x, t, y_u, y_f, 1e-3)
```

#### Step 6: Analysis of the error

In this section we want to analyze the error of our algorithm using two different ways and its time complexity.

```python
res = np.zeros((5,25))
timing = np.zeros((5,25))
warnings.filterwarnings("ignore")
for k in range(5):
    for n in range(25):
        start_time = time.time()
        (x, t, y_u, y_f) = get_simulated_data(n)
        m = minimize(nlml, np.random.rand(4), args=(x, t, y_u, y_f, 1e-7), method="Nelder-Mead")
        res[k][n] = m.x[2]
        timing[k][n] = time.time() - start_time
```

###### Plotting the error in our estimate for $\nu$ (Error = $| \nu_{estimate} - \nu_{true} |$):

```python
lin = np.linspace(1, res.shape[1], res.shape[1])

for i in range(res.shape[0]):       # This is the number of runs
    plt.plot(lin, np.abs(res[i,:]))
    plt.ylabel('Error')
    plt.xlabel('Number of data points')
    plt.show()
```

All in one plot:

```python
lin = np.linspace(1, res.shape[1], res.shape[1])

for i in range(res.shape[0]):
    plt.plot(lin, np.abs(res[i,:]))
    plt.ylabel('Error')
    plt.xlabel('Number of data points')

est = np.repeat(0.2, len(lin))
plt.plot(lin, est, color='blue', linestyle='dashed')
plt.show()
```

We see that for n sufficiently large (in this case $n \geq 8$), we can assume the error to be bounded by 0.2.

It seems to be difficult to (even roughly) describe the limiting behavior of the error w.r.t. the number of data points

###### Plotting the error between the solution and the approximative solution:

Another approach of plotting the error is by calculating the difference between the approximative solution and the true solution.

That is: Let $\tilde{c}$ be the parameter, resulting from our algorithm. Set $\Omega := \{(x_i, t_i) \; \vert \; x_i \in x, t_i \in t\} \subseteq [0,1] \times [0,1]$
Then we can calculate the solution of the PDE

\begin{align}
    \frac{d^2}{dt^2}\tilde{u}(x,t) - \tilde{c}\frac{d^2}{dx^2}\tilde{u}(x,t) = 0.
\end{align}

and set the error to $\lVert \tilde{u}(x,t) - u(x,t) \rVert_{\Omega}$. The norm can be chosen freely.

In our case, finding the solution to a given $\tilde{c}$ is again very simple. It is given by $\tilde{u}(x,t) = u(x,\tilde{c}t) = (x-\tilde{c}t)^2 + \sin(x+\tilde{c}t)$.

We thus get:

\begin{align}
\lVert \tilde{u}(x,t) - u(x,t) \rVert_{\Omega} = \lVert (x-\tilde{c}t)^2 + \sin(x+\tilde{c}t) - (x-t)^2 - \sin(x+t) \rVert_{\Omega}
\end{align}

With the $L^2$-norm, this is

\begin{align}
(\sum_{(x_i,t_i) \in \Omega} \vert (x_i-\tilde{c}t_i)^2 + \sin(x_i+\tilde{c}t_i) - (x_i-t_i)^2 - \sin(x_i+t_i) \vert^2 )^{1/2}
\end{align}

```python
lin = np.linspace(1, res.shape[1], res.shape[1])
ones = np.ones(res.shape[1])
diff = np.ndarray(res.shape[1])

for i in range(res.shape[0]):
    for j in range(res.shape[1]):
        diff[j] = np.linalg.norm((x - res[i,j]t)**2 + np.sin(x+res[i,j]t) - (x-t)**2 - np.sin(x + t))
    plt.plot(lin, diff)
    plt.ylabel('Error')
    plt.xlabel('Number of data points')
    plt.show()
```

###### Plotting the execution time:

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

for i in range(timing.shape[0]):
    plt.plot(lin, timing[i,:])
    plt.ylabel('Execution time in seconds')
    plt.xlabel('Number of data points')
    plt.show()
```

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

for i in range(timing.shape[0]):
    plt.plot(lin, timing[i,:])
    plt.ylabel('Execution time in seconds')
    plt.xlabel('Number of data points')

est = lin**(1.25)
plt.plot(lin, est, color='blue', linestyle='dashed')
plt.show()
```

Curiously, the time complexity seems to be around $\mathcal{O}(n^{5/4})$ (blue-dashed line).

Assuming an equal amount of function evaluations in the Nelder-Mead algorithm for different values of n,
we would expect a time complexity of $\mathcal{O}(n^3)$, due to the computation of the inverse of an $n\times n$-matrix in every evaluation of $\textit{nlml}$.

```python

```


```python
import matplotlib.pyplot as plt
from functools import partial
import numpy as np

```

```python
def q(lE, lX, eX):
    return (lE - 1)/(lX*eX + lE)
```

```python
R = np.arange(-1,1.1,.1)
lE = 1
lX = 1
arr1 = q(2, 1, R)
plt.plot(arr1)
arr2 = q(2, 1, R)
plt.plot(arr2)
plt.axvline(10, c='k', alpha=.5)
plt.show()
```

```python
lE = 2.1
lX = 2
def get_q(lE, lX, R):
    for r in R:
        yield(q(lE,lX,R,r))
def Q(lE,lX,R):
    return np.array(list(get_q(lE,lX,R)))
Q(lE,lX,R)
q(lE,lX,R[2],R)
q(2.1,2,.1,-.9)
```

    9.166666666666677

```python
plt.imshow(Q(lE,lX,R), cmap='viridis')
plt.colorbar()
plt.show()
```

```python
import sympy as sp

x = sp.Symbol('x')
l = sp.Function('l')(x)
k = sp.Symbol('kappa')

#f = (l-1)/(k*x + l)

A = sp.diff((l-1)/(k*x + l), x)
```

```python
A.simplify().factor(sp.diff(l, x), k)
```

$\displaystyle \frac{\kappa x \frac{d}{d x} l{\left(x \right)} + \kappa \left(1 - l{\left(x \right)}\right) + \frac{d}{d x} l{\left(x \right)}}{\left(\kappa x + l{\left(x \right)}\right)^{2}}$

```python
S = np.arange(1,5,.2)
plt.plot((S - 1)/S, marker='.')
plt.show()
```

```python
plt.plot(R + 2)
plt.show()
```

```python
R
```

    array([-1.00000000e+00, -9.00000000e-01, -8.00000000e-01, -7.00000000e-01,
           -6.00000000e-01, -5.00000000e-01, -4.00000000e-01, -3.00000000e-01,
           -2.00000000e-01, -1.00000000e-01, -2.22044605e-16,  1.00000000e-01,
            2.00000000e-01,  3.00000000e-01,  4.00000000e-01,  5.00000000e-01,
            6.00000000e-01,  7.00000000e-01,  8.00000000e-01,  9.00000000e-01,
            1.00000000e+00])

```python
import sympy as sp
l = sp.Symbol('lambda')
x = sp.Symbol('x')
L = sp.Function('L')(x)

A = sp.diff((L - 1 )/(l*x  + L), x).simplify()
```

```python
A
```

$\displaystyle \frac{- \left(\lambda + \frac{d}{d x} L{\left(x \right)}\right) \left(L{\left(x \right)} - 1\right) + \left(\lambda x + L{\left(x \right)}\right) \frac{d}{d x} L{\left(x \right)}}{\left(\lambda x + L{\left(x \right)}\right)^{2}}$

```python
eq = sp.fraction(A)[0]
B = sp.solve(eq, sp.diff(L, x))
B[0]
```

$\displaystyle \frac{\lambda \left(L{\left(x \right)} - 1\right)}{\lambda x + 1}$

Test algebra

```python
l_E = sp.Symbol('l_E')
l_X = sp.Symbol('l_X')
q = sp.Symbol('q')
n = sp.Symbol('n')
P_E = sp.Symbol('P_E')
e_E = sp.Symbol('epsilon_E')

PnL_h = n/P_E   * (q * x * l + e_E *(l_E * (1 - q) - 1)  )
PnL_u = PnL_h.subs(q, 1).simplify()
PnL_u
(PnL_u - PnL_h).simplify().factor(l)
```

$\displaystyle \frac{n \left(1 - q\right) \left(- \epsilon_{E} l_{E} + \lambda x\right)}{P_{E}}$

```python
PnL_h
```

$\displaystyle \frac{n \left(\epsilon_{E} \left(l_{E} \left(1 - q\right) - 1\right) + l_{X} \lambda q\right)}{P_{E}}$

```python

```


 Calibration Notebook

Author: Leonardo Assis Morais

Calibrate the TES detector using area measurements.

The output of this notebook is a .csv file with the counting thresholds

required to convert TES area information to photon-number information.

Use the notebook Counting Photons.ipynb to obtain photon-number

information from your raw TES area data.

Summary:
1) Getting data
2) Importing data
3) Calibration
4) Sanity Check
5) Additional Feature: plotting other characteristics

# Getting the calibration data

You will need to have the calibration data on your computer

for this notebook to work properly.

In order to get the data, you will need to use a ftp protocol.

In Mac, I can press CMD + K and type the following address:
 ftp://smp-qtlab11.instrument.net.uq.edu.au

This wiil grant you access to the folder ’captures’ where all

measurements made with the FPGA are stored. Copy the folder with

your data set to your machine in a folder that you know.

You will need to know this address to run the notebook below.

```python
import numpy as np
import matplotlib.pyplot as plt
import tes
```

```python
# for development/debugging of the tes package
#%load_ext autoreload
%reload_ext autoreload
```

# Importing data

```python
from tes.data import CaptureData

# edit the path here with the folder where your data is located
datapath = ('/Users/leo/TES_package_and_data/TES3_80mK_820nm_10khz_'
            '50ns_2_6V_1MBW_peak1_h078df2b2/drive-pulse_BL-pulse_'
            'threshold-slope_max')

# function to get the data from files
data = CaptureData(datapath)
# boolean mask for events in channel 0 - the photon detections
ch0 = data.mask(0)
# boolean mask for events in channel 1 -  the laser drive pulses.
ch1 = data.mask(1)
# this is the relative time since the last event
times = data.time
heights = data.rise[ch0]['height'][:,0]
areas = data.area[ch0]
minima = data.rise[ch0]['minimum'][:,0]
lengths = data.pulse_length[ch0]
rise_time = data.rise[ch0]['rise_time'][:,0]
```

# Calibration

Using the cell below to find the position of the peaks of the histogram.

The data and model will be plotted so the user can make a sanity check of this fit first guess.

```python
%matplotlib inline
from tes.calibration import guess_histogram, plot_guess

# histogram parameters
BIN_NUMBER = 1000
WIN_LENGTH = 25
MINIMUM = 7
MAX_AREA = 1e7

# figure parameters
FIG_WIDTH = 15
FIG_HEIGHT = 9
YLIM = [1e0, 3.5e5] # limits for y axis

# calculating educated fitting
counts, smooth_hist, bin_centre, max_i, max_list = guess_histogram(areas,
                                                         BIN_NUMBER,
                                                         WIN_LENGTH,
                                                         MINIMUM,
                                                         MAX_AREA)

# plotting educated fitting with maxima points
fig, ax = plt.subplots(figsize=[FIG_WIDTH,FIG_HEIGHT])
plot_guess(ax, counts, smooth_hist, bin_centre, max_i)
print("The number of peaks found in this data set was {}.".format(len(max_i)))
# plotting details
ax.set_ylim(YLIM)
ax.set_xlim(-0.1e6,np.max(ax.get_xticks()))
ax.grid('on')
```

## Fitting the data with a mixture model of gaussian distributions

Used lmfit package for fitting. Documentation, tutorials, and further informations here:
https://lmfit.github.io/lmfit-py/

The model is composed of a sum of 16 Gaussian distributions:

\begin{equation}
    M(A_1, \mu_1, \sigma_1, ..., A_{16}, \mu_{16}, \sigma_{16}) = \sum_{n=1}^{16} G_n(a),
\end{equation}

where

\begin{equation}
    G_n(a) = \frac{A_n}{\sqrt{2\pi} \sigma_n} \exp \bigg( \frac{-(a - \mu_n)^2}{2 \sigma_n^2}\bigg),
\end{equation}

$a$ is the area, $\sigma_n^2$ is the gaussian variance, $A_n$ is the amplitude, $\mu_n$ is the mean.

## Creating the histogram

```python
from tes.calibration import area_histogram

MAX_AREA = 1e7
BIN_NUMBER = 45000

bin_centre, counts, error, bin_width = area_histogram(MAX_AREA, BIN_NUMBER, areas)
```

    The number of points in the analysed data is: 14884856 .
    We analyse data up to: 10000000.0 area units.
    The number of bins in the analysed data is: 45000 .
    The size of each bin is: 222.22222222222223 .

## Fitting

```python
from tes.calibration import residual_gauss
import lmfit

# requires max_list from guess_histogram
MAX_IDX = len(max_i) # number of distributions to be used+1
INIT_SCALE = 40889
INIT_AMP = 75751

params = lmfit.Parameters()

[params.add(r'scale{}'.format(idx), value=INIT_SCALE)
 for idx in range(1, MAX_IDX)]
[params.add(r'loc{}'.format(idx), value=max_list[idx])
 for idx in range(1,MAX_IDX)]
[params.add(r'amp{}'.format(idx), value=INIT_AMP)
 for idx in range(1,MAX_IDX)]

gauss_fit = lmfit.minimize(residual_gauss, params, args=(bin_centre, counts, error,MAX_IDX))
```

```python
# uncomment to see fit report
# print(lmfit.fit_report(gauss_fit))
```

```python
# requires area_histogram
from tes.calibration import plot_area, gaussian_model
%matplotlib inline

# creating the model
model_gauss = gaussian_model(gauss_fit, counts, bin_centre, MAX_IDX)

# plotting the figure
width = 11
golden_ratio = (np.sqrt(5.0)-1.0)/2.0

fig, ax = plt.subplots(figsize=[width, width*golden_ratio])

# edit plot steps to avoid plotting all bins
plot_steps = 1
plot_area(ax, bin_centre, counts, error, model_gauss, plot_steps)

# uncomment to save fit/edit figure path
# fig.savefig('./Chapter3/Figures/area_histogram_16.pdf', bbox_inches='tight',
#             dpi=300)
```

Given the area histogram above, the function below will find the

position of the counting thresholds by positioning them at

the intersection point of two adjacent normalised distributions.

```python
from tes.calibration import find_thresholds

# requires: gauss_fit, max_i
# returns: dist, thresholds
# objective: from fitting obtained in last section, obtain normalised distributions
#            for plots and position of counting thresholds

# normally you won't need to change the variable below.
# see help(find_thresholds) for more info.
const = 3e4
dist, thresholds = find_thresholds(gauss_fit, max_i, bin_centre, counts, const)
```

# Saving counting thresholds

Save all counting thresholds in a .csv file to be used for converting area information into photon-number information.

```python
import csv

folder_name = '/Users/leo/Desktop/'
file_name = 'calibration'
with open(folder_name+file_name+'.csv', mode='w') as count_file:
    fock_numbers = [i for i in range(0, len(thresholds)+1)]
    writer = csv.writer(count_file)
    count_file.write(str(datapath) + "\n")
    writer.writerow(fock_numbers)
    writer.writerow(thresholds)
```

# Sanity Check: Plotting thresholds and normalised distributions

```python
from tes.calibration import correct_xticks, plot_normalised
# Creating the figure

GOLDEN_RATIO = (np.sqrt(5.0)-1.0)/2.0
FIG_WIDTH = 11
FIG_HEIGHT = FIG_WIDTH*GOLDEN_RATIO
STEP_LABEL = 2 # every other threshold has tick label (x-axis)
LAST_THRESH = 5.6e6
YMIN = -0.001e-5
YMAX = 1.1e-5

fig, ax = plt.subplots(figsize=[FIG_WIDTH, FIG_HEIGHT])

plot_normalised(ax, max_i, bin_centre, dist, thresholds)
# fixing xtick labels
expx = correct_xticks(ax)
expx = expx-1
ax.set_xticks(thresholds[::STEP_LABEL])
tick_values = thresholds/10**expx
tick_labels = [r'${:2.1f}$'.format(tick) for tick in tick_values]
ax.set_xticklabels(tick_labels[::STEP_LABEL]);
ax.set_xlabel(r'Pulse Area $(\times 10^{{ {} }})$ (arb. un.)'.format(expx))
ax.set_xlim(0, LAST_THRESH)
ax.set_ylim(YMIN, YMAX)
```

# Other components

If you want to see the data for different components,

you can use the code below.

```python
%matplotlib inline
from tes.calibration import plot_histogram

# histogram parameters
BIN_NUMBER = 300
measurement = 'Length'

# figure parameters
FIG_WIDTH = 15
FIG_HEIGHT = 9

# plotting educated fitting with maxima points
fig, ax = plt.subplots(figsize=[FIG_WIDTH,FIG_HEIGHT])
plot_histogram(ax, lengths, BIN_NUMBER, measurement)
```

END OF NOTEBOOK


```python
import matplotlib.pyplot as plt
import matplotlib.patches as patches
import sympy
import numpy as np
import warnings
warnings.filterwarnings('ignore')
plt.style.use("seaborn-muted")
```

# Surplus

## Consumer Surplus

A competitive market is comprised of producers who make goods and services and the consumers who purchase and use them. Through market forces, these consumers and producers agree on a common price for the good or service in question. We will assume that there is no price discrimination and all consumers face the same price. This is typical but not always the case in the real world. Although all consumers face the same market price, consumers are different in how much they individually value a good. We say that consumers have a maximum price that they are willing to pay for a good, and any price marginally higher than this price will dissuade the consumer from participating in the market. This max WTP (willingness to pay) price varies among entities based on their desire for the good, which in turn can be based on how much of the good they already have.

Consider the market for electricity. Among consumers we have entities such as households, commercial buildings, factories, and so on. A factory would likely have a very high WTP for electricity because the opportunity costs for factories to not operate are very high. Capital is expensive, employees still have to be paid, and it is inconvenient to have to stop and start up machinery frequently. Thus, for a factory it is preferable to always have a reliable supply of electricity to continue operations and this need is reflected in the WTP. Contrast this with households, who certainly value having electricity, but should electricity become prohibitively expensive, probably would decide to cut back on usage as the drawbacks of not having electricity aren't as severe compared to the factory above.

## Producer Surplus

Producers experience a similar characteristic. A producer has a minimum price at which it is willing to produce a good based on its costs. Any market price less than this price would dissuade a producer from supplying its good. Again, in the electricity example, we have several power plants that produce electricity, but each inherently does so at different costs. Imagine and contrast the operating costs of a solar farm with a coal plant, or a newer, more efficient natural gas plant versus an older one.

Putting all of these concepts together we arrive at the idea of economic welfare. Suppose electricity costs 10 cents per kWh. On the demand side, imagine a factory who's WTP is 30 cents/kWh. This factory enjoys a consumer surplus of 20 cents/kWh, in other words, it's paying 20 cents less per kWh than what it would be willing to pay. A household might have a WTP of 15 cents/kWh. Here the household's surplus is only 5 cents/kWh. We can also imagine a consumer whose WTP is less than the market price and thus doesn't participate in the market. Imagine for some reason that cryptocurrency prices have dropped to the point that they aren't worth the electricity it takes to mine them. In this case, we might have an idle or non-existent crypto-farm (a place with a lot of computing power) due to electricity being too expensive. On the producer side, maybe we have a solar plant which is operating at the market price, but a natural gas plant that is idling because the price of supplying electricity isn't sufficient to make up for operating costs.

Combining the surpluses of all individual consumers and producers yields the market consumer surplus and producer surplus. As the market price fluctuates, certain comsumers and producers enter and exit the market, and the total surplus varies. Note from the above examples that a consumer is not always an individual, it can be a firm buying from another firm. We now explore further.

## Example

We create a consumer class with a WTP characteristic, and a list of consumers with WTP from 10 to 1. The binary function `demand` indicates whether the consumer participates in the market at a given price or not.

```python
class Consumer:
    def __init__(self, WTP):
        self.WTP = WTP
    def demand(self, price):
        if price <= self.WTP:
            return 1
        else:
            return 0
    def surplus(self, price):
        if price <= self.WTP:
            return self.WTP - price
        else:
            return 0

consumers = [Consumer(x) for x in range(10,0,-1)]

print([x.demand(6) for x in consumers])
```

    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]

For a market price of 6, we have 5 consumers who participate and 5 who don't. Now let's make a matrix of the lists of participants for each market price between 1 and 10.

```python
for price in range(10,0,-1):
    print([x.demand(price) for x in consumers])
```

    [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    [1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
    [1, 1, 1, 0, 0, 0, 0, 0, 0, 0]
    [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
    [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]
    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0]
    [1, 1, 1, 1, 1, 1, 1, 1, 0, 0]
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 0]
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

Notice the vague resemblance to a downward-sloping demand curve. The left-most consumer, with a WTP of 10, always participates for these prices. The right-most consumer only participates at a market price of 1. Now lets try and find participation for each price point, starting at 10.

```python
print("Market Price:", "Market Participants")
print("")
for price in range(10,0,-1):
    if price == 10:
        print(str(price) + ":", sum([x.demand(price) for x in consumers]))
    else:
        print(str(price) + ": ", sum([x.demand(price) for x in consumers]))
```

    Market Price: Market Participants

    10: 1
    9:  2
    8:  3
    7:  4
    6:  5
    5:  6
    4:  7
    3:  8
    2:  9
    1:  10

Here, each number represents the total number of market participants as the market price *decreases* starting from 10. Notice as the price decreases, participation increases in this demand case. Now lets find actual surplus.

```python
for price in range(10,0,-1):
    print([x.surplus(price) for x in consumers])
```

    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    [2, 1, 0, 0, 0, 0, 0, 0, 0, 0]
    [3, 2, 1, 0, 0, 0, 0, 0, 0, 0]
    [4, 3, 2, 1, 0, 0, 0, 0, 0, 0]
    [5, 4, 3, 2, 1, 0, 0, 0, 0, 0]
    [6, 5, 4, 3, 2, 1, 0, 0, 0, 0]
    [7, 6, 5, 4, 3, 2, 1, 0, 0, 0]
    [8, 7, 6, 5, 4, 3, 2, 1, 0, 0]
    [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Instead of printing a binary 1 or 0 indicating market participation, we've displayed each participant's actual surplus value. Similarly, let's find total surplus per price point.

```python
print("Market Price:", "Consumer Surplus")
print("")
for price in range(10,0,-1):
    if price == 10:
        print(str(price) + ":", sum([x.surplus(price) for x in consumers]))
    else:
        print(str(price) + ": ", sum([x.surplus(price) for x in consumers]))
```

    Market Price: Consumer Surplus

    10: 0
    9:  1
    8:  3
    7:  6
    6:  10
    5:  15
    4:  21
    3:  28
    2:  36
    1:  45

Clearly there must be an opposite "force" at play here, otherwise all prices would converge to 0 as consumers maximize their surplus (more on maximization later). Naturally, we must also consider the producers who sell their product to the consumers. We essentially repeat the exercise above, but now instead of a consumer class with individual willingness to pay, we have a producer class with some minimal market price at which production can occur.

```python
class Producer:
    def __init__(self, WTA):
        self.WTA = WTA
    def supply(self, price):
        if price >= self.WTA:
            return 1
        else:
            return 0
    def surplus(self, price):
        if price >= self.WTA:
            return price - self.WTA
        else:
            return 0

producers = [Producer(x) for x in range(1,11)]

for price in range(10,0,-1):
    print([x.surplus(price) for x in producers])
```

    [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
    [8, 7, 6, 5, 4, 3, 2, 1, 0, 0]
    [7, 6, 5, 4, 3, 2, 1, 0, 0, 0]
    [6, 5, 4, 3, 2, 1, 0, 0, 0, 0]
    [5, 4, 3, 2, 1, 0, 0, 0, 0, 0]
    [4, 3, 2, 1, 0, 0, 0, 0, 0, 0]
    [3, 2, 1, 0, 0, 0, 0, 0, 0, 0]
    [2, 1, 0, 0, 0, 0, 0, 0, 0, 0]
    [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Looks familiar, but with an opposite slope! Here we've captured the idea of producer surplus. At a market price of 10, the leftmost producer is very happy with a surplus of 9, as in this case that producer is actually able to produce and sell at a price of 1 but is able to operate at a price of 10.

### Some Thoughts

Before we continue, let's take a moment to think about the meaning and significance of our findings. Firms that can produce at lower market prices than their peers seem to be better off in the sense that they enjoy higher surplus. This minimum production price is based on the costs of operation the firm experiences, so naturally it seems that firms that can operate at lower costs do better. Certainly, if market prices decrease, more inefficent firms would be the first to shut down while these low operating cost firms continue to do business. This idea is very important in economics: Firms that can reduce their costs are rewarded with higher surplus. This is pretty much how society advances, at least in an economics context. Production methods continually to improve, and less efficient firms must either follow suit or shut down as prices decrease, to the benefit of consumers.

However, what would the equivalent be for the consumer side of things? We've discussed the idea of willingness to pay, and initially it might seem that in our perfectly-competitive market environment, only the consumers who most need a good or service will be the first to get it, as their WTP is the highest. We might think that resources are efficiently allocated in this way. Most of the time this is likely the case, but we've made an assumption while reaching this conclusion; an assumption that doesn't necessarily hold. We have assumed that a person with high willingness to pay also has at least an equally high *ability* to pay. In reality, this might not be the case. A hungry person might have high WTP for a serving of food, but if this person lacks the means to pay for this food, his willingness to pay won't do him much good. In this scenario, our earlier exercise reflects willingness to pay with a simultaneous ability to pay as well. While this week isn't about the ethics of certain types of markets and whether they achieve their goals, it's important to keep in mind that in these ideal exercises, an efficient economy with rational pricing should reflect consumers' willingness to pay, whereas in reality this might not actually be the case.

## Note on the Demand and Supply Curves

As pointed out above, the matrix we saw with rows of surpluses and columns of prices resembles the demand curve in the sense that we can see a diagonal line separating participants from non-participants. This is no coincidence. This idea is essentially what the demand and supply curves depict, except that due to there usually being many participants in a market, we simplify the concept to a continuous curve as opposed to a set of discrete values. This is helpful not only for visualization, but as we will soon see we can use these curves to find rates of change, which will prove to be useful as well.

Earlier we had a matrix of each individual's surplus at each price point, and the overall surplus at each price point. Notice how as the price decreased, surplus increased. Let's see this exact same concept illustrated on a familiar demand curve. Take a few moments to adjust the slider controlling the market price to see how consumer surplus behaves.

```python
def demand(price):
    p = sympy.Symbol("p")
    consumers = [Consumer(x) for x in range(0,11)]
    demand_equation = 10 - p
    prices = [x for x in range(0,11)]
    demand_Q = [demand_equation.subs(p,x) for x in prices]

    plt.figure(figsize = [11,7])
    plt.plot(prices, demand_Q)
    plt.plot(demand_equation.subs(p,price), price, 'ro')

    triangle1 = patches.Polygon([[demand_equation.subs(p,price),price], [0,price], [0,10]], True, color='g')
    current_axis = plt.gca()
    current_axis.add_patch(triangle1)

    plt.xlabel("Demand Quantity", fontsize=14)
    plt.ylabel("Price", fontsize=14)
    plt.title("Demand Curve with Consumer Surplus Shaded", fontsize=18)
    plt.show()
    print("Consumer surplus is equal to: "+str(sum([person.surplus(price) for person in consumers])))

#slider1 = widgets.IntSlider(min=1, max=10, step=1, value=5)
#display(widgets.interactive(demand, price=slider1))
demand(5)
```

Producer surplus with the supply curve works exactly the same way but mirrored to reflect the fact that producers gain surplus from higher prices instead of lower.

```python
def supply(price):
    p = sympy.Symbol("p")
    producers = [Producer(x) for x in range(1,11)]
    supply_equation = p
    prices = [x for x in range(0,11)]
    supply_Q = [supply_equation.subs(p,x) for x in prices]

    plt.figure(figsize = [11,7])
    plt.plot(prices, supply_Q, 'r')
    plt.plot(supply_equation.subs(p,price), price, 'ro')

    triangle1 = patches.Polygon([[supply_equation.subs(p,price),price], [0,price], [0,0]], True, color='g')
    current_axis = plt.gca()
    current_axis.add_patch(triangle1)

    plt.xlabel("Supply Quantity", fontsize=14)
    plt.ylabel("Price", fontsize=14)
    plt.title("Supply Curve with Producer Surplus Shaded", fontsize=18)
    plt.show()
    print("Producer surplus is equal to: "+str(sum([person.surplus(price) for person in producers])))

#slider1 = widgets.IntSlider(min=1, max=10, step=1, value=5)
#display(widgets.interactive(supply, price=slider1))
supply(5)
```

Here we used a demand curve of $10-P$ and a supply curve of $P$. Now explore how the surplus changes with shifts in the curves below. `demandParam` adjusts the intercept of the demand curve, `supplyParam` adjusts the slope of the supply curve.

```python
def equilibrium(demandParam, supplyParam):
    p = sympy.Symbol("p")
    demand_equation = demandParam - p
    supply_equation = p * (supplyParam / 10)
    price_end = sympy.solve(demand_equation)[0]
    prices = []
    demand_Q = []
    supply_Q = []
    for price in range(0, price_end + 1):
        prices += [price]
        demand_Q += [demand_equation.subs(p,price)]
        supply_Q += [supply_equation.subs(p,price)]

    equilibrium_P = sympy.solve(demand_equation-supply_equation)[0]
    equilibrium_Q = demand_equation.subs(p,equilibrium_P)

    plt.figure(figsize = [11,7])
    plt.plot(demand_Q,prices, label="Demand")
    plt.plot(supply_Q,prices, 'r', label="Supply")
    plt.legend()
    plt.plot(equilibrium_Q,equilibrium_P, 'ro')

    triangle1 = patches.Polygon([[equilibrium_Q,equilibrium_P], [0,equilibrium_P], [0,price_end]], True, color="green")
    triangle2 = patches.Polygon([[equilibrium_Q,equilibrium_P], [0,equilibrium_P], [0,0]], True, color="lightgreen")
    current_axis = plt.gca()
    current_axis.add_patch(triangle1)
    current_axis.add_patch(triangle2)

    plt.xlabel("Supply and Demand Quantity", fontsize=14)
    plt.ylabel("Price", fontsize=14)
    plt.title("Supply and Demand Curves with Surplus Shaded", fontsize=18)
    plt.ylim(0, 15)
    plt.xlim(0, 10)
    plt.show()
    print("The equilibrium price is "+str(round(equilibrium_P,2))+" and equilibrium quantity is "+str(round(equilibrium_Q,2))+".")
    print("The consumer surplus at this equilibrium "+str((price_end-equilibrium_P)*(equilibrium_Q)*.5))
    print("The producer surplus at this equilibrium "+str((equilibrium_P)*(equilibrium_Q)*.5))

#slider1 = widgets.IntSlider(min=5, max=15,step=1,value=10)
#slider2 = widgets.IntSlider(min=5, max=15,step=1,value=10)
#display(widgets.interactive(equilibrium, demandParam=slider1, supplyParam=slider2))
equilibrium(10, 10)
```


author: leezeeyee
date: 2021/4/15
link: [github](https://github.com/easilylazy/pattern-recognition)

initial: can't using solve_qp in qpsolevers: Q should be positive
因为把svm转化成QP问题时构造的矩阵不是正定的，但增加一个近似0的偏置数据即可解决

```python
from numpy import array, dot
from qpsolvers import solve_qp

M = array([[1., 2., 0.], [-8., 3., 2.], [0., 1., 1.]])
P = dot(M.T, M)  # this is a positive definite matrix
q = dot(array([3., 2., 3.]), M).reshape((3,))
G = array([[1., 2., 1.], [2., 0., 1.], [-1., 2., -1.]])
h = array([3., 2., -2.]).reshape((3,))
A = array([1., 1., 1.])
b = array([1.])

x = solve_qp(P, q, G, h, A, b)
print("QP solution: x = {}".format(x))
```

    QP solution: x = [ 0.30769231 -0.69230769  1.38461538]

```python
import numpy as np
import matplotlib.pyplot as plt
import qpsolvers
```

## QP solver

$$
\begin{equation}
\begin{array}{ll}
\underset{x}{\operatorname{minimize}} & \frac{1}{2} x^{T} P x+q^{T} x \\
\text { subject to } & G x \leq h \\
& A x=b \\
& l b \leq x \leq u b
\end{array}
\end{equation}
$$

```python
def addSingleBias(X):
    N=X.shape[0]
    X=np.reshape(X,[N,1])
    bias_vector=np.ones([1,1])
    return np.row_stack((bias_vector,X))
def addBias(X):
    '''
    为维数为N,d的向量X添加偏置向量
    Args:
        X(numpy.ndarray):shape为[N,d]
    Returns:
        numpy.ndarray:shape为[N,d+1]，且第一列列向量全为1
    '''

    try:
        N=X.shape[0]
        d=X.shape[1]
        if d == 1:
            return addSingleBias(X)
        bias_vector=np.ones([N,1])
        return np.column_stack((bias_vector,X))

    except:
        return addSingleBias(X)
```

## loadData

```python
#中国沿海城市的坐标集
X1 = np.array([
    [119.1,34.4],
    [121.3,29.5],
    [120.2,36.1],
    [119.4,39.6],
    [113.5,22.3],
    [118.1,24.5],
    [121.4,34.5],
    [114.1,22.6],
    [110.1,20.3],
    [123.3,38.6],
    [117.2,39.1]])
X1_add = np.array([
    [114,30],
    [116,24],
    [117,32]])
#日本沿海城市的坐标集
X2 = np.array([
    [130.3,32.3],
    [130.6,33.6],
    [139.4,35.3],
    [135.1,34.4],
    [136.6,35.1],
    [130.4,33.6],
    [132.3,34.2],
    [130.5,33.4],
    [129.9,32.8],
    [135.3,34.4],
    [139.5,35.4]])
X2_add = np.array([
    [141,38],
    [138,36],
    [135,35]])
island=np.asarray([123,25])
D=np.r_[X1,X2]
Y=np.r_[np.ones([len(X1),1]),-np.ones([len(X2),1])]
D_add=np.r_[X1,X1_add,X2,X2_add]
Y=np.r_[np.ones([len(X1),1]),-np.ones([len(X2),1])]
Y_add=np.r_[np.ones([len(X1)+len(X1_add),1]),-np.ones([len(X2)+len(X2_add),1])]
```

## basic
原始数据，皆为临海城市

```python
d=2
I=np.eye(d)
smallBias=1e-6
Q= np.block([[smallBias,smallBias*np.ones([1,d])],
                [smallBias*np.ones([d,1]),I]])
p=np.zeros(d+1)
```

```python
A=-np.multiply(addBias(D),Y)
C=-np.ones(len(A))
```

```python
x = solve_qp(Q, p, A, C)#G, h, A, b)
# solve_qp(solver= CVXOPT, OSQP and quadprog))
print("QP solution: x = {}".format(x))
```

    QP solution: x = [30.16075914 -0.25397711  0.05581915]

```python
x
```

    array([30.16075914, -0.25397711,  0.05581915])

### support vector

```python
np.dot(x,addBias(D).transpose())
```

    array([ 1.83226347,  1.        ,  1.64778119,  2.04632989,  2.57912364,
            1.53363104,  1.25369802,  2.44348311,  3.33100754,  1.        ,
            2.57716997, -1.12950042, -1.13312866, -3.27323472, -2.23137036,
           -2.57326263, -1.08233324, -1.53139827, -1.11889478, -1.        ,
           -2.28216578, -3.29305052])

```python
support_vector_site=np.where(np.abs(np.abs(np.dot(x,addBias(D).transpose()))-1)<=1e-6)[0]
for site in support_vector_site:
    print(D[site])
```

    [121.3  29.5]
    [123.3  38.6]
    [129.9  32.8]

### result

```python
print(np.dot(x,addBias(island)))
if np.dot(x.transpose(),addBias(island))>0:
    print('China')
else:
    print('Japan')
```

    [0.31705275]
    China

### display

```python
class Point():
    def __init__(self,x,y):
        self.x=x
        self.y=y
W1=x.reshape(3)
# print(W1)
k=-W1[1]/W1[2]
b=-W1[0]/W1[2]
scale=1000

plt.figure(figsize=(7,5))
P1=Point(scale,k*scale+b)
P2=Point(-scale,-k*scale+b)
plt.scatter(D[:len(X1),0],D[:len(X1),1],marker = 'o', s=20, cmap = plt.cm.Spectral)
plt.scatter(D[len(X1):,0],D[len(X1):,1],marker = 'o', s=20, cmap = plt.cm.Spectral)
plt.scatter(D[support_vector_site][:,0],D[support_vector_site][:,1],marker = '+', s=50, cmap = plt.cm.Spectral)
plt.scatter(island[0],island[1],marker = '*', s=100, cmap = plt.cm.Spectral)
plt.plot([P1.x,P2.x],[P1.y,P2.y],'-')
plt.legend(['Classify plane','China','Japan','support vector','island'])
plt.grid(True)
plt.xlim((100, 150))
plt.ylim((0, 60))
plt.show()
```

## pro
增加非沿海城市数据

```python
d=2
I=np.eye(d)
smallBias=1e-6
Q= np.block([[smallBias,smallBias*np.ones([1,d])],
                [smallBias*np.ones([d,1]),I]])
p=np.zeros(d+1)
```

```python
A=-np.multiply(addBias(D_add),Y_add)
C=-np.ones(len(A))
```

```python
x = solve_qp(Q, p, A, C)#G, h, A, b)
# solve_qp(solver= CVXOPT, OSQP and quadprog))
print("QP solution: x = {}".format(x))
```

    QP solution: x = [30.16075914 -0.25397711  0.05581915]

### support vector

```python
support_vector_site=np.where(np.abs(np.abs(np.dot(x,addBias(D).transpose()))-1)<=1e-6)[0]
for site in support_vector_site:
    print(D[site])
```

    [121.3  29.5]
    [123.3  38.6]
    [129.9  32.8]

### result

```python
print(np.dot(x,addBias(island)))
if np.dot(x.transpose(),addBias(island))>0:
    print('China')
else:
    print('Japan')
```

    [0.31705275]
    China

### display

```python
class Point():
    def __init__(self,x,y):
        self.x=x
        self.y=y
W1=x.reshape(3)
# print(W1)
k=-W1[1]/W1[2]
b=-W1[0]/W1[2]
scale=1000

plt.figure(figsize=(7,5))
P1=Point(scale,k*scale+b)
P2=Point(-scale,-k*scale+b)
plt.scatter(D[:len(X1),0],D[:len(X1),1],marker = 'o', s=20, cmap = plt.cm.Spectral)
plt.scatter(D[len(X1):,0],D[len(X1):,1],marker = 'o', s=20, cmap = plt.cm.Spectral)
plt.scatter(D[support_vector_site][:,0],D[support_vector_site][:,1],marker = '+', s=50, cmap = plt.cm.Spectral)
plt.scatter(island[0],island[1],marker = '*', s=100, cmap = plt.cm.Spectral)
plt.plot([P1.x,P2.x],[P1.y,P2.y],'-')
plt.xlim((100, 150))
plt.ylim((0, 60))
plt.show()
```

```python

```