# Week 2

# Lecture 3 - Aug 31

## Least Squares by Gradient Descent

We left off last week needing to minimize a loss function for linear regression, i.e. the minimization problem below.

$$\min\limits_w\,L(w)=\min\limits_w\,\|Xw-y\|^2$$

We will use the method of **gradient descent** to find an approximate solution. Gradient descent is a very quick method exploiting some pretty simple ideas from multivariable calculus. While it will only find an approximate answer at best, it is practically good enough in most cases. The benefits are that it is computationally cheap and can be used for many other useful loss functions.

Gradient descent (and it's sped-up version, **stochastic gradient descent** or **SGD**) is *heavily* used in machine learning. Along with backpropagation, SGD is the primary method used for training neural networks. In fact, the first practical neural networks were written about in the machine learning literature under a class of methods called "gradient-based learning" due to the primacy of SGD and related methods.

## Some Ideas from Multivariate Calculus

First, we need just a few ideas from multivariate calculus. This is quite minimal, but you learn more details about these topics in sections 4.3 (partial derivatives), 4.6 (gradients), and 4.7 (multivariate optimization) of *Calculus Volume 3* by Strang.

The ideas we need for gradient descent include:

* If we have a differentiable function of several variables, like our loss function $L(w) = L(w_0, ..., w_d)$, we can define the **partial derivatives** with respect to each of these variables as

    $$L_{w_i}=\frac{\partial L}{\partial w_i}=\lim\limits_{h\to 0}\frac{L(w+he_i) - L(w)}{h},$$

    where $e_i$ is a $(d+1)$-vector with all 0s except for a 1 in the $i$th component.  Geometrically, this partial derivative is the slope of $L$ if we go in the direction of $e_i$.

* To **minimize a multivariable function** by hand, we need to find critical points, which are points $w$ where *all* partial derivatives are 0 and compare which ones give the lowest outputs. In numerical algorithms, must settle for approximations that are "nearly" critical points.

* If we put these partial derivatives into a vector of $d+1$ variables, we call that a **gradient**, which we denote

    $$
    \nabla L(w)
    =\begin{pmatrix}
    L_{w_0}(w) \\
    \vdots \\
    L_{w_d}(w)
    \end{pmatrix}
    $$

* The **directional derivative** of a function in the direction of a unit vector $u$ starting from a point $w$.

    $$D_u L(w) = \lim\limits_{h\to 0}\frac{L(w+hu) - L(w)}{h},$$

    which is the slope in the direction of the vector $u$. Since $L$ is differentiable, this directional derivative will be defined for all directions leaving from $w$. In 1D, there are just two directions: left of right. In 2D, we have directional derivatives at every angle in a circle around the point.

* A common theorem says the **directional derivative is maximized in the direction of the gradient** at each point, so the gradient gives the direction of the *steepest ascent* in the function $L$. Similarly, the direction of the *steepest descent* is $-\nabla L(w)$, the opposite direction.

### The Geometry of Gradient Descent

We will discuss the geometry of gradient-based methods in class, but let's discuss a general outline of how gradient descent works, setting aside the stochastic version for now. The goal of gradient descent is to approximately solve the minimization problem

$$\min\limits_{w}\,L(w)$$

by finding (approximate) critical values by making a guess for the location of a critical value, taking a small step in the opposite direction as the gradient, and repeating this over and over until, hopefully, we reach a minimum value.

The steps are:

0. Make a guess for the critical value -- $w^0$
1. Compute the gradient of $L$ at $w^0$
2. Take a small step to $w^1 = w^0 - \alpha\nabla L\left(w^0\right)$
3. Compute the gradient of $L$ at $w^1$
4. Take a small step to $w^2 = w^1 - \alpha\nabla L\left(w^1\right)$
5. (repeat until the gradient gets close to $(0, ..., 0)$)

This $\alpha>0$ is a number that will be used in the algorithm as a multiplier of the steps the method will take. This is called the **learning rate**.

This idea seems plausible from the calculus ideas above because we just keep switching directions and making a step in the direction of the steepest downward path--the opposite direction as the gradient--until we reach a good place. This is a "greedy" algorithm because it just picks the quickest step in each iteration, which is fast, but it is likely to land in the first minimum it finds, which may or may not be optimal.

If you had two parameters, $L$ would be like a 3D curved surface. A nice visual to have in mind is a rain drop falling on a huge leaf. The droplet of water will move in the steepest downward direction due to gravity--but this direction *changes* as the drip follows the contours of the leaf. This is what gradient descent does.

Will the drip land in the physically lowest altitude part of the leaf? Maybe, but maybe not. If the rain drop lands on the edge, it will probably just roll off the edge. If the leaf has a few different "sinks," different initial locations of the rain drop might cause it to land in these different ones, some of which have lower altitudes than others. Now, if there is heavy rain and lots of rain drops land on the leaf, we can be pretty sure *some* of them will reach the lowest-altitude sink.

From this analogy, you might get the idea that we can make several initial guesses and run it to be more confident we will find the global minimum and not just a local minimum.

In the end, if some of our initial guesses are good choices, the step size $\alpha$ is not too big or too small, and the loss function is pretty well-behaved, the method will converge approximately to a local minimum.

### Implementing Gradient Descent

Before writing code for gradient descent, let's import some libraries.

```python
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import normalize
from sklearn.preprocessing import scale
import matplotlib.pyplot as plt
import pandas as pd

# increase the width of boxes in the notebook file (this is only cosmetic)
np.set_printoptions(linewidth=180)
```

Now, let's implement gradient descent and test it.

Gradient descent will need a few inputs:

* A function to minimize $f$
* A starting point $x_0$
* A learning rate $\alpha$
* A small number $h$ (the variable that goes to 0 in the definition of the derivative)
* A small positive value that we can use for a stopping condition for the derivative being sufficiently small (the tolerance)
* A maximum number of iterations

```python
# estimate the gradient
def computeGradient(f, x, h):
    n = len(x)
    gradient = np.zeros(n)

    for counter in range(n):
        xUp = x.copy()
        xUp[counter] += h
        gradient[counter] = (f(xUp) - f(x))/h

    return gradient

# run gradient descent ant output the
def gradientDescent(f, x0, alpha, h, tolerance, maxIterations):
    # set x equal to the initial guess
    x = x0

    # take up to maxIterations number of steps
    for counter in range(maxIterations):
        # update the gradient
        gradient = computeGradient(f, x, h)

        # stop if the norm of the gradient is near 0
        if np.linalg.norm(gradient) < tolerance:
            print('Gradient descent took', counter, 'iterations to converge')
            print('The norm of the gradient is', np.linalg.norm(gradient))
            # return the approximate critical value x
            return x

        # if we do not converge, print a message
        elif counter == maxIterations-1:
            print("Gradient descent failed")
            print('The gradient is', gradient)
            # return x, sometimes it is still pretty good
            return x

        # take a step in the opposite direction as the gradient
        x -= alpha*gradient
```

Let's test it on some simple functions

```python
f = lambda x : x[0]**2

x = gradientDescent(f,[2],0.4,0.4,0.000001,10000)
print(x, f(x))
```

    Gradient descent took 10 iterations to converge
    The norm of the gradient is 4.5055999998641627e-07
    [-0.19999977] 0.03999990988805076

```python
f = lambda x: np.sin(x)

x = gradientDescent(f,[2],0.5,0.5,0.0001,10000)
print(x, f(x))
```

    Gradient descent took 18 iterations to converge
    The norm of the gradient is 6.221664077488143e-05
    [4.46232611] [-0.96889687]

Let's implement a class for linear regression using gradient descent to estimate $w$.

```python
class LeastSquaresGradient:

    # fit the model to the data
    def fit(self, X, y, w0, alpha, h, tolerance, maxIterations):
        self.n = X.shape[0]
        self.d = X.shape[1]
        self.h = h
        self.alpha = alpha
        self.initialGuess = w0

        # save the training data
        self.data = np.hstack((np.ones([self.n, 1]), X))

        # save the training labels
        self.outputs = y

        # find the w values that minimize the sum of squared errors via gradient descent
        X = self.data
        L = lambda w: ((X @ w).T - y.T) @ (X @ w - y)
        self.w = self.gradientDescent(L, self.initialGuess, self.alpha, self.h, tolerance, maxIterations)

    # predict the output from testing data
    def predict(self, X):
        # initialize an empty matrix to store the predicted outputs
        yPredicted = np.empty([X.shape[0],1])

        # append a column of ones at the beginning of X
        X = np.hstack((np.ones([X.shape[0],1]), X))

        # apply the function f with the values of w from the fit function to each testing datapoint (rows of X)
        for row in range(X.shape[0]):
            yPredicted[row] = self.w @ X[row,]

        return yPredicted

    # run gradient descent to minimize the loss function
    def gradientDescent(self, f, x0, alpha, h, tolerance, maxIterations):
        # set x equal to the initial guess
        x = x0

        # take up to maxIterations number of steps
        for counter in range(maxIterations):
            # update the gradient
            gradient = self.computeGradient(f, x, h)

            # stop if the norm of the gradient is near 0
            if np.linalg.norm(gradient) < tolerance:
                print('Gradient descent took', counter, 'iterations to converge')
                print('The norm of the gradient is', np.linalg.norm(gradient))
                # return the approximate critical value x
                return x

            # if we do not converge, print a message
            elif counter == maxIterations-1:
                print("Gradient descent failed")
                print('The gradient is', gradient)
                # return x, sometimes it is still pretty good
                return x

            # take a step in the opposite direction as the gradient
            x -= alpha*gradient

    # estimate the gradient
    def computeGradient(self, f, x, h):
        n = len(x)
        gradient = np.zeros(n)

        for counter in range(n):
            xUp = x.copy()
            xUp[counter] += h
            gradient[counter] = (f(xUp) - f(x))/h

        return gradient
```

Let's try it!

```python
X = np.array([[6], [7], [8], [9], [7]])
y = np.array([1, 2, 3, 3, 4])

# instantiate an least squares object, fit to data, predict data
model = LeastSquaresGradient()

print('Fitting the model...\n')
model.fit(X, y, [0, 0], alpha = 0.001, h = 0.001, tolerance = 0.01, maxIterations = 100000)

predictions = model.predict(X)

# print the predictions
print('\nThe predicted y values are', predictions.T[0])

# print the real y values
print('The real y values are', y)

# print the w values
parameters = model.w
print('The w values are', parameters)

# plot the training points
plt.scatter(X, y, label = 'Data')

# plot the fitted model with the data
xModel = np.linspace(6,10,100)
yModel = parameters[0] + parameters[1]*xModel

# write a string for the formula
lineFormula = 'y={:.3f}+{:.3f}x'.format(parameters[0], parameters[1])

# plot the model
plt.plot(xModel, yModel, 'r', label = lineFormula)

# add a legend
plt.legend()

# return quality metrics
print('\nThe r^2 score is', r2_score(y, predictions))
print('The mean squared error is', mean_squared_error(y, predictions))
print('The mean absolute error is', mean_absolute_error(y, predictions),'\n')
```

Clearly, the model seems to work. It's not a great fit, although the data is not really linear, so the model cannot fit it well.

## Cross-Validation: Train, Dev, and Test Datasets

(see the lecture notes: in Python, we can use the `train_test_split` function from the `scikit-learn` library to randomly assign datasets to training and testing sets. If we need a dev set, we can run it twice!)

## Example: High School Graduate Rates in US States

Let's try to use least squares on a real dataset. The CSV file in `../data/US_State_data.csv` contains data from each U.S. state.

We would like to predict the output variable included, the high school graduation rate, from some input variables: including the crime rate (per 100,000 persons), the violent crime rate (per 100,000 persons), average teacher salary, student-to-teacher ratio, education expenditure per student, population density, and median household income.

This means we have 50 examples (one for each state), 7 input (predictor) variables, and one output (response) variable. In order to use the formula we derived above to attack the problem with least squares, we need to find the matrices $X$ and $y$.

```python
# import the data from the csv file to an numpy array
data = pd.read_csv('../data/US_State_Data.csv', sep=',').to_numpy()

X = np.array(data[:,1:8], dtype=float)
y = np.array(data[:,8], dtype=float)

# split the data into training and test sets
(trainX, testX, trainY, testY) = train_test_split(X, y, test_size = 0.25, random_state = 1)

trainX = scale(trainX)
testX = scale(testX)

# instantiate a least squares model
model = LeastSquaresGradient()

# fit the model to the training data (find the w parameters)
print('Fitting the model...\n')
model.fit(trainX, trainY, [0, 0, 0, 0, 0, 0, 0, 0], alpha = 0.001, h = 0.001, tolerance = 0.01, maxIterations = 100000)

# return the predicted outputs for the datapoints in the training set
trainPredictions = model.predict(trainX)

# print the coefficient of determination r^2
print('\nThe r^2 score is', r2_score(trainY, trainPredictions))

# print quality metrics
print('\nThe mean absolute error on the training set is', mean_absolute_error(trainY, trainPredictions))

# return the predicted outputs for the datapoints in the test set
predictions = model.predict(testX)

# print the predictions
print('\nThe predicted y values for the test set are', np.round(predictions.T[0],0))

# print the real y values
print('The real y values for the test set are     ', testY)

# print the weights
print('\nThe weights are', model.w)

# print quality metrics
print('\nThe mean absolute error on the test set is', mean_absolute_error(testY, predictions), '\n')
```

    Fitting the model...

    Gradient descent took 834 iterations to converge
    The norm of the gradient is 0.009954123437901363

    The r^2 score is 0.40352475296449164

    The mean absolute error on the training set is 3.6674122809135645

    The predicted y values for the test set are [82. 80. 89. 85. 81. 81. 85. 90. 78. 84. 83. 80. 83.]
    The real y values for the test set are      [70. 83. 83. 81. 76. 87. 89. 89. 78. 78. 84. 80. 79.]

    The weights are [82.97247297 -3.76322781  0.79836794 -1.79592293 -1.16423823 -1.61290262  2.114449    0.85703448]

    The mean absolute error on the test set is 4.004933566374094

## Comments on Gradient Descent

* We must be careful with the $h$ and tolerance hyperparameters to be sure gradient descent will converge.

* Gradient descent in our implementation above does not actually require any derivatives since we only used approximate derivatives.

* If we knew formulas for the derivatives, we could compute them exactly to let the step size be exactly proportional to $\nabla L$. This would drastically reduce the number of times we compute the loss function and increase speed.

* We will use exact derivatives as well as other approaches to speed up gradient descent when we start to build huge neural networks.

* Gradient descent and related methods are the main driver of many machine learning problems that are based on to minimizing a loss function (least squares and neural networks, among others)

## Regularization

**Regularization** is an approach of adjusting a model in various ways to deal with collinearity among other problems. Another important effect of regularization is that it can sometimes reduce overfitting, a problem where we fit the model to the training data too strongly causing it to perform badly on test data.

For example, in the image below, a regularized model would look more like the green curve, which is much simpler and *probably* captures the real dynamics of the system generating the data better than the blue curve, even though they both fit the training data (the red dots) perfectly.

In general, regularization typically sacrifices some training accuracy through approaches that reduce the dimension of the parameter space, shrink some parameters, and adjust loss functions in such a way that it improves how well the model generalizes to test data and other unknown data.

We will want to add regularization to our toolboxfor reducing overfitting whenever we learn by minimizing a loss function, which is what neural networks do. There are a few common regularization methods use in linear regression, which mirror its usage in neural networks, where they are commonly referred to as **weight decay**, which we will discuss below.

## Ridge Regression

One issue that can cause overfitting is when parameters get far larger than other parameters, which can over-emphasize certain features. **Ridge regression** (also called **Tikhonov regularization** or $L^2$ **regularization**) offers a partial remedy for the problems of overfitting by adjusting the loss function to "encourage" the $w_i$'s to be smaller by **penalizing** large $w_i$'s. To do this, instead of the ordinary least squares loss function

$$L(w)=\sum\limits_{i=1}^n \left(\hat{f}(x_i)-y_i\right)^2= \|Xw-y\|_2^2,$$

we add another term to the loss function to get

$$L_{\text{ridge}}(\lambda,w)=\sum\limits_{i=1}^n \left(\hat{f}(x_i)-y_i\right)^2 + \lambda\sum\limits_{i=1}^dw_i^2=\|Xw-y\|_2^2+\lambda\|w\|_2^2,$$

for some constant $\lambda\geq 0$. So, here, if $w_i$ is large, the loss will be larger. Therefore, when we minimize the loss function, it will push $w_i$'s toward 0 unless there is a really good reason not to do so. That's why we say we *penalize* large $w$ values.

The value $\lambda$ is a hyperparameter of ridge regression. As $\lambda\to 0$, ridge regression approaches ordinary least squares. If $\lambda$ is very large, optimizing the loss function will force the $w$ values to be small. So, the larger we make $\lambda$, the more pressure we put on the $w$ parameters to shrink.

### Note

For ridge regression, you typically should normalize the data before minimizing the loss function. Otherwise, different scaling will cause minimization to penalize some variables more than others.

## LASSO Regression and Elastic Net Regression

**LASSO** (least absolute shrinkage and selection operator) regression is very similar to ridge regression, but it can otherwise be called **$L^1$ regularization** because it adds an $L^1$ penalty to the size of the $w$ parameters, so the loss function is

$$L_{\text{lasso}}(\lambda,w)=\sum\limits_{i=1}^n \left(\hat{f}(x_i)-y_i\right)^2 + \lambda\sum\limits_{i=1}^d |w_i|=\|Xw-y\|_2^2+\lambda\|w\|_1.$$

**Elastic net** regression combines both $L^1$ and $L^2$ regularization as a linear combination. So, the elastic net loss function is

$$L_{\text{elastic}}(\lambda_1,\lambda_2,w)=\sum\limits_{i=1}^n \left(\hat{f}(x_i)-y_i\right)^2 + \lambda_1\sum\limits_{i=1}^d |w_i| + \lambda_2\sum\limits_{i=1}^d w_i^2=\|Xw-y\|_2^2+\lambda_1\|w\|_1+\lambda_2\|w\|_2^2,$$

so it contains two hyperparameters $\lambda_1\geq 0$ and $\lambda_2\geq 0$ controlling how large the $L^1$ and $L^2$ penalties are.

## Ridge vs. LASSO vs. Elastic Net Regression

Ridge, LASSO, and elastic net regression aim to accomplish most of the same tasks:

1. Predict outputs when there are linearly dependent variables
2. Reduce the dimension of the data (practically speaking)
3. improve how well a model can generalize to test data and beyond

So, which one should we use? That's not really an easy question to answer because the usefulness of a model depends on how well it can generalize to unknown datapoints, but they are unknown, so... we don't know much about them!

There are some differences between $L^1$ and $L^2$ penalties that can guide our testing, although only testing is likely to tell us what will work better, practically speaking.

* A loss function with an $L^1$ penalty is NOT differentiable when any $w_i=0$, so we have to be careful with gradient-based methods. A method called soft-thresholding is often used to send parameters directly to 0 if the gradient method brings a $w_i$ sufficiently close close to 0.

* A loss function with an $L^2$ penalty will not cause parameters to go to 0, but $L^1$ can. (We will talk about some of the geometry of why this is true in class. It's also discussed in some videos about ridge and LASSO regression.)

* There is no simple formula for $w$ minimizing the loss functions in LASSO or elastic net regression, so numerical optimization, such as gradient descent, must be used.

* There is a matrix expression for $w$ minimizing the loss function in ridge regression, but gradient descent can be used as well.

This means $L^1$ can totally eliminate variables, which can be good or bad depending on what we are modeling. If the output is well-predicted by only a few variables, this is good. If we need lots of variables to predict the output, this is bad.

These notes only provide a short intro to regularization techniques, but you can read more from the (free) classic book *Elements of Statistical Learning* by Hastie, et. al., in section 3.4.

## Implementing Regularized Linear Regression

To implement these methods, we simply adjust the loss function, and gradient descent will still work. Previously, we had

`L = lambda w: ((X @ w).T - y.T) @ (X @ w - y)`

For ridge regression, simply add a hyperparameter $\lambda_2$ times the sum of squares of the weights:

`L = lambda w: ((W @ w).T - y.T) @ (X @ x - y) + l2 * w.T @ w`

For LASSO regression, simply add a hyperparameter $\lambda_1$ times the sum of absolute values of the weights:

`L = lambda w: ((W @ w).T - y.T) @ (X @ x - y) + l1 * np.sum(np.abs(w))`

For elastic net, add both parts:

`L = lambda w: ((W @ w).T - y.T) @ (X @ x - y) + l1 * np.sum(np.abs(w)) + l2 * w.T @ w`

# Lecture 4 - Sept 2

### Classification Problems

**Classification problems** are problems where we would like to take datapoints and assign them to an appropriate **class** based on some examples that have known classes.

### Examples

* If we have a dataset of medical records where each datapoint has a single patient's age, weight, blood pressure, status as a smoker or not, and other information and each patient is known to either have or not have kidney disease. We might want take data from a new patient and predict if he or she is likely to develop kidney disease.

* If we have a dataset of labeled images of cat and dogs, we might want to take a new image and classify whether it has a dog or a cat. (How does Google image search know how to find pictures of what you search?)

* If we have a dataset of audio files, each of which is a jazz, classical, rock, pop, or hip-hop song, we might want to predict the genre of a new audio file. (Spotify does this kind of analysis to recommend songs based on your listening history.)

* If we have a dataset of traffic logs on a network, some known to be infected by a specific virus and some are not, we might want to use this information to classify a new traffic log as likely to be infected or not.

* If we have a dataset of sounds of people speaking along with transcripts of the words, we might want to classify the words spoken into a microphone. (Think Siri!)

In all of these cases, the information can be represented as a point in the $n$-dimensional real space.

* The medical records would have numbers for age, weight, and blood pressure and a binary digit for non-smoker or smoker.

* The dog/cat images might have three channels for a picture, meaning three numbers for each pixel (the red, green, and blue levels) like the bird picture below.

* The audio files might have numbers specifying the type of sound for the song many times per second.

* The network traffic logs might have numbers of packets transferred, file size, ports, addresses, the content of the packets, etc.

* The audio files might have numbers specifying the type of sound for a word many times per second.

The blue lines indicate the beginning of a word and the red lines indicate the ends of words.

In all mature applications, there are likely preprocessing steps done before the classification is done.

I chose these applications to demonstrate two things: (1) classification problems are interesting and useful in almost every area of study and (2) a huge class of classification problems have much in common mathematically. All apply to datapoints, although some types of data may have far more dimensions than others--a medical record may only have 10 to 12 numbers, but a 12-megapixel photo from the latest iPhone would have $4,000\times 3,000\times 3=36,000,000$ numbers, 3 for each pixel).

### The Math of a Classification Problem

To exploit the similarities, let's abstract away the specifics of the applications for now and think about how to describe a classification problem mathematically. Consider a $d$-dimensional point, or vector, $x_1\in\mathbb{R}^d$ and denote $x=(x_{11},x_{12},...,x_{1d})$. $x$ is a member one of $k$ classes $C=\{c_1, c_2, ..., c_k\}$. We call the point $x_1$ an **example** and we call the class the **label** of $x_1$.

The goal of a classification problem is to find a function $M:\mathbb{R}^d\to C$ mapping each example $x_1$ to its class $y_1=M(x_1)$ and will generalize to successfully classify new, unlabeled datapoints with high accuracy.

This will segment the space $\mathbb{R}^d$ into sets $X_j=\{x_1\in\mathbb{R}^d | M(x_1)=c_j\}$ corresponding to each class. In the image below, for example, the space $\mathbb{R}^2$ is partitioned into three sets colored red, blue, and green.

The colored points are labeled examples and the $\mathbb{R}^2$ space is colored by the class to be assigned to points in different regions. (image from Wikipedia)

### Classification Algorithms

There are many algorithms used for classification. Some of the most popular include

* Bayes and naive Bayes
* $k$-nearest neighbors
* decision trees
* logistic regression
* support vector machines
* neural networks (many types)

as well as tree-based algorithms that systematically apply an ensemble of different classifiers. Any of the methods above are good choices and there are pros and cons of each, but we only consider tiny neural networks and logistic regression this week.

An array of classifiers are used on a few datasets in the code below from [scikit-learn](https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html). I ask you not to focus on the code, but look at the diagram it generates. The diagram shows how different classifiers come to quite different results at classifying 2D points into the red and blue classes.

```python
print(__doc__)

# Code source: Gaël Varoquaux
#              Andreas Müller
# Modified for documentation by Jaques Grobler
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

h = .02  # step size in the mesh

names = ["Nearest Neighbors", "Linear SVM", "RBF SVM", "Gaussian Process",
         "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
         "Naive Bayes", "QDA"]

classifiers = [
    KNeighborsClassifier(3),
    SVC(kernel="linear", C=0.025),
    SVC(gamma=2, C=1),
    GaussianProcessClassifier(1.0 * RBF(1.0)),
    DecisionTreeClassifier(max_depth=5),
    RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
    MLPClassifier(alpha=1, max_iter=1000),
    AdaBoostClassifier(),
    GaussianNB(),
    QuadraticDiscriminantAnalysis()]

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,
                           random_state=1, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(noise=0.3, random_state=0),
            make_circles(noise=0.2, factor=0.5, random_state=1),
            linearly_separable
            ]

figure = plt.figure(figsize=(27, 9))
i = 1
# iterate over datasets
for ds_cnt, ds in enumerate(datasets):
    # preprocess dataset, split into training and test part
    X, y = ds
    X = StandardScaler().fit_transform(X)
    X_train, X_test, y_train, y_test = \
        train_test_split(X, y, test_size=.4, random_state=42)

    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    if ds_cnt == 0:
        ax.set_title("Input data")
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright,
               edgecolors='k')
    # Plot the testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.6,
               edgecolors='k')
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
        clf.fit(X_train, y_train)
        score = clf.score(X_test, y_test)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        if hasattr(clf, "decision_function"):
            Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
        else:
            Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)

        # Plot the training points
        ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright,
                   edgecolors='k')
        # Plot the testing points
        ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright,
                   edgecolors='k', alpha=0.6)

        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        if ds_cnt == 0:
            ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1

plt.tight_layout()
plt.show()
```

### Measuring Success of Classifiers

 For a specific class labeled 1, we denote

$c_{00}$ is the number of test examples in class 0 classified in class 0 (true negatives)

$c_{01}$ is the number of test examples in class 0 classified in class 1 (false positives)

$c_{10}$ is the number of test examples in class 1 classified in class 0 (false negatives)

$c_{11}$ is the number of test examples in class 1 classified in class 1 (true positives)

These numbers are sometimes included in what is called a **confusion matrix**

$$
\begin{align}
\begin{pmatrix}
c_{00} & c_{01} \\
c_{10} & c_{11}
\end{pmatrix}
\end{align}
$$

There are a few different measures of success in common usage. Each one can be computed for each class.

* Accuracy

$$
\begin{align}
\text{Accuracy}=\frac{\text{# of true classifications}}{\text{# of test examples}}=\frac{c_{00}+c_{11}}{c_{00}+c_{01}+c_{10}+c_{11}}
\end{align}
$$

* Precision

$$
\begin{align}
\text{Precision}=\frac{\text{# of true positives}}{\text{# of positive classifications}}=\frac{c_{11}}{c_{01}+c_{11}}
\end{align}
$$

* Recall

$$
\begin{align}
\text{Recall}=\frac{\text{# of true positives}}{\text{# of positive examples}}=\frac{c_{11}}{c_{10}+c_{11}}
\end{align}
$$

* F1 Score

$$
\begin{align}
F_1=\frac{2\text{(precision)(recall)}}{\text{precision + recall}}
\end{align}
$$

It is convenient to use the classification_report function from the sklearn.metrics module with inputs of the test set labels and then the labels predicted by the classifier. This will return the precision and recall for each class, accuracy, and some additional quality metrics.

## A One-Node Neural Network for Binary Classification

Next, we show how classification can be done for a binary classification problem with a neural network of just one node, similar to the way we represented linear regression as a one-node neural network. The method is conventionally known as **logistic regression** if a very specific loss function is used, as we will see later.

In a binary classification problem, each datapoint is assumed to belong to class 0 or it belongs to class 1. Therefore, for a datapoint $x_i$ will be mapped to $y_i=0$ or $y_i=1$.

As usual, suppose we have a modified data matrix, labels, and weights:

$$
X=\begin{pmatrix}
1 & x_{11} & x_{12} & \cdots & x_{1d}\\
1 & x_{21} & x_{22} & \cdots & x_{2d}\\
\vdots & \vdots & \vdots & \ddots & \vdots\\
1 & x_{n1} & x_{n2} & \cdots & x_{nd}
\end{pmatrix}
\hspace{2cm}y=\begin{pmatrix}
y_1 \\ y_2 \\ \vdots \\ y_n
\end{pmatrix}
\hspace{2cm}w=\begin{pmatrix}
w_0 \\ w_1 \\ \vdots \\ w_d
\end{pmatrix}
$$

We hope to train our model to map $x_i$ values to their corresponding targets $y_i$. With linear regression, the predicted $y$ values ($\hat{y}$) were computed as $Xw$, which can be any real number, depending on the input data $X$ and weights $w$. This was good for regression, where we tried to predict a numerical value.

However, with classification, we will predict a probability the datapoint belongs to class 1 which must be a number between 0 and 1 and round it to the nearest integer to predict the class. So, the question becomes: how can we "squish" this $Xw$ into $[0,1]$? The **sigmoid** or **logisitc** function does precisely this: its domain is $\mathbb{R}$, but its range is just $[0,1]$. It has some other nice properties that will help us in the future.

The sigmoid function is defined as

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Therefore, we can attempt to solve the classification problem using a very similar minimization problem that we used with linear regression, except our predicted probabilities will be computed as $\sigma(Xw)$, with $\sigma$ applied elementwise:

$$L(w)=\|\hat{y}-y\|_2^2=\|\sigma(Xw)-y\|_2^2$$

We will write a simple Python implementation of the method using gradient descent below.

First, let's import some libraries.

```python
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sn

from sklearn import datasets
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
```

Next, let's write a class for the logistic classifier. We only need to lightly modify the `LinearRegressionGradient` class from our last class session to use the sigmoid function.

```python
class LogisticClassifierGradient:

    # fit the model to the data
    def fit(self, X, y, w0, alpha, h, tolerance, maxIterations):
        self.n = X.shape[0]
        self.d = X.shape[1]
        self.h = h
        self.alpha = alpha
        self.initialGuess = w0

        # standardize the data
        X = self.standardize(X)

        # save the training data and add a column of 1s to it
        self.data = np.hstack((np.ones([self.n, 1]), X))

        # save the training labels
        self.outputs = y

        # find the w values that minimize the sum of squared errors via gradient descent
        X = self.data
        L = lambda w: (self.sigmoid((X @ w)).T - y.T) @ (self.sigmoid(X @ w) - y)
        self.w = self.gradientDescent(L, self.initialGuess, self.alpha, self.h, tolerance, maxIterations)

    # predict the output from testing data
    def predict(self, X):
        # standardize the data
        X = self.standardize(X)

        # initialize an empty matrix to store the predicted outputs
        yPredicted = np.empty([X.shape[0],1])

        # append a column of ones at the beginning of X
        X = np.hstack((np.ones([X.shape[0],1]), X))

        # apply the function f with the values of w from the fit function to each testing datapoint (rows of X)
        for row in range(X.shape[0]):
            yPredicted[row] = np.round(self.sigmoid(self.w @ X[row,]))

        return yPredicted

    # run gradient descent to minimize the loss function
    def gradientDescent(self, f, x0, alpha, h, tolerance, maxIterations):
        # set x equal to the initial guess
        x = x0

        # take up to maxIterations number of steps
        for counter in range(maxIterations):
            # update the gradient
            gradient = self.computeGradient(f, x, h)

            # stop if the norm of the gradient is near 0
            if np.linalg.norm(gradient) < tolerance:
                print('Gradient descent took', counter, 'iterations to converge')
                print('The norm of the gradient is', np.linalg.norm(gradient))
                # return the approximate critical value x
                return x

            # if we do not converge, print a message
            elif counter == maxIterations-1:
                print("Gradient descent failed")
                print('The norm of the gradient is', np.linalg.norm(gradient))
                # return x, sometimes it is still pretty good
                return x

            # take a step in the opposite direction as the gradient
            x -= alpha*gradient

    # estimate the gradient
    def computeGradient(self, f, x, h):
        n = len(x)
        gradient = np.zeros(n)
        fx = f(x)

        for counter in range(n):
            xUp = x.copy()
            xUp[counter] += h
            gradient[counter] = (f(xUp) - fx)/h

        return gradient

    def sigmoid(self, z): return 1.0 / (1 + np.exp(-z))

    def standardize(self, X):
        n = X.shape[0]

        # normalize all the n features of X.
        for i in range(n):
            X = (X - X.mean(axis=0))/X.std(axis=0)

        return X
```

### Example: Breast Cancer Detection

```python
breastcancer = datasets.load_breast_cancer()
print(breastcancer['DESCR'])
```

    .. _breast_cancer_dataset:

    Breast cancer wisconsin (diagnostic) dataset
    --------------------------------------------

    **Data Set Characteristics:**

        :Number of Instances: 569

        :Number of Attributes: 30 numeric, predictive attributes and the class

        :Attribute Information:
            - radius (mean of distances from center to points on the perimeter)
            - texture (standard deviation of gray-scale values)
            - perimeter
            - area
            - smoothness (local variation in radius lengths)
            - compactness (perimeter^2 / area - 1.0)
            - concavity (severity of concave portions of the contour)
            - concave points (number of concave portions of the contour)
            - symmetry
            - fractal dimension ("coastline approximation" - 1)

            The mean, standard error, and "worst" or largest (mean of the three
            worst/largest values) of these features were computed for each image,
            resulting in 30 features.  For instance, field 0 is Mean Radius, field
            10 is Radius SE, field 20 is Worst Radius.

            - class:
                    - WDBC-Malignant
                    - WDBC-Benign

        :Summary Statistics:

        ===================================== ====== ======
                                               Min    Max
        ===================================== ====== ======
        radius (mean):                        6.981  28.11
        texture (mean):                       9.71   39.28
        perimeter (mean):                     43.79  188.5
        area (mean):                          143.5  2501.0
        smoothness (mean):                    0.053  0.163
        compactness (mean):                   0.019  0.345
        concavity (mean):                     0.0    0.427
        concave points (mean):                0.0    0.201
        symmetry (mean):                      0.106  0.304
        fractal dimension (mean):             0.05   0.097
        radius (standard error):              0.112  2.873
        texture (standard error):             0.36   4.885
        perimeter (standard error):           0.757  21.98
        area (standard error):                6.802  542.2
        smoothness (standard error):          0.002  0.031
        compactness (standard error):         0.002  0.135
        concavity (standard error):           0.0    0.396
        concave points (standard error):      0.0    0.053
        symmetry (standard error):            0.008  0.079
        fractal dimension (standard error):   0.001  0.03
        radius (worst):                       7.93   36.04
        texture (worst):                      12.02  49.54
        perimeter (worst):                    50.41  251.2
        area (worst):                         185.2  4254.0
        smoothness (worst):                   0.071  0.223
        compactness (worst):                  0.027  1.058
        concavity (worst):                    0.0    1.252
        concave points (worst):               0.0    0.291
        symmetry (worst):                     0.156  0.664
        fractal dimension (worst):            0.055  0.208
        ===================================== ====== ======

        :Missing Attribute Values: None

        :Class Distribution: 212 - Malignant, 357 - Benign

        :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian

        :Donor: Nick Street

        :Date: November, 1995

    This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
    https://goo.gl/U2Uwz2

    Features are computed from a digitized image of a fine needle
    aspirate (FNA) of a breast mass.  They describe
    characteristics of the cell nuclei present in the image.

    Separating plane described above was obtained using
    Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree
    Construction Via Linear Programming." Proceedings of the 4th
    Midwest Artificial Intelligence and Cognitive Science Society,
    pp. 97-101, 1992], a classification method which uses linear
    programming to construct a decision tree.  Relevant features
    were selected using an exhaustive search in the space of 1-4
    features and 1-3 separating planes.

    The actual linear program used to obtain the separating plane
    in the 3-dimensional space is that described in:
    [K. P. Bennett and O. L. Mangasarian: "Robust Linear
    Programming Discrimination of Two Linearly Inseparable Sets",
    Optimization Methods and Software 1, 1992, 23-34].

    This database is also available through the UW CS ftp server:

    ftp ftp.cs.wisc.edu
    cd math-prog/cpo-dataset/machine-learn/WDBC/

    .. topic:: References

       - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction
         for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on
         Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
         San Jose, CA, 1993.
       - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and
         prognosis via linear programming. Operations Research, 43(4), pages 570-577,
         July-August 1995.
       - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques
         to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994)
         163-171.

```python
breastcancer = datasets.load_breast_cancer()

# find the data and labels
X = breastcancer.data
Y = breastcancer.target

# split the data into train and test sets
trainX, testX, trainY, testY = train_test_split(X, Y, test_size = 0.25)

# build the logistic classifier
model = LogisticClassifierGradient()

# fit the Bayes classifier to the training data
model.fit(trainX, trainY, [0] * (X.shape[1] + 1), alpha = 0.001, h = 0.001,
          tolerance = 0.001, maxIterations = 10000)

# predict the labels of the training set
predictedY = model.predict(trainX)

# print quality metrics
print('\nTrain Classification Report:\n\n', classification_report(trainY, predictedY))

# predict the labels of the test set
predictedY = model.predict(testX)

# print quality metrics
print('\nTest Classification Report:\n\n', classification_report(testY, predictedY))
print('\nTest Confusion Matrix:\n')

sn.heatmap(confusion_matrix(testY, predictedY), annot = True)
```

Even though our model is a tiny neural network, it still makes quite good predictions! It correctly classified 99% of training tumor data as benign or malignant.

Even more impressive, it correctly classifies 96% of the test data, which is data the model was not allowed to see during training. The model successfully **generalized** to data it has never seen!


```python
from pythreejs import *
```

```python
import numpy as np
from IPython.display import display
from traittypes import Array
from ipywidgets import HTML, Text, Widget
from traitlets import link, dlink, Any, HasTraits
```

```python
class Surf(HasTraits):

    function = Any()
    points = Array()

    def __init__(self, **kwargs):
        self.xmin = kwargs.pop('xmin', -1)
        self.xmax = kwargs.pop('xmax', 1)
        self.ymin = kwargs.pop('ymin', -1)
        self.ymax = kwargs.pop('ymax', 1)

        self.grid_size = kwargs.pop('grid_size', 20)

        self.points = kwargs.pop('points', np.zeros((1, 3)))
        self.observe(self.update_points, 'points')

        self.point_size = kwargs.pop('size', 0.05)

        self.function = kwargs.pop('function', lambda xx, yy: xx**2 - yy**2)
        self.observe(self.update_surf, 'function')

        self.surf_color = kwargs.pop('surf_color', 'viridis')
        self.line_color = kwargs.pop('line_color', 'black')
        self.point_colors = kwargs.pop('point_colors', ['red'])
        self.cam_position = [0, 3, 3]
        self.cam_up = [0, 0, 1]

        self.plot(**kwargs)
        self.draw(**kwargs)

    def plot(self, **kwargs):
        x = np.linspace(self.xmin, self.xmax, self.grid_size)
        y = np.linspace(self.xmin, self.xmax, self.grid_size)
        xx, yy = np.meshgrid(x, y)
        self.z = self.function(xx, yy)

    def update_surf(self, new):
        self.plot()
        self.surface_geom.z = list(self.z[::-1].flat)
        self.surface.material = LambertMaterial(map=height_texture(self.z[::-1], self.surf_color))

    def update_points(self, new):
        self.draw_points()
        self.marks = [self.surface, self.surfgrid] + self.dots + [AmbientLight(color='#777777')]
        self.scene.children = self.marks

    def draw_surf(self, **kwargs):
        self.surface_geom = SurfaceGeometry(z=list(self.z[::-1].flat),
                         width=2 * self.xmax,
                         height=2 * self.xmax,
                         width_segments=self.grid_size - 1,
                         height_segments=self.grid_size - 1)
        self.surface = Mesh(geometry=self.surface_geom,
                         material=LambertMaterial(map=height_texture(self.z[::-1], self.surf_color)))
        self.surfgrid = SurfaceGrid(geometry=self.surface_geom, material=LineBasicMaterial(color=self.line_color))

    def draw_points(self, **kwargs):
        self.dots = []
        for i in range(self.points.shape[0]):
            self.dots.append(Mesh(geometry=SphereGeometry(radius=self.point_size),
                                    material=LambertMaterial(color=self.point_colors[i % len(self.point_colors
                                                                                )]),
                                    position=self.points[i, :].tolist()))

    def draw(self, **kwargs):
        self.draw_surf(**kwargs)
        self.draw_points(**kwargs)

        self.marks = [self.surface, self.surfgrid] + self.dots + [AmbientLight(color='#777777')]

        self.scene = Scene(children=self.marks)
        self.cam = PerspectiveCamera(position=self.cam_position, up=self.cam_up,
                              children=[DirectionalLight(color='white', position=[3, 5, 1], intensity=0.6)])

        self.figure = Renderer(camera=self.cam,
                                 scene = self.scene, controls=[OrbitControls(controlling=self.cam)])
        return
```

```python
a = Surf()
a.figure
```

```python
a.function = lambda x, y: x**3 - 3*x*(y**2)
```

```python
a.points = np.array([[0, 0, 1], [0, 0, -1]])
```

```python
from sympy import *
```

```python
def evaluate_gradient(func, point, variables=['x', 'y']):
    var = []
    grad = []
    for i in variables:
        var.append(symbols(i))

    if len(point) != len(variables):
        raise ValueError('Length of points doesnt match the number of variables')

    for i in range(len(var)):
        diff_func = lambdify(variables, diff(sympify(func), var[i]))
        grad.append(diff_func(*point))
    return np.array(grad)
```

```python
func = 'x**3 - 3*x*(y**2)'
```

```python
evaluate_gradient(func, [2, 1])
```

    array([  9, -12])

```python
loc = [.2,.2]
```

```python
z=a.function(loc[0],loc[1])
```

```python
a.points = np.array([[loc[0],loc[1],z]])
```

```python
g = evaluate_gradient(func, loc)
```

```python
step_size = .1
```

```python
loc = loc - step_size * g
```

```python
a.points.append([0,0,1])
```

```python
a.points.shape
```

    (1, 3)


 # Milestone 1

 First milestone for CS207 Fall 2018 Project Group 1.

## Introduction

Differentiation is ubiquitous in almost all aspects of computer science, mathematics, and physics. It is used for numeric root-finding as in Newton's Method, and used for optimization with different forms of gradient descent.
However, calculating analytic derivatives is difficult and can lead to exponentially growing abstract syntax trees, which makes finding the derivative infeasible in many cases.
Similarly, calculating the derivative numerically using the limit definition runs into numeric instabilities due to limited machine precision.
Automatic differentiation addresses both of these issues - it uses the chain rule and the fact that computers calculate any function as a sequence of elementary operations to find the derivative.

## Background

Automatic differentiation relies heavily on the principles of chain rule differentiation. A graph of elementary functions is built to calculate the values of more complex functions. Using the chain rule on the graph of elementary functions, the value of the derivative at each node can also be calculated. This gives us the ability to calculate the values of functions and their derivatives, no matter how complex, to near machine precision (a significant advantage compared to alternatives such as finite differences).

The chain rule tells us that:
\begin{align}
\frac{d}{dx}(f(g(x))) &= f'(g(x))g'(x)\\
\end{align}

Since each step in our graph is just a combination of linear operations, we can find the derivative at a node by considering the value and derivative of the expressions at the previous node. By starting with an initial 'seed' vector for the derivative (often set to 1), we can find the derivative in any desired search direction.

Below is an example of constructing a graph to find the exact values of a function and its derivative. The function we used was:

$$f\left(x, y, z\right) = \dfrac{1}{xyz} + \sin\left(\dfrac{1}{x} + \dfrac{1}{y} + \dfrac{1}{z}\right)$$

We worked through this, starting with trace elements $x_1$ for $x$,  $x_2$ for $y$ and  $x_3$ for $z$. We wanted to solve this function at $(x, y, z) = (1, 2, 3)$.

| Trace | Elementary Function | Current Value | Elementary Function Derivative | $\nabla_{x}$ Value  | $\nabla_{y}$ Value  | $\nabla_{z}$ Value  |
| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: |
| $x_{1}$ | $x$ | 1 | $\dot{x}$ | 1 | 0 | 0 |
| $x_{2}$ | $y$ | 2 | $\dot{y}$ | 0 | 1 | 0 |
| $x_{3}$ | $z$ | 3 | $\dot{z}$ | 0 | 0 | 1 |
| $x_{4}$ | $1/x_{1}$ | 1 | $-\dot{x}_{1}/x_{1}^{2}$ | $-1$ | $0$ | $0$ |
| $x_{5}$ | $1/x_{2}$ | 1/2 | $-\dot{x}_{2}/x_{2}^{2}$ | $0$ | $-1/4$ | $0$ |
| $x_{6}$ | $1/x_{3}$ | 1/3 |  $-\dot{x}_{3}/x_{3}^{2}$ | $0$ | $0$ | $-1/9$ |
| $x_{7}$ | $x_{4} + x_{5} + x_{6}$ | 11/6 |$\dot{x}_{4} + \dot{x}_{5} + \dot{x}_{6}$ | -1 | -0.25 | -0.11 |
| $x_{8}$ | $sin(x_{7})$ | 0.966 |$\dot{x}_{7}cos(x_{7})$ | 0.260 | 0.065 | 0.029 |
| $x_{9}$ | $x_{4}x_{5}x_{6}$| 1/6 |$\dot{x}_{4}x_{5}x_{6} + \dot{x}_{5}x_{4}x_{6} + \dot{x}_{6}x_{4}x_{5} $ |-0.167 | -0.083  | -0.056 |
| $x_{10}$ | $x_{8} + x_{9}$ | 1.132 |$\dot{x}_{8} + \dot{x}_{9}$ | 0.093| -0.018  | -0.027 |

This isn't a very complicated function, but it shows how we can use the most basic of functions to create a graph allowing us to find exact values and gradients.

## Usage

Since most expressions and applications that require automatic differentiation are not constructed dynamically, we will start by building the computational graph statically. Then, to perform computations and get the derivative, we feed our computational graph some inputs. A sample usage would look like:

```python
>>> import ad
>>> x = ad.Variable('x')
>>> f = 10.0 + 5.0 * (x ** 2.0) - 3.0 * x
>>> f.eval({'x': 5.0})
120.0
>>> f.d
47.0
```

Multiple inputs could also be provided. For example,

```python
>>> x = ad.Variable('x')
>>> y = ad.Variable('y')
>>> f = x * y
>>> f.eval({'x': 5.0, 'y': 2.0})
10.0
>>> f.d
{x: 2.0, y: 5.0}
```

## Software Organization

### Directory Structure

Our library currently only does Automatic Differentiation. Therefore, we only have one module, which we will call `ad`. We will add additional modules for our extension once we decide what we will be doing for that.

```bash
cs207project
├── LICENSE
├── README
├── ad
│   ├── __init__.py
│   ├── ad.py
│   ├── mat_ops.py
│   ├── plots.py
│   ├── simple_ops.py
│   └── tests
│       ├── test_eval.py
│       ├── test_forward.py
│       ├── test_mat_ops.py
│       └── test_simple_ops.py
└── setup.py
```

Our test suite will use TravisCI for continual integration and Coveralls for testing code coverage. Also, we will distribute our package through PyPI.

## Implementation

In our implementation, “Expression" would be the core data structure. Everything in the computational graph is an expression.

The attributes of an expression includes its gradient and the links to its children. The methods of an expression includes "eval" (get the value of the expression) and "d" (get the gradient of the expression). Dunder methods like \__add__(), \__sub__(), \__mul__(), and \__truediv__()  will be implemented for ease of use.

Classes that we are going to implement for constructing the computational graph include:

* Expression(object)
* Variable(Expression)
* Constant(Expression)

Classes that we will use for operations take in one or two instances of an `Expression` class and returns a new `Expression` class with the operation applied. These will store the children so we can traverse down our computational graph. The initial classes will be:

* Unop(Expression)
* Binop(Expression)
* Addition(Binop)
* Subtraction(Binop)
* Multiplication(Binop)
* Division(Binop)
* Sin(Unop)
* Cos(Unop)
* Exp(Unop)
* Log(Unop)

The `Unop` class is a base class that stores 1 child and `Binop` stores two children. This implementation should be able to handle vector functions of vectors by using multiple `Variable` objects in our function.

## External Dependencies

We will mainly be using `numpy` as an external dependency for mathematical and vector operations.


# Blending High Extraction and Low Extraction Flours

The [Tartine method][Tartine3] calls for wheat flours at various extraction levels which
are not easy to purchase. They can, however, be created by blending low and
high extraction flour. The
[Hamelman Challenge, Tech. Note: High Extraction Flour][Hamelman Challenge]
provides a method for blending whole wheat (100% extraction) and typical bread
flour (~70% extraction) to achieve the desired High Extraction Flour (85%).

I found a [bread flour][Azure Bread Flour] from [Azure Standard] with a flavor
and color I enjoy. The Unifine milling process reduces the bran and germ to small enough particles that they are not noticeable in the texture. Oven spring is good, but the air-pockets are not large, unlike [Tartine][Tartine3] bread. The texture is good but a little dense.

The [Azure Bread Flour] extraction (90%) is higher than the recommended 85% for High-Extraction and 75% for Medium-Strong. So assuming my turning and forming skills are not to blame (*dubious assumption*), I'm going to try blending flours  to achieve the recommended extraction.

Blending the bread flour with All-Purpose flour (70–75% extraction) provides a simple solution, but we'll have to guess at the extraction rate since it is not usually reported on the label. [Azure Standard] has an [unenriched All-Purpose Flour][Azure All-Purpose Flour], which we'll assume is 72% extraction. Since the target extraction rate ranges between 75 and 85%, assuming 72% for the All-Purpose flour seems a reasonable middle-ground.

We can avoid any questions about whether enriching is healthy since the bread flour and the whole grain flours typical of [Tartine][Tartine3] recipes should compensate for any nutrients removed from the All-Purpose flour. The amount of All-Purpose flour required is a small fraction (~6%) of the total flour.

Taking an approach similar to the [Hamelman Challenge], we'll estimate extraction by adding the weight of All-Purpose flour to the total grain weight (denominator of the equation). Since we're not adding any bran or germ, the numerator only includes the weight of bread flour.

[Tartine3]: https://books.google.com/books/about/Tartine_Book_No_3.html?id=iS_UAAAAQBAJ "Tartine Book No. 3: Modern Ancient Classic Whole"

[Hamelman Challenge]: http://hamelmanchallenge.blogspot.com/2010/06/tech-note-high-extraction-flour.html "Tech. Note: High Extraction Flour"

[Azure Bread Flour]: https://www.azurestandard.com/shop/product/food/flour/white/wheat/unbleached-bread-flour-ultra-unifine-organic/20260?package=FL403 "Azure Market Organics Unbleached Bread Flour (Ultra Unifine), Organic"

[Azure Standard]: https://www.azurestandard.com "Azure Standard of Healthy and Abundant Living"

[Azure All-Purpose Flour]: https://www.azurestandard.com/shop/product/food/flour/unbleached-flour/all-purpose-flour-unbleached-organic/11444?package=FL080 "Azure Market Organics All Purpose Flour Unbleached, Organic"

```python
from IPython.display import display, Math, Latex
from sympy.plotting import plot
import sympy as sym

sym.init_printing() # LaTeX formatted output

%matplotlib inline
```

The math is a fairly straightforward mass-balance or weighted average:

Let:

$b$
:   Subscript for bread flour

$a$
:   Subscript for All-Purpose flour

$T$
:   Subscript for Target (desired) extraction flour

$m$
:   mass of flour

$E$
:   Extraction fraction (i.e., percentage/100)—if you're using a spreadsheet, just type `%` after the percentage.

$$
\begin{aligned}
    \frac{m_b}{E_b} + \frac{m_a}{E_a} &= \frac{m_T}{E_T} \\
          m_b + m_a &= m_T \\
\end{aligned}
$$
The first equation is the total mass of grain, and the second is the mass of flour.

Using a little bit of the Algebra, we were never going to use after high school (actually sympy is doing the work), we'll subtract the left side from the right to put the equations into a format sympy can solve:

```python
ET, Eb, Ea, mT = sym.symbols('E_T, E_b, E_a, m_T')
ma, mb = sym.symbols('m_a, m_b')

E1 =  mT / ET - mb / Eb - ma /Ea
E2 = mT - mb - ma
sols = sym.solve([E1, E2], [mb, ma])

sols[mb] = sols[mb].simplify()

display(Math(data=r'\text{The equations become:}'),
        Math(data=sym.latex(E1)),
        Math(data=sym.latex(E2)),
        Math(data=r'\text{The solution is:}'),
        Math(data=sym.latex(sols))
       )

mb = (sols[mb]).factor()
ma = (sols[ma]).factor()
```

$\displaystyle \text{The equations become:}$

$\displaystyle - \frac{m_{b}}{E_{b}} - \frac{m_{a}}{E_{a}} + \frac{m_{T}}{E_{T}}$

$\displaystyle m_{T} - m_{a} - m_{b}$

$\displaystyle \text{The solution is:}$

$\displaystyle \left \{ m_{a} : \frac{E_{a} m_{T} \left(E_{T} - E_{b}\right)}{E_{T} \left(E_{a} - E_{b}\right)}, \quad m_{b} : - \frac{E_{b} m_{T} \left(E_{T} - E_{a}\right)}{E_{T} \left(E_{a} - E_{b}\right)}\right \}$

```python
subs = dict([
             ('E_b', 0.90),
             ('E_a', 0.72),
             ('m_T', 500.0),
            ])
plt = plot(ma.subs(subs), mb.subs(subs), ('E_T', 0.72, 0.90), show=False, )
plt[0].line_color = 'firebrick'
plt[0].label = '$m_a$'
plt[1].label = '$m_b$'
plt.xlabel = 'Extraction Rate, $E_T$'
plt.xlim = (0.70, 0.9)
plt.ylabel = 'Flour Mass (g)'
plt.axis_center = (0.70, 0)
plt.legend = True
plt.show()
```

## Example 1: High-Extraction Flour Blend

For 500 g ($m_T$) of high-extraction flour:

```python
subs = dict([('E_T', 0.85),
             ('E_b', 0.90),
             ('E_a', 0.72),
             ('m_T', 500.0),
            ])

def display_blend(subs=subs, ma=ma, mb=mb):
    '''Display the bread and All-Purpose flour blend'''

    result = dict([('m_a', round(ma.subs(subs),0)),
                   ('m_b', round(mb.subs(subs),0))])
    result['m_T'] = result['m_a'] + result['m_b']

    display(Math(data=r'\text{For:}'), Math(data=sym.latex(subs)),
            Math(data=r'\text{the flour blend should be:}'),
            Math(data=sym.latex(result)))

    assert result['m_T'] == subs['m_T']

display_blend()
```

$\displaystyle \text{For:}$

$\displaystyle \left \{ E_T : 0.85, \quad E_a : 0.72, \quad E_b : 0.9, \quad m_T : 500.0\right \}$

$\displaystyle \text{the flour blend should be:}$

$\displaystyle \left \{ m_T : 500.0, \quad m_a : 118.0, \quad m_b : 382.0\right \}$

## Example 2: Medium-Strong Wheat Flour Blend

For 500 g ($m_T$) of medium-strong wheat flour:

```python
subs = dict([('E_T', 0.75),
             ('E_b', 0.90),
             ('E_a', 0.72),
             ('m_T', 500.0),
            ])

display_blend(subs=subs)
```

$\displaystyle \text{For:}$

$\displaystyle \left \{ E_T : 0.75, \quad E_a : 0.72, \quad E_b : 0.9, \quad m_T : 500.0\right \}$

$\displaystyle \text{the flour blend should be:}$

$\displaystyle \left \{ m_T : 500.0, \quad m_a : 400.0, \quad m_b : 100.0\right \}$

# Results

Time to bake some bread and see....


# Two Loop TDEM

```python
from geoscilabs.base import widgetify
import geoscilabs.em.InductionLoop as IND
from ipywidgets import interact, FloatSlider, FloatText
```

## App Parameter Descriptions

Below are the adjustable parameters for widgets within this notebook:

* $I_p$: Transmitter current amplitude [A]
* $a_{Tx}$: Transmitter loop radius [m]
* $a_{Rx}$: Receiver loop radius [m]
* $x_{Rx}$: Receiver x position [m]
* $z_{Rx}$: Receiver z position [m]
* $\theta$: Receiver normal vector relative to vertical [degrees]
* $R$: Resistance of receiver loop [$\Omega$]
* $L$: Inductance of receiver loop [H]
* $f$: Specific frequency [Hz]
* $t$: Specific time [s]

## Background Theory: Induced Currents due to a Step-Off Primary Signal

Consider the case in the image below, where a circular loop of wire ($Tx$) caries a time-varying current $I_p (t)$. According to the Biot-Savart law, this produces a time-varying primary magnetic field. The time-varying nature of the corresponding magnetic flux which passes through the receiver coil ($Rx$) generates an induced secondary current $I_s (t)$, which depends on the coil's resistance ($R$) and inductance ($L$). Here, we will provided final analytic results associated with the app below. A full derivation can be found at the bottom of the page.

For a step-off primary current of the form $I_p (t) = I_0 u(-t)$, the secondary current carried by ($Rx$) is given by:

\begin{equation}
I_s (t) = \frac{I_0 A \beta_n}{L} \, \textrm{e}^{-Rt/L} \, u(t)
\end{equation}

where $A$ is the area of $Rx$, $\beta$ contains the geometric information pertaining to the problem and $u(t)$ is the unit-step function.

```python
# RUN TRANSIENT WIDGET
widgetify(IND.fcn_TDEM_Widget,I=FloatSlider(min=0.01, max=100., value=1., step=1., continuous_update=False, description = "$I_0$"),\
                        a1=FloatSlider(min=1., max=20., value=10., step=1., continuous_update=False, description = "$a_{Tx}$"),\
                        a2=FloatSlider(min=1., max=20., value=5., step=1., continuous_update=False, description = "$a_{Rx}$"),\
                        xRx=FloatSlider(min=-15., max=15., value=0., step=1., continuous_update=False, description = "$x_{Rx}$"),\
                        zRx=FloatSlider(min=-15., max=15., value=-8., step=1., continuous_update=False, description = "$z_{Rx}$"),\
                        azm=FloatSlider(min=-90., max=90., value=0., step=10., continuous_update=False, description = "$\\theta$"),\
                        logR=FloatSlider(min=0, max=6, value=2, step=1., continuous_update=False, description = "$log_{10}(R)$"),\
                        logL=FloatSlider(min=-7, max=-2, value=-2, step=1., continuous_update=False, description = "$log_{10}(L)$"),\
                        logt=FloatSlider(min=-6, max=-2, value=-4, step=1., continuous_update=False, description = "$log_{10}(t)$"))

```

    MyApp(children=(FloatSlider(value=1.0, continuous_update=False, description='$I_0$', min=0.01, step=1.0), Floa…

## Supporting Derivation for the Transient Response

Consider a transmitter loop which carries a primary current of the form:
\begin{equation}
I_p(t) = I_0 \, u(-t)
\end{equation}
where $u(t)$ is the unit-step function. According to the Biot-Savart law, this results in a primary magnetic field:
\begin{equation}
\mathbf{B_p} (\mathbf{r},t) =  \boldsymbol{\beta} \, I_0 \, t(-t)  \;\;\;\; \textrm{where} \;\;\;\;\; \boldsymbol{\beta} = \frac{\mu_0}{4 \pi} \int_C \frac{d \mathbf{l} \times \mathbf{r'}}{|\mathbf{r'}|^2}
\end{equation}
where $\boldsymbol{\beta}$ contains the problem geometry. Assume the magnetic field is homogeneous through the receiver loop. The primary field generates an EMF within the receiver loop equal to:
\begin{equation}
EMF = - i\omega \Phi \;\;\;\;\; \textrm{where} \;\;\;\;\; \Phi =  A \beta_n I_0 \, u(-t)
\end{equation}
where $A$ is the area of the receiver loop and $\beta_n$ is the component of $\boldsymbol{\beta}$ along $\hat n$. The EMF induces a secondary current $I_s$ within the receiver loop. Instead of solving the problem directly in the time-domain, we will apply the inverse Fourier transform to frequency-domain solutions. The secondary current induced in the receiver loop can be obtained through a convolution:
\begin{equation}
I_s (t) = h(t) \otimes u(-t)
\end{equation}
where $h(t)$ is the inverse Fourier transform of the harmonic solution:
\begin{equation}
h(t) = \mathbb{F}^{-1} \big [ I_s (\omega) \big ] = - \frac{I_0 A \beta_n}{L} \frac{d}{dt} \bigg [ \textrm{e}^{-Rt/L} \, u(t) \bigg ]
\end{equation}
Thus the secondary current induced in the receiver loop is given by:
\begin{equation}
I_s (t) = \frac{I_0 A \beta_n}{L} \, \textrm{e}^{-Rt/L} \, u(t)
\end{equation}

```python

```


```python
import sympy as sp
from sympy import poly
import numpy as np
import math
import matplotlib.pyplot as plt
class TransferFunctionModel(object):
    #TransferFunction Model, num and den is coefficent
    def __init__(self,num,den,tau = 0):
        self.num = num
        self.den = den
        self.tau = tau
        pass

    def freqres(self, w, unwarp = False):

        b = self.num
        a = self.den
        tau = self.tau
        s = 1j * w
        h = np.polyval(b, s) * np.exp(-tau * s) / np.polyval(a, s)
        print(np.absolute(h))
        h = np.complex64(h)
        amp = 20 * np.log10(np.absolute(h))
        pha = None
        if unwarp:
            pha = np.unwrap(np.arctan2(h.imag, h.real)) * 180 / math.pi
        else:
            pha = np.arctan2(h.imag, h.real) * 180 / math.pi
        return amp, pha

    def plot(self, freq = None):
        if freq is None:
            freq = np.linspace(1.0,10,10)
        print(freq)
        amp,pha = self.freqres(freq,True)
        plt.semilogx(freq,amp,label="Amp")
        plt.semilogx(freq,pha,label="Pha")

class TransferFunctionParamModel(object):
    #A TransferFunction with unknown parameters

    def __init__(self,num,den,tau = 0):
        self.num = num
        self.den = den
        self.tau = tau
        self.s = sp.symbols('s')

    def transfer_function_by_dict(self, sym_dict):
        s = self.s
        num_formula =  self.num.subs(sym_dict)
        num = poly(num_formula,s).all_coeffs()

        den_formula =  self.den.subs(sym_dict)
        den = poly(den_formula,s).all_coeffs()

        return TransferFunctionModel(num,den,self.tau)

    def get_unknown_param_list(self):
        syms = self.num.atoms(sp.Symbol)
        syms.update(self.den.atoms(sp.Symbol))
        syms.remove(self.s)
        return syms
```

```python
s,a,b,c = sp.symbols('s a b c')
num = c*s+0
den = a*s**2 + b*s

tfparam = TransferFunctionParamModel(num,den)

tf = tfparam.transfer_function_by_dict({
    a : 2,
    b:  3,
    c : 5
})
tf.__dict__
tf.plot()
```

```python
tfparam.get_unknown_param_list()
```

    {a, b, c}

```python

```


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""
    # YOUR CODE HERE
    r = []
    for i in range(0, len(a)):
        if (i == len(a)-1 or a[i]>a[i+1]) and a[i]>a[i-1]:
            r.append(i)
    return np.array(r)
```

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
# YOUR CODE HERE
h =  pi_digits_str
p = []
for i in h:
    p.append(int(i))
e = np.array(p)
g = find_peaks(e)
y = np.diff(g)
plt.hist(y, bins = 90)
plt.ylabel('minanma spacing')
plt.xlabel('minima')
```

```python
assert True # use this for grading the pi digits histogram
```

```python

```

```python

```


### This is to study the min-sum approximation in channel decoding
\begin{equation}
2\tanh^{-1}{\lbrack\prod_{v'\in M(c)\backslash v}\tanh{(\frac{u^t_{v'\rightarrow c} }{2})}\rbrack}
\approx
\min_{v'\in M(c)\backslash v}|u^t_{v'\rightarrow c}|\prod_{v'\in M(c)\backslash v}\text{sign}{(u^t_{v'\rightarrow c} )}.
\end{equation}

```python
import numpy as np
from functools import partial
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def minsum(x):
    return np.min(np.abs(x)) * np.prod(np.sign(x))
```

```python
def itanh_prodtanh(x):
    return 2 * np.arctanh(np.prod(np.tanh(x/2)))
```

```python
np.random.seed(2020)
```

```python
for n in range(1, 10):
    x = np.random.random(n) * 2 - 1  # fall into range [-1,1]
    res1 = minsum(x)
    res2 = itanh_prodtanh(x)

    #print("prod(tanh(x/2))", np.prod(np.tanh(x/2)))

    if n == 2:
        A, B = x
        offset = np.log( (1+np.exp(-np.abs(A+B))) / (1+np.exp(-np.abs(A-B))) )
        print(n, "minsum={:.4f}, target={:.4f}, offset={:.4f}, minsum+offset={:.4f}".format(
            res1, res2, offset, res1+offset))
    else:
        print(n, "minsum={:.4f}, target={:.4f}".format(
            res1, res2))
```

    1 minsum=0.9726, target=0.9726
    2 minsum=0.0195, target=0.0070, offset=-0.0125, minsum+offset=0.0070
    3 minsum=-0.3262, target=-0.0200
    4 minsum=-0.3134, target=-0.0078
    5 minsum=-0.2887, target=-0.0009
    6 minsum=-0.1229, target=-0.0002
    7 minsum=-0.0067, target=-0.0000
    8 minsum=-0.0479, target=-0.0000
    9 minsum=-0.1153, target=-0.0000

```python
def mean_err(f, info, Ns, repeated=3000, scaled_norm=1):
    """
    Measure the mean approximation error.

    Args:
        f: approximation function takes given vector and returns output value
        info: the print message to include
        Ns: a list of input vector dimension
        repeated: number of simulation times
        scaled_norm: the sampling range [-1,1] * scaled_norm

    Returns:
        A dictionary containing the mean error, Ns, mean target values and approx values
    """

    output = {'mean_err':[], '|target|':[], 'f(x)':[], 'Ns': Ns}
    for n in Ns:
        err = 0
        target = 0
        fx = 0
        for _ in range(repeated):
            x = np.random.random(n) * 2 - 1  # fall into range [-1,1]
            x = x * scaled_norm
            res1 = f(x)
            res2 = itanh_prodtanh(x)
            err += np.abs(res1 - res2)
            target += np.abs(res2)
            fx += np.abs(res1)
        mean_err = err / repeated
        target = target / repeated
        fx = fx / repeated
        print("{}, n={}, mean err={:.5f}, |target|={:.5f}, f(x)={:.5f}".format(
              info, n, mean_err, target, fx))

        output['mean_err'].append(mean_err)
        output['|target|'].append(target)
        output['f(x)'].append(fx)
    return output
```

```python
ms = mean_err(minsum, "minsum", range(1,10)) #10000 samples from [-1,1]
```

    minsum, n=1, mean err=0.00000, |target|=0.50908, f(x)=0.50908
    minsum, n=2, mean err=0.21595, |target|=0.11429, f(x)=0.33024
    minsum, n=3, mean err=0.22411, |target|=0.02765, f(x)=0.25176
    minsum, n=4, mean err=0.19310, |target|=0.00671, f(x)=0.19981
    minsum, n=5, mean err=0.17197, |target|=0.00171, f(x)=0.17369
    minsum, n=6, mean err=0.14154, |target|=0.00039, f(x)=0.14193
    minsum, n=7, mean err=0.12546, |target|=0.00009, f(x)=0.12554
    minsum, n=8, mean err=0.11124, |target|=0.00002, f(x)=0.11127
    minsum, n=9, mean err=0.09765, |target|=0.00001, f(x)=0.09765

### This is to study the offset minsum approximation
\begin{equation}
2\tanh^{-1}{\lbrack\prod_{v'\in M(c)\backslash v}\tanh{(\frac{u^t_{v'\rightarrow c} }{2})}\rbrack}
\approx
\max(\min_{v'\in M(c)\backslash v}|u^t_{v'\rightarrow c}|-\beta, 0)
\prod_{v'\in M(c)\backslash v}\text{sign}{(u^t_{v'\rightarrow c} )}
\end{equation}

```python
def offsetminsum(x, offset):
    return max(0, np.min(np.abs(x)) - offset) * np.prod(np.sign(x))
```

```python
oms = {}
os = [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
for offset in os:
    f = partial(offsetminsum, offset=offset)
    oms[offset] = mean_err(f, "offsetminsum={:.2f}".format(offset), range(1,10), scaled_norm=1)
```

    offsetminsum=0.00, n=1, mean err=0.00000, |target|=0.50083, f(x)=0.50083
    offsetminsum=0.00, n=2, mean err=0.21821, |target|=0.11712, f(x)=0.33533
    offsetminsum=0.00, n=3, mean err=0.22166, |target|=0.02725, f(x)=0.24891
    offsetminsum=0.00, n=4, mean err=0.19592, |target|=0.00679, f(x)=0.20271
    offsetminsum=0.00, n=5, mean err=0.16406, |target|=0.00156, f(x)=0.16562
    offsetminsum=0.00, n=6, mean err=0.14421, |target|=0.00039, f(x)=0.14459
    offsetminsum=0.00, n=7, mean err=0.12434, |target|=0.00009, f(x)=0.12443
    offsetminsum=0.00, n=8, mean err=0.11017, |target|=0.00002, f(x)=0.11019
    offsetminsum=0.00, n=9, mean err=0.10020, |target|=0.00001, f(x)=0.10021
    offsetminsum=0.10, n=1, mean err=0.09504, |target|=0.50190, f(x)=0.40687
    offsetminsum=0.10, n=2, mean err=0.13275, |target|=0.11320, f(x)=0.23926
    offsetminsum=0.10, n=3, mean err=0.13354, |target|=0.02665, f(x)=0.15835
    offsetminsum=0.10, n=4, mean err=0.11136, |target|=0.00671, f(x)=0.11750
    offsetminsum=0.10, n=5, mean err=0.08787, |target|=0.00161, f(x)=0.08932
    offsetminsum=0.10, n=6, mean err=0.06549, |target|=0.00038, f(x)=0.06582
    offsetminsum=0.10, n=7, mean err=0.05362, |target|=0.00009, f(x)=0.05370
    offsetminsum=0.10, n=8, mean err=0.04159, |target|=0.00002, f(x)=0.04161
    offsetminsum=0.10, n=9, mean err=0.03678, |target|=0.00001, f(x)=0.03679
    offsetminsum=0.20, n=1, mean err=0.17915, |target|=0.49601, f(x)=0.31686
    offsetminsum=0.20, n=2, mean err=0.08055, |target|=0.11306, f(x)=0.16587
    offsetminsum=0.20, n=3, mean err=0.08248, |target|=0.02794, f(x)=0.10299
    offsetminsum=0.20, n=4, mean err=0.06110, |target|=0.00658, f(x)=0.06562
    offsetminsum=0.20, n=5, mean err=0.04231, |target|=0.00159, f(x)=0.04330
    offsetminsum=0.20, n=6, mean err=0.02923, |target|=0.00037, f(x)=0.02943
    offsetminsum=0.20, n=7, mean err=0.02054, |target|=0.00009, f(x)=0.02058
    offsetminsum=0.20, n=8, mean err=0.01507, |target|=0.00002, f(x)=0.01508
    offsetminsum=0.20, n=9, mean err=0.01064, |target|=0.00001, f(x)=0.01065
    offsetminsum=0.30, n=1, mean err=0.25358, |target|=0.49644, f(x)=0.24287
    offsetminsum=0.30, n=2, mean err=0.05926, |target|=0.11724, f(x)=0.11538
    offsetminsum=0.30, n=3, mean err=0.04923, |target|=0.02812, f(x)=0.06190
    offsetminsum=0.30, n=4, mean err=0.03104, |target|=0.00660, f(x)=0.03313
    offsetminsum=0.30, n=5, mean err=0.01804, |target|=0.00150, f(x)=0.01835
    offsetminsum=0.30, n=6, mean err=0.01186, |target|=0.00039, f(x)=0.01191
    offsetminsum=0.30, n=7, mean err=0.00766, |target|=0.00010, f(x)=0.00766
    offsetminsum=0.30, n=8, mean err=0.00352, |target|=0.00002, f(x)=0.00352
    offsetminsum=0.30, n=9, mean err=0.00273, |target|=0.00001, f(x)=0.00273
    offsetminsum=0.40, n=1, mean err=0.32244, |target|=0.51452, f(x)=0.19208
    offsetminsum=0.40, n=2, mean err=0.06107, |target|=0.11734, f(x)=0.07250
    offsetminsum=0.40, n=3, mean err=0.03058, |target|=0.02812, f(x)=0.03274
    offsetminsum=0.40, n=4, mean err=0.01611, |target|=0.00657, f(x)=0.01576
    offsetminsum=0.40, n=5, mean err=0.00740, |target|=0.00153, f(x)=0.00700
    offsetminsum=0.40, n=6, mean err=0.00454, |target|=0.00040, f(x)=0.00443
    offsetminsum=0.40, n=7, mean err=0.00180, |target|=0.00010, f(x)=0.00175
    offsetminsum=0.40, n=8, mean err=0.00104, |target|=0.00002, f(x)=0.00103
    offsetminsum=0.40, n=9, mean err=0.00049, |target|=0.00000, f(x)=0.00049
    offsetminsum=0.50, n=1, mean err=0.37315, |target|=0.49555, f(x)=0.12240
    offsetminsum=0.50, n=2, mean err=0.07458, |target|=0.11389, f(x)=0.04044
    offsetminsum=0.50, n=3, mean err=0.02426, |target|=0.02746, f(x)=0.01629
    offsetminsum=0.50, n=4, mean err=0.00951, |target|=0.00680, f(x)=0.00658
    offsetminsum=0.50, n=5, mean err=0.00328, |target|=0.00157, f(x)=0.00238
    offsetminsum=0.50, n=6, mean err=0.00123, |target|=0.00037, f(x)=0.00098
    offsetminsum=0.50, n=7, mean err=0.00077, |target|=0.00010, f(x)=0.00071
    offsetminsum=0.50, n=8, mean err=0.00022, |target|=0.00002, f(x)=0.00020
    offsetminsum=0.50, n=9, mean err=0.00007, |target|=0.00001, f(x)=0.00007
    offsetminsum=0.60, n=1, mean err=0.41940, |target|=0.49699, f(x)=0.07759
    offsetminsum=0.60, n=2, mean err=0.09376, |target|=0.11629, f(x)=0.02253
    offsetminsum=0.60, n=3, mean err=0.02374, |target|=0.02765, f(x)=0.00609
    offsetminsum=0.60, n=4, mean err=0.00707, |target|=0.00687, f(x)=0.00217
    offsetminsum=0.60, n=5, mean err=0.00193, |target|=0.00159, f(x)=0.00065
    offsetminsum=0.60, n=6, mean err=0.00066, |target|=0.00039, f(x)=0.00032
    offsetminsum=0.60, n=7, mean err=0.00009, |target|=0.00008, f(x)=0.00000
    offsetminsum=0.60, n=8, mean err=0.00005, |target|=0.00002, f(x)=0.00003
    offsetminsum=0.60, n=9, mean err=0.00002, |target|=0.00001, f(x)=0.00001
    offsetminsum=0.70, n=1, mean err=0.45122, |target|=0.49459, f(x)=0.04337
    offsetminsum=0.70, n=2, mean err=0.10314, |target|=0.11191, f(x)=0.00877
    offsetminsum=0.70, n=3, mean err=0.02593, |target|=0.02759, f(x)=0.00182
    offsetminsum=0.70, n=4, mean err=0.00627, |target|=0.00641, f(x)=0.00042
    offsetminsum=0.70, n=5, mean err=0.00163, |target|=0.00160, f(x)=0.00015
    offsetminsum=0.70, n=6, mean err=0.00037, |target|=0.00037, f(x)=0.00001
    offsetminsum=0.70, n=7, mean err=0.00011, |target|=0.00009, f(x)=0.00002
    offsetminsum=0.70, n=8, mean err=0.00002, |target|=0.00002, f(x)=0.00000
    offsetminsum=0.70, n=9, mean err=0.00000, |target|=0.00000, f(x)=0.00000
    offsetminsum=0.80, n=1, mean err=0.48467, |target|=0.50395, f(x)=0.01928
    offsetminsum=0.80, n=2, mean err=0.11388, |target|=0.11673, f(x)=0.00286
    offsetminsum=0.80, n=3, mean err=0.02770, |target|=0.02803, f(x)=0.00033
    offsetminsum=0.80, n=4, mean err=0.00651, |target|=0.00655, f(x)=0.00007
    offsetminsum=0.80, n=5, mean err=0.00163, |target|=0.00163, f(x)=0.00000
    offsetminsum=0.80, n=6, mean err=0.00038, |target|=0.00038, f(x)=0.00000
    offsetminsum=0.80, n=7, mean err=0.00009, |target|=0.00009, f(x)=0.00000
    offsetminsum=0.80, n=8, mean err=0.00002, |target|=0.00002, f(x)=0.00000
    offsetminsum=0.80, n=9, mean err=0.00001, |target|=0.00001, f(x)=0.00000
    offsetminsum=0.90, n=1, mean err=0.49407, |target|=0.49949, f(x)=0.00541
    offsetminsum=0.90, n=2, mean err=0.11550, |target|=0.11589, f(x)=0.00039
    offsetminsum=0.90, n=3, mean err=0.02826, |target|=0.02826, f(x)=0.00000
    offsetminsum=0.90, n=4, mean err=0.00658, |target|=0.00658, f(x)=0.00000
    offsetminsum=0.90, n=5, mean err=0.00154, |target|=0.00154, f(x)=0.00000
    offsetminsum=0.90, n=6, mean err=0.00039, |target|=0.00039, f(x)=0.00000
    offsetminsum=0.90, n=7, mean err=0.00009, |target|=0.00009, f(x)=0.00000
    offsetminsum=0.90, n=8, mean err=0.00002, |target|=0.00002, f(x)=0.00000
    offsetminsum=0.90, n=9, mean err=0.00001, |target|=0.00001, f(x)=0.00000

```python
def plotcurve(res, os, y='mean_err', title='10000 samples from [-1,1]'):
    fig, ax = plt.subplots()
    for o in os:
        ax.plot(res[o]['Ns'], res[o][y], label="{:.2f}".format(o))
    ax.legend()
    ax.set_ylabel(y)
    ax.set_xlabel('N')
    ax.set_title(title)
    plt.show()
```

```python
plotcurve(oms, os, title='Offset Minsum on 3000 samples from [-1,1]')
```

```python
oms2 = {}
os = np.arange(0.12, 0.32, 0.02)
for offset in os:
    f = partial(offsetminsum, offset=offset)
    oms2[offset] = mean_err(f, "offsetminsum={:.2f}".format(offset), range(1,len(os)), scaled_norm=1)
```

    offsetminsum=0.12, n=1, mean err=0.11220, |target|=0.49441, f(x)=0.38221
    offsetminsum=0.12, n=2, mean err=0.12500, |target|=0.11830, f(x)=0.23335
    offsetminsum=0.12, n=3, mean err=0.12434, |target|=0.02802, f(x)=0.14951
    offsetminsum=0.12, n=4, mean err=0.10268, |target|=0.00683, f(x)=0.10872
    offsetminsum=0.12, n=5, mean err=0.07728, |target|=0.00165, f(x)=0.07870
    offsetminsum=0.12, n=6, mean err=0.05797, |target|=0.00038, f(x)=0.05829
    offsetminsum=0.12, n=7, mean err=0.04317, |target|=0.00009, f(x)=0.04324
    offsetminsum=0.12, n=8, mean err=0.03483, |target|=0.00002, f(x)=0.03485
    offsetminsum=0.12, n=9, mean err=0.02915, |target|=0.00001, f(x)=0.02915
    offsetminsum=0.14, n=1, mean err=0.13007, |target|=0.49555, f(x)=0.36548
    offsetminsum=0.14, n=2, mean err=0.11625, |target|=0.12053, f(x)=0.22357
    offsetminsum=0.14, n=3, mean err=0.11035, |target|=0.02748, f(x)=0.13396
    offsetminsum=0.14, n=4, mean err=0.09061, |target|=0.00677, f(x)=0.09629
    offsetminsum=0.14, n=5, mean err=0.06740, |target|=0.00162, f(x)=0.06870
    offsetminsum=0.14, n=6, mean err=0.04922, |target|=0.00039, f(x)=0.04951
    offsetminsum=0.14, n=7, mean err=0.03661, |target|=0.00009, f(x)=0.03668
    offsetminsum=0.14, n=8, mean err=0.03039, |target|=0.00002, f(x)=0.03041
    offsetminsum=0.14, n=9, mean err=0.02155, |target|=0.00001, f(x)=0.02155
    offsetminsum=0.16, n=1, mean err=0.14749, |target|=0.49543, f(x)=0.34793
    offsetminsum=0.16, n=2, mean err=0.09812, |target|=0.11385, f(x)=0.19327
    offsetminsum=0.16, n=3, mean err=0.10342, |target|=0.02778, f(x)=0.12673
    offsetminsum=0.16, n=4, mean err=0.07572, |target|=0.00648, f(x)=0.08085
    offsetminsum=0.16, n=5, mean err=0.06030, |target|=0.00163, f(x)=0.06152
    offsetminsum=0.16, n=6, mean err=0.04323, |target|=0.00042, f(x)=0.04353
    offsetminsum=0.16, n=7, mean err=0.03063, |target|=0.00009, f(x)=0.03070
    offsetminsum=0.16, n=8, mean err=0.02292, |target|=0.00002, f(x)=0.02294
    offsetminsum=0.16, n=9, mean err=0.01676, |target|=0.00000, f(x)=0.01677
    offsetminsum=0.18, n=1, mean err=0.16310, |target|=0.49673, f(x)=0.33363
    offsetminsum=0.18, n=2, mean err=0.09029, |target|=0.11504, f(x)=0.18356
    offsetminsum=0.18, n=3, mean err=0.09448, |target|=0.02816, f(x)=0.11669
    offsetminsum=0.18, n=4, mean err=0.06440, |target|=0.00634, f(x)=0.06896
    offsetminsum=0.18, n=5, mean err=0.04949, |target|=0.00162, f(x)=0.05060
    offsetminsum=0.18, n=6, mean err=0.03745, |target|=0.00040, f(x)=0.03773
    offsetminsum=0.18, n=7, mean err=0.02366, |target|=0.00009, f(x)=0.02371
    offsetminsum=0.18, n=8, mean err=0.01827, |target|=0.00002, f(x)=0.01828
    offsetminsum=0.18, n=9, mean err=0.01487, |target|=0.00001, f(x)=0.01487
    offsetminsum=0.20, n=1, mean err=0.18095, |target|=0.50451, f(x)=0.32356
    offsetminsum=0.20, n=2, mean err=0.08475, |target|=0.11716, f(x)=0.17353
    offsetminsum=0.20, n=3, mean err=0.08115, |target|=0.02747, f(x)=0.10113
    offsetminsum=0.20, n=4, mean err=0.06461, |target|=0.00694, f(x)=0.06948
    offsetminsum=0.20, n=5, mean err=0.04328, |target|=0.00164, f(x)=0.04427
    offsetminsum=0.20, n=6, mean err=0.02842, |target|=0.00037, f(x)=0.02862
    offsetminsum=0.20, n=7, mean err=0.02088, |target|=0.00009, f(x)=0.02093
    offsetminsum=0.20, n=8, mean err=0.01220, |target|=0.00002, f(x)=0.01220
    offsetminsum=0.20, n=9, mean err=0.00978, |target|=0.00001, f(x)=0.00978
    offsetminsum=0.22, n=1, mean err=0.19455, |target|=0.49471, f(x)=0.30016
    offsetminsum=0.22, n=2, mean err=0.07651, |target|=0.11475, f(x)=0.15673
    offsetminsum=0.22, n=3, mean err=0.07436, |target|=0.02778, f(x)=0.09299
    offsetminsum=0.22, n=4, mean err=0.05602, |target|=0.00687, f(x)=0.06027
    offsetminsum=0.22, n=5, mean err=0.03475, |target|=0.00157, f(x)=0.03559
    offsetminsum=0.22, n=6, mean err=0.02322, |target|=0.00036, f(x)=0.02337
    offsetminsum=0.22, n=7, mean err=0.01602, |target|=0.00009, f(x)=0.01606
    offsetminsum=0.22, n=8, mean err=0.01173, |target|=0.00002, f(x)=0.01174
    offsetminsum=0.22, n=9, mean err=0.00851, |target|=0.00000, f(x)=0.00851
    offsetminsum=0.24, n=1, mean err=0.21069, |target|=0.50102, f(x)=0.29033
    offsetminsum=0.24, n=2, mean err=0.07089, |target|=0.11672, f(x)=0.14733
    offsetminsum=0.24, n=3, mean err=0.06651, |target|=0.02779, f(x)=0.08408
    offsetminsum=0.24, n=4, mean err=0.04964, |target|=0.00707, f(x)=0.05363
    offsetminsum=0.24, n=5, mean err=0.03204, |target|=0.00161, f(x)=0.03279
    offsetminsum=0.24, n=6, mean err=0.01897, |target|=0.00037, f(x)=0.01909
    offsetminsum=0.24, n=7, mean err=0.01490, |target|=0.00009, f(x)=0.01493
    offsetminsum=0.24, n=8, mean err=0.00902, |target|=0.00002, f(x)=0.00903
    offsetminsum=0.24, n=9, mean err=0.00616, |target|=0.00001, f(x)=0.00616
    offsetminsum=0.26, n=1, mean err=0.22479, |target|=0.49931, f(x)=0.27452
    offsetminsum=0.26, n=2, mean err=0.06662, |target|=0.11654, f(x)=0.13544
    offsetminsum=0.26, n=3, mean err=0.05752, |target|=0.02809, f(x)=0.07282
    offsetminsum=0.26, n=4, mean err=0.04128, |target|=0.00673, f(x)=0.04452
    offsetminsum=0.26, n=5, mean err=0.02609, |target|=0.00154, f(x)=0.02667
    offsetminsum=0.26, n=6, mean err=0.01797, |target|=0.00040, f(x)=0.01809
    offsetminsum=0.26, n=7, mean err=0.01087, |target|=0.00009, f(x)=0.01089
    offsetminsum=0.26, n=8, mean err=0.00797, |target|=0.00002, f(x)=0.00798
    offsetminsum=0.26, n=9, mean err=0.00539, |target|=0.00001, f(x)=0.00539
    offsetminsum=0.28, n=1, mean err=0.24137, |target|=0.50143, f(x)=0.26006
    offsetminsum=0.28, n=2, mean err=0.05948, |target|=0.11241, f(x)=0.11937
    offsetminsum=0.28, n=3, mean err=0.05331, |target|=0.02781, f(x)=0.06766
    offsetminsum=0.28, n=4, mean err=0.03591, |target|=0.00690, f(x)=0.03865
    offsetminsum=0.28, n=5, mean err=0.02220, |target|=0.00156, f(x)=0.02264
    offsetminsum=0.28, n=6, mean err=0.01447, |target|=0.00039, f(x)=0.01454
    offsetminsum=0.28, n=7, mean err=0.00892, |target|=0.00009, f(x)=0.00893
    offsetminsum=0.28, n=8, mean err=0.00544, |target|=0.00002, f(x)=0.00544
    offsetminsum=0.28, n=9, mean err=0.00436, |target|=0.00001, f(x)=0.00435
    offsetminsum=0.30, n=1, mean err=0.25463, |target|=0.50295, f(x)=0.24832
    offsetminsum=0.30, n=2, mean err=0.05883, |target|=0.11447, f(x)=0.11290
    offsetminsum=0.30, n=3, mean err=0.04790, |target|=0.02763, f(x)=0.06008
    offsetminsum=0.30, n=4, mean err=0.03455, |target|=0.00706, f(x)=0.03709
    offsetminsum=0.30, n=5, mean err=0.01949, |target|=0.00161, f(x)=0.01991
    offsetminsum=0.30, n=6, mean err=0.01260, |target|=0.00039, f(x)=0.01265
    offsetminsum=0.30, n=7, mean err=0.00757, |target|=0.00010, f(x)=0.00757
    offsetminsum=0.30, n=8, mean err=0.00436, |target|=0.00002, f(x)=0.00436
    offsetminsum=0.30, n=9, mean err=0.00261, |target|=0.00001, f(x)=0.00260

```python
plotcurve(oms2, os, title='Offset Minsum on 3000 samples from [-1,1]')
```

### This is to study the scaled minsum approximation
\begin{equation}
2\tanh^{-1}{\lbrack\prod_{v'\in M(c)\backslash v}\tanh{(\frac{u^t_{v'\rightarrow c} }{2})}\rbrack}
\approx
w\times
\min_{v'\in M(c)\backslash v}|u^t_{v'\rightarrow c}|\prod_{v'\in M(c)\backslash v}\text{sign}{(u^t_{v'\rightarrow c} )}
\end{equation}

```python
def scaledminsum(x, coef):
    return coef * np.min(np.abs(x)) * np.prod(np.sign(x))
```

```python
sms = {}
ss = np.arange(0, 1, 0.1)
for coef in ss:
    f = partial(scaledminsum, coef=coef)
    sms[coef] = mean_err(f, "scaledminsum={:.2f}".format(coef), range(1,len(ss)))
```

    scaledminsum=0.00, n=1, mean err=0.50617, |target|=0.50617, f(x)=0.00000
    scaledminsum=0.00, n=2, mean err=0.11756, |target|=0.11756, f(x)=0.00000
    scaledminsum=0.00, n=3, mean err=0.02757, |target|=0.02757, f(x)=0.00000
    scaledminsum=0.00, n=4, mean err=0.00646, |target|=0.00646, f(x)=0.00000
    scaledminsum=0.00, n=5, mean err=0.00161, |target|=0.00161, f(x)=0.00000
    scaledminsum=0.00, n=6, mean err=0.00039, |target|=0.00039, f(x)=0.00000
    scaledminsum=0.00, n=7, mean err=0.00009, |target|=0.00009, f(x)=0.00000
    scaledminsum=0.00, n=8, mean err=0.00002, |target|=0.00002, f(x)=0.00000
    scaledminsum=0.00, n=9, mean err=0.00001, |target|=0.00001, f(x)=0.00000
    scaledminsum=0.10, n=1, mean err=0.44458, |target|=0.49398, f(x)=0.04940
    scaledminsum=0.10, n=2, mean err=0.08487, |target|=0.11850, f(x)=0.03375
    scaledminsum=0.10, n=3, mean err=0.01007, |target|=0.02784, f(x)=0.02507
    scaledminsum=0.10, n=4, mean err=0.01344, |target|=0.00694, f(x)=0.02038
    scaledminsum=0.10, n=5, mean err=0.01519, |target|=0.00160, f(x)=0.01679
    scaledminsum=0.10, n=6, mean err=0.01377, |target|=0.00038, f(x)=0.01416
    scaledminsum=0.10, n=7, mean err=0.01223, |target|=0.00009, f(x)=0.01233
    scaledminsum=0.10, n=8, mean err=0.01095, |target|=0.00002, f(x)=0.01097
    scaledminsum=0.10, n=9, mean err=0.00982, |target|=0.00000, f(x)=0.00982
    scaledminsum=0.20, n=1, mean err=0.39693, |target|=0.49617, f(x)=0.09923
    scaledminsum=0.20, n=2, mean err=0.05042, |target|=0.11381, f(x)=0.06563
    scaledminsum=0.20, n=3, mean err=0.02234, |target|=0.02851, f(x)=0.05084
    scaledminsum=0.20, n=4, mean err=0.03347, |target|=0.00670, f(x)=0.04017
    scaledminsum=0.20, n=5, mean err=0.03165, |target|=0.00163, f(x)=0.03328
    scaledminsum=0.20, n=6, mean err=0.02850, |target|=0.00039, f(x)=0.02889
    scaledminsum=0.20, n=7, mean err=0.02460, |target|=0.00009, f(x)=0.02468
    scaledminsum=0.20, n=8, mean err=0.02247, |target|=0.00002, f(x)=0.02250
    scaledminsum=0.20, n=9, mean err=0.01975, |target|=0.00001, f(x)=0.01976
    scaledminsum=0.30, n=1, mean err=0.34719, |target|=0.49599, f(x)=0.14880
    scaledminsum=0.30, n=2, mean err=0.02816, |target|=0.11712, f(x)=0.10084
    scaledminsum=0.30, n=3, mean err=0.04688, |target|=0.02769, f(x)=0.07457
    scaledminsum=0.30, n=4, mean err=0.05293, |target|=0.00669, f(x)=0.05961
    scaledminsum=0.30, n=5, mean err=0.04882, |target|=0.00164, f(x)=0.05045
    scaledminsum=0.30, n=6, mean err=0.04242, |target|=0.00038, f(x)=0.04280
    scaledminsum=0.30, n=7, mean err=0.03744, |target|=0.00009, f(x)=0.03753
    scaledminsum=0.30, n=8, mean err=0.03387, |target|=0.00002, f(x)=0.03390
    scaledminsum=0.30, n=9, mean err=0.03013, |target|=0.00001, f(x)=0.03014
    scaledminsum=0.40, n=1, mean err=0.30153, |target|=0.50255, f(x)=0.20102
    scaledminsum=0.40, n=2, mean err=0.02284, |target|=0.11610, f(x)=0.13308
    scaledminsum=0.40, n=3, mean err=0.07422, |target|=0.02863, f(x)=0.10284
    scaledminsum=0.40, n=4, mean err=0.07325, |target|=0.00661, f(x)=0.07986
    scaledminsum=0.40, n=5, mean err=0.06476, |target|=0.00158, f(x)=0.06634
    scaledminsum=0.40, n=6, mean err=0.05721, |target|=0.00038, f(x)=0.05760
    scaledminsum=0.40, n=7, mean err=0.05036, |target|=0.00009, f(x)=0.05046
    scaledminsum=0.40, n=8, mean err=0.04448, |target|=0.00002, f(x)=0.04451
    scaledminsum=0.40, n=9, mean err=0.04008, |target|=0.00001, f(x)=0.04009
    scaledminsum=0.50, n=1, mean err=0.24950, |target|=0.49899, f(x)=0.24950
    scaledminsum=0.50, n=2, mean err=0.05134, |target|=0.11664, f(x)=0.16798
    scaledminsum=0.50, n=3, mean err=0.09654, |target|=0.02736, f(x)=0.12391
    scaledminsum=0.50, n=4, mean err=0.09246, |target|=0.00647, f(x)=0.09893
    scaledminsum=0.50, n=5, mean err=0.08101, |target|=0.00156, f(x)=0.08257
    scaledminsum=0.50, n=6, mean err=0.07114, |target|=0.00039, f(x)=0.07153
    scaledminsum=0.50, n=7, mean err=0.06150, |target|=0.00009, f(x)=0.06159
    scaledminsum=0.50, n=8, mean err=0.05662, |target|=0.00002, f(x)=0.05665
    scaledminsum=0.50, n=9, mean err=0.05013, |target|=0.00001, f(x)=0.05013
    scaledminsum=0.60, n=1, mean err=0.19592, |target|=0.48980, f(x)=0.29388
    scaledminsum=0.60, n=2, mean err=0.08428, |target|=0.11425, f(x)=0.19854
    scaledminsum=0.60, n=3, mean err=0.12016, |target|=0.02731, f(x)=0.14747
    scaledminsum=0.60, n=4, mean err=0.11401, |target|=0.00669, f(x)=0.12069
    scaledminsum=0.60, n=5, mean err=0.10024, |target|=0.00166, f(x)=0.10190
    scaledminsum=0.60, n=6, mean err=0.08501, |target|=0.00038, f(x)=0.08539
    scaledminsum=0.60, n=7, mean err=0.07509, |target|=0.00009, f(x)=0.07518
    scaledminsum=0.60, n=8, mean err=0.06680, |target|=0.00002, f(x)=0.06682
    scaledminsum=0.60, n=9, mean err=0.06072, |target|=0.00001, f(x)=0.06072
    scaledminsum=0.70, n=1, mean err=0.14691, |target|=0.48969, f(x)=0.34278
    scaledminsum=0.70, n=2, mean err=0.11827, |target|=0.11689, f(x)=0.23516
    scaledminsum=0.70, n=3, mean err=0.14707, |target|=0.02738, f(x)=0.17445
    scaledminsum=0.70, n=4, mean err=0.13648, |target|=0.00690, f(x)=0.14338
    scaledminsum=0.70, n=5, mean err=0.11378, |target|=0.00157, f(x)=0.11535
    scaledminsum=0.70, n=6, mean err=0.10038, |target|=0.00040, f(x)=0.10078
    scaledminsum=0.70, n=7, mean err=0.08667, |target|=0.00010, f(x)=0.08676
    scaledminsum=0.70, n=8, mean err=0.07904, |target|=0.00002, f(x)=0.07907
    scaledminsum=0.70, n=9, mean err=0.06993, |target|=0.00001, f(x)=0.06993
    scaledminsum=0.80, n=1, mean err=0.09816, |target|=0.49079, f(x)=0.39263
    scaledminsum=0.80, n=2, mean err=0.15310, |target|=0.11912, f(x)=0.27221
    scaledminsum=0.80, n=3, mean err=0.17391, |target|=0.02807, f(x)=0.20199
    scaledminsum=0.80, n=4, mean err=0.15372, |target|=0.00686, f(x)=0.16059
    scaledminsum=0.80, n=5, mean err=0.12994, |target|=0.00160, f(x)=0.13155
    scaledminsum=0.80, n=6, mean err=0.11540, |target|=0.00039, f(x)=0.11579
    scaledminsum=0.80, n=7, mean err=0.10262, |target|=0.00010, f(x)=0.10272
    scaledminsum=0.80, n=8, mean err=0.08705, |target|=0.00002, f(x)=0.08707
    scaledminsum=0.80, n=9, mean err=0.07977, |target|=0.00001, f(x)=0.07977
    scaledminsum=0.90, n=1, mean err=0.04998, |target|=0.49977, f(x)=0.44979
    scaledminsum=0.90, n=2, mean err=0.18520, |target|=0.11594, f(x)=0.30114
    scaledminsum=0.90, n=3, mean err=0.19549, |target|=0.02720, f(x)=0.22269
    scaledminsum=0.90, n=4, mean err=0.17015, |target|=0.00634, f(x)=0.17650
    scaledminsum=0.90, n=5, mean err=0.14992, |target|=0.00164, f(x)=0.15156
    scaledminsum=0.90, n=6, mean err=0.12672, |target|=0.00038, f(x)=0.12710
    scaledminsum=0.90, n=7, mean err=0.11001, |target|=0.00009, f(x)=0.11010
    scaledminsum=0.90, n=8, mean err=0.10157, |target|=0.00002, f(x)=0.10160
    scaledminsum=0.90, n=9, mean err=0.08857, |target|=0.00001, f(x)=0.08858

```python
plotcurve(sms, ss, title='Scaled Minsum on 3000 samples from [-1,1]')
```

```python
fig, ax = plt.subplots()

ax.plot(oms[0.4]['Ns'], oms[0.4]['mean_err'], label="offset={:.2f}".format(0.4))
ax.plot(oms[0.3]['Ns'], oms[0.3]['mean_err'], label="offset={:.2f}".format(0.3))
ax.plot(oms[0.2]['Ns'], oms[0.2]['mean_err'], label="offset={:.2f}".format(0.2))

ax.plot(sms[0.4]['Ns'], sms[0.4]['mean_err'], label="scaled={:.2f}".format(0.4))
ax.plot(sms[ss[3]]['Ns'], sms[ss[3]]['mean_err'], label="scaled={:.2f}".format(ss[3]))
ax.plot(sms[0.2]['Ns'], sms[0.2]['mean_err'], label="scaled={:.2f}".format(0.2))

ax.legend()
ax.set_ylabel('mean_err')
ax.set_xlabel('N')
plt.show()
```

```python

```