# PC lab 6: Linear model selection - Regularization

We have already seen linear regression to tackle regression problems. With linear regression, we model a continous outcome as a linear function of the features:

\begin{equation}
\hat{y} = w_{0}x_{0} + w_{1}x_{1} + ... + w_{p}x_{p} = \sum\limits_{i=0}^{p}w_{i}x_{i}
\end{equation}

This works well when there are a lot of training observations and when the number of features (the dimensionality of the problem) is not too large. However, there are a couple of situations where ordinary linear regression might give problems:

* When the number of features $p$ becomes large with respect to the number of observations $n$, the variance of the model weights estimated by linear regression increases, which might result in poor predictive performance. Futhermore, there is no longer an analytical solution provided by least squares when $p > n$.
* It is possible that there are a lot of uninteresting or redundant features. If we want a sparse and interpretable model, we might want to do feature selection to reduce $p$.

In this lab, we will cover two solutions to the problems above: subset selection and regularization methods.

## Subset selection methods

In subset selection, we only use a subset of the features that are available. The goal is to come up with a model that is sparse and that generalizes better to unseen data. There are two main strategies for subset selection: in *best subset selection*, we fit all the $p \choose k$ models for each $k = 1, 2.. p$ and retain the best model for each $k$. Finally, we select the model that performs best on some measure that controls for overfitting:

This becomes quickly unfeasible for large values of $p$. Therefore, an alternative approach is to perform *stepwise selection*, which explores a much smaller set of feature combinations. Stepwise selection can be performed either backward or forward. For large $p$ they are the only computationally feasible subset selection methods.

Finally, it is important to account for the fact that the MSE (or, equivalently, the RSS) will always go down on the training data as we add more and more features. To select the best model out of several candidates, it is important to have an estimate of the test error of each model. This can be done indirectly by using a metric that penalizes for model complexity such as the AIC or the adjusted $R^2$. Another option is to use cross-validation to get an estimate of the test error.

**Let's apply subset selection on two datasets.** The first dataset contains features of mixtures used to produce concrete. The goal is to predict the compressive strength of the concrete.

```python
# Read in the data
import pandas as pd

concretedata = pd.read_table('concreteComprStrength.txt', delim_whitespace=True, header=0, index_col=None)
concretedata.head()
```

Run the code below to perform best subset selection with linear regression. The code implements algorithm 6.1 from the book as shown above. For each number of features $k$, the code will print the number of feature combinations that is explored. We fit a model for each combination, so there is quite some computation involved.

```python
import scipy as scipy
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_validate
from itertools import combinations
from matplotlib import pyplot as plt
%matplotlib inline

# Specify features and labels
X = concretedata.drop(['compressiveStrength'], axis=1)
y = concretedata['compressiveStrength']

# Dicts to store the feature combinations and their scores
scores_perNfeat = {'mean':[], 'max':[], 'var':[]}
best_feature_perNfeat = []
# Define splitter object outside loop to ensure same splits over loop
splitter = KFold(n_splits=5, shuffle=True, random_state=None)

# For k in 1,...,p
for k in np.arange(1,X.shape[1]+1):
    print('Number of combinations  with {} features: {}'.format(k, scipy.special.comb(X.shape[1],k)))
    best_score_k = 0
    scores_k_comb = []
    # Create and loop over all the possible feature combinations given k
    for c in combinations(X.columns, k):
        # Select features from data
        X_c = X[list(c)]
        # Define model
        model = LinearRegression()
        # Fit model and compute score on training data
        scores = cross_validate(model, X_c.values, y.values, cv=splitter)
        mean_cv_score = np.mean(scores['test_score'])
        # Save scores
        scores_k_comb.append(mean_cv_score)
        if mean_cv_score > best_score_k:
            best_features = c
            best_score_k = mean_cv_score
    # Save scores based on # features

    scores_perNfeat['max'].append(np.max(scores_k_comb))
    scores_perNfeat['var'].append(np.var(scores_k_comb))
    scores_perNfeat['mean'].append(np.mean(scores_k_comb))
    # Save best feature combination
    best_feature_perNfeat.append(best_features)

#Get variance of scores within grouped scores
for i, score in enumerate(scores_perNfeat['max']):
    print("Best CV score using {} features: {}".format(i+1, score))

fig, ax = plt.subplots(1,1,figsize=(10,8), sharey=True)

ax.errorbar(np.arange(len(scores_perNfeat['mean']))+1, scores_perNfeat['mean'], scores_perNfeat['var'], label='mean score')
ax.scatter(np.arange(len(scores_perNfeat['max']))+1, scores_perNfeat['max'], alpha=0.5, label='max score')
ax.set_xlabel('#features')
ax.set_ylabel('R^2')
fig.legend()
print('Best feature combination: {}'.format(best_feature_perNfeat[np.argmax(scores_perNfeat['max'])]))
```

We will do the same thing for a second dataset. This time, the features are measurements from a flow cytometry experiment. The 'SC' features measure scatter, and say something about the morphologhy of the cells (FSC: forwad scatter, SSC: sideway scatter). The 'FL' features are fluorescence features from different parts of the spectrum. There are two possible bacterial species present in the dataset. The goal is to classify the correct species based on the measurements from the flow cytometer. Species number one corresponds to *Pseudomonas putida*, while species number 6 is *Brachybacterium faecium*. We will use logistic regression to do the classification.

```python
# Read in the data
bacterialdata = pd.read_csv('fc_data_new.csv', index_col=0)
bacterialdata = bacterialdata.drop(['Width', 'Time'], axis=1).reset_index(drop=True)
```

```python
bacterialdata.head()
```

```python
from sklearn.linear_model import LogisticRegression
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# Specify features and labels
X = bacterialdata.drop(['species'], axis=1)
y = bacterialdata['species']

# Dicts to store the feature combinations and their scores
scores_perNfeat = {'mean':[], 'max':[], 'var':[]}
best_feature_perNfeat = []
# Define splitter object outside loop to ensure same splits over loop
splitter = KFold(n_splits=5, shuffle=True, random_state=None)

# For k in 1,...,p
for k in np.arange(1,X.shape[1]+1):
    print('Number of combinations  with {} features: {}'.format(k, scipy.special.comb(X.shape[1],k)))
    best_score_k = 0
    scores_k_comb = []
    # Create and loop over all the possible feature combinations given k
    for c in combinations(X.columns, k):
        # Select features from data
        X_c = X[list(c)]
        # Define model
        model = LogisticRegression() # or model = LDA()
        # Fit model and compute score on training data
        scores = cross_validate(model, X_c.values, y.values, cv=splitter, n_jobs=4)
        mean_cv_score = np.mean(scores['test_score'])
        # Save scores
        scores_k_comb.append(mean_cv_score)
        if mean_cv_score > best_score_k:
            best_features = c
            best_score_k = mean_cv_score
    # Save scores based on # features

    scores_perNfeat['max'].append(np.max(scores_k_comb))
    scores_perNfeat['var'].append(np.var(scores_k_comb))
    scores_perNfeat['mean'].append(np.mean(scores_k_comb))
    # Save best feature combination
    best_feature_perNfeat.append(best_features)

#Get variance of scores within grouped scores
for i, score in enumerate(scores_perNfeat['max']):
    print("Best CV score using {} features: {}".format(i+1, score))

fig, ax = plt.subplots(1,1,figsize=(10,8), sharey=True)

ax.errorbar(np.arange(len(scores_perNfeat['mean']))+1, scores_perNfeat['mean'], scores_perNfeat['var'], label='mean score')
ax.scatter(np.arange(len(scores_perNfeat['max']))+1, scores_perNfeat['max'], alpha=0.5, label='max score')
ax.set_xlabel('#features')
ax.set_ylabel('accuracy')
fig.legend()
print('Best feature combination: {}'.format(best_feature_perNfeat[np.argmax(scores_perNfeat['max'])]))
```

Exercise

Implement stepwise selection for features as explained at the start of the notebook

```python
from sklearn.linear_model import LogisticRegression

# Specify features and labels
X = bacterialdata.drop(['species'], axis=1)
y = bacterialdata['species']

# Lists to store the feature combinations and their scores
scores_perNfeat = {'mean':[], 'max':[], 'var':[]}
best_feature_perNfeat = []

features = [] # start with empty set of features

# Define splitter object outside loop to ensure same splits over loop
splitter = KFold(n_splits=3, shuffle=True, random_state=None)

# For k in 1,...,p
for k in np.arange(1,X.shape[1]+1):
    best_score_k = 0
    scores_k_comb = []
    # Create and loop over all the possible features to add
    for feature_idx in np.arange(0,X.shape[1]):
        ... # make sure that you don't features are not added that are already present in 'features'
        temp_features = features + [feature_idx]
        # Select features from data
        X_c = X.iloc[:, temp_features]
        # Define model
        model = LogisticRegression()
        # Fit model and compute score on training data
        scores = cross_validate(model, X_c.values, y.values, cv=splitter, n_jobs=4)
        mean_cv_score = np.mean(scores['test_score'])
        # Save scores
        scores_k_comb.append(mean_cv_score)
        if mean_cv_score > best_score_k:
            best_feature = ...
            best_score_k = ...
    features.append(best_feature)
    # Save scores based on # features
    scores_perNfeat['max'].append(np.max(scores_k_comb))
    scores_perNfeat['var'].append(np.var(scores_k_comb))
    scores_perNfeat['mean'].append(np.mean(scores_k_comb))
    # Save best feature combination
    best_feature_perNfeat.append(features[:])

#Get variance of scores within grouped scores
for i, score in enumerate(scores_perNfeat['max']):
    print("Best training score using {} features: {}".format(i+1, score))

fig, ax = plt.subplots(1,1,figsize=(10,8), sharey=True)
ax.errorbar(np.arange(len(scores_perNfeat['mean']))+1, scores_perNfeat['mean'], scores_perNfeat['var'], label='tune_data')
ax.scatter(np.arange(len(scores_perNfeat['max']))+1, scores_perNfeat['max'], alpha=0.5)
ax.set_xlabel('#features')
ax.set_ylabel('accuracy')
fig.legend()
print('final set of features: {}'.format(set(best_feature_perNfeat[np.argmax(scores_perNfeat['max'])])))
best_feature_perNfeat
```

## Regularization methods: ridge regression and the lasso

Regularization is one of the most important concepts in machine learning to avoid overfitting. It comes in many forms. In linear regression, regularization techniques typically constrain the coefficient estimates. In return for a little extra bias, this reduces the variance of the coefficient esimates. The two main shrinkage techniques are **ridge regression** and the **lasso**.

**Ridge regression penalizes the sum of the squares of the model weights by adding a term to the MSE loss function**:

The lasso does a similar thing, but penalizes the absolute value of the model coefficients. The effect of both approaches is that the model coefficients are shrunk towards zero, resulting in less overfitting and less variance in the predictions (at the cost of a little more bias). We will apply both models on two datasets.

### Linear models for high dimensional data

We will apply ridge regression on the Communities and Crime Data Set. The dataset contains 123 population statistics on 1994 communities. We would like to predict the number of violent crimes per 100000 inhabitants. This is the final column of the dataframe. Of the 123 features, a lot contain missing values, so we will drop these columns and use only 99 features.

```python
import pandas as pd
import numpy as np

data = pd.read_csv('./communities.csv')

# Drop columns with missing values
data = data.dropna(axis=1)
data.head()
```

Exercise

Use linear regression and ridge regression to predict the number of violent crimes per 100,000 inhabitants. Use 5-fold cross-validation to evaluate both models. The scikit-learn implementation of RidgeCV automatically performs cross-validation to tune the hyperparameter that determines the amount of regularization, so you don't need to implement a second cross-validation loop. Which model performs best?

**Before we apply ridge regression, it's important to make sure that all the features are on the same scale. If one of the features is on a completely different scale (let's say, income can be measured in dollars or in thousands of dollars), this might lead the ridge regression coefficient to change substantially because of the penalty term in the optimization problem. We can make sure that all the features are on the same scale by dividing them by their respective standard deviation: (see book p. 232)**

\begin{equation}
\tilde{x}_{ij} = \frac{x_{ij}}{\sqrt{\frac{1}{n}\sum_{i=1}^{n}(x_{ij} - \bar{x}_{j})^2}}
\end{equation}

We can do this with the ```StandardScaler``` from scikit-learn. We will do the scaling each time in the cross-validation loop using only training data statistics.

```python
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
```

```python
# Select X and y
y = data['ViolentCrimesPerPop'].values
X = data.drop(['ViolentCrimesPerPop'], axis=1).values

kf = KFold(n_splits=5)

linreg_scores = []
ridge_scores = []

for train_index, test_index in kf.split(X):

    ... # extract train and test splits from train_index & test_index

    ... # call and fit scaler

    linregmodel = LinearRegression()
    ridgemodel = RidgeCV(alphas=np.logspace(-3,3,100))

    ... # fit linear regression model and append scores to linreg_scores

    ... # fit ridgemodel and append scores to ridge_scores

    print('Regularization parameter: {}'.format(ridgemodel.alpha_))

print('Average validation fold R² of linear regression: {}'.format(np.mean(linreg_scores)))
print('Average validation fold R² of ridge regression: {}'.format(np.mean(ridge_scores)))

# ** solution
```

Now suppose that we don't have 99 features, but four times as many features. And suppose that a lot of features are correlated. This situation is very common in lots of datasets. We will mimic this situation by adding correlated features to our original feature matrix:

```python
X_1 = X + np.random.normal(loc=0, scale=0.05, size=(X.shape))
X_2 = X + np.random.normal(loc=0, scale=0.1, size=(X.shape))
X_3 = X + np.random.normal(loc=0, scale=0.01, size=(X.shape))
X_expanded = np.concatenate((X, X_1, X_2, X_3), axis=1)

X_expanded.shape
```

Exercise

Repeat the comparison between ridge regression and linear regression from above, but with the new feature matrix that contains correlated features. What happens with the performance of linear regression? What happens with the regularization parameter? Now, add even more correlated features and repeat the analysis.

```python
kf = KFold(n_splits=5)

linreg_scores = []
ridge_scores = []

for train_index, test_index in kf.split(X_expanded):
    ... # Repeat from previous but with X_expanded
    # consider using a larger range of alphas for RidgeCV, think for yourself why?

print('Average validation fold R² of linear regression: {}'.format(np.mean(linreg_scores)))
print('Average validation fold R² of ridge regression: {}'.format(np.mean(ridge_scores)))
# ** solution
```

Finally, it's interesting to look at how the amount of regularization in ridge regression and lasso regression affects the magnitudes of the fitted weights.

```python
from sklearn.linear_model import Ridge, Lasso
from matplotlib import pyplot as plt
%matplotlib inline

y = data['ViolentCrimesPerPop'].values
X = data.drop(['ViolentCrimesPerPop'], axis=1).values

# Scale X
#scaler = StandardScaler(with_mean=False)
#x = scaler.fit_transform(X)
x = X
alphas = np.logspace(-5,2,100)

ridge_coefficients = np.ndarray(shape=(50, len(alphas)))
lasso_coefficients = np.ndarray(shape=(50, len(alphas)))

for i,a in enumerate(alphas):
    ridgemodel = Ridge(alpha=a)
    lassomodel = Lasso(alpha=a, max_iter=10000)

    ridgemodel.fit(X,y)
    lassomodel.fit(X,y)

    ridge_coefficients[:, i] = ridgemodel.coef_[:50]
    lasso_coefficients[:, i] = lassomodel.coef_[:50]
```

```python
fig, ((ax1, ax2)) = plt.subplots(figsize=(15,10), nrows=2, sharex=True)

for c in range(ridge_coefficients.shape[0]):
    pd.Series(ridge_coefficients[c,:]).plot(ax=ax1)
    pd.Series(lasso_coefficients[c,:]).plot(ax=ax2)

ax2.set_xlabel('Regularization parameter').set_fontsize(20)
ax1.set_ylabel('Model weights')
ax2.set_ylabel('Model weights')
ax1.set_title('Model weights for increasing regularization - Ridge regression').set_fontsize(20)
ax2.set_title('Model weights for increasing regularization - Lasso').set_fontsize(20)
ax2.get_xaxis().set_ticks(alphas);
```

Exercise

Make sure you understand the plots above. What is the main difference between ridge regression and the lasso?

### Feature selection with the lasso

Suppose that we are interested in selecting only a couple of features out of a high dimensional dataset. Let's fit ridge regression and lasso regression on the data and look at the model coefficients for both models:

```python
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.linear_model import LassoCV
from sklearn.model_selection import train_test_split

ridgemodel = RidgeCV(cv=5)
lassomodel = LassoCV(cv=5, max_iter=10000)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Scale X
scaler = StandardScaler(with_mean=False)
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

ridgemodel.fit(X_train, y_train)
ridgescore = np.round(ridgemodel.score(X_test, y_test),2)
lassomodel.fit(X_train, y_train)
lassoscore = np.round(lassomodel.score(X_test, y_test),2)

# Plot of the coefficients for ridge regression
fig, ((ax1, ax2)) = plt.subplots(figsize=(40,30), nrows=2)
pd.Series(ridgemodel.coef_).plot(kind='bar', ax=ax1)
ax1.set_title('Ridge regression coefficients. Test set R²: {}'.format(ridgescore)).set_fontsize(40)
ax1.get_xaxis().set_ticks([])
ax1.set_xlabel('Features')

# Plot of the coefficients for the lasso
pd.Series(lassomodel.coef_).plot(kind='bar', ax=ax2)
ax2.set_title('Lasso regression coefficients. Test set R²: {}'.format(lassoscore)).set_fontsize(40)
ax2.get_xaxis().set_ticks([])
ax2.set_xlabel('Features');
```

The lasso applies regularization by constraining the sum of the absolute values of the model coefficients (the L1-norm). A result of this is that a lot of model coefficients are set to zero: the lasso performs **feature selection**. This is not the case for ridge regression: the model weights are rarely set to zero. Feature selection is a nice property if we want an interpretable model. Let's list the features with non-zero coefficients in the lasso:

```python
import seaborn as sns
sns.set_style('whitegrid')

fig, ax = plt.subplots(figsize=(12,15))
sns.barplot(x= pd.Series(lassomodel.coef_[lassomodel.coef_ != 0]),
            y=data.drop(['ViolentCrimesPerPop'], axis=1).columns[lassomodel.coef_ != 0],
            palette=['steelblue' if n > 0 else 'coral' for n in lassomodel.coef_[lassomodel.coef_ != 0]],
            ax=ax);
ax.set_title('Lasso model coefficients for the Community Crime dataset').set_fontsize(20);
```

### Regularization methods for $n < p$ data

In high dimensional data, we often have more features than observations. This is often called the $n < p$ scenario. In this situation, linear regression breaks down: the variance on the weight estimates blows up and the model will fail on unseen data. Both ridge regression and the lasso are valuable solutions here.

We will work with a dataset that contains spectral measurements on food samples. The target variables are the water, fat and protein content of the samples.

```python
data.head()
```

```python
data = pd.read_csv('./meatNIR1000.csv',header=None)
colnames = pd.read_csv('./meatNIR1000.colnames', header=None)
data.columns = colnames.values[0]

data.head()
```

```python
# Read in the data
data = pd.read_csv('./meatNIR1000.csv',header=None)
colnames = pd.read_csv('./meatNIR1000.colnames', header=None)
data.columns = colnames.values[0]
W = data['water']
F = data['fat']
P = data['protein']

data = data.drop(['water', 'fat', 'protein'], axis=1)
data.head()
```

We have 240 observations, but 500 features, so we are in a $n < p$ setting:

```python
data.shape
```

```python
# let's plot some random data points

fig, ax = plt.subplots(figsize=(20,10))
for i in range(20):
    ax.plot(data.iloc[np.random.choice(range(len(data)))])
ax.set_xlabel('Wavelength').set_fontsize(20)
ax.set_ylabel('Absorbance').set_fontsize(20)
```

Exercise

Try to fit a linear regression model to predict the fat content of a sample on the entire dataset. Look at the model coefficients. What happens?

```python
... # import, fit

... # look at the first twentycoefficients of the model
```

Exercise

Compare the performance of linear regression with ridge regression or with the lasso. Use 5-fold cross-validation to evaluate both models.

```python
import warnings
warnings.filterwarnings('ignore')

#** solution
from sklearn.linear_model import LassoCV, RidgeCV
# Select X and y
y = F
X = data.values

kf = KFold(n_splits=5)

linreg_scores = []
ridge_scores = []
lasso_scores = []

for train_index, test_index in kf.split(X):
    ... # Define X_train, X_test, y_train, y_test

    ... # scaler fit & transform

    ... # Fit linear, ridge and lasso

    ... # Append test scores to the lists

print('Average validation fold R² of linear regression: {}'.format(np.mean(linreg_scores)))
print('Average validation fold R² of ridge regression: {}'.format(np.mean(ridge_scores)))
print('Average validation fold R² of lasso regression: {}'.format(np.mean(lasso_scores)))
#**solution
```


# Leaky Integration
As already described in the Introduction to Neuron Models, the current state $x$ of a neuron $i$ not only depends on the inputs $U$ from the pre-synaptic layer $j$ but also on previous activations $\theta$. This is because a neuron does not respond instantaneously to new inputs. This can be addressed by modelling a neuron as leaky integrator.

Recalling our neuron model form the introduction, we use the following equation to model one neuron:
\begin{align}
x_i &= W_{i,j} \cdot U_j
\end{align}

As we now introduce a previous activation, we need to introduce time. To model discrete neuron states we can formulate the next state of our neuron at $t+1$ as:

\begin{align}
x_i(t+1) &= W_{i,j} \cdot U_j(t)
\end{align}

To include the previous activation $\theta_i(t)$ we need to somehow define how much the next state $x_i(t+1)$ depends on the current inputs $U_j(t)$ and how much it depends on the previous activation $\theta_i(t)$. We do so by introducing a leak-rate $\lambda_i$ for our neuron. This leak-rate defines ho much of the current activation fades till the next state. If the leak-rate is $0$, the next state solely depends on the current activation, if the leak-rate is $1$ the next state solely depends on the current inputs.

    Figure 1: Symbolic representation of leaky integration.

So we can re-formulate our neuron state equation as:

\begin{align}
x_i(t+1) &= \lambda_i \cdot \left(W_{i,j} \cdot U_j(t)\right) + (1-\lambda_i) \cdot \theta_i(t)
\end{align}

Let's see how this affects the dynamic of one neuron by modelling it in python. We use the class for a single neuron, previously defined in the introduction and extend it by the leaking rate in line 9 and 14 (we cannot use lambda here as this is reserved for lambda expressions in python):

```python
import numpy as np

class single_neuron_leaky:

    def __init__(self, n_inputs: int, leak_rate: float,  seed: int = 42):
        np.random.seed(seed)
        self.x = 0
        self.theta = 0
        self.leak = leak_rate
        self.u = np.zeros([n_inputs, 1])
        self.w = np.random.uniform(low=-1, high=1, size=[1, n_inputs])

    def activate(self, input_vector):
        self.x = self.leak*np.dot(self.w, input_vector) + (1-self.leak)*self.theta
        self.theta = np.tanh(self.x)

    def step(self, input_vector):
        self.activate(input_vector)
        return self.theta
```

We now use only one input with zeros and a step of magnitude $1$ for time-steps $5$ to $10$ (lines 4-5). We also set the weight for this input to $1$ (line 14) and calculate the output for $30$ steps (line 1) and leak-rates from $1$ to $0$ in steps of $0.2$ (line 9).

```python
length = 30

# set th einput at t=5 to 1
u = np.zeros(length)
u[5:10] = 1

# testing leak rates from 1 to 0 in
# steps of 0.2
leak_rates = [1, 0.8, 0.6, 0.4, 0.2, 0]
o = np.zeros([len(leak_rates), length])

for n, lr in enumerate(leak_rates):
    neuron = single_neuron_leaky(1, lr)
    neuron.w = np.ones(1)
    for i in range(length):
        o[n, i] = neuron.step(u[i])
```

```python
from matplotlib import pyplot as plt
from matplotlib import rc

# Set plot parameters
rc('font', **{'family': 'sans-serif', 'sans-serif': ['Latin Modern Sans']})
params = {'font.size': 16,
          'figure.figsize': [20, 10],
          'text.usetex': True,
          'legend.fontsize': 'x-large',
          'axes.labelsize': 'x-large',
          'axes.titlesize':'x-large',
          'xtick.labelsize':'x-large',
          'ytick.labelsize':'x-large'}
plt.rcParams.update(params)

# plot the stuff
colors = ['#0072b2', '#e69f00', '#f0e442', '#009e73', '#56b4e9', '#d55e00', '#cc79a7', '#000000']
fig, ax = plt.subplots()
for n, lr in enumerate(leak_rates):
    ax.plot(o[n,:], color=colors[n], linewidth=3, label=r"$\theta(t), \lambda = %.1f $" % lr)
ax.plot(u, color='black', linewidth=3, linestyle='dashed', label=r"$u(t)$")

ax.legend()
ax.set_xlabel(r"$t$")
```

Here we can recognise how the neuron responds directly to the input for a leak-rate of 1, while this response gets slower and slower for lower leak-rates. We can now continue to extend this model for one neuron to a whole layer of neurons.

As each neuron has unique properties, we also need to introduce unique leak-rates for each neuron within a modelled layer of neurons. We can do so by formulating these leak-rates as a square-matrix $\Lambda$ (capital $\lambda$): On the main-diagonal of this square matrix we put the leak-rates for the neurons. All other elements are $0$. Let's note this down in detail in a small example with three inputs $U$ and two neurons:

\begin{align}
    X(t+1) &= \Lambda \cdot \left(W \cdot U(t)\right) + (I-\Lambda) \cdot \Theta(t) \\
           &=\Lambda \cdot \left(
    \begin{bmatrix}
        w_{1,1} & w_{1,2} & w_{1,3} \\
        w_{2,1} & w_{2,2} & w_{2,3}
    \end{bmatrix}
    \cdot
    \begin{bmatrix}
        u_1 \\
        u_2 \\
        u_3
    \end{bmatrix}
    \right) + (I-\Lambda) \cdot \Theta(t) \\
    &=
    \begin{bmatrix}
        \lambda_1 & 0 \\
        0 & \lambda_2
    \end{bmatrix}
    \cdot
    \begin{bmatrix}
        w_{1,1} u_1 + w_{1,2} u_2 + w_{1,3} u_3 \\
        w_{2,1} u_1 + w_{2,2} u_2 + w_{2,3} u_3
    \end{bmatrix}
    +
    \begin{bmatrix}
        1-\lambda_1 & 0 \\
        0 & 1-\lambda_2
    \end{bmatrix}
    \cdot
    \begin{bmatrix}
        \theta_1(t) \\
        \theta_2(t)
    \end{bmatrix} \\
    &=
    \begin{bmatrix}
        \lambda_1 \cdot (w_{1,1} u_1 + w_{1,2} u_2 + w_{1,3} u_3) \\
        \lambda_2 \cdot (w_{2,1} u_1 + w_{2,2} u_2 + w_{2,3} u_3)
    \end{bmatrix}
    +
    \begin{bmatrix}
        (1-\lambda_1) \cdot \theta_1(t) + 0 \cdot \theta_2(t) \\
        0 \cdot \theta_1(t) + (1-\lambda_2) \cdot \theta_2(t)
    \end{bmatrix}
\end{align}

Using this matrix notation, we can still use the dot-product for calculation, which is a benefit when it comes to programming and it has other advantages as we will see later. But first let's change our class for one neuron layer, to add leaky integration:

First we add a random initialised leak matrix by using the numpy `diag` function (line 9). As we do not want to calculate $I - \Lambda$ on every time-step, we calculate the recurrent leak also on initialisation (line 10). To set custom leak rates we also add a `set_leak` function which takes a numpy array of leak rates as input:

```python
class layer_leaky:

    def __init__(self, n_inputs: int, n_outputs: int, seed: int = 42):
        np.random.seed(seed)
        self.x = np.zeros([n_outputs, 1])
        self.theta = np.zeros([n_outputs, 1])
        self.u = np.zeros([n_inputs, 1])
        self.w = np.random.uniform(low=-1, high=1, size=[n_outputs, n_inputs])
        self.leak = np.diag(np.random.uniform(low=0, high=1, size=n_outputs))
        self.leak_rec = np.identity(n_outputs)-self.leak

    def set_leak(self, leak_rates: np.array):
        n_outputs = self.x.shape[0]
        self.leak = np.diag(leak_rates)
        self.leak_rec = np.identity(n_outputs)-self.leak

    def activate(self, input_vector):
        self.x = np.dot(self.leak, np.dot(self.w, input_vector)) + np.dot(self.leak_rec, self.theta)
        self.theta = np.tanh(self.x)

    def step(self, input_vector):
        self.activate(input_vector)
        return self.theta
```

We use the same three inputs for two neurons as in the previous post. To compare the behaviour of this neuron layer, we initialise one layer with random leak-rates (line 15) and one layer with a leak rate of $1$ (lines 16-17), thereby creating the equivalent to a layer without leak:

```python
# number of time-steps
length = 400

# inputs to the network as different sine waves
u0 = np.sin((np.arange(length) * np.pi / 10) + np.pi / 2)
u1 = 0.8*np.sin((np.arange(length) * np.pi / 15) + np.pi / 4)
u2 = 0.5*np.sin((np.arange(length) * np.pi / 20) + np.pi)

# stack the three inputs, so each row is the input for one time-step
u = np.stack([u0, u1, u2], axis=0)

o_1 = np.zeros([2, length])
o_2 = np.zeros([2, length])

neurons_1 = layer_leaky(3,2)
neurons_2 = layer_leaky(3,2)
neurons_2.set_leak(np.ones(2))

for i in range(length):
    o_1[:,i:i+1] = neurons_1.step(u[:,i:i+1])
    o_2[:,i:i+1] = neurons_2.step(u[:,i:i+1])
```

```python
# plot the stuff
fig, axes = plt.subplots(2, 1)
axes[0].set_title(r"Inputs")
axes[0].plot(u0, color="blue", linewidth=3, label=r"$u_1$")
axes[0].plot(u1, color="red", linewidth=3, label=r"$u_2$")
axes[0].plot(u2, color="black", linewidth=3, label=r"$u_3$")
axes[0].legend()
axes[1].set_title(r"Outputs")
axes[1].plot(o_2[0,:], color="black", linewidth=3, linestyle="dashed", label=r"$\theta_1, \lambda_1=1$")
axes[1].plot(o_1[0,:], color="black", linewidth=3, label=r"$\theta_1, \lambda_1=%.2f$" % neurons_1.leak[0,0])
axes[1].plot(o_2[1,:], color="red", linewidth=3, linestyle="dashed", label=r"$\theta_2, \lambda_2=1$")
axes[1].plot(o_1[1,:], color="red", linewidth=3, label=r"$\theta_2, \lambda_2=%.2f$" % neurons_1.leak[1,1])
axes[1].legend(loc=4)
axes[1].set_xlabel(r"time-steps")
```

The resulting output shows how the first neuron with a leak-rate far below $1$ ($0.06$) exhibits completely different dynamics compared to the previous configuration with a leak of $1$ (black). The second neuron with a leak-rate close to $1$ ($0.87$) expectedly shows almost the same behaviour as with a leak-rate of $1$ (red). Here we can observe how the leak-rate controls the dynamics of the neuron and speeds it up for high leak-rates and slows it down for small leak-rates. We can envision this property as simple low-pass filters:

To test the filter properties of leaky-integrating neurons, we can use a chirp from a high to a low frequency (from `scipy.signal.chirp`). The output of the neuron then tells us at which frequncy it responds for the given leak rate. Hence, we create a chirp from $100$ to $1$ $Hz$ in $1000$ steps for one second (lines 5-6) and test the response for lower leak rates (line 8):

```python
from scipy.signal import chirp

length= 1001

t = np.linspace(0, 1, length)
u = chirp(t, f0=100, f1=1, t1=1, method='linear')

leak_rates = [0.6, 0.4, 0.2, 0.1, 0.05, 0.02]
o = np.zeros([len(leak_rates), length])

for n, lr in enumerate(leak_rates):
    neuron = single_neuron_leaky(1, lr)
    neuron.w = np.ones(1)
    for i in range(length):
        o[n, i] = neuron.step(u[i])
```

```python
# Set plot parameters
rc('font', **{'family': 'sans-serif', 'sans-serif': ['Latin Modern Sans']})
params = {'font.size': 16,
          'figure.figsize': [20, 10],
          'text.usetex': True,
          'legend.fontsize': 'x-large',
          'axes.labelsize': 'x-large',
          'axes.titlesize':'x-large',
          'xtick.labelsize':'x-large',
          'ytick.labelsize':'x-large'}
plt.rcParams.update(params)

# plot the stuff
colors = ['#0072b2', '#e69f00', '#f0e442', '#009e73', '#56b4e9', '#d55e00', '#cc79a7', '#000000']
fig, ax = plt.subplots()
ax.plot(u, color='gray', linewidth=5, label=r"$u(t)$")
for n, lr in enumerate(leak_rates):
    scale = 1 # np.max([np.abs(np.min(o[n,:])), np.max(o[n,:])])
    ax.plot(o[n,:]/scale, color=colors[n], linewidth=3, label=r"$\theta(t), \lambda = %.2f $" % lr)

ax.legend()
ax.set_xlabel(r"$t (ms)$")
```

This shows us how a neuron with a leak-rate of $0.6$ responds with almost a constant amplitude to the chirp, while neurons with lower leak rates start to respond later.

At the end of this post I want to make another remark on why we modelled the leak rate as a matrix instead of a vector:
In the second half of our layer state-equation we use activations of other neurons $\Theta(t)$ to calculate the future states. As all other elements than the main-diagonal of the leak-rate matrix are zero, we only include the activation $\theta_i$ of one neuron in its own state update but not in the state update of other neurons. Setting the elements outside the main diagonal different from zero woul e.g. include $\theta_2$ in the state update of neuron $1$. This enables us to model recurrent connections as we will see in the next posts.

```python

```


# Lecture 16

## Exponential Distribution, Memoryless Property

## Exponential Distribution

### Description

Real-valued distribution describing wait times, survival times. Rate parameter $\lambda \gt 0$.

Continuous analog of the geometric distribution.

Unique in that the exponential distribution has the memoryless property.

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.xkcd()

x = np.linspace(0,5, 500)

_, ax = plt.subplots()

lambdas = [1.5, 1.0, 0.5]
for l in lambdas:
    y = l * np.e ** (l*-x)
    ax.plot(x, y, label="$\lambda = ${}".format(l))
ax.grid(True, which='both')
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')

legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

plt.show()
```

### Notation

$X \sim Expo(\lambda)$

### Parameters

$\lambda$ - rate parameter where $\lambda \gt 0$

### Probability density function

\begin{align}
  f(x) &=
  \begin{cases}
    \lambda e^{-\lambda x}, &\text{ if } x \ge 0 \\
    0, &\text{ otherwise }
  \end{cases}
\end{align}

### Cumulative distribution function

\begin{align}
  F(x) &= \int_{0}^{x}  \lambda e^{-\lambda t} \, dt \\
         &= \lambda \int_{0}^{x} e^{-\lambda t} \, dt &\text{ let } u = -\lambda t \text{, } du =  -\lambda \, dt \\
         &= \int - e^{u} \, du \\
         &= - e^{u} \\
         &= \left. - e^{-\lambda t} \right|_{0}^{x} \\
         &= \boxed{ 1 - e^{-\lambda x} }
\end{align}

### Standardized Exponential Distribution

If we let $Y = \lambda X$, then $Y \sim \mathbb{Expo}(1)$.

You can compare this with the standardized Normal.

Proof

\begin{align}
  P(Y \le y) &= P\left(X \le \frac{y}{\lambda} \right) \\
             &= 1 - e^{-\lambda \frac{y}{\lambda}} &\text{ just plugging } \frac{y}{\lambda} \text{ into the CDF above} \\
             &= 1 - e^{-y} &\text{ which is the CDF of } \mathbb{Expo}(1) ~~~~ \blacksquare \\
\end{align}

We will next find the mean and variance of $\mathbb{Expo}(1)$, and then derive the general case mean and variance afterwards.

### Mean and variance of $\mathbb{Expo}(1)$

Let $Y \sim \mathbb{Expo}(1)$, find $\mathbb{E}(Y)$ and $\mathbb{Var}(Y)$.

\begin{align}
  \mathbb{E}(Y) &= \int_{0}^{\infty} y\,e^{-y}\,dy & &\text{ let } u = y \text{, } du = dy \\
    & & &\text{ and let } dv = e^{-y}\,dy \text{, } v = -e^{-y} \\
    &= \underbrace{ \left. -y e^{^y} \right\vert_{0}^{\infty}}_{\text{evaluates to }0} + \underbrace{\int_{0}^{\infty} e^{-y}\,dy}_{\text{PDF of }\mathbb{Expo}(1)} \\
    &= \boxed{1} \\
    \\\\
  \mathbb{Var}(Y) &= \mathbb{E}(Y^2) - \mathbb{E}Y ^2\\
    &= \int_{0}^{\infty} y^{2}\,e^{-y}\,dy \,- 1^2 & &\text{ let } u = y^{2} \text{, } du = 2y\,dy \\
    & & &\text{ and let } dv = e^{-y}\,dy \text{, } v = -e^{-y} \\
    &= \left. -y^{2}\,e^{^y} \right\vert_{0}^{\infty} + \int_{0}^{\infty} 2y\,e^{-y}\,dy \,-1 \\
    &= 0 + 2 - 1 \\
    &= \boxed{1}
\end{align}

### Mean and variance of $\mathbb{Expo}(\lambda)$

We can derive the mean and variance of $\mathbb{Expo}(\lambda)$ from that of $\mathbb{Expo}(1)$.

From $Y = \lambda X$ we have $X = \frac{y}{\lambda}$.

\begin{align}
  \mathbb{E}(X) &= \mathbb{E}\left(\frac{Y}{\lambda}\right) \\
                &= \frac{1}{\lambda} \, \mathbb{E}(Y) \\
                &= \boxed{ \frac{1}{\lambda} } \\
  \\\\
  \mathbb{Var}(X) &= \mathbb{Var}\left(\frac{Y}{\lambda}\right) \\
                  &= \frac{1}{\lambda^2} \, \mathbb{Var}(Y) \\
                  &= \boxed{ \frac{1}{\lambda^2} } \\
\end{align}

### Memorylessness Property

If you have a random variable representing a wait-time (continuous), the memorylessness property means that no matter how long you have already waited, the probability that you will have to wait an _additional_ time $t$ is the same as if you were starting fresh from 0 (irrespective of the time you already spent waiting).

Fact: $\mathbb{Expo}(\lambda)$ is the only distribution with the memorylessness property.

\begin{align}
  P(X \ge s+t | X \ge s) &= P(X \ge t) \\
\end{align}

The survival function is the random variable that describes how long something might live/exist, in constrast to that for a waiting time. In other words, $P(X \ge s)$ is the probability that some object of interest _lasts longer than_ a continuous time $s$.

\begin{align}
  P(X \ge s) &= 1 - P(X \le s) \\
             &= 1 - (1 - e^{-\lambda s}) \\
             &= e^{-\lambda s} & \quad \text{ the survival function}
\end{align}

And so using this survival function in an equation using the definition of conditional probability, we have:

\begin{align}
  P(X \ge s+t | X \ge s) &= \frac{P(X \ge s+t \text{, }X \ge s)}{P(X \ge s)} \\
                         &= \frac{P(X \ge s+t)}{P(X \ge s)} & \quad \text{ since } P(X \ge s) \text{ is redundant} \\
                         &= \frac{e^{-\lambda (s+t)}}{e^{-\lambda s}} & \quad \text{ ratio of survival functions} \\
                         &= e^{-\lambda t} \\
                         &= P(X \ge t) & \quad \blacksquare
\end{align}

### Useful corollary of the Memorylessness Property

This is a brief introduction to conditional expectation, which is just like conditional probability.

Given $X \sim \mathbb{Expo}(\lambda)$, what is the expected wait-time if we have already waited for some time $a$?

\begin{align}
  \mathbb{E}(X | X \gt a) &= a + \mathbb{E}(X - a|X \gt a) & \quad \text{ by linearity} \\
                          &= a + \frac{1}{\lambda} & \quad \text{ by the memorylessness property}
\end{align}


##### Algorithms implemented in Python from the book "Discrete Mathematics" by Dossey, Otto, Spence and Eynden

```python
import numpy as np
import random
from sympy import *
```

### Chapter 1

#### Algorithm for evaluating $x^n$

```python
def eval(x, n):
    P=x
    k=1
    while k<n:
        P*=x
        k+=1
    return P

eval(3, 4)
```

    81

#### Polynomial Evaluation Algorithm: computes $P(x)=a_nx^n + a_{n-1}x^{n-1} +...+ a_0$, given the non-negative integer $n$ and real numbers $x$, $a_0$, $a_1$, ..., $a_n$

```python
def P(x, a):
    # a is the list of the constants: a0, a1, ..., an
    S=a[0]
    k=1
    n=len(a)-1
    while k<=n:
        S+=a[k]*x**k
        k+=1
    return S
P(2, [4, 3, -2, 5])
```

    42

#### Horner's Polynomial Evaluation Algorithm

```python
def HP(x, a):
    n=len(a)-1
    S=a[n]
    k=1
    while k<=n:
        S=x*S+a[n-k]
        k+=1
    return S
HP(2, [4, 3, -2, 5])
```

    42

#### Next Subset Algorithm

```python
def NS(a):
    """a is a string of 0s and 1s corresponding to a subset of n elements"""
    n=len(a)
    a=list(a)
    k=n-1
    while k>=0 and a[k]=='1':
        k-=1
    if k>=0:
        a[k]='1'
        for j in range(k+1, n):
            a[j]='0'
        return ''.join(a)
    else: return 'The string contains all 1s'
NS('110')
```

    '111'

#### Bubble Sort Algorithm

```python
def BS(a):
    """A is a list of unsorted real numbers"""
    n=len(a)-1
    for j in range(n):
        for k in range(n-1, j-1, -1):
            if a[k+1]<a[k]:
                t=a[k+1]
                a[k+1]=a[k]
                a[k]=t
    return a

BS([1.2, 3.2, 9.0, 3.2, 0.5, 0.43, 8.44])
```

    [0.43, 0.5, 1.2, 3.2, 3.2, 8.44, 9.0]

#### Revised Polynomial Evaluation Algorithm

```python
def RP(x, a):
    S=a[0]
    y=1
    k=1
    n=len(a)-1
    while k<=n:
        y=x*y
        S+=y*a[k]
        k+=1
    return S
RP(2, [4, 3, -2, 5])
```

    42

 ### Chapter 3

#### The Euclidean Algorithm

```python
def gcd(m, n):
    """m and n are both non-negative integers"""
    while n!=0:
        m, n = n, m%n
    return m

gcd(427, 154)
```

    7

#### The Extended Euclidean Algorithm

```python
def exgcd(m, n):
    r=[m, n]
    x=[1, 0]
    y=[0, 1]
    i=1
    while r[i]!=0:
        i+=1
        q=r[i-2]//r[i-1]
        x+=[x[i-2]-q*x[i-1],]
        y+=[y[i-2]-q*y[i-1],]
        r+=[r[i-2]%r[i-1],]
    return (r[i-1], x[i-1], y[i-1])
exgcd(101, 1120)
```

    (1, -499, 45)

#### The Modular Exponentiation Algorithm: Given positive integers P, E and n, this algorithm computes the remainder when $P^E$ is divided by $n$.

```python
def ME(P, E, n):
    r2=1
    p=P
    e=E
    while e!=0:
        Q=e//2
        R=e%2
        r1=p**2%n
        if R==1: r2=(r2*p)%n
        p=r1
        e=Q
    return r2

ME(582, 621,1189)
```

    90

#### Check Matrix Row Decoding Algorithm

```python
def CMRD(codeword, generatorMatrix):
    A=generatorMatrix
    w=codeword
    k=len(A)
    n=len(A[0])
    J=np.copy(A[:, k:])
    A_star=np.append(J, np.eye(len(J[0]), dtype=int), axis=0)
    print(A_star)
    s=(w.dot(A_star))%2
    print(s)
    if sum(s)==0: return w[:k]
    if list(s) in A_star.tolist():
        i=np.where(np.all(A_star==s,axis=1))
        w[i]=(w[i]+1)%2
        return w[:k]
    if list(s) not in A_star.tolist(): return "unknown"

A=np.array(([1, 0, 0, 1, 0, 1, 0], [0, 1, 0, 1, 1, 0, 1], [0, 0, 1, 0, 1, 1, 1]))
w1=np.array([0, 0, 1, 0, 1, 1, 1])
w2=np.array([1, 0, 1, 1, 1, 0, 0])
w3=np.array([0, 1, 1, 1, 0, 1, 0])
w4=np.array([1, 0, 0, 0, 1, 1, 1])
w5=np.array([1, 1, 0, 0, 0, 1, 0])
#CMRD(w5, A)

#ex 3.6, 29
B=np.array(([1, 0, 1, 1, 0], [0, 1, 0, 1, 1])) # Generator matrix
c=np.array([1, 0, 1, 1, 0]) # received word
CMRD(c, B)
```

    [[1 1 0]
     [0 1 1]
     [1 0 0]
     [0 1 0]
     [0 0 1]]
    [0 0 0]

    array([1, 0])

### Chapter 4

#### Euler circuit Algorithm

```python
V=['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J']
Ed={'a':['A', 'B'], 'c':['B', 'E'], 'f':['E', 'D'], 'b':['D', 'A'], 'd':['E', 'C'], 'e':['C', 'F'], 'g':['F', 'E'], 'h':['E', 'G'], 'k':['H', 'G'], 'm':['H', 'J'], 'l':['J', 'H'], 'j':['H', 'E']}

def EulerCircuit(V, Ed):
    E1=list(Ed.keys())
    E2=[Ed[i] for i in Ed]
    U=random.choice(V)
    R=[i for i in E2 if U in i]
    #print(R)

    st=random.choice(R)
    ind=E2.index(st)
    C={}
    #print(C)

    if st[0]!=U: st=st[::-1]
    C[E1[ind]]=st
    del E1[ind]
    del E2[ind]
    #print(E1)
    #print(E2)

    while st[1]!=U:
        u=st[1]
        R=[i for i in E2 if u in i]
        st=random.choice(R)
        ind=E2.index(st)
        if st[0]!=u: st=st[::-1]
        C[E1[ind]]=st
        del E1[ind]
        del E2[ind]

    while len(E1)!=0 and len(E2)!=0:
        E21=sympy.flatten(E2)
        E3=list(C.keys())
        E4=[C[i] for i in C]
        E41=sympy.flatten(E4)
        L=[] # store the vertices whose edges are not listed in C
        L=list(set([i for i in E41 if E21.count(i)>0]))
        #print(L)
        u= random.choice(L)

        R=[i for i in E2 if u in i]
        st=random.choice(R)
        ind=E2.index(st)
        P={}

        if st[0]!=u: st=st[::-1]
        P[E1[ind]]=st
        del E1[ind]
        del E2[ind]

        while st[1]!=u:
            u1=st[1]
            R=[i for i in E2 if u1 in i]
            st=random.choice(R)
            ind=E2.index(st)
            if st[0]!=u1: st=st[::-1]
            P[E1[ind]]=st
            del E1[ind]
            del E2[ind]

        x=list(P.keys())
        y=[P[i] for i in P]
        z=[i for i in E4 if i[1]==u]
        #print(E3)
        #print(E4)
        #print(x, y, z)
        ra=random.choice(z)
        k1=E4.index(ra)
        E3[k1+1:k1+1]=x
        E4[k1+1:k1+1]=y
        C={}
        for i in range(len(E3)): C[E3[i]]=E4[i]
        #print(P)

    return C

#EulerCircuit(V, Ed)

Ed={'a':['U', 'V'], 'b':['U', 'V'], 'c':['U', 'A'], 'd':['A', 'V'], 'e':['U', 'V']}
V=['A', 'U', 'V']
print(Ed)
EulerCircuit(V, Ed)
```

    {'a': ['U', 'V'], 'b': ['U', 'V'], 'c': ['U', 'A'], 'd': ['A', 'V'], 'e': ['U', 'V']}

    {'a': ['V', 'U'],
     'b': ['U', 'V'],
     'e': ['V', 'U'],
     'c': ['U', 'A'],
     'd': ['A', 'V']}

#### Breadth-First Search Algorithm

```python
#Ed=[['S', 'A'], ['S', 'B'], ['A', 'E'], ['E', 'C'], ['A', 'C'], ['C', 'B'], ['F', 'E'], ['F', 'H'], ['C', 'H'], ['H', 'I'], ['I', 'J'], ['J', 'G'], ['G', 'D'], ['D', 'B']]
#V=['S', 'A', 'B', 'E', 'F', 'H', 'I', 'J', 'G', 'D', 'B', 'C']
#start='S'
#end='I'

Ed=[['S', 'A'], ['S', 'D'], ['D', 'B'], ['G', 'D'], ['B', 'G'], ['E', 'C'], ['E', 'F'], ['C', 'F'], ['E', 'G'], ['G', 'H'], ['F', 'H'], ['F', 'T'], ['H', 'I'], ['T', 'I']]
V=['S', 'A', 'D', 'B', 'G', 'E', 'C', 'F', 'G', 'H', 'I', 'T']
start='S'
end='T'

def BFSA(Ed, V, start, end):
    st=[start]
    L={st[0]:[0, '-']}

    k=0

    while len(V)>1:
        k+=1
        st1=[]
        for s in st:
            del V[V.index(s)]
            I=[j for j in Ed if (j[0]==s or j[1]==s)]
            for i in I:
                del Ed[Ed.index(i)]
            I1=[i[::-1] if i[1]==s else i for i in I]
            I2=[i for i in I1 if (i[0] in st and i[1] not in st) or (i[0] not in st and i[1] in st) ]
            for [i, j] in I2:
                L[j]=[k, s]
                st1+=[j, ]
        st=list(set(st1))

    print(L)

    path=end
    pr=L[end][1]
    while pr!='-':
        path+=pr
        pr=L[pr][1]

    return path[::-1]

BFSA(Ed, V, start, end)
```

    {'S': [0, '-'], 'A': [1, 'S'], 'D': [1, 'S'], 'B': [2, 'D'], 'G': [2, 'D'], 'E': [3, 'G'], 'H': [3, 'G'], 'C': [4, 'E'], 'F': [4, 'H'], 'I': [4, 'H'], 'T': [5, 'F']}

    'SDGHFT'

#### Dijkstra's Algorithm

```python
#Ed=[['S', 'A', 3], ['S', 'B', 1], ['C', 'A', 2], ['C', 'B', 3], ['D', 'B', 5], ['A', 'B', 1], ['C', 'E', 3], ['D', 'C', 1], ['D','E', 1]]
#V=['A', 'S', 'B', 'D', 'C', 'E']
#st='S'
#end='E'

##Ex 4.3, 6
#Ed=[['S', 'C', 2], ['S', 'E', 4], ['E', 'C', 1], ['C', 'F', 3], ['C', 'D', 5], ['F', 'E', 1], ['F', 'G', 2], ['D', 'F', 3], ['D', 'G', 1], ['D', 'H',1], ['G', 'H', 3], ['A', 'H', 3], ['E', 'J', 2], ['G', 'J', 3]]
#V=['S', 'C', 'D', 'E', 'F', 'G', 'H', 'A', 'J']
#st='S'
#end='A'

##Ex 4.3, 7
Ed=[['S', 'C', 3], ['S', 'E', 2], ['S', 'H', 1], ['C', 'E', 1], ['E', 'H', 1], ['F', 'C', 3], ['F', 'H', 2], ['F', 'D', 2], ['C', 'D', 2], ['F', 'B', 3], ['B', 'H', 5], ['G', 'D', 1], ['G', 'B', 1], ['G', 'A', 2], ['A', 'D', 5], ['A', 'B', 4]]
V=['S', 'C', 'E', 'H', 'D', 'F', 'G', 'B', 'A']
st='S'
end='A'

def DA(Ed, V, st, end):
    """Each of the entries in Ed has form: ['vertex1', 'vertex2', weight of the edge joining vertex1 and vertex2]"""
    E1=[]
    E2=[]
    for i in Ed:
        E1+=[[i[0], i[1]], ]
        E2+=[i[2], ]
    P={st:[0, '-']}
    V1=V[:]
    del V1[V1.index(st)]
    temp={}
    I=[]
    for i in V1:
        if [i, st] in E1:
            ind=E1.index([i, st])
            temp[i]=[E2[ind], st]
            I+=[ind,]

        elif [st, i] in E1:
            ind=E1.index([st, i])
            temp[i]=[E2[ind], st]
            I+=[ind,]

        elif (not [i, st] in E1) or (not [st, i] in E1):
            temp[i]=[oo, st]

    for ind in sorted(I, reverse=True):
        del E1[ind]
        del E2[ind]

    t1=list(temp.keys())
    t2=[temp[i] for i in temp]

    while len(t2)!=0:
        temp1={}
        I=[]
        Min=min(t2)
        Ind=t2.index(Min)
        st_old=st
        st=t1[Ind]
        label_st=Min
        P[st]=label_st
        del V1[V1.index(st)]

        for i in V1:
            if [i, st] in E1:
                old_label=temp[i]
                o1=old_label[0]
                o2=old_label[1]
                ind=E1.index([i, st])
                if o1<=label_st[0]+E2[ind]: temp1[i]=old_label
                else:
                    Minimum=min([o1, label_st[0]+E2[ind]])
                    temp1[i]=[Minimum, st]
                    I+=[ind,]

            elif [st, i] in E1:
                old_label=temp[i]
                o1=old_label[0]
                o2=old_label[1]
                ind=E1.index([st, i])
                if o1<=label_st[0]+E2[ind]: temp1[i]=old_label
                else:
                    Minimum=min([o1, label_st[0]+E2[ind]])
                    temp1[i]=[Minimum, st]
                    I+=[ind,]

            elif (not [i, st] in E1) or (not [st, i] in E1):
                temp1[i]=temp[i]

        for ind in sorted(I, reverse=True):
            del E1[ind]
            del E2[ind]
        temp=temp1

        t1=list(temp.keys())
        t2=[temp[i] for i in temp]

    path=end
    pr=P[end][1]
    if P[end][0]==oo: return 'No shortest path exists'
    while pr!='-':
        path+=pr
        pr=P[pr][1]
    print(P)
    return path[::-1]

DA(Ed, V, st, end)

```

    {'S': [0, '-'], 'H': [1, 'S'], 'E': [2, 'S'], 'F': [3, 'H'], 'C': [3, 'S'], 'D': [5, 'F'], 'G': [6, 'D'], 'B': [6, 'H'], 'A': [8, 'G']}

    'SHFDGA'


Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel → Restart) and then **run all cells** (in the menubar, select Cell → Run All).

Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE", as well as your name and collaborators below:

```python
NAME = "Prabal Chowdhury"
COLLABORATORS = ""
```

# **CSE330 Lab: Hermite Interpolation**

Hermite Interpolation is an example of a variant of the interpolation problem, where the interpolant matches one or more derivatives of  f  at each of the nodes, in addition to the function values.

## **Importing the necessary libraries**

```python
import numpy as np
import matplotlib.pyplot as plt
from itertools import combinations
from numpy.polynomial import Polynomial
```

## **Creating the components for Hermite interpolation**

For the case of Hermite Interpolation, we look for a polynomial that matches both  $f'(x_i)$  and  $f(x_i)$  at the nodes  $x_i = x_0,\dots,x_n$ . Say you have  $n+1$  data points, $(x_0,y_0),(x_1,y_1),x_2,y_2),…,(x_n,y_n)$  and you happen to know the first-order derivative at all of these points, namely,  $(x_0, y_0 ^\prime ), (x_1, y_1 ^\prime ), x_2, y_2 ^\prime ), \dots ,(x_n, y_n ^\prime )$ . According to hermite interpolation, since there are  $2n+2$  conditions;  $n+1$  for  $f(x_i)$  plus  $n+1$  for  $f′(x_i)$ ; you can fit a polynomial of order  $2n+1$ .

General form of a  $2n+1$  degree Hermite polynomial:

$$p_{2n+1} = \sum_{k=0}^{n} \left(f(x_k)h_k(x) + f'(x_k)\hat{h}_k(x)\right), \tag{1}$$

where  $h_k$  and  $\hat{h}_k$  are defined using Lagrange basis functions by the following equations:

$$h_k(x) = (1-2(x-x_k)l^\prime_k(x_k))l^2_k(x_k), \tag{2}$$

and
$$\hat{h}_k(x) = (x-x_k)l^2_k(x_k), \tag{3}$$

where the Lagrange basis function being:
$$l_k(x) = \prod_{j=0, j\neq k}^{n} \frac{x-x_j}{x_k-x_j}. \tag{4}$$

**Note** that, we can rewrite Equation  $(2)$  in this way,
\begin{align}
h_k(x) &= \left(1-2(x-x_k)l^\prime_k(x_k) \right)l^2_k(x_k) \\
&= \left(1 - 2xl^\prime_k(x_k) + 2x_kl^\prime_k(x_k) \right)l^2_k(x_k) \\
&= \left(1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x \right) l^2_k(x_k) \tag{5}
\end{align}

Replacing  $l′_k(x_k)$  with  m , we get:
$$h_k(x) = (1 - 2xm + 2x_km)l^2_k(x_k). \tag{6}$$
## **Tasks:**
* The functions: l(k, x), `h(k, x)` and `h_hat(k, x)` calculate the corresponding
$l_k$,  $h_k$, and  $\hat{h}_k$, respectively.

* Function `l(k, x)` has already been defined for you.
* Your task is to complete the `h(k, x)`, `h_hat(k, x)`, and `hermit(x, y, y_prime)` functions.

Later we will draw some plots to check if the code is working.

### **Part 1: Calculate $l_k$**

This function uses the following equation to calculate $l_k(x)$ and returns a polynomial:
$$l_k(x) = \prod_{j=0, j\neq k}^{n} \frac{x-x_j}{x_k-x_j}$$

```python
def l(k, x):
    n = len(x)
    assert (k < len(x))

    x_k = x[k]
    x_copy = np.delete(x, k)

    denominator = np.prod(x_copy - x_k)

    coeff = []

    for i in range(n):
        coeff.append(sum([np.prod(x) for x in combinations(x_copy, i)]) * (-1)**(i) / denominator)

    coeff.reverse()

    return Polynomial(coeff)
```

### **Part 2: Calculate  $h_k$**

This function calculates  $h_k(x)$  using the following equation:
$$h_k(x) = \left(1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x \right) l^2_k(x_k).$$

This equation is basically a multiplication of two polynomials.

First polynomial:  $1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x$

Second polynomial:  $l^2_k(x_k)$ .

The `coeff` variable should contain a python list of coefficient values for the **first** polynomial of the equation. These coefficient values are used to create a polynomial `p`.

```python
def h(k, x):
    # initialize with None. Replace with appropriate values/function calls
    # initialize with None. Replace with appropriate values/function calls
    l_k = l(k, x)
    l_k_sqr = l_k * l_k
    l_k_prime = l_k.deriv(1)
    coeff = [1 + 2 * x[k] * l_k_prime(x[k]), -2 * l_k_prime(x[k])]
    p = Polynomial(coeff)

    # --------------------------------------------
    # YOUR CODE HERE

    # --------------------------------------------

    return p * l_k_sqr
```

```python
# Test case for the h(k, x) function

x = [3, 5, 7, 9]
k = 2
h_test = h(k, [3, 5, 7, 9])
h_result = Polynomial([-2.5, 0.5]) * (l(k, x) ** 2)

assert Polynomial.has_samecoef(h_result, h_test)
assert h_result == h_test
```

### **Part 3: Calculate  $\hat{h}_k$**
This function calculates  $\hat{h}_k(x)$  using the following equation:

$$\hat{h}_k(x) = (x-x_k)l^2_k(x_k).$$

This equation is also a multiplication of two polynomials.

First polynomial:  $x-x_k$ .

Second polynomial:  $l^2_k(x_k)$ .

The `coeff` variable should contain a python list of coefficient values for the **first** polynomial of the equation. These coefficient values are used to create a polynomial `p`.

```python
def h_hat(k, x):
    # Initialize with none
    l_k = l(k, x)
    l_k_sqr = l_k * l_k
    coeff = [-x[k], 1]
    p = Polynomial(coeff)

    # --------------------------------------------
    # YOUR CODE HERE

    # --------------------------------------------

    return p * l_k_sqr
```

```python
# Test case for the h(k, x) function

x = [3, 5, 7, 9]
k = 2
h_test = h_hat(k, [3, 5, 7, 9])
h_result = Polynomial([-7, 1]) * (l(k, x) ** 2)

assert Polynomial.has_samecoef(h_result, h_test)
assert h_result == h_test
```

### **Part 4: The Hermite Polynomial**
This function uses the following equation:

$$p_{2n+1} = \sum_{k=0}^{n} \left(f(x_k)h_k(x) + f'(x_k)\hat{h}_k(x)\right).$$

The polynomial denoted by the equation is calculated by the variable `f`.

```python
def hermit(x, y, y_prime):
    assert len(x) == len(y)
    assert len(y) == len(y_prime)

    f = Polynomial([0.0])
    # --------------------------------------------
    # YOUR CODE HERE
    for i in range(len(x)):
        f += y[i] * h(i, x) + y_prime[i] * h_hat(i, x)
    # --------------------------------------------
    return f
```

### **Testing our methods by plotting graphs.**
**Note:**

* For each of the 5 plots, there will be 2 curves plotted: one being the original function, and the other being the interpolated curve.
* The original functions are displayed in orange color, while the hermite interpolated curves are in blue.
* `x`, `y`, and `y_prime` contain  $x_i$, $f(x_i)$, and  $f'(x_i)$  of the given nodes of the original function  $f$ .

Upon calling the `hermit()` function, it returns a polynomial `f`. For example, for plot 1, it is called `f3`.

In general, a polynomial may look like the following:  $f = 1 + 2x + 3x^2$. Next, we pass in a number of  $x$  values to the polynomial by calling the `.linspace()` function on the polynomial object using `f.linspace()`. This function outputs a tuple, which is stored in a variable called data. First element of data contains a 1D numpy array of  $x_i$  values generated by `linspace()`, and the second element of data contains a 1D numpy array of the corresponding  $y_i$  values outputted by our example polynomial:  $f = 1 + 2x + 3x^2$.

Using `test_x`, we generate a range of  $x_i$  values to plot the original function, and `test_y` contains the corresponding  $y_i$  values of the original function. For the first plot, our original function is the sine curve.

For all the plots:

`plt.plot(test_x, test_y)` plots the original function.

`plt.plot(data[0], data[1])` plots the interpolated polynomial.

```python
pi      = np.pi
x       = np.array([0.0, pi/2.0,  pi, 3.0*pi/2.0])
y       = np.array([0.0,    1.0, 0.0,       -1.0])
y_prime = np.array([1.0,    0.0, 1.0,        0.0])
```

**Plot 1**: trying to interpolate a sine curve `(np.sin())` using first 2 nodes in `x` and `y`, and their corresponding derivative in `y_prime`.

```python
n      = 1
f3     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f3.linspace(n=50, domain=[-3, 3])
test_x = np.linspace(-3, 3, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()
np.testing.assert_allclose(data[1][20:32], test_y[20:32], atol=0.7, rtol=1.4)
```

**Plot 2**: trying to interpolate a sine curve `(np.sin())` using first 3 nodes in `x` and `y` and their corresponding derivative in `y_prime`.

```python
n      = 2
f5     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f5.linspace(n=50, domain=[-0.7, 3])
test_x = np.linspace(-2*pi, 2*pi, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(test_x, test_y) # 25-
plt.plot(data[0], data[1]) # 10-33
plt.show()

data = f5.linspace(n=50, domain=[0, 3])
test_x = np.linspace(0, 3, 50, endpoint=True)
test_y = np.sin(test_x)
np.testing.assert_allclose(data[1], test_y, atol=0.5, rtol=1.7)
```

**Plot 3**: trying to interpolate a sine curve `(np.sin())` using first 4 nodes in `x` and `y` and their corresponding derivative in `y_prime`.

```python
n      = 3
f7     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f7.linspace(n=50, domain=[-0.3, 3])
test_x = np.linspace(-2*pi, 2*pi, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()

data = f7.linspace(n=50, domain=[0, 3])
test_x = np.linspace(0, 3, 50, endpoint=True)
test_y = np.sin(test_x)
np.testing.assert_allclose(data[1], test_y, atol=0.8, rtol=1.9)
```

**Plot 4**: trying to interpolate an exponential curve `(np.exp())` using all nodes in `x` and `y` and their corresponding derivatives in `y_prime`.

```python
#defining new set of given node information: x, y and y'
x       = np.array([0.0, 1.0,          2.0       ])
y       = np.array([1.0, 2.71828183,  54.59815003])
y_prime = np.array([0.0, 5.43656366, 218.39260013])

f7      = hermit( x, y, y_prime)
data    = f7.linspace(n=50, domain=[-0.5, 2.2])
test_x  = np.linspace(-0.5, 2.2, 50, endpoint=True)
test_y  = np.exp(test_x**2)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()

np.testing.assert_allclose(test_y[27:47], data[1][27:47], atol=3, rtol=0.4)
```

**Plot 5:** trying to interpolate  $y = (x-3)^2 + 1$  using all nodes in `x` and `y` and their corresponding derivatives in `y_prime`.

For this plot you might be able to see only one curve due to the two curves overlapping. This means that our polynomial is accurately interpolating the original function.

```python
#defining new set of given node information: x, y and y'
x       = np.array([1.0, 3.0, 5.0])
y       = np.array([5.0, 1.0, 5.0])
y_prime = np.array([-4.0, 0.0, 4.0])

f7      = hermit( x, y, y_prime)
data    = f7.linspace(n=50, domain=[-10, 10])
test_x  = np.linspace(-10, 10, 50, endpoint=True)
test_y  = (test_x-3)**2 + 1

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()

np.testing.assert_allclose(test_y, data[1], atol=0.1, rtol=0.1)
```


# Workshop 8 optional material: Simulating dynamical systems

Physical systems that evolve in time can be easily (but not always quickly) simulated on a computer.   All you need to know is the function that takes the system from its current state to a future state:

$S_2 = f(S_1, dt)$

In the above equation, $S_1$ and $S_2$ represent the current and future states, respectively.  $S_2$ is the state of the system after a short amount of time, $dt$.  The function $f$, which encodes the dynamics of the system, often depends on the current state and the choice of $dt$.  Choosing a smaller time step, $dt$, generally improves the accuracy of any simulation.

For example, the position of a particle, $x(t)$, in one dimension can be written as a function of its velocity $v$:

$\frac{dx}{dt} = v$

You could find $x(t)$ using differential equation solving methods, or you could just simulate the motion of the particle.  To do this, we can express the dynamics as:

\begin{align}
\ x_2 & = f(x_1, dt) \\
\ & = x_1 + v*dt \\
\end{align}

```python
import matplotlib.pyplot as plt

velocity = 1.0
dt = 0.01

def update(x, t):
    '''This is like the function f above; you should
       think of the "state" of the system as its position
       at some time.  We update its state according to the
       dynamics of the system, and I prefer to also update
       the value of time in this same function--but this could
       also be done in the loop below.'''

    # update the state and time
    x = x + velocity*dt
    t = t + dt

    # return the updated values
    return x, t
```

```python
x = 5 # initial x position
t = 0 # initial time

# Some lists to store the positions and times
x_values = [x]  # I like putting the initial values in the list
t_values = [t]

# This is where the actual "simulation" occurs; the update
# function is used in a loop with however many iterations we want
for i in range(100):

    x, t = update(x, t)
    x_values.append(x)
    t_values.append(t)

# Finally, some plotting to visualize the results
plt.plot(x_values, t_values)
plt.ylabel('position')
plt.xlabel('time')
plt.show()
```

Wow! It's a straight line.  You probably could've guessed that or used some fancy, built-in ODE solving libraries.  But the point is, if you can write down an `update( )` function for some system, then you can simulate it--whether or not there's an analytical way to find a solution.

There's a chance that your capstone project is just a variant of this problem, so I don't want to steal away too many project ideas in this notebook.  Once you start simulating a physical system, it's good to consider the accuracy of your simulated "solution".  If you don't have an exact solution to compare against, you can adjust the chosen value of $dt$ and see how it changes the simulation.

## Exercise 1

Change the code above to simulate a particle under constant acceleration in one dimension.  The `update` function should now also update and return the velocity of the particle.  Plot the position of the particle as a function of time using at least two different choices of time step $dt$.  It may be helpful to include $dt$ as an input parameter to the `update` function so you can easily adjust $dt$ in your simulations.

```python

```

Stochastic systems (systems with some randomness) are another great use-case for this method.  [Brownian motion](https://en.wikipedia.org/wiki/Brownian_motion) describes the random motion of a particle (like a piece of dust) in a fluid.  The thermal motion of the fluid molecule bumps the dust particle around in a seemingly random, but well-described, process.  (In theory you could also model this by keeping track of the motion of many tiny fluid particles colliding with each other, the walls of some container, and the large dust particle.)

Below, I've written down an update function for the position of a particle undergoing Brownian motion.

```python
import numpy as np
import matplotlib.pyplot as plt

def update(x, t, dt):

    x = x + np.sqrt(dt)*np.random.randn()
    t = t + dt

    return x, t
```

```python
x = 0
t = 0
x_values = [x]
t_values = [t]

for i in range(100):

    x, t = update(x, t, 0.1)

    x_values.append(x)
    t_values.append(t)

plt.plot(t_values, x_values)
plt.show()
```

## Exercise 2

Repeat the Brownian motion simulation many (~1000) times and plot the distribution of final positions.  The mean of the distribution should be zero, but there is some variance to this distribution.  How does the variance and/or standard deviation of this distribution scale with time?

```python

```

## Viral spreading

For a more complicated example, here's a simulation of viral spreading following the mathematics of [this paper](https://www.pnas.org/content/pnas/111/46/E4911.full.pdf). You can play with the parameter $\mu$ (mu) to see how it controls the nature of the spreading.

```python
import numpy as np

prob = 0.4  # Probability of infection
L = 2000  # Size of grid

# The distribution from which "jumps" are drawn
def jump(y, mu, L, C):
    return np.power((y*(np.power(L, -mu) - np.power(C, -mu)) + np.power(C, -mu)), -1/mu)

# function that updates the state of the population and list of infected individuals
def update(population, infected,  mu, L, C):
    for inf in infected:
        if np.random.random() > prob:
            start_x = inf[0]
            start_y = inf[1]
            angle = np.random.uniform(0, 2*np.pi)
            dist = jump(np.random.random(), mu, L, C)
            end_x = int(start_x + np.cos(angle)*dist) % L
            end_y = int(start_y + np.sin(angle)*dist) % L
            if population[end_x, end_y] == 0:
                population[end_x, end_y] = 1
                infected.append([end_x, end_y])

    return population, infected

```

```python
population = np.zeros((int(L),int(L)))
population[1000,1000] = 1

infected = [[1000,1000]]

mu = 1.8
C = 1.5
pending = []

ts = []
rs = []
populations = np.zeros((30, int(L), int(L)))

# The simulation -- each iteration is more computationally expensive than the last,
# so be careful changing the number of iterations
for i in range(26):
    populations[i] = population
    population, infected = update(population, infected, mu, L, C)
```

```python
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import cm,rc
from matplotlib.colors import ListedColormap, LinearSegmentedColormap

dark = cm.get_cmap('Dark2_r', 256)
newcolors = dark(np.linspace(0, 1, 256))
white = np.array([1,1,1,1])
newcolors[:25, :] = white
newcmp = ListedColormap(newcolors)

fig = plt.figure(figsize=(8,8))

a = populations[25]  # You can change this number to get an earlier/later state of the population

im = plt.imshow(a, interpolation='none', aspect='auto', vmin=0, vmax=1, cmap = newcmp)
plt.show()
```


## Learn Calculus with Python

#### start

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

x = np.arange(0,100,0.1)
y = np.cos(x)

plt.plot(x,y)
plt.show()
```

#### normal function

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def f(x):
    return x**3 / (25)

f(30)
f(10)
```

    1080.0

    40.0

```python
# num -> smooth ( how many dots )
x = np.linspace(-30,30,num=1000)
y = f(x)

plt.plot(x,y)
```

#### exp

```python
def exp(x):
    return np.e**x

exp(2)
np.exp(2)

xz = np.linspace(1, 20, num=100)
yz = exp(xz)

plt.plot(xz, yz)
```

```python
def exp2(x):
    sum = 0
    for k in range(100):
        sum += float(x**k)/np.math.factorial(k)
    return sum

exp2(1)
exp(1)
```

    2.7182818284590455

    2.718281828459045

#### log

```python
# pick 100 items between 1 and 500 isometricly
x = np.linspace(1,500,100,endpoint=False)

y1 = np.log2(x)
y2 = np.log(x)
y3 = np.log10(x)

plt.plot(x,y1,'red',x,y2,'green',x,y3,'blue')
```

#### trigonometric

```python
pi_val = np.pi
pi_range = np.linspace(-2*pi_val,2*pi_val )

plt.plot(
    pi_range,
    np.sin(pi_range)
)

plt.plot(
    pi_range,
    np.cos(pi_range)
)
```

#### f(g(x))

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
f = lambda x:x+20
g = lambda x:x**2
h = lambda x:f(g(x))

x = np.array(range(-30,30))
plt.plot(x,h(x),'bs')
```

#### f-1(x)

```python
w = lambda x: x**2
winv = lambda x: np.sqrt(x)

x = np.linspace(0,2,100)

plt.plot(x,w(x),'b',x,winv(x),'r',x,x,'g-.')
```

#### *higher order functions*

```python
def horizontal_shift(f,W):
    return lambda x: f(x-W)

g = lambda x:x**2

x = np.linspace(-20,20,100)
shifted_g = horizontal_shift(g,5)

plt.plot(x,g(x),'b',x,shifted_g(x),'r')
```

#### Euler's Formula

```python
# 即'欧拉公式'

rules_of_imaginary_number = '''
i^0 = 1   i^1 = i   i^2 = -1   i^3 = -i
i^4 = 1   i^5 = i   i^6 = -1   i^7 = -i '''

euler_equation = '''
e^(i*x) = cos(x) + i*sin(x)
e^(i*pi) + 1 = 0  <= (if x=pi) '''

# --- sympy ---

from sympy import Symbol,expand,E,I

z = Symbol('z',real=True)
expand(E**(I*z),complex=True)

# --- numpy ---

x = np.linspace(-np.pi,np.pi)
lhs = np.e**(1j*x)
rhs = np.cos(x) + 1j*np.sin(x)

sum(lhs==rhs)
len(x)
```

    I*sin(z) + cos(z)

    50

    50

#### Higher Derivatives

```python
from sympy.abc import x,y

f = x**4

f.diff(x,2)
f.diff(x).diff(x)
```

    12*x**2

    12*x**2

#### Ordinary Differential Equations

```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *

t = Symbol('t')
c = Symbol('c')
domain = np.linspace(-3,3,100)

v = t**3-3*t-6
a = v.diff()
```

```python
for p in np.linspace(-2,2,20):
    slope = a.subs(t,p)
    intercept = solve(slope*p+c-v.subs(t,p),c)[0]
    lindomain = np.linspace(p-1,p+1,20)
    plt.plot(lindomain,slope*lindomain+intercept,'red',linewidth=1)

plt.plot(domain,[v.subs(t,i) for i in domain],linewidth=3)
```


# Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is an optimization method that is applicable to problems where
the objective function is given as the sum of several terms
$$
E(w) = E_1(w) + E_2(w) + \dots + E_N(w)
$$

Many models in machine learning lead to optimization objectives that have this form. For example, when $E_i(w)$ is the prediction error on example $i$ when using parameter $w$, the total error $E(w)$ is the sum of individual terms.

SGD is an algorithm that can be used in this case. We make the following observations:

* We can work with the average error
$$
\frac{1}{N}E(w) = \frac{1}{N}(E_1(w) + E_2(w) + \dots + E_N(w))
$$
and the true average gradient is
$$
\frac{1}{N} \nabla E(w) = \frac{1}{N}\nabla  E_1(w) + \frac{1}{N}\nabla E_2(w) + \dots + \frac{1}{N}\nabla E_N(w) \equiv g_{[N]}(w) = g(w)
$$

* In a gradient descent algorithm, the computational burden will come from calculating the gradient for each data point in a big dataset. Remember that after taking a gradient descent, step the parameter will change and we will need to calculate the gradient again from scratch. If the data set is large, we could approximately calculate the true average gradient as

$$
\frac{1}{N} \nabla E(w) \approx \frac{1}{B}\sum_{i\in \Omega}\nabla  E_i(w) \equiv g_\Omega
$$
Here, $\Omega$ is a random subset of $\{1,\dots,N\}$ with $B$ elements.

* We actually don't need to calculate the true average gradient $\frac{1}{N} \nabla E(w)$ exactly. In the extreme case, by just choosing $B=1$, that is just computing the gradient for a single data point, we get an unbiased but noisy estimate of the true gradient. Taking $B$ larger, reduces the variance of the gradient estimate.

$$
g = \left\langle{g_\Omega}\right\rangle
$$

# Maximum Likelihood Estimation

Suppose we wish to estimate the parameter $\theta$, given conditionally independent observations
$$
p(x| \theta) = \prod_{i=1}^N p(x_i| \theta)
$$
The maximum-likelihood estimator $\theta^*$ maximizes the log likelihood function
\begin{align}
\theta^* = \arg\max_\theta \mathcal{L}(\theta)
\end{align}

The likelihood function can be written as the sum
$$
\mathcal{L}(\theta) = \sum_{i=1}^N \mathcal{L}_i(\theta)
$$

$\mathcal{L}_i(\theta) \equiv \log p(x_i| \theta)$

## Example: Estimating the variance parameter of a Gaussian

Assume for $i=1\dots N$, we have conditionally independent observations
$$
x_i \sim \mathcal{N}(x; 0, \theta)
$$

The likelihood contribution of a single data point is
$$\log p(x_i|\theta) = \mathcal{L}_i =  - \frac{1}{2} \frac{x_i^2}{\theta} -\frac{1}{2}\log 2\pi \theta$$

The total log-likelihood is the sum of individual data terms
$$\log p(x|\theta) = \mathcal{L}(\theta) = \sum_{i=1}^N \mathcal{L}_i =  - \frac{1}{2} \frac{\sum_{i=1}^N x_i^2}{\theta} -\frac{N}{2}\log 2\pi \theta$$

$$\frac{\partial \mathcal{L}(\theta)}{\partial \theta} = g(\theta) = -\frac{N}{2 \theta} + \frac{1}{2} \frac{\sum_i x_i^2}{\theta^2}$$

This problem has a solution in closed form:

\begin{align}                                                                                g(\theta) & = 0 = -\frac{N}{2 \theta} + \frac{1}{2} \frac{\sum_i x_i^2}{\theta^2} \\
                                                                                \theta^* & = \frac{\sum_i x_i^2}{N}
                                                                                \end{align}

## Recursive estimation

Let's denote the estimate at $N$'th step as
\begin{align}
\theta^*_{N} & = \frac{\sum_{i=1}^N x_i^2}{N} \\
N \theta^*_{N} & = \sum_{i=1}^{N-1} x_i^2 + x_{N}^2 \\
\theta^*_{N} & = \frac{1}{N}\sum_{i=1}^{N-1} x_i^2 + \frac{1}{N} x_{N}^2 \\
 & = \frac{N-1}{N} \theta^*_{N-1} + \frac{1}{N} x_{N}^2 \\
& = \theta^*_{N-1} + \frac{1}{N} (x_{N}^2 - \theta^*_{N-1})
\end{align}

This recursion leads to the following algorithm
\begin{align}
\theta^*_{1}  & = x_1^2  \\
\text{for } & \tau=2,3,\dots & \\
&\theta^*_{\tau}  = \theta^*_{\tau-1} + \frac{1}{\tau} (x_{\tau}^2 - \theta^*_{\tau-1})
\end{align}

Generate sample data from the true model

```python
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import scipy as sc
import pandas as pd

mpl.rc('font',**{'size': 20, 'family':'sans-serif','sans-serif':['Helvetica']})
mpl.rc('text', usetex=True)

N = 1000
x = np.random.randn(N)

def g(th, N, s):
    return -0.5/th + 0.5*s/N/th**2

```

# Exact recursive maximum likelihood

```python
th_star = np.mean(x**2)
th = 0
TH = []
for n in range(N):
    th = th + (x[n]**2 - th)/(n+1)
    TH.append(th)

plt.figure(figsize=(12,3))
plt.plot(TH)
plt.axhline(th_star, ls=':')
plt.show()
```

The following panel illustrates the true gradient of the likelihood $\mathcal{L}$ with
estimates obtained on random subsets denoted by $\Omega$. The expectation of the random gradients
is the true gradient

$$
g(\theta) = \left\langle{g_\Omega(\theta)}\right\rangle
$$

```python

from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
from IPython.display import clear_output, display, HTML
from matplotlib import rc

def plot_batch(B=1, NumOfBatches=10):
    th = np.linspace(0.4,2)

    for J in range(NumOfBatches):
        s = np.sum(np.random.choice(x, size=B,replace=False)**2)
        plt.plot(th, g(th, B, s), ls='-', alpha=0.3, color='blue')

    s = np.sum(x**2)
    plt.plot(th, g(th, N, s), ls=':', linewidth=4, color='red',label='true')

    plt.gca().set_ylim([-0.2,1])
    plt.grid()

    plt.xlabel('$\\theta$/Parameter')
    plt.ylabel('$g$/Gradient Estimate')
    #plt.legend(['$g_\Omega$','$g(\\theta)$'])
    plt.title('True gradient versus mini-batch')
    plt.show()

res = interact(plot_batch, NumOfBatches=(10,400,10), B=(1,500,10))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

# SGD

* Local search algorithms have the form :
\begin{align}
w_0 & = \text{some initial value} \\
\text{for}\;\; & \tau = 1, 2,\dots \\
& w_\tau = w_{\tau-1} + \eta_\tau s_\tau
\end{align}

$\eta_\tau$ is a learning rate and $s_\tau$ is a search direction.

At each iteration $\tau$, minibatch SGD selects random $i$ from $1\dots N$. Set the search direction
$$
s_\tau = \nabla\mathcal{L}_i(\theta_{\tau-1})
$$

# Minibatch SGD

At each iteration $\tau$, minibatch SGD selects a random subset $\Omega_\tau$ of size $B_\tau$ of $1\dots N$. Set the search direction
$$
s_\tau = \frac{1}{B_\tau}\sum_{i\in \Omega_\tau} \nabla\mathcal{L}_i(\theta_{\tau-1})
$$

When the batch size is one ($B=1$), the minibatch SGD algorithm is SGD.

\begin{align}
g_\Omega(\theta)/B & = -\frac{1}{2 \theta} + \frac{1}{2} \frac{\sum_{i\in \Omega} x_i^2}{B\theta^2} \\
& = \frac{1}{2\theta^2}\left(-\theta + \frac{1}{B}\sum_{i\in \Omega} x_i^2\right)
\end{align}

```python

th_true = np.sum(x**2)/N

def plot_path(B=1, eta=1., seed=0):
    np.random.seed(seed)
    theta = 1.5

    MAX_ITER = 1000

    TH = []
    TH.append(theta)
    for tau in range(MAX_ITER):
        eta_tau = eta/(tau+1)
        x_omega = np.random.choice(x, size=B)
        theta = theta + eta_tau*(-0.5/theta + 0.5*np.sum(x_omega**2)/B/theta**2 )
        TH.append(theta)

    TH = np.array(TH)
    plt.figure(figsize=(12,3))
    plt.plot(TH)
    plt.axhline(th_true, ls=':',color='r')
    plt.ylabel('$\\theta_\\tau$/Estimate at $\\tau$')
    plt.xlabel('$\\tau$/Iteration')

    plt.ylim([0.75, 1.5])

    plt.show()

res = interact(plot_path, B=(1,100), eta=(0.01,5,0.01), seed=(0,100))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

A model for outliers

$$
x_i \sim \frac{1}{2}\mathcal{N}(x; 0, \theta) + \frac{1}{2}\mathcal{N}(x; 0, 100)
$$

```python
def log_gauss_pdf(x,mu,v):
    return -0.5*np.log(2*np.pi*v) - 0.5*(x-mu)**2/v

x = np.linspace(-10,10,200)
v = 0.1
v0 = 40.
m = 3.

y = np.log(0.006*np.exp(log_gauss_pdf(x,m,v)) + 0.994*np.exp(log_gauss_pdf(x,0,v0)))

plt.plot(x,y)
plt.show()

```

```python
import torch
import torch.autograd
from torch.autograd import Variable

X_np = np.array([1, 2.5, -3.2, -1.1, 2.2, 2])
X = Variable(torch.from_numpy(X_np).double(), requires_grad=False)
```

```python
# Implementation
v  = Variable(torch.DoubleTensor([3.3]), requires_grad=True)
m0 = Variable(torch.DoubleTensor([0.]), requires_grad=False)
m  = Variable(torch.DoubleTensor([3.]), requires_grad=False)
v0 = Variable(torch.DoubleTensor([40.]), requires_grad=False)
c  = Variable(torch.DoubleTensor([0.006,0.994]), requires_grad=False)

def loglik(X, m, v, m0, v0, c):
    return torch.log(c[0]*torch.exp(log_gauss_pdf(X,m,v))
                     + c[1]*torch.exp(log_gauss_pdf(X,m0,v0)) )

def log_gauss_pdf(x,mu,v):
#    return -torch.log(v) - (x-mu)*(x-mu)/v
    return -0.5*torch.log(2*np.pi*v) - 0.5*torch.pow(x-mu,2)/v

eta = 2
MAX_ITER = 200
V = []
L = []
for epoch in range(MAX_ITER):

    # Compute the loglikelihood
    LL = torch.sum(loglik(X, m, v, m0, v0, c))
    L.append(LL.data)
    # Compute the gradients by automated differentiation
    LL.backward()

    # The gradient ascent step
    v.data.add_(eta*v.grad.data)
    V.append(1*v.data.numpy())

    #print(v.grad.data)
    # Reset the gradients, as otherwise they are accumulated in v.grad
    v.grad.zero_()

print(v.data.numpy())

plt.plot(V)
plt.show()
plt.plot(L)
plt.show()

```

```python

v  = Variable(torch.DoubleTensor([0.05]), requires_grad=True)
m0 = Variable(torch.DoubleTensor([0.]),  requires_grad=False)
m  = Variable(torch.DoubleTensor([3.]),  requires_grad=False)
v0 = Variable(torch.DoubleTensor([40.]), requires_grad=False)
c  = Variable(torch.DoubleTensor([0.006,0.994]), requires_grad=False)

def loglik(X, m, v, m0, v0, c):
    return c[0]*torch.exp(log_gauss_pdf(X,m,v)) + c[1]*torch.exp(log_gauss_pdf(X,m0,v0))

LL = torch.sum(loglik(X, m, v, m0, v0, c))

LL.backward()

print(v.grad.data)

v.grad.zero_()

```

    1.00000e-03 *
      2.2835
    [torch.DoubleTensor of size 1]

    Variable containing:
     0
    [torch.DoubleTensor of size 1]

```python
def fun(v):
    return torch.sin(v)*v**2 + 2*v

v  = Variable(torch.linspace(-3,3,100), requires_grad=True)
LL = torch.sum(fun(v))
LL.backward()

plt.plot(v.data.numpy(), v.grad.data.numpy())
plt.plot(v.data.numpy(), fun(v).data.numpy(), 'r:')

plt.show()

v.grad.zero_()
```

```python
torch.linspace(-3,3,100)
```

     0.9093
    [torch.DoubleTensor of size 1]


# Tutorial - SymPy in 10 minutes

## Introduction

In this tutorial we will learn the basics of [Jupyter](http://jupyter.org/) and [SymPy](https://es.wikipedia.org/wiki/SymPy). SymPy is a Python library for symbolic computation. It provides computer algebra capabilities either as a standalone application, as a library to other applications, or live on the web as [SymPy Live](http://live.sympy.org/) or [SymPy Gamma](http://www.sympygamma.com/). Sympy is similar to other CAS (Computer Algebra Software) like Mathematica, Maple or Maxima.

A more complete tutorial can be found at [http://docs.sympy.org/latest/tutorial/index.html](http://docs.sympy.org/latest/tutorial/index.html).

Before using SymPy we should load it, like any other Python libary. We will use

```python
init_session()
```

to make some imports, this will help us in its interactive use.

For scripting it would be better to do the imports diffferently, for example

```python
import sympy as sym
```

and then call the functions from SymPy in the following manner

```python
x = sym.Symbols("x")
expr = sym.cos(x)**2 + 3*x
deriv = expr.diff(x)
```

where we computed the derivative of $cos^2(x) + 3x$, that should be $-2\sin(x)\cos(x) + 3$.

For further information on the Jupyter Notebook you can refer to the [User Manual](https://athena.brynmawr.edu/jupyter/hub/dblank/public/Jupyter%20Notebook%20Users%20Manual.ipynb).

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
```

```python
init_session()
plt.style.use("seaborn-notebook")
```

    IPython console for SymPy 1.2 (Python 3.6.5-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.2/

Let us start with some simple calculations. Belowe we have a _code cell_ with an addition. Place the cursor on it and press SHIFT + ENTER to evaluate it.

```python
1 + 3
```

Let us make some calculations

```python
factorial(5)
```

```python
Out[4]* 10
```

```python
1 // 3
```

```python
1 / 3
```

```python
S(1) / 3
```

We can evaluate an expression to its floating point version

```python
sqrt(2*pi)
```

```python
float(Out[9])
```

In the previous line we used the expression **Out[8]** that stores the output from the evaluation in cell 8 (In[8]). We can also store expressions as variables, just as any Python variable.

```python
radius = 10
height = 100
area = pi * radius**2
volume = area * height
```

```python
volume
```

```python
float(volume)
```

So far, we have just used SymPy as a calculator. Let us try more advanced calculations

Definite and indefinite integrals

```python
integrate(sin(x), x)
```

```python
integrate(sin(x), (x, 0, pi))
```

We can define a function and integrate it

```python
f = lambda x: x**2 + 5
```

```python
f(5)
```

```python
integrate(f(z), z)
```

```python
integrate(1/(x**2 + y), x)
```

If we assume that the denominator is positive, the expression can be factorized further

```python
a = symbols("a", positive=True)
integrate(1/(x**2 + a), x)
```

We just learnt the basics, we can try some examples now.

**Note:** If you want to know more about a specific function you can use ``help()`` or the IPython magic command ``??``

```python
help(integrate)
```

    Help on function integrate in module sympy.integrals.integrals:

    integrate(*args, **kwargs)
        integrate(f, var, ...)

        Compute definite or indefinite integral of one or more variables
        using Risch-Norman algorithm and table lookup. This procedure is
        able to handle elementary algebraic and transcendental functions
        and also a huge class of special functions, including Airy,
        Bessel, Whittaker and Lambert.

        var can be:

        - a symbol                   -- indefinite integration
        - a tuple (symbol, a)        -- indefinite integration with result
                                        given with `a` replacing `symbol`
        - a tuple (symbol, a, b)     -- definite integration

        Several variables can be specified, in which case the result is
        multiple integration. (If var is omitted and the integrand is
        univariate, the indefinite integral in that variable will be performed.)

        Indefinite integrals are returned without terms that are independent
        of the integration variables. (see examples)

        Definite improper integrals often entail delicate convergence
        conditions. Pass conds='piecewise', 'separate' or 'none' to have
        these returned, respectively, as a Piecewise function, as a separate
        result (i.e. result will be a tuple), or not at all (default is
        'piecewise').

        **Strategy**

        SymPy uses various approaches to definite integration. One method is to
        find an antiderivative for the integrand, and then use the fundamental
        theorem of calculus. Various functions are implemented to integrate
        polynomial, rational and trigonometric functions, and integrands
        containing DiracDelta terms.

        SymPy also implements the part of the Risch algorithm, which is a decision
        procedure for integrating elementary functions, i.e., the algorithm can
        either find an elementary antiderivative, or prove that one does not
        exist.  There is also a (very successful, albeit somewhat slow) general
        implementation of the heuristic Risch algorithm.  This algorithm will
        eventually be phased out as more of the full Risch algorithm is
        implemented. See the docstring of Integral._eval_integral() for more
        details on computing the antiderivative using algebraic methods.

        The option risch=True can be used to use only the (full) Risch algorithm.
        This is useful if you want to know if an elementary function has an
        elementary antiderivative.  If the indefinite Integral returned by this
        function is an instance of NonElementaryIntegral, that means that the
        Risch algorithm has proven that integral to be non-elementary.  Note that
        by default, additional methods (such as the Meijer G method outlined
        below) are tried on these integrals, as they may be expressible in terms
        of special functions, so if you only care about elementary answers, use
        risch=True.  Also note that an unevaluated Integral returned by this
        function is not necessarily a NonElementaryIntegral, even with risch=True,
        as it may just be an indication that the particular part of the Risch
        algorithm needed to integrate that function is not yet implemented.

        Another family of strategies comes from re-writing the integrand in
        terms of so-called Meijer G-functions. Indefinite integrals of a
        single G-function can always be computed, and the definite integral
        of a product of two G-functions can be computed from zero to
        infinity. Various strategies are implemented to rewrite integrands
        as G-functions, and use this information to compute integrals (see
        the ``meijerint`` module).

        The option manual=True can be used to use only an algorithm that tries
        to mimic integration by hand. This algorithm does not handle as many
        integrands as the other algorithms implemented but may return results in
        a more familiar form. The ``manualintegrate`` module has functions that
        return the steps used (see the module docstring for more information).

        In general, the algebraic methods work best for computing
        antiderivatives of (possibly complicated) combinations of elementary
        functions. The G-function methods work best for computing definite
        integrals from zero to infinity of moderately complicated
        combinations of special functions, or indefinite integrals of very
        simple combinations of special functions.

        The strategy employed by the integration code is as follows:

        - If computing a definite integral, and both limits are real,
          and at least one limit is +- oo, try the G-function method of
          definite integration first.

        - Try to find an antiderivative, using all available methods, ordered
          by performance (that is try fastest method first, slowest last; in
          particular polynomial integration is tried first, Meijer
          G-functions second to last, and heuristic Risch last).

        - If still not successful, try G-functions irrespective of the
          limits.

        The option meijerg=True, False, None can be used to, respectively:
        always use G-function methods and no others, never use G-function
        methods, or use all available methods (in order as described above).
        It defaults to None.

        Examples
        ========

        >>> from sympy import integrate, log, exp, oo
        >>> from sympy.abc import a, x, y

        >>> integrate(x*y, x)
        x**2*y/2

        >>> integrate(log(x), x)
        x*log(x) - x

        >>> integrate(log(x), (x, 1, a))
        a*log(a) - a + 1

        >>> integrate(x)
        x**2/2

        Terms that are independent of x are dropped by indefinite integration:

        >>> from sympy import sqrt
        >>> integrate(sqrt(1 + x), (x, 0, x))
        2*(x + 1)**(3/2)/3 - 2/3
        >>> integrate(sqrt(1 + x), x)
        2*(x + 1)**(3/2)/3

        >>> integrate(x*y)
        Traceback (most recent call last):
        ...
        ValueError: specify integration variables to integrate x*y

        Note that ``integrate(x)`` syntax is meant only for convenience
        in interactive sessions and should be avoided in library code.

        >>> integrate(x**a*exp(-x), (x, 0, oo)) # same as conds='piecewise'
        Piecewise((gamma(a + 1), -re(a) < 1),
            (Integral(x**a*exp(-x), (x, 0, oo)), True))

        >>> integrate(x**a*exp(-x), (x, 0, oo), conds='none')
        gamma(a + 1)

        >>> integrate(x**a*exp(-x), (x, 0, oo), conds='separate')
        (gamma(a + 1), -re(a) < 1)

        See Also
        ========

        Integral, Integral.doit

```python
integrate??
```

## Examples

### Solving algebraic equations

Right now, there are two main options for solution of algebraic systems, namely:
[``solveset`` and ``solve``](http://docs.sympy.org/latest/tutorial/solvers.html).
The preferred method is ``solveset`` (see this
[explanation](http://docs.sympy.org/latest/modules/solvers/solveset.html), although
there are systems that can be solved using ``solve`` and not ``solveset``.

To solve equations using ``solveset``:

```python
a, b, c = symbols("a b c")
solveset(a*x**2 + b*x + c, x)
```

We should enter the equations as equated to 0, or as an equation

```python
solveset(Eq(a*x**2 + b*x, -c), x)
```

Currently solveset is not capable of solving the following types of equations:

- Non-linear multivariate system
- Equations solvable by LambertW (Transcendental equation solver).

solve can be used for such cases:

```python
solve([x*y - 1, x - 2], x, y)
```

```python
solve(x*exp(x) - 1, x )
```

### Linear Algebra

We use ``Matrix`` to create matrices. Matrices can contain expressions. And we use the method ``.inv()`` to compute the inverse and ``*`` to multiply matrices.

```python
A = Matrix([
        [1, -1],
        [1, sin(c)]
    ])
display(A)
```

$$\left[\begin{matrix}1 & -1\\1 & \sin{\left (c \right )}\end{matrix}\right]$$

```python
B = A.inv()
display(B)
```

$$\left[\begin{matrix}\frac{\sin{\left (c \right )}}{\sin{\left (c \right )} + 1} & \frac{1}{\sin{\left (c \right )} + 1}\\- \frac{1}{\sin{\left (c \right )} + 1} & \frac{1}{\sin{\left (c \right )} + 1}\end{matrix}\right]$$

```python
A * B
```

$$\left[\begin{matrix}\frac{\sin{\left (c \right )}}{\sin{\left (c \right )} + 1} + \frac{1}{\sin{\left (c \right )} + 1} & 0\\0 & \frac{\sin{\left (c \right )}}{\sin{\left (c \right )} + 1} + \frac{1}{\sin{\left (c \right )} + 1}\end{matrix}\right]$$

This should be the identity matrix, let us simplify the expression. There are several simplification functions, and ``simplify`` is the most general one. Simplifying is a complicated matter... if you are uncertain; use ``simplify``.

```python
simplify(A * B)
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

### Plotting

We can make 2D and 3D plots

```python
%matplotlib inline
from sympy.plotting import plot3d
```

```python
plot(sin(x), (x, -pi, pi));
```

```python
monkey_saddle = x**3 - 3*x*y**2
p = plot3d(monkey_saddle, (x, -2, 2), (y, -2, 2))
```

### Derivatives and differential equations

We can use the function ``diff`` or the method ``.diff()`` to compute derivatives.

```python
f = lambda x: x**2
```

```python
diff(f(x), x)
```

```python
f(x).diff(x)
```

```python
g = lambda x: sin(x)
```

```python
diff(g(f(x)), x)
```

Yes, SymPy knows about the chain rule.

To finish, let us solve a second order ODE

$$ u''(t) + \omega^2 u(t) = 0$$

```python
u = symbols("u", cls=Function)
omega = symbols("omega", positive=True)
```

```python
ode = u(t).diff(t, 2) + omega**2 * u(t)
dsolve(ode, u(t))
```

## Turning Sympy expression into evaluable functions

``lambdify`` provides convenient functions to transform sympy expressions to lambda functions which can be used to calculate numerical values very fast.

Let's try a first example

```python
f = lambdify(x, x**2, "numpy")
f(3)
```

```python
f(np.array([1, 2, 3]))
```

    array([1, 4, 9])

We can try a more difficult one

```python
fun = diff(sin(x)*cos(x**3) - sin(x)/x, x)
fun
```

```python
fun_numpy = lambdify(x, fun, "numpy")
```

and then evaluate it for some points in, let's say, $[0, 5]$

```python
pts = np.linspace(0, 5, 1000)
fun_pts = fun_numpy(pts + 1e-6) # To avoid division by 0
```

```python
plt.plot(pts, fun_pts);
```

## References

- Žiga Lenarčič. ["10 minute (wx)Maxima tutorial:"](https://andrejv.github.io/wxmaxima/tutorials/10minute.zip), (2008). Accessed on: July 25, 2017.
- Borland, David, and Russell M. Taylor II. ["Rainbow color map (still) considered harmful."](https://data3.mprog.nl/course/15%20Readings/40%20Reading%204/Borland_Rainbow_Color_Map.pdf) IEEE computer graphics and applications 27.2 (2007): 14-17.

The following cell change the style of the notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```