# Linear models for classification problems

Given a training set of $N$ samples, $D = \{(\boldsymbol{x_1} , y_1 ), \ldots , (\boldsymbol{x_N} , y_N )\}$ , where $\boldsymbol{x_i}$ is a multidimensional input vector with dimension $P$ and class label (target or response).

Multiclass Classification problems can be seen as several binary classification problems $y_i \in \{0, 1\}$ where the classifier aims to discriminate the sample of the current class (label 1) versus the samples of other classes (label 0).

Therfore, for each class the classifier seek for a vector of parameters $\boldsymbol{w}$ that performs a linear combination of the input variables, $\boldsymbol{x}^T \boldsymbol{w}$. This step performs a **projection** or a **rotation** of input sample into a good discriminative one-dimensional sub-space, that best discriminate sample of current class vs sample of other classes.

This score (a.k.a decision function) is tranformed, using the nonlinear activation funtion $f(.)$,  to a "posterior probabilities" of class 1: $p(y=1|\boldsymbol{x}) = f(\boldsymbol{x}^T \boldsymbol{w})$, where, $p(y=1|\boldsymbol{x}) = 1 - p(y=0|\boldsymbol{x})$.

The decision surfaces (orthogonal hyperplan to $\boldsymbol{w}$) correspond to $f(x)=\text{constant}$, so that $\boldsymbol{x}^T \boldsymbol{w}=\text{constant}$ and hence the decision surfaces are linear functions of $\boldsymbol{x}$, even if the function $f(.)$ is nonlinear.

A thresholding of the activation (shifted by the bias or intercept) provides the predicted class label.

The vector of parameters, that defines the discriminative axis, minimizes an **objective function** $J(\boldsymbol{w})$ that is a sum of of **loss function** $L(\boldsymbol{w})$ and some penalties on the weights vector $\Omega(\boldsymbol{w})$.
$$
\min_{\boldsymbol{w}}~J = \sum_i L(y_i, f(\boldsymbol{x_i}^T\boldsymbol{w})) + \Omega(\boldsymbol{w}),
$$

## Fisher's linear discriminant with equal class covariance

This geometric method does not make any probabilistic assumptions, instead it relies on distances. It looks for the **linear projection** of the data points onto a vector, $\boldsymbol{w}$, that maximizes the between/within variance ratio, denoted $F(\boldsymbol{w})$. Under a few assumptions, it will provide the same results as linear discriminant analysis (LDA), explained below.

Suppose two classes of observations, $C_0$ and $C_1$, have means $\boldsymbol{\mu_0}$ and $\boldsymbol{\mu_1}$ and the same total within-class scatter ("covariance") matrix,

\begin{align}
    \boldsymbol{S_W} &= \sum_{i\in C_0} (\boldsymbol{x_i} - \boldsymbol{\mu_0})(\boldsymbol{x_i} - \boldsymbol{\mu_0})^T + \sum_{j\in C_1} (\boldsymbol{x_j} - \boldsymbol{\mu_1})(\boldsymbol{x_j} -\boldsymbol{\mu_1})^T\\
        &= \boldsymbol{X_c}^T \boldsymbol{X_c},
\end{align}

where $\boldsymbol{X_c}$ is the $(N \times P)$ matrix of data centered on their respective means:

$$
\boldsymbol{X_c} = \begin{bmatrix}
          \boldsymbol{X_0} -  \boldsymbol{\mu_0} \\
          \boldsymbol{X_1} -  \boldsymbol{\mu_1}
      \end{bmatrix},
$$

where $\boldsymbol{X_0}$ and $\boldsymbol{X_1}$ are the $(N_0 \times P)$ and $(N_1 \times P)$ matrices of samples of classes $C_0$ and $C_1$.

Let $\boldsymbol{S_B}$ being the scatter "between-class" matrix, given by

$$
    \boldsymbol{S_B} = (\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )^T.
$$

The linear combination of features $\boldsymbol{w}^T x$ have means $\boldsymbol{w}^T \mu_i$ for $i=0,1$, and variance $\boldsymbol{w}^T
\boldsymbol{X^T_c} \boldsymbol{X_c} \boldsymbol{w}$. Fisher defined the separation between these two distributions to be the ratio of the
variance between the classes to the variance within the classes:

\begin{align}
F_{\text{Fisher}}(\boldsymbol{w}) &= \frac{\sigma_{\text{between}}^2}{\sigma_{\text{within}}^2}\\
                     &= \frac{(\boldsymbol{w}^T \boldsymbol{\mu_1} - \boldsymbol{w}^T \boldsymbol{\mu_0})^2}{\boldsymbol{w}^T  X^T_c \boldsymbol{X_c} \boldsymbol{w}}\\
                     &= \frac{(\boldsymbol{w}^T (\boldsymbol{\mu_1} - \boldsymbol{\mu_0}))^2}{\boldsymbol{w}^T  X^T_c \boldsymbol{X_c} \boldsymbol{w}}\\
                     &= \frac{\boldsymbol{w}^T (\boldsymbol{\mu_1} - \boldsymbol{\mu_0}) (\boldsymbol{\mu_1} - \boldsymbol{\mu_0})^T w}{\boldsymbol{w}^T X^T_c \boldsymbol{X_c} \boldsymbol{w}}\\
                     &= \frac{\boldsymbol{w}^T \boldsymbol{S_B} w}{\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w}}.
\end{align}

### The Fisher most discriminant projection

In the two-class case, the maximum separation occurs by a projection on the $(\boldsymbol{\mu_1} - \boldsymbol{\mu_0})$ using the Mahalanobis
metric $\boldsymbol{S_W}^{-1}$, so that

$$
    \boldsymbol{w} \propto \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0}).
$$

#### Demonstration

Differentiating $F_{\text{Fisher}}(w)$ with respect to $w$ gives

\begin{align*}
    \nabla_{\boldsymbol{w}}F_{\text{Fisher}}(\boldsymbol{w}) &= 0\\
    \nabla_{\boldsymbol{w}}\left(\frac{\boldsymbol{w}^T \boldsymbol{S_B} w}{\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w}}\right) &= 0\\
    (\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w})(2 \boldsymbol{S_B} \boldsymbol{w}) - (\boldsymbol{w}^T \boldsymbol{S_B} \boldsymbol{w})(2 \boldsymbol{S_W} \boldsymbol{w}) &= 0\\
    (\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w})(\boldsymbol{S_B} \boldsymbol{w}) &= (\boldsymbol{w}^T \boldsymbol{S_B} \boldsymbol{w})(\boldsymbol{S_W} \boldsymbol{w})\\
    \boldsymbol{S_B} \boldsymbol{w} &= \frac{\boldsymbol{w}^T \boldsymbol{S_B} \boldsymbol{w}}{\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w}}(\boldsymbol{S_W} \boldsymbol{w})\\
    \boldsymbol{S_B} \boldsymbol{w} &= \lambda (\boldsymbol{S_W} \boldsymbol{w})\\
    \boldsymbol{S_W}^{-1}{\boldsymbol{S_B}} \boldsymbol{w} &= \lambda  \boldsymbol{w}.
\end{align*}

Since we do not care about the magnitude of $\boldsymbol{w}$, only its direction, we replaced the scalar factor $(\boldsymbol{w}^T \boldsymbol{S_B} \boldsymbol{w}) / (\boldsymbol{w}^T \boldsymbol{S_W} \boldsymbol{w})$ by $\lambda$.

In the multiple-class case, the solutions $w$ are determined by the eigenvectors of $\boldsymbol{S_W}^{-1}{\boldsymbol{S_B}}$ that correspond to the $K-1$ largest eigenvalues.

However, in the two-class case (in which $\boldsymbol{S_B} = (\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )^T$) it is easy to show that $\boldsymbol{w} = \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0})$ is the unique eigenvector of $\boldsymbol{S_W}^{-1}{\boldsymbol{S_B}}$:

\begin{align*}
    \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )^T \boldsymbol{w} &= \lambda  \boldsymbol{w}\\
    \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )(\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )^T \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0}) &= \lambda  \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0}),
\end{align*}

where here $\lambda = (\boldsymbol{\mu_1} - \boldsymbol{\mu_0} )^T \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0})$. Which leads to the result

$$
\boldsymbol{w} \propto \boldsymbol{S_W}^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0}).
$$

### The separating hyperplane

The separating hyperplane is a $P-1$-dimensional hyper surface, orthogonal to the projection vector, $w$. There is no single best way to find the origin of the plane along $w$, or equivalently the classification threshold that determines whether a point should be classified as belonging to $C_0$ or to $C_1$. However, if the projected points have roughly the same distribution, then the threshold can be chosen as the hyperplane exactly between the projections of the two means, i.e. as

$$
T = \boldsymbol{w} \cdot \frac{1}{2}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0}).
$$

```python
%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn import datasets
import sklearn.linear_model as lm
import sklearn.metrics as metrics

np.set_printoptions(precision=2)
pd.set_option('precision', 2)
```

## Linear discriminant analysis (LDA)

Linear discriminant analysis (LDA) is a probabilistic generalization of Fisher's linear discriminant. It uses Bayes' rule to fix the threshold based on prior probabilities of classes.

1. First compute the class-**conditional distributions** of $\boldsymbol{x}$ given class $C_k$: $p(x|C_k) = \mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu_k}, \boldsymbol{S_W})$. Where $\mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu_k}, \boldsymbol{S_W})$ is the multivariate Gaussian distribution defined over a P-dimensional vector $x$ of continuous variables, which is given by

$$
\mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu_k}, \boldsymbol{S_W}) = \frac{1}{(2\pi)^{P/2}|\boldsymbol{S_W}|^{1/2}}\exp\{-\frac{1}{2} (\boldsymbol{x} - \boldsymbol{\mu_k})^T \boldsymbol{S_W}^{-1}(x - \boldsymbol{\mu_k})\}
$$

2. Estimate the **prior probabilities** of class $k$, $p(C_k) = N_k/N$.

3. Compute **posterior probabilities** (ie. the probability of a each class given a sample) combining conditional with priors using Bayes' rule:

$$
p(C_k|\boldsymbol{x}) = \frac{p(C_k) p(\boldsymbol{x}|C_k)}{p(\boldsymbol{x})}
$$

Where $p(x)$ is the marginal distribution obtained by suming of classes:
As usual, the denominator
in Bayes’ theorem can be found in terms of the quantities appearing in the
numerator, because

$$
p(x) = \sum_k p(\boldsymbol{x}|C_k)p(C_k)
$$

4. Classify $\boldsymbol{x}$ using the Maximum-a-Posteriori probability: $C_k= \arg \max_{C_k} p(C_k|\boldsymbol{x})$

LDA is a **generative model** since the class-conditional distributions cal be used to generate samples of each classes.

LDA is useful to deal with imbalanced group sizes (eg.: $N_1 \gg N_0$) since priors probabilities can be used to explicitly re-balance the classification by setting $p(C_0) = p(C_1) = 1/2$ or whatever seems relevant.

LDA can be generalised to the multiclass case with $K>2$.

With  $N_1 = N_0$, LDA lead to the same solution than Fisher's linear discriminant.

### Exercise

How many parameters are required to estimate to perform a LDA ?

```python
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# Dataset 2 two multivariate normal
n_samples, n_features = 100, 2
mean0, mean1 = np.array([0, 0]), np.array([0, 2])
Cov = np.array([[1, .8],[.8, 1]])
np.random.seed(42)
X0 = np.random.multivariate_normal(mean0, Cov, n_samples)
X1 = np.random.multivariate_normal(mean1, Cov, n_samples)
X = np.vstack([X0, X1])
y = np.array([0] * X0.shape[0] + [1] * X1.shape[0])

# LDA with scikit-learn
lda = LDA()
proj = lda.fit(X, y).transform(X)
y_pred_lda = lda.predict(X)

errors =  y_pred_lda != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y_pred_lda)))
```

## Logistic regression

Logistic regression is called a generalized linear models. ie.: it is a linear model with a link function that maps the output of linear multiple regression to the posterior probability of class $1$ $p(1|x)$ using the logistic sigmoid function:

$$
p(1|\boldsymbol{w, x_i}) = \frac{1}{1 + \exp(-\boldsymbol{w} \cdot \boldsymbol{x_i})}
$$

```python
def logistic(x): return 1 / (1 + np.exp(-x))

x = np.linspace(-6, 6, 100)
plt.plot(x, logistic(x))
plt.grid(True)
plt.title('Logistic (sigmoid)')
```

Logistic regression is a **discriminative model** since it focuses only on the posterior probability of each class $p(C_k|x)$. It only requires to estimate the $P$ weights of the $\boldsymbol{w}$ vector. Thus it should be favoured over LDA with many input features. In small dimension and balanced situations it would provide similar predictions than LDA.

However imbalanced group sizes cannot be explicitly controlled. It can be managed using a reweighting of the input samples.

```python
logreg = lm.LogisticRegression(penalty='none').fit(X, y)
# This class implements regularized logistic regression.
# C is the Inverse of regularization strength.
# Large value => no regularization.

logreg.fit(X, y)
y_pred_logreg = logreg.predict(X)

errors =  y_pred_logreg != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y_pred_logreg)))
print(logreg.coef_)
```

### Exercise

Explore the ``Logistic Regression`` parameters and proposes a solution in cases of highly imbalanced training dataset $N_1 \gg N_0$  when we know that in reality both classes have the same probability $p(C_1) = p(C_0)$.

## Losses

### Negative log likelihood or cross-entropy

The **Loss function** for sample $i$ is the negative log of the probability:
$$
L(\boldsymbol{w, x_i}, y_i) = \begin{cases}
        -\log(p(1|w, \boldsymbol{x_i}))  & \text{if } y_i = 1
        \\
        -\log(1 - p(1|w, \boldsymbol{x_i})  & \text{if } y_i = 0
        \end{cases}
$$

For the whole dataset $\boldsymbol{X}, \boldsymbol{y} = \{\boldsymbol{x_i}, y_i\}$ the loss function to minimize $L(\boldsymbol{w, X, y})$ is the negative negative log likelihood (nll) that can be simplied using a 0/1 coding of the label in the case of binary classification:

\begin{align}
L(\boldsymbol{w, X, y}) &= -\log \mathcal{L}(\boldsymbol{w, X, y}) \\
&= -\log \Pi_i\{p(1|\boldsymbol{w}, \boldsymbol{x_i})^{y_i} (1 - p(1|\boldsymbol{w}, \boldsymbol{x_i})^{(1 - y_i)}\}\\
&=  \sum_i\{y_i \log p(1|\boldsymbol{w}, \boldsymbol{x_i}) + (1 - y_i) \log(1 - p(1|\boldsymbol{w}, \boldsymbol{x_i}))\},
\end{align}

This is known as the **cross-entropy** between the true label $y$ and the predicted probability $p$.

For the logistic regression case, we have:
$$
L(\boldsymbol{w, X, y}) = \sum_i\{y_i \boldsymbol{w \cdot x_i} - \log(1 + \exp(\boldsymbol{w \cdot x_i}))\}
$$

This is solved by numerical method using the gradient of the loss:
$$
\partial\frac{L(\boldsymbol{w, X, y})}{\partial\boldsymbol{w}} = \sum_i \boldsymbol{x_i} (y_i - p(1|\boldsymbol{w}, \boldsymbol{x_i}))
$$

See also [Scikit learn doc](https://scikit-learn.org/stable/modules/sgd.html#mathematical-formulation)

### Hinge loss or $\ell_1$ loss

TODO

## Overfitting

VC dimension (for Vapnik–Chervonenkis dimension) is a measure of the **capacity** (complexity, expressive power, richness, or flexibility) of a statistical classification algorithm, defined as the cardinality of the largest set of points that the algorithm can shatter.

Theorem: Linear classifier in $R^P$ have VC dimension of $P+1$. Hence in dimension two ($P=2$) any random partition of 3 points can be learned.

## Regularization using penalization of coefficients

The penalties use in regression are also used in classification. The only difference is the loss function generally the negative log likelihood (cross-entropy) or the hinge loss. We will explore:

- Ridge (also called $\ell_2$) penalty: $\|\mathbf{w}\|_2^2$. It shrinks coefficients toward 0.
- Lasso (also called $\ell_1$) penalty: $\|\mathbf{w}\|_1$. It performs feature selection by setting some coefficients to 0.
- ElasticNet (also called $\ell_1\ell_2$) penalty: $\alpha \left(\rho~\|\mathbf{w}\|_1 + (1-\rho)~\|\mathbf{w}\|_2^2 \right)$. It performs selection of group of correlated features by setting some coefficients to 0.

```python
# Dataset with some correlation
X, y = datasets.make_classification(n_samples=100, n_features=10,
                                          n_informative=5, n_redundant=3,
                                          n_classes=2, random_state=3, shuffle=False)

lr = lm.LogisticRegression(penalty='none').fit(X, y)

l2 = lm.LogisticRegression(penalty='l2', C=.1).fit(X, y)  # lambda = 1 / C!

# use solver 'saga' to handle L1 penalty
l1 = lm.LogisticRegression(penalty='l1', C=.1, solver='saga').fit(X, y)  # lambda = 1 / C!

l1l2 = lm.LogisticRegression(penalty='elasticnet',  C=.1, l1_ratio=0.5, solver='saga').fit(X, y)  # lambda = 1 / C!

pd.DataFrame(np.vstack((lr.coef_, l2.coef_, l1.coef_, l1l2.coef_)),
             index=['lr', 'l2', 'l1', 'l1l2'])
```

## Ridge Fisher's linear classification ($\ell_2$-regularization)

When the matrix $\boldsymbol{S_W}$ is not full rank or $P \gg N$, the The Fisher most discriminant projection estimate of the is not unique. This can be solved using a biased version of $\boldsymbol{S_W}$:
$$
\boldsymbol{S_W}^{Ridge} = \boldsymbol{S_W} + \lambda \boldsymbol{I}
$$

where $I$ is the $P \times P$ identity matrix. This leads to the regularized (ridge) estimator of the Fisher's linear discriminant analysis:
$$
    \boldsymbol{w}^{Ridge} \propto (\boldsymbol{S_W} + \lambda \boldsymbol{I})^{-1}(\boldsymbol{\mu_1} - \boldsymbol{\mu_0})
$$

Increasing $\lambda$ will:

- Shrinks the coefficients toward zero.
- The covariance will converge toward the diagonal matrix, reducing the contribution of the pairwise covariances.

## Ridge logistic regression ($\ell_2$-regularization)

The **objective function** to be minimized is now the combination of the logistic loss (negative log likelyhood) $-\log \mathcal{L}(\boldsymbol{w})$ with a penalty of the L2 norm of the weights vector. In the two-class case, using the 0/1 coding we obtain:

$$
\min_{\boldsymbol{w}}~\text{Logistic ridge}(\boldsymbol{w}) = -\log \mathcal{L}(\boldsymbol{w, X, y}) + \lambda~\|\boldsymbol{w}\|^2
$$

```python
from sklearn import linear_model
lrl2 = linear_model.LogisticRegression(penalty='l2', C=.1)
# This class implements regularized logistic regression. C is the Inverse of regularization strength.
# Large value => no regularization.

lrl2.fit(X, y)
y_pred_l2 = lrl2.predict(X)
prob_pred_l2 = lrl2.predict_proba(X)

print("Probas of 5 first samples for class 0 and class 1:")
print(prob_pred_l2[:5, :])

print("Coef vector:")
print(lrl2.coef_)

# Retrieve proba from coef vector
probas = 1 / (1 + np.exp(- (np.dot(X, lrl2.coef_.T) + lrl2.intercept_))).ravel()
print("Diff", np.max(np.abs(prob_pred_l2[:, 1] - probas)))

errors =  y_pred_l2 != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y)))
```

## Lasso logistic regression ($\ell_1$-regularization)

The **objective function** to be minimized is now the combination of the logistic loss $-\log \mathcal{L}(\boldsymbol{w})$ with a penalty of the L1 norm of the weights vector. In the two-class case, using the 0/1 coding we obtain:

$$
\min_{\boldsymbol{w}}~\text{Logistic Lasso}(w) = -\log \mathcal{L}(\boldsymbol{w, X, y}) + \lambda~\|\boldsymbol{w}\|_1
$$

```python
from sklearn import linear_model
lrl1 = lm.LogisticRegression(penalty='l1', C=.1, solver='saga') # lambda = 1 / C!

# This class implements regularized logistic regression. C is the Inverse of regularization strength.
# Large value => no regularization.

lrl1.fit(X, y)
y_pred_lrl1 = lrl1.predict(X)

errors =  y_pred_lrl1 != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y_pred_lrl1)))

print("Coef vector:")
print(lrl1.coef_)
```

## Ridge linear Support Vector Machine ($\ell_2$-regularization)

Support Vector Machine seek for separating hyperplane with maximum margin to enforce robustness against noise. Like logistic regression it is a **discriminative method** that only focuses of predictions.

Here we present the non separable case of Maximum Margin Classifiers with $\pm 1$ coding (ie.: $y_i \ \{-1, +1\}$). In the next figure the legend aply to samples of "dot" class.

Linear SVM for classification (also called SVM-C or SVC) minimizes:

$$
\begin{array}{lll}
\text{min}   & \text{Linear SVM}(\boldsymbol{w}) &= \text{penalty}(w) +  C~\text{Hinge loss}(w)\\
             & & = \|w\|_2^2 + C~\sum_i^N\xi_i\\
\text{with}  & \forall i & y_i (w \cdot \boldsymbol{x_i}) \geq 1 - \xi_i
\end{array}
$$

Here we introduced the slack variables: $\xi_i$, with $\xi_i = 0$ for points that are on or inside the correct margin boundary and $\xi_i = |y_i - (w \ cdot  \cdot \boldsymbol{x_i})|$ for other points. Thus:

1. If $y_i (w \cdot \boldsymbol{x_i}) \geq 1$ then the point lies outside the margin but on the correct side of the decision boundary. In this case $\xi_i=0$. The constraint is thus not active for this point. It does not contribute to the prediction.

2. If $1 > y_i (w \cdot \boldsymbol{x_i}) \geq 0$ then the point lies inside the margin and on the correct side of the decision boundary. In this case $0<\xi_i \leq 1$. The constraint is active for this point. It does contribute to the prediction as a support vector.

3. If $0 <  y_i (w \cdot \boldsymbol{x_i})$) then the point is on the wrong side of the decision boundary (missclassification). In this case $0<\xi_i > 1$. The constraint is active for this point. It does contribute to the prediction as a support vector.

This loss is called the hinge loss, defined as:

$$
\max(0, 1 - y_i~ (w \cdot \boldsymbol{x_i}))
$$

So linear SVM is closed to Ridge logistic regression, using the hinge loss instead of the logistic loss. Both will provide very similar predictions.

```python
from sklearn import svm

svmlin = svm.LinearSVC(C=.1)
# Remark: by default LinearSVC uses squared_hinge as loss
svmlin.fit(X, y)
y_pred_svmlin = svmlin.predict(X)

errors =  y_pred_svmlin != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y_pred_svmlin)))
print("Coef vector:")
print(svmlin.coef_)
```

## Lasso linear Support Vector Machine ($\ell_1$-regularization)

Linear SVM for classification (also called SVM-C or SVC) with l1-regularization

$$
\begin{array}{lll}
\text{min}   & F_{\text{Lasso linear SVM}}(w) &= ||w||_1 + C~\sum_i^N\xi_i\\
\text{with}  & \forall i & y_i (w \cdot \boldsymbol{x_i}) \geq 1 - \xi_i
\end{array}
$$

```python
from sklearn import svm

svmlinl1 = svm.LinearSVC(penalty='l1', dual=False)
# Remark: by default LinearSVC uses squared_hinge as loss

svmlinl1.fit(X, y)
y_pred_svmlinl1 = svmlinl1.predict(X)

errors =  y_pred_svmlinl1 != y
print("Nb errors=%i, error rate=%.2f" % (errors.sum(), errors.sum() / len(y_pred_svmlinl1)))
print("Coef vector:")
print(svmlinl1.coef_)
```

## Exercise

Compare predictions of Logistic regression (LR) and their SVM counterparts, ie.: L2 LR vs L2 SVM and L1 LR vs L1 SVM

- Compute the correlation between pairs of weights vectors.

- Compare the predictions of two classifiers using their decision function:

    * Give the equation of the decision function for a linear classifier, assuming that their is no intercept.
    * Compute the correlation decision function.
    * Plot the pairwise decision function of the classifiers.

- Conclude on the differences between Linear SVM and logistic regression.

## Elastic-net classification ($\ell_1\ell_2$-regularization)

The **objective function** to be minimized is now the combination of the logistic loss $\log L(\boldsymbol{w})$ or the hinge loss with combination of L1 and L2 penalties. In the two-class case, using the 0/1 coding we obtain:

\begin{align}
\min~\text{Logistic enet}(\boldsymbol{w}) &= -\log \mathcal{L}(\boldsymbol{w, X, y}) + \alpha~\left(\rho~\|\boldsymbol{w}\|_1 + (1-\rho)~\|\boldsymbol{w}\|_2^2 \right)\\
\min~\text{Hinge enet}(\boldsymbol{w}) &= \text{Hinge loss}(\boldsymbol{w}) + \alpha~\left(\rho~\|\boldsymbol{w}\|_1 + (1-\rho)~\|\boldsymbol{w}\|_2^2 \right)
\end{align}

```python
# Use SGD solver
enetlog = lm.SGDClassifier(loss="log", penalty="elasticnet",
                            alpha=0.1, l1_ratio=0.5, random_state=42)
enetlog.fit(X, y)

# Or saga solver:
# enetloglike = lm.LogisticRegression(penalty='elasticnet',
#                                    C=.1, l1_ratio=0.5, solver='saga')

enethinge = lm.SGDClassifier(loss="hinge", penalty="elasticnet",
                            alpha=0.1, l1_ratio=0.5, random_state=42)
enethinge.fit(X, y)

print("Hinge loss and logistic loss provide almost the same predictions.")
print("Confusion matrix")
metrics.confusion_matrix(enetlog.predict(X), enethinge.predict(X))

print("Decision_function log x hinge losses:")
_ = plt.plot(enetlog.decision_function(X),
             enethinge.decision_function(X), "o")
```

## Classification performance evaluation metrics

source: https://en.wikipedia.org/wiki/Sensitivity_and_specificity

Imagine a study evaluating a new test that screens people for a disease. Each person taking the test either has or does not have the disease. The test outcome can be positive (classifying the person as having the disease) or negative (classifying the person as not having the disease). The test results for each subject may or may not match the subject's actual status. In that setting:

- True positive (TP): Sick people correctly identified as sick

- False positive (FP): Healthy people incorrectly identified as sick

- True negative (TN): Healthy people correctly identified as healthy

- False negative (FN): Sick people incorrectly identified as healthy

- **Accuracy** (ACC):

    ACC = (TP + TN) / (TP + FP + FN + TN)

- **Sensitivity** (SEN) or **recall** of the positive class or true positive rate (TPR) or hit rate:

    SEN = TP / P = TP / (TP+FN)

- **Specificity** (SPC) or **recall** of the negative class or true negative rate:

    SPC = TN / N = TN / (TN+FP)

- **Precision** or positive predictive value (PPV):

    PPV = TP / (TP + FP)

- **Balanced accuracy** (bACC):is a useful performance measure is the balanced accuracy which avoids inflated performance estimates on imbalanced datasets (Brodersen, et al. (2010). "The balanced accuracy and its posterior distribution"). It is defined as the arithmetic mean of sensitivity and specificity, or the average accuracy obtained on either class:

    bACC = 1/2 * (SEN + SPC)

- F1 Score (or F-score) which is a weighted average of precision and recall are usefull to deal with imballaced datasets

The four outcomes can be formulated in a 2×2 contingency table or confusion matrix https://en.wikipedia.org/wiki/Sensitivity_and_specificity

For more precision see: http://scikit-learn.org/stable/modules/model_evaluation.html

```python
from sklearn import metrics
y_pred = [0, 1, 0, 0]
y_true = [0, 1, 0, 1]

metrics.accuracy_score(y_true, y_pred)

# The overall precision an recall
metrics.precision_score(y_true, y_pred)
metrics.recall_score(y_true, y_pred)

# Recalls on individual classes: SEN & SPC
recalls = metrics.recall_score(y_true, y_pred, average=None)
recalls[0] # is the recall of class 0: specificity
recalls[1] # is the recall of class 1: sensitivity

# Balanced accuracy
b_acc = recalls.mean()

# The overall precision an recall on each individual class
p, r, f, s = metrics.precision_recall_fscore_support(y_true, y_pred)
```

### Significance of classification rate

P-value associated to classification rate. Compared the number of correct classifications (=accuracy $\times N$) to the null hypothesis of Binomial distribution of parameters $p$ (typically 50% of chance level) and $N$ (Number of observations).

Is 65% of accuracy a significant prediction rate among 70 observations?

Since this is an exact, **two-sided** test of the null hypothesis, the p-value can be divided by 2 since we test that the accuracy is superior to the chance level.

```python
import scipy.stats

acc, N = 0.65, 70
pval = scipy.stats.binom_test(x=int(acc * N), n=N, p=0.5) / 2
print(pval)
```

### Area Under Curve (AUC) of Receiver operating characteristic (ROC)

Some classifier may have found a good discriminative projection $w$. However if the threshold to decide the final predicted class is poorly adjusted, the performances will highlight an high specificity and a low sensitivity or the contrary.

In this case it is recommended to use the AUC of a ROC analysis which basically provide a measure of overlap of the two classes when points are projected on the discriminative axis. For more detail on ROC and AUC see:https://en.wikipedia.org/wiki/Receiver_operating_characteristic.

```python
score_pred = np.array([.1 ,.2, .3, .4, .5, .6, .7, .8])
y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1])
thres = .9
y_pred = (score_pred > thres).astype(int)

print("With a threshold of %.2f, the rule always predict 0. Predictions:" % thres)
print(y_pred)
metrics.accuracy_score(y_true, y_pred)

# The overall precision an recall on each individual class
r = metrics.recall_score(y_true, y_pred, average=None)
print("Recalls on individual classes are:", r, "ie, 100% of specificity, 0% of sensitivity")

# However AUC=1 indicating a perfect separation of the two classes
auc = metrics.roc_auc_score(y_true, score_pred)
print("But the AUC of %.2f demonstrate a good classes separation." % auc)
```

## Imbalanced classes

Learning with discriminative (logistic regression, SVM) methods is generally based on minimizing the misclassification of training samples, which may be unsuitable for imbalanced datasets where the recognition might be biased in favor of
the most numerous class. This problem can be addressed with a generative approach, which typically requires
more parameters to be determined leading to reduced performances in high dimension.

Dealing with imbalanced class may be addressed by three main ways (see Japkowicz and Stephen (2002) for a review), resampling, reweighting and one class learning.

In **sampling strategies**, either the minority class is oversampled or majority class is undersampled or some combination of the two is deployed. Undersampling (Zhang and Mani, 2003) the majority class would lead to a poor usage of the left-out samples. Sometime one cannot afford such strategy since we are also facing a small sample size problem even for the majority class.
Informed oversampling, which goes beyond a trivial duplication of minority class samples, requires the estimation of class conditional distributions in order to generate synthetic samples. Here generative models are required. An alternative, proposed in (Chawla et al., 2002) generate samples along the line segments joining any/all of the k minority class nearest neighbors. Such procedure blindly generalizes the minority area without regard to the majority class, which may be particularly problematic with high-dimensional and potentially skewed class distribution.

**Reweighting**, also called cost-sensitive learning, works at an algorithmic level by adjusting the costs of the various classes to counter the class imbalance. Such reweighting can be implemented within SVM (Chang and Lin, 2001) or logistic regression (Friedman et al., 2010) classifiers. Most classifiers of Scikit learn offer such reweighting possibilities.

The ``class_weight`` parameter can be positioned into the ``"balanced"`` mode which uses the values of $y$ to automatically adjust weights inversely proportional to class frequencies in the input data as $N / (2 N_k)$.

```python
# dataset
X, y = datasets.make_classification(n_samples=500,
                           n_features=5,
                           n_informative=2,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=2,
                           random_state=1,
                           shuffle=False)

print(*["#samples of class %i = %i;" % (lev, np.sum(y == lev)) for lev in np.unique(y)])

print('# No Reweighting balanced dataset')
lr_inter = linear_model.LogisticRegression(C=1)
lr_inter.fit(X, y)
p, r, f, s = metrics.precision_recall_fscore_support(y, lr_inter.predict(X))
print("SPC: %.3f; SEN: %.3f" % tuple(r))
print('# => The predictions are balanced in sensitivity and specificity\n')

# Create imbalanced dataset, by subsampling sample of class 0: keep only 10% of
# class 0's samples and all class 1's samples.
n0 = int(np.rint(np.sum(y == 0) / 20))
subsample_idx = np.concatenate((np.where(y == 0)[0][:n0], np.where(y == 1)[0]))
Ximb = X[subsample_idx, :]
yimb = y[subsample_idx]
print(*["#samples of class %i = %i;" % (lev, np.sum(yimb == lev)) for lev in
        np.unique(yimb)])

print('# No Reweighting on imbalanced dataset')
lr_inter = linear_model.LogisticRegression(C=1)
lr_inter.fit(Ximb, yimb)
p, r, f, s = metrics.precision_recall_fscore_support(yimb, lr_inter.predict(Ximb))
print("SPC: %.3f; SEN: %.3f" % tuple(r))
print('# => Sensitivity >> specificity\n')

print('# Reweighting on imbalanced dataset')
lr_inter_reweight = linear_model.LogisticRegression(C=1, class_weight="balanced")
lr_inter_reweight.fit(Ximb, yimb)
p, r, f, s = metrics.precision_recall_fscore_support(yimb,
                                                     lr_inter_reweight.predict(Ximb))
print("SPC: %.3f; SEN: %.3f" % tuple(r))
print('# => The predictions are balanced in sensitivity and specificity\n')
```

## Confidence interval cross-validation

Confidence interval CI classification accuracy measured by cross-validation:

## Exercise

### Fisher linear discriminant rule

Write a class ``FisherLinearDiscriminant`` that implements the Fisher's linear discriminant analysis. This class must be compliant with the scikit-learn API by providing two methods:
- ``fit(X, y)`` which fits the model and returns the object itself;
- ``predict(X)`` which returns a vector of the predicted values.
Apply the object on the dataset presented for the LDA.


```python
import numpy as np
import matplotlib.pylab as plt
import pandas as pd
```

# Gompertzian Growth

The Gompertz model is an empirical growth model, based on the observation that population decline can be faster than a decaying exponential, which implies that the death rate must be increasing with age (because a constant death rate would result in exponential decrease).

Mathematically, the Gompertz model can be expressed as a system of two coupled differential equations

\begin{align}
      \frac{dN}{dt} &=G(t)N(t) \tag{1}\\
      \frac{dG}{dt} &=-r G(t) \tag{2}
\end{align}

where
 - the population $N(t)$ grows or shrinks over time with time-dependent growth factor $G(t)$
 - the growth factor $G(t)$ decays exponentially over time with constant rate $r$.

Here, we explore two ways to solve Gompertz model numerically:

In the first approach, we make use of the known solution to ODE (2) to solve ODE (1) as a function of population $N$ and time $t$.
Then, we solve the system of coupled ODEs directly, without relying on partial analytic solutions of the problem.

## Use known analytic expression of G(t) and solve single ODE

We already know the solution to ODE (2):

$$G(t)=G_0\exp(-r\,t)$$

with initial condition $G_0$.

This allows us to rewrite ODE (1) by substituting the functional form of $G(t)$:
$$\frac{dN}{dt}=G(t)N(t) = G_0\exp(-r\,t)\, N(t) \tag{3}$$

Now we have reduced the system of coupled ODEs that define the Gompertz model to a single ODE that depends on time $t$ and population $N$.

This problem can be solved with the tools that we are already familiar with.

```python
# Gompertz ODE with growth rate r and initial growth factor G0
def gompertz_single_ode(t, N):
    r  = 0.3
    G0 = 1
    return G0*np.exp(-r*t)*N

def solve_euler(f, t, y_0):
    y = np.zeros(t.shape)
    y[0] = y_0
    for i in range(0, len(t)-1):
        y[i+1] = y[i] + (t[i+1]-t[i]) * f(t[i],y[i])
    return y

t_single_ode = np.arange(0, 20, 0.1)
y_numeric_single_ode = solve_euler(gompertz_single_ode, t_single_ode, y_0=100)

plt.plot(t_single_ode, y_numeric_single_ode,'b' ,label="numeric solution, single ODE")
plt.legend()
```

## Solve System of Coupled ODEs

Alternatively, we can solve the system of coupled differential equations numerically:

\begin{align}
      \frac{dN}{dt} &=G(t)N(t) &=& &f(t, N, G) &=& N'\\
      \frac{dG}{dt} &=-r G(t)  &=& &g(t, N, G)     &=& G'
\end{align}

The basic idea of the Euler method was to approximate an unknown function at the next evaluation time point using its current value and its derivative.
Here, we need to approximate two functions simultaneously in each integration step $i$:

\begin{align}
      N_{i+1} &= N_i + \Delta_i\, f(t_i, N_i, G_i)\\
      G_{i+1} &= G_i + \Delta_i\, g(t_i, N_i, G_i)\\
\end{align}

For implementation, we need a computational function that returns the values $f(t, N, G)$ and $g(t, N, G)$ for given $t, N, G$.
Ideally, we would like this function to use the same call signature as the ODE defining functions that we had used before, i.e. one argument for function values ($N, G$), and one for time.

This can be achieved by vectorization:

Instead of accepting only a single scalar as input for the  current function value, we permit a k-dimensional array as input argument ($k=2$ in this example, i.e. `[N, G]`).
Likewise, instead of returning only the derivative $y'$, we return an array of derivative values, one for each ODE in our system of ODEs (here: `[N', G']`).

---
**Exercise:**

1. Define a vectorized ODE function with call signature `f(t, y)`, where `t` is an array of evaluation (time) points values and `y` an array of function values.
 The length of array `y` will correspond to the number of ODEs ($k$) in the coupled system.
 The ODE defining function should return an array `y_p` of same length as `y`.

2. Adapt your implementation of the Euler method to be able to work with this vectorized ODE function.
   The initial value argument `y_0` will now take an array of $k$ initial values, one for each of the ODEs. The solver function should return an array that contains the solution of all $k$ ODEs at all $n$ integration steps.

3. Solve for $N(t)$ and $G(t)$ using initial values $G_0=1$, $N_0=100$ and parameter $r=0.3$ between $t=0$ and $t=20$.

4. Plot your results and compare to the result of the previous solution approach, and the analytic solution:

    $$N(t)=N_0e^{-\frac{G_0}{r}(e^{-r\,t} -1)}$$
5. Find a suitable integration stepsize.

6. Verify that the system reaches its carrying capacity $K$ for $t\to+\infty$ and that your solution fulfills the following relation:

    $$N_0e^{\frac{G_0}{r}}=K$$
---

```python
# identify y[0] with N and y[1] with G
def gompertz_coupled_ode(t, y):
    r = 0.3
    y_p = np.zeros(y.shape)
    y_p[0] = y[0] * y[1]
    y_p[1] = -r*y[1]
    return y_p

def solve_euler_ndim(f, t, y_0):
    y = np.zeros((len(y_0),len(t)))
    y[:,0] = y_0
    for i in range(0, len(t)-1):
        y[:,i+1] = y[:,i] + (t[i+1]-t[i]) * f(t[i], y[:,i])
    return y

# initial values
G0 = 1
N0 = 100
# time
t_coupled_ode = np.arange(0, 20, 0.1)

y_numeric_coupled_ode = solve_euler_ndim(gompertz_coupled_ode, t_coupled_ode, y_0=np.array([N0,G0]))
#y_numeric_coupled_ode
```

```python
# plotting
r=0.3
analytic = N0*np.exp(-G0/r * (np.exp(-r*t_coupled_ode)-1))
K = np.exp(G0/r)*N0

fig, axes = plt.subplots(2, 1, figsize=(10,8), sharex=True)

axes[0].plot(t_coupled_ode, y_numeric_coupled_ode[0,:], 'g', label='N(t) - coupled ODE')
axes[0].plot(t_single_ode, y_numeric_single_ode, 'b', label='N(t) - single ODE')
axes[0].plot(t_single_ode, analytic, 'k--', label='N(t) - analytic')
axes[0].axhline(y=K, color='r', linestyle=':', label='expected carrying capacity')
axes[0].set_xlabel("t")
axes[0].set_ylabel("N(t)")
axes[0].legend()

axes[1].plot(t_coupled_ode, y_numeric_coupled_ode[1,:], 'm', label='G(t) - coupled ODE')
axes[1].set_xlabel("t")
axes[1].set_ylabel("G(t)")
axes[1].legend()
plt.show()
```

## Solve Systems of Coupled ODEs with `scipy.integrate.solve_ivp`

`solve_ivp` solves coupled systems *by default*, and always expects an array of initial values `y0` as input.

Applied to the Gompertz model:

```python
from scipy.integrate import solve_ivp

# as in 1.2
def gompertz_coupled_ode(t, y):
    r = 0.3
    y_p = np.zeros(y.shape)
    y_p[0] = y[0] * y[1]
    y_p[1] = -r*y[1]
    return y_p

# initial values
G0 = 1
N0 = 100
# time
t_0 = 0
t_N = 20

sol = solve_ivp(fun=gompertz_coupled_ode,
                t_span=[t_0, t_N], y0=[N0, G0],
                rtol=1E-6)
```

```python
# plotting
r=0.3
analytic = N0*np.exp(-G0/r * (np.exp(-r*t_coupled_ode)-1))
K = np.exp(G0/r)*N0

fig, axes = plt.subplots(2, 1, figsize=(10,8), sharex=True)

axes[0].plot(sol.t, sol.y[0,:], 'g', label='N(t) - coupled ODE (solve_ivp)')
axes[0].plot(t_coupled_ode, analytic, 'k--', label='N(t) - analytic')
axes[0].axhline(y=K, color='r', linestyle=':', label='expected carrying capacity')

axes[0].set_xlabel("t")
axes[0].set_ylabel("N(t)")
axes[0].legend()

axes[1].plot(sol.t, sol.y[1,:], 'm', label='G(t) - coupled ODE')
axes[1].set_xlabel("t")
axes[1].set_ylabel("G(t)")
axes[1].legend()
plt.show()
```

###### About
This notebook is part of the *biosci670* course on *Mathematical Modeling and Methods for Biomedical Science*.
See https://github.com/cohmathonc/biosci670 for more information and material.


# Measuring amplitude

### George Tzanetakis, University of Victoria

In this notebook we will explore different ways of measuring the amplitude of a sinusoidal signal. The use of the inner product to estimate the amplitude of a sinusoids in the presence of noise and other sinusoids will also be covered. As usual we start by defining a sinusoid generation function.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import IPython.display as ipd

# generate a discrete time sinusoidal signal with a specified frequency and duration
def sinusoid(freq=440.0, dur=1.0, srate=44100.0, amp=1.0, phase = 0.0):
    t = np.linspace(0,dur,int(srate*dur))
    data = amp * np.sin(2*np.pi*freq *t+phase)
    return data
```

One way of measuring the amplitude of an audio signal is by finding the maximum value. As long the array of samples contains a few cycles of a sinusoidal signal this estimation works well.

```python
def peak_amplitude(data):
    return np.max(data)
```

Let's check it out:

```python
freq = 550
data = sinusoid(freq, 0.5, amp =4.0)
print('Peak amplitude = %2.2f ' % peak_amplitude(data))
```

    Peak amplitude = 4.00

Now let's define a function that returns the Root of the Mean Squarred (RMS) amplitude. For an array containing a few cycles of a sinusoid signal we can estimate the RMS amplitude as follows:

```python
def rms_amplitude(data):
    rms_sum = np.sum(np.multiply(data,data))
    rms_sum /= len(data)
    return np.sqrt(rms_sum) * np.sqrt(2.0)
```

Let's check out that this method of estimation also works:

```python
freq = 550
data = sinusoid(freq, 0.5, amp =8.0)
print('Rms amplitude = %2.2f' %  rms_amplitude(data))
```

    Rms amplitude = 8.00

Now let's look at estimating the amplitude based on taking the dot product of two sinusoids.
Unlike the peak and RMS methods of estimating amplitude this method requires knowledge of the
frequency (and possibly phase) of the underlying sinusoid. However, it has the advantage that it is much more robust when there is interferring noise or other sinusoidal signals with other frequencies.

```python
def dot_amplitude(data1, data2):
    dot_product = np.dot(data1, data2)
    return 2 * (dot_product / len(data1))
```

First lets confirm that this amplitude estimation works for a single sinusoid

```python
data = sinusoid(300, 0.5, amp =4.0)
basis = sinusoid(300, 0.5, amp = 1)
print('Dot product amplitude = %2.2f' % dot_amplitude(data, basis))
plt.figure()
plt.plot(data[1:1000])
plt.plot(basis[1:1000])
```

Now lets add some noise to our signal. Notice that the dot-amplitude estimation works reliably, the RMS still does ok but the peak amplitude gets really affected by the added noise. Notice that the dot product amplitude estimation requires knowledge of the frequency to create the appropriate basis signal

```python
noise = np.random.normal(0, 1.0, len(data))
mix = data + noise
plt.figure()
plt.plot(data[1:1000])
plt.plot(noise[1:1000])
plt.plot(mix[1:1000])
plt.plot(basis[1:1000])
print('Dot product amplitude = %2.2f' % dot_amplitude(mix, basis))
print('Peak amplitude = %2.2f' % peak_amplitude(mix))
print('RMS amplitude = %2.2f' % rms_amplitude(mix))
```

```python
data_other = sinusoid(500, 0.5, amp = 3.0)
mix = data + data_other
plt.figure()
plt.plot(data_other[1:1000])
plt.plot(data[1:1000])
plt.plot(mix[1:1000])
plt.plot(basis[1:1000])
print('Dot product amplitude = %2.2f' % dot_amplitude(mix, basis))
print('Peak amplitude = %2.2f' % peak_amplitude(mix))
print('RMS amplitude = %2.2f' % rms_amplitude(mix))
```

To summarize, if we know the frequency of the sinusoid we are interested in, we can use the inner product with a sinusoid of the same frequency and phase as a robust way to estimate the amplitude in the presence of interferring noise and/or sinusoidal signals of different frequencies. If we don't know the phase we can use an iterative approach of trying every possible phase and selecting the one that gives the highest amplitude estimate - the brute force approach we talked about in a previous notebook.

However, there is a simpler approach to estimating both the amplitude and the phase of a sinusoidal signal of known frequency.

It is based on the following identity:
\begin{equation} a \sin(x) + b \cos(x) = R \sin (x + \theta)
\end{equation}
where
$ R = \sqrt{(a^2 + b^2)} \;\; \text{and} \;\; \theta = \tan^{-1} \frac{b}{a} $

So basically we can represent a sinusoidal signal of a particular amplitude and phase as a weighted sum (with appropriate weights $ a \;\; \text{and}\;\; b$ of a sine signal and a cosine signal. So to estimate the amplitude and phase of a sinusoid of known frequency we can take the inner product with a pair of sine and cosine signals of the same frequncy. Let's see how this would work. We will see later that these pairs of sines and cosine signals are what are called basis functions of the Discrete Fourier Transform.

```python
srate = 8000
amplitude = 3.0
k = 1000
phase = k * (2 * np.pi / srate)
print('Original amplitude  = %2.2f' % amplitude)
print('Original phase = %2.2f' % phase)

data = sinusoid(300, 0.5, amp =amplitude, phase = phase)
plt.plot(data[1:1000])
basis_sin = sinusoid(300, 0.5, amp = 1)
basis_cos = sinusoid(300, 0.5, amp = 1, phase = np.pi/2)

a = dot_amplitude(data, basis_sin)
b = dot_amplitude(data, basis_cos)
estimated_phase = np.arctan(b/a)
estimated_magnitude = np.sqrt(a*a+b*b)
print('Estimated Magnitude = %2.2f' % estimated_magnitude)
print('Estimated Phase = %2.2f' % estimated_phase)

```

```python

```


```julia
] activate .
```

[Vandermonde matrix:](https://en.wikipedia.org/wiki/Vandermonde_matrix)
\begin{align}V=\begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}\end{align}

```julia
using PyCall
```

```julia
np = pyimport("numpy")
```

    PyObject <module 'numpy' from 'C:\\Users\\carsten\\Anaconda3\\lib\\site-packages\\numpy\\__init__.py'>

```julia
np.vander(1:5, increasing=true)
```

    5×5 Array{Int32,2}:
     1  1   1    1    1
     1  2   4    8   16
     1  3   9   27   81
     1  4  16   64  256
     1  5  25  125  625

The source code for this function is [here](https://github.com/numpy/numpy/blob/v1.16.1/numpy/lib/twodim_base.py#L475-L563). It calls `np.multiply.accumulate` which is implemented in C [here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/ufunc_object.c#L3678). However, this code doesn't actually perform the computation, it basically only checks types and stuff. The actual kernel that gets called is [here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/loops.c.src#L1742). This isn't even C code but a template for C code which is used to generate type specific kernels.

Overall, this setup only supports a limited set of types, like `Float64`, `Float32`, and so forth.

Here is a simple Julia implementation (taken from [Steve's Julia intro](https://web.mit.edu/18.06/www/Fall17/1806/julia/Julia-intro.pdf))

```julia
function vander(x::AbstractVector{T},  n=length(x)) where T
    m = length(x)
    V = Matrix{T}(undef, m, n)
    for j = 1:m
        V[j,1] = one(x[j])
    end
    for i= 2:n
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

    vander (generic function with 2 methods)

```julia
vander(1:5)
```

    5×5 Array{Int64,2}:
     1  1   1    1    1
     1  2   4    8   16
     1  3   9   27   81
     1  4  16   64  256
     1  5  25  125  625

# A quick benchmark

```julia
using BenchmarkTools, Plots
```

```julia
ns = exp10.(range(1, 4, length=30));
```

```julia
tnp = Float64[]
tjl = Float64[]
for n in ns
    x = 1:n |> collect
    push!(tnp, @belapsed np.vander($x) samples=3 evals=1)
    push!(tjl, @belapsed vander($x) samples=3 evals=1)
end
```

```julia
plot(ns, tnp./tjl, m=:circle, xscale=:log10, xlab="matrix size", ylab="NumPy time / Julia time", legend=:false)
```

```julia
savefig("vandermonde.pdf")
savefig("vandermonde.svg")
savefig("vandermonde.png")
```

Note that the clean and concise Julia implementation is beating the numpy implementation for small and is on-par for large matrix sizes!

At the same time, the Julia code is generic and works for arbitrary types!

```julia
vander(Int32[4, 8, 16, 32])
```

    4×4 Array{Int32,2}:
     1   4    16     64
     1   8    64    512
     1  16   256   4096
     1  32  1024  32768

```julia
vander(["this", "is", "a", "test"])
```

    4×4 Array{String,2}:
     ""  "this"  "thisthis"  "thisthisthis"
     ""  "is"    "isis"      "isisis"
     ""  "a"     "aa"        "aaa"
     ""  "test"  "testtest"  "testtesttest"

```julia
vander([true, false, false, true])
```

    4×4 Array{Bool,2}:
     true   true   true   true
     true  false  false  false
     true  false  false  false
     true   true   true   true

```julia

```


# [SymPy](http://www.sympy.org/en/index.html)
A Python library for symbolic mathematics. Among others things, it provides:
1. [Symbolic simplification](http://docs.sympy.org/latest/tutorial/simplification.html).
2. [Calculus (derivatives, integrals, limits, and series expansions)](http://docs.sympy.org/latest/tutorial/calculus.html).
3. [Algebraic solver](http://docs.sympy.org/latest/tutorial/solvers.html).
4. [Matrix operations](http://docs.sympy.org/latest/tutorial/matrices.html).
5. [Combinatorics](http://docs.sympy.org/latest/modules/combinatorics/index.html)
6. [Cryptography](http://docs.sympy.org/latest/modules/crypto.html).

Table of Contents

1  Example

## Example

```python
try:
    from sympy import init_session
except:
    !pip3 install sympy
    from sympy import init_session

init_session(use_latex='matplotlib')
```

    IPython console for SymPy 1.5.1 (Python 3.7.7-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
# https://github.com/AeroPython/Taller-Aeropython-PyConEs16
expr = cos(x)**2 + sin(x)**2
expr
```

```python
simplify(expr)
```

```python
expr.subs(x, y**2)
```

```python
expr = (x + y) ** 2
expr
```

```python
expr = expr.expand()
expr
```

```python
expr = expr.factor()
expr
```

```python
expr = expr.integrate(x)
expr
```

```python
expr = expr.diff(x)
expr
```


# Computing the Riemann Constant Vector

This iPython notebook contains the example code presented in the paper [Computing the Riemann Constant Vector](http://www.cswiercz.info/assets/files/rcv.pdf) by Bernard Deconinck, Matthew S. Patterson, and Chris Swierczewski.

First, we construct the Riemann surface, $X$, defined by the plane algebraic curve

$$
f(x,y) = x^2 y^3 - x^4 + 1.
$$

We computationally determine its genus as well as a basis $\{\tilde{\omega}_1, \ldots, \tilde{\omega}_g\}$ of Abelian differentials of the first kind, $\Omega_X^1$, defined on the surface $X$.

```python
from sympy.abc import x,y
from abelfunctions import (RiemannSurface, RiemannTheta, Jacobian,
                           AbelMap, RiemannConstantVector, puiseux)

f = x**2*y**3 - x**4 + 1
X = RiemannSurface(f,x,y)
g = X.genus()
omega = X.holomorphic_differentials()

print 'differentials:'
for omegai in omega:
    print omegai
print 'genus:', g
```

    differentials:
    1/3*x**2*y**2
    x/3*x**2*y**2
    x*y/3*x**2*y**2
    x**2/3*x**2*y**2
    genus: 4

Next, we compute a canonical basis of cycles $\{a_1, \ldots, a_g, b_1, \ldots, b_g\}$ of the first homology group $H_1(X, \mathbb{Z})$. That is, every cycle on $X$ can be written as a linear combination of these cycles.

```python
%matplotlib inline

a = X.a_cycles()
b = X.b_cycles()

# use 256 points to plot the x- and y-parts of the path
figx = a[0].plot_x(256, color='blue', linewidth=2, linestyle='dashed')
figy = a[0].plot_y(256, color='green', linewidth=2)
```

Using these data we can compute the period matrix of the surface:

$$
\tau = [A \; B] \in \mathbb{C}^{g \times 2g}
$$

where

$$
A_{ij} = \oint_{a_j} \tilde{\omega}_i, \quad \text{and} \quad B_{ij} = \oint_{a_j} \tilde{\omega}_i
$$

If using a *normalized* basis of differentials, $\{\omega_1, \ldots, \omega_g\}$, the period matrix is of the form

$$
\tau = [I \; \Omega] \in \mathbb{C}^{g \times 2g}.
$$

This is equivalent to setting $\Omega = A^{-1}B$ and, in fact, since Abelfunctions returns a non-normalized basis this is how the matrix $\Omega$ is computed. We computationally verify that $\Omega$ is a "Riemann matrix": a symmetric matrix with positive definite imaginary part.

```python
# for brevity, we only print the first four significant digits
import numpy
numpy.set_printoptions(precision=4, suppress=True)

tau = X.period_matrix()
A = tau[:g,:g]
B = tau[:g,g:]
Omega = X.riemann_matrix() # returns A**(-1)*B

print A
print
print B
print
print Omega
```

    [[ 0.2800+1.045j   0.2800-0.485j  -1.8100+1.045j   0.0000-0.j    ]
     [ 0.6625-1.1475j  0.6625+0.3825j -0.6625-1.1475j  0.0000+1.53j  ]
     [-0.8347+0.4819j -0.8347+0.4819j  0.8347-1.4457j  0.0000+1.9276j]
     [-1.0450+0.28j   -1.0450+1.81j   -0.4850+0.28j    0.0000+0.j    ]]

    [[-0.2800+0.485j   0.2800-1.045j   0.0000-2.09j    0.7650-1.325j ]
     [ 0.6625+0.3825j -0.6625+0.3825j  0.0000-0.765j   0.0000-1.53j  ]
     [-0.8347+0.4819j  0.8347-1.4457j  0.0000-0.9638j -1.6694-0.9638j]
     [ 1.0450-1.81j   -1.0450-0.28j   -0.0000-0.56j    0.7650-1.325j ]]

    [[ 0.3934+0.795j  -0.7541-0.3691j -0.4426-0.0284j  0.2049+0.2697j]
     [-0.7541-0.3691j  0.2787+0.8518j  0.0984+0.1988j -0.4344-0.1562j]
     [-0.4426-0.0284j  0.0984+0.1988j -0.3770+0.6815j -0.9180+0.4543j]
     [ 0.2049+0.2697j -0.4344-0.1562j -0.9180+0.4543j -1.2787+0.8802j]]

```python
symmetric_error = numpy.linalg.norm(Omega - Omega.T)
imag_part_evals = numpy.linalg.eigvals(Omega.imag)

print 'error:', symmetric_error
print 'evals:', imag_part_evals
```

    error: 3.54420172099e-10
    evals: [ 1.4038  1.1654  0.4294  0.21  ]

## Places and Divisors

Places $P \in X$ are represented using a Puiseux series. If $x$ and $y$ are the affine coordinates of the underlying curve $f$ then we write

$$
P =
\begin{cases}
x(t) = \alpha + \lambda t^e \\
y(t) = \sum_i \beta_i t^{n_i}
\end{cases}
$$

In the case when $e = 1$ (an "unramified place") and $x \neq \infty$ a place is synonymous with a tuple $(\alpha, \beta) \in \mathbb{C}^2$ such that $f(\alpha, \beta) = 0$.

A divisor $\mathcal{D}$ is a *formal sum* of places, $\mathcal{D} = \sum_i n_i P_i$, where each of the $n_i$ are the *multiplicities* of the places $P_i$ in the divisor. Below we compute local representation of several places on the Riemann surface and construct some divisors from them.

```python
places_above_zero = X(0)
print places_above_zero
```

    [(-t**3, -1/t**2 + O(1))]

```python
print 'Places:'
places_above_two = X(2)
for P in places_above_two:
    print P

print 'Puiseux:'
series_at_two = puiseux(f,x,y,2)
for p in series_at_two:
    print p
```

    Places:
    (2, RootOf(4*_y**3 - 15, 0))
    (2, RootOf(4*_y**3 - 15, 1))
    (2, RootOf(4*_y**3 - 15, 2))
    Puiseux:
    (t + 2, RootOf(4*_y**3 - 15, 0) + O(t**2))
    (t + 2, RootOf(4*_y**3 - 15, 1) + O(t**2))
    (t + 2, RootOf(4*_y**3 - 15, 2) + O(t**2))

```python
P = places_above_zero[0]
Q = places_above_two[0]
D = 3*P + Q

print 'Divisor:', D
print 'Degree:', D.degree
```

    Divisor: (2, RootOf(4*_y**3 - 15, 0)) + 3(-t**3, -1/t**2 + O(1))
    Degree: 4

```python
D2 = sum(places_above_two)
print D2
```

    (2, RootOf(4*_y**3 - 15, 0)) + (2, RootOf(4*_y**3 - 15, 2)) + (2, RootOf(4*_y**3 - 15, 1))

## The Abel Map

Given a fixed *base place* $P_0 \in X$, the Abel Map $A : X \to J(X)$ is defined

$$
A(P) = \left( \int_{P_0}^P \omega_1, \ldots, \int_{P_0}^P \omega_g \right)
$$

where the path of integration from $P_0$ to $P$ is the same for each Abelian differential of the first kind. The Abel map is linear over divisors. That is, if $\mathcal{D} = \sum_i n_iP_i$ then

$$
A(\mathcal{D}) = \sum_i n_i A(P_i).
$$

When we want to change or make the back point explicit we write $A(P_0,\mathcal{D})$. Below we evaluate the Abel map at the places and divisors constructed above.

```python
J = Jacobian(X)   # reduces vectors modulo lattice ZZ^g + Omega ZZ^g
z1 = AbelMap(P)   # Abel map from P0 to P
z2 = AbelMap(Q)   # Abel map from P0 to Q
z3 = AbelMap(P,Q) # Abel map from P to Q
print z1
print z2
print z3

# numerically verify that A(P,Q) = A(P0,Q) - A(P0,P)
print numpy.linalg.norm( J((z2-z1) - z3) )
```

    [-0.5261+0.0864j  0.0669+0.6392j -0.7495+1.1037j -1.5030+1.0356j]
    [-0.3875+0.1157j -0.0290+0.4437j -0.4532+0.7774j -0.9721+0.6732j]
    [ 0.1468-0.0985j  0.8467+0.6989j  0.0996+1.0083j -1.1003+0.8159j]
    6.48145569903e-16

```python
w = AbelMap(D)
print w

# verify that w = 3*z1 + z2 mod period lattice
z = J(3*z1 + z2)
print numpy.linalg.norm(w-z)
```

    [ 0.0670-0.1361j  0.9421+0.7429j -0.4887+0.7663j -1.5057+0.6992j]
    1.3608726004e-15

## Canonical and Valuation Divisors

Let $P = (x(t), y(t))$ be the Puiseux series representation of a place $P \in X$. Given a meromorphic differential $\nu$ we can write the "localization of $\nu$ at $P$" as a Laurent series in the local parameter $t$:

$$
\nu\big|_P = (ct^n + \cdots) dt.
$$

Then the "valuation" of $\nu$ at $P$ is

$$
\text{val}(\nu,P) = n.
$$

The **valuation divisor** $(\nu) \in \text{Div}(X)$ of $\nu$ is the divisor

$$
(\nu) = \sum_{P \in X} \text{val}(\nu,P) P.
$$

A divisor is **canonical** if and only if it's the valuation divisor of a *holomorphic* differential on $X$. All canonical divisors are of degree $2g-2$. Below, we compute a canonical divisor for each holmomrphic differential basis element $\tilde{\omega}_i$ and verify that their degree is $6$.

```python
C0 = omega[0].valuation_divisor()
for place,multiplicity in C0:
    print multiplicity, place
print 'Degree:', C0.degree
```

    6 (t**(-3), t**(-2) + O(1))
    Degree: 6

```python
C1 = omega[1].valuation_divisor()
for place,multiplicity in C1:
    print multiplicity, place
print 'Degree:', C1.degree
```

    3 (-t**3, -1/t**2 + O(1))
    3 (t**(-3), t**(-2) + O(1))
    Degree: 6

```python
C2 = omega[2].valuation_divisor()
for place,multiplicity in C2:
    print multiplicity, place
print 'Degree:', C2.degree
```

    1 (t**3/4 + 1, t + O(t**3))
    1 (-t**3*RootOf(_x**2 + 1, 0)/4 + RootOf(_x**2 + 1, 0), t + O(t**3))
    1 (-t**3/4 - 1, t + O(t**3))
    1 (t**(-3), t**(-2) + O(1))
    1 (-t**3, -1/t**2 + O(1))
    1 (-t**3*RootOf(_x**2 + 1, 1)/4 + RootOf(_x**2 + 1, 1), t + O(t**3))
    Degree: 6

```python
C3 = omega[3].valuation_divisor()
for place,multiplicity in C3:
    print multiplicity, place
print 'Degree:', C3.degree
```

    6 (-t**3, -1/t**2 + O(1))
    Degree: 6

## Riemann Constant Vector

The Riemann constant vector satisfies the following two theorems:

**Theorem 1:** $\mathcal{C}$ is a canonical divisor if any only if $K(P_0) \equiv -A(P_0,\mathcal{C})$.

**Theorem 2:** $\theta(W,\Omega) = 0$ if and only if $W = A(P_0,\mathcal{D}) + K(P_0)$ where $\mathcal{D}$ is a degree $g-1$ divisor.

We compute $K$ below and verify that these two theorems are satisfied.

```python
K = RiemannConstantVector # alias the RCV function for brevity
P0 = X.base_place()
print K(P0)
```

    [ 0.8488+0.7203j -0.5941-0.1146j -0.7432+0.8913j -0.8189+1.1381j]

```python
z = J(2*K(P0) + AbelMap(C3))
print z
```

    [ 0.+0.j -0.+0.j -0.+0.j  0.+0.j]

```python
W = K(P0)
v = RiemannTheta.oscillatory_part(W,Omega)
print abs(v)
```

    2.59621137233e-08

```python
D = sum(places_above_two)
W = J(AbelMap(D) + K(P0))
v = RiemannTheta.oscillatory_part(W, Omega)
print abs(v)
```

    9.39107997133e-10

Additional Code
===============

The following code generates Figure 1 in the paper "Computing the Riemann Constant Vector".

```python
import matplotlib.pyplot as plt
from abelfunctions.riemann_constant_vector import initialize_half_lattice_vectors
```

```python
h = initialize_half_lattice_vectors(X)
V = 0.5*AbelMap(C3)

oscillatory_parts = [
    RiemannTheta.oscillatory_part(J(hi.T-V),Omega)
    for hi in h
]
oscillatory_magnitudes = sorted(map(abs,oscillatory_parts))
```

```python
%matplotlib inline

fig = plt.figure()
ax = fig.add_subplot(111)
ax.semilogy(range(256), oscillatory_magnitudes,
            linestyle='', marker='d',markerfacecolor='grey',markersize=8)
ax.axis([-32,256+32,1e-9,10])
ax.set_xticks([-32] + [32*k for k in range(10)])
ax.set_xticklabels([''] + [str(32*k) for k in range(9)] + [''])
ax.grid(True)
fig.set_figwidth(9)
fig.set_figheight(4)

fig.show()
```

```python

```

notebooks/Riemann Constant Vector Paper.ipynb


```python
%matplotlib inline
import numpy as np
from matplotlib.pyplot import *

import sympy as sym
import pandas as pd
```

Consider the following one-dimensional PDE:
$$
-u_{xx}(x) = f(x)\quad\mathrm{ in }\ \Omega = (0, \pi)
$$
$$
u(x) = 0, \quad\mathrm{ on }\ \partial\Omega = \{0, \pi\}
$$

Given the following $4^{th}$ order finite difference approximation of the second order derivative:

$$u_{xx}(x_i) = \frac{-u_{i-2}+16u_{i-1}-30u_i+16u_{i+1}-u_{i+2}}{12h^2}$$

Implement a function that given the domain interval, the forcing function, the number of discretization points, the boundary conditions, returns the matrix $A$ and the the right hand side $b$.

```python
def finDif( omega, f, n, bc ):
    '''
    Given the domain interval omega,
    the forcing function f,
    the number of discretization points n,
    the boundary conditions bc,
    finDif returns the matrix A and the the right hand side b
    '''
    assert n>1
    assert omega[1]>omega[0]
    x=np.linspace(omega[0],omega[1],n)
    h = (omega[1]-omega[0])/(n-1)
    a2 = np.ones((n-2,)) # Offdiagonal entries
    a1 = -16 * np.ones((n-1,)) # Offdiagonal entries
    a0 = 30 * np.ones((n,)) # Diagonal entries
    A = (np.diag(a2, -2) + np.diag(a1, -1) + np.diag(a0, 0 )+ np.diag(a1, 1 )+ np.diag(a2, 2))/(12)
    b=f(x)
    # Change first row of the matrix A
    A[0,:] = 0
    A[:,0] = 0
    A[0,0] = 1
    b[0] = bc[0]/(h**2)
    # Change last row of the matrix A
    A[-1,:] = 0
    A[:,-1] = 0
    A[-1,-1] = 1
    b[-1] = bc[1]/(h**2)
    A = A / ( h**2 )
    return A, b
```

Call the function using:

```python
omega = [0, np.pi]
f = lambda x : np.sin(x)
n=100
bc = [0,0]
A, b = finDif(omega, f, n, bc)
```

Implement two functions that compute the LU and the Cholesky factorization of the system matrix $A$

```python
def LU(A):
    '''
    LU returns the LU factorization of the system matrix A
    '''
    A = A.copy()
    N=len(A)
    for k in range(N-1):
        if (abs(A[k,k]) < 1e-15):
            raise RuntimeError("Null pivot")
        A[k+1:N,k] /= A[k,k]
        for j in range(k+1,N):
            A[k+1:N,j] -= A[k+1:N,k]*A[k,j]
    L=np.tril(A)
    for i in range(N):
        L[i,i]=1.0
    U = np.triu(A)
    return L, U

L, U = LU(A)
```

```python
def cholesky(A):
    '''
    cholesky returns the Cholesky factorization of the system matrix A
    '''
    A = A.copy()
    N = len(A)
    for k in range(N-1):
        A[k,k] = np.sqrt(A[k,k])
        A[k+1:N,k] = A[k+1:N,k]/A[k,k]
        for j in range(k+1,N):
            A[j:N,j] = A[j:N,j] - A[j:N,k]*A[j,k]
    A[-1,-1] = np.sqrt(A[-1,-1])
    L=np.tril(A)
    return L, L.transpose()

HT, H = cholesky(A)
```

Implement forward and backward substitution functions to exploit the developed factorization methods to solve the derived linear system of equations.

```python
def L_solve(L,rhs):
    x = np.zeros_like(rhs)
    N = len(L)
    x[0] = rhs[0]/L[0,0]
    for i in range(1,N):
        x[i] = (rhs[i] - np.dot(L[i, 0:i], x[0:i]))/L[i,i]
    return x
```

```python
def U_solve(U,rhs):
    x = np.zeros_like(rhs)
    N=len(U)
    x[-1] = rhs[-1]/U[-1,-1]
    for i in reversed(range(N)):
        x[i] = (rhs[i] - np.dot(U[i, i+1:N], x[i+1:N]))/U[i,i]
    return x
```

Solve the derived linear system using the implemented functions and plot the computed solution:

```python
x=np.linspace(omega[0],omega[1],n)
# My solution
w = L_solve(L,b)
u = U_solve(U,w)
# Numpy solution
u_ex = np.linalg.solve(A, b)
# Lets plot
_ = plot(x,u,'o-b', label="LU")
_ = plot(x,u_ex,'r', label="numpy")
legend()
show()
print(np.linalg.norm(u-u_ex,2)/np.linalg.norm(u,2))
```

Considering the new domain $\Omega = (0,1)$ and the forcing term $f(x) = x(1-x)$ with B.C. $u(x) = 0$, on $\partial \Omega = {0,1}$ produce a plot and a table where you show the decay of the error w.r.t. the number of grid points.
(The analytical solution for the above problems is $u_{an} = \frac{x^4}{12} - \frac{x^3}{6} + \frac{x}{12}$)

```python
def calculate_error(omega, f, bc, Nmin, Nmax, sol_exact):
    '''
    calculate the error between the LU solution and the function sol_exact
    given:
    the interval omega
    the forcing force f
    the boundary condictions bc
    as a function of n integer in between Nmin and Nmax
    '''
    errors = []
    N=[]
    error = 0.
    for n in range(Nmin,Nmax):
        x=np.linspace(omega[0],omega[1],n)
        A, b = finDif(omega, f, n, bc)
        L, U = LU(A)
        w = L_solve(L,b)
        u = U_solve(U,w)
        #u = np.linalg.solve(A, b)
        u_ex =sol_exact(x)
        error=max(abs(u-u_ex))
        errors.append(error)
        N.append(n)
    return N,errors

sol_exact = lambda x :  (x**4)/12. - (x**3)/6. + x/12
omega2 = [0,1]
f2 = lambda x : x*(1-x)
bc2 = [0,0]
m,errors=calculate_error(omega2, f2, bc2, 5, 101, sol_exact)
semilogy(errors, label="error")
title("Decay of the error w.r.t. the number of grid points")
legend()
df = pd.DataFrame({'n' : m, 'error' : errors})
display(df)
# df.style
```

Exploit the derived LU factorizations to compute the condition number of the system's matrix $A$ using the original problem formulation.

```python
def IPM(A, x0, mu, tol, nmax ):
    '''
    Power method with shift (or inverse power method):
    given the matrix A and vector x0, together with the shift mu
    IPM returns eigenvalue close to mu and the relative eigenvector
    with tolerance tol or after nmax iterations
    '''
    M = A - mu * np.identity(len(A))
    L,U = LU(M)
    err = tol + 1.
    it = 0
    q = x0/(np.linalg.norm(x0,2))
    while it < nmax and err > tol :
        y = L_solve(L,q)
        x = U_solve(U,y)
        q = x/(np.linalg.norm(x,2))
        z = np.dot(A,q)
        l = np.dot(q.T,z)
        err =np.linalg.norm(z-l*q,2)
        it += 1
    return l,q

def PM(A, z0, tol, nmax):
    '''
    Power method:
    given A and z0
    PM returns the maximum eigenvalue and the associated eigenvector
    with tolerance tol or after nmax iterations
    '''
    q = z0/(np.linalg.norm(z0,2))
    it = 0
    err = tol + 1.
    while it < nmax and err > tol:
        z = np.dot(A,q)
        l = np.dot(q.T,z)
        err = np.linalg.norm(z-l*q,2)
        q = z/(np.linalg.norm(z,2))
        it += 1
    return l,q

def condNumb(A, eps, tol, nmax):
    '''
    condNumb returns the conditional number of the matrix A:
    it uses the power method to find the max eigenvalue
    and the Inverse power method with shift to find the minimum eigenvalue.
    The tolerance is tol and nmax is the max number of iterations.
    eps is a regulator for the Inverse power method.
    Actually, we look for the minimum eigenvalue close to zero (regulated by eps)
    as our matrix is symmetric positve definite
    '''
    n = np.shape(A)[0]
    z0 = np.random.rand(n)
    lmin, _ = IPM(A, z0, eps, tol, nmax)
    lmax, _ = PM( A, z0, tol, nmax)
    condNu = lmax/lmin
    return condNu

print(condNumb(A, 1e-10, 1e-10, 10000 ))
print(np.linalg.cond(A))
```

    5278.068747699028
    5278.068747922936

Implement a preconditioned Conjugant Gradient method to solve the original linear system of equations using an iterative method:

```python
def conjugate_gradient(A, b, P, nmax, eps):
    '''
    Preconditioned Conjugant Gradient method:
    Given the matrix A, the vector b and the preconditioner matrix P
    conjugate_gradient returns the solution of the original system
    after nmax iterations or with tolerance eps
    '''
    N=len(A)
    x = np.zeros_like(b)
    tol = eps + 1
    it = 0
    r = b - np.dot(A,x)
    rho_old = 1.
    p_old = np.zeros_like(b)
    while (it < nmax and tol > eps):
        it += 1
        z = np.linalg.solve(P,r)
        rho = np.dot(r,z)
        if (it > 1):
            beta = rho/rho_old
            p = z + beta*p_old
        else:
            p = z
        q = np.dot(A,p)
        alpha = rho/(np.dot(p,q))
        x += p*alpha
        r -= q*alpha
        p_old = p
        rho_old = rho
        tol = np.linalg.norm(r,2)
    return x

# conjugate gradient solution
sol_conjugate_gradient = conjugate_gradient(A,b, np.identity(len(A)), np.shape(A)[0], 1e-10)
x=np.linspace(omega[0],omega[1],n)
# LU solution
L,U = LU(A)
w = L_solve(L,b)
u = U_solve(U,w)
# Numpy solution
u_ex = np.linalg.solve(A, b)
# Lets plot
_ = plot(x,sol_conjugate_gradient,'ro', label="conjugate gradient")
_ = plot(x,u_ex,'b', label="numpy")
legend()
show()
_ = plot(x,sol_conjugate_gradient,'ro', label="conjugate gradient")
_ = plot(x,u,'g', label="LU")
legend()
show()

print(np.linalg.norm(sol_conjugate_gradient-u_ex,2)/np.linalg.norm(u_ex,2))
print(np.linalg.norm(sol_conjugate_gradient-u,2)/np.linalg.norm(u,2))
```

Consider the following time dependent variation of the PDE starting from the orginal problem formulation:
$$
u'(t)-u_{xx} = \alpha(t)f(x)
$$

for $t\in [0,T]$, with $\alpha(t) = \cos(t)$ and $T = 6\pi$

Use the same finite difference scheme to derive the semi-discrete formulation and solve it using a forward Euler's method.

Plot the time dependent solution solution at $x = \pi/2$, $x=1$,
$x=\pi$

```python
def fe_pde(omega_x, omega_t, bc, y0, h, n, f, eps ):
    '''
    Given the intervals omega_t and omega_x
    with boundary conditions y0 and bc
    h is the time distance between two consecutive nodes
    n is the number of spatial discretization points
    f is the forcing function
    eps a small parameter to calculate the timestep correctly
    fe_pde returns the solutions of the pde together with the timesteps and xsteps
    '''
    assert n > 0
    assert h > 0
    A, b = finDif(omega_x, f, n, bc)
    timesteps = np.arange(omega_t[0],omega_t[1]+eps, h)
    m = len(timesteps)
    sol = np.zeros((m,n))
    Au = np.zeros((m,n))
    xsteps=np.linspace(omega_x[0],omega_x[1],n)
    sol[0] = y0
    for i in range(0,m-1):
        Au[i] = np.dot(A,sol[i])
        sol[i+1] = sol[i]  + h *(b*np.cos(timesteps[i])- Au[i] )
    return sol, timesteps, xsteps

def sol_x_fixed(sol, omega_x, x):
    assert x>= omega_x[0]
    assert x<= omega_x[1]
    n = np.shape(sol)[1]
    assert n>0
    x_index = int((n-1)*(x-omega_x[0])//(omega_x[1]-omega_x[0]))
    #if x_index<0:
    #    x_index=0
    #if x_index>=n:
    #    x_index=n-1
    sol_x_fixed = sol[:,x_index]
    return sol_x_fixed

def sol_t_fixed(sol, omega_t, t):
    assert t>= omega_t[0]
    assert t<= omega_t[1]
    m = np.shape(sol)[0]
    assert m>0
    t_index = int((m-1)*(t-omega_t[0])//(omega_t[1]-omega_t[0]))
    #if t_index<0:
    #    t_index=0
    #if t_index>=m:
    #    t_index=m-1
    sol_t_fixed = sol[t_index,:]
    return sol_t_fixed

### Boundary condition y0 = 0

f = lambda x : np.sin(x)
omega_x = [0,np.pi]
omega_t = [0,6*np.pi]
n=100
x=np.linspace(omega[0],omega[1],n)
z0 = np.ones_like(x)
bc = [0,0]
A, b = finDif(omega, f, n, bc)
l_max,_ = PM(A,z0,1e-15,15000)
h = 0.0001
assert h < 2/ abs(l_max)
y0 = 0
sol, tsteps, xsteps = fe_pde(omega_x, omega_t, bc, y0, h, n, f, 1e-10)

# Comparison with exact solution
t= tsteps
sol_t = sol_x_fixed(sol, omega_x, 1)
plot(t,sol_t, 'blue')
plot(t,-(1/2)*np.exp(-t)*np.sin(1)+1/2*np.cos(t)*np.sin(1)+1/2*np.sin(t)*np.sin(1),'black',linestyle='dotted', label="analytical")
legend()
show()

sol_t = sol_x_fixed(sol, omega_x,np.pi/2)
plot(t,sol_t,'green')
plot(t,-(1/2)*np.exp(-t)*np.sin(np.pi/2)+1/2*np.cos(t)*np.sin(np.pi/2)+1/2*np.sin(t)*np.sin(np.pi/2),'black',linestyle='dotted', label="analytical")
legend()
show()

sol_t = sol_x_fixed(sol, omega_x,np.pi)
plot(t,sol_t, 'red')
plot(t,-(1/2)*np.exp(-t)*np.sin(np.pi)+1/2*np.cos(t)*np.sin(np.pi)+1/2*np.sin(t)*np.sin(np.pi),'black',linestyle='dotted', label="analytical")
legend()
show()

### Boundary condition y0 = sin(x)

f = lambda x : np.sin(x)

omega_x = [0,np.pi]
omega_t = [0,6*np.pi]
n=100
x=np.linspace(omega[0],omega[1],n)
bc = [0,0]
A, b = finDif(omega, f, n, bc)
l_max,_ = PM(A,z0,1e-15,15000)
h = 0.0003
assert h < 2/ abs(l_max)
y0 = np.sin(x)

sol, tsteps, xsteps = fe_pde(omega_x, omega_t, bc, y0, h, n, f, 1e-10)

# Comparison with exact solution
t= tsteps
sol_t = sol_x_fixed(sol, omega_x, 1)
plot(t,sol_t, 'blue')
plot(t,(1/2)*np.exp(-t)*np.sin(1)+1/2*np.cos(t)*np.sin(1)+1/2*np.sin(t)*np.sin(1),'black',linestyle='dotted', label="analytical")
legend()
show()

sol_t = sol_x_fixed(sol, omega_x, np.pi/2)
plot(t,sol_t,'green')
plot(t,(1/2)*np.exp(-t)*np.sin(np.pi/2)+1/2*np.cos(t)*np.sin(np.pi/2)+1/2*np.sin(t)*np.sin(np.pi/2),'black',linestyle='dotted', label="analytical")
legend()
show()

sol_t = sol_x_fixed(sol, omega_x, np.pi)
plot(t,sol_t, 'red')
plot(t,(1/2)*np.exp(-t)*np.sin(np.pi)+1/2*np.cos(t)*np.sin(np.pi)+1/2*np.sin(t)*np.sin(np.pi),'black',linestyle='dotted', label="analytical")
legend()
show()
## Few differences between analytic solution and numerical solution but of the order of 1e-16
```

Given the original $Au = b$ system, implement an algorithm to compute the eigenvalues and eigenvectors of the matrix $A$. Exploit the computed LU factorization

```python
def lu_eigvalues(A, tol, nmax):
    """
    Given the matrix A lu_eigvalues returns the eigenvalues
    with tolerance tol or after nmax iterations
    """
    A_new=A.copy()
    n = np.shape(A_new)[0]
    eigvals_old = np.diag(A_new)
    it = 0
    err = tol + 1.
    while(it < nmax and err > tol ):
        L, U = LU(A_new)
        A_new = np.matmul(U,L)
        eigvals_new = np.diag(A_new)
        err = np.linalg.norm(eigvals_new-eigvals_old,2)
        it = it + 1
        eigvals_old = eigvals_new
    return eigvals_new

def eig(A, eps, tol_eigevalues, nmax_eigevalues, tol_IPM, nmax_IPM):
    """
    eig returns eigenvalues and eigenvectors of A
    eig first finds the approximated eigenvalues using LU decomposition (lueigvals)
    with tolerance tol_eigevalues or using nmax_eigevalues iterations
    Then we use the IPM function to find the eigenvalues,
    close to the approximated eigenvalue (regulated by eps),
    and the relative eigenvectors
    with tolerance tol_IPM or using nmax_IPM iterations
    """
    n = np.shape(A)[0]
    z0 = np.random.rand(n)
    eigvals = np.zeros(n)
    eigvecs = np.zeros((n,n))
    lueigvals = lu_eigvalues(A, tol_eigevalues, nmax_eigevalues)
    for i, qre in enumerate(lueigvals):
        eigvals[i], eigvecs[i] = IPM(A, z0, qre + eps, tol_IPM, nmax_IPM )
    return eigvals, eigvecs

l,v = eig(A, 1e-10, 1e-10, 3000, 1e-10, 10000 )
l_numpy,v_numpy = np.linalg.eig(A)
print(max(abs(np.sort(l)-np.sort(l_numpy))))
print()
print(np.sort(v)-np.sort(v_numpy))
# It takes some time...
# The algorith is not that efficient so in order to have good accuracy for the eigenvalues I need many iterations
# Still it is not enough to resolve correctly the eigenvectors

```

    1.5006662579253316e-11

    [[-6.75407314e-25 -6.58353206e-25 -6.33976992e-25 ...  6.88137895e-25
       3.81118185e-01 -7.54736733e-02]
     [ 5.87127558e-04  6.27310410e-04  8.44491526e-04 ... -5.67468240e-05
      -3.55711170e-04 -6.09596892e-04]
     [ 1.32406417e-04 -1.88459072e-04  3.70312990e-04 ... -4.96275872e-05
      -1.34983203e-04 -2.40706189e-04]
     ...
     [-5.88838847e-04 -1.00888656e-03 -1.26785559e-03 ... -2.20569435e-04
      -8.34925547e-05  6.30041547e-05]
     [ 2.29808483e-04 -1.17553961e-03 -1.79866824e-03 ... -1.46720716e-01
      -1.42627391e-01 -1.42649860e-01]
     [-6.75407314e-25 -6.58353206e-25 -6.33976992e-25 ...  6.88137895e-25
       3.81118185e-01 -7.54736733e-02]]

Compute the inverse of the matrix A exploiting the derived LU factorization

```python
def inverse(A):
    """
    inverse returns the inverse of the matrix A
    """
    m = np.shape(A)[0]
    Ainv= np.ones_like(A)
    L,U = LU(A)
    for j in range(m):
        e=np.zeros((m,))
        e[j]=1
        winv = L_solve(L,e)
        uinv = U_solve(U,winv)
        Ainv[:,j]=uinv
    return Ainv

Ainv = inverse(A)
Ainv_numpy = np.linalg.inv(A)
print(np.linalg.norm(Ainv-Ainv_numpy,2))
```

    6.492862786734044e-14

Consider the following Cauchy problem
$$
\begin{cases}
y'= -ty^2 \quad 0\le t \le 2\\
y(0) = 1
\end{cases}
$$
Implement a Backward Euler's method in a suitable function and solve the resulting non-linear equation using a Newton's method.

```python
def newton(f, f_prime, x0, eps, n_max):
    '''
    Newton's method:
    f: function
    f_prime: derivative of the function
    x0: initial point
    eps: tolerance
    n_max: max number of iterations
    '''
    err = abs(f(x0))
    it = 0
    x = x0
    while (err > eps and it < n_max):
        if abs(f_prime(x)) < 1e-12:
            raise RuntimeError("f_prime(x) is close to zero")
        x_new = x - f(x)/f_prime(x)
        err = abs(f(x_new))
        x = x_new
        it += 1
    return x

def fe(t0, tf, h, y0, eps=1e-10):
    '''
    Forward Euler's method:
    given
    t0:  initial time
    tf:  final time
    h:   time distance between two consecutive nodes
    y0:  boundary condition at y[t0]
    eps: small epsilon to calculate the time steps correctly
    fe returns the solution of the ODE and the timesteps
    '''
    timesteps = np.arange(t0,tf+eps, h)
    sol = np.zeros_like(timesteps)
    sol[0] = y0
    for i in range(1,len(sol)):
        sol[i] = sol[i-1] - h * timesteps[i-1] * sol[i-1]**2
    return sol, timesteps

def be(t0, tf, h, y0, tol=1e-10, nmax=1000, eps=1e-10):
    """
    Backward Euler's method:
    given
    t0: initial time
    tf: final time
    h: the time distance between two consecutive nodes
    y0: boundary condition at y[t0]
    tol: tolerance for the iterations
    nmax: max number of iterations
    eps: small epsilon to calculate the time steps correctly
    be returns the solution of the ODE and the timesteps
    """
    timesteps = np.arange(t0, tf + eps, h)
    sol = np.zeros_like(timesteps)
    sol[0] = y0
    it = 0
    for i in range(len(sol)-1):
        t = sym.symbols('t')
        f_sym = t - sol[i] + h * timesteps[i+1] * t**2
        f_prime_sym = sym.diff(f_sym,t)
        f = sym.lambdify(t, f_sym, 'numpy')
        f_prime = sym.lambdify(t, f_prime_sym, 'numpy')
        sol[i+1] = newton(f, f_prime, sol[i], tol, nmax)
    return sol, timesteps

l = -5.
t0 = 0.
tf = 2.
y0 = 1.
s = np.linspace(t0,tf,100)
h=0.1
#h=0.01
f_exact = lambda x: 2/(2 + x**2)
f_fe, t = fe(t0, tf, h, y0)
f_be, t = be(t0, tf, h, y0)
_ = plot(t, f_fe, 'r', label="forward euler")
_ = plot(t, f_be, 'g', label="backward euler")
_ = plot(t, f_exact(t), 'b', label="exact")
legend()
error1 = max(abs(f_exact(t) - f_fe))
error2 = max(abs(f_exact(t) - f_be))
print(error1, error2)
```

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from sympy import Matrix, eye, ones
from graphviz import Digraph, Graph
import graphviz
```

```python
def is_binary(M): return all([M[i,j] == 0 or M[i,j] == 1 for j in range(M.shape[1]) for i in range(M.shape[0])])
def is_symmetric(M): return M == M.T

def plot(G):
    if is_symmetric(G).all(): return plot_graph(G)
    else: return plot_digraph(G)

def plot_graph(G):
    dot = Graph(comment="Matrix graph", engine='sfdp', format='png')
    dot.attr(size='6,6')
    dot.attr(overlap='false')
    dot.attr(fontsize='12')
    is_bin = is_binary(G)
    for i in range(G.shape[0]):
        dot.node(str(i), str(i))
        for j in range(i, G.shape[1]):
            if G[i,j] != 0:
                if is_bin: dot.edge(str(i), str(j))
                else: dot.edge(str(i), str(j), str(G[i,j]))
    return graphviz.Source(dot)

def plot_digraph(G):
    dot = Digraph(comment="Matrix graph", engine='sfdp', format='png')
    dot.attr(size='6,6')
    dot.attr(overlap='false')
    dot.attr(fontsize='12')
    is_bin = is_binary(G)
    for i in range(G.shape[0]):
        dot.node(str(i), str(i))
        for j in range(G.shape[1]):
            if G[i,j] != 0:
                if is_bin: dot.edge(str(i), str(j))
                else: dot.edge(str(i), str(j), str(G[i,j]))
    return graphviz.Source(dot)

def plot_mst(G, MST):
    dot = Graph(comment="Matrix graph", engine='sfdp', format='png')
    dot.attr(size='6,6')
    dot.attr(overlap='false')
    dot.attr(fontsize='12')
    is_bin = is_binary(G)
    for i in range(G.shape[0]):
        dot.node(str(i), str(i))
        for j in range(i, G.shape[1]):
            if G[i,j] != 0:
                if is_bin: dot.edge(str(i), str(j), color='red' if MST[i, j] != 0 else 'black')
                else: dot.edge(str(i), str(j), str(G[i,j]), color='red' if MST[i, j] != 0 else 'black')
    return graphviz.Source(dot)

def generate_undirect_graph(n):
    G = np.array([ np.floor(np.random.random(n)*10) for _ in range(n)])
    G = G - np.diagonal(G) * np.eye(n)
    return G * G.T
```

```python
%time

def mst(G):
    if not is_symmetrical(G):
        raise ValueError('G must be undirected.')
    n = G.shape[0]
    #G = generate_undirect_graph(n)
    V = list(range(G.shape[0]))
    D = {(x,y):G[x,y] for x in range(G.shape[0]) for y in range(G.shape[1]) if G[x,y] != 0}
    D = sorted(D.items(), key=lambda kv: kv[1], reverse=True)
    MST = np.reshape(np.zeros(n*n), (n,n))
    weight = 0
    while len(D) > 0 and not all(V):
        k, d = D.pop(); x, y = k
        if V[x] != V[y]:
            mi, ma = min(V[x], V[y]), max(V[x], V[y])
            V[x] = V[y] = mi
            MST[x,y] = d; MST[y,x] = d
            weight += d
            for i in range(len(V)):
                if V[i] == ma: V[i] = mi

    return MST, weight
```

    Wall time: 0 ns

```python
MST, weight = mst(G)
print('Weight of MST is {}'.format(weight))
```

    Weight of MST is 81.0

# Weighted undirected graph

```python
plot(G)
```

```python
plot_mst(G, MST)
```

```python
plot(MST)
```

# Unweighted undirected graph

```python
M = np.array(np.ones(n) - np.eye(n))
M
```

    array([[0., 1., 1., 1., 1., 1., 1., 1.],
           [1., 0., 1., 1., 1., 1., 1., 1.],
           [1., 1., 0., 1., 1., 1., 1., 1.],
           [1., 1., 1., 0., 1., 1., 1., 1.],
           [1., 1., 1., 1., 0., 1., 1., 1.],
           [1., 1., 1., 1., 1., 0., 1., 1.],
           [1., 1., 1., 1., 1., 1., 0., 1.],
           [1., 1., 1., 1., 1., 1., 1., 0.]])

```python
MST, weight = mst(M)
plot_mst(M, MST)
```


```python
# https://colab.research.google.com/github/kassbohm/wb-snippets/blob/master/ipynb/TEM_11/Selbst/2.6.ipynb

from sympy import *

EI, q1, q2, l = var("EI, q1, q2, l")
C1, C2 = var("C1, C2")
x, xi = var("x, xi")

pprint("\nA:")
A = q1/2*l + (q2 - q1)/6*l
pprint(A)

qx = (q2 - q1)*x/l + q1
Rx = (qx - q1)*x/2
R1 = q1*x

Mx = -R1*x/2 - Rx*x/3 + x*A
tmp = Mx
tmp = apart(tmp,x)

pprint("\nM(x):")
# - EI w'' = M(x):
Mx = -R1*x/2 - Rx*x/3 + x*A
tmp = apart(Mx,x)
pprint(tmp)

# - EI w':
tmp = integrate(tmp,x) + C1
# - EI w:
tmp = integrate(tmp,x) + C2

pprint("\n\n- EI w / l⁴:")
w = - tmp/EI
tmp = - EI*w/l**4
tmp = tmp.expand()
tmp = tmp.subs(x, xi*l)
pprint(tmp)
pprint("\nwith  ξ = x/l.")

# eq1: w(x=0)=0
eq1 = Eq( w.subs(x,0), 0)
# eq2: w(x=l)=0
eq2 = Eq( w.subs(x,l), 0)

sol = solve([eq1, eq2],[C1, C2])

C1s = sol[C1]
C2s = sol[C2]

pprint("\nC1:")
pprint(C1s)

pprint("\nC2:")
pprint(C2s)

w = w.subs({C1: C1s, C2: C2s})

pprint("\n\n360 EI w / l⁴:")
w = w * 360 * EI / l**4
w = w.subs(x, xi*l)
w = w.expand()
pprint(collect(w,xi))

# A:
# l⋅q₁     ⎛  q₁   q₂⎞
# ──── + l⋅⎜- ── + ──⎟
#  2       ⎝  6    6 ⎠
#
# M(x):
#       2                      3
#   q₁⋅x      ⎛l⋅q₁   l⋅q₂⎞   x ⋅(q₁ - q₂)
# - ───── + x⋅⎜──── + ────⎟ + ────────────
#     2       ⎝ 3      6  ⎠       6⋅l
#
#
# - EI w / l⁴:
#                 5       4       3       5       3
# C₁⋅ξ   C₂   q₁⋅ξ    q₁⋅ξ    q₁⋅ξ    q₂⋅ξ    q₂⋅ξ
# ──── + ── + ───── - ───── + ───── - ───── + ─────
#   3     4    120      24      18     120      36
#  l     l
#
# with  ξ = x/l.
#
# C1:
#   3
# -l ⋅(8⋅q₁ + 7⋅q₂)
# ──────────────────
#        360
#
# C2:
# 0
#
#
# 360 EI w / l⁴:
#        4    5                   3
# 15⋅q₁⋅ξ  + ξ ⋅(-3⋅q₁ + 3⋅q₂) + ξ ⋅(-20⋅q₁ - 10⋅q₂) + ξ⋅(8⋅q₁ + 7⋅q₂)

```


```python
import numpy as np
import matplotlib.pyplot as plt
```

## Superposition of waves

### Example: 1
\begin{equation}
y=a+a\cos2\theta+a\cos4\theta +a\cos6\theta+... +\cos2n\theta+..............
\end{equation}

\begin{equation}
y=a\sum_{n=0}^N{\cos2n\theta}
\end{equation}

```python
step = 1000  #step
a=50  #amplitude
theta = np.arange(0,4*np.pi,4*np.pi/step)
```

```python
plt.figure(figsize = [20,6])
for n in range(4):
    y = a*np.cos(2*n*theta)
    plt.plot(theta,y, label= "a*cos("+str(2*n)+r"$\theta$)")

plt.xlabel("theta")
plt.ylabel("Displacement")
plt.legend()
plt.show()
```

```python
plt.figure(figsize = [15,4])
y =a* sum([np.sin(2*n*theta) for n in [i for i in range(6)]])
plt.plot(theta,y)
plt.xlabel(r"$\theta$",fontsize =14,color='b')
plt.ylabel("Displacement",fontsize =14,color='b')
plt.title('Superpostion of Waves ',fontsize =18,color='r')
#plt.tight_layout()
plt.savefig("image/wave1.pdf") # dpi dot per inch
plt.show()
```

### Example: 2
\begin{equation}
y=a\sin\theta+a\sin3\theta +a\sin5\theta+... +a\sin(2n+1)\theta+.......
\end{equation}

\begin{equation}
y=a\sum_{n=0}^N{\sin(2n+1)\theta}
\end{equation}

```python
plt.figure(figsize = [20,6])
for n in range(4):
    y = a*np.sin((2*n+1)*theta)
    plt.plot(theta,y, label= "a*sin("+str(2*n+1)+r"$\theta$)")

plt.xlabel("theta ")
plt.ylabel("y")
plt.legend()
plt.show()
```

```python
plt.figure(figsize = [15,4])
y = sum([a*np.sin((2*n+1)*theta) for n in [i for i in range(6)]])
plt.plot(theta,y,label ="wave")
plt.legend()
plt.xlabel(r"$\theta$",fontsize =14,color='b')
plt.ylabel("y",fontsize =14,color='b')
plt.title('Superposition of Waves ',fontsize =18,color='r')
plt.tight_layout()
plt.savefig("image/wave2.pdf", dpi = 300) # dpi dot per inch
plt.show()
```

---------------

## Fourier Series:
\begin{equation}
f(x)=\frac{a_0}{2}+\sum_{n=1}^\infty{[a_n\cos(\frac{n\pi x}{L})+b_n\sin(\frac{n\pi x}{L})]}
\end{equation}

### 1 Full wave rectifier
\begin{equation}
f(x)=\frac{2}{\pi}-\frac{4}{\pi}\sum_{n=0}^N{\frac{\cos nx}{n^2-1}}
\end{equation}
here n is even

```python
x=np.arange(0,4*np.pi,4*np.pi/step)
```

```python
plt.figure(figsize = [20,6])
for i in range(6):
    n=2*i
    y = (2/np.pi)-(4/np.pi)*np.cos(n*x)/(n**2-1)
    plt.plot(x,y, label= "2/pi - 4/pi"+str(1/(n**2-1))+"sin("+str(n)+"x)")

plt.xlabel("x")
plt.ylabel("f(x)")
plt.legend()
plt.show()
```

```python
plt.figure(figsize = [15,4])
y =(2/np.pi)-(4/np.pi)* sum([np.cos(n*x)/(n**2-1) for n in [i for i in range(10)if i%2 ==0 ]])
plt.plot(theta,y)
plt.xlabel("x",fontsize =14,color='b')
plt.ylabel("f(x)",fontsize =14,color='b')
plt.title('Full wave rectifier ',fontsize =18,color='r')
plt.tight_layout()
plt.savefig("image/fullwave.pdf", dpi = 300) # dpi dot per inch
plt.show()
```

### 2 well
\begin{equation}
f(x)=\frac{\pi ^2}{3}+4\sum_{n=1}^N{\frac{(-1)^n}{n^2}\cos nx}
\end{equation}

```python
plt.figure(figsize = [20,6])
for n in range(1,6):
    y = (np.pi**2/3)+4*np.cos(n*x)/n**2
    plt.plot(x,y, label= r"$\pi^2$/3+ 4"+str((-1)**n/n**2)+"cos("+str(n)+"x)")

plt.xlabel("x")
plt.ylabel("f(x)")
plt.legend()
plt.show()
```

```python
plt.figure(figsize = [15,4])
y =(np.pi**2/3)+4* sum([np.cos(n*x)/n**2 for n in [i for i in range(1,10)]])
plt.plot(x,y)
plt.xlabel("x",fontsize =14,color='b')
plt.ylabel("f(x)",fontsize =14,color='b')
plt.title('Well',fontsize =18,color='r')
plt.tight_layout()
plt.savefig("image/well.pdf", dpi = 300) # dpi dot per inch
plt.show()
```

### 3 Home Work(Triangular wave)
\begin{equation}
f(x)=\frac{8}{\pi ^2}\sum_{n=1,3,5..}^N{\frac{(-1)^{(n-1)/2}}{n^2}\sin(\frac{n\pi x}{L})}
\end{equation}

```python
L = 1
xs =1000
x = np.arange(0,2,10/xs)
plt.figure(figsize = [15,4])
for n in [i for i in range(10) if i%2 !=0]:
    y = (8/np.pi**2)*(-1)**(0.5*(n-1))*np.sin(n*np.pi*x/float(L))/n**2
    plt.plot(x,y, label = str(n)+" wave")
plt.xlabel('x')
plt.ylabel('f(x)')
plt.legend()
plt.show()
```

```python
x = np.arange(0,10,10/xs)
plt.figure(figsize = [15,4])
y = (8/np.pi**2)*sum([(-1)**(0.5*(n-1))*np.sin(n*np.pi*x/float(L))/n**2 for n in [i for i in range(20) if i%2 !=0]])
plt.plot(x,y)
plt.xlabel("x",fontsize =14,color='b')
plt.ylabel("f(x)",fontsize =14,color='b')
plt.title('Triangular wave',fontsize =18,color='r')
plt.tight_layout()
plt.savefig("image/triangularl.pdf", dpi = 300) # dpi dot per inch
plt.show()
```

```python

```