```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

The features from a particular dataset that will ultimately prove important for
machine learning can be difficult to know ahead of time.  This is especially
true for problems that do not have a strong physical underpinning. The
row-dimension of the input matrix ($X$) for fitting data in Scikit-learn is the
number of samples and the column dimension is the number of features. There may
be a large number of column dimensions in this matrix, and the purpose of
dimensionality reduction is to somehow reduce these to only those columns that
are important for the machine learning task.

Fortunately, Scikit-learn provides some powerful tools to help uncover the most
relevant features.  Principal Component Analysis (PCA) consists of taking
the input $X$ matrix and (1) subtracting the mean, (2) computing the covariance
matrix, and (3) computing the eigenvalue decomposition of the covariance
matrix. For example, if $X$ has more columns than is practicable for a
particular learning method, then PCA can reduce the number of columns to a more
manageable number.  PCA is widely used in statistics and other areas beyond
machine learning, so it is worth examining what it does in some detail. First,
we need the decomposition module from Scikit-learn.

```python
from sklearn import decomposition
import numpy as np
pca = decomposition.PCA()
```

 Let's create some very simple data and apply PCA.

```python
x = np.linspace(-1,1,30)
X = np.c_[x,x+1,x+2] # stack as columns
pca.fit(X)
print(pca.explained_variance_ratio_)
```

    [1.00000000e+00 2.73605815e-32 8.35833807e-34]

**Programming Tip.**

The `np.c_` is a shorcut method for creating stacked column-wise arrays.

 In this example, the columns are just constant offsets of the first
column. The *explained variance ratio* is the percentage of the variance
attributable to the transformed columns of `X`. You can think of this as the
information that is relatively concentrated in each column of the transformed
matrix `X`. [Figure](#fig:pca_001) shows the graph of this dominant
transformed column in the bottom panel.  Note that a constant offset in each of
the columns does not change its respective variance and thus, as far as PCA  is
concerned, the three columns are identical from an information standpoint.

```python
%matplotlib inline

from matplotlib.pylab import subplots
fig,axs = subplots(2,1,sharex=True,sharey=True)
ax = axs[0]
_=ax.plot(x,X[:,0],'-k',lw=3)
_=ax.plot(x,X[:,1],'--k',lw=3)
_=ax.plot(x,X[:,2],':k',lw=3)
ax=axs[1]
_=ax.plot(x,pca.fit_transform(X)[:,0],'-k',lw=3)
#ax.tick_params(labelsize='x-large')
fig.savefig('fig-machine_learning/pca_001.png')
```

The top panel shows the columns of the feature matrix and the bottom panel shows the dominant component that PCA has extracted.

To make this more interesting, let's change the slope of each of the columns as
in the following,

```python
X = np.c_[x,2*x+1,3*x+2,x] # change slopes of columns
pca.fit(X)
print(pca.explained_variance_ratio_)
```

    [1.00000000e+00 3.26962032e-33 3.78960782e-34 2.55413064e-35]

 However, changing the slope did not impact the explained variance
ratio. Again, there is still only one dominant column. This means that PCA is
invariant to both constant offsets and scale changes. This works for functions
as well as simple lines,

```python
x = np.linspace(-1,1,30)
X = np.c_[np.sin(2*np.pi*x),
          2*np.sin(2*np.pi*x)+1,
          3*np.sin(2*np.pi*x)+2]
pca.fit(X)
print(pca.explained_variance_ratio_)
```

    [1.00000000e+00 3.70493694e-32 2.51542007e-33]

 Once again, there is only one dominant column, which is shown in the
bottom panel of [Figure](#fig:pca_002). The top panel shows the individual
columns of the feature matrix. To sum up, PCA is able to identify and eliminate
features that are merely linear transformations of existing features. This also
works when there is additive noise in the features, although more samples are
needed to separate the uncorrelated noise from between features.

The top panel shows the columns of the feature matrix and the bottom panel shows the dominant component that PCA has computed.

```python
fig,axs = subplots(2,1,sharex=True,sharey=True)
ax = axs[0]
_=ax.plot(x,X[:,0],'-k')
_=ax.plot(x,X[:,1],'--k')
_=ax.plot(x,X[:,2],':k')
ax=axs[1]
_=ax.axis(xmin=-1,xmax=1)
_=ax.plot(x,pca.fit_transform(X)[:,0],'-k')
# ax.tick_params(labelsize='x-large')
fig.savefig('fig-machine_learning/pca_002.png')
```

```python
fig,axs=subplots(1,2,sharey=True)
ax=axs[0]
_=ax.axis(xmin=-2,xmax=12)
x1 = np.arange(0, 10, .01/1.2)
x2 = x1+ np.random.normal(loc=0, scale=1, size=len(x1))
X = np.c_[(x1, x2)]
good = (x1>5) | (x2>5)
bad = ~good
_=ax.plot(x1[good],x2[good],'o',alpha=.3,color='gray')
_=ax.plot(x1[bad],x2[bad],'ok',alpha=.3)
_=ax.set_title("original data space")
_=ax.set_xlabel("x")
_=ax.set_ylabel("y")

_=pca.fit(X)
Xx=pca.fit_transform(X)
ax=axs[1]
# ax.set_aspect(1/1.6)
_=ax.plot(Xx[good,0],Xx[good,1]*0,'o',alpha=.3,color='gray')
_=ax.plot(Xx[bad,0],Xx[bad,1]*0,'ok',alpha=.3)
_=ax.set_title("PCA-reduced data space")
_=ax.set_xlabel(r"$\hat{x}$")
_=ax.set_ylabel(r"$\hat{y}$")
fig.savefig('fig-machine_learning/pca_003.png')
```

To see how PCA can simplify machine learning tasks, consider [Figure](#fig:pca_003) wherein the two classes are separated along the diagonal.
After PCA, the transformed data lie along a single axis where the two classes
can be split using a one-dimensional interval, which greatly simplifies the
classification task. The class identities are preserved under PCA because the
principal component is along the same direction that the classes are separated.
On the other hand, if the classes are separated along the direction
*orthogonal* to the principal component, then the two classes become mixed
under PCA and the classification task becomes much harder. Note that in both
cases, the `explained_variance_ratio_` is the same because the explained
variance ratio does not account for class membership.

The left panel shows the original two-dimensional data space of two easily distinguishable classes and the right panel shows the reduced the data space transformed using PCA. Because the two classes are separated along the principal component discovered by PCA, the classes are  preserved under the transformation.

```python

fig,axs=subplots(1,2,sharey=True)
ax=axs[0]
x1 = np.arange(0, 10, .01/1.2)
x2 = x1+np.random.normal(loc=0, scale=1, size=len(x1))
X = np.c_[(x1, x2)]
good = x1>x2
bad = ~good
_=ax.plot(x1[good],x2[good],'ow',alpha=.3)
_=ax.plot(x1[bad],x2[bad],'ok',alpha=.3)
_=ax.set_title("original data space")
_=ax.set_xlabel("x")
_=ax.set_ylabel("y")

_=pca.fit(X)
Xx=pca.fit_transform(X)
ax=axs[1]
_=ax.plot(Xx[good,0],Xx[good,1]*0,'ow',alpha=.3)
_=ax.plot(Xx[bad,0],Xx[bad,1]*0,'ok',alpha=.3)
_=ax.set_title("PCA-reduced data space")
_=ax.set_xlabel(r"$\hat{x}$")
_=ax.set_ylabel(r"$\hat{y}$")

fig.savefig('fig-machine_learning/pca_004.png')
```

As compared with [Figure](#fig:pca_003), the two classes differ along the coordinate direction that is orthogonal to the principal component. As a result, the two classes are no longer distinguishable after transformation.

PCA works by decomposing the covariance matrix of the data using the Singular
Value Decomposition (SVD). This decomposition exists for all matrices and
returns the following factorization for an arbitrary matrix $\mathbf{A}$,

$$
\mathbf{A} =  \mathbf{U} \mathbf{S}  \mathbf{V}^T
$$

 Because of the symmetry of the covariance matrix, $\mathbf{U} =
\mathbf{V}$. The elements of the diagonal matrix $\mathbf{S}$ are the singular
values of $\mathbf{A}$ whose squares are the eigenvalues of $\mathbf{A}^T
\mathbf{A}$. The eigenvector matrix $\mathbf{U}$ is orthogonal: $\mathbf{U}^T
\mathbf{U} =\mathbf{I}$. The singular values are in decreasing order so that
the first column of $\mathbf{U}$ is the axis corresponding to the largest
singular value. This is the first dominant column that PCA identifies. The
entries of the covariance matrix are of the form $\mathbb{E}(x_i x_j)$ where
$x_i$ and $x_j$ are different features [^covariance]. This means that the
covariance matrix is filled with entries that attempt to uncover mutually
correlated relationships between all pairs of columns of the feature matrix.
Once these have been tabulated in the covariance matrix, the SVD finds optimal
orthogonal transformations to align the components along the directions most
strongly associated with these correlated relationships. Simultaneously,
because orthogonal matrices have columns of unit-length, the SVD collects the
absolute squared lengths of these components into the $\mathbf{S}$ matrix.  In
our example above in [Figure](#fig:pca_003), the two feature vectors were obviously correlated along the
diagonal, meaning that PCA selected that diagonal direction as the principal
component.

[^covariance]: Note that these entries are constructed from the data
using an estimator of the covariance matrix because we do not have
the full probability densities at hand.

We have seen that PCA is a powerful dimension reduction method that is
invariant to linear transformations of the original feature space. However,
this method performs poorly with transformations that are nonlinear. In that
case, there are a wide range of extensions to PCA, such as Kernel PCA, that are
available in Scikit-learn, which allow for embedding parameterized
non-linearities into the PCA at the risk of overfitting.

## Independent Component Analysis

Independent Component Analysis (ICA) via the `FastICA` algorithm is also
available in Scikit-learn. This method is fundamentally different from PCA
in that it is the small differences between components that are emphasized,
not the large principal components.  This method is adopted from signal
processing.  Consider a matrix of signals ($\mathbf{X}$) where the rows are
the samples and the columns are the different signals. For example, these
could be EKG signals from multiple leads on a single patient. The analysis
starts with the following model,

$$
\begin{equation}
\mathbf{X} =  \mathbf{S}\mathbf{A}^T
\label{eq:ICA} \tag{1}
\end{equation}
$$

 In other words, the observed signal matrix is an unknown mixture
($\mathbf{A}$) of some set of conformable, independent random sources
$\mathbf{S}$,

$$
\mathbf{S}=\left[ \mathbf{s}_1(t),\mathbf{s}_2(t),\ldots,\mathbf{s}_n(t)\right]
$$

 The distribution on the random sources is otherwise unknown, except
there can be at most one Gaussian source, otherwise, the mixing matrix
$\mathbf{A}$ cannot be identified because of technical reasons.  The problem in
ICA is to find $\mathbf{A}$ in Equation [1](#eq:ICA) and thereby un-mix the
$s_i(t)$ signals, but this cannot be solved without a strategy to reduce the
inherent arbitrariness in this formulation.

To make this concrete, let us simulate the situation with the following code,

```python
np.random.seed(123456)
```

```python
from numpy import matrix, c_, sin, cos, pi
t = np.linspace(0,1,250)
s1 = sin(2*pi*t*6)
s2 =np.maximum(cos(2*pi*t*3),0.3)
s2 = s2 - s2.mean()
s3 = np.random.randn(len(t))*.1
# normalize columns
s1=s1/np.linalg.norm(s1)
s2=s2/np.linalg.norm(s2)
s3=s3/np.linalg.norm(s3)
S =c_[s1,s2,s3] # stack as columns
# mixing matrix
A = matrix([[  1,  1,1],
            [0.5, -1,3],
            [0.1, -2,8]])
X= S*A.T # do mixing
```

```python
fig,axs=subplots(3,2,sharex=True)
fig.set_size_inches((8,8))
X = np.array(X)

_=axs[0,1].plot(t,-X[:,0],'k-')
_=axs[1,1].plot(t,-X[:,1],'k-')
_=axs[2,1].plot(t,-X[:,2],'k-')
_=axs[0,0].plot(t,s1,'k-')
_=axs[1,0].plot(t,s2,'k-')
_=axs[2,0].plot(t,s3,'k-')

_=axs[2,0].set_xlabel('$t$',fontsize=18)
_=axs[2,1].set_xlabel('$t$',fontsize=18)
_=axs[0,0].set_ylabel('$s_1(t)$        ',fontsize=18,rotation='horizontal')
_=axs[1,0].set_ylabel('$s_2(t)$        ',fontsize=18,rotation='horizontal')
_=axs[2,0].set_ylabel('$s_3(t)$        ',fontsize=18,rotation='horizontal')
for ax in axs.flatten():
    _=ax.yaxis.set_ticklabels('')

_=axs[0,1].set_ylabel('        $X_1(t)$',fontsize=18,rotation='horizontal')
_=axs[1,1].set_ylabel('        $X_2(t)$',fontsize=18,rotation='horizontal')
_=axs[2,1].set_ylabel('        $X_3(t)$',fontsize=18,rotation='horizontal')
_=axs[0,1].yaxis.set_label_position("right")
_=axs[1,1].yaxis.set_label_position("right")
_=axs[2,1].yaxis.set_label_position("right")

fig.savefig('fig-machine_learning/pca_008.png')
```

The left column shows the original signals and the right column shows the mixed signals. The object of ICA is to recover the left column from the right.

 The individual signals ($s_i(t)$) and their mixtures ($X_i(t)$) are
shown in [Figure](#fig:pca_008). To recover the individual signals using ICA,
we use the `FastICA` object and fit the parameters on the `X` matrix,

```python
from sklearn.decomposition import FastICA
ica = FastICA()
# estimate unknown S matrix
S_=ica.fit_transform(X)
```

 The results of this estimation are shown in [Figure](#fig:pca_009),
showing that ICA is able to recover the original signals from the observed
mixture. Note that ICA is unable to distinguish the signs of the recovered
signals or preserve the order of the input signals.

```python
fig,axs=subplots(3,2,sharex=True)
fig.set_size_inches((8,8))
X = np.array(X)

_=axs[0,1].plot(t,-S_[:,2],'k-')
_=axs[1,1].plot(t,-S_[:,1],'k-')
_=axs[2,1].plot(t,-S_[:,0],'k-')
_=axs[0,0].plot(t,s1,'k-')
_=axs[1,0].plot(t,s2,'k-')
_=axs[2,0].plot(t,s3,'k-')

_=axs[2,0].set_xlabel('$t$',fontsize=18)
_=axs[2,1].set_xlabel('$t$',fontsize=18)
_=axs[0,0].set_ylabel('$s_1(t)$        ',fontsize=18,rotation='horizontal')
_=axs[1,0].set_ylabel('$s_2(t)$        ',fontsize=18,rotation='horizontal')
_=axs[2,0].set_ylabel('$s_3(t)$        ',fontsize=18,rotation='horizontal')
for ax in axs.flatten():
    _=ax.yaxis.set_ticklabels('')

_=axs[0,1].set_ylabel('        $s_1^\prime(t)$',fontsize=18,rotation='horizontal')
_=axs[1,1].set_ylabel('        $s_2^\prime(t)$',fontsize=18,rotation='horizontal')
_=axs[2,1].set_ylabel('        $s_3^\prime(t)$',fontsize=18,rotation='horizontal')
_=axs[0,1].yaxis.set_label_position("right")
_=axs[1,1].yaxis.set_label_position("right")
_=axs[2,1].yaxis.set_label_position("right")

fig.savefig('fig-machine_learning/pca_009.png')
```

The left column shows the original signals and the right column shows the signals that ICA was able to recover. They match exactly, outside of a possible sign change.

To develop some intuition as to how ICA accomplishes this feat, consider the
following two-dimensional situation with two uniformly distributed independent
variables, $u_x,u_y \sim \mathcal{U}[0,1]$. Suppose we apply the
following orthogonal rotation matrix to these variables,

$$
\begin{bmatrix}
u_x^\prime \\\
u_y^\prime
\end{bmatrix}=
\begin{bmatrix}
\cos(\phi) & -\sin(\phi) \\\
\sin(\phi) & \cos(\phi)
\end{bmatrix}
\begin{bmatrix}
u_x \\\
u_y
\end{bmatrix}
$$

The left panel shows two classes labeled on the $u_x,u_y$ uniformly independent random variables. The right panel shows these random variables after a rotation, which removes their mutual independence and makes it hard to separate the two classes along the coordinate directions.

 The so-rotated variables $u_x^\prime,u_y^\prime$ are no longer
independent, as shown in [Figure](#fig:pca_005). Thus, one way to think about
ICA is as a search through orthogonal matrices so that the independence is
restored. This is where the prohibition against Gaussian distributions arises.
The two dimensional Gaussian distribution of independent variables is
proportional the following,

$$
f(\mathbf{x})\propto\exp(-\frac{1}{2}\mathbf{x}^T \mathbf{x})
$$

 Now, if we similarly rotated the $\mathbf{x}$ vector as,

$$
\mathbf{y} = \mathbf{Q} \mathbf{x}
$$

 the resulting density for $\mathbf{y}$ is obtained by plugging in
the following,

$$
\mathbf{x} = \mathbf{Q}^T \mathbf{y}
$$

 because the inverse of an orthogonal matrix is its transpose, we obtain

$$
f(\mathbf{y})\propto\exp(-\frac{1}{2}\mathbf{y}^T \mathbf{Q}\mathbf{Q}^T \mathbf{y})=\exp(-\frac{1}{2}\mathbf{y}^T \mathbf{y})
$$

 In other words, the transformation is lost on the $\mathbf{y}$
 variable. This means that ICA cannot search over orthogonal transformations if
 it is blind to them, which explains the restriction of Gaussian random
 variables. Thus, ICA is a method that seeks to maximize the non-Gaussian-ness
 of the transformed random variables.  There are many methods to doing this,
 some of which involve cumulants and others that use the
*negentropy*,

$$
\mathcal{J}(Y) = \mathcal{H}(Z)-\mathcal{H}(Y)
$$

 where $\mathcal{H}(Z)$ is the information entropy of the
Gaussian random variable $Z$ that has the same variance as $Y$. Further
details would take us beyond our scope, but that is the outline of how
the FastICA algorithm works.

The implementation of this method in Scikit-learn includes two different ways
of extracting more than one independent source component. The *deflation*
method iteratively extracts one component at a time using a incremental
normalization step. The *parallel* method also uses the single-component method
but carries out normalization of all the components simultaneously, instead of
for just the newly computed component.  Because ICA extracts
independent components, a whitening step is used beforehand to balance the
correlated components from the data matrix. Whereas PCA returns
uncorrelated components along dimensions optimal for Gaussian random
variables, ICA returns components that are as far from the Gaussian density
as possible.

The left panel on [Figure](#fig:pca_005) shows the orignal uniform random
sources. The white and black colors distinguish between two classes. The right
panel shows the mixture of these sources, which is what we observe as input
features. The top row of [Figure](#fig:pca_006) shows the PCA (left) and ICA
(right) transformed data spaces.  Notice that ICA is able to un-mix the two
random sources whereas PCA transforms along the dominant diagonal. Because ICA
is able to preserve the class membership, the data space can be reduced to two
non-overlapping sections, as shown. However, PCA cannot achieve a similiar
separation because the classes are mixed along the dominant diagonal that PCA
favors as the main component in the decomposition.

The panel on the top left shows two classes in a plane after a  rotation. The bottom left panel shows the result of dimensionality reduction using PCA, which causes mixing between the two classes. The top right panel shows the ICA transformed output and the lower right panel shows that, because ICA was able to un-rotate the data, the lower dimensional data maintains the separation between the classes.

For a good principal component analysis treatment, see  [[richert2013building]](#richert2013building),
[[alpaydin2014introduction]](#alpaydin2014introduction), [[cuesta2013practical]](#cuesta2013practical), and
[[izenman2008modern]](#izenman2008modern). Independent Component Analysis is discussed in more detail
in [[hyvarinen2004independent]](#hyvarinen2004independent).


# Question 1

Considérons le problème
\begin{align*}
\min_{x, y}\ & x+y \\
\text{tel que } & x - y = 1 \\
& -y = 0
\end{align*}
Résolvez le problème en appliquant la méthode à deux phases.
Que pouvez-vous en conclure?

## Solution

Notons que le problème a pour solution triviale $(1,0)$.
Appliquons néanmoins la phase 1, en construisant le problème artificiel
\begin{align*}
\min_{x, y, s_1, s_2}\ & s_1+s_2 \\
\text{tel que } & x - y + s_1 = 1 \\
& -y + s_2 = 0
\end{align*}
Le tableau du simplexe donne
$$
\begin{matrix}
1 & -1 & 1 & 0 & 1 \\
0 & -1 & 0 & 1 & 0 \\
-1 & 2 & 0 & 0 & -1
\end{matrix}
$$
Dès lors $x$ entre la base et $s_1$ en sort.
Après pivot, nous avons
$$
\begin{matrix}
1 & -1 & 1 & 0 & 1 \\
0 & -1 & 0 & 1 & 0 \\
0 & 1 & 1 & 0 & 0
\end{matrix}
$$
Le simplexe s'arrête donc avec la solution de base $x = 1$, $s_2 = 0$, et la valeur optimale $0$ pour le problème artificielle.
$y$ n'est pas dans la solution de base, mais le problème est néanmoins réalisable car la fonction objectif du problème artificielle est nulle, ce qui est possible car la solution trouvée est dégénérée. Comme $y$ est hors base, $y = 0$ et donc nous avons comme solution réalisable du problème original $(1,0)$.
On peut faire un pivot pour faire entrer $y$ dans la base et sortir $s_2$. Le pivot donne le tableau
$$
\begin{matrix}
1 & 0 & 1 & -1 & 1 \\
0 & 1 & 0 & -1 & 0 \\
0 & 0 & 1 & 1 & 0
\end{matrix}
$$
Si on part de ce tableau pour le problème de départ, nous obtenons
$$
\begin{matrix}
1 & 0 & 1 \\
0 & 1 & 0 \\
1 & 1 & 0
\end{matrix}
$$
et après annulation des coûts réduits
$$
\begin{matrix}
1 & 0 & 1 \\
0 & 1 & 0 \\
0 & 0 & -1
\end{matrix}
$$
Nous retrouvons bien la solution optimale $(1,0)$.

## Meilleure solution

Nous avons ignoré le fait que $x$ est une variable isolée. En exploitant ce point, nous contruisons le problème artificiel
\begin{align*}
\min_{x, y, s}\ & s \\
\text{tel que } & x - y = 1 \\
& -y + s = 0
\end{align*}
Le tableau du simplexe donne alors
$$
\begin{matrix}
1 & -1 & 0 & 1 \\
0 & -1 & 1 & 0 \\
0 & 0 & 1 & 0
\end{matrix}
$$
et après annulation du coût réduit de $s$,
$$
\begin{matrix}
1 & -1 & 0 & 1 \\
0 & -1 & 1 & 0 \\
0 & 1 & 0 & 0
\end{matrix}
$$

Tous les coûts réduits sont positifs, et on retrouve la solution $x = 1$, $s = 0$. À nouveau, $y$ n'est pas dans la base, qui est cependant dégénérée, et comme précédemment, nous pouvons effectuer un pivot pour l'insérer.

$$
\begin{matrix}
1 & 0 & -1 & 1 \\
0 & 1 & -1 & 0 \\
0 & 0 & 1 & 0
\end{matrix}
$$

La phase 2 se déroule alors comme précédemment.

# Question 2: intra A05

Soit le problème suivant
\begin{align*}
&\min\ z = -8x-6y\\
&\begin{aligned}
\text{s.à. }
& 5x + 3y \le 30,\\
&x+3y \le 18,\\
& x,y \ge 0.
\end{aligned}
\end{align*}

## (a)

$\textbf{Résoudre le problème graphiquement.
}$

```julia
using Plots
```

    (process:15068): GLib-GIO-WARNING **: 14:04:29.214: Unexpectedly, UWP app `6760NGPDFLab.PDFX_1.2.9.0_x64__sbe4t8mqwq93a' (AUMId `6760NGPDFLab.PDFX_sbe4t8mqwq93a!App') supports 45 extensions but has no verbs

```julia
g1 = (x -> 10-5/3*x ≥ 0 ? 10-5/3*x : NaN)  # NaN: Not a Number
g2 = (x -> 6-1/3*x ≥ 0 ? 6-1/3*x : NaN)
g3 = (x -> 0)
objOpt = (x -> -4/3*x+9 ≥ 0 ? -4/3*x+9 : NaN)
```

    #7 (generic function with 1 method)

```julia
x1=0:0.01:7
pyplot()
plot()

plot(x1,g1,fillrange = 0,
         fillalpha = 0.3,
         fillcolor = :blue,
         color=:blue,
         label="y=10-5/3*x")

plot!(x1,g2,fillrange = 0,
         fillalpha = 0.3,
         fillcolor = :red,
         color=:red,
         label="y=6-1/3*x")

plot!(x1,g3,fillrange = 0,
         fillalpha = 0.3,
         fillcolor = :green,
         color=:green,
         label="y=0")

plot!(x1,objOpt, color=:black, linewidth=2,
    linestyle=:solid, label="-8x-6y = 9")

scatter!([3],[5],label="Optimum", color=:red)

plot!(
    #size=(800, 600),
    xticks = 0:1:7,
    yticks = 0:1:10,

    ylabel = "y",
    xlabel = "x",

    legend=:topright,
    title  = "Programme linéaire"
    )
```

## (b)

$\textbf{Résoudre le problème avec la méthode du simplexe.
}$

Avant de résoudre le problème avec le simplexe, il faut le mettre sous forme standard.
\begin{align*}
&\min\ z = -8x-6y\\
&\begin{aligned}
\text{s.a. } & 5x + 3y + s_1 = 30,\\
&x+3y + s_2 = 18,\\
& x,y,s_1,s_2 \ge 0.
\end{aligned}
\end{align*}

\begin{align*}
\begin{array}{c|cccc|c}
    & x & y & s_1 & s_2 & \text{t.d.}\\
\hline
s_1 & \color{red}5 & 3 & 1 & 0 & 30\\
s_2 & 1 & 3 & 0 & 1 & 18\\
\hline
-z & -8 & -6 & 0 & 0 & 0
\end{array}
\end{align*}

$x$ entre et $s_1$ sort

\begin{align*}
\begin{array}{c|cccc|c}
    & x & y & s_1 & s_2 & \text{t.d.}\\
\hline
x & 1 & 3/5 & 1/5 & 0 & 6\\
s_2 & 0 & \color{red}{12/5} & -1/5 & 1 & 12\\
\hline
-z  & 0 & -6/5 & 8/5 & 0 & 48
\end{array}
\end{align*}

$y$ entre et $s_2$ sort

\begin{align*}
\begin{array}{c|cccc|c}
    & x & y & s_1 & s_2 & \text{t.d.}\\
\hline
x & 1 & 0 & 1/4 & -1/4 & 3\\
y & 0 & 1 & -1/12 & 5/12 & 5\\
\hline
-z & 0 & 0 & 3/2 & 1/2 & 54
\end{array}
\end{align*}

La solution obtenue est $x = 3$, $y=5$, $s_1 = s_2 = 0$ et la valeur optimale est $z^{\star} = -54$.

## (c)

$\textbf{Trouver l'inverse de la base optimale.
}$

L'inverse de la base optimale $B_{\star}^{-1} $ se lit directement dans le
dernier tableau du simplexe sous les colonnes associées au variable d'écart $s_1$
et $s_2$ introduites.
\begin{align*}
B_{\star}^{-1} =
\begin{pmatrix}
1/4 & -1/4\\
-1/12 & 5 /12
\end{pmatrix}
\end{align*}

## (d)

$\textbf{Trouver le vecteur des multiplicateurs pour la base optimale.
}$

Le vecteur des multiplicateurs se lit au signe près directement dans le dernier tableau du simplexe, les coûts
réduits des variables d'écarts. Autrement, si on note $\pi$ le vecteur des multiplicateurs optimaux, on rappelle
que $\pi = C_{B} B_{\star}^{-1}$ où $c_{B_\star} = \left(c_x, c_y \right) = \left(-8, -6 \right)$.
Ainsi, $\pi = \left(-3/2, -1/2 \right)$

## (e)

$\textbf{Est-ce que la base reste optimale si on change le terme de droite pour
}$
\begin{align*}
\tilde{b} =
\begin{pmatrix}
24\\
18
\end{pmatrix}?
\end{align*}

$B_{\star}^{-1} $ reste réalisable si $B_{\star}^{-1} \tilde{b} \ge 0$.
\begin{align*}
B_{\star}^{-1} \tilde{b} =
\begin{pmatrix}
3/2\\
11/2
\end{pmatrix}
\end{align*}
Les coûts réduits associées à cette base n'ont pas été modifiés. Il restent positifs et $B_{\star} $ demeure optimale.
La nouvelle valeur optimale est $\tilde{z} = -45$.

## (f)

$\textbf{Est-ce que la base reste optmimale si on change le terme de droite pour
}$
\begin{align*}
\bar{b} =
\begin{pmatrix}
30\\
16
\end{pmatrix}
\end{align*}

$B_{\star}^{-1} $ reste réalisable si $B_{\star}^{-1} \tilde{b} \ge 0$.
\begin{align*}
B_{\star}^{-1} \bar{b} =
\begin{pmatrix}
7/2\\
25/6
\end{pmatrix}
\end{align*}
Les coûts réduits associés à cette base n'ont pas été modifié. Il restent positifs. $B_{\star} $ demeure
optimale. La nouvelle valeur optimale est $\bar{z} = -53$.

## (g)

$\textbf{Ajoutons une nouvelle contrainte au problème qui devient alors}$
\begin{align*}
&\min\ z = -8x-6y\\
&\begin{aligned}
\text{s.a. } & 5x + 3y + s_1 = 30,\\
&x+3y+ s_2 = 18,\\
&y + s_3 = 5,\\
& x,y,s_1,s_2,s_3 \ge 0.
\end{aligned}
\end{align*}
$\textbf{Quelle est maintenant la solution de ce problème?}$

Au lieu de reprendre le simplexe, on exploite l'information associé à $B_{\star} $.
Pour ce nouveau problème, une base est de rang $3$. La nouvelle base est

\begin{align*}
\tilde{B} =
\begin{array}[b]{cccc}
x & y & s_3 \\
\left(
\begin{array}{c}
5\\
1\\
0
\end{array}
\right.
&
\begin{array}{c}
3\\
3\\
1
\end{array}
&
\left.
\begin{array}{c}
0\\
0\\
1
\end{array}
\right)
\end{array}
\end{align*}
Au lieu d'inverser $\tilde{B}$ puis de retrouver le tableau à partir du dernier tableau de la partie b,
on retrouvera le tableau associé à $\tilde{B}$.

\begin{align*}
\begin{array}{c|ccccc|c}
    & x & y & s_1 & s_2 & s_3 & \text{t.d.}\\
\hline
x & 1 & 0 & 1/4 & -1/4 & 0 & 3\\
y & 0 & 1 & -1/12 & 5/12 & 0 & 5\\
s_3 & 0 & 1 & 0 & 0 & 1 & 5\\
\hline
-z & 0 & 0 & 3/2 & 1/2 & 0 & 54
\end{array}
\end{align*}

Pour respecter le fait que $y$ est une variable de base, il faut rendre $y$ isolé, ce qui peut être obtenu en modifiant le tableau ainsi
\begin{align*}
\begin{array}{c|ccccc|c}
  & x & y & s_1 & s_2 & s_3 & \text{t.d.} \\
\hline
x & 1 & 0 & 1/4 & -1/4 & 0 & 3\\
y & 0 & 1 & -1/12 & 5/12 & 0 & 5\\
s_3 & 0 & 0 & 1/12 & -5/12 & 1 & 0 \\
\hline
-z & 0 & 0 & 3/2 & 1/2 & 0 & 54
\end{array}
\end{align*}

La solution obtenue est $x = 3$, $y=5$, $s_1 = s_2 = s_3 = 0$ et la valeur optimale est $z^{\star} = -54$.

# Question 3
$\textbf{Utiliser le simplexe révisé pour résoudre}$
\begin{align*}
&\min\ z = -4x_1 -3x_2 - x_3 -2x_4\\
&\begin{aligned}
\text{s.a. }
& 4x_1 + 2x_2 + x_3 + x_4 + x_5 = 5.\\
&3x_1 + x_2 + 2x_3 +x_4 +x_6 = 4.\\
&x_j \ge 0,\ j=1,\ldots,6.
\end{aligned}
\end{align*}

Soient
\begin{align*}
A=
\begin{pmatrix}
4 & 2 & 1 & 1 & 1 & 0 \\
3 & 1 & 2 & 1 & 0 & 1
\end{pmatrix}, \quad
b =
\begin{pmatrix}
5\\4
\end{pmatrix}
\end{align*}

$\textbf{Itération 1}$

Soit
\begin{align*}
B_0^{-1} =
\begin{pmatrix}
1 & 0\\
0 & 1
\end{pmatrix},
\end{align*}
Pour commencer, on trouve le nouveau multiplicateur $\pi$ et on calcul les coûts réduits
\begin{gather*}
\pi_0^t = c_B^tB_0^{-1} =
\begin{pmatrix}
0 & 0
\end{pmatrix}\\[6pt]
\bar{c}^t = c^t - \pi_0^t A =
\begin{pmatrix}
-4 & -3 & -1 & -2 & 0 & 0
\end{pmatrix}
\end{gather*}

La variable d'entrée est donc $x_1$.
Pour trouver la variable de sortie, on a besoin de la colonne de $A$ associée à $x_1$ que l'on notera $A_1$.
\begin{align*}
\bar{A}_1 = B_0^{-1} A_1 =
\begin{pmatrix}
4\\
3
\end{pmatrix}
\end{align*}
Comme $\min\left\{5/4, 4/3 \right\} = 5/4$, $x_5$ sort de la base.
On met à jour $B_0^{-1} $ et en même temps on met à jour les termes de droite et la valeur de l'objectif

\begin{align*}
\begin{array}{ccccc}
    & x_5 & x_6 & b & x_1\\
x_5 & 1 & 0 & 5 & {\color{red}4}\\
x_6 & 0 & 1 & 4 & 3\\
-z & 0 & 0 & 0 & -4
\end{array}
\end{align*}
On fait un pivot
\begin{align*}
\begin{array}{cccc}
 & x_5 & x_6 & b \\
x_1 & 1/4 & 0 & 5/4\\
x_6 & -3/4 & 1 & 1/4\\
-z  & 1 & 0 & 5
\end{array}
\end{align*}
On a maintenant
\begin{align*}
B_1^{-1} =
\begin{pmatrix}
1/4 & 0\\
-3/4 & 1
\end{pmatrix}
\end{align*}
et on recommence une nouvelle itération.

$\textbf{Itération 2}$

Pour commencer, on trouve le nouveau multiplicateur $\pi$ et on calcul les coûts réduits
\begin{gather*}
\pi_1^t = c_B^tB_1^{-1} =
\begin{pmatrix}
-1 & 0
\end{pmatrix}\\[6pt]
\bar{c}^t = c^t - \pi_1^t A =
\begin{pmatrix}
0 & -1 & 0 & -1 & 1 & 0
\end{pmatrix}
\end{gather*}

Prenons $x_2$ comme variable d'entrée.
Pour trouver la variable de sortie, on a besoin de la colonne de $A$ associée à $x_2$ que l'on notera $A_2$.
\begin{align*}
\bar{A}_2 = B_1^{-1} A_2 =
\begin{pmatrix}
1/2\\
-1/2
\end{pmatrix}
\end{align*}
Et $\min\left\{ (5/4)*2 \right\} = 5/2$, alors $x_1$ sort de la base.
On met à jour $B_1^{-1} $ et en même temps on met à jour les termes de droite et la valeur de l'objectif

\begin{align*}
\begin{array}{ccccc}
    & x_5 & x_6 & b & x_2\\
x_1 & 1/4 & 0 & 5/4 & \color{red}{1/2}\\
x_6 & -3/4 & 1 & 1/4 & -1/2\\
-z & 1 & 0 & 5 & -1
\end{array}
\end{align*}
On fait un pivot
\begin{align*}
\begin{array}{cccc}
 & x_5 & x_6 & b \\
x_2 & 1/2 & 0 & 5/2\\
x_6 & -1/2 & 1 & 3/2\\
-z  & 3/2 & 0 & 15/2
\end{array}
\end{align*}
On a maintenant
\begin{align*}
B_2^{-1} =
\begin{pmatrix}
1/2 & 0\\
-1/2 & 1
\end{pmatrix}
\end{align*}
et on recommence une nouvelle itération.

$\textbf{Itération 3}$
Pour commencer, on trouve le nouveau multiplicateur $\pi$ et on calcul les coûts réduits
\begin{gather*}
\pi_2^t = c_B^tB_2^{-1} =
\begin{pmatrix}
-3/2 & 0
\end{pmatrix}\\[6pt]
\bar{c}^t = c^t - \pi_2^t A =
\begin{pmatrix}
2 & 0 & 1/2 & -1/2 & 3/2 & 0
\end{pmatrix}
\end{gather*}

La variable d'entrée est donc $x_4$.
Pour trouver la variable de sortie, on a besoin de la colonne de $A$ associée à $x_4$ que l'on notera $A_4$.
\begin{align*}
\bar{A}_4 = B_2^{-1} A_4 =
\begin{pmatrix}
1/2\\
1/2
\end{pmatrix}
\end{align*}
Et $\min\left\{5, 3 \right\} = 3$, alors $x_6$ sort de la base.
On met à jour $B_2^{-1} $ et en même temps on met à jour les termes de droite et la valeur de l'objectif

\begin{align*}
\begin{array}{ccccc}
    & x_5 & x_6 & b & x_1\\
x_2 & 1/2 & 0 & 5/2 & 1/2\\
x_6 & -1/2 & 1 & 3/2 & \color{red}{1/2}\\
-z & 3/2 & 0 & 15/2 & 1/2
\end{array}
\end{align*}
On fait un pivot
\begin{align*}
\begin{array}{cccc}
 & x_5 & x_6 & b \\
x_2 & 1 & -1 & 1\\
x_4 & -1 & 2 & 3\\
-z  & 1 & 1 & 9
\end{array}
\end{align*}
On a maintenant
\begin{align*}
B_3^{-1} =
\begin{pmatrix}
1 & -1\\
-1 & 2
\end{pmatrix}
\end{align*}
et on recommence une nouvelle itération.

$\textbf{Itération 4}$

Pour commencer, on trouve le nouveau multiplicateur $\pi$ et on calcul les coûts réduits
\begin{gather*}
\pi_3^t = c_B^tB_3^{-1} =
\begin{pmatrix}
-1 & -1
\end{pmatrix}\\[6pt]
\bar{c}^t = c^t - \pi_3^t A =
\begin{pmatrix}
3 & 0 & 1 & 0 & 1 & 1
\end{pmatrix}
\end{gather*}

Tous les coûts réduits sont positifs. La base $B_3$ est donc optimale. La solution optimale associée à $B_3$ est $x_2 = 1$ et $x_4 = 2$  (variable de base) et $x_1 = x_3 = x_5 = x_6 = 0$ (variable hors base).
La valeur optimale est $-9$.

# Question 4

Question 5 de l'examen intra A2011.

On a le problème suivant
\begin{align*}
\begin{array}{l c c c c c c c}
  \min\  z  = &  -3x_1    & - &  x_2   & - &  3x_3  &  &   \\
  \text{tel que }   &  2x_1     & + &  1x_2  & + &  x_3   &  \leq  & 2  \\
            &  x_1      & + &  2x_2  & + &  3x_3  &  \leq  & 5  \\
            &  2x_1     & + &  2x_2  & + &  x_3   &  \leq  & 6  \\
            & x_1 && x_2 && x_3 & \geq & 0      \\
\end{array}
\end{align*}

**(a)** Transformez le problème sous forme standard.

**(b)** À une itération du simplexe révisé on obtient
\begin{equation}
\begin{array}{ccccccc}
x_1 &\frac{1}{2} & 0 & 0 & 1 \\
x_5 &\frac{-1}{2} & 1 & 0 & 4 \\
x_6 &-1 & 0 & 1 & 4 \\
-z &\frac{3}{2} & 0 & 0 & 3 \\
\end{array}
\end{equation}
où dans la dernière ligne on a mis **l'opposé du vecteur des multiplicateurs** $\lambda^t$ $\left(\lambda^t=c^t_B B^{-1} \right)$.

Quelle est la base associée à cette itération?

**(c)** En partant du tableau du simplexe révisé en **(b)**, continuer l'algorithme du simplexe révisé jusqu'à l'optimum.

**(d)**
Soit le nouveau coût initial $\hat{c}_2 = c_2+\delta$, où $c_2=-1$ est le coût initial associé à $x_2$. Pour quelle valeur de $\delta$ la solution trvouvée en $\textbf{(b)}$ demeure-t-elle optimale?

## Solution

**(a)** Pour transformer le problème sous forme standard, on introduit des variables d'écarts $x_4$, $x_5$ et $x_6$.

\begin{align*}
\begin{array}{l c c c c c c c c c c c c c}
  \min  z  = &  -3x_1    & - &  x_2   & - &  3x_3  &   &       &   &     &  &      &  &   \\
  \text{tel que }    &  2x_1     & + &  x_2  & + &  x_3   & + &  x_4  &   &     &  &      &= & 2  \\
            &  x_1      & + &  2x_2  & + &  3x_3  &   &       & + & x_5 &  &      &= & 5  \\
            &  2x_1     & + &  2x_2  & + &  x_3   &   &       &   &     & +&  x_6 &= & 6  \\
            &  x_1, && x_2, && x_3, && x_4, && x_5, && x_6 & \geq & 0
\end{array}
\end{align*}

**(b)**
La base B est composée des colonnes associées aux variables $x_1$, $x_5$ et $x_6$.
$$
B = \begin{pmatrix}
    2 & 0 & 0 \\
    1 & 1 & 0 \\
    2 & 0 & 1
\end{pmatrix}
$$

**(c)**
Pour savoir si nous devons poursuivre la résolution du problème, nous
calculons d'abord les coûts réduits:

Nous disposons dans le tableau de
l'inverse de la base, et du vecteur multiplicateur $\lambda$: $\lambda^t = \left(\frac{-3}{2}, 0, 0 \right)$.

Pour calculer les coûts réduits :

$\bar c^t = c^t - \lambda^t A $

$\bar c^t$ = (0, $\frac{1}{2}$, $\frac{-3}{2}$, $\frac{3}{2}$, 0 0)

$\bar c_3 < 0$ la base courante n'est pas optimale. $x_3$ rentre dans la base.

Déterminons la variable de sortie:

$y_3 = B^{-1}A_3$

$$
y_3 = \begin{pmatrix}
      1/2  \\
      5/2  \\
      0
\end{pmatrix}
$$
De plus, $\min \left\{ \frac{1}{1/2}, \frac{4}{5/2} \right\} = 8/5$.
Ainsi $x_5$ sort de la base.

\begin{align*}
\begin{array} { c c c c c|  c}
        &    &   &   &   &  y_3  \\
\hline
 x_1    & 1/2 & 0 & 0 & 1 & 1/2  \\
 x_5    & -1/2 & 1 & 0 & 4 & 5/2 \\
 x_6    & -1  & 0 & 1 & 4 &  0 \\
\hline
 -z     & 3/2 & 0 & 0 & 3 & -3/2 \\
\hline
\end{array}
\end{align*}

Les opérations sur les lignes se résument ainsi:
$L'_1 \leftarrow L_1 - \frac{1}{2} L'_2$ \\
$L'_2 \leftarrow \frac{2}{5} L_2$ \\
$L'_3 \leftarrow L_3$ \\
$L'_4 \leftarrow L_4 + \frac{3}{2} L' _2$

Le nouveau tableau du simplexe révisé est donc le suivant:

\begin{align*}
\begin{array}{|c| c c c |c|}
\hline
 x_1  & 3/5  & -1/5 & 0 & 1/5  \\
 x_3  & -1/5 & 2/5  & 0 & 8/5  \\
 x_6  & -1   & 0    & 1 & 4    \\
\hline
 -z   & 6/5  & 3/5  & 0 & 27/5 \\
\hline
\end{array}
\end{align*}

Dès lors,
$$
B^{-1} = \begin{pmatrix}
    3/5 & -1/5 & 0 \\
    -1/5 & 2/5 & 0 \\
    -1 & 0 & 1
\end{pmatrix}
$$
et
$$
\lambda^t = c_b^t B^{-1} = (-6/5 , -3/5, 0).
$$
et nous pouvons calculer les coûts réduits $\bar c^t = c^t - \lambda^tA$:
$$
\bar c^t = (0, 7/5, 0, 6/5, 3/5, 0).
$$
Tous les coûts réduits sont positifs ou nuls. La base courante est donc optimale.
Une solution optimale est donnée par :
$x_1^* = 1/5$, $x_2^* = 0$, $x_3^* = 8/5$, $x_4^* = 0$, $x_5^* = 0$, $x_6^* = 4$, de valeur optimale -27/5.

**(d)**
La solution demeure optimale tant que $(-1 + \delta) - \lambda A_2 \geq 0$.
Alors la solution demeure réalisable pour tout $\delta \geq -7/5$.

### Question supplémentaire : Dualité

**(e)**
Formuler le dual du problème et vérifier que $\lambda = (-1.2, -0.6, 0)$ est une solution réalisable de ce problème.

Le dual de (P) est:

\begin{align*}
\begin{array}{c c c c c c c c}
\max    & 2 \lambda_1  &  + & 5 \lambda_2  &  +  & 6 \lambda_3  &        &    \\
  \text{tel que }   & 2 \lambda_1  & +  &   \lambda_2  &  +  & 2 \lambda_3  &  \leq  & -3 \\
       &  \lambda_1   & +  & 2 \lambda_2  &  +  & 2 \lambda_3  &  \leq  & -1 \\
       & \lambda_1  & +  & 3 \lambda_2  &  +  &  \lambda_3   &  \leq  & -3 \\
       &  \lambda_1   &    &            &     &            &  \leq  & 0 \\
       &            &    &   \lambda_2  &     &            &  \leq  & 0 \\
       &            &    &            &     & \lambda_3    &  \leq  & 0 \\
\end{array}
\end{align*}

Le vecteur $\lambda = (-1.2, -0.6, 0)$ est égal au vecteur des multiplicateurs
optimaux (-6/5, -3/5, 0).  Le vecteur des multiplicateurs étant solution
réalisable et optimale du problème dual, on a vérifié que $\lambda$ est
réalisable dans le dual.


# CatBoost learning to rank on Microsoft dataset

```python
from catboost import CatBoost, Pool, MetricVisualizer
from copy import deepcopy
import numpy as np
import os
import pandas as pd
```

### Ranking problem

Let's introduce a notation:
* Let $Q = \{q_1, \dots, q_n\}$ be the set of groups
* $D_q = \{d_{q1}, \dots, d_{qm}\}$ -- set of objects retrieved for a group $q$
* $L_q = \{l_{q1}, \dots, l_{qm}\}$ -- relevance labels for the objects from the set $D_q$

Every object $d_{qi}$ is represented in the vector space of features, describing the associatinos between the group and the object.

So every group is associated with set of objects. For example, group is a query and object is a document if we are ranking documents for a search query.

The goal is to learn the ranking function $f = f(d_{qi})$, such that the ranking of objects $d_{qi}$ for all groups from $Q$ based on their scores $x_{qi} = f(d_{qi})$, is as close as possible to the ideal ranking from the editorial judgements $l_{qi}$.

### Ranking quality metrics:
* __Precision__
    $$ \mbox{P}=\frac{|\{\mbox{relevant docs}\}\cap\{\mbox{retrieved docs}\}|}{|\{\mbox{retrieved docs}\}|} $$
* __Recall__
    $$ \mbox{R}=\frac{|\{\mbox{relevant docs}\}\cap\{\mbox{retrieved docs}\}|}{|\{\mbox{relevant docs}\}|} $$

    Notation $@k$ means that metric is calculated on the first $k$ documents from ranking list.

    For example, if 1,2,5,7,9 is the ranks of relevant documents (enumerations starts from number 1) from ten retrivied then $P@5$ will be $\frac{3}{5}$.

* __Mean average precision (MAP)__
    $$\frac{1}{|Q|}\sum_{q \in Q} \frac{1}{|\mbox{relevant docs in } D_q|} \sum_{k} P@k(q) \times rel(q, k) $$

    Where $rel(q, k)$ is a relevance label of the document at k-th position in our ranking of $D_q$. This metric calculates average precision for a query weighted with document relevances and then calculate mean between all queries.

* __Discounted cumulative gain (DCG)__
    $$\sum_{k=1}^{mq} \frac{2 ^ {l_{qk}}}{\log_2(k+1)}$$

    This metric takes into account user behavior: user attention is high on the top and then nonlinear decrease to the end.

* __NDCG__ - normalized DCG = DCG $~ / ~$ IDCG, where IDCG is a maximum possible value of DCG with given set of relevance labels.

* __AverageGain__ - represents the average value of the label values for objects with the defined top  label values.

* __[PFound](https://tech.yandex.com/catboost/doc/dg/references/pfound-docpage/#pfound)__

More on wiki: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)

Parameter $@k$ for every metric can be specified through metric parameter "top", for example "NDCG:top=10", would mean NDCG@10.

### Download part of [MSRank](https://www.microsoft.com/en-us/research/project/mslr/) dataset from CatBoost datasets storage

```python
from catboost.datasets import msrank_10k
train_df, test_df = msrank_10k()

X_train = train_df.drop([0, 1], axis=1).values
y_train = train_df[0].values
queries_train = train_df[1].values

X_test = test_df.drop([0, 1], axis=1).values
y_test = test_df[0].values
queries_test = test_df[1].values
```

### Dataset analysis

__Number of documents__

```python
num_documents = X_train.shape[0]
print(num_documents)
```

__Number of features__

```python
X_train.shape[1]
```

__Relevance labels statistics__

0 - irrelevant, 1 - highly relevant. Table represents number of documents for each value.

```python
from collections import Counter
Counter(y_train).items()
```

For calculation such metrics as NDCG and PFound relevances should be in segment \[0,1\].

```python
max_relevance = np.max(y_train)
y_train /= max_relevance
y_test /= max_relevance
```

__Number of queries__

```python
num_queries = np.unique(queries_train).shape[0]
num_queries
```

### Creation of CatBoost pools

```python
train = Pool(
    data=X_train,
    label=y_train,
    group_id=queries_train
)

test = Pool(
    data=X_test,
    label=y_test,
    group_id=queries_test
)
```

### You can also create pools from files

```python
data_dir = './msrank'

if not os.path.exists(data_dir):
    os.makedirs(data_dir)

train_file = os.path.join(data_dir, 'train.csv')
test_file = os.path.join(data_dir, 'test.csv')

train_df.to_csv(train_file, index=False, header=False)
test_df.to_csv(test_file, index=False, header=False)
```

```python
description_file = os.path.join(data_dir, 'dataset.cd')
with open(description_file, 'w') as f:
    f.write('0\tLabel\n')
    f.write('1\tQueryId\n')
```

```python
Pool(data=train_file, column_description=description_file, delimiter=',')
```

### Attention: all objects in dataset must be grouped by group_id

For example, if the dataset consits of five documents
\[d1, d2, d3, d4, d5\] with corresponding queries \[q1, q2, q2, q1, q2\] then the dataset should be look like:

$$\begin{pmatrix}
    d_1, q_1, f_1\\
    d_4, q_1, f_4\\
    d_2, q_2, f_2\\
    d_3, q_2, f_3\\
    d_5, q_2, f_5\\
\end{pmatrix} \hspace{6px} \texttt{or} \hspace{6px}
\begin{pmatrix}
    d_2, q_2, f_2\\
    d_3, q_2, f_3\\
    d_5, q_2, f_5\\
    d_1, q_1, f_1\\
    d_4, q_1, f_4\\
\end{pmatrix}$$

where $f_i$ is feature vector of i-th document.

### Reducing problem to machine learning task

The first and simplest idea is to try predicting document relevance $l_q$ minimizing RMSE.

$$\frac{1}{N}\sqrt{ \sum_q \sum_{d_{qk}} \left(f(d_{qk}) - l_{qk} \right)^2 }$$

```python
default_parameters = {
    'iterations': 2000,
    'custom_metric': ['NDCG', 'PFound', 'AverageGain:top=10'],
    'verbose': False,
    'random_seed': 0,
}

parameters = {}
```

```python
def fit_model(loss_function, additional_params=None, train_pool=train, test_pool=test):
    parameters = deepcopy(default_parameters)
    parameters['loss_function'] = loss_function
    parameters['train_dir'] = loss_function

    if additional_params is not None:
        parameters.update(additional_params)

    model = CatBoost(parameters)
    model.fit(train_pool, eval_set=test_pool, plot=True)

    return model
```

Lets train the simplest model and also demonstrate precision/recall metrics from introduction.

```python
model = fit_model('RMSE', {'custom_metric': ['PrecisionAt:top=10', 'RecallAt:top=10', 'MAP:top=10']})
```

### Group weights parameter
Suppose we know that some queries are more important than others for us.

The word "importance" used here in terms of accuracy or quality of CatBoost prediction for given queries.

You can pass this additional information for learner using a ``group_weights`` parameter.

Under the hood, CatBoost uses this weights in loss function simply multiplying it on a group summand.

So the bigger weight $\rightarrow$ the more attention for query.

Let's show an example of training procedure with random query weights.

```python
def create_weights(queries):
    query_set = np.unique(queries)
    query_weights = np.random.uniform(size=query_set.shape[0])
    weights = np.zeros(shape=queries.shape)

    for i, query_id in enumerate(query_set):
        weights[queries == query_id] = query_weights[i]

    return weights

train_with_weights = Pool(
    data=X_train,
    label=y_train,
    group_weight=create_weights(queries_train),
    group_id=queries_train
)

test_with_weights = Pool(
    data=X_test,
    label=y_test,
    group_weight=create_weights(queries_test),
    group_id=queries_test
)

fit_model(
    'RMSE',
    additional_params={'train_dir': 'RMSE_weigths'},
    train_pool=train_with_weights,
    test_pool=test_with_weights
)
```

### A special case: top-1 prediction

Someday you may face with a problem $-$ you will need to predict the top one most relevant object for a given query.

For this purpose CatBoost has a mode called __QuerySoftMax__.

Suppose our dataset contain a binary target: 1 $-$ mean best document for a query, 0 $-$ others.

We will maximize the probability of being the best document for given query.

MSRANK dataset doesn't contain binary labels, but for example of method __QuerySoftMax__ we convert it to that format,
 choosing a best document for every query.

```python
def get_best_documents(labels, queries):
    query_set = np.unique(queries)
    num_queries = query_set.shape[0]
    by_query_arg_max = {query: -1 for query in query_set}

    for i, query in enumerate(queries):
        best_idx = by_query_arg_max[query]
        if best_idx == -1 or labels[best_idx] < labels[i]:
            by_query_arg_max[query] = i

    binary_best_docs = np.zeros(shape=labels.shape)
    for arg_max in by_query_arg_max.values():
        binary_best_docs[arg_max] = 1.

    return binary_best_docs
```

```python
best_docs_train = get_best_documents(y_train, queries_train)
best_docs_test = get_best_documents(y_test, queries_test)

train_with_weights = Pool(
    data=X_train,
    label=best_docs_train,
    group_id=queries_train,
    group_weight=create_weights(queries_train)
)

test_with_weights = Pool(
    data=X_test,
    label=best_docs_test,
    group_id=queries_test,
    group_weight=create_weights(queries_test)
)

fit_model(
    'QuerySoftMax',
    additional_params={'custom_metric': 'AverageGain:top=1'},
    train_pool=train_with_weights,
    test_pool=test_with_weights
)
```

### Reducing ploblem, step 2

Now lets look at example of documents relevance:

$$
    \begin{align}
    labels(q_1) &= \begin{bmatrix}
           4 \\
           3 \\
           3 \\
           1
         \end{bmatrix},
    labels(q_2) &= \begin{bmatrix}
           2 \\
           1 \\
           1 \\
           0
         \end{bmatrix}
   \end{align}
$$

This means that with RMSE loss function we pay more attention to q1 than q2.

To avoid this problem we introduce into RMSE a coefficient $c_q$ which depends only on query (and if fact equals to the mean of the difference between prediction and label).

$$\frac{1}{N}\sqrt{ \sum_q \sum_{d_{qk}} \left(f(d_{qk}) - l_{qk} - \color{red}{c_{q}} \right)^2 }$$

```python
fit_model('QueryRMSE')
```

### Reducing problem, step 3

Since the goal of ranking is to predict a list of documents (which can be generated from given document relevances) RMSE loss function doesn't take into account relations between documents: the first is better than second, second is better than third and fifth etc.

We can easily bring this information into the loss function, reducing problem not to regression but classification for two documents $(d_i, d_j)$ -- does $i$th better than $j$th or not.

So we minimize the negative loglikelihood:

$$ - \sum_{i,j \in Pairs} \log \left( \frac{1}{1 + \exp{-(f(d_i) - f(d_j))}} \right) $$

Methods based on pair comparisons called __pairwise__ in CatBoost this objective called __PairLogit__.

There's no need to change the dataset CatBoost generate the pairs for us. The number of generating pairs managed via parameter max_size.

```python
fit_model('PairLogit')
```

Also we can to specify the pairs directly. There are two ways to do that:

1. Two-dimensional matrix with shape=(num_pairs, 2) $\rightarrow$ (winner_id, loser_id): list, numpy.array, pandas.DataFrame.
2. Path two the input file that contains pair descriptions:
    * Row format: $\texttt{[winner index, loser index, pair weight]}$

```python
def read_groups(file_name):
    groups = {}
    group_ids = []

    with open(file_name) as f:
        for doc_id, line in enumerate(f):
            line = line.split(',')[:2]

            label, query_id = tuple(map(float, line))
            if query_id not in groups:
                groups[query_id] = []
            groups[query_id].append((doc_id, label))

            group_ids.append(query_id)

    return groups, group_ids

train_groups, train_group_ids = read_groups(train_file)
assert num_queries == len(train_groups)
```

```python
pairs = []

for group in train_groups.values():
    for i in range(len(group)):
        for j in range(i, len(group)):
            if i == j:
                continue
            doc_i, relevance_i = group[i]
            doc_j, relevance_j = group[j]
            if relevance_i < relevance_j:
                pairs.append((doc_j, doc_i))
            else:
                pairs.append((doc_i, doc_j))

pairs_file = os.path.join(data_dir, 'pairs.csv')

with open(pairs_file, 'w') as f:
    for pair in pairs:
        f.write(str(pair[0]) + '\t' + str(pair[1]) + '\t1\n')
```

```python
pool1 = Pool(data=X_train, label=y_train, group_id=train_group_ids, pairs=pairs)
pool2 = Pool(data=train_file, column_description=description_file, pairs=pairs_file, delimiter=',')
```

### Reducing problem, step 3.1

Thus we know that $f(d_{qk})$ is a ensemble of trees, we can accurately solve the minimization task from step 3.

This method called __PairLogitPairwise__.

```python
fit_model('PairLogitPairwise')
```

### Reducing problem, step 4

Previous loss function directly minimize the number of pairs $(d_i, d_j)$ where $l_i > l_j$ but $f(d_i) < f(d_j)$, simply said the number of incorrectly placed documents.

Since the user attention is high on the first documents and low on last the incorrect switch of the first two documents and last two has different cost.

In steps 3 and 3.1 user can set the weight for pair.

Method __YetiRank__ take this effect into account and generates weights for pairs according to their positions ([paper](https://cache-mskstoredata08.cdn.yandex.net/download.yandex.ru/company/to_rank_challenge_with_yetirank.pdf)).

$$ - \sum_{i,j \in Pairs} \color{red}{w_{ij}} \log \left( \frac{1}{1 + \exp{-(f(d_i) - f(d_j))}} \right) $$

```python
fit_model('YetiRank')
```

### Step 4.1

As in step 3.1 __YetiRankPairwise__ is slower than __YetiRank__, but gives more accurate results.

```python
fit_model('YetiRankPairwise')
```

```python
widget = MetricVisualizer(['RMSE', 'QueryRMSE', 'PairLogit', 'PairLogitPairwise', 'YetiRank', 'YetiRankPairwise'])
widget.start()
```

### Simple classification

Very fast $\rightarrow$ very slow; Simple method $\rightarrow$ complex method; Low quality $\rightarrow$ high quality.

1. RMSE
2. QueryRMSE
3. PairLogit
4. PairLogitPairwise
5. YetiRank
6. YetiRankPairwise

Besides our classification, the quality of the concrete method may depend on your dataset.

Look on NDCG metric of method YetiRank $-$ it's underfitted.

```python
fit_model('YetiRank', {'train_dir': 'YetiRank-lr-0.3', 'learning_rate': 0.3})
```

```python
widget = MetricVisualizer(['YetiRank', 'YetiRank-lr-0.3'])
widget.start()
```

### Additional parameters

__Metric period__

Period in iterations of calculation metrics. This parameter can speed up training process.

```python
fit_model('YetiRank', {'metric_period': 50})
```

__Task type__

You can significantly speed up training procedure switching to gpu.

```python
fit_model('YetiRank', {'task_type': 'GPU'})
```


# Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

### Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2018-10-05 11:12:05 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets transfer our data to a smaller range
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. It returns an array, so called __popt__, which consists of optimal values for the parameters so that the sum of the squared residuals of __sigmoid(xdata, *popt) - ydata__ is minimized.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.453018, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    Mean absolute error: 0.03
    Residual sum of squares (MSE): 0.00
    R2-score: 0.82

```python
plt.plot(train_x, train_y, 'ro', label='data')
plt.plot(test_x,test_y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

Double-click __here__ for the solution.

## Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: [SPSS Modeler](http://cocl.us/ML0101EN-SPSSModeler).

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at [Watson Studio](https://cocl.us/ML0101EN_DSX)

### Thanks for completing this lesson!

Notebook created by: Saeed Aghabozorgi

Copyright © 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/).


# Simple Rotations

Author: Doug <sweetser@alum.mit.edu>

A deep leason from special relativity is that all measurements start as events in space-time. A measurment such as the length of a rod would appear to only involve space. Yet one needs a signal from both ends that gets received by the observer. And who is the observer? Every other system in the Universe is allowed. Since observers can be in very different states, that means the raw data these different observers collect from the same signals can be different. Special relaitivty helps determine what can be agreed upon, namely the interval for two inertial reference frame observers.

## Spatial Rotations

In this iPython notebook, the one thing quaternions are known for - doing 3D spatial rotations - will be examined. When working with quaternions, it is not possible to work in just three spatial dimensions. There is always a fourth dimension, namely time. Even when thinking about spatial rotations, one must work with events in space-time.

Create a couple of quaternions to play with.

```python
%%capture
from Q_tool_devo import Q8;
U=Q8([1,2,-3,4])
V=Q8([4,-2,3,1])
R=Q8([5,6,7,-8])
```

Use the Q8 class that places these 4 numbers in 8 slots like so:

```python
print(U)
print(R)
```

    (1, 0)_I0,2  (2, 0)_i1,3  (0, 3)_j1,3  (4, 0)_k1,3  Q
    (5, 0)_I0,2  (6, 0)_i1,3  (7, 0)_j1,3  (0, 8)_k1,3  Q

If you are unfamiliar with this notation, the $I^2 = -1,\, i^3=-i,\, j^3=-j,\, k^3=-k$. Only positive numbers are used, with additive inverse put in these placeholders.

To do a rotation, one needs to pre-multiply by a quaternion, then post-multiply by its inverse. By doing so, the norm of R will not change because quaternions are a normed division algebra. A quaternion times its inverse has a norm of unity.

```python
def rotate_R_by_U(R, U):
    """Given a space-time number R, rotate it by Q."""
    return U.triple_product(R, U.invert())

R_rotated = rotate_R_by_U(R,U)
```

Should we expect the first term to change? Look into the triple product first term where I use Capital variable for 3-vectors to simplify the presentation:

$$\begin{align}(u, W)&(t, R)(u, -W)/(u^2 + W \cdot W)\\
&= (u t - W \cdot R, u R + W t + W \times R)(u, -W)/(u^2 + W \cdot W)\\
&=(u^2 t - uW \cdot R + u W \cdot R + W \cdot W t - W \cdot W \times R, ...)/(u^2 + W \cdot W)\\
&=\left(\frac{u^2 + W \cdot W}{u^2 + W \cdot W}t,...\right) = (t, ...)\end{align}$$

Another way to see this is that the norm of $||R||$ does not change. That means the scalar plust the 3-vector norm do not change. While the 3-vector can shift who has values, the first term only has one term so should not change.

Now look at the rotated event.

```python
print(R_rotated)
print(R_rotated.reduce())
```

    (13.033333333333333, 8.033333333333333)_I0,2  (4.933333333333334, 16.266666666666666)_i1,3  (13.233333333333334, 8.9)_j1,3  (10.466666666666667, 11.8)_k1,3  QxQxQ.^-1
    (5.0, 0)_I0,2  (0, 11.333333333333332)_i1,3  (4.333333333333334, 0)_j1,3  (0, 1.333333333333334)_k1,3  QxQxQ.^-1.reduce

The first term does change! At least in the non-reduced Q8 format, there is a change because it is composed of the positive and negative terms we saw in the algebra problem. For example, there is the vector identity W.WxR=0. The cross product makes a vector that is 90 degrees to both W and R. The dot product of that cross product with W is zero because nothing is in the direction of W anymore. This shows up algebraically because the 6 terms of the cross product have three positive terms and three negative terms that exactly cancel when dotted to W. But the values remain in the $I^0$ and $I^2$ terms until Q8 is reduced.

When the Q8 is reduced, it ends up being a 5 as expected. This may be of interest because we keep more information about the change with the eight positions to fill in the Q8 representation (none of which are empty after the rotation).

We expect the square of the norms to be identical in the reduce form:

```python
print(R.norm_squared())
print(R_rotated.norm_squared())
print(R_rotated.norm_squared().reduce())
```

    (174, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  Q.norm_squared
    (1026.4666666666667, 852.4666666666667)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  QxQxQ.^-1.norm_squared
    (174.0, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  QxQxQ.^-1.norm_squared.reduce

If squared, the reduced interval should be the same too:

```python
print(R.square().reduce())
print(R_rotated.square())
print(R_rotated.square().reduce())
```

    (0, 124)_I0,2  (60, 0)_i1,3  (70, 0)_j1,3  (0, 80)_k1,3  Q.Q_sq.reduce
    (877.4666666666667, 1001.4666666666667)_I0,2  (389.94666666666666, 503.28)_i1,3  (487.94222222222226, 444.60888888888894)_j1,3  (462.4177777777778, 475.7511111111112)_k1,3  QxQxQ.^-1.QxQxQ.^-1_sq
    (0, 124.0)_I0,2  (0, 113.33333333333331)_i1,3  (43.333333333333314, 0)_j1,3  (0, 13.333333333333371)_k1,3  QxQxQ.^-1.QxQxQ.^-1_sq.reduce

But what should we make of these non-reduced calculations? Here is my speculation. In classical physics, one always, always, always uses the reduced form of a Q8 quaternion measurement. Classical physics involves one thing doing something. Physics gets odd when dealing with relativistic quantum feild theory. That is a rare sport played only when a one packet of protons collides with another inside an atom smasher. In those entirely odd situations, one must start thinking about multiple particles because we cannot know what happened, there is too much energy around, so we sum over all possible histories.

It is simple to move an event to another place in space-time the same distance from the origin. Because it is a transient event, it feels fleeting, which it should.

## Rotations as a Well-behaved Function

Do rotations preserve the group structure of quaternions with multiplication? If it did, then:

$$\rm{Rot}(V*U) R = \rm{Rot}(V) *\rm{Rot}(U) R $$

The product of V\*U into the rotation function is identical to doing one after the other.

```python
product_UV = rotate_R_by_U(R, V.product(U))
product_rotations = rotate_R_by_U(rotate_R_by_U(R, V), U)
print(product_UV)
print(product_rotations)
print(product_UV.reduce())
print(product_rotations.reduce())
```

    (38.55777777777778, 33.55777777777777)_I0,2  (32.315555555555555, 39.871111111111105)_i1,3  (37.84, 34.41777777777778)_j1,3  (40.64222222222222, 31.686666666666667)_k1,3  QxQxQxQxQ.^-1
    (38.55777777777778, 33.55777777777778)_I0,2  (37.92666666666667, 34.260000000000005)_i1,3  (41.19555555555556, 31.06222222222222)_j1,3  (33.297777777777775, 39.03111111111111)_k1,3  QxQxQxQ.^-1xQ.^-1
    (5.000000000000007, 0)_I0,2  (0, 7.55555555555555)_i1,3  (3.4222222222222243, 0)_j1,3  (8.955555555555556, 0)_k1,3  QxQxQxQxQ.^-1.reduce
    (5.0, 0)_I0,2  (3.6666666666666643, 0)_i1,3  (10.133333333333336, 0)_j1,3  (0, 5.733333333333334)_k1,3  QxQxQxQ.^-1xQ.^-1.reduce

This looks well-behaved because the the U and V if the U and V form a product before being applied, it results in the same answer as doing one after the other. I was a bit surprised this work without having to reduce the results.

## A Rotation of Time and Space

A rotation in time is commonly called a boost. The idea is that one gets a boost in speed, and that will change measurements of both time and distance. If one rushes toward the source of a signal, both the measurement of time and distance will get shorter in a way that keeps the interval the same.

There are published claims in the literature that a boost cannot be done with real-valued quaternions. This may be because people followed the form of rotations in space too closely. It is true that swapping hyperbolic cosines for cosines, and hyperbolic sines for sines does not create a Lorentz boost. Rotations are known as a compact Lie group while boosts form a group that is not compact. A slightly more complicated combination of the hyperbolic trig functions does do the work:

$$\begin{align*} b \rightarrow b' = &(\cosh(\alpha), \sinh(\alpha) (t, R) (\cosh(\alpha), -\sinh(\alpha) \\&- \frac{1}{2}(((\cosh(\alpha), \sinh(\alpha) (\cosh(\alpha), \sinh(\alpha) (t,R))^* -((\cosh(\alpha), -\sinh(\alpha) (\cosh(\alpha), -\sinh(\alpha) (t,R))^*)\\
&=(\cosh(\alpha) t - \sinh(\alpha) R, \cosh(\alpha) R - \sinh(\alpha) t)\end{align*}$$

```python
R_boosted=R.boost(0.01,0.02, 0.003)
print("boosted: {}".format(R_boosted.reduce()))
print(R.square().reduce())
print(R_boosted.square())
print(R_boosted.square().reduce())
```

    boosted: (4.65291333288644, 0)_I0,2  (5.903470866671137, 0)_i1,3  (6.80694173334227, 0)_j1,3  (0, 8.028958739998657)_k1,3  Q.boost.reduce
    (0, 124)_I0,2  (60, 0)_i1,3  (70, 0)_j1,3  (0, 80)_k1,3  Q.Q_sq.reduce
    (332.92404269673744, 456.9240426967374)_I0,2  (168.7613021070281, 113.8246254953465)_i1,3  (193.37159851076186, 130.0273786162631)_j1,3  (146.4900636716656, 221.20616201273532)_k1,3  Q.boost.Q.boost_sq
    (0, 123.99999999999994)_I0,2  (54.93667661168159, 0)_i1,3  (63.344219894498764, 0)_j1,3  (0, 74.71609834106971)_k1,3  Q.boost.Q.boost_sq.reduce

The reduced interval is $124 \,I^2$, whether boosted or not. The norm will shrink because all the number are a little smaller, no longer quite (5, 6, 7, 8).

```python
print(R.norm_squared().reduce())
print(R_boosted.norm_squared())
print(R_boosted.norm_squared().reduce())
```

    (174, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  Q.norm_squared.reduce
    (478.5736451800898, 311.27444021338505)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  Q.boost.norm_squared
    (167.29920496670474, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  Q.boost.norm_squared.reduce

## Rotations in Space and Time

Quaternions are just numbers. This makes combining transformations trivial. A measurement can be rotated and boosted. The only thing that should be unchanged is the interval:

```python
R_rotated_and_boosted = R_rotated.boost(0.01,0.02, 0.003)
print("rotated and boosted: {}".format(R_rotated_and_boosted.reduce()))
print(R.square().reduce())
print(R_rotated_and_boosted.square())
print(R_rotated_and_boosted.square().reduce())
```

    rotated and boosted: (5.066457160027788, 0)_I0,2  (0, 11.43399790493362)_i1,3  (4.132004190132783, 0)_j1,3  (0, 1.3635327048134158)_k1,3  QxQxQ.^-1.boost.reduce
    (0, 124)_I0,2  (60, 0)_i1,3  (70, 0)_j1,3  (0, 80)_k1,3  Q.Q_sq.reduce
    (4220.926982084568, 4344.926982084568)_I0,2  (1987.3008154686133, 2103.1605365750006)_i1,3  (2147.719618485641, 2105.850374056915)_j1,3  (2132.0756217717103, 2145.892181842178)_k1,3  QxQxQ.^-1.boost.QxQxQ.^-1.boost_sq
    (0, 124.0)_I0,2  (0, 115.85972110638727)_i1,3  (41.86924442872623, 0)_j1,3  (0, 13.816560070467858)_k1,3  QxQxQ.^-1.boost.QxQxQ.^-1.boost_sq.reduce

Because of the rotation, the z value was larger. It is a safe bet that the norm turns out to be smaller as happened before:

```python
print(R.norm_squared().reduce())
print(R_rotated_and_boosted.norm_squared())
print(R_rotated_and_boosted.norm_squared().reduce())
```

    (174, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  Q.norm_squared.reduce
    (4370.5959702389655, 4195.257993930171)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  QxQxQ.^-1.boost.norm_squared
    (175.33797630879417, 0)_I0,2  (0, 0)_i1,3  (0, 0)_j1,3  (0, 0)_k1,3  QxQxQ.^-1.boost.norm_squared.reduce

## Ratios at Work

An angle is a ratio, this side over that side. The velocity used in a boost is a ratio of distance over time. Say we have a reference observer who measures the interval between two events. If another observer see the same events, but was standing on his head, we migh expect the headstand to change how the crazy observer places his numbers into the three spatial slots. Yet the two observers should agree to the inteval, and they do.

The same happens if there is an observer travelling at a constant velocity relative to the referene observer. It is not surprizing that the math machinery is different because a spatial rotation is different from moving along at certain velocity.

Combining the spatial rotations and boosts can be done, creating messy results, except for the interval that remains the same.

```python
print(R.product(U).dif(U.product(R)))
```

    (70, 70)_I0,2  (72, 64)_i1,3  (22, 102)_j1,3  (28, 92)_k1,3  QxQ-QxQ

```python
print(R.vahlen_conj().product(U.vahlen_conj()).dif(U.vahlen_conj().product(R.vahlen_conj())))
```

    (70, 70)_I0,2  (72, 64)_i1,3  (22, 102)_j1,3  (28, 92)_k1,3  Q.vc-xQ.vc--Q.vc-xQ.vc-

```python
print(R.vahlen_conj("'").product(U.vahlen_conj("'")).dif(U.vahlen_conj("'").product(R.vahlen_conj("'"))))
```

    (70, 70)_I0,2  (64, 72)_i1,3  (102, 22)_j1,3  (28, 92)_k1,3  Q.vc'xQ.vc'-Q.vc'xQ.vc'

```python

```


# Introduction to Meijer $G$-functions

This notebook provides introductory material on the Meijer-$G$ functions based on our paper *"Demystifying Black-box Models with Symbolic Metamodels"* submitted to **NeurIPS 2019** by *Ahmed M. Alaa and Mihaela van der Schaar*. The notebook provides the formal definition of Meijer-$G$ functions, along with examples for how to use our Python implementation of the $G$-function in the context of symbolic computation.

## What are Meijer $G$-functions?

The **Meijer G-functions** are a family of univariate functions, each of them determined by finitely many indexes. The $G$-function is a linear combination of certain special functions of standard type, and hence it is often use to solve indefinite integrals [1]. Despite their useful properties, $G$-functions are not very well known in the mathematical community. In this notebook, we provide a brief background on Meijer $G$-functions, presenting its formal definition along with its basic properties. In addition, we show how to use our python implementation of $G$-functions in symbolic computations.

Formally, the Meijer $G$-function is a univariate function given by a line integral in the complex plane [2, 3]:

$G^{m,n}_{p, q}\left(^{a_1,\ldots,a_p}_{b_1,\ldots,b_q}\,\Big|\, x\right) = \frac{1}{2\pi i} \int_{\mathcal{L}} \frac{\prod^m_{j=1} \Gamma(b_j - s)\prod^n_{j=1} \Gamma(1 - a_j + s)}{\prod^q_{j=m+1} \Gamma(1 - b_j + s)\prod^p_{j=n+1} \Gamma(a_j + s)}\, x^s\, ds$

where $\Gamma(.)$ is the Gamma function, $0 \leq m \leq q$, $0 \leq n \leq p$, where $m$, $n$, $p$ and $q$ are integer numbers, $a_k - b_j \neq 1, 2, 3,.\,.\,.$ for $k = 1, 2,\,.\,.\,., n$, $j = 1, 2\,.\,.\,., m$ and $x \neq 0$. This means that no pole of any $\Gamma(b_j - s)$ coincides with any pole of any $\Gamma(1 - a_k + s)$. The integration path $\mathcal{L}$ separates the poles of the factors $\Gamma(b_j - s)$ from those of the factors $\Gamma(1 - a_k + s)$.

As shown in the Figure above, there are three cases in which integration path $\mathcal{L}$ can be chosen to separate the poles and the zeros as follows [4]:

- $L$ goes from $-i\infty$ to $i\infty$.
- $L$ is a loop that starts at $i\infty$ on a line parallel to the $+$ve real axis, encircles the poles of the factors $\Gamma(b_j - s)$ and returns to $i\infty$ on another line parallel to the $+$ve real axis.
- $L$ is a loop that starts at $i\infty$ on a line parallel to the $-$ve real axis, encircles the poles of the factors $\Gamma(1 - a_k + s)$ and returns to $\infty$ on another line parallel to the $-$ve real axis.

When more than one of Cases 1, 2, and 3 is applicable the same value is obtained for the $G$-function. The conditions for convergence of the line integral can be established by applying Stirling's asymptotic approximation to the gamma functions $\Gamma(b_j - s)$ and $\Gamma(1 - a_k + s)$  in the integrand. As a consequence of the definition above, the Meijer $G$-function is an analytic function of $x$ (with possible exception of the origin $x$ = 0 and the unit circle $|x| = 1$).

## How to use Meijer $G$-functions in PySymbolic?

**PySymbolic** is a symbolic computation library that we developed in order to implement symbolic metamodeling. This is basically a wrapper over **SymPy** that provides all the relevant functionalities needed for metamodeling, in addition to a high-level API that makes symbolic comptations easier.

To see how can we evaluate Meijer $G$-functions in PySymbolic, we first import the **MeijerG** class from the **models.special_functions** module as follows:

```python
from pysymbolic.models.special_functions import MeijerG
```

In order to create an instance of a Meijer $G$-function, we only need to provide it with the poles and the zeros through two input arguments as follows:

**MeijerG(theta=[a_1,..., a_n,..., a_p, b_1,...,b_m,...,b_q, scale], order=[m, n, p, q])**

Here the arguments **theta** and **order** correspond to the actual values of the poles and zeros, and the indexes $m$, $n$, $p$ and $q$, respectively. The **scale** variable corresponds to a real number by which the input $x$ is scaled. To demonstrate the above, let us create an object holding the following Meijer $G$-function:

$G^{0,1}_{3, 1}\left(^{2, 2, 2}_{1}\,\Big|\, 5x\right)$

To create the corresponding MeijerG object, we define the inputs as theta = [2,2,2,1,5] since $a_1=a_2=a_3 = 2$, and $b_1=1$, whereas the scaling factor is 5. Thus, we have the following:

```python
MeijerG(theta=[2, 2, 2, 1, 5], order=[0, 1, 3, 1])
```

Now to recover the expression above in a symbolic format, we use the "expression" method:

```python
My_meijerg_object = MeijerG(theta=[2, 2, 2, 1, 5], order=[0, 1, 3, 1])

My_meijerg_object.expression()
```

As we can see above, it turns out that this particular selection of poles and zeros reduces the Meijer $G$-function to an identity function! This is the power of Meijer $G$-functions: for different selections of poles and zeros, they reduce to familiar functional forms.

In addition to the **theta** and **order** inputs, the **MeijerG** object takes two other optional arguments:

- **evaluation_mode**: When evaluating the value numerical value of a Meijer $G$-function at a specific value of $x$, this argument determines the way numerical evaluation is carried out. The options include ['eval','numpy', 'theano', 'cython']. The 'eval' option applies direct symbolic computation, which is the slowest approach. 'numpy' and 'theano' apply numeric evaluation using numpy arrays or theano graphs, whereas 'cython' creates an intermediate C code to enable fast computation of the Meijer $G$-function. The default setting is 'numpy'.
- **approximation_order**: When approximating the Meijer $G$-function for intermediate computation, we use a Taylor series approximation. This argument determines the number of terms in a Taylor expansion to include in the approximation, and is set to a default value of 15.

As mentioned earlier, the main strength of the Meijer $G$-function is that they reduce to familiar functional forms for different selections of poles and zeros. Notable examples include the following:

- $G^{0,1}_{3, 1}\left(^{2, 2, 2}_{1}\,\Big|\, x\right) = x$ (which we have demonstrated above).

- $G^{1,0}_{0, 1}\left(^{-}_{0}\,\Big|\, x\right) = e^{-x}$.

- $G^{1,2}_{2, 2}\left(^{1,1}_{1,0}\,\Big|\, x\right) = \log(x+1)$.

- $G^{1,2}_{2, 2}\left(^{1,1}_{1,1}\,\Big|\, x\right) = \frac{x}{(x+1)}$.

In what follows, we show how can we recover all the identities above using the MeijerG object. To enable printing in LaTex format inside the notebook, we use the "init_print" command in sympy.

```python
from sympy import *
init_printing()
```

```python
identity_function    = MeijerG(theta=[2, 2, 2, 1, 1], order=[0, 1, 3, 1])
exponential_function = MeijerG(theta=[0, 1], order=[1, 0, 0, 1])
logarithmic_function = MeijerG(theta=[1,1,1,0,1], order=[1, 2, 2, 2])
rational_function    = MeijerG(theta=[1,1,1,1,1], order=[1, 2, 2, 2])
```

Now let us check if the magic works!

```python
identity_function.expression()
```

```python
exponential_function.expression()
```

```python
logarithmic_function.expression()
```

```python
rational_function.expression()
```

### References

[1] Richard Beals and Jacek Szmigielski. Meijer g-functions: a gentle introduction. *Notices of the AMS*, 60(7):866–872, 2013.

[2] CS Meijer. On the G-function. *North-Holland*, 1946.

[3] CS Meijer. Uber whittakersche bezw. besselsche funktionen und deren produkte (english translation: About whittaker and bessel functions and their products). *Nieuw Archief voor Wiskunde*, 18(2):10–29, 1936.

[4] I. Gradshteyn and I. Ryzhik. Table of integrals, series, and products. *Academic press*, 2014.

```python

```


#Simple cartridge model
## Overview
The goal of this notebook is to create a simple model of firearm cartridge pressures and forces.

- Simple Internal Ballistics Model
  - Chamber pressure curve
  - Bullet dynamics
  - Chamber wall friction
- .22lr model
- 9mm model
- 357 model (TBD)
- 40 S&W (TBD)

## Simple Internal Ballistics
This simple model is a function of three main physical effects:
1. Rapidly rising powder combustion pressure
1. The combustion gas expanding as the bullet accelerates down the barrel
1. The bolt thrust as a function of chamber pressure and case friction

###Chamber Pressure Curve
The chamber pressure is governed by two main effects, the peak combustion pressure of the powder and the expansion of the gasses while the bullet accelerates down the barrel.

Chinn and Quickload tend to model the initial rise in powder combustion pressure as an exponential:

While [Varmit Al](http://www.varmintal.com/) shows some very nice pressure curves with more of an "S" shape to them initially:

For this model we choose to fit a [Sigmoid Function](http://en.wikipedia.org/wiki/Sigmoid_function):

$$S(t) = \frac{1}{1+\mathrm{e}^{-\alpha t}}$$

The curve below represents what the peak chamber pressure would be assuming that the bullet and bolt are frozen from moving.  This is not the same as peak chamber pressure during normal operation.  Once the bullet is allowed to accelerate down the barrel, the chamber pressures will be much smaller as the gas volume increases.

```
%pylab inline
from IPython.display import Math
from curves import sigmoid
import units as u

# sigmoid curve parameters
t_peak  = 5e-4                    # s, time for 50% sigmoid pressure
alpha_s = 6.0 / t_peak            # Combustion pressure timeconstant, 0-peak
p_peak  = 40 * u.ksi              # Peak combustion pressure

#Time axis
Ts = 1e-6
Tf = 3.0e-3
N = int(Tf / Ts)
t = array(range(0,N))*Ts

# Render the base curve
plot(t, p_peak/u.ksi*sigmoid(t, alpha_s, t_peak))
title('Combustion pressure vs time')
xlabel('Time, s')
ylabel('Pressure, ksi')
axis([0, 3e-3, 0, 45])
grid()
show()

```

### Bullet Dynamics
For the purposes of this analysis we will assume that the bullet is a mass pushed through a frictionless tube with a perfect gas seal.
$$
F_{bullet} = P_{chamber} A_{bullet}
$$
As the bullet accelerates down the barrel the pressure in the chamber will decrease as the gas expands.  We will assume [isentropic expansion](http://en.wikipedia.org/wiki/Isentropic_process) which is governed by
$$
\frac{p_1}{p_0} = \left( \frac{V_0}{V_1}\right) ^{\gamma}
$$

Where $\gamma$ is the ratio of specific heats.  Here $\gamma$ = 1.3 which should be in the ball park for hot air and $CO_2$.

## 22LR pressure curve
The following simulation combines the pressure curve with the bullet dynamics to create a more familiar chamber pressure cure.  The model is calibrated to correlate with Varmit Al's 22lr data.  The model assumes a 24.5" barrel, and a 40 grain .22lr bullet.

```
from cartridges import C22_Al
import units as u

c = C22_Al()

X = zeros([4,N])        # x, x_d, pressure, F_case
for i in range(N-1):
    x = X[0,i]
    x_d = X[1,i]
    x_dd = X[2,i] * c.a_chamber / c.m_bullet

    p_combust = c.p_peak*sigmoid(t[i], c.alpha_s, c.t_peak)
    if x >  24.5 * u.inch:
        p_combust = 0
    p_k = p_combust*pow(c.v_case / (c.v_case + x*c.a_chamber), c.gama)

    F_case = p_k * c.a_chamber

    X[0,i+1] = x + x_d*Ts
    X[1,i+1] = x_d + x_dd*Ts
    X[2,i+1] = p_k
    X[3,i+1] = F_case

F_bolt = array(X[3,:])

figure(figsize=(12,8), dpi=150)
subplot(2,2,1)
plot(t,X[2,:] /u.ksi)
title('Chamber Pressure')
xlabel('Time, s')
ylabel('Chamber Pressure, ksi')
axis([0, 3e-3, 0, 8])
grid()

subplot(2,2,2)
plot(t, X[1,:]/u.ft)
title('Bullet Velocity')
xlabel('Time, s')
ylabel('Velocity, ft/s')
axis([0, 3e-3, 0, 1200])
grid()

show()
```

This model looks qualitatively pretty good.  After choosing the rise time of the sigmoid function, the peak pressure was hand tuned until the the bullet velocity matched. Alternatively the pressures could have been tuned to match the peak pressure more closely at the expense of a lower bullet velocity.

### Chamber wall pressure and friction
For bolt action and falling block actions, the above model is sufficient.  For blowback or other actions where the case may move while under pressure, it is helpful to have an estimate for the friction between the case wall and the chamber wall.  This model assumes that the chamber is smooth (not fluted.)

First we will look at the yield and burst pressures for the case using the [thin wall hoop stress equation](http://en.wikipedia.org/wiki/Cylinder_stress#Hoop_stress), then calculate the force between the brass and the chamber wall and finally estimate the frictional force acting between the case and the chamber.

The hoop stress equation is:
$$
\sigma_{\theta} = \frac{Pd}{2t}
$$
Where $\sigma_{\theta}$ is the stress on the brass, $P$ is the chamber pressure, $d$ is the chamber diameter and $t$ is the case wall thickness.

For brass, the [yield stress](http://en.wikipedia.org/wiki/Ultimate_tensile_strength) is assumed to be ~29ksi and the ultimate strength is estimated to be ~80ksi.
$$
P_{yield} = \frac{2\sigma_{yield}t}{d} \\
P_{burst} = \frac{2\sigma_{uts}t}{d}
$$

```
p_yield = c.yield_case * 2.0*c.t_case / c.d_case
p_uts = c.uts_case * 2.0*c.t_case / c.d_case
print 'Case yield pressure: %.2f ksi' % (p_yield / u.ksi)
print 'Case burst pressure: %.2f ksi' % (p_uts / u.ksi)
```

    Case yield pressure: 2.86 ksi
    Case burst pressure: 7.86 ksi

Even for the relatively low velocity .22lr round, the case will deform significantly and rupture without support from the chamber.

The force acting between the case wall and the chamber wall is estimated as:
$$
\begin{align}
F_{wall} & = P_{case \_ wall} A_{case} \\
& = (P_{chamber} - P_{yield}) \pi d_{case} l_{case}
\end{align}
$$
Where $P_{case \_ wall}$ is the net pressure acting between the case and the chamber walls, $d_{case}$ is the diameter of the case and $l_{case}$ is the length of case acting on the wall (forming the cylindrical area).

The frictional force is then:
$$
F_{friction} = \mu F_{wall}
$$

We can now add this effect to the above simulation:

```
X = zeros([4,N])        # x, x_d, pressure, F_case_wall
for i in range(N-1):
    x = X[0,i]
    x_d = X[1,i]
    x_dd = X[2,i] * c.a_chamber / c.m_bullet

    p_combust = c.p_peak*sigmoid(t[i], c.alpha_s, c.t_peak)
    if x > 24.5 * u.inch:
        p_combust = 0
    p_k = p_combust*pow(c.v_case / (c.v_case + x*c.a_chamber), c.gama)

    F_case = p_k * c.a_chamber

    # Case wall friction removes force
    if p_k < p_yield:
        F_friction = 0
    else:
        a_case = pi*c.d_case*c.l_case
        F_friction = (p_k - p_yield) * a_case * c.u_cw

    F_case =p_k * c.a_chamber - F_friction
    if F_case < 0:
        F_case = 0

    X[0,i+1] = x + x_d*Ts
    X[1,i+1] = x_d + x_dd*Ts
    X[2,i+1] = p_k
    X[3,i+1] = F_case

figure(figsize=(12,8), dpi=150)
subplot(2,2,1)
plot(t, F_bolt.T / u.lbf, label='w/o friction')
plot(t,X[3,:] / u.lbf, label='w friction')
legend(loc=0)
title('Bolt Force')
#xlabel('Time, s')
ylabel('Bolt Force, lbf')
axis([0, 2.5e-3, 0, 350])
grid()

subplot(2,2,2)
plot(t, X[1,:]/u.ft)
title('Bullet Velocity')
#xlabel('Time, s')
ylabel('Velocity, ft/s')
#axis([0, 2.5e-3, 0, 1200])
grid()

show()

```

The bolt force figure tells the story here. Once the pressure is high enough for the case to yield, the chamber begins supporting the brass and a frictional force develops between the two. Even with a relatively low coefficient of friction (0.1 in this case) the larger area of the case wall affords a relatively large force for the same chamber pressure when compared to the force on the case base.  In this case of a .22lr the peak force applied to the bolt is reduced significantly by the friction between the case and the chamber.

## 9mm Cartridge Model
The process is repeated for the 9mm Luger cartridge below.

```
from cartridges import C9mm
import units as u

c = C9mm()

l_barrel = 16.0*u.inch
(t,Y) = c.sim(False, l_barrel)
(t,Z) = c.sim(True, l_barrel)

figure(figsize=(12,8), dpi=150)
subplot(2,2,1)
plot(Y[0,:] / u.inch ,Y[2,:] / u.ksi)
title('Chamber Pressure Vs Bullet Position')
ylabel('Pressure, kPsi')
axis([0, l_barrel / u.inch, 0, 40])
grid()

subplot(2,2,2)
plot(t, Y[2,:]/u.ksi)
title('Chamber Pressure vs Time')
ylabel('Pressure, kPsi')
axis([0, 1.5e-3, 0, 40])
grid()

subplot(2,2,3)
plot(Y[0,:] / u.inch, Y[1,:] / u.ft)
title('Bullet Velocity Vs Position')
xlabel('Bullet Position, inchs')
ylabel('Bullet Velocity, ft/s')
axis([0, l_barrel/u.inch, 0, 1600])
grid()

subplot(2,2,4)
plot(t, Y[1,:] / u.ft)
title('Bullet Velocity Vs Time')
xlabel('Time, s')
ylabel('Bullet Velocity ft/s')
axis([0, 1.5e-3, 0, 1600])
grid()

show()
```

```
figure(figsize=(12,8), dpi=150)
subplot(2,2,1)
plot(t,Y[3,:] / u.lbf, label='w/o friction')
plot(t,Z[3,:] / u.lbf, label='w friction')
legend(loc=0)
title('Bolt Force')
xlabel('Time, s')
ylabel('Bolt Force, lbf')
axis([0, 1.5e-3, 0, 4000])
grid()

subplot(2,2,2)
plot(t, Y[1,:]/u.ft)
title('Bullet Velocity')
xlabel('Time, s')
ylabel('Velocity, ft/s')
axis([0, 1.5e-3, 0, 1600])
grid()
```

## Next steps
- .38 Special and .357 magnum models
- .40 S&W model

```
css_path = 'custom.css'
from ipynb_utils import inject_css2
from IPython.display import HTML
HTML(inject_css2(css_path))
```

<style>
#notebook_panel { /* main background */
    background: #fff;
    color: #f6f6f6;
}

div #notebook { /* center the content */
    background: #fff;
    width: 800px;
    margin: auto;
    padding-left: 10px;
    padding-right: 10px;
    padding-top: 10px;
}

#notebook li { /* More space between bullet points */
    margin-top:6pt;
}

.rendered_html *li{  /* More space between bullet points */
    margin-top:6pt;
}

div.container { /* make the widths match */
    width: 780px;
}

/*The cell is the fundamemntal container for each chunk of work, markdown*/
/*code, etc. */
div.cell {
    width: 760px;
}

div.CodeMirror { /* code input area */
    font-family: "Inconsolata-dz", monospace;
    font-size: 12pt;
}

div.cell.code_cell { /* Code output area, print statements, etc */
    font-size: 12pt;
    line-height: 115%;
    background: #fff;
    border: none;
    border-radius: 4px;
    padding-top: 1ex;
}

/*area that contains rendered markdown*/
div.text_cell_render{
    font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
    font-size: 14pt;
    line-height: 130%;
    margin-left:auto;
    margin-right:auto;
}

.rendered_html *+ol, *+ul{ /* Remove margin at top of lists*/
    margin-top:0;
}

.rendered_html h1 {
    margin: 0;
    padding: 0;
}

.rendered_html h2 {
    margin: 0;
    padding: 0;
}

.rendered_html h3 {
    margin: 0;
    padding: 0;
}

div.input_area { /* box around box with code */
    border: none;
    background: #f5f5f5;
    border: 1px solid #ccc;
    border-radius: 5px;
    padding-top: 0.5ex;
    padding-bottom: 0.5ex;
    padding-left: 0.5em;
}

div.prompt { /* remove In/Out prompt */
    display: none;
}

div.output_subarea { /* remove margin in front of output */
    border: none;
}

div.cell.border-box-sizing.code_cell.running {
    /* draw border around running cells */
    border: 3px dotted #f33;
}

/* header colours and fonts */
h1 { color: #444; }
h2 { color: #444; }
h3 { color: #444; }
h4 { color: #444; }
h5 { color: #444; }
h6 { color: #444; }

/* remove big margins around title */
div.text_cell_render.border-box-sizing.rendered_html {
    margin: 0;
    padding: 0;
    margin-left: 1ex;
}

</style>

```
# %load_ext autoreload
# %autoreload 2
# %pylab inline
```