```python
import pandas as pd
import numpy as np
import math
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import norm
import scipy.stats
import random
import copy
```

**Data**

* Currently using the Broward County data set from ProPublica (https://github.com/propublica/compas-analysis/blob/master/compas-scores.csv).
* Inlcudes defendant sex, age, # priors, charge, COMPAS score, and recidivism status.
* Does not include judge's decision about the defendant.  Martina tried getting this information, but had to query each defendant individually.
* Right now, we are using synthetic data to get around this hurdle -- we estimate the judge's decision (detain / release) via linear regression on defendant age and # priors.
* Alternative datasets to consider for the future.
 * VSVC:  Stevenson used in https://www.econstor.eu/bitstream/10419/215249/1/dp12853.pdf.  "The primary data source is the VCSC, which collects and maintains records on all felony sentences in the state of Virginia since 1995."  Contains defendant profile, risk score (not COMPAS), and judge's sentencing decisions.

```python
# import data
df_data = pd.read_csv('people.csv')

# only keep relevant fields, only keep rows with recidivism recorded
fields = ['race', 'age', 'priors_count', 'decile_score', 'is_recid']
df_data = df_data[fields]
df_data = df_data[df_data['is_recid'] >= 0] # -1 signals no value
df_data = df_data[df_data['decile_score'] >= 0] # -1 signals no value
df_data.head()
```

      race
      age
      priors_count
      decile_score
      is_recid

      0
      Other
      69
      0
      1
      0

      2
      African-American
      34
      0
      3
      1

      3
      African-American
      24
      4
      4
      1

      4
      African-American
      23
      1
      8
      0

      5
      Other
      43
      2
      1
      0

**Model Specifications**

**Variables**

$ \{x_{1}, x_{2},...x_{k}\} \in X$, Observed features of defendant

$\mu_{prior}$, Judge's mean prior belief about a defendant; $\textit{most probable}$ risk prediction.

$\sigma_{prior}$, Standard deviation of judge's prior belief on a defendant; corresponds to a judge's $\textit{uncertainty}$ about prediction

$\mu_{ra}$, Observed algorithmic risk assessment prediction; mean of normally-distributed perceived anchor information

$\sigma_{ra}$, Perceived anchor $\textit{influence}$; Corresponds to std of risk assessment (when perceived as containing a lot of information std will be lower and vice versa).

$q$, Parameter mapping $\mu_{prior},\sigma_{prior}$ onto $\mu_{post},\sigma_{post}$

$scale$, Influence of anchor; lower (closer to zero) = more influence; think this parameter could have been Martina's issue. She had the scale specified such that $\mu_{prior} = \mu_{post}$.

$\mu_{post}$, Judge's mean posterior belief

$\sigma_{post}$, Standard deviation of judge's posterior belief

$\tau$, Decision making threshold

$\phi_{prior}(\tau;\mu_{prior},\sigma_{prior})$, Gaussian distribution representing judge's prior belief

$\phi_{post}(\tau;\mu_{post},\sigma_{post})$, Gaussian distribution representing the judge's posterior belief

$\Phi_{post}(\tau;\mu_{post},\sigma_{post})$, Probability of drawing a belief from the posterior less than threshold $\tau$

$\hat{y}$, The judge's belief about the probability that the defendant will recidivate.

$y$, Judge's observed decision on defendant, which we just generated synthetically.

**Procedure**

Given {$\beta, b, \sigma _{prior}, \tau$, scale}, want to calculate:

\begin{equation} \mu_{prior} = \beta X + b \end{equation}

\begin{equation}
d = (\mu_{ra}-\mu_{prior})^2
\end{equation}

\begin{equation}
I = e^{-scale*d}
\end{equation}

\begin{equation}
\mu_{post} = I\cdot\mu_{ra} + (1-I)\cdot\mu_{prior}
\end{equation}

\begin{equation}
\sigma_{post} = \sigma_{prior}e^{-I}
\end{equation}

\begin{equation}
\phi(\tau; \mu_{post},\sigma_{post}) = \mathcal{N}(\tau;\mu_{post},\sigma_{post})
\end{equation}

\begin{equation}
\Phi(\tau;\mu_{post},\sigma_{post}) = \int_0^{\tau} \phi_{post}(\tau; \mu_{post},\sigma_{post})
\end{equation}

\begin{equation}
\hat{y}=1-\Phi(\tau;\mu_{post},\sigma_{post})
\end{equation}

**Remarks and Questions**
* $\hat{y}$ cannot equal 0 or 1, otherwise the loss function will return nan; currently fixing this issue by adding 1e-10 of noise, an arbitrary number.

```python
# functions to create synthetic data
def calc_mu_prior(df, features, parameters):
    B = parameters['B']
    b = parameters['b']

    mu_prior = b + df[features[0]]*B[0] + df[features[1]]*B[1] # make pretty later
    df['mu_prior'] = mu_prior
    return mu_prior

def calc_I(df, parameters):
    scale = parameters['scale']

    difference = (df['mu_prior'] - df['decile_score']).abs()
    I = [1-(1/(1+np.exp(-scale*d))) for d in difference]
    df['difference'] = difference
    df['I'] = I
    return difference, I

def calc_mu_post(df, parameters):
    q = parameters['q']

    mu_post = df['I']*(df['mu_prior']*(q/(q+1)) + df['decile_score']/(q+1)) + (1-df['I'])*df['mu_prior']
    df['mu_post'] = mu_post
    return mu_post

def calc_sigma_post(df, parameters):
    q = parameters['q']
    sd_prior = parameters['sigma_prior']

    sd_post = [(I[i]*(sd_prior*np.sqrt(q/(q+1))))+((1-I[i])*sd_prior) for i in range(len(mu_prior))]
    df['sigma_post'] = sd_post

    return sd_post

def calc_y(df, parameters):
    tau = parameters['tau']

    mu_post = df['mu_post'].values
    sigma_post = df['sigma_post'].values
    y_pred = [1 - norm.cdf(tau, loc=mu_post[i], scale=sigma_post[i]) for i in range(len(mu_post))]
    y = [1 if p > 0.5 else 0 for p in y_pred]
    df['y_pred'] = y_pred
    df['y'] = y
```

```python
# data
scale = 0.1
syn_params = {'B': np.array([[0.1], [1]]),
              'b': 1,
              'q': .5,
              'sigma_prior': 3,
              'tau': 6,
              'scale': scale # lower = more influence
             }
mu_ra = data['decile_score'].values

# calcs
features = ['age', 'priors_count']
calc_mu_prior(df_data, features, syn_params)
calc_I(df_data, syn_params)
calc_mu_post(df_data, syn_params)
calc_sigma_post(df_data, syn_params)
calc_y(df_data, syn_params)
df_data.head()
```

      race
      age
      priors_count
      decile_score
      is_recid
      mu_prior
      difference
      I
      mu_post
      sigma_post
      y_pred
      y

      0
      Other
      69
      0
      1
      0
      7.9
      6.9
      0.334033
      6.363448
      2.630888
      0.554938
      1

      2
      African-American
      34
      0
      3
      1
      4.4
      1.4
      0.465057
      3.965947
      2.472775
      0.205374
      0

      3
      African-American
      24
      4
      4
      1
      7.4
      3.4
      0.415809
      6.457499
      2.422930
      0.574883
      1

      4
      African-American
      23
      1
      8
      0
      4.3
      3.7
      0.408541
      5.307735
      2.447981
      0.388669
      0

      5
      Other
      43
      2
      1
      0
      7.3
      6.3
      0.347511
      5.840456
      2.565107
      0.475203
      0

**Fairness Criteria of Original Predictions**

1. Overall accuracy equity
2. Predictive parity
3. Calibration
4. Statistical parity
5. Equal FPRs and FNRs

```python
# restrict to black and white defendants, add columns for TP, TN, FP, FN
df_data = df_data[df_data.race.isin(['African-American', 'Caucasian'])]
df_data.loc[(df_data['is_recid'] == 1) & (df_data['y'] == 1), 'TP'] = 1
df_data.loc[(df_data['is_recid'] == 0) & (df_data['y'] == 0), 'TN'] = 1
df_data.loc[(df_data['is_recid'] == 0) & (df_data['y'] == 1), 'FP'] = 1
df_data.loc[(df_data['is_recid'] == 1) & (df_data['y'] == 0), 'FN'] = 1
df_data.head()
```

      race
      age
      priors_count
      decile_score
      is_recid
      mu_prior
      difference
      I
      mu_post
      sigma_post
      y_pred
      y
      ra_new
      ra_change
      TP
      TN
      FP
      FN

      2
      African-American
      34
      0
      3
      1
      4.4
      1.4
      0.465057
      3.965947
      2.472775
      0.205374
      0
      7.840438
      4.840438
      NaN
      NaN
      NaN
      1.0

      3
      African-American
      24
      4
      4
      1
      7.4
      3.4
      0.415809
      6.457499
      2.422930
      0.574883
      1
      4.033073
      0.033073
      1.0
      NaN
      NaN
      NaN

      4
      African-American
      23
      1
      8
      0
      4.3
      3.7
      0.408541
      5.307735
      2.447981
      0.388669
      0
      8.461149
      0.461149
      NaN
      1.0
      NaN
      NaN

      7
      Caucasian
      41
      14
      6
      1
      19.1
      13.1
      0.212487
      17.244282
      2.444862
      0.999998
      1
      -42.550876
      NaN
      1.0
      NaN
      NaN
      NaN

      9
      Caucasian
      39
      0
      1
      0
      4.9
      3.9
      0.403717
      3.850335
      2.547805
      0.199410
      0
      7.624679
      6.624679
      NaN
      1.0
      NaN
      NaN

```python
# overall accuracy equity
accuracy = (df_data['is_recid']==df_data['y']).astype(int).groupby(df_data['race']).mean()
accuracy.plot.bar(x='race', y='accuracy', rot=0)
```

```python
# predictive parity
d = pd.DataFrame(df_data.groupby(['y_pred','race']).agg({'is_recid': 'mean'}))
d = d.reset_index()
im = sns.scatterplot(data=d, x='y_pred', y='is_recid', hue='race')
im.set(ylim=(0,1))
```

```python
# compare distributions
df_data['mu_post'].hist(bins=10, grid=False)
```

```python
# compare confusion matrices
cm = pd.crosstab(df_data['y'], df_data.loc[df_data['race'] == 'African-American']['is_recid'],
                               rownames=['Predicted'], colnames=['Actual'], normalize='columns')
ax = plt.axes()
plt.figure(figsize=(5,5))
sns.heatmap(cm, annot=True, fmt='.2f', cbar=False, ax=ax)
ax.set_title('African-American Defendants')
plt.show()

cm = pd.crosstab(df_data['y'], df_data.loc[df_data['race'] == 'Caucasian']['is_recid'],
                               rownames=['Predicted'], colnames=['Actual'], normalize='columns')
ax = plt.axes()
plt.figure(figsize=(5,5));
sns.heatmap(cm, annot=True, fmt='.2f', cbar=False, ax=ax)
ax.set_title('Caucasian Defendants')
plt.show()
```

```python
# find nearest ra to change decision, if exists
def find_new_ra(df, params):
    tau = params['tau']
    q = params['q']
    sigma_prior = params['sigma_prior']
    mu_prior = df['mu_prior'].values
    ra_orig = df['decile_score'].values
    y_orig = df['y'].values

    ra_new = list()
    ra_change = list()

    # iterate through defendants
    # for i in range(0, 5):
    for i in range(0, len(ra_orig)):
        # try all possible risk scores, see if any change outcome
        ra_cands = list()
        for ra in range(1, 11):
            d = np.abs(ra-mu_prior[i])
            I = 1-1/(1+np.exp(-scale*d))
            mu_post = I*((mu_prior[i]*(q/(q+1)))+(ra/(q+1)))+(1-I)*mu_prior[i]
            sigma_post = (I*(sigma_prior*np.sqrt(q/(q+1))))+(1-I)*sigma_prior
            y_pred = 1 - norm.cdf(tau, loc=mu_post, scale=sigma_post)
            y_new = 1 if y_pred > 0.5 else 0
            # see if different results
            if(y_new != y_orig[i]):
                ra_cands.append((ra, np.abs(ra - ra_orig[i])))
            # print(ra, ra_orig[i], mu_prior[i], mu_post, sigma_post, y_pred, y_new, y_orig[i])
        if(len(ra_cands) == 0):
            ra_new.append(np.NaN)
            ra_change.append(np.NaN)
        else:
            ra_opt = min(ra_cands, key = lambda t: t[1])
            ra_new.append(ra_opt[0])
            ra_change.append(ra_opt[1])

    df['ra_new'] = ra_new
    df['ra_change'] = ra_change
    return(df)

find_new_ra(df_data, syn_params)
```

      race
      age
      priors_count
      decile_score
      is_recid
      mu_prior
      difference
      I
      mu_post
      sigma_post
      y_pred
      y
      ra_new
      ra_change
      TP
      TN
      FP
      FN

      2
      African-American
      34
      0
      3
      1
      4.4
      1.4
      0.465057
      3.965947
      2.472775
      0.205374
      0
      NaN
      NaN
      NaN
      NaN
      NaN
      1.0

      3
      African-American
      24
      4
      4
      1
      7.4
      3.4
      0.415809
      6.457499
      2.422930
      0.574883
      1
      1.0
      3.0
      1.0
      NaN
      NaN
      NaN

      4
      African-American
      23
      1
      8
      0
      4.3
      3.7
      0.408541
      5.307735
      2.447981
      0.388669
      0
      NaN
      NaN
      NaN
      1.0
      NaN
      NaN

      7
      Caucasian
      41
      14
      6
      1
      19.1
      13.1
      0.212487
      17.244282
      2.444862
      0.999998
      1
      NaN
      NaN
      1.0
      NaN
      NaN
      NaN

      9
      Caucasian
      39
      0
      1
      0
      4.9
      3.9
      0.403717
      3.850335
      2.547805
      0.199410
      0
      10.0
      9.0
      NaN
      1.0
      NaN
      NaN

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      11748
      Caucasian
      23
      0
      7
      0
      3.3
      3.7
      0.408541
      4.307735
      2.419784
      0.242168
      0
      NaN
      NaN
      NaN
      1.0
      NaN
      NaN

      11749
      African-American
      56
      0
      1
      0
      6.6
      5.6
      0.363547
      5.242756
      2.595973
      0.385258
      0
      5.0
      4.0
      NaN
      1.0
      NaN
      NaN

      11751
      African-American
      63
      5
      1
      1
      12.3
      11.3
      0.244161
      10.460653
      2.628265
      0.955169
      1
      NaN
      NaN
      1.0
      NaN
      NaN
      NaN

      11753
      Caucasian
      22
      3
      7
      1
      6.2
      0.8
      0.480011
      6.456006
      2.413485
      0.574930
      1
      5.0
      2.0
      1.0
      NaN
      NaN
      NaN

      11755
      Caucasian
      29
      2
      3
      0
      5.9
      2.9
      0.428004
      5.072526
      2.463512
      0.353278
      0
      7.0
      4.0
      NaN
      1.0
      NaN
      NaN

9309 rows × 18 columns

```python
# nudge risk score for defendants to acheive target accuracy rate
# select subset such that average ra_change is minimized
def optimize_accuracy(df, target):
    n = len(df['y'])
    num_changed = math.ceil(target*n - (df['TP'].sum()+df['TN'].sum()))
    ppl_changed = df.nsmallest(num_changed, 'ra_change')
    return(ppl_changed)

accuracy_rates = np.linspace(0.60, 1, 20)
avg_change = []
max_change = []

for rate in accuracy_rates:
    df_new = optimize_accuracy(df_data, rate)
    if(len(df_new) != math.ceil(rate*len(df_data['y']) - (df_data['TP'].sum()+df_data['TN'].sum()))):
        avg_change.append(np.nan)
        max_change.append(np.nan)
    else:
        avg_change.append(df_new['ra_change'].mean())
        max_change.append(df_new['ra_change'].max())

plt.figure(figsize=(10, 5))
plt.plot(accuracy_rates, avg_change, linestyle='--', marker='o')
plt.title('Average change to RA')
plt.xlabel('Accuracy rate')
plt.ylabel('Average change to RA')
plt.show()

plt.figure(figsize=(10, 5))
plt.plot(accuracy_rates, max_change, linestyle='--', marker='o')
plt.title('Max change to RA')
plt.xlabel('Accuracy rate')
plt.ylabel('Max change to RA')
plt.show()
```

```python
df_data['FP'].sum()+df_data['FN'].sum()
```

    3717.0

```python
# nudge risk score for defendants to acheive target FPR for group
# select subset such that average ra_change is minimized
def optimize_FPR(df, target):
    df_FPs = df.loc[df['FP'] == 1]
    df_FPs = df.loc[df['race'] == 'African-American']
    num_changed = math.ceil(df['FP'].sum() - df['TN'].sum()*target/(1-target))
    ppl_changed = df_FPs.nsmallest(num_changed, 'ra_change')
    return(ppl_changed)

FPRs = np.linspace(0, .5, 20)
avg_change = []
max_change = []

for rate in FPRs:
    df_new = optimize_FPR(df_data, rate)
    if(len(df_new) != math.ceil(df_data['FP'].sum() - df_data['TN'].sum()*rate/(1-rate))):
        avg_change.append(np.NaN)
        max_change.append(np.NaN)
    else:
        avg_change.append(df_new['ra_change'].mean())
        max_change.append(df_new['ra_change'].max())

plt.figure(figsize=(10, 5))
plt.plot(FPRs, avg_change, linestyle='--', marker='o')
plt.title('Average change to RA')
plt.xlabel('Black FPR')
plt.ylabel('Average change to RA')
plt.show()

plt.figure(figsize=(10, 5))
plt.plot(FPRs, max_change, linestyle='--', marker='o')
plt.title('Max change to RA')
plt.xlabel('Black FPR')
plt.ylabel('Max change to RA')
plt.show()
```

```python

```


# Dropout and maxout
In this lab we will explore the methods of [dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf), a regularisation method which stochastically drops out activations from the model during training, and [maxout](http://www.jmlr.org/proceedings/papers/v28/goodfellow13.pdf), another non-linear transformation that can be used in multiple layer models. This is based on material covered in the [fifth lecture slides](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp05-hid.pdf).

```python
import sys
mlpracticalDir = '/home/student/Dropbox/msc_Artificial_Intelligence/mlp_Machine_Learning_Practical/mlpractical'
sys.path.append(mlpracticalDir)
```

```python
import numpy as np
```

## Exercise 1: Implementing a dropout layer

During training the forward propagation through a dropout layer produces outputs where a subset of the input dimensions are set to zero ('dropped out'). The dimensions to be dropped out are randomly sampled for each new batch, with each dimension having a probability $p$ of being included and the inclusion (or not) of each dimension independent of all the others. If the inputs to a dropout layer are $D$ dimensional vectors then we can represent the dropout operation by an elementwise multiplication by a $D$ dimensional *binary mask* vector $\boldsymbol{m} = \left[m_1 ~ m_2 ~\dots~ m_D\right]^{\rm T}$ where $m_d \sim \text{Bernoulli}(p) ~~\forall d \in \lbrace 1 \dots D\rbrace$.

As a first step implement a `random_binary_mask` function in the cell below to generate a binary mask array of a specified shape, where each value in the outputted array is either a one with probablity `prob_1` or zero with probability `1 - prob_1` and all values are sampled independently.

```python
rng = np.random#.RandomState(seed=0)

prob_1 = 0.8

rng.binomial(1, prob_1, (3,3))
```

    array([[1, 1, 1],
           [1, 1, 1],
           [1, 1, 1]])

```python
def random_binary_mask(prob_1, shape, rng):
    """Generates a random binary mask array of a given shape.

    Each value in the outputted array should be an indepedently sampled
    binary value i.e in {0, 1} with the probability of each value
    being 1 being equal to `prob_1`.

    Args:
        prob_1: Scalar value in [0, 1] specifying probability each
            entry in output array is equal to one.
        shape: Shape of returned mask array.
        rng (RandomState): Seeded random number generator object.

    Returns:
        Random binary mask array of specified shape.
    """
    #raise NotImplementedError()
    return rng.binomial(1, prob_1, shape)
```

Test your `random_binary_mask` function using the cell below (if your implementation is incorrect you will get an `AssertionError` - look at what the assert statement is checking for a clue as to what is wrong).

```python
import numpy as np
test_shapes = [(1, 1000), (10, 10, 10)]
test_probs = [0.1, 0.5, 0.7]
for i in range(10):
    for shape in test_shapes:
        for prob in test_probs:
            output = random_binary_mask(prob, shape, np.random)
            # Check generating correct shape output
            assert output.shape == shape
            # Check all outputs are binary values
            assert np.all((output == 1.) | (output == 0.))
            # Check proportion equal to one plausible
            # This will be noisy so there is a chance this will error
            # even for a correct implementation
            assert np.abs(output.mean() - prob) < 0.1
```

Given a randomly sampled binary mask $\boldsymbol{m}$, the outputs $\lbrace \boldsymbol{y}^{(b)} \rbrace_{b=1}^B$ of the stochastic forward propagation through a dropout layer given a batch of inputs $\lbrace \boldsymbol{x}^{(b)} \rbrace_{b=1}^B$ can be calculated by simply performing an elementwise multiplication of the inputs with the mask

\begin{equation}
  y^{(b)}_d = m_k x^{(b)}_d \qquad \forall d \in \lbrace 1 \dots D \rbrace
\end{equation}

The corresponding partial derivatives required for implementing back-propagation through a dropout layer are

\begin{equation}
  \frac{\partial y^{(b)}_k}{\partial x^{(b)}_d} =
  \begin{cases}
      m_k & \quad k = d \\
      0   & \quad k \neq d
  \end{cases}
  \qquad \forall k,\,d \in \lbrace 1 \dots D \rbrace
\end{equation}

As discussed in the lecture slides, when using a model trained with dropout at test time dimensions are no longer stochastically dropped out and instead all activations are deterministically fed forward through the model. So that the expected (mean) outputs of each layer are the same at test and training we scale the forward propagated inputs during testing by $p$ the probability of each dimension being included in the output. If we denote the deterministically forward-propagated batch of outputs of a dropout layer at test time as $\lbrace \boldsymbol{z}^{(b)} \rbrace_{b=1}^B$ then we have

\begin{equation}
  z^{(b)}_d =
  \mathbb{E}\left[ y^{(b)}_d \right] =
  \sum_{m_d \in \lbrace 0,1 \rbrace} \left( \mathbb{P}\left[\mathrm{m}_d = m_d\right] m_d x^{(b)}_d \right) =
  (p) (1) x^{(b)}_d  + (1-p) (0) x^{(b)}_d =
  p x^{(b)}_d  \qquad \forall d \in \lbrace 1 \dots D \rbrace
\end{equation}

To allow switching between this stochastic training time behaviour and deterministic test time behaviour, a new abstract `StochasticLayer` class has been defined in the `mlp.layers` module. This acts similarly to the layer objects we have already encountered other than adding an extra boolean argument `stochastic` to the `fprop` method interface. When `stochastic = True` (the default) a stochastic forward propagation should be caculated, for dropout this corresponding to $\boldsymbol{x}^{(b)} \to \boldsymbol{y}^{(b)}$ above. When `stochastic = False` a deterministic forward-propagation corresponding to the expected output of the stochastic forward-propagation should be calculated, for dropout this corresponding to $\boldsymbol{x}^{(b)} \to \boldsymbol{z}^{(b)}$ above.

Using the skeleton `DropoutLayer` class definition below, implement the `fprop` and `bprop` methods. You may wish to store the binary mask used in the forward propagation as an attribute of the class for use in back-propagation - it is fine to assume that the `fprop` and `bprop` will always be called in sync.

```python
from mlp.layers import DropoutLayer
# from mlp.layers import StochasticLayer

# class DropoutLayer(StochasticLayer):
#     """Layer which stochastically drops input dimensions in its output."""

#     def __init__(self, rng=None, incl_prob=0.5):
#         """Construct a new dropout layer.

#         Args:
#             rng (RandomState): Seeded random number generator.
#             incl_prob: Scalar value in (0, 1] specifying the probability of
#                 each input dimension being included in the output.
#         """
#         super(DropoutLayer, self).__init__(rng)

#         assert incl_prob > 0. and incl_prob <= 1.

#         self.incl_prob = incl_prob

#         self.rng = np.random if rng is None else rng

#     def fprop(self, inputs, stochastic=True):
#         """Forward propagates activations through the layer transformation.

#         Args:
#             inputs: Array of layer inputs of shape (batch_size, input_dim).
#             stochastic: Flag allowing different deterministic
#                 forward-propagation mode in addition to default stochastic
#                 forward-propagation e.g. for use at test time. If False
#                 a deterministic forward-propagation transformation
#                 corresponding to the expected output of the stochastic
#                 forward-propagation is applied.

#         Returns:
#             outputs: Array of layer outputs of shape (batch_size, output_dim).
#         """
#         #raise NotImplementedError()

#         #return inputs
#         #print random_binary_mask(self.incl_prob, inputs.shape, self.rng)
#         #print inputs.shape
#         #return self.incl_prob * random_binary_mask(self.incl_prob, inputs.shape, self.rng) * inputs

# #         if stochastic:
# #             self.binaryMask = random_binary_mask(self.incl_prob, inputs.shape, self.rng)
# #             return self.binaryMask * inputs
# #         else:
# #             return self.incl_prob * inputs

#         self.binaryMask = random_binary_mask(self.incl_prob, inputs.shape, self.rng) if\
#             stochastic else np.ones(inputs.shape)

#         outputs = self.binaryMask * inputs

#         if stochastic:
#             return outputs
#         else:
#             return self.incl_prob * outputs

#     def bprop(self, inputs, outputs, grads_wrt_outputs):
#         """Back propagates gradients through a layer.

#         Given gradients with respect to the outputs of the layer calculates the
#         gradients with respect to the layer inputs. This should correspond to
#         default stochastic forward-propagation.

#         Args:
#             inputs: Array of layer inputs of shape (batch_size, input_dim).
#             outputs: Array of layer outputs calculated in forward pass of
#                 shape (batch_size, output_dim).
#             grads_wrt_outputs: Array of gradients with respect to the layer
#                 outputs of shape (batch_size, output_dim).

#         Returns:
#             Array of gradients with respect to the layer inputs of shape
#             (batch_size, input_dim).
#         """
#         #raise NotImplementedError()
#         #print .shape
#         return self.binaryMask * grads_wrt_outputs

#     def __repr__(self):
#         return 'DropoutLayer(incl_prob={0:.1f})'.format(self.incl_prob)
```

Test your implementation by running the cell below (if your implementation is incorrect you will get an `AssertionError` - look at what the assert statement is checking for a clue as to what is wrong).

```python
seed = 31102016
rng = np.random.RandomState(seed)
test_incl_probs = [0.1, 0.5, 0.7]
input_shape = (5, 10)
for incl_prob in test_incl_probs:
    layer = DropoutLayer(rng, incl_prob)
    inputs = rng.normal(size=input_shape)
    grads_wrt_outputs = rng.normal(size=input_shape)
    for t in range(100):
        outputs = layer.fprop(inputs, stochastic=True)
        # Check outputted array correct shape
        assert outputs.shape == inputs.shape
        # Check all outputs are either equal to inputs or zero
        assert np.all((outputs == inputs) | (outputs == 0))
        grads_wrt_inputs = layer.bprop(inputs, outputs, grads_wrt_outputs)
        # Check back-propagated gradients only non-zero for non-zero outputs
        assert np.all((outputs != 0) == (grads_wrt_inputs != 0))
        assert np.all(grads_wrt_outputs[outputs != 0] == grads_wrt_inputs[outputs != 0])
    det_outputs = layer.fprop(inputs, stochastic=False)
    # Check deterministic fprop outputs are correct shape
    assert det_outputs.shape == inputs.shape
    # Check deterministic fprop outputs scaled correctly
    assert np.allclose(det_outputs, incl_prob * inputs)
```

### Optional extension

Above we assumed the same dropout mask was applied to each input in a batch, as specified in the lecture slides. In practice sometimes a different mask is sampled for each input. As an extension you could try implementing this per-input form of dropout either by defining a new layer or adding an extra argument to the constructor of the above layer which allows you to switch between the two forms.

## Exercise 2: Training with dropout

Experiment with training models with dropout layers to classify MNIST digits. Code has been provided below as a starting point for setting up the model objects though feel free to use any additional adaptive learning rules or learning rule schedulers you wrote during the coursework instead. You may also wish to change the model architecture to use a larger model with more parameters in which the regularisation provided by dropout is likely to have a more pronounced effect. You will probably also find that models with dropout generally need to be trained over more epochs than those without (can you suggest why this might be?).

You should training with a few different `incl_prob` settings for the dropout layers and try to establish how the values chosen affect the training performance. You may wish to experiment with using a different dropout probability at the input than for the intermediate layers (why?).

You may wish to start reading through and implementing exercise 3 while waiting for training runs to complete.

#### my answer

Well since dropout erases some of the nodes of the neural network then you would expect to train the network gradually.
In addition since the dropout can be imagined as a combination of an exponential number of networks then you would imagine that all this training of all these networks requires more time.

Yes indeed the probability of the input must be higher than the probability of the intermediate layers because we must give the chance to the neural network to see all possible variations of our inputs.
If we use a low probability for the input is like turning off many of the pixels of our input images and we will not give "the whole picture" to our neural network. It might miss some important features.

```python
import numpy as np
import logging
from mlp.data_providers import MNISTDataProvider
from mlp.models import MultipleLayerModel
from mlp.layers import ReluLayer, AffineLayer
from mlp.errors import CrossEntropySoftmaxError
from mlp.initialisers import GlorotUniformInit, ConstantInit
from mlp.learning_rules import MomentumLearningRule
from mlp.optimisers import Optimiser
import matplotlib.pyplot as plt
%matplotlib inline

# Seed a random number generator
seed = 31102016
rng = np.random.RandomState(seed)

# Set up a logger object to print info about the training run to stdout
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers = [logging.StreamHandler()]

# Create data provider objects for the MNIST data set
train_data = MNISTDataProvider('train', batch_size=50, rng=rng)
valid_data = MNISTDataProvider('valid', batch_size=50, rng=rng)
```

## Simple testing (not experimenting) with Annealed Dropout Implementation

```python
from mlp.schedulers import AnnealedDropoutScheduler
```

```python
print "aa: " + ", ".join(["ee", "tt"])
```

    aa: ee, tt

```python
# Probability of input being included in output in dropout layer
startInputInclProb = 0.8
startHiddenInclProb = 0.4
num_epochs = 100

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, startInputInclProb),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),

    DropoutLayer(rng, startHiddenInclProb),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),

    DropoutLayer(rng, startHiddenInclProb),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(model, error, learning_rule, train_data, valid_data, data_monitors,
                      schedulers=[
        AnnealedDropoutScheduler(model = model,
                         startInputInclProb=startInputInclProb,
                         startHiddenInclProb=startHiddenInclProb,
                         epochs = num_epochs,
                         holdOnPercent=0.2)
    ])

stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

## Lab starts here

```python
# Probability of input being included in output in dropout layer
incl_prob = 0.5

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, incl_prob),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

num_epochs = 100
stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

Yes indeed after 100 epochs it has not reached at a high place, so it seems that dropout needs more time

```python
# Probability of input being included in output in dropout layer
incl_prob = 0.15
num_epochs = 100

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, incl_prob),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

It seems that with very low probability we have underfitted

```python
# Probability of input being included in output in dropout layer
incl_prob = 0.85
num_epochs = 100

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, incl_prob),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

We have managed to achieve better regularization meaning steady error which is not increasing as the epochs go by. We have also managed to achieve better final accuracy at 97%

```python
# Probability of input being included in output in dropout layer
incl_prob = 0.5
num_epochs = 200

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, incl_prob),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

Even after 200 epochs we didn't get any actual improvement in final accuracy. We could blame the learning rule here as well.

```python
# Probability of input being included in output in dropout layer
initial_prob = 0.9
incl_prob = 0.5
num_epochs = 100

input_dim, output_dim, hidden_dim = 784, 10, 125

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng, gain=2.**0.5)
biases_init = ConstantInit(0.)

# Create three affine layer model with rectified linear non-linearities
# and dropout layers before every affine layer
model = MultipleLayerModel([
    DropoutLayer(rng, initial_prob),
    AffineLayer(input_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, hidden_dim, weights_init, biases_init),
    ReluLayer(),
    DropoutLayer(rng, incl_prob),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

Still here it seems that we have not achieved better results than before since we have probably regularized too soon.

But we should note that when comparing the above with the case where the probability was 0.5 for all layers including the inputs (here we have the inputs to have 0.9 probability) we got from 93% to more than 95% final training accuracy and we have not overfitted

## Exercise 3: Implementing maxout

[Maxout](http://www.jmlr.org/proceedings/papers/v28/goodfellow13.pdf) can be considered a generalisation of the rectified linear transformation implemented in the previous lab.

For a rectified linear (`Relu`) layer the forward propagation corresponds to

\begin{equation}
    y^{(b)}_k =
    \max\left\lbrace 0,\,x^{(b)}_k \right\rbrace
\end{equation}

i.e. each output corresponds to an pairwise maximum of a constant (0) and the input.

Instead of taking the maximum of the input and a constant, we could instead consider taking the maximum over sets of inputs of a fixed size $s$.

\begin{equation}
    y^{(b)}_k =
    \max\left\lbrace x^{(b)}_{(k-1)s + 1},\, x^{(b)}_{(k-1)s + 2},\, \dots ,\, x^{(b)}_{ks} \right\rbrace
\end{equation}

If these inputs $x^{(b)}_d$ are themselves the outputs of an affine layer, then this corresponds to taking the maximum of a series of affine functions of the previous layer outputs. Like a rectified linear layer this leads to piecewise linear input-output relationships (which have well-behaved gradients which do not suffer from the saturation problems of logistic sigmoid / hyperbolic tangent transformations) but unlike the rectified linear case we do not end force a portion of the outputs to be zero.

Experimentally this form of transformation has been found to give good performance, with the name *maxout* chosen because the *out*put is the *max*imum of a set of inputs. Maxout is also commonly used with dropout layers however note they are not directly related - maxout defines a deterministic non-linear transformation which can help improve the representational capacity and trainability of models; dropout defines a stochastic transformation which is mainly aimed at regularising a model to reduce overfitting.

Using layers which take the maximum of fixed sized sets of inputs is also a common technique in models with convolutional layers which we will cover later in the course, with here the layer commonly being termed a *max-pooling* layer (with there being natural generalisation to other choices of reduction functions over pools such as the mean). We will adopt this terminology here for a layer implementing the transformation described above and we will be able to reuse our code implementing this maximum operation when experimenting with convolutional models.

The partial derivatives of this max-pooling transformation are sparse (lots of values are zero), with only the partial derivative of the output of a pool with respect to the maximum input in the pool non-zero. This can be expressed as

\begin{equation}
    \frac{\partial y^{(b)}_k}{\partial x^{(b)}_d} =
    \begin{cases}
      1 & \quad (k-1)s + 1 \leq d  \leq ks \quad\textrm{and} &x^{(b)}_d = \max\left\lbrace x^{(b)}_{(k-1)s + 1},\, x^{(b)}_{(k-1)s + 2},\, \dots ,\, x^{(b)}_{ks} \right\rbrace \\
      0 & \quad \textrm{otherwise}
    \end{cases}.
\end{equation}

Using these definitions implement the `fprop` and `bprop` methods of the skeleton `MaxPoolingLayer` class below.

Some hints

  * One way of organising the inputs into non-overlapping pools is using the `numpy.reshape` function.
  * The `numpy.max` function has an `axis` argument which allows you specify the axis (dimension) of the input array to take the maximum over.
  * It may help to construct a binary mask corresponding to the definitions of the partial derivatives above to allow you to implement the `bprop` method.
  * As with the `DropoutLayer` it is fine to temporarily store values calculated in the `fprop` method as attributes of the object (e.g. `self.val = val`) to use in the `bprop` method (although you don't necessarily need to do this).

```python
test_inputs = np.array([[-3, -4, 5, 8], [0, -2, 3, -8], [1, 5, 3, 2]])
shape = test_inputs.shape
#shape[-1] = 3
#shape = list(shape)
#shape[-1] = 3
#shape[:len(shape)-1] + (shape[-1]/2, 2)
#test_inputs.max(axis=-1)
newShape = shape[:len(shape)-1] + (shape[-1]/2, 2)

reshaped = np.reshape(test_inputs, newShape)
reshaped
```

    array([[[-3, -4],
            [ 5,  8]],

           [[ 0, -2],
            [ 3, -8]],

           [[ 1,  5],
            [ 3,  2]]])

```python
#maxes = reshaped.max(axis=-1)
maxes = np.amax(reshaped,axis=-1, keepdims=True)
maxes
```

    array([[[-3],
            [ 8]],

           [[ 0],
            [ 3]],

           [[ 5],
            [ 3]]])

```python
reshaped.argmax()
```

    3

```python
#np.argmax(reshaped, axis=-1)
args = reshaped.argmax(axis=-1)
args
#reshaped[0,0,1]
```

    array([[0, 1],
           [0, 0],
           [1, 0]])

```python
np.unravel_index(args, reshaped.shape)
```

    (array([[0, 0],
            [0, 0],
            [0, 0]]), array([[0, 0],
            [0, 0],
            [0, 0]]), array([[0, 1],
            [0, 0],
            [1, 0]]))

```python
np.unravel_index(np.array([3,4]), (4,2))
```

    (array([1, 2]), array([1, 0]))

```python
test = np.array([0,1])
result = np.array([[1, 0], [0, 1]])
```

```python
#np.apply_along_axis(lambda elem: )
s = 2
```

```python
a = np.zeros(s)
#a[0] = 1
a
```

    array([ 0.,  0.])

```python
reshaped == maxes
```

    array([[[ True, False],
            [False,  True]],

           [[ True, False],
            [ True, False]],

           [[False,  True],
            [ True, False]]], dtype=bool)

```python
(reshaped == maxes)*1
```

    array([[[1, 0],
            [0, 1]],

           [[1, 0],
            [1, 0]],

           [[0, 1],
            [1, 0]]])

```python
mask = np.array([[[0, 0, 0, 1]], [[0, 0, 1, 0]], [[0, 1, 0, 0]]])
mask
```

    array([[[0, 0, 0, 1]],

           [[0, 0, 1, 0]],

           [[0, 1, 0, 0]]])

```python
incomings = np.array([[10], [5], [-3]])
print incomings.shape
incomings
```

    (3, 1)

    array([[10],
           [ 5],
           [-3]])

```python
mask.shape
```

    (3, 1, 4)

```python
shape
```

    (3, 4)

```python
incomingsNewShape = incomings.shape + (1,)
incomingsNewShape
```

    (3, 1, 1)

```python
incomingsD = np.reshape(incomings, incomingsNewShape)
incomingsD
```

    array([[[10]],

           [[ 5]],

           [[-3]]])

```python
result = mask * incomingsD
result
```

    array([[[ 0,  0,  0, 10]],

           [[ 0,  0,  5,  0]],

           [[ 0, -3,  0,  0]]])

```python
np.squeeze(result)
```

    array([[ 0,  0,  0, 10],
           [ 0,  0,  5,  0],
           [ 0, -3,  0,  0]])

```python
originallyShaped = np.reshape(mask, shape)
```

```python
originallyShaped
```

    array([[0, 0, 0, 1],
           [0, 0, 1, 0],
           [0, 1, 0, 0]])

```python
originallyShaped * incomings
```

    array([[ 0,  0,  0, 10],
           [ 0,  0,  5,  0],
           [ 0, -3,  0,  0]])

```python
mask * incomings[np.newaxis]
```

    array([[[ 0,  0,  0, 10],
            [ 0,  0,  0,  5],
            [ 0,  0,  0, -3]],

           [[ 0,  0, 10,  0],
            [ 0,  0,  5,  0],
            [ 0,  0, -3,  0]],

           [[ 0, 10,  0,  0],
            [ 0,  5,  0,  0],
            [ 0, -3,  0,  0]]])

SECTION 2

```python
mask = np.array([[[1, 0],  [0, 1]], [[1, 0],  [1, 0]], [[0, 1],  [1, 0]]])
mask
```

    array([[[1, 0],
            [0, 1]],

           [[1, 0],
            [1, 0]],

           [[0, 1],
            [1, 0]]])

```python
grads = np.array([[3, -1], [2, 5], [5, 3]])
print grads.shape
grads
```

    (3, 2)

    array([[ 3, -1],
           [ 2,  5],
           [ 5,  3]])

```python
mask.shape
```

    (3, 2, 2)

```python
grads.shape
```

    (3, 2)

```python
grads[np.newaxis]
```

    array([[[ 3, -1],
            [ 2,  5],
            [ 5,  3]]])

```python
gradsD = np.reshape(grads, (3, 2, 1))
gradsD
```

    array([[[ 3],
            [-1]],

           [[ 2],
            [ 5]],

           [[ 5],
            [ 3]]])

```python
result = mask * gradsD
result
```

    array([[[ 3,  0],
            [ 0, -1]],

           [[ 2,  0],
            [ 5,  0]],

           [[ 0,  5],
            [ 3,  0]]])

```python
np.reshape(result, (3,4))
```

    array([[ 3,  0,  0, -1],
           [ 2,  0,  5,  0],
           [ 0,  5,  3,  0]])

https://www.quora.com/How-are-gradients-computed-through-max-pooling-or-maxout-units-in-neural-networks

```python
from mlp.layers import Layer

class MaxPoolingLayer(Layer):

    def __init__(self, pool_size=2):
        """Construct a new max-pooling layer.

        Args:
            pool_size: Positive integer specifying size of pools over
               which to take maximum value. The outputs of the layer
               feeding in to this layer must have a dimension which
               is a multiple of this pool size such that the outputs
               can be split in to pools with no dimensions left over.
        """
        assert pool_size > 0
        self.pool_size = pool_size

    def fprop(self, inputs):
        """Forward propagates activations through the layer transformation.

        This corresponds to taking the maximum over non-overlapping pools of
        inputs of a fixed size `pool_size`.

        Args:
            inputs: Array of layer inputs of shape (batch_size, input_dim).

        Returns:
            outputs: Array of layer outputs of shape (batch_size, output_dim).
        """
        assert inputs.shape[-1] % self.pool_size == 0, (
            'Last dimension of inputs must be multiple of pool size')
        #return np.maximum(0, inputs) #from relu
        shape = inputs.shape

        newShape = shape[:len(shape)-1] + (shape[-1]/self.pool_size, self.pool_size)

        pools = np.reshape(inputs, newShape)

        self.binaryMask = (np.amax(pools, axis=-1, keepdims=True) == pools)*1

        # here we want by column or by row? Are we using multiple instances or multiple attributes ?????
        # it seems like multiple attributes
        return pools.max(axis=-1)

    def bprop(self, inputs, outputs, grads_wrt_outputs):
        """Back propagates gradients through a layer.

        Given gradients with respect to the outputs of the layer calculates the
        gradients with respect to the layer inputs.

        Args:
            inputs: Array of layer inputs of shape (batch_size, input_dim).
            outputs: Array of layer outputs calculated in forward pass of
                shape (batch_size, output_dim).
            grads_wrt_outputs: Array of gradients with respect to the layer
                outputs of shape (batch_size, output_dim).

        Returns:
            Array of gradients with respect to the layer inputs of shape
            (batch_size, input_dim).
        """
        #raise NotImplementedError()

        #print inputs.shape
        #print grads_wrt_outputs.shape
        #print self.binaryMask

        #maskOriginallyShaped = np.reshape(self.binaryMask, inputs.shape)
        #print maskOriginallyShaped

        grads_wrt_outputs_plusOneDimension = np.reshape(grads_wrt_outputs, grads_wrt_outputs.shape + (1,))

        grads_wrt_inputs_malformed = np.squeeze(self.binaryMask * grads_wrt_outputs_plusOneDimension)

        grads_wrt_inputs = np.reshape(grads_wrt_inputs_malformed, inputs.shape)

        return grads_wrt_inputs

        #we have a max in every column

    def __repr__(self):
        return 'MaxPoolingLayer(pool_size={0})'.format(self.pool_size)
```

Test your implementation by running the cell below.

```python
test_inputs = np.array([[-3, -4, 5, 8], [0, -2, 3, -8], [1, 5, 3, 2]])
test_outputs_1 = np.array([[8], [3], [5]])
test_grads_wrt_outputs_1 = np.array([[10], [5], [-3]])
test_grads_wrt_inputs_1 = np.array([[0, 0, 0, 10], [0, 0, 5, 0], [0, -3, 0, 0]])
test_outputs_2 = np.array([[-3, 8], [0, 3], [5, 3]])
test_grads_wrt_outputs_2 = np.array([[3, -1], [2, 5], [5, 3]])
test_grads_wrt_inputs_2 = np.array([[3, 0, 0, -1], [2, 0, 5, 0], [0, 5, 3, 0]])
layer_1 = MaxPoolingLayer(4)
layer_2 = MaxPoolingLayer(2)
# Check fprop with pool_size = 4
assert np.allclose(layer_1.fprop(test_inputs), test_outputs_1)
# Check bprop with pool_size = 4
assert np.allclose(
    layer_1.bprop(test_inputs, test_outputs_1, test_grads_wrt_outputs_1),
    test_grads_wrt_inputs_1
)
# Check fprop with pool_size = 2
assert np.allclose(layer_2.fprop(test_inputs), test_outputs_2)
# Check bprop with pool_size = 2
assert np.allclose(
    layer_2.bprop(test_inputs, test_outputs_2, test_grads_wrt_outputs_2),
    test_grads_wrt_inputs_2
)
```

## Exercise 4: Training with maxout

Use your `MaxPoolingLayer` implementation in a multiple layer models to experiment with how well maxout networks are able to classify MNIST digits. As with the dropout training exercise, code has been provided below as a starting point for setting up the model objects, but again feel free to substitute any components.

If you have time you may wish to experiment with training a model using a combination of maxout and dropout or another regularisation method covered in the last lab notebook.

```python
import numpy as np
import logging
from mlp.data_providers import MNISTDataProvider
from mlp.models import MultipleLayerModel
from mlp.layers import AffineLayer
from mlp.errors import CrossEntropySoftmaxError
from mlp.initialisers import GlorotUniformInit, ConstantInit
from mlp.learning_rules import MomentumLearningRule
from mlp.optimisers import Optimiser
import matplotlib.pyplot as plt
%matplotlib inline

# Seed a random number generator
seed = 31102016
rng = np.random.RandomState(seed)

# Set up a logger object to print info about the training run to stdout
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers = [logging.StreamHandler()]

# Create data provider objects for the MNIST data set
train_data = MNISTDataProvider('train', batch_size=50, rng=rng)
valid_data = MNISTDataProvider('valid', batch_size=50, rng=rng)
```

```python
# Size of pools to take maximum over
pool_size = 2

input_dim, output_dim, hidden_dim = 784, 10, 100

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng)
biases_init = ConstantInit(0.)

# Create three affine layer model interleaved with max-pooling layers
model = MultipleLayerModel([
    AffineLayer(input_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

num_epochs = 100
stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

The above settings for maxout seem to work very awesome except the fact that it in ~15%. But the validation accuracy remains at very good levels ~98.3%

```python
# Size of pools to take maximum over
pool_size = 3

input_dim, output_dim, hidden_dim = 784, 10, 100

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng)
biases_init = ConstantInit(0.)

# Create three affine layer model interleaved with max-pooling layers
model = MultipleLayerModel([
    AffineLayer(input_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

num_epochs = 100
stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```
For pool size = 3 we have the same behaviour as before ~98.3 accuracy even if at some point peaked at ~98.4. The overfitting is worse in this case and it happens within 90 epochs. The validation error is almost the same as the error

```python
# Size of pools to take maximum over
pool_size = 4

input_dim, output_dim, hidden_dim = 784, 10, 100

# Use Glorot initialisation scheme for weights and zero biases
weights_init = GlorotUniformInit(rng=rng)
biases_init = ConstantInit(0.)

# Create three affine layer model interleaved with max-pooling layers
model = MultipleLayerModel([
    AffineLayer(input_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, hidden_dim * pool_size, weights_init, biases_init),
    MaxPoolingLayer(pool_size),
    AffineLayer(hidden_dim, output_dim, weights_init, biases_init)
])

# Multiclass classification therefore use cross-entropy + softmax error
error = CrossEntropySoftmaxError()

# Use a momentum learning rule - you could use an adaptive learning rule
# implemented for the coursework here instead
learning_rule = MomentumLearningRule(0.02, 0.9)

# Monitor classification accuracy during training
data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

optimiser = Optimiser(
    model, error, learning_rule, train_data, valid_data, data_monitors)

num_epochs = 100
stats_interval = 5

stats, keys, run_time = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

# Plot the change in the validation and training set error over training.
fig_1 = plt.figure(figsize=(8, 4))
ax_1 = fig_1.add_subplot(111)
for k in ['error(train)', 'error(valid)']:
    ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_1.legend(loc=0)
ax_1.set_xlabel('Epoch number')

# Plot the change in the validation and training set accuracy over training.
fig_2 = plt.figure(figsize=(8, 4))
ax_2 = fig_2.add_subplot(111)
for k in ['acc(train)', 'acc(valid)']:
    ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
              stats[1:, keys[k]], label=k)
ax_2.legend(loc=0)
ax_2.set_xlabel('Epoch number')
```

The amazing thing with the maxout when pool size = 4 is that it is indeed slower because it involves more connections and again overfits within 10 epochs but the final (overfitted) validation error is lower than the two previous experience. Also the training accuracy is slightly but consistently higher at ~98.4%

```python

```


## SurfinPy

#### Tutorial 2 - Introducing temperature dependence

In tutorial 1 we generated a phase diagram at 0K. However this is not representative of normal conditions. Temperature is an important consideration for materials chemists and we may wish to evaluate the state of a solid electrolyte at the operating temperature or synthesis conditions. In order to overcome this and introduce temperature we need to modify the equation for our surface energy in tutorial 1. This explanation will again be using $TiO_2$

\begin{align}
\gamma_{Surf} & = \frac{1}{2A} \Bigg( E_{TiO_2}^{slab} - \frac{nTi_{Slab}}{nTi_{Bulk}} E_{TiO_2}^{Bulk} \Bigg) - \Gamma_O \mu_O - \Gamma_{H_2O} \mu_{H_2O} - n_O \mu_O (T) - n_{H_2O} \mu_{H_2O} (T)
\end{align}

where A is the surface area, $E_{TiO_2}^{slab}$ is the DFT energy of the slab, $nTi_{Slab}$ is the number of cations in the slab, $nTi_{Bulk}$ is the number of cations in the bulk unit cell, $E_{TiO_2}^{Bulk}$ is the DFT energy of the bulk unit cell and

\begin{align}
\Gamma_O & = \frac{1}{2A} \Bigg( nO_{Slab} - \frac{nO_{Bulk}}{nTi_{Bulk}}nTi_{Slab}  \Bigg) ,
\end{align}

\begin{align}
\Gamma_{H_2O} & = \frac{nH_2O}{2A} ,
\end{align}

$nO_{Slab}$ is the number of anions in the slab, $nO_{Bulk}$ is the number of anions in the bulk, $nH_2O$ is the number of adsorbing water molecules and $n_O$ is the number of defective oxygen. $\Gamma_O$ / $\Gamma_{H_2O}$ is the excess oxygen / water at the surface and $\mu_O$ / $\mu_{H_2O}$ is the oxygen / water chemcial potential. $\mu_{H_2O} $(T) / $\mu_O (T)$ are terms to correct the chemcial potential of oxygen and water based on thermochemical data from the NIST_JANAF database, making the chemical potential a temperature dependent term.

\begin{align}
\mu_O (T) &  = \frac{1}{2} \mu_O (T) (0 K , DFT) +  \frac{1}{2} \mu_O (T) (0 K , EXP) +  \frac{1}{2} \Delta G_{O_2} ( \Delta T, Exp)
\end{align}

where $\mu_O$ (T) (0 K , DFT) is the 0K free energy of an isolated oxygen moleculeevaluated with DFT, $\mu_O$ (T) (0 K , EXP) is the 0 K experimental  Gibbs energy for oxygen gas and $\Delta$ $G_{O_2}$ ( $\Delta$ T, Exp) is the Gibbs energy defined at temperature T as

\begin{align}
\Delta G_{O_2} ( \Delta T, Exp) &  = \frac{1}{2} [H(T, {O_2}) -  H(0 K, {O_2})] -  \frac{1}{2} T[S(T, {O_2}])
\end{align}

```python
import matplotlib.pyplot as plt
from surfinpy import mu_vs_mu
from surfinpy import utils as ut
from surfinpy import data
```

In order to calculate our $\Delta$ $G_{O_2}$ ( $\Delta$ T, Exp) values we need to use experimental data from the NIST-JANAF database. As a user you need to download the tables for the species you are interested in ( In our case oxygen and water). Surfinpy has a function that can read this data, assuming it is in the correct format and calculate the temperature correction for you. Provide the path to the file and the temperature that you want as an index.

```python
Oxygen_exp = ut.fit_nist("O2.txt")[298]
Water_exp = ut.fit_nist("H2O.txt")[298]
```

-9.08 is the DFT energy of an oxygen molecule, -0.86 is the zero point energy and Oxygen_exp is the experimental free energy at 298 K.

```python
Oxygen_corrected = (-9.08 + -0.86 + Oxygen_exp) / 2
Water_corrected = -14.84 + 0.55 + Water_exp
print(Oxygen_corrected)
print(Water_corrected)
```

    -5.2427609629166
    -14.77234481554258

Oxygen_corrected and Water_corrected are now temperature dependent terms corresponding to a temperature of 298 K. The resulting phase diagram will now be at a temperature of 298 K.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 2, energy = -780.0, funits = 4)

pure = data.DataSet(cation = 24, x = 48, y = 0, area = 60.0, energy = -575.0, label = "0.00 $TiO_2$", nspecies = 1)
H2O = data.DataSet(cation = 24, x = 48, y = 2, area = 60.0, energy = -612.0, label = "0.16 $TiO_2$", nspecies = 1)
H2O_2 = data.DataSet(cation = 24, x = 48, y = 4, area = 60.0, energy = -640.0, label = "0.32 $TiO_2$", nspecies = 1)
H2O_3 = data.DataSet(cation = 24, x = 48, y = 8, area = 60.0, energy = -676.0, label = "0.64 $TiO_2$", nspecies = 1)
Vo = data.DataSet(cation = 24, x = 46, y = 0, area = 60.0, energy = -558.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_1 = data.DataSet(cation = 24, x = 46, y = 2, area = 60.0, energy = -594.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_2 = data.DataSet(cation = 24, x = 46, y = 4, area = 60.0, energy = -624.0, label = "0.16 $TiO_1.9$", nspecies = 1)
H2O_Vo_3 = data.DataSet(cation = 24, x = 46, y = 6, area = 60.0, energy = -640.0, label = "0.32 $TiO_1.9$", nspecies = 1)
H2O_Vo_4 = data.DataSet(cation = 24, x = 46, y = 8, area = 60.0, energy = -670.0, label = "0.64 $TiO_1.9$", nspecies = 1)

data = [pure, Vo, H2O, H2O_Vo_1, H2O_2, H2O_Vo_2, H2O_3, H2O_Vo_3,  H2O_Vo_4]

```

```python
deltaX = {'Range': [ -12, -6],  'Label': 'O'}
deltaY = {'Range': [ -19, -12], 'Label': 'H_2O'}
```

```python
system =  mu_vs_mu.calculate(data, bulk, deltaX, deltaY, x_energy=Oxygen_corrected, y_energy=Water_corrected)
```

```python
ax = system.plot_phase(temperature=298, set_style="fast", colourmap="RdBu", cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_3.png", dpi=600)
plt.show()
```


# Axion-photon coupling limits vs axion mass

Axion mass-PQ scale relation:
\begin{equation}
m_{a}=5.70(7) \mu \mathrm{eV}\left(\frac{10^{12} \,\mathrm{GeV}}{f_{a}}\right)
\end{equation}

Axion-photon coupling
\begin{equation}
g_{a \gamma} \equiv \frac{\alpha}{2 \pi} \frac{C_{a\gamma}}{f_{a}}=2.0 \times 10^{-16} C_{a \gamma} \frac{m_{a}}{\mu \mathrm{eV}} \mathrm{GeV}^{-1}
\end{equation}

Model dependent constant:
\begin{equation}
C_{a\gamma} \equiv \frac{E}{N} - 1.92(4) =
\begin{cases}
  -1.92, & {\rm KSVZ} \\
  0.75, & {\rm DFSZ\,I} \\
  -1.25, & {\rm DFSZ\,II}
\end{cases}
\end{equation}

```python
% matplotlib inline
from numpy import *
import matplotlib.pyplot as plt
from PlotFuncs import FigSetup, AxionPhoton, MySaveFig, BlackHoleSpins

fig, ax = FigSetup(Shape='Rectangular', ylab='$|g_{a\gamma}|$ [GeV$^{-1}$]', mathpazo=True)

AxionPhoton.QCDAxion(ax, C_center=abs(5 / 3 - 1.92) * (44 / 3 - 1.92) / 2, C_width=0.7, vmax=1.1)
AxionPhoton.Cosmology(ax)
AxionPhoton.StellarBounds(ax)
AxionPhoton.Haloscopes(ax)
AxionPhoton.Helioscopes(ax)
AxionPhoton.LSW(ax)
AxionPhoton.LowMassAstroBounds(ax)
AxionPhoton.ALPdecay(ax)
AxionPhoton.NeutronStars(ax)
BlackHoleSpins(ax, (1 / 137) * (1 / (2 * pi)), [2.1e-12, 1.5e-17], rotation=-90, alpha=0.3)

# this is bad, need a better solution to this
plt.arrow(0.265, 0.535, 0, 0.035, transform=fig.transFigure, figure=fig,
          length_includes_head=True, lw=1,
          head_width=0.007, head_length=0.016, overhang=0.13,
          edgecolor='crimson', facecolor='crimson', clip_on=True)

MySaveFig(fig, 'AxionPhoton')
```

```python
fig, ax = FigSetup(Shape='Rectangular', ylab='$|g_{a\gamma}|$ [GeV$^{-1}$]', mathpazo=True)

AxionPhoton.QCDAxion(ax, text_on=False)
AxionPhoton.Cosmology(ax)
AxionPhoton.StellarBounds(ax)
AxionPhoton.Haloscopes(ax, projection=True)
AxionPhoton.Helioscopes(ax, fs=23, projection=True)
AxionPhoton.LSW(ax, projection=True)
AxionPhoton.LowMassAstroBounds(ax, projection=True)
AxionPhoton.ALPdecay(ax, projection=True)
AxionPhoton.NeutronStars(ax)
BlackHoleSpins(ax, (1 / 137) * (1 / (2 * pi)), [2.1e-12, 1.5e-17], rotation=-90, alpha=0.3)

MySaveFig(fig, 'AxionPhoton_with_Projections')
```

```python
% matplotlib inline
from numpy import *
import matplotlib.pyplot as plt
from PlotFuncs import BlackHoleSpins, AxionPhoton, MySaveFig

fig, ax = FigSetup(Shape='Wide', ylab='$|C_{a\gamma}|$', mathpazo=True,
                   g_min=0.01, g_max=100000.0, m_min=1e-9, m_max=1,
                   tickdir='in', TopAndRightTicks=True)

AxionPhoton.QCDAxion(ax, RescaleByMass=True, fs=20, KSVZ_on=False, DFSZ_on=True, DFSZ_label_mass=1e-2)
AxionPhoton.ADMX(ax, fs=25, RescaleByMass=True, projection=True)
AxionPhoton.ADMX_SLIC(ax, RescaleByMass=True)
AxionPhoton.RBF_UF(ax, fs=20, RescaleByMass=True)
AxionPhoton.HAYSTAC(ax, fs=15, RescaleByMass=True, col='violet')
AxionPhoton.CAPP(ax, fs=17, RescaleByMass=True, col='deeppink')
AxionPhoton.DMRadio(ax, RescaleByMass=True)
AxionPhoton.QUAX(ax, fs=17, RescaleByMass=True)
AxionPhoton.RADES(ax, fs=12, RescaleByMass=True)
AxionPhoton.MADMAX(ax, fs=20, RescaleByMass=True)
AxionPhoton.ALPHA(ax, fs=20, RescaleByMass=True)
AxionPhoton.ORGAN(ax, fs=20, RescaleByMass=True, projection=True)
AxionPhoton.KLASH(ax, fs=20, RescaleByMass=True)
AxionPhoton.BRASS(ax, fs=18, RescaleByMass=True)
AxionPhoton.TOORAD(ax, fs=12, RescaleByMass=True)
AxionPhoton.LAMPOST(ax, fs=18, RescaleByMass=True)
AxionPhoton.NeutronStars(ax, fs=25, RescaleByMass=True, text_shift=[0.4, 0.4])
AxionPhoton.ALPS(ax, projection=True, RescaleByMass=True)
AxionPhoton.Helioscopes(ax, col='royalblue', fs=35, projection=True, RescaleByMass=True)

MySaveFig(fig, 'AxionPhoton_Rescaled')
```

```python
% matplotlib inline
from numpy import *
import matplotlib.pyplot as plt
from PlotFuncs import BlackHoleSpins, AxionPhoton, MySaveFig

fig, ax = FigSetup(Shape='Wide', ylab='$E/N - 1.92$', mathpazo=True,
                   g_min=0.01, g_max=100000.0, m_min=1e-9, m_max=1,
                   tickdir='in', TopAndRightTicks=True)

AxionPhoton.QCDAxion(ax, RescaleByMass=True, fs=20, text_on=False, DFSZ_on=False, KSVZ_on=False)
ax.axhline(44 / 3 - 1.92, color='k', lw=5, zorder=0)
ax.axhline(abs(5 / 3 - 1.92), color='k', lw=5, zorder=0)
ax.axhline(44 / 3 - 1.92, color='orange', lw=3, zorder=0)
ax.axhline(abs(5 / 3 - 1.92), color='orange', lw=3, zorder=0)
plt.gcf().text(0.14, 0.22, r'$E/N = 5/3$', fontsize=20)
plt.gcf().text(0.14, 0.48, r'$E/N = 44/3$', fontsize=20)

AxionPhoton.ADMX(ax, fs=25, RescaleByMass=True, projection=False)
AxionPhoton.RBF_UF(ax, fs=20, RescaleByMass=True)
AxionPhoton.HAYSTAC(ax, fs=15, RescaleByMass=True, col='violet')
AxionPhoton.CAPP(ax, fs=17, RescaleByMass=True, col='deeppink')
AxionPhoton.RADES(ax, fs=12, RescaleByMass=True)
AxionPhoton.QUAX(ax, fs=17, RescaleByMass=True)
AxionPhoton.ADMX_SLIC(ax, RescaleByMass=True)
AxionPhoton.ORGAN(ax, fs=20, RescaleByMass=True, projection=False)
AxionPhoton.NeutronStars(ax, fs=25, RescaleByMass=True, text_shift=[0.4, 0.4])
AxionPhoton.Helioscopes(ax, col='royalblue', fs=35, projection=False, RescaleByMass=True)

MySaveFig(fig, 'AxionPhoton_Rescaled_NoProjections')
```

```python

```


```
def check_goldbach_for_num(n, primes_set):
    for i in primes_set:
        if (n-i)in primes_set:
            return (True)

```

```
from sympy.ntheory.generate import primerange
primes = set(primerange(50,1000))
check_goldbach_for_num(116, primes)
```


Heavily based on TensorFlow Quantum
[MNIST example](https://www.tensorflow.org/quantum/tutorials/mnist)

```python
%matplotlib inline

import tensorflow as tf
import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np
import seaborn as sns
import collections

# visualization tools
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

```python
# Loading raw data

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Rescale the images from [0,255] to the [0.0,1.0] range.
x_train, x_test = x_train[..., np.newaxis]/255.0, x_test[..., np.newaxis]/255.0

print("Number of original training examples:", len(x_train))
print("Number of original test examples:", len(x_test))
```

    Number of original training examples: 60000
    Number of original test examples: 10000

Dataset filtering

```python
def filter_36(x, y):
    keep = (y == 3) | (y == 6)
    x, y = x[keep], y[keep]
    y = y == 3
    return x,y

x_train, y_train = filter_36(x_train, y_train)
x_test, y_test = filter_36(x_test, y_test)

print("Number of filtered training examples:", len(x_train))
print("Number of filtered test examples:", len(x_test))

print(y_train[0])

plt.imshow(x_train[0, :, :, 0])
plt.colorbar()
```

Post-processing of data

```python
# Downscaling
x_train_small = tf.image.resize(x_train, (4,4)).numpy()
x_test_small = tf.image.resize(x_test, (4,4)).numpy()

print(y_train[0])

plt.imshow(x_train_small[0,:,:,0], vmin=0, vmax=1)
plt.colorbar()

def remove_contradicting(xs, ys):
    mapping = collections.defaultdict(set)
    orig_x = {}
    # Determine the set of labels for each unique image:
    for x,y in zip(xs,ys):
        orig_x[tuple(x.flatten())] = x
        mapping[tuple(x.flatten())].add(y)

    new_x = []
    new_y = []
    for flatten_x in mapping:
      x = orig_x[flatten_x]
      labels = mapping[flatten_x]
      if len(labels) == 1:
          new_x.append(x)
          new_y.append(next(iter(labels)))
      else:
          # Throw out images that match more than one label.
          pass

    num_uniq_3 = sum(1 for value in mapping.values() if len(value) == 1 and True in value)
    num_uniq_6 = sum(1 for value in mapping.values() if len(value) == 1 and False in value)
    num_uniq_both = sum(1 for value in mapping.values() if len(value) == 2)

    print("Number of unique images:", len(mapping.values()))
    print("Number of unique 3s: ", num_uniq_3)
    print("Number of unique 6s: ", num_uniq_6)
    print("Number of unique contradicting labels (both 3 and 6): ", num_uniq_both)
    print()
    print("Initial number of images: ", len(xs))
    print("Remaining non-contradicting unique images: ", len(new_x))

    return np.array(new_x), np.array(new_y)

x_train_nocon, y_train_nocon = remove_contradicting(x_train_small, y_train)
```

```python
THRESHOLD = 0.5

x_train_bin = np.array(x_train_nocon > THRESHOLD, dtype=np.float32)
x_test_bin = np.array(x_test_small > THRESHOLD, dtype=np.float32)

# This would not be beneficial
_ = remove_contradicting(x_train_bin, y_train_nocon)

def convert_to_circuit(image):
    """Encode truncated classical image into quantum datapoint."""
    values = np.ndarray.flatten(image)
    qubits = cirq.GridQubit.rect(4, 4)
    circuit = cirq.Circuit()
    for i, value in enumerate(values):
        if value:
            circuit.append(cirq.X(qubits[i]))
    return circuit

x_train_circ = [convert_to_circuit(x) for x in x_train_bin]
x_test_circ = [convert_to_circuit(x) for x in x_test_bin]

SVGCircuit(x_train_circ[0])
```

    Number of unique images: 193
    Number of unique 3s:  80
    Number of unique 6s:  69
    Number of unique contradicting labels (both 3 and 6):  44

    Initial number of images:  10338
    Remaining non-contradicting unique images:  149

Neural network

```python
x_train_tfcirc = tfq.convert_to_tensor(x_train_circ)
x_test_tfcirc = tfq.convert_to_tensor(x_test_circ)

class CircuitLayerBuilder():
    def __init__(self, data_qubits, readout):
        self.data_qubits = data_qubits
        self.readout = readout

    def add_layer(self, circuit, gate, prefix):
        for i, qubit in enumerate(self.data_qubits):
            symbol = sympy.Symbol(prefix + '-' + str(i))
            circuit.append(gate(qubit, self.readout)**symbol)

demo_builder = CircuitLayerBuilder(data_qubits = cirq.GridQubit.rect(4,1),
                                   readout=cirq.GridQubit(-1,-1))

circuit = cirq.Circuit()
demo_builder.add_layer(circuit, gate = cirq.XX, prefix='xx')
SVGCircuit(circuit)
```

```python
def create_quantum_model():
    """Create a QNN model circuit and readout operation to go along with it."""
    data_qubits = cirq.GridQubit.rect(4, 4)  # a 4x4 grid.
    readout = cirq.GridQubit(-1, -1)         # a single qubit at [-1,-1]
    circuit = cirq.Circuit()

    # Prepare the readout qubit.
    circuit.append(cirq.X(readout))
    circuit.append(cirq.H(readout))

    builder = CircuitLayerBuilder(
        data_qubits = data_qubits,
        readout=readout)

    # Then add layers (experiment by adding more).
    builder.add_layer(circuit, cirq.XX, "xx1")
    builder.add_layer(circuit, cirq.ZZ, "zz1")

    # Finally, prepare the readout qubit.
    circuit.append(cirq.H(readout))

    return circuit, cirq.Z(readout)

model_circuit, model_readout = create_quantum_model()

```

```python
# Build the Keras model.
model = tf.keras.Sequential([
    # The input is the data-circuit, encoded as a tf.string
    tf.keras.layers.Input(shape=(), dtype=tf.string),
    # The PQC layer returns the expected value of the readout gate, range [-1,1].
    tfq.layers.PQC(model_circuit, model_readout),
])

y_train_hinge = 2.0*y_train_nocon-1.0
y_test_hinge = 2.0*y_test-1.0

def hinge_accuracy(y_true, y_pred):
    y_true = tf.squeeze(y_true) > 0.0
    y_pred = tf.squeeze(y_pred) > 0.0
    result = tf.cast(y_true == y_pred, tf.float32)

    return tf.reduce_mean(result)

model.compile(
    loss=tf.keras.losses.Hinge(),
    optimizer=tf.keras.optimizers.Adam(),
    metrics=[hinge_accuracy])

print(model.summary())

```

    Model: "sequential_3"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    pqc_1 (PQC)                  (None, 1)                 32
    =================================================================
    Total params: 32
    Trainable params: 32
    Non-trainable params: 0
    _________________________________________________________________
    None

```python
EPOCHS = 3
BATCH_SIZE = 32

NUM_EXAMPLES = len(x_train_tfcirc)

x_train_tfcirc_sub = x_train_tfcirc[:NUM_EXAMPLES]
y_train_hinge_sub = y_train_hinge[:NUM_EXAMPLES]

qnn_history = model.fit(
      x_train_tfcirc_sub, y_train_hinge_sub,
      batch_size=32,
      epochs=EPOCHS,
      verbose=1,
      validation_data=(x_test_tfcirc, y_test_hinge))

qnn_results = model.evaluate(x_test_tfcirc, y_test)
```

    Epoch 1/3
    324/324 [==============================] - 54s 166ms/step - loss: 0.6229 - hinge_accuracy: 0.7803 - val_loss: 0.3569 - val_hinge_accuracy: 0.8311
    Epoch 2/3
    324/324 [==============================] - 53s 162ms/step - loss: 0.3794 - hinge_accuracy: 0.8132 - val_loss: 0.3410 - val_hinge_accuracy: 0.8337
    Epoch 3/3
    324/324 [==============================] - 54s 166ms/step - loss: 0.3544 - hinge_accuracy: 0.8392 - val_loss: 0.3312 - val_hinge_accuracy: 0.8750
    62/62 [==============================] - 2s 40ms/step - loss: 0.3312 - hinge_accuracy: 0.8750

```python
# Classical
def create_classical_model():
    # A simple model based off LeNet from https://keras.io/examples/mnist_cnn/
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Conv2D(32, [3, 3], activation='relu', input_shape=(28,28,1)))
    model.add(tf.keras.layers.Conv2D(64, [3, 3], activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
    model.add(tf.keras.layers.Dropout(0.25))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(128, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.5))
    model.add(tf.keras.layers.Dense(1))
    return model

model = create_classical_model()
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

model.summary()
```

    Model: "sequential_4"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d_2 (Conv2D)            (None, 26, 26, 32)        320
    _________________________________________________________________
    conv2d_3 (Conv2D)            (None, 24, 24, 64)        18496
    _________________________________________________________________
    max_pooling2d_1 (MaxPooling2 (None, 12, 12, 64)        0
    _________________________________________________________________
    dropout_2 (Dropout)          (None, 12, 12, 64)        0
    _________________________________________________________________
    flatten_2 (Flatten)          (None, 9216)              0
    _________________________________________________________________
    dense_4 (Dense)              (None, 128)               1179776
    _________________________________________________________________
    dropout_3 (Dropout)          (None, 128)               0
    _________________________________________________________________
    dense_5 (Dense)              (None, 1)                 129
    =================================================================
    Total params: 1,198,721
    Trainable params: 1,198,721
    Non-trainable params: 0
    _________________________________________________________________

```python
model.fit(x_train,
          y_train,
          batch_size=128,
          epochs=1,
          verbose=1,
          validation_data=(x_test, y_test))

cnn_results = model.evaluate(x_test, y_test)
```

    95/95 [==============================] - 5s 55ms/step - loss: 0.0470 - accuracy: 0.9851 - val_loss: 0.0045 - val_accuracy: 0.9990
    62/62 [==============================] - 0s 4ms/step - loss: 0.0045 - accuracy: 0.9990

```python
def create_fair_classical_model():
    # A simple model based off LeNet from https://keras.io/examples/mnist_cnn/
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Flatten(input_shape=(4,4,1)))
    model.add(tf.keras.layers.Dense(2, activation='relu'))
    model.add(tf.keras.layers.Dense(1))
    return model

model = create_fair_classical_model()
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

model.summary()

```

    Model: "sequential_5"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten_3 (Flatten)          (None, 16)                0
    _________________________________________________________________
    dense_6 (Dense)              (None, 2)                 34
    _________________________________________________________________
    dense_7 (Dense)              (None, 1)                 3
    =================================================================
    Total params: 37
    Trainable params: 37
    Non-trainable params: 0
    _________________________________________________________________

```python
model.fit(x_train_bin,
          y_train_nocon,
          batch_size=128,
          epochs=20,
          verbose=2,
          validation_data=(x_test_bin, y_test))

fair_nn_results = model.evaluate(x_test_bin, y_test)

```

    Epoch 1/20
    81/81 - 0s - loss: 0.6853 - accuracy: 0.5249 - val_loss: 0.6839 - val_accuracy: 0.4868
    Epoch 2/20
    81/81 - 0s - loss: 0.6661 - accuracy: 0.5249 - val_loss: 0.6604 - val_accuracy: 0.4868
    Epoch 3/20
    81/81 - 0s - loss: 0.6304 - accuracy: 0.5249 - val_loss: 0.6162 - val_accuracy: 0.4868
    Epoch 4/20
    81/81 - 0s - loss: 0.5754 - accuracy: 0.5249 - val_loss: 0.5584 - val_accuracy: 0.4868
    Epoch 5/20
    81/81 - 0s - loss: 0.5219 - accuracy: 0.5249 - val_loss: 0.5126 - val_accuracy: 0.4868
    Epoch 6/20
    81/81 - 0s - loss: 0.4811 - accuracy: 0.5249 - val_loss: 0.4769 - val_accuracy: 0.4868
    Epoch 7/20
    81/81 - 0s - loss: 0.4491 - accuracy: 0.5249 - val_loss: 0.4469 - val_accuracy: 0.4868
    Epoch 8/20
    81/81 - 0s - loss: 0.4225 - accuracy: 0.7050 - val_loss: 0.4208 - val_accuracy: 0.7937
    Epoch 9/20
    81/81 - 0s - loss: 0.3995 - accuracy: 0.8266 - val_loss: 0.3975 - val_accuracy: 0.8049
    Epoch 10/20
    81/81 - 0s - loss: 0.3797 - accuracy: 0.8323 - val_loss: 0.3780 - val_accuracy: 0.8079
    Epoch 11/20
    81/81 - 0s - loss: 0.3629 - accuracy: 0.8373 - val_loss: 0.3606 - val_accuracy: 0.8095
    Epoch 12/20
    81/81 - 0s - loss: 0.3483 - accuracy: 0.8421 - val_loss: 0.3457 - val_accuracy: 0.8120
    Epoch 13/20
    81/81 - 0s - loss: 0.3356 - accuracy: 0.8436 - val_loss: 0.3326 - val_accuracy: 0.8216
    Epoch 14/20
    81/81 - 0s - loss: 0.3245 - accuracy: 0.8492 - val_loss: 0.3214 - val_accuracy: 0.8222
    Epoch 15/20
    81/81 - 0s - loss: 0.3148 - accuracy: 0.8493 - val_loss: 0.3114 - val_accuracy: 0.8222
    Epoch 16/20
    81/81 - 0s - loss: 0.3063 - accuracy: 0.8524 - val_loss: 0.3026 - val_accuracy: 0.8293
    Epoch 17/20
    81/81 - 0s - loss: 0.2989 - accuracy: 0.8535 - val_loss: 0.2947 - val_accuracy: 0.8293
    Epoch 18/20
    81/81 - 0s - loss: 0.2923 - accuracy: 0.8535 - val_loss: 0.2880 - val_accuracy: 0.8293
    Epoch 19/20
    81/81 - 0s - loss: 0.2867 - accuracy: 0.8541 - val_loss: 0.2820 - val_accuracy: 0.8293
    Epoch 20/20
    81/81 - 0s - loss: 0.2815 - accuracy: 0.8554 - val_loss: 0.2764 - val_accuracy: 0.8684
    62/62 [==============================] - 0s 388us/step - loss: 0.2764 - accuracy: 0.8684

```python
# Comparison
qnn_accuracy = qnn_results[1]
cnn_accuracy = cnn_results[1]
fair_nn_accuracy = fair_nn_results[1]

sns.barplot(["Quantum", "Classical, full", "Classical, fair"],
            [qnn_accuracy, cnn_accuracy, fair_nn_accuracy])
```


### Formation d'Ingénieur en Génie Mécanique
#### Mécanique des fluides (code UE MQ07 niveau L3)
---

* Rapide tutoriel sur les [Notebook Jupyter](../../Tuto-Perso-Jupyter/Tuto-Jupyter.md)

---

## Vanne radiale

Une vanne radiale est fermée , elle est constituée d'une portion de cylindre d'axe $Oy$ de rayon $R$=2m, de
longueur $L$=5m et d'angle de 40° sur laquelle à gauche s'exerce la pression de l'eau et à droite la pression
atmosphérique. La vanne a une masse $M$, $G$ est son centre de gravité et la vanne est articulée autour d'un axe
horizontal passant par le point $O$ le haut de la vanne en contact avec l'eau est à la profondeur $h$=2m le bas de la vanne reposant sur le sol est à $2 + 2*\sin(40)= 3,28m$ de profondeur un point $M$ sur la surface cylindrique est repéré par l'angle $\theta$ mesuré à partir du haut de la vanne

1. Exprimer la pression absolue exercée par l'eau sur la vanne en $M$ à partir de l'équation fondamentale de l'hydrostatique, puis la pression relative
2. Représenter sur un schéma l'effort $\mathrm{d}F$ exercé en $M$ par la pression sur un élément de surface $\mathrm{d}S$
3. Calculer la résultante de la pression relative de l'eau sur la vanne
4. En isolant la vanne, déterminer la masse minimale de la vanne nécessaire pour maintenir le contact de la vanne avec le sol Mθzx

### 1) Pression exercée par l'eau sur la vanne :

* En appliquant la loi de l'hydrostatique entre le point $O$ et le point $M$, avec $\vec{z}$ dirigé vers le haut en prenant l'origine à la surface de l'eau :

$$P(O) + \rho g z_o = P(M) + \rho g z_M\Longrightarrow P_{atm} = p(z_M) + \rho g z_M\Longrightarrow p(z_M) = P_{atm} - \rho g z_M$$

Ici $z_M = -\left(h+R\sin(\theta)\right)$, donc :

$$P(M) = p(z_M) = P_{atm} + \rho g\left(h + R\sin(\theta)\right)$$

* En tenant compte de la pression atmosphérique s'exerçant sur l'eau et à l'extérieur de la vanne, la pression relative en $M$ (ou différence de pression) $\Delta P_M$ s'écrit :

$$\Delta P_M = P_M - P_{atm} = \rho g\left(h + R\sin(\theta)\right)$$

### 2) Effort $\mathrm{d}F$ exercé en $M$ par la pression sur un élément de surface $\mathrm{d}S$:

* L'élément de surface $\mathrm{d}S$ de la face cylindrique de la vanne radiale en contact avec l'eau s'écrit (l'axe $\vec{y}$ étant orienté suivant l'axe du cylindre, donc pependiculaire à la figure):

$$\mathrm{d}S = R\mathrm{d}\theta \mathrm{d}y$$
Comme dans ce problème la pression est constante suivante l'axe $\vec{y}$ on peut écrire que , avec $L$ largeur de la vanne :

$$\mathrm{d}S = R\mathrm{d}\theta L$$
Et l'effort élémentaire dû à la pression relative (ou différence de pression en $M$) s'écrit :

$$\mathrm{d}\vec{F}=-\Delta P_M \mathrm{d}S\vec{n} = -\rho g\left(h + R\sin(\theta)\right)R\mathrm{d}\theta L\vec{n}$$
Ici $\vec{n}$ est la normale "sortante" à l'élément de surface $\mathrm{d}S$ (i.e orientée vers l'extérieur de $\mathrm{d}S$, vers l'eau), vecteur unitaire tel que :

$$\vec{n}=\frac{\vec{OM}}{\Vert\vec{OM}\Vert}$$

Donc dans le repère de la figure :

$$\vec{n}=-\cos(\theta)\vec{x} -\sin(\theta)\vec{z}$$
et
$$\mathrm{d}\vec{F} = \rho g\left(h + R\sin(\theta)\right)R\mathrm{d}\theta L\left(\cos(\theta)\vec{x} + \sin(\theta)\vec{z}\right)$$

### 3)  Résultante des forces de pression sur la vanne :

* Le torseur de l'action mécanique de la pression de l'eau sur la vanne, s'écrit :

$$
\left\lbrace\mathcal{T}_{\text{Eau}\to\text{Vanne}}\right\rbrace:
\left\lbrace
\begin{array}{c}
\vec{\mathcal{R}}_{\text{Eau}\to\text{Vanne}}\\
\vec{\mathcal{M}}_{O_{\text{Eau}\to\text{Vanne}}}
\end{array}
\right\rbrace_{O}=
\left\lbrace
\begin{array}{c}
\iint\limits_S\mathrm{d}\vec{F}\\
\iint\limits_S\mathrm{d}\vec{OM}\wedge\mathrm{d}\vec{F}
\end{array}
\right\rbrace_{O}
$$

On remarque le fait d'écrire les composantes du torseur au point $O$, nous permet de conclure que le moment résultant des forces de pression sur la vanne $\vec{\mathcal{M}}_{O\text{Eau}\to\text{Vanne}}$ est null car $\forall M\in S$, $\vec{OM}\wedge\mathrm{d}\vec{F}=\vec{O}$, puisque $\vec{OM}$ et $\mathrm{d}\vec{F}$ sont colinéaires.

 * Pour calculer la résultante, il suffit de calculer :

$$\vec{\mathcal{R}}_{\text{Eau}\to\text{Vanne}} = \iint\limits_S\mathrm{d}\vec{F} = \int\limits_0^{2\theta_0}\rho g\left(h + R\sin(\theta)\right)R L\left(\cos(\theta)\vec{x} + \sin(\theta)\vec{z}\right)\mathrm{d}\theta$$

où $\theta_0=20°$.

 * On obtient les composantes de la résultante $\vec{\mathcal{R}}_{\text{Eau}\to\text{Vanne}}$ :
     * Suivant $\vec{x}$ :

     $$R_x=\frac{\rho g LR}{2}\left[ R + \left( 2h\sin(2\theta_0) - R\cos^2(2\theta_0)\right)\right]$$

     * Suivant $\vec{z}$ :

     $$R_z=\rho g LR\left[ 1 + \left( R - \frac{R\sin(2\theta_0)\cos(2\theta_0)}{2} - h\cos(2\theta_0)\right)\right]$$

### 4) Equilibre de la vanne et masse minimale.

* Si on isole la vanne, le bilan des actions mécaniques extérieures à la vanne fait apparaître :
    * Le torseur des actions de contact de l'articulation en $O$ (liaison pivot) :

    $$\left\lbrace\mathcal{T}_{\text{Pivot}\to\text{Vanne}}\right\rbrace:
    \left\lbrace
    \begin{array}{c}
    \vec{\mathcal{R}}_{\text{Pivot}\to\text{Vanne}}=X_O\vec{x}+Z-O\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{O}$$

    * Le torseur des actions de contact du sol sur la vanne agissant en $H$ :

    $$\left\lbrace\mathcal{T}_{\text{Sol}\to\text{Vanne}}\right\rbrace:
    \left\lbrace
    \begin{array}{c}
    \vec{\mathcal{R}}_{\text{Sol}\to\text{Vanne}}=Y_H\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{H}$$

    * Le torseur de l'action mécanique de la pression de l'eau en :

    $$\left\lbrace\mathcal{T}_{\text{Eau}\to\text{Vanne}}\right\rbrace:
    \left\lbrace
    \begin{array}{c}
    \vec{\mathcal{R}}_{\text{Eau}\to\text{Vanne}}=R_x\vec{x}+R_z\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{O}$$

    * Le torseur du poids de la vanne en $G$ :

    $$\left\lbrace\mathcal{T}_{\text{Poids}\to\text{Vanne}}\right\rbrace:
    \left\lbrace
    \begin{array}{c}
    \vec{\mathcal{P}}=-mg\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{G}$$

    * Le PFS en $O$ s'écrit :

    $$
    \left\lbrace
    \begin{array}{c}
    X_O\vec{x}+Z-O\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{O}+
    \left\lbrace
    \begin{array}{c}
    R_H\vec{z}\\
    \vec{OH}\wedge Y_H\vec{z}
    \end{array}
    \right\rbrace_{H}+
    \left\lbrace
    \begin{array}{c}
    R_x\vec{x}+R_z\vec{z}\\
    \vec{0}
    \end{array}
    \right\rbrace_{O}+
    \left\lbrace
    \begin{array}{c}
    -mg\vec{z}\\
    \vec{OG}\wedge(-mg\vec{z})
    \end{array}
    \right\rbrace_{O}=
    \lbrace\vec{0}\rbrace
    $$

    Avec $\vec{OH}=-R(\cos(2\theta_0)\vec{x} + \sin(2\theta_0)\vec{z})$ et $\vec{OG}=-a(\cos(\theta_0)\vec{x} + \sin(\theta_0)\vec{z})$ ($a=1.5$m), ce PFS donne les 3 équations scalaires :

    $$
    \left\lbrace
    \begin{array}{rl}
    R_x + X_O &=0\\
    R_y + Y_O + Y_H -mg &=0\\
    amg\cos(\theta_0)  - RY_H\cos(2\theta_0) &= 0
    \end{array}
    \right.$$

* On tire du PFS :

$$Y_H=\frac{amg\cos(\theta_0)}{R\cos(2\theta_0)} \quad X_O=-R_x \quad Y_O = -R_y + mg\left(1-\frac{ag\cos(\theta_0)}{R\cos(2\theta_0)}\right)$$

On voit que le PFS est vérfié quelque soit la valeur de la masse $m$, donc l'équilibre est possible indépendamment de la masse $m$ de vanne.

----

#### Bonus : calcul symbolique des composantes de la résultante avec la librairie `sympy` du langage `python`.

```python
# On importe la libraire "sympy" -> docs sur https://www.sympy.org
from sympy import *
# On déclare les variables "symboliques" à utiliser :
# Données :
rho,g,teta,teta0,a,R,h,P,L = symbols('rho g theta theta_0 a R h P L')
# Inconnues :
XO,YO,YH = symbols('X_O Y_O Y_H')
```

```python
# On défini les expressions symbolique des composantes de la
# force élémentaire due à la différence de pression sur la vanne
dFx = rho*g*R*L*(h+R*sin(teta))*cos(teta)
dFy = rho*g*R*L*(h+R*sin(teta))*sin(teta)
display(dFx,dFy)
```

$\displaystyle L R g \rho \left(R \sin{\left(\theta \right)} + h\right) \cos{\left(\theta \right)}$

$\displaystyle L R g \rho \left(R \sin{\left(\theta \right)} + h\right) \sin{\left(\theta \right)}$

```python
# Intégration symbolique de dFx et dFy entre 0 et 2\theta_0 et affichage du résultat :
Fx=integrate(dFx,(teta,0,2*teta0))
Fy=integrate(dFy,(teta,0,2*teta0))
display(Fx,Fy)
```

$\displaystyle \frac{L R^{2} g \rho}{2} + L R g \rho \left(- \frac{R \cos^{2}{\left(2 \theta_{0} \right)}}{2} + h \sin{\left(2 \theta_{0} \right)}\right)$

$\displaystyle L R g h \rho + L R g \rho \left(R \theta_{0} \sin^{2}{\left(2 \theta_{0} \right)} + R \theta_{0} \cos^{2}{\left(2 \theta_{0} \right)} - \frac{R \sin{\left(2 \theta_{0} \right)} \cos{\left(2 \theta_{0} \right)}}{2} - h \cos{\left(2 \theta_{0} \right)}\right)$