# PerfForesightConsumerType: Perfect foresight consumption-saving

```python
# Initial imports and notebook setup, click arrow to show
from HARK.ConsumptionSaving.ConsIndShockModel import PerfForesightConsumerType
from HARK.utilities import plotFuncs
from time import clock
import matplotlib.pyplot as plt
import numpy as np
mystr = lambda number : "{:.4f}".format(number)
```

The module $\texttt{HARK.ConsumptionSaving.ConsIndShockModel}$ concerns consumption-saving models with idiosyncratic shocks to (non-capital) income.  All of the models assume CRRA utility with geometric discounting, no bequest motive, and income shocks are fully transitory or fully permanent.

$\texttt{ConsIndShockModel}$ currently includes three models:
1. A very basic "perfect foresight" model with no uncertainty.
2. A model with risk over transitory and permanent income shocks.
3. The model described in (2), with an interest rate for debt that differs from the interest rate for savings.

This notebook provides documentation for the first of these three models.
$\newcommand{\CRRA}{\rho}$
$\newcommand{\DiePrb}{\mathsf{D}}$
$\newcommand{\PermGroFac}{\Gamma}$
$\newcommand{\Rfree}{\mathsf{R}}$
$\newcommand{\DiscFac}{\beta}$

## Statement of perfect foresight consumption-saving model

The $\texttt{PerfForesightConsumerType}$ class the problem of a consumer with Constant Relative Risk Aversion utility
${\CRRA}$
\begin{equation}
U(C) = \frac{C^{1-\CRRA}}{1-\rho},
\end{equation}
has perfect foresight about everything except whether he will die between the end of period $t$ and the beginning of period $t+1$, which occurs with probability $\DiePrb_{t+1}$.  Permanent labor income $P_t$ grows from period $t$ to period $t+1$ by factor $\PermGroFac_{t+1}$.  The consumer faces no artificial borrowing constraint and is able to borrow against the entire future stream of income he will receive.

At the beginning of period $t$, the consumer has an amount of market resources $M_t$ (which includes both market wealth and currrent income) and must choose how much of those resources to consume $C_t$ and how much to retain in a riskless asset $A_t$, which will earn return factor $\Rfree$. The agent's flow of future utility $U(C_{t+n})$ from consumption is geometrically discounted by factor $\DiscFac$ per period. If the consumer dies, he receives zero utility flow for the rest of time.

The agent's problem can be written in Bellman form as:

\begin{eqnarray*}
V_t(M_t,P_t) &=& \max_{C_t}~U(C_t) ~+ \DiscFac (1 - \DiePrb_{t+1}) V_{t+1}(M_{t+1},P_{t+1}), \\
& s.t. & \\
A_t &=& M_t - C_t, \\
M_{t+1} &=& \Rfree A_t + Y_{t+1}, \\
Y_{t+1} &=& P_{t+1}, \\
P_{t+1} &=& \PermGroFac_{t+1} P_t.
\end{eqnarray*}

The consumer's problem is characterized by a coefficient of relative risk aversion $\CRRA$, an intertemporal discount factor $\DiscFac$, an interest factor $\Rfree$, and age-varying sequences of the permanent income growth factor $\PermGroFac_t$ and survival probability $(1 - \DiePrb_t)$.

While it does not reduce the computational complexity of the problem (as permanent income is deterministic, given its initial condition $P_0$), HARK represents this problem with *normalized* variables (represented in lower case), dividing all real variables by permanent income $P_t$ and utility levels by $P_t^{1-\CRRA}$.  The Bellman form of the model thus reduces to:

\begin{eqnarray*}
v_t(m_t) &=& \max_{c_t}~U(c_t) ~+ \DiscFac (1 - \DiePrb_{t+1}) \PermGroFac_{t+1}^{1-\CRRA} v_{t+1}(m_{t+1}), \\
& s.t. & \\
a_t &=& m_t - c_t, \\
m_{t+1} &=& \Rfree/\PermGroFac_{t+1} a_t + 1.
\end{eqnarray*}

## Solution method for PerfForesightConsumerType

Because of the assumptions of CRRA utility, no risk other than mortality, and no artificial borrowing constraint, the problem has a closed form solution.  In fact, the consumption function is perfectly linear, and the value function composed with the inverse utility function is also linear.  The mathematical solution of this model is described in detail in the lecture notes [PerfForesightCRRA](http://econ.jhu.edu/people/ccarroll/public/lecturenotes/consumption/PerfForesightCRRA).

The one period problem for this model is solved by the function $\texttt{solveConsPerfForesight}$, which creates an instance of the class $\texttt{ConsPerfForesightSolver}$. To construct an instance of the class $\texttt{PerfForesightConsumerType}$, several parameters must be passed to its constructor as shown in the table below.

## Example parameter values to construct an instance of PerfForesightConsumerType

| Parameter | Description | Code | Example value | Time-varying? |
| :---: | --- | --- | --- | --- |
| $\DiscFac$ |Intertemporal discount factor  | $\texttt{DiscFac}$ | $0.96$ |  |
| $\CRRA $ |Coefficient of relative risk aversion | $\texttt{CRRA}$ | $2.0$ | |
| $\Rfree$ | Risk free interest factor | $\texttt{Rfree}$ | $1.03$ | |
| $1 - \DiePrb_{t+1}$ |Survival probability | $\texttt{LivPrb}$ | $[0.98]$ | $\surd$ |
|$\PermGroFac_{t+1}$|Permanent income growth factor|$\texttt{PermGroFac}$| $[1.01]$ | $\surd$ |
|$T$| Number of periods in this type's "cycle" |$\texttt{T_cycle}$| $1$ | |
|(none)| Number of times the "cycle" occurs |$\texttt{cycles}$| $0$ | |

Note that the survival probability and income growth factor have time subscripts; likewise, the example values for these parameters are *lists* rather than simply single floats.  This is because those parameters are *time-varying*: their values can depend on which period of the problem the agent is in.  All time-varying parameters *must* be specified as lists, even if the same value occurs in each period for this type.

The last two parameters in the table specify the "nature of time" for this type: the number of (non-terminal) periods in this type's "cycle", and the number of times that the "cycle" occurs.  *Every* subclass of $\texttt{AgentType}$ uses these two code parameters to define the nature of time.  Here, $\texttt{T_cycle}$ has the value $1$, indicating that there is exactly one period in the cycle, while $\texttt{cycles}$ is $0$, indicating that the cycle is repeated in *infinite* number of times-- it is an infinite horizon model, with the same "kind" of period repeated over and over.

In contrast, we could instead specify a life-cycle model by setting $\texttt{T_cycle}$ to $1$, and specifying age-varying sequences of income growth and survival probability.  In all cases, the number of elements in each time-varying parameter should exactly equal $\texttt{T_cycle}$.

The parameter $\texttt{AgentCount}$ specifies how many consumers there are of this *type*-- how many individuals have these exact parameter values and are *ex ante* homogeneous.  This information is not relevant for solving the model, but is needed in order to simulate a population of agents, introducing *ex post* heterogeneity through idiosyncratic shocks.  Of course, simulating a perfect foresight model is quite boring, as there are *no* idiosyncratic shocks other than death!

The cell below defines a dictionary that can be passed to the constructor method for $\texttt{PerfForesightConsumerType}$, with the values from the table here.

```python
PerfForesightDict = {
    # Parameters actually used in the solution method
    "CRRA" : 2.0,                          # Coefficient of relative risk aversion
    "Rfree" : 1.03,                        # Interest factor on assets
    "DiscFac" : 0.96,                      # Default intertemporal discount factor
    "LivPrb" : [0.98],                     # Survival probability
    "PermGroFac" :[1.01],                  # Permanent income growth factor

    # Parameters that characterize the nature of time
    "T_cycle" : 1,                         # Number of periods in the cycle for this agent type
    "cycles" : 0                           # Number of times the cycle occurs (0 --> infinitely repeated)
}
```

## Solving and examining the solution of the perfect foresight model

With the dictionary we have just defined, we can create an instance of $\texttt{PerfForesightConsumerType}$ by passing the dictionary to the class (as if the class were a function).  This instance can then be solved by invoking its $\texttt{solve}$ method.

```python
PFexample = PerfForesightConsumerType(**PerfForesightDict)
PFexample.cycles = 0
PFexample.solve()
```

The $\texttt{solve}$ method fills in the instance's attribute $\texttt{solution}$ as a time-varying list of solutions to each period of the consumer's problem.  In this case, $\texttt{solution}$ will be a list with exactly one instance of the class $\texttt{ConsumerSolution}$, representing the solution to the infinite horizon model we specified.

```python
print(PFexample.solution)
```

Each element of $\texttt{solution}$ has a few attributes. To see all of them, we can use the \texttt{vars} built in function:

the consumption functions reside in the attribute $\texttt{cFunc}$ of each element of $\texttt{ConsumerType.solution}$.  This method creates a (time varying) attribute $\texttt{cFunc}$ that contains a list of consumption functions.

```python
print(vars(PFexample.solution[0]))
```

The two most important attributes of a single period solution of this model are the (normalized) consumption function $\texttt{cFunc}$ and the (normalized) value function $\texttt{vFunc}$.  Let's plot those functions near the lower bound of the permissible state space (the attribute $\texttt{mNrmMin}$ tells us the lower bound of $m_t$ where the consumption function is defined).

```python
print('Linear perfect foresight consumption function:')
mMin = PFexample.solution[0].mNrmMin
plotFuncs(PFexample.solution[0].cFunc,mMin,mMin+10.)
```

```python
print('Perfect foresight value function:')
plotFuncs(PFexample.solution[0].vFunc,mMin+0.1,mMin+10.1)
```

An element of $\texttt{solution}$ also includes the (normalized) marginal value function $\texttt{vPfunc}$, and the lower and upper bounds of the marginal propensity to consume (MPC) $\texttt{MPCmin}$ and $\texttt{MPCmax}$.  Note that with a linear consumption function, the MPC is constant, so its lower and upper bound are identical.

## Simulating the perfect foresight consumer model

Suppose we wanted to simulate many consumers who share the parameter values that we passed to $\texttt{PerfForesightConsumerType}$-- an *ex ante* homogeneous *type* of consumers.  To do this, our instance would have to know *how many* agents there are of this type, as well as their initial levels of assets $a_t$ and permanent income $P_t$.

### Setting simulation parameters

Let's fill in this information by passing another dictionary to $\texttt{PFexample}$ with simulation parameters.  The table below lists the parameters that an instance of $\texttt{PerfForesightConsumerType}$ needs in order to successfully simulate its model using the $\texttt{simulate}$ method.

| Description | Code | Example value |
| :---: | --- | --- |
| Number of consumers of this type | $\texttt{AgentCount}$ | $10000$ |
| Number of periods to simulate | $\texttt{T_sim}$ | $120$ |
| Mean of initial log (normalized) assets | $\texttt{aNrmInitMean}$ | $-6.0$ |
| Stdev of initial log  (normalized) assets | $\texttt{aNrmInitStd}$ | $1.0$ |
| Mean of initial log permanent income | $\texttt{pLvlInitMean}$ | $0.0$ |
| Stdev of initial log permanent income | $\texttt{pLvlInitStd}$ | $0.0$ |
| Aggregrate productivity growth factor | $\texttt{PermGroFacAgg}$ | $1.0$ |
| Age after which consumers are automatically killed | $\texttt{T_age}$ | $None$ |

We have specified the model so that initial assets and permanent income are both distributed lognormally, with mean and standard deviation of the underlying normal distributions provided by the user.

The parameter $\texttt{PermGroFacAgg}$ exists for compatibility with more advanced models that employ aggregate productivity shocks; it can simply be set to 1.

In infinite horizon models, it might be useful to prevent agents from living extraordinarily long lives through a fortuitous sequence of mortality shocks.  We have thus provided the option of setting $\texttt{T_age}$ to specify the maximum number of periods that a consumer can live before they are automatically killed (and replaced with a new consumer with initial state drawn from the specified distributions).  This can be turned off by setting it to $\texttt{None}$.

The cell below puts these parameters into a dictionary, then gives them to $\texttt{PFexample}$.  Note that all of these parameters *could* have been passed as part of the original dictionary; we omitted them above for simplicity.

```python
SimulationParams = {
    "AgentCount" : 10000,                  # Number of agents of this type
    "T_sim" : 120,                         # Number of periods to simulate
    "aNrmInitMean" : -6.0,                 # Mean of log initial assets
    "aNrmInitStd"  : 1.0,                  # Standard deviation of log initial assets
    "pLvlInitMean" : 0.0,                  # Mean of log initial permanent income
    "pLvlInitStd"  : 0.0,                  # Standard deviation of log initial permanent income
    "PermGroFacAgg" : 1.0,                 # Aggregate permanent income growth factor
    "T_age" : None,                        # Age after which simulated agents are automatically killed
}

PFexample(**SimulationParams) # This implicitly uses the assignParameters method of AgentType
```

To generate simulated data, we need to specify which variables we want to track the "history" of for this instance.  To do so, we set the $\texttt{track_vars}$ attribute of our $\texttt{PerfForesightConsumerType}$ instance to be a list of strings with the simulation variables we want to track.

In this model, valid elments of $\texttt{track_vars}$ include $\texttt{mNrmNow}$, $\texttt{cNrmNow}$, $\texttt{aNrmNow}$, and $\texttt{pLvlNow}$.  Because this model has no idiosyncratic shocks, our simulated data will be quite boring.

### Generating simulated data

Before simulating, the $\texttt{initializeSim}$ method must be invoked.  This resets our instance back to its initial state, drawing a set of initial $\texttt{aNrmNow}$ and $\texttt{pLvlNow}$ values from the specified distributions and storing them in the attributes $\texttt{aNrmNow_init}$ and $\texttt{pLvlNow_init}$.  It also resets this instance's internal random number generator, so that the same initial states will be set every time $\texttt{initializeSim}$ is called.  In models with non-trivial shocks, this also ensures that the same sequence of shocks will be generated on every simulation run.

Finally, the $\texttt{simulate}$ method can be called.

```python
PFexample.track_vars = ['mNrmNow']
PFexample.initializeSim()
PFexample.simulate()
```

Each simulation variable $\texttt{X}$ named in $\texttt{track_vars}$ will have the *history* of that variable for each agent stored in the attribute $\texttt{X_hist}$ as an array of shape $(\texttt{T_sim},\texttt{AgentCount})$.  To see that the simulation worked as intended, we can plot the mean of $m_t$ in each simulated period:

```python
plt.plot(np.mean(PFexample.mNrmNow_hist,axis=1))
plt.xlabel('Time')
plt.ylabel('Mean normalized market resources')
plt.show()
```

A perfect foresight consumer can borrow against the PDV of his future income-- his human wealth-- and thus as time goes on, our simulated agents approach the (very negative) steady state level of $m_t$ while being steadily replaced with consumers with roughly $m_t=1$.

The slight wiggles in the plotted curve are due to consumers randomly dying and being replaced; their replacement will have an initial state drawn from the distributions specified by the user.  To see the current distribution of ages, we can look at the attribute $\texttt{t_age}$.

```python
N = PFexample.AgentCount
F = np.linspace(0.,1.,N)
plt.plot(np.sort(PFexample.t_age),F)
plt.xlabel('Current age of consumers')
plt.ylabel('Cumulative distribution')
plt.show()
```

The distribution is (discretely) exponential, with a point mass at 120 with consumers who have survived since the beginning of the simulation.

One might wonder why HARK requires users to call $\texttt{initializeSim}$ before calling $\texttt{simulate}$: Why doesn't $\texttt{simulate}$ just call $\texttt{initializeSim}$ as its first step?  We have broken up these two steps so that users can simulate some number of periods, change something in the environment, and then resume the simulation.

When called with no argument, $\texttt{simulate}$ will simulate the model for $\texttt{T_sim}$ periods.  The user can optionally pass an integer specifying the number of periods to simulate (which should not exceed $\texttt{T_sim}$).

In the cell below, we simulate our perfect foresight consumers for 80 periods, then seize a bunch of their assets (dragging their wealth even more negative), then simulate for the reamining 40 periods.

```python
PFexample.initializeSim()
PFexample.simulate(80)
PFexample.aNrmNow += -5. # Adjust all simulated consumers' assets downward by 5
PFexample.simulate(40)

plt.plot(np.mean(PFexample.mNrmNow_hist,axis=1))
plt.xlabel('Time')
plt.ylabel('Mean normalized market resources')
plt.show()
```

```python

```


```python
import numpy as np
import numba
import matplotlib.pyplot as plt
import sympy as sym
plt.style.use('presentation.mplstyle')
%matplotlib notebook

def d2np(d):

    names = []
    numbers = ()
    dtypes = []
    for item in d:
        names += item
        if type(d[item]) == float:
            numbers += (d[item],)
            dtypes += [(item,float)]
        if type(d[item]) == int:
            numbers += (d[item],)
            dtypes += [(item,int)]
        if type(d[item]) == np.ndarray:
            numbers += (d[item],)
            dtypes += [(item,np.float64,d[item].shape)]
    return np.array([numbers],dtype=dtypes)
```

```python
psi_ds,psi_qs,psi_dr,psi_qr = sym.symbols('psi_ds,psi_qs,psi_dr,psi_qr')
i_ds,i_qs,i_dr,i_qr = sym.symbols('i_ds,i_qs,i_dr,i_qr')
L_s,L_r,L_m = sym.symbols('L_s,L_r,L_m')

eq_ds = (L_s+L_m)*i_ds + L_m*i_dr - psi_ds
eq_qs = (L_s+L_m)*i_qs + L_m*i_qr - psi_qs
eq_dr = (L_r+L_m)*i_dr + L_m*i_ds - psi_dr
eq_qr = (L_r+L_m)*i_qr + L_m*i_qs - psi_qr

s = sym.solve([eq_ds,eq_qs,eq_dr,eq_qr],[i_ds,i_qs,i_dr,i_qr])

for item in s:
    print(item, '=', s[item])
```

    i_ds = (-L_m*psi_dr + L_m*psi_ds + L_r*psi_ds)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_qs = (-L_m*psi_qr + L_m*psi_qs + L_r*psi_qs)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_dr = (L_m*psi_dr - L_m*psi_ds + L_s*psi_dr)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_qr = (L_m*psi_qr - L_m*psi_qs + L_s*psi_qr)/(L_m*L_r + L_m*L_s + L_r*L_s)

```python
# [1] T. Demiray, F. Milano, and G. Andersson,
# “Dynamic phasor modeling of the doubly-fed induction generator under unbalanced conditions,” 2007 IEEE Lausanne POWERTECH, Proc., no. 2, pp. 1049–1054, 2007.

@numba.jit(nopython=True, cache=True)
def dfim(struct,i):

    x_idx = struct[i]['dfim_idx']

    L_m = struct[i]['L_m']
    L_r = struct[i]['L_r']
    L_s = struct[i]['L_s']
    R_r = struct[i]['R_r']
    R_s = struct[i]['R_s']

    Dt  = struct[i]['Dt']

    psi_ds = struct[i]['psi_ds']
    psi_qs = struct[i]['psi_qs']

    psi_dr = float(struct[i]['x'][x_idx+0,0])
    psi_qr = float(struct[i]['x'][x_idx+1,0])

    v_ds = struct[i]['v_ds']
    v_qs = struct[i]['v_qs']
    v_dr = struct[i]['v_dr']
    v_qr = struct[i]['v_qr']

    omega_r = struct[i]['omega_r']
    omega_s = struct[i]['omega_s']

    sigma = (omega_s - omega_r)/omega_s

    i_qr = (L_m*psi_qr - L_m*psi_qs + L_s*psi_qr)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_qs = (-L_m*psi_qr + L_m*psi_qs + L_r*psi_qs)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_dr = (L_m*psi_dr - L_m*psi_ds + L_s*psi_dr)/(L_m*L_r + L_m*L_s + L_r*L_s)
    i_ds = (-L_m*psi_dr + L_m*psi_ds + L_r*psi_ds)/(L_m*L_r + L_m*L_s + L_r*L_s)

    psi_qs = (R_s*i_ds - v_ds)/omega_s
    psi_ds = (v_qs - R_s*i_qs)/omega_s

    dpsi_dr = v_dr - R_r*i_dr + sigma*omega_s*psi_qr
    dpsi_qr = v_qr - R_r*i_qr - sigma*omega_s*psi_dr

    tau_e  = psi_qr*i_dr - psi_dr*i_qr

    struct[i]['f'][x_idx+0,0] = float(dpsi_dr)
    struct[i]['f'][x_idx+1,0] = float(dpsi_qr)

    struct[i]['i_ds'] = i_ds
    struct[i]['i_qs'] = i_qs
    struct[i]['i_dr'] = i_dr
    struct[i]['i_qr'] = i_qr

    struct[i]['psi_ds'] = psi_ds
    struct[i]['psi_qs'] = psi_qs
    struct[i]['psi_dr'] = psi_dr
    struct[i]['psi_qr'] = psi_qr

    struct[i]['tau_e'] = tau_e
    struct[i]['sigma'] = sigma

    struct[i]['p_s'] = 3.0/2.0*(v_ds*i_ds + v_qs*i_qs)
    struct[i]['q_s'] = 3.0/2.0*(v_ds*i_qs - v_qs*i_ds)

    return tau_e

@numba.jit(nopython=True, cache=True)
def wecs_mech_1(struct,i):

    x_idx = struct[i]['mech_idx']
    omega_t = struct[i]['x'][x_idx,0]  # rad/s
    tau_t   = struct[i]['tau_t']
    tau_r   = struct[i]['tau_r']

    J_t  = struct[i]['J_t']
    N_tr = struct[i]['N_tr']
    Dt   = struct[i]['Dt']

    domega_t = 1.0/J_t*(tau_t - N_tr*tau_r)

    omega_r = N_tr*omega_t

    struct[i]['f'][x_idx,0] = domega_t

    struct[i]['omega_r'] = omega_r
    struct[i]['omega_t'] = omega_t

    return omega_t

```

```python
@numba.jit(nopython=True, cache=True)
def dfig_ctrl_r_1(struct,i):

    x_idx = struct[i]['ctrl_r']
    xi_ixr = struct[i]['x'][x_idx+0,0]  # rad/s
    xi_iyr = struct[i]['x'][x_idx+1,0]  # rad/s

    L_m = struct[i]['L_m']
    L_s = struct[i]['L_s']

    i_ds = struct[i]['i_ds']
    i_qs = struct[i]['i_qs']
    i_dr = struct[i]['i_dr']
    i_qr = struct[i]['i_qr']
    i_xr_ref = struct[i]['i_xr_ref']
    i_yr_ref = struct[i]['i_yr_ref']

    i_dm = L_s/L_m*i_ds + i_dr
    i_qm = L_s/L_m*i_qs + i_qr

    theta_s = np.arctan2(i_qm,i_dm)

    struct[i]['theta_s'] = theta_s

    i_xr = i_dr*np.cos(theta_s) + i_qr*np.sin(theta_s)
    i_yr = i_qr*np.cos(theta_s) - i_dr*np.sin(theta_s)

    error_xr = i_xr_ref - i_xr
    error_yr = i_yr_ref - i_yr

    v_xr = 0.1*error_xr + 5.0*xi_ixr
    v_yr = 0.1*error_yr + 5.0*xi_iyr

    dxi_ixr = error_xr
    dxi_iyr = error_yr

    v_dr = v_xr*np.cos(theta_s) - v_yr*np.sin(theta_s)
    v_qr = v_xr*np.sin(theta_s) + v_yr*np.cos(theta_s)

    struct[i]['i_xr'] = i_xr
    struct[i]['i_yr'] = i_yr
    struct[i]['v_dr'] = v_dr
    struct[i]['v_qr'] = v_qr

    struct[i]['f'][x_idx+0,0] = dxi_ixr
    struct[i]['f'][x_idx+1,0] = dxi_iyr
    return theta_s
```

```python
Omega_b = 2.0*np.pi*50.0
S_b = 1.0e6
U_b = 690.0
Z_b = U_b**2/S_b
#nu_w =np.linspace(0.1,15,N)
H = 2.0

# H = 0.5*J*Omega_t_n**2/S_b
S_b = 2.0e6
Omega_t_n = 1.5
J_t = 2*H*S_b/Omega_t_n**2

#Z_b = 1.0
#Omega_b = 1.0
d =dict(R_r = 0.01*Z_b,
        R_s = 0.01*Z_b,
        L_r = 0.08*Z_b/Omega_b,
        L_s = 0.1*Z_b/Omega_b,
        L_m = 3.0*Z_b/Omega_b,
        psi_ds = 0.0,
        psi_qs = 0.0,
        p_s = 0.0,
        q_s = 0.0,
        psi_dr = 0.0,
        psi_qr = 0.0,
        i_ds   = 0.0,
        i_qs   = 0.0,
        i_dr   = 0.0,
        i_qr   = 0.0,
        v_ds   = 0.0,
        v_qs   = 0.0,
        v_dr   = 0.0,
        v_qr   = 0.0,
        omega_r   = Omega_b*0.99,
        omega_s   = Omega_b,
        sigma   = 0.0,
        tau_e   = 0.0,
        x = np.zeros((5,1)),
        f = np.zeros((5,1)),
        Dt = 0.0,
        J_t = J_t,
        omega_t = 0.0,
        tau_t = 0.0,
        tau_r = 0.0,
        N_tr = 20.0,
        dfim_idx = 0,
        mech_idx = 2,
        ctrl_r = 3,
        theta_s = 0.0,
        i_xr = 0.0,
        i_yr = 0.0,
        i_xr_ref = 0.0,
        i_yr_ref = 0.0,
        xi_ixr = 0.0,
        xi_iyr = 0.0,
        )

struct = d2np(d)
dfim(struct,0)
wecs_mech_1(struct,0)
dfig_ctrl_r_1(struct,0)

```

    0.0

```python
struct = d2np(d)

@numba.jit(nopython=True, cache=True)
def f_eval(struct):
    dfim(struct,0)
    wecs_mech_1(struct,0)
    dfig_ctrl_r_1(struct,0)
    return struct[0]['f'][:]

#@numba.jit(nopython=True, cache=True)
def run(struct):
    struct[0]['x']= np.zeros((5,1))
    struct[0]['v_ds'] = 500.0
    struct[0]['v_qs'] = 0.0
    struct[0]['tau_t'] = 0.0
    struct[0]['x'][2,0] = Omega_b*0.9/struct[0]['N_tr']

    Dt = 1.0e-3
    N_steps = 10000
    Theta_s = np.zeros((N_steps,1))
    P_s = np.zeros((N_steps,1))
    Q_s = np.zeros((N_steps,1))
    I_xr = np.zeros((N_steps,1))
    I_yr = np.zeros((N_steps,1))
    V_dr = np.zeros((N_steps,1))
    V_qr = np.zeros((N_steps,1))
    Tau_e = np.zeros((N_steps,1))
    T     = np.zeros((N_steps,1))
    X = np.zeros((N_steps,5))
    struct[0]['i_xr_ref'] = 1000.0
    struct[0]['i_yr_ref'] = 0.0
    for it in range(N_steps):
        t = Dt*float(it)

        # perturbations
        if t>1.0:
            struct[0]['tau_t'] = 100e3
        if t>2.0:
            struct[0]['i_xr_ref'] = 2000.0
        if t>5.0:
            struct[0]['i_yr_ref'] = 0.0

        # solver
        f1 = np.copy(f_eval(struct))
        x1 = np.copy(struct[0]['x'][:])
        struct[0]['x'] = np.copy(x1 + Dt*f1)
        struct[0]['x'] = np.copy(x1 + 0.5*Dt*(f1 + f_eval(struct)))

        # interconections
        struct[0]['tau_r'] =  struct[0]['tau_e']

        # outputs
        Tau_e[it,0] = struct[0]['tau_e']
        X[it,:] = np.copy(struct[0]['x'][:]).T
        T[it,0] = float(t)
        P_s[it,0] = struct[0]['p_s']
        Q_s[it,0] = struct[0]['q_s']
        I_xr[it,0] = struct[0]['i_xr']
        I_yr[it,0] = struct[0]['i_yr']
        V_dr[it,0] = struct[0]['v_dr']
        V_qr[it,0] = struct[0]['v_qr']
        Theta_s[it,0] = struct[0]['theta_s']
    return T,X,Tau_e,P_s,Q_s,Theta_s,I_xr,I_yr,V_dr,V_qr

```

```python
T,X,Tau_e,P_s,Q_s,Theta_s,I_xr,I_yr,V_dr,V_qr = run(struct)
```

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
axes.plot(T,V_dr)
axes.plot(T,V_qr)
fig.savefig('dfim_tau_e.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

    /home/jmmauricio/bin/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:1743: UserWarning: This figure includes Axes that are not compatible with tight_layout, so its results might be incorrect.
      warnings.warn("This figure includes Axes that are not "

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
axes.plot(T,I_xr)
axes.plot(T,I_yr)
fig.savefig('dfim_tau_e.svg', bbox_inches='tight')
```

    <IPython.core.display.Javascript object>

    /home/jmmauricio/bin/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:1743: UserWarning: This figure includes Axes that are not compatible with tight_layout, so its results might be incorrect.
      warnings.warn("This figure includes Axes that are not "

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
axes.plot(T,Tau_e)
fig.savefig('dfim_tau_e.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

    /home/jmmauricio/bin/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:1743: UserWarning: This figure includes Axes that are not compatible with tight_layout, so its results might be incorrect.
      warnings.warn("This figure includes Axes that are not "

```python
fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(8, 8), sharex = True)
axes[0].plot(T,P_s/1e6, label='$\sf p_s$')
axes[0].plot(T,Q_s/1e6, label='$\sf q_s$')
axes[0].legend()
fig.savefig('dfim_tau_e.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

    /home/jmmauricio/bin/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py:1743: UserWarning: This figure includes Axes that are not compatible with tight_layout, so its results might be incorrect.
      warnings.warn("This figure includes Axes that are not "

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
axes.plot(T,X[:,2]*20)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7effcc9c6400>]

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
axes.plot(T,Theta_s)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7effccbb3da0>]

```python

```

```python

```

```python

```

```python

```

```python

```


# Cálculo e clasificación de puntos críticos

Con todas as ferramentas que xa levamos revisado nas anteriores prácticas, o cálculo de puntos críticos e a súa clasificación mediante o criterio que involucra á matriz Hessiana de funcións de dúas variables diferenciables é moi sinxelo usando o módulo **Sympy**. No caso de puntos críticos basta calcular o gradiente da función e resolver un sistema de dúas ecuacións (habitualmente non lineal) e para a clasificación dos puntos críticos se deben inspeccionar os autovalores da matriz Hessiana (cálculo que tamén está dispoñible en **Sympy**).

Como aplicación do cálculo e identificación de máximos e mínimos relativos, revisaremos como interpretar como un problema de optimización o axuste polinomial mediante mínimos cadrados dun conxunto de puntos unidimensional.

### Obxectivos:

- Cálculo de puntos críticos
- Clasificación de puntos críticos: matriz Hessiana
- Problema de optimización: axuste polinomial mediante mínimos cadrados

## Cálculo de puntos críticos
Nesta práctica usaremos tanto o módulo **Sympy**, como tamén **Numpy** e **Matplotlib**. Así que debemos importalos para o resto do guión de prácticas:

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

Como xa aconteceu en prácticas anteriores, debemos facer unha implementación propia para calcular o gradiente dunha función escalar $f$. Para iso usaremos a relación que xa coñecemos entre a matriz Xacobiana $Df$ dunha función escalar e o vector (columna) gradiente $\nabla f$, isto é $\nabla f=Df^{t}$:

```python
gradient = lambda f, v: sp.transpose(sp.Matrix([f]).jacobian(v))
```

Como xa estudamos nas sesións expositivas, que se supoñemos que a función de dúas variables $f$ é diferenciable, o cálculo de puntos críticos realízase tendo en conta que o plano tanxente á superficie que define a función son planos horizontais nos extremos relativos da función, é dicir, naqueles puntos onde as derivadas parciais de $f$ son nulas. Vexamos isto cun exemplo no que $f(x,y)=-x^3 +4xy-2y^2+1$:

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x e y
f = sp.Lambda((x,y), -x**3 +4*x*y-2*y**2+1)

# Cálculo de puntos críticos
grad_f = gradient(f(x,y),(x,y))
sol = sp.solve((sp.Eq(grad_f[0],0),sp.Eq(grad_f[1],0)),(x,y))
display('Critical points for x and y:', sol)
```

    'Critical points for x and y:'

    [(0, 0), (4/3, 4/3)]

Para comprobar visualmente o tipo de puntos críticos que posúe esta función, a podemos representar gráficamente:

```python
p = sp.plotting.plot3d(f(x,y), (x, -2, 2), (y, -2, 2), show=False)
p.xlabel='x'
p.ylabel='y'
p.zlabel='z'
p.show()
```

### **Exercicio 8.1**
Calcula os puntos críticos e representa graficamente a función:
$$
f(x,y) =  \left(\frac12-x^2+y^2\right)e^{1-x^2-y^2}
$$
na rexión $(x,y)\in[-4,4]\times[-4,4]$.

```python
# O TEU CÓDIGO AQUÍ
```

## Clasificación de puntos críticos: matriz Hessiana
O cálculo da matriz Hessiana co módulo **Sympy** é inmediato xa que basta usar o comando `sp.hessian`. Unha vez feito isto, para calcular os autovalores desta matriz e decidir atendendo aos seus valores sen os puntos críticos son máximos relativos, mínimos relativos ou puntos de sela, soamnete debemos empregar o método `eigenvals` que está accesible na clase de obxectos tipo `sp.Matrix`:

```python
H = sp.Lambda((x,y), sp.hessian(f(x,y), (x,y)))
display('Hessian matrix', H(x,y))

# Clasificación do primeiro punto crítico: (0,0)
eigs = H(*sol[0]).eigenvals()
display('Eigenvalues for point (0,0)', np.double([*eigs]))

# Clasificación do segundo punto crítico: (4/3,4/3)
eigs = H(*sol[1]).eigenvals()
display('Eigenvalues for point (4/3,4/3)', np.double([*eigs]))
```

    'Hessian matrix'

$\displaystyle \left[\begin{matrix}- 6 x & 4\\4 & -4\end{matrix}\right]$

    'Eigenvalues for point (0,0)'

    array([-6.47213595,  2.47213595])

    'Eigenvalues for point (4/3,4/3)'

    array([-10.47213595,  -1.52786405])

### **Exercicio 8.2**
Clasifica os puntos críticos obtidos no exercicio 8.1, que correspondían á función:
$$
f(x,y) =  \left(\frac12-x^2+y^2\right)e^{1-x^2-y^2}
$$
na rexión $(x,y)\in[-4,4]\times[-4,4]$.

```python
# O TEU CÓDIGO AQUÍ
```

## Axuste polinomial mediante mínimos cadrados
Dado un conxunto de puntos nun plano $(x_1,y_1), (x_2,y_2),\ldots, (x_{m},y_{m})$, é habitual que se trate atopar cal é o mellor polinomio de grao $N$ que minimiza o erro cadrático medio entre os datos proporcionados e os valores do polinomio axustado. No caso, dunh polinomio de grado $1$, o polinomio se escribe como $p(x)=ax+b$ e o anterior problema redúcese a atopar $(a^*,b^*)$ tal que se minimiza a función error:
$$
\mathrm{error}(a^*,b^*)=\min_{(a,b)\in\mathbb{R}^{2}}\mathrm{error}(a,b)
$$
onde
$$
\mathrm{error}(a,b)=\sum_{i=1}^{m}(ax_i+b-y_i)^2.
$$
Como calquera outro problema de minimización sen restriccións, para a súa resolución se deben calcular os puntos críticos da función erro e despois comprobar que se trata dun mínimo relativo (que será absoluto xa que a función erro tende a infinito cando $a$ ou $b$ tenden a $\pm\infty$). Vexamos este cálculo cun exemplo concreto. En primeiro lugar introducimos os datos:

```python
# Datos
xdata = np.array([0.0, 1.0, 2.0, 3.0,  4.0,  5.0])
ydata = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
```

A continuación, definimos a función error, calculamos os puntos críticos e mediante a matriz Hessiana determinamos que se trata dun mínimo relativo.

```python
# Erro a minimizar no método de mínimos cadrados
a, b = sp.symbols('a b', real=True)
error = sp.Lambda((a,b),sum((a*xi + b - yi)**2 for xi,yi in zip(xdata, ydata)))
display('Error function', error(a,b))

# Cálculo de puntos críticos
grad_error = gradient(error(a,b),(a,b))
sol = sp.solve((sp.Eq(grad_error[0],0),sp.Eq(grad_error[1],0)),(a,b))
display('Critical points for a and b:', sol)

# Clasificación de puntos críticos: autovalores da matriz Hessiana
H = sp.hessian(error(a,b), (a,b))
display('Hessian matrix', H)
eigs = H.eigenvals()
display('Eigenvalues', [*eigs])
display('Eigenvalues', np.double([*eigs]))
```

    'Error function'

$\displaystyle b^{2} + \left(1.0 a + b - 0.8\right)^{2} + \left(2.0 a + b - 0.9\right)^{2} + \left(3.0 a + b - 0.1\right)^{2} + \left(4.0 a + b + 0.8\right)^{2} + \left(5.0 a + b + 1.0\right)^{2}$

    'Critical points for a and b:'

    {a: -0.302857142857143, b: 0.757142857142857}

    'Hessian matrix'

$\displaystyle \left[\begin{matrix}110.0 & 30.0\\30.0 & 12\end{matrix}\right]$

    'Eigenvalues'

    [61 - sqrt(3301), sqrt(3301) + 61]

    'Eigenvalues'

    array([  3.54567031, 118.45432969])

Dado que o proceso de axuste polinomial de datos é unha tarefa moi recurrente, existen métodos numéricos dedicados ao seu cálculo (tanto nunha como en varias dimensións). En particular, o módulo **Numpy** tamén ten un ferramenta directa para facer este axuste co comando `np.polyfit`. Comprobemos que os coeficientes que calcula son os mesmos que se obteñen cos cálculos en **Sympy**:

```python
# Axuste por mínimos cadrados dun polinomio de orde 1
z = np.polyfit(xdata, ydata, 1)
display('Values for a and b', z)
```

    'Values for a and b'

    array([-0.30285714,  0.75714286])

Adicionalmente, o comando `np.polyfit` permite facer o axuste usando un polinomo de calquera orde. No que segue, se calcula e representa graficamente o axuste polinomial de orde $3$:

```python
# Axuste por mínimos cadrados dun polinomio de orde 3
z = np.polyfit(xdata, ydata, 3)

# Define un polinomio en Sympy a partir dos seus coeficientes
x = sp.symbols('x', real=True)
P = sp.Lambda(x,sum((a*x**i for i,a in enumerate(z[::-1]))))

# Representación gráfica
pol = sp.lambdify(x,P(x),"numpy")
plt.plot(xdata, ydata, '.', label='Data')
xp = np.linspace(-1.,6.,100)
plt.plot(xp, pol(xp), '-', label='Fitting')
plt.xlim(-1,6)
plt.ylim(-2,2)
plt.legend()
plt.show()
```

### **Exercicio 8.3**
Sobre os datos usados anteriormente, sábese que no punto $x=4.5$, o valor é $-1.01$. Usa diferente orde de polinomios para axustar os datos, por exemplo $N=3, 5, 10, 20, 30$. Calcula o erro que se comete neste axuste no punto $x=4.5$:
- Ao aumentar a orde polinomial, mellorase o erro?
- Cal é o valor de $N$ co que se comete menor erro no axuste para o punto $x=4.5$?
- Ao introducir o novo dato do punto $x=4.5$ e usar $N<4$: apreciase algunha diferencia na curva axustada?

```python
# O TEU CÓDIGO AQUÍ
```


# Train Model

It is finally time to use the features we have created to train some classifiers to predict Physics Nobel Laureates. We will be training the following classifiers:
- [Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression)
- [Support Vector Machine](https://en.wikipedia.org/wiki/Support_vector_machine)
- [Random Forest](https://en.wikipedia.org/wiki/Random_forest)

These classifiers are chosen due to their simplicity, appropriateness for the input features and target data types, ease of interpretability and their well-established performance on a range of classification tasks.

We will be selecting [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter) by evaluating the performance of the models on the validation set. This will be done in a principled manner using a couple of performance measures. The first of these, the [Matthews Correlation Coefficient (MCC)](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient), you have already seen during the creation of our [baseline model](5.0-baseline-model.ipynb). We will discuss the second measure in detail, and explain why it is needed, when we get to that point. OK let's get going!

```python
import ast
import operator
import warnings

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import matthews_corrcoef
from sklearn.model_selection import ParameterGrid
from sklearn.model_selection import ParameterSampler
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

from src.data.progress_bar import progress_bar
from src.features.features_utils import convert_categoricals_to_numerical
from src.features.features_utils import convert_target_to_numerical
from src.models.metrics_utils import confusion_matrix_to_dataframe
from src.models.metrics_utils import mcc_auc_score
from src.models.metrics_utils import mcc_curve

%matplotlib inline
```

## Reading in the Data

First let's read in both sets of training and validation features and targets. We will also read in the sample weights that we created for covariate shift adaptation. Also let's convert the categorical fields to a numerical form that is suitable for building machine learning models.

```python
train_features = pd.read_csv('../data/processed/train-features.csv')
X_train = convert_categoricals_to_numerical(train_features)
X_train.head()
```

```python
sample_weights = pd.read_csv('../models/train-features-sample-weights.csv')
sample_weights.head()
```

```python
train_features_topics = pd.read_csv('../data/processed/train-features-topics.csv')
X_train_topics = convert_categoricals_to_numerical(train_features_topics)
X_train_topics.head()
```

```python
sample_weights_topics = pd.read_csv('../models/train-features-topics-sample-weights.csv')
sample_weights_topics.head()
```

```python
train_target = pd.read_csv('../data/processed/train-target.csv', index_col='full_name', squeeze=True)
y_train = convert_target_to_numerical(train_target)
y_train.head()
```

```python
validation_features = pd.read_csv('../data/processed/validation-features.csv')
X_validation = convert_categoricals_to_numerical(validation_features)
X_validation.head()
```

```python
validation_features_topics = pd.read_csv('../data/processed/validation-features-topics.csv')
X_validation_topics = convert_categoricals_to_numerical(validation_features_topics)
X_validation_topics.head()
```

```python
validation_target = pd.read_csv('../data/processed/validation-target.csv', index_col='full_name',
                                squeeze=True)
y_validation = convert_target_to_numerical(validation_target)
y_validation.head()
```

## Hyperparameter Selection

The hyperparameters of the models that we will be fitting are critical to their predictive performance. We will use [grid search](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search) to select them in a principled manner. The optimal hyperparameter values will be chosen according to the set of values that maximize the MCC on the validation set. The function below will be used to accomplish this task.

```python
def evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=LogisticRegression(),
    param_grid=ParameterGrid(dict(C=np.logspace(-5, 15, 21, base=2.0))), score_func=matthews_corrcoef,
    greater_score_is_better=True, solver='lbfgs', sample_weight=None, max_iter=1000,
    random_state=None, n_jobs=None, name='classifier', progress_bar=None):

    """Evaluate an `sklearn` classifier over a parameter grid using a scoring function.

    Args:
        X_train ({array-like, sparse matrix}, shape (n_samples, n_features)): Training features matrix,
            where n_samples is the number of samples and n_features is the number of features.
        y_train (array-like, shape (n_samples,)): Training data target vector.
        X_validation ({array-like, sparse matrix}, shape (n_samples, n_features)): Validation features
        matrix, where n_samples is the number of samples and n_features is the number of features.
        y_validation (array-like, shape (n_samples,)): Validation data target vector.
        clf (sklearn.base.BaseEstimator, optional): Defaults to LogisticRegression(). Classifier.
        param_grid (sklearn.model_selection.ParameterGrid, optional): Defaults to
            ParameterGrid(dict(C=np.logspace(-5, 15, 21, base=2.0))). Grid of parameters with a discrete
            number of values for each.
        score_func (callable, optional): Defaults to matthews_corrcoef. Score function (or loss function)
            with signature score_func(y, y_pred, **kwargs).
        greater_score_is_better (bool, optional): Defaults to True. Whether score_func is a score function
            (default), meaning high is good, or a loss function, meaning low is good. In the latter case,
            the scorer object will sign-flip the outcome of the score_func.
        solver (str, optional): Defaults to 'lbfgs'. Algorithm to use in the optimization problem.
        sample_weight (array-like, shape (n_samples,), optional): Defaults to None. Array of weights
            that are assigned to individual samples. If not provided, then each sample is given unit
            weight.
        max_iter (int, optional): Defaults to 1000. Maximum number of iterations taken for the solvers
            to converge.
        random_state (int, RandomState instance or None, optional): Defaults to None. The seed of the
            pseudo random number generator to use when shuffling the data.
        n_jobs (int or None, optional): Defaults to None. Number of CPU cores used when parallelizing.
            None means 1 unless in a joblib.parallel_backend context. -1 means using all processors.
        name (str, optional): Defaults to 'classifier'. Name of classifier.
        progress_bar (progressbar.ProgressBar, optional): Defaults to None. Progress bar.

    Returns:
        dict: Results containing the name, best classifier estimator, best parameters, best score,
            training scores, and validation scores.

    Raises:
        NotImplementedError: If classifier is not LogisticRegression, SVC or RandomForestClassifier.

        """

    if progress_bar:
        progress_bar.start()

    train_scores = {}
    validation_scores = {}
    classifiers = {}
    num_iters = 0
    for params in param_grid:

        num_iters += 1
        if progress_bar:
            progress_bar.update(num_iters)

        # fit the model to training set
        if isinstance(clf, LogisticRegression):
            classifier = LogisticRegression(
                penalty=params.get('penalty', 'l2'), C=params.get('C', 1.0), solver=solver,
                random_state=random_state, class_weight=params.get('class_weight'), max_iter=max_iter)
        elif isinstance(clf, SVC):
            classifier = SVC(
                C=params.get('C', 1.0), kernel=params.get('kernel', 'rbf'),
                gamma=params.get('gamma', 'auto_deprecated'), random_state=random_state,
                class_weight=params.get('class_weight'), max_iter=max_iter)
        elif isinstance(clf, RandomForestClassifier):
            classifier = RandomForestClassifier(
                n_estimators=params.get('n_estimators', 'warn'),
                max_features=params.get('max_features', 'auto'),
                min_samples_leaf=params.get('min_samples_leaf', 1), random_state=random_state,
                class_weight=params.get('class_weight'), n_jobs=n_jobs)
        else:
            raise NotImplementedError
        classifier.fit(X_train, y_train, sample_weight=sample_weight)
        classifiers[str(params)] = classifier

        # predict on validation set and evaluate scores
        y_train_predict = classifier.predict(X_train)
        y_validation_predict = classifier.predict(X_validation)
        with warnings.catch_warnings():  # ignore runtime warnings caused by zero MCC
            warnings.filterwarnings('ignore', category=RuntimeWarning)
            train_scores[str(params)] = score_func(y_true=y_train, y_pred=y_train_predict)
            validation_scores[str(params)] = score_func(y_true=y_validation,
                                                        y_pred=y_validation_predict)

    if progress_bar:
        progress_bar.finish()

    # find the best scoring model
    sorted_validation_scores = sorted(
        validation_scores.items(), key=operator.itemgetter(1), reverse=greater_score_is_better)
    best_params = ast.literal_eval(sorted_validation_scores[0][0])
    best_score = sorted_validation_scores[0][1]
    best_classifier = classifiers[str(best_params)]

    # issue warning about LogisticRegression classifiers with infeasibly large or small coefficients
    # (corresponds to odds lower than p = 0.001 or larger than p = 0.999)
    # see https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-how-do-i-interpret-odds-ratios-in-logistic-regression/
    if isinstance(best_classifier, LogisticRegression):
        odds_ratio = np.exp(best_classifier.coef_.ravel())
        max_odds_ratio = np.max(odds_ratio)
        if max_odds_ratio > 0.999 / (1.0 - 0.999):
            print('Warning: The best classifier Logistic regression coefficients are infeasibly large. '
                  'The maximum odds ratio is {}-1.'.format(max_odds_ratio))

    # return results
    results = dict(name=name, best_classifier=best_classifier, best_params=best_params,
                   best_score=best_score, train_scores=train_scores, validation_scores=validation_scores)
    return results

def print_best_classifier(results):
    """Print the best classifier.

    Args:
        results (dict): Results of classifier evaluation.
    """

    print(results['name'])
    print('Best params: ', results['best_params'])
    print('Training score: ', round(results['train_scores'][str(results['best_params'])], 3))
    print('Validation score: ', round(results['best_score'], 3))
```

It's now time to select the best parameters for the two feature sets with and without the sample weights.

### Logistic Regression (LR)

The hyperparameters to be selected for the logistic regression model are:
- The `penalty`, which is used to specify whether the $L1$ or $L2$ norms are used in the [regularization](https://en.wikipedia.org/wiki/Regularization_(mathematics)). The latter favors sparse solutions and naturally performs feature selection.
- `C`, the inverse of regularization strength. Smaller values specify stronger regularization.
- `class_weight`, the weights associated with the classes. It penalizes mistakes in samples of a class with its associated class_weight. So a higher value indicates more emphasis is put on a class.

Let's perform the grid search now.

```python
penalty = ['l1', 'l2']
Cs= np.logspace(-5, 15, 21, base=2.0)
class_weight = ([{0: weight, 1: 1.0 - weight} for weight in np.linspace(0.0, 1.0, 21)] +
                 [{0: 1.0, 1: 1.0}] + ['balanced'])
param_grid = ParameterGrid(dict(penalty=penalty, C=Cs, class_weight=class_weight))

clf = LogisticRegression()
solver = 'liblinear'
bar = progress_bar(len(param_grid), banner_text_begin='Running: ', banner_text_end=' param sets')
```

```python
logit_results = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid, solver=solver,
    random_state=0, name='LR', progress_bar=bar)
print_best_classifier(logit_results)
```

```python
logit_results_weights = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid, solver=solver,
    sample_weight=sample_weights['weight'], random_state=1, name='LR + sample weights', progress_bar=bar)
print_best_classifier(logit_results_weights)
```

From the warning message it is clear that the odds ratio is impractically large and consequently we cannot trust the model. This appears to be caused by some combination of `sample_weight` and `class_weight`. Note that the above warning appears for all values of the `class_weight` parameter when $L1$ regularization is used. It does not appear when $L2$ regularization is used. Furthermore, the performance of the model is independent of the `class_weight` under $L2$ regularization. So let's try again excluding these parameters from the grid search ($L2$ regularization is used by default).

```python
logit_results_weights = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=ParameterGrid(dict(C=Cs)),
    solver=solver, sample_weight=sample_weights['weight'], random_state=1, name='LR + sample weights',
    progress_bar=bar)
print_best_classifier(logit_results_weights)
```

OK this looks much better now.

```python
logit_results_topics = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid,
    solver=solver, random_state=2, name='LR (topics)', progress_bar=bar)
print_best_classifier(logit_results_topics)
```

```python
logit_results_topics_weights = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid,
    solver=solver, sample_weight=sample_weights_topics['weight'], random_state=3,
    name='LR (topics) + sample weights', progress_bar=bar)
print_best_classifier(logit_results_topics_weights)
```

We can make the following observations about the results:
- All of the models selected a much larger `class_weight` for the positive class than the negative class (except for the model where we excluded this hyperparameter from the grid search). It seems like the choice of this hyperparamter is important.
- Models fitted with the original features are overfitting and those with the topics features are underfitting (the validation MCC's are higher than the training MCCs).
- Applying strong regularization does not improve performance for the original features.

### Support Vector Machine (SVM)

The hyperparameters to be selected for the support vector machine model are:
- The regularization parameter `C` of the error term. This parameter trades off correct classification of training examples against maximization of the separating hyperplane's margin. For larger values of `C`, a smaller margin will be accepted if the separating hyperplane is better at classifying training points correctly. Lower values of `C` encourage a larger margin at the cost of misclassifying more training points.
- `class_weight`, as defined above for logistic regression.

Note that the `kernel` parameter, which is used to specify the kernel type to be used in the algorithm, will be fixed as the *linear* kernel. The reason for not considering the *RBF* or *poly* kernels is interpretability of the model. For the *RBF* and *poly* kernels, the [separating hyperplane and the weights that define it exist in a transformed space](https://stackoverflow.com/questions/21260691/how-to-obtain-features-weights) that is not directly related to the input feature space. OK let's perform the grid search now.

```python
Cs = np.logspace(-5, 10, 16, base=2.0)
class_weight = ([{0: weight, 1: 1.0 - weight} for weight in np.linspace(0.0, 1.0, 21)] +
                 [{0: 1.0, 1: 1.0}] + ['balanced'])
param_grid = ParameterGrid(dict(kernel=['linear'], C=Cs, class_weight=class_weight))

clf=SVC()
max_iter = -1
bar = progress_bar(len(param_grid), banner_text_begin='Running: ', banner_text_end=' param sets')
```

```python
svm_results = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid, random_state=4,
    max_iter=max_iter, name='SVM', progress_bar=bar)
print_best_classifier(svm_results)
```

```python
svm_results_weights = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid,
    sample_weight=sample_weights['weight'], random_state=5, max_iter=max_iter,
    name='SVM + sample weights', progress_bar=bar)
print_best_classifier(svm_results_weights)
```

```python
svm_results_topics = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid,
    random_state=6, max_iter=max_iter, name='SVM (topics)', progress_bar=bar)
print_best_classifier(svm_results_topics)
```

```python
svm_results_topics_weights = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid,
    sample_weight=sample_weights_topics['weight'], random_state=7, max_iter=max_iter,
    name='SVM (topics) + sample weights', progress_bar=bar)
print_best_classifier(svm_results_topics_weights)
```

We can make the following observations about the results:
- Once again, since all of the models selected a larger `class_weight` for the positive class than the negative class, we can see that the choice of this hyperparameter is very important. Interestingly, the balanced `class_weight` is chosen for the topics features.
- Again we see overfitting and underfitting of the models. However, for the classifiers fitted with the sample weights, the effect is far less pronounced than for the logistic regression models.
- The sample-weighted classifiers seem to favor larger values of C (i.e. smaller margin) whereas the unweighted ones facvor smaller values of C (i.e larger margin).

### Random Forest (RF)

There are a multitude of hyperparameters that can be selected for a random forest model. The hyperparameters control the randomness of the forest. The aim is to set them such that reasonable predictive power of individual trees is achieved without excessive correlation between the trees (i.e. bias-variance trade-off). Based on the advice of *Probst et al.* in [Hyperparameters and Tuning Strategies for Random Forest](https://arxiv.org/pdf/1804.03515.pdf), we will restrict ourselves to the following most influential hyperparameters:

- The number of features `max_features` to consider when looking for the best split. Lower values lead to more diverse and less correlated trees, which results in more stable aggregation of predictions. However, lower values also lead to the individual trees being weaker predictors of the target variable.
- The minimum number of samples required to be at a leaf node `min_samples_leaf`. Lower values lead to
trees of larger depth, which means that more splits are performed until the terminal nodes. More splits means the model is more complex and this can potentially lead to overfitting.
- `class_weight`, as defined above for logistic regression.

Note that the number of trees in the forest `n_estimators` will be fixed at 500. Following the recommendations of *Probst et al.*, "the number of trees should be set high: the higher the number of trees, the better the results in terms of performance and precision of variable importances. However, the improvement obtained by adding trees diminishes as more and more trees are added." 500 trees in the forest seems like a reasonable compromise based on the stability of the solution and computational resources available.

Let's perform the grid search now. Note that this takes a considerable amount of time due to the number of trees in the forest and the large hyperparameter space being searched. To reduce the computation time, we will perform a **random search** over one-quarter of the values in the hyperparameter space, by drawing randomly from a uniform distribution. *Bergstra and Bengio* show in [Random Search for Hyper-Parameter Optimization](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf) that for neural networks, random search is more efficient in searching good hyperparameter specifications than grid search.

```python
n_estimators = [500]
max_features = [0.01] + list(np.linspace(0.05, 0.95, 7)) + ['sqrt', None]
min_samples_leaf = np.logspace(0, 6, 7, base=2.0).astype('int64')

class_weight = ([{0: weight, 1: 1.0 - weight} for weight in np.linspace(0.0, 1.0, 11)] +
                 [{0: 1.0, 1: 1.0}] + ['balanced', 'balanced_subsample'])
param_grid = dict(n_estimators=n_estimators, max_features=max_features,
                  min_samples_leaf=min_samples_leaf, class_weight=class_weight)
param_grid_sampler = ParameterSampler(param_grid, n_iter=int(0.25 * len(list(ParameterGrid(param_grid)))),
                                      random_state=8)

clf=RandomForestClassifier()
n_jobs = -1
bar = progress_bar(len(param_grid_sampler), banner_text_begin='Running: ', banner_text_end=' param sets')
```

```python
rf_results = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid_sampler, random_state=9,
    n_jobs=n_jobs, name='RF', progress_bar=bar)
print_best_classifier(rf_results)
```

```python
rf_results_weights = evaluate_classifier(
    X_train, y_train, X_validation, y_validation, clf=clf, param_grid=param_grid_sampler,
    random_state=10, sample_weight=sample_weights['weight'], n_jobs=n_jobs, name='RF + sample weights',
    progress_bar=bar)
print_best_classifier(rf_results_weights)
```

```python
rf_results_topics = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid_sampler,
    random_state=11, n_jobs=n_jobs, name='RF (topics)', progress_bar=bar)
print_best_classifier(rf_results_topics)
```

```python
rf_results_topics_weights = evaluate_classifier(
    X_train_topics, y_train, X_validation_topics, y_validation, clf=clf, param_grid=param_grid_sampler,
    sample_weight=sample_weights_topics['weight'], random_state=12, n_jobs=n_jobs,
    name='RF (topics) + sample weights', progress_bar=bar)
print_best_classifier(rf_results_topics_weights)
```

We can make the following observations about the results:
- Once again, since all of the models selected a much larger `class_weight` for the positive class than the negative class, we can see that the choice of this hyperparameter is very important.
- The original features classifiers seem to favor lower values of `min_samples_leaf` (i.e. deeper trees) and are overfitting considerably.
- The `max_features` is low for the majority of the classifiers, which means the individual trees are diverse, but weak predictors.

## Model Selection

Despite the Matthews Correlation Coefficient (MCC) being a balanced measure, which is computed using the entire confusion matrix, it is not a suitable metric for choosing between classifiers. The reason is that its value depends on the **threshold**, $T$, that is used for separating the positive from the negative class. As such, it is quite possible that two classifiers can be made to perform identically by simply adjusting one of their thresholds. To avoid such artifacts, a nonparametric performance measure such as a [Receiver Operator Characteristic](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) (ROC) curve is generally applied.

The ROC curve is constructed by using different values of the threshold, $T$,
to plot the [false positive rate](https://en.wikipedia.org/wiki/False_positive_rate) (FPR) (or alternatively the [true negative rate](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)) (TNR = 1 - FPR) on the x-axis against the [true positive rate](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) (TPR) on the y-axis. The [area under the ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve) is typically used as a measure of the predictive performance of a classifier. An alternative, is to construct the [precision-recall](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html) (PR) curve by using different values of the threshold, $T$, to plot the [recall](https://en.wikipedia.org/wiki/Precision_and_recall) on the x-axis against the [precision](https://en.wikipedia.org/wiki/Precision_and_recall) on the y-axis. In this case, the [average precision](https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html) is typically used as a measure of the predictive performance of a classifier. For [imbalanced datasets the PR curve is more informative](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4349800/).

We would prefer not to resort to new measures and would like to continue working directly with the MCC. Following the methodology of *Zhou and Jakobsson* in [Predicting Protein-Protein Interaction by the MirrortreeMethod: Possibilities and Limitations](https://www.researchgate.net/publication/259354929_Predicting_Protein-Protein_Interaction_by_the_Mirrortree_Method_Possibilities_and_Limitations) we can construct an MCC curve. This curve plots different values of the threshold, $T$, on the x-axis against the [Matthews Correlation Coefficient](https://en.wikipedia.org/wiki/Matthews_correlation_coefficient) (MCC) on the y-axis. The area under the MCC curve seems like an intuitive measure of the predictive performance of a classifier and is defined by:

\begin{equation}
AUC_{MCC} = \int_{-\infty}^{\infty} MCC(T) dT
\end{equation}

However, the problem with this measure is that it is not easy to discern good performance versus bad performance. So let's normalize by the area under the MCC curve assuming $MCC \equiv 1$ for *all* thresholds, $T$. We can now define a new measure known as the **normalized area under the MCC curve**:

\begin{equation}
NAUC_{MCC} = \frac{AUC_{MCC}}{AUC_{MCC \equiv 1}}
\end{equation}

As a result, the interpretation of the $NAUC_{MCC}$ is analogous to that of the MCC. It has an upper limit of +1 indicating a perfect prediction, a lower limit of -1 indicating total disagreement between prediction and observation and a mid value of 0 representing a random prediction. To the best of our knowledge this is the first time the normalized area under the MCC curve has been defined and used in a study.

An interesting case is the case of a probabilistic classfier where $T \in [0, 1]$. By definition, for such a classifier, $AUC_{MCC \equiv 1} = 1$ and $NAUC_{MCC} = AUC_{MCC} = \int_{0}^{1} MCC(T) dT$. So we see that the AUC is already normalized. For all other classifiers we should perform the normalization by integrating over the finite range of thresholds.

Now let's proceed and plot the MCC curves for each of the classifiers, evaluate their normalized AUC and see how they perform compared to our [baseline classifier](5.0-baseline-model.ipynb). Note that for the support vector machine (SVM) classifiers, we will normalize the thresholds $T \in [0, 1]$ so that their performance can be visualized on the same scale as the logistic regression (LR) and random forest (RF) classifiers.

```python
def plot_mccs(
    clfs, X, linestyle='solid', colors = ['black', 'blue', 'green', 'red', 'orange', 'purple'],
    normalize_thresholds=True, ax=None):

    """Plot the Matthew Correlation Coefficients (MCC) as a function of the classification threshold for
        a list of classifiers.

    Args:
        clfs (list): Classifiers.
        X ({array-like, sparse matrix}, shape (n_samples, n_features)): Features matrix, where n_samples
            is the number of samples and n_features is the number of features.
        linestyle (str, optional): Defaults to 'solid'. Linestyle to plot.
        colors (list, optional): Defaults to ['black', 'blue', 'green', 'red', 'orange', 'purple']. Colors
            to plot the lines in. They will be colored according to this order.
        normalize_thresholds (bool, optional): Defaults to True. Normalized the thresholds to lie in [0, 1].
        ax (matplotlib.axes.Axes, optional): Defaults to None. axes.

    Raises:
        NotImplementedError: Raises if classifier does not have `predict_proba` or `decision_function`
            attribute.

    Returns:
        matplotlib.axes.Axes): axes.
    """

    if not ax:
        fig, ax = plt.subplots(figsize=(10, 8))

    for color, clf in enumerate(clfs):
        best_classifier = clf['best_classifier']
        if hasattr(best_classifier, 'predict_proba'):
            y_score = best_classifier.predict_proba(X)[:, 1]
            probability = True
        elif hasattr(best_classifier, 'decision_function'):
            y_score = best_classifier.decision_function(X)
            probability = False
        else:
            raise NotImplementedError
        mccs, _, _, thresholds = mcc_curve(y_true=y_validation, y_score=y_score,
                                           probability=probability)
        mcc_auc = mcc_auc_score(y_true=y_validation, y_score=y_score, sample_weight=None,
                                probability=probability)

        if normalize_thresholds and (np.min(thresholds) < 0.0 or np.max(thresholds) > 1.0):
            scaler = MinMaxScaler()
            thresholds = scaler.fit_transform(thresholds.reshape(-1, 1))

        ax.plot(thresholds, mccs, linestyle=linestyle, color=colors[color],
                label=clf['name'] + ' - {} NAUC'.format(round(mcc_auc, 3)))

    ax.set_xlabel('Classification Threshold')
    ax.set_ylabel('Matthews Correlation Coefficient (MCC)')
    ax.set_title('Matthews Correlation Coefficient (MCC) vs Classification Threshold')
    ax.legend()

    return ax
```

```python
classifiers = [logit_results, logit_results_weights, svm_results, svm_results_weights, rf_results,
               rf_results_weights]
classifiers_topics = [logit_results_topics, logit_results_topics_weights, svm_results_topics,
                      svm_results_topics_weights, rf_results_topics, rf_results_topics_weights]
```

```python
ax = plot_mccs(classifiers, X_validation)
ax = plot_mccs(classifiers_topics, X_validation_topics, linestyle='dashed', ax=ax)
baseline_mcc = matthews_corrcoef(y_validation, X_validation.num_workplaces_at_least_2)
ax.axhline(y=baseline_mcc, linestyle='-.',
           label='Baseline' + ' - {} MCC'.format(round(baseline_mcc, 3)))
ax.set_xlim(0.0, 1.0)
ax.set_ylim(-0.2, 1.0)
ax.set_xlabel('Normalized Classification Threshold')
ax.set_title('Matthews Correlation Coefficient (MCC) vs Normalized Classification Threshold')
ax.legend(ncol=2);
```

We can make the following observations about the chart:

1. All of the models beat the naive baseline classifier over at least some range of classification thresholds. As such we can conclude that machine learning is appropriate for this task.
2. All $NAUC_{MCC}$'s are significantly greater than zero. This lends further support to the first point. It looks like there is some signal in the data for predicting the target. However, the low values indicate that the majority of the classifiers are weak.
3. The logistic regression classifiers perform better than the random forest classifiers, which in turn, perform better than the support vector machine classifiers.
4. Performance of the classifiers trained with the topics features are worse than those trained with the original features. It seems like some important information was lost during topic modeling.
5. The effect of the sample weights varies with classifier type and feature type. Using sample weights improves the performance of the SVM classifier when using the original features. However, sample weights worsen the performance when topics features are used or the classifier is a logistic regression or random forest. Although, the difference is very small for the original features.

Clearly we should select the logistic regression classifier as it is the best performer based on $NAUC_{MCC}$.

## Optimal Classification Threshold

Our work is not quite done yet as we still have to choose an optimal classification threshold for the **LR classifier**. Below we plot the classification threshold against the Matthews Correlation Coefficient (MCC), true positive rate (TPR) and true negative rate (TNR) to help us with this selection. Naturally, there is a tradeoff between the TPR and the TNR. Minimizing false positives (i.e. maximizing the TNR) is more important than minimizing false negatives (i.e. maximizing the TPR) when classifying the physicists as laureates and non-laureates. This is because we are trying to gain insight into any kinds of biases that may involved when a physicist is awarded the Nobel Prize. So any threshold to the right of the intersection of the TPR and TNR would satisfy this criteria. However, we must also ensure that we retrieve as many of the true positives as possible so that the conclusions we draw are based on a large as possible sample of *actual* laureates. So intuitively, the threshold corresponding to the maximum value of the MCC, seems to be a good choice for the optimal classification threshold.

```python
def plot_mcc_tpr_tnr(clf, X, ax=None):
    """Plot the Matthew Correlation Coefficients (MCC), true positive rate (TPR) and true negative rate
        (TNR) as a function of the classification threshold for a classifier.

    Args:
        clf ([type]): [description]
        X ({array-like, sparse matrix}, shape (n_samples, n_features)): Features matrix, where n_samples
            is the number of samples and n_features is the number of features.
        ax (matplotlib.axes.Axes, optional): Defaults to None. axes.

    Raises:
        NotImplementedError: Raises if classifier does not have `predict_proba` or `decision_function`
            attribute.

    Returns:
        matplotlib.axes.Axes): axes.

        float: Threshold corresponding to maximum MCC.
    """

    if not ax:
        fig, ax = plt.subplots(figsize=(8, 6))

    best_classifier = clf['best_classifier']
    if hasattr(best_classifier, 'predict_proba'):
        y_score = best_classifier.predict_proba(X)[:, 1]
        probability = True
    elif hasattr(best_classifier, 'decision_function'):
        y_score = best_classifier.decision_function(X)
        probability = False
    else:
        raise NotImplementedError
    mcc, tnr, tpr, thresholds = mcc_curve(y_true=y_validation, y_score=y_score, probability=probability)

    ax.plot(thresholds, mcc, color='black', label='Matthews Correlation Coefficient (MCC)')
    ax.plot(thresholds, tpr, color='blue', label='True Positive Rate (TPR)')
    ax.plot(thresholds, tnr, color='green', label='True Negative Rate (TNR)')

    max_mcc_idx = np.argmax(mcc)
    ax.axvline(thresholds[max_mcc_idx], color='red', linestyle='dashed', linewidth=1.0,
               label='Max MCC = {} (threshold = {})'.format(round(np.max(mcc), 3),
                                                            round(thresholds[max_mcc_idx], 3)))
    ax.set_xlim(0.0, 1.0)
    ax.set_ylim(0.0, 1.0)

    ax.set_xlabel('Classification Threshold')
    ax.set_title('Matthews Correlation Coefficient (MCC) / True Positive Rate (TPR) \n'
                 '/ False Negative Rate (FNR) vs Classification Threshold')
    ax.legend()

    return ax, thresholds[max_mcc_idx]
```

```python
_, optimal_threshold = plot_mcc_tpr_tnr(logit_results, X_validation);
```

Or alternatively, if you prefer, the confusion matrix and the classification report of this classifier tell a similar story to that described above.

```python
best_classifier = logit_results['best_classifier']
y_pred = (best_classifier.predict_proba(X_validation)[:, 1] > optimal_threshold).astype('int64')
display(confusion_matrix_to_dataframe(confusion_matrix(y_validation, y_pred)))
print(classification_report(y_validation, y_pred))
```

## Persisting the Best Classifier Parameters

Now we have the best classifier, let's persist it's parameters and associated metadata, which will allow us to reconstruct the classifier at any point in the future. Note that we are not persisting the actual sklearn estimator using [joblib](https://pypi.org/project/joblib/) or [pickle](https://docs.python.org/3/library/pickle.html) as we would like to avoid any [compatibility or security issues](https://stackabuse.com/scikit-learn-save-and-restore-models/#compatibilityissues).

```python
best_classifier_params = pd.Series(
    type(best_classifier), name=logit_results['name'].replace(' ', '_'), index=['estimator'])
best_classifier_params['params'] = best_classifier.get_params()
best_classifier_params['threshold'] = optimal_threshold
best_classifier_params
```

```python
best_classifier_params.to_csv('../models/' + best_classifier_params.name + '.csv', header=True)
```

As a sanity check, let's recreate the estimator and make sure that we get the same results as before.

```python
best_classifier_params_check = pd.read_csv('../models/' + best_classifier_params.name + '.csv',
                                           squeeze=True, index_col=0)
best_classifier_params_check
```

```python
best_classifier_check = LogisticRegression(**ast.literal_eval(best_classifier_params_check.params))
best_classifier_check.fit(X_train, y_train)
```

```python
assert(np.array_equal(best_classifier_check.predict(X_validation), best_classifier.predict(X_validation)))
np.testing.assert_allclose(best_classifier_check.predict_proba(X_validation),
                           best_classifier.predict_proba(X_validation))
y_pred_check = (
    (best_classifier_check.predict_proba(
        X_validation)[:, 1] > best_classifier_params.threshold).astype('int64'))
assert(np.array_equal(y_pred_check, y_pred))
```

Great, everything looks good.


```python
import numpy as np
import sympy as sp
import adaptive_trapezint as p1
import sinesum1 as p2
import centered_diff as p3
import find_primes as p4
```

# Homework 2
Michael Seaman

2/12/16

## Problem 3.8: Adaptive Trapazoid Approximation
Using the trapazoid approximation to find areas under the curve, we can get a good guess at bounded integration, however, we can find how many trapazoids we want to use  by specifying a maximum error $\epsilon$ and then using the following equation:

$n = (b-a)\sqrt{\frac{(b-a) max\left|f''(x)\right|}{12\epsilon}}$

In this example, "some_function" is $f(x) = x^{4} + x^{2}$ evaluated from 0 to 20.

Actual result? $$\int_{0}^{20} (x^4 + x^2) dx = 642667 $$

```python
def some_function(x):
    return x**4 + x**2
print(p1.adaptive_trapezint(some_function, 0, 20))
```

    642665.509138

## Problem 3.15: Fourier series Approximation
The function we're trying to model is a piecewise function following:

$f(t) =    \left\{
\begin{array}{ll}
      1 & 0<t<T/2 \\
      0 & t = T/2 \\
      -1 & T/2<t<T \\
\end{array}
\right.$

We're trying to approximate it with the sinosoidal Fourier approximation:
$S(t;n) = \frac{4}{\pi}\sum_{i=0}^{n}\frac{1}{2i-1}sin(\frac{2(2i+1)\pi t}{T})$

We will see how accurate S is at approximating f by trying different vaules of n (the number of sinewaves used to approximate) and with different vaues for $\alpha$ where $\alpha=\frac{t}{T}$.

```python
p2.print_error_results()
```

    alphas:    0.01    0.25    0.49
    n: 1      0.92    -0.273    0.92
    n: 3      0.762    -0.103    0.762
    n: 5      0.609    -0.0631    0.609
    n: 10      0.267    0.0318    0.267
    n: 30      -0.145    0.0106    -0.145
    n: 100      0.0501    0.00318    0.0501

## Problem 3.18 Numerical differentiation

The "diff" function follows the formula:

$f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}$

in order to approximate the derivative of f at x using a very small value for h.

Below we use the functions:

$f_{1}(x) = e^{x}$ at $x=0$

$f_{2}(x) = e^{-2x^{2}}$ at $x=0$

$f_{3}(x) = cos{x}$ at $x=2\pi$

$f_{4}(x) = ln{x}$ at $x=1$

and calculate the difference of our differentiating function's output with h = .01 with the actual result.

```python
p3.application()
```

    Result: 1.00001666675, answer: 1, difference -1.66667499921e-05
    Result: 0.0, answer: 0, difference 0.0
    Result: 0.0, answer: 0, difference 0.0
    Result: 1.00003333533, answer: 1, difference -3.33353334772e-05

## 3.34 Finding Prime numbers
This application uses the sieve of Eratosthenes algorithm to find pirme numbers less than or equal to the input. The algorithm checks off multiples of the lowest candidate that is still prime and removes them from the candidates; then moving up to the next.

```python
p4.find_primes(10)
```

    [2, 3, 5, 7]

```python
print(p4.find_primes(500))
```

    [2,
     3,
     5,
     7,
     11,
     13,
     17,
     19,
     23,
     29,
     31,
     37,
     41,
     43,
     47,
     53,
     59,
     61,
     67,
     71,
     73,
     79,
     83,
     89,
     97,
     101,
     103,
     107,
     109,
     113,
     127,
     131,
     137,
     139,
     149,
     151,
     157,
     163,
     167,
     173,
     179,
     181,
     191,
     193,
     197,
     199,
     211,
     223,
     227,
     229,
     233,
     239,
     241,
     251,
     257,
     263,
     269,
     271,
     277,
     281,
     283,
     293,
     307,
     311,
     313,
     317,
     331,
     337,
     347,
     349,
     353,
     359,
     367,
     373,
     379,
     383,
     389,
     397,
     401,
     409,
     419,
     421,
     431,
     433,
     439,
     443,
     449,
     457,
     461,
     463,
     467,
     479,
     487,
     491,
     499]


About      Issues      Tutorials      Documentation

Tutorial 3: Intro to Quantum Chemistry

---

This tutorial shows how to select a quantum chemistry method, visualize orbitals, and analyze the electronic wavefunction.

```python
%matplotlib inline
import numpy as np
from matplotlib.pylab import *
try: import seaborn  #optional, makes plots look nicer
except ImportError: pass

import moldesign as mdt
from moldesign import units as u
```

## I. Build and minimize minimal basis set hydrogen

### A. Build the molecule
This cell builds H2 by creating the two atoms, and explicitly setting their positions.

**Try editing this cell to**:
 * Create HeH+
 * Create H3+
 * Change the atoms' initial positions

```python
atom1 = mdt.Atom('H')
atom2 = mdt.Atom('H')
atom1.bond_to(atom2,1)
atom2.x = 2.0 * u.angstrom

h2 = mdt.Molecule([atom1,atom2], name='H2', charge=0)
h2.draw(height=300)
```

### B. Run a hartree-fock calculation
The next cell adds the RHF energy model to our molecule, then triggers a calculation.

**Try editing this cell to**:
 * Change the atomic basis
 * Get a list of other available energy models (type `mdt.models.` and then hit the `[tab]` key)

```python
h2.set_energy_model(mdt.models.RHF, basis='3-21g')
h2.calculate()

print('Calculated properties:', h2.properties.keys())

print('Potential energy:', h2.potential_energy)
```

### C. Visualize the orbitals
After running the calculation, we have enough information to visualize the molecular orbitals.

```python
h2.draw_orbitals()
```

### D. Minimize the energy
Here, we'll run a quick energy minimization then visualize how the hydrogen nuclei AND the atomic wavefunctions changed.

```python
minimization = h2.minimize(frame_interval=1, nsteps=10)
minimization.draw_orbitals()
```

# II. Analyzing the wavefunction

The wavefunction created during QM calculations will be stored as an easy-to-analyze python object:

```python
wfn = h2.wfn
wfn
```

## A. Molecular orbital data
First, let's examine the molecular orbitals. The overlaps, fock matrix, coefficents, and density matrix are all available as 2D numpy arrays (with units where applicable).

We'll specifically look at the "canonical" orbitals that result from Hartree-Fock calculations.

```python
mos = wfn.orbitals.canonical
mos
```

MOs are, of course, a linear combination of AOs:

\begin{equation} \left| \text{MO}_i \right \rangle = \sum_j c_{ij} \left| \text{AO}_j \right\rangle \end{equation}

The coefficient $c_{ij}$ is stored at `mos.coeffs[i,j]`

```python
mos.coeffs
```

Most MO sets are orthogonal; their overlaps will often be the identity matrix (plus some small numerical noise)

```python
mos.overlaps
```

By definition, the fock matrix should be orthogonal as well; the orbital energies are on its diagonal.

```python
matshow(mos.fock.value_in(u.eV), cmap=cm.seismic)
colorbar(label='fock element/eV')
title('Fock matrix')
```

The `MolecularOrbitals` class also offers several methods to transform operators into different bases. For instance, the `overlap` method creates an overlap matrix between the AOs and MOs, where `olap[i,j]` is the overlap between MO _i_ and AO _j_:
\begin{equation}
\text{olap[i,j]} = \left\langle MO_i \middle| AO_j \right \rangle
\end{equation}

```python
olap = mos.overlap(wfn.aobasis)
olap
```

Various other matrices are available from this this object, such as the two-electron Hamiltonian terms:

```python
matshow(mos.h2e.value_in(u.eV), cmap=cm.inferno)
colorbar(label='2-electron hamiltonian term / eV')
title('2-electron hamiltonian')
```

## B. Individual orbitals

You can work with inidividual orbitals as well. For instance, to get a list (in order) of our four atomic orbitals (i.e., the basis functions):

```python
aos = wfn.orbitals.atomic
aos[:]
```

Let's grab the lowest orbital and examine some of its properties:

```python
orb = aos[0]
print('Name:', orb.name)
print('Energy:', orb.energy)
```

Orbital objects also give you access to various matrix elements:

```python
ha_1s = aos[0]
hb_1s = aos[3]

print('Overlap between 1s orbitals: ', ha_1s.overlap(hb_1s))
print('Fock element between 1s orbitals', ha_1s.fock_element(hb_1s))
```

## C. Basis functions

An object representing the wavefunction's basis functions is available at `wfn.aobasis`

```python
wfn.aobasis
```

It stores a list of `AtomicBasisFunction` objects:

```python
basis_function = wfn.aobasis[0]
print('Name:', basis_function.name)
print('Angular quantum number:', basis_function.l)
print('Azimuthal quantum number:', basis_function.m)
print('Centered at:', basis_function.center)
```

Each basis function is defined as a linear combination of "primitive" 3D Gaussian functions:

```python
basis_function.primitives
```

And these primitives can themselves be examined:

```python
primitive = basis_function.primitives[0]
print(primitive)
print("Coeff:", primitive.coeff)
print("Alpha:", primitive.alpha)
```

```python

```

moldesign/_notebooks/Tutorial 3. Quantum Chemistry.ipynb


```python
import math
import plotting
```

    IOPub data rate exceeded.
    The notebook server will temporarily stop sending output
    to the client in order to avoid crashing it.
    To change this limit, set the config variable
    `--NotebookApp.iopub_data_rate_limit`.

# Trigonometry

## Right triangles

The study of the ratios of sides of triangles.

### Sine
$$\sin=\frac{\text{opposite}}{\text{hypotenuse}}$$

### Cosine
$$\cos=\frac{\text{adjacent}}{\text{hypotenuse}}$$

### Tangent
$$ \tan = \frac{\text{opposite}}{\text{adjacent}}$$

### Mnemonic

The mnemonic **soh-cah-toa**.

### Example

Suppose we have a triangle:

     A
     |\
     | \
     |55\
     |   \
    4|    \ ?
     |     \
     |      \
     |       \
     |________\
     B         C

Given $\Delta ABC$, find $AC$.

First we determine which trigonometric ratio to use. We want to calculate the hypotenuse and we know the length of a side that is adjacent to angle $A$. The ratio that contains both is $\cos$.

$$
\begin{align}
\cos(E) &= \frac{\text{adjacent}}{\text{hypotenuse}} \\
\cos(55) &= \frac{4}{AC} \\
AC \cdot \cos(55) &= 4 \\
AC &= \frac{4}{\cos(55)}
\end{align}
$$

```python
4 / math.cos(math.radians(55))
```

    6.973787182484391

$AC \approx 6.97$

### Excersises

```python
8.2 / math.sin(math.radians(72))
```

    8.62199023875379

```python
3 * math.tan(math.radians(37))
```

    2.2606621503083826

### Finding angles

If it is a right triangle $\Delta ABC$ and angle $\angle A$ is known, it is trivial to solve for $\angle B$:

$$\begin{align}
\angle A + \angle B + 90^\circ &= 180^\circ \\
\angle B &= 180^\circ - 90^\circ - \angle A
\end{align}$$

### Inverse trig functions

$$
\begin{align}
\sin(\theta) = \frac{ \text{opposite} }{ \text{hypotenuse} } &\rightarrow \sin^{-1} \left( \frac{ \text{opposite} }{ \text{hypotenuse} } \right) = \theta \\
\cos(\theta) = \frac{ \text{adjacent}}{\text{hypotenuse}} &\rightarrow \cos^{-1} \left( \frac{ \text{adjacent}}{\text{hypotenuse}} \right) = \theta \\
\tan(\theta) = \frac{ \text{opposite}}{\text{adjacent}} &\rightarrow \tan^{-1} \left( \frac{ \text{opposite}}{\text{adjacent}} \right) = \theta
\end{align}
$$

**Example**

     A
     |\
     |?\
     |  \
     |   \
    4|    \ 5
     |     \
     |      \
     |       \
     |________\
     B         C

We have the adjacent to $\angle A$ and the hypotenuse. So we use $\cos$:

$$ \cos^{-1}\left(\frac{4}{5}\right) = \angle A $$

```python
math.degrees(math.acos(4/5))
```

    36.86989764584401

$\angle A \approx 36.86$

### Triangle similarity

For two triangles $\Delta ABC$ and $\Delta DEF$ which both have the same angle $\theta$:

     B
     |\ 90-θ
     | \
     |  \        E
     |   \       |\ 90-θ
     |    \      | \
     |     \     |  \
     |      \    |__θ\
     |       \   F    D
     |_______θ\
     C         A

We can find the following ratios:

$$ \frac{BC}{BA}=\frac{EF}{ED} \qquad \frac{AC}{AB}=\frac{DF}{DE} \qquad \frac{BC}{CA}=\frac{EF}{DF} $$

Which can be generalized to (from the view of $\theta$):

$$ \frac{\text{opposite}}{\text{hypotenuse}} \qquad \frac{\text{adjacent}}{\text{hypotenuse}} \qquad \frac{\text{opposite}}{\text{adjacent}}$$

Now we formarly define these ratios to get:

$$ \sin\theta=\frac{\text{opposite}}{\text{hypotenuse}} \qquad \cos\theta=\frac{\text{adjacent}}{\text{hypotenuse}} \qquad \tan\theta=\frac{\text{opposite}}{\text{adjacent}}$$

### Cofunctions

**Sine and cosine**
$$\begin{align}
\sin(\theta) &= \cos(90^\circ - \theta) \\
\cos(\theta) &= \sin(90^\circ - \theta)
\end{align}$$

**Tangent and cotangent**
$$\begin{align}
\tan(\theta) &= \cot(90^\circ - \theta) \\
\cot(\theta) &= \tan(90^\circ - \theta)
\end{align}$$

**Secant and cosecant**
$$\begin{align}
\sec(\theta) &= \csc(90^\circ - \theta) \\
\csc(\theta) &= \sec(90^\circ - \theta)
\end{align}$$

### Trig ratios of special triangles

There are two special triangles:
* 45-45-90
* 30-60-90

**30-60-90**

The longer leg is $x\sqrt{3}$ and the hypotenuse is twice as long as the shortest leg $2x$.

**45-45-90**

The hypotenuse is $x\sqrt{2}$.

**Deriving ratios**

We can use those triangles to derive the trig ratios for $30^\circ$, $45^\circ$, and $60^\circ$:

$$\begin{array}{c|lcr}
 & \cos(\theta) & \sin(\theta) & \tan(\theta) \\
 \hline
 \theta = 30^\circ & \frac{\sqrt{3}}{2} & \frac{1}{2} & \frac{\sqrt{3}}{3} \\
 \theta = 45^\circ & \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2} & 1 \\
 \theta = 60^\circ & \frac{1}{2} & \frac{\sqrt{3}}{2} & \sqrt{3}
\end{array}$$

### Pythagorean trig identity

When we have a triangle:

           /|
          / |
         /  |
      c /   |
       /    | b
      /     |
     /      |
    /_______|
        a

We can relate this to the Pythagorean theorem by doing so:

$$\begin{align}
\sin\theta = \frac{b}{c} &\qquad \cos\theta = \frac{a}{c} \\
\sin^2\theta = \frac{b^2}{c^2} &\qquad \cos^2\theta = \frac{a^2}{c^2} \\
\sin^2\theta + \cos^2\theta &= \frac{b^2}{c^2} + \frac{a^2}{c^2} \\
\end{align}$$

Now we have $\frac{b^2}{c^2} + \frac{a^2}{c^2} = \frac{a^2+b^2}{c^2}$ and by the Pythagorean theorem $a^2 + b^2 = c^2$, the result is $\frac{c^2}{c^2}=1$.

This gives us the _Pythagorean trig identity_:

$$\sin^2\theta + \cos^2\theta = 1$$

### Reciprocal trig ratios

**Cosecant (csc)**

$$\csc(A)=\frac{\text{hypotenuse}}{\text{opposite}} \qquad \rightarrow \qquad \csc(A)=\frac{1}{\sin(A)}$$

**Secant (sec)**

$$\sec(A)=\frac{\text{hypotenuse}}{\text{adjacent}} \qquad \rightarrow \qquad \sec(A)=\frac{1}{\cos(A)}$$

**Cotangent (cot)**

$$ \tan(A)=\frac{\text{adjacent}}{\text{opposite}} \qquad \rightarrow \qquad \tan(A)=\frac{1}{\tan(A)}$$

**Example**

To find the $\csc(45^\circ)$ we first draw the 45-45-90 triangle. From the triangle we find that $\sin(45^\circ) = \frac{1}{\sqrt{2}}$. The $\csc$ is reciprocal of $\sin$ so we can swap the fraction. The answer is $\sqrt{2}$.

## General triangles

### Law of Sines

$$\frac{\sin a}{A} = \frac{\sin b}{B} = \frac{\sin c}{C}$$

**Example**

Suppose we have a triangle, find $m\angle B$ using the _Law of Sines_.

            A
           /\
       11 /  \ 12
         /    \
        /73____\
       C        B

First we define:
$$ \frac{\sin(B)}{b}=\frac{\sin(C)}{c}$$

Multiply both sides by $b$:
$$ \sin(B) = \frac{b}{c}\sin(C)$$

Plug-in the values:
$$ \sin(B) = \frac{11}{12}\sin(73)$$

Evaluate the inverse sin to find $B$:
$$ B = \sin^{-1}\left(\frac{11}{12}\sin(73)\right)$$

Which is:
$$ B \approx 61^\circ$$

**Sidenote**

If an angle is _obtuse_ ($\angle A> 90^\circ$) we need to use the fact that $\sin(\theta) = \sin(180^\circ - \theta)$.

### Law of Cosines

$$c^2 = a^2 + b^2 - 2ab \cos \theta$$

**Example**

Suppose we have a triangle, find $a$ using the _Law of Cosines_:

           /\
          /  \
         /θ=87\
    b=12/      \c=9
       /        \
      /          \
     /____________\
           a=?

Plug-in the values and solve for $a$:

$$\begin{align}
a^2 &= 144 + 81 - 2(12)(9)\cos 87^\circ \\
a^2 &= 225 - 216 \cos 87^\circ \\
a &= \sqrt{225 - 216 \cos 87^\circ} \\
a &\approx 14.6
\end{align}$$

**Python examples**

```python
def side(theta,b,c):
    rtheta = math.radians(theta)
    return math.sqrt(b**2+c**2-2*b*c*math.cos(rtheta))

side(87,12,9)
```

    14.618325261517617

```python
def angle(a,b,c):
    rad = math.acos((a**2+b**2-c**2)/(2*a*b))
    return math.degrees(rad)

angle(50,60,20)
```

    18.194872338766785

Let's calculate the angles for a triangle with sides $9$, $12$, and $14.6$. The $c$ term in the function is for which side the angle is calculated.

```python
angle(9,12,14.6)
```

    86.85776755589895

```python
angle(12,14.6,9)
```

    37.98910087977064

```python
angle(9,14.6,12)
```

    55.15313156433045

Check the results:

```python
55.15+37.98+86.85
```

    179.98

## Unit circle

### Radians

* $2\pi$ radians = $360$ degrees.
* $\pi$ radian = $180$ degrees.

**Radians to Degrees**
$$1 \text{radian} = \frac{180}{\pi}\text{degrees}$$

```python
def d2r(degree):
    return degree * math.pi / 180
```

```python
d2r(180)
```

    3.141592653589793

```python
print(str(d2r(45) / math.pi) + 'π');
```

    0.25π

**Degrees to Radians**
$$\frac{\pi}{180}\text{radians}=1\text{degree}$$

```python
def r2d(rad):
    return rad / math.pi * 180
```

```python
r2d(math.pi)
```

    180.0

**Unit circle**

```python

```

Applied Math/KhanAcademy/.ipynb_checkpoints/KA - Trigonometry-checkpoint.ipynb