# PerfForesightConsumerType: Perfect foresight consumption-saving

```python
# Initial imports and notebook setup, click arrow to show

from copy import copy

import matplotlib.pyplot as plt
import numpy as np

from HARK.ConsumptionSaving.ConsIndShockModel import PerfForesightConsumerType
from HARK.utilities import plotFuncs

mystr = lambda number: "{:.4f}".format(number)
```

The module $\texttt{HARK.ConsumptionSaving.ConsIndShockModel}$ concerns consumption-saving models with idiosyncratic shocks to (non-capital) income.  All of the models assume CRRA utility with geometric discounting, no bequest motive, and income shocks are fully transitory or fully permanent.

$\texttt{ConsIndShockModel}$ currently includes three models:
1. A very basic "perfect foresight" model with no uncertainty.
2. A model with risk over transitory and permanent income shocks.
3. The model described in (2), with an interest rate for debt that differs from the interest rate for savings.

This notebook provides documentation for the first of these three models.
$\newcommand{\CRRA}{\rho}$
$\newcommand{\DiePrb}{\mathsf{D}}$
$\newcommand{\PermGroFac}{\Gamma}$
$\newcommand{\Rfree}{\mathsf{R}}$
$\newcommand{\DiscFac}{\beta}$

## Statement of perfect foresight consumption-saving model

The $\texttt{PerfForesightConsumerType}$ class the problem of a consumer with Constant Relative Risk Aversion utility
${\CRRA}$
\begin{equation}
U(C) = \frac{C^{1-\CRRA}}{1-\rho},
\end{equation}
has perfect foresight about everything except whether he will die between the end of period $t$ and the beginning of period $t+1$, which occurs with probability $\DiePrb_{t+1}$.  Permanent labor income $P_t$ grows from period $t$ to period $t+1$ by factor $\PermGroFac_{t+1}$.

At the beginning of period $t$, the consumer has an amount of market resources $M_t$ (which includes both market wealth and currrent income) and must choose how much of those resources to consume $C_t$ and how much to retain in a riskless asset $A_t$, which will earn return factor $\Rfree$.  The consumer cannot necessarily borrow arbitarily; instead, he might be constrained to have a wealth-to-income ratio at least as great as some "artificial borrowing constraint" $\underline{a} \leq 0$.

The agent's flow of future utility $U(C_{t+n})$ from consumption is geometrically discounted by factor $\DiscFac$ per period. If the consumer dies, he receives zero utility flow for the rest of time.

The agent's problem can be written in Bellman form as:

\begin{eqnarray*}
V_t(M_t,P_t) &=& \max_{C_t}~U(C_t) ~+ \DiscFac (1 - \DiePrb_{t+1}) V_{t+1}(M_{t+1},P_{t+1}), \\
& s.t. & \\
A_t &=& M_t - C_t, \\
A_t/P_t &\geq& \underline{a}, \\
M_{t+1} &=& \Rfree A_t + Y_{t+1}, \\
Y_{t+1} &=& P_{t+1}, \\
P_{t+1} &=& \PermGroFac_{t+1} P_t.
\end{eqnarray*}

The consumer's problem is characterized by a coefficient of relative risk aversion $\CRRA$, an intertemporal discount factor $\DiscFac$, an interest factor $\Rfree$, and age-varying sequences of the permanent income growth factor $\PermGroFac_t$ and survival probability $(1 - \DiePrb_t)$.

While it does not reduce the computational complexity of the problem (as permanent income is deterministic, given its initial condition $P_0$), HARK represents this problem with *normalized* variables (represented in lower case), dividing all real variables by permanent income $P_t$ and utility levels by $P_t^{1-\CRRA}$.  The Bellman form of the model thus reduces to:

\begin{eqnarray*}
v_t(m_t) &=& \max_{c_t}~U(c_t) ~+ \DiscFac (1 - \DiePrb_{t+1}) \PermGroFac_{t+1}^{1-\CRRA} v_{t+1}(m_{t+1}), \\
& s.t. & \\
a_t &=& m_t - c_t, \\
a_t &\geq& \underline{a}, \\
m_{t+1} &=& \Rfree/\PermGroFac_{t+1} a_t + 1.
\end{eqnarray*}

## Solution method for PerfForesightConsumerType

Because of the assumptions of CRRA utility, no risk other than mortality, and no artificial borrowing constraint, the problem has a closed form solution.  In fact, the consumption function is perfectly linear, and the value function composed with the inverse utility function is also linear.  The mathematical solution of this model is described in detail in the lecture notes [PerfForesightCRRA](http://econ.jhu.edu/people/ccarroll/public/lecturenotes/consumption/PerfForesightCRRA).

The one period problem for this model is solved by the function $\texttt{solveConsPerfForesight}$, which creates an instance of the class $\texttt{ConsPerfForesightSolver}$. To construct an instance of the class $\texttt{PerfForesightConsumerType}$, several parameters must be passed to its constructor as shown in the table below.

## Example parameter values to construct an instance of PerfForesightConsumerType

| Parameter | Description | Code | Example value | Time-varying? |
| :---: | --- | --- | --- | --- |
| $\DiscFac$ |Intertemporal discount factor  | $\texttt{DiscFac}$ | $0.96$ |  |
| $\CRRA $ |Coefficient of relative risk aversion | $\texttt{CRRA}$ | $2.0$ | |
| $\Rfree$ | Risk free interest factor | $\texttt{Rfree}$ | $1.03$ | |
| $1 - \DiePrb_{t+1}$ |Survival probability | $\texttt{LivPrb}$ | $[0.98]$ | $\surd$ |
|$\PermGroFac_{t+1}$|Permanent income growth factor|$\texttt{PermGroFac}$| $[1.01]$ | $\surd$ |
|$\underline{a}$|Artificial borrowing constraint|$\texttt{BoroCnstArt}$| $None$ |  |
|$(none)$|Maximum number of gridpoints in consumption function |$\texttt{aXtraCount}$| $200$ |  |
|$T$| Number of periods in this type's "cycle" |$\texttt{T_cycle}$| $1$ | |
|(none)| Number of times the "cycle" occurs |$\texttt{cycles}$| $0$ | |

Note that the survival probability and income growth factor have time subscripts; likewise, the example values for these parameters are *lists* rather than simply single floats.  This is because those parameters are *time-varying*: their values can depend on which period of the problem the agent is in.  All time-varying parameters *must* be specified as lists, even if the same value occurs in each period for this type.

The artificial borrowing constraint can be any non-positive $\texttt{float}$, or it can be $\texttt{None}$ to indicate no artificial borrowing constraint.  The maximum number of gridpoints in the consumption function is only relevant if the borrowing constraint is not $\texttt{None}$; without an upper bound on the number of gridpoints, kinks in the consumption function will propagate indefinitely in an infinite horizon model if there is a borrowing constraint, eventually resulting in an overflow error.  If there is no artificial borrowing constraint, then the number of gridpoints used to represent the consumption function is always exactly two.

The last two parameters in the table specify the "nature of time" for this type: the number of (non-terminal) periods in this type's "cycle", and the number of times that the "cycle" occurs.  *Every* subclass of $\texttt{AgentType}$ uses these two code parameters to define the nature of time.  Here, $\texttt{T_cycle}$ has the value $1$, indicating that there is exactly one period in the cycle, while $\texttt{cycles}$ is $0$, indicating that the cycle is repeated in *infinite* number of times-- it is an infinite horizon model, with the same "kind" of period repeated over and over.

In contrast, we could instead specify a life-cycle model by setting $\texttt{T_cycle}$ to $1$, and specifying age-varying sequences of income growth and survival probability.  In all cases, the number of elements in each time-varying parameter should exactly equal $\texttt{T_cycle}$.

The parameter $\texttt{AgentCount}$ specifies how many consumers there are of this *type*-- how many individuals have these exact parameter values and are *ex ante* homogeneous.  This information is not relevant for solving the model, but is needed in order to simulate a population of agents, introducing *ex post* heterogeneity through idiosyncratic shocks.  Of course, simulating a perfect foresight model is quite boring, as there are *no* idiosyncratic shocks other than death!

The cell below defines a dictionary that can be passed to the constructor method for $\texttt{PerfForesightConsumerType}$, with the values from the table here.

```python
PerfForesightDict = {
    # Parameters actually used in the solution method
    "CRRA": 2.0,  # Coefficient of relative risk aversion
    "Rfree": 1.03,  # Interest factor on assets
    "DiscFac": 0.96,  # Default intertemporal discount factor
    "LivPrb": [0.98],  # Survival probability
    "PermGroFac": [1.01],  # Permanent income growth factor
    "BoroCnstArt": None,  # Artificial borrowing constraint
    "aXtraCount": 200,  # Maximum number of gridpoints in consumption function
    # Parameters that characterize the nature of time
    "T_cycle": 1,  # Number of periods in the cycle for this agent type
    "cycles": 0,  # Number of times the cycle occurs (0 --> infinitely repeated)
}
```

## Solving and examining the solution of the perfect foresight model

With the dictionary we have just defined, we can create an instance of $\texttt{PerfForesightConsumerType}$ by passing the dictionary to the class (as if the class were a function).  This instance can then be solved by invoking its $\texttt{solve}$ method.

```python
PFexample = PerfForesightConsumerType(**PerfForesightDict)
PFexample.cycles = 0
PFexample.solve()
```

The $\texttt{solve}$ method fills in the instance's attribute $\texttt{solution}$ as a time-varying list of solutions to each period of the consumer's problem.  In this case, $\texttt{solution}$ will be a list with exactly one instance of the class $\texttt{ConsumerSolution}$, representing the solution to the infinite horizon model we specified.

```python
print(PFexample.solution)
```

    [<HARK.ConsumptionSaving.ConsIndShockModel.ConsumerSolution object at 0x000001ECC95BD700>]

Each element of $\texttt{solution}$ has a few attributes. To see all of them, we can use the \texttt{vars} built in function:

the consumption functions reside in the attribute $\texttt{cFunc}$ of each element of $\texttt{ConsumerType.solution}$.  This method creates a (time varying) attribute $\texttt{cFunc}$ that contains a list of consumption functions.

```python
print(vars(PFexample.solution[0]))
```

    {'cFunc': <HARK.interpolation.LinearInterp object at 0x000001ECC95BD970>, 'vFunc': <HARK.ConsumptionSaving.ConsIndShockModel.ValueFunc object at 0x000001ECC95BDEE0>, 'vPfunc': <HARK.ConsumptionSaving.ConsIndShockModel.MargValueFunc object at 0x000001ECC95BD7C0>, 'vPPfunc': <HARK.utilities.NullFunc object at 0x000001ECC95BD370>, 'mNrmMin': -50.49994992551661, 'hNrm': 50.49994992551661, 'MPCmin': 0.04428139169919579, 'MPCmax': 0.04428139169919579, 'mNrmSS': -50.49991081984768}

The two most important attributes of a single period solution of this model are the (normalized) consumption function $\texttt{cFunc}$ and the (normalized) value function $\texttt{vFunc}$.  Let's plot those functions near the lower bound of the permissible state space (the attribute $\texttt{mNrmMin}$ tells us the lower bound of $m_t$ where the consumption function is defined).

```python
print("Linear perfect foresight consumption function:")
mMin = PFexample.solution[0].mNrmMin
plotFuncs(PFexample.solution[0].cFunc, mMin, mMin + 10.0)
```

```python
print("Perfect foresight value function:")
plotFuncs(PFexample.solution[0].vFunc, mMin + 0.1, mMin + 10.1)
```

An element of $\texttt{solution}$ also includes the (normalized) marginal value function $\texttt{vPfunc}$, and the lower and upper bounds of the marginal propensity to consume (MPC) $\texttt{MPCmin}$ and $\texttt{MPCmax}$.  Note that with a linear consumption function, the MPC is constant, so its lower and upper bound are identical.

### Liquidity constrained perfect foresight example

Without an artificial borrowing constraint, a perfect foresight consumer is free to borrow against the PDV of his entire future stream of labor income-- his "human wealth" $\texttt{hNrm}$-- and he will consume a constant proportion of his total wealth (market resources plus human wealth).  If we introduce an artificial borrowing constraint, both of these features vanish.  In the cell below, we define a parameter dictionary that prevents the consumer from borrowing *at all*, create and solve a new instance of $\texttt{PerfForesightConsumerType}$ with it, and then plot its consumption function.

```python
LiqConstrDict = copy(PerfForesightDict)
LiqConstrDict["BoroCnstArt"] = 0.0  # Set the artificial borrowing constraint to zero

LiqConstrExample = PerfForesightConsumerType(**LiqConstrDict)
LiqConstrExample.cycles = 0  # Make this type be infinite horizon
LiqConstrExample.solve()

print("Liquidity constrained perfect foresight consumption function:")
plotFuncs(LiqConstrExample.solution[0].cFunc, 0.0, 10.0)
```

At this time, the value function for a perfect foresight consumer with an artificial borrowing constraint is not computed nor included as part of its $\texttt{solution}$.

## Simulating the perfect foresight consumer model

Suppose we wanted to simulate many consumers who share the parameter values that we passed to $\texttt{PerfForesightConsumerType}$-- an *ex ante* homogeneous *type* of consumers.  To do this, our instance would have to know *how many* agents there are of this type, as well as their initial levels of assets $a_t$ and permanent income $P_t$.

### Setting simulation parameters

Let's fill in this information by passing another dictionary to $\texttt{PFexample}$ with simulation parameters.  The table below lists the parameters that an instance of $\texttt{PerfForesightConsumerType}$ needs in order to successfully simulate its model using the $\texttt{simulate}$ method.

| Description | Code | Example value |
| :---: | --- | --- |
| Number of consumers of this type | $\texttt{AgentCount}$ | $10000$ |
| Number of periods to simulate | $\texttt{T_sim}$ | $120$ |
| Mean of initial log (normalized) assets | $\texttt{aNrmInitMean}$ | $-6.0$ |
| Stdev of initial log  (normalized) assets | $\texttt{aNrmInitStd}$ | $1.0$ |
| Mean of initial log permanent income | $\texttt{pLvlInitMean}$ | $0.0$ |
| Stdev of initial log permanent income | $\texttt{pLvlInitStd}$ | $0.0$ |
| Aggregrate productivity growth factor | $\texttt{PermGroFacAgg}$ | $1.0$ |
| Age after which consumers are automatically killed | $\texttt{T_age}$ | $None$ |

We have specified the model so that initial assets and permanent income are both distributed lognormally, with mean and standard deviation of the underlying normal distributions provided by the user.

The parameter $\texttt{PermGroFacAgg}$ exists for compatibility with more advanced models that employ aggregate productivity shocks; it can simply be set to 1.

In infinite horizon models, it might be useful to prevent agents from living extraordinarily long lives through a fortuitous sequence of mortality shocks.  We have thus provided the option of setting $\texttt{T_age}$ to specify the maximum number of periods that a consumer can live before they are automatically killed (and replaced with a new consumer with initial state drawn from the specified distributions).  This can be turned off by setting it to $\texttt{None}$.

The cell below puts these parameters into a dictionary, then gives them to $\texttt{PFexample}$.  Note that all of these parameters *could* have been passed as part of the original dictionary; we omitted them above for simplicity.

```python
SimulationParams = {
    "AgentCount": 10000,  # Number of agents of this type
    "T_sim": 120,  # Number of periods to simulate
    "aNrmInitMean": -6.0,  # Mean of log initial assets
    "aNrmInitStd": 1.0,  # Standard deviation of log initial assets
    "pLvlInitMean": 0.0,  # Mean of log initial permanent income
    "pLvlInitStd": 0.0,  # Standard deviation of log initial permanent income
    "PermGroFacAgg": 1.0,  # Aggregate permanent income growth factor
    "T_age": None,  # Age after which simulated agents are automatically killed
}

# This implicitly uses the assignParameters method of AgentType
PFexample(**SimulationParams)
```

To generate simulated data, we need to specify which variables we want to track the "history" of for this instance.  To do so, we set the $\texttt{track_vars}$ attribute of our $\texttt{PerfForesightConsumerType}$ instance to be a list of strings with the simulation variables we want to track.

In this model, valid elments of $\texttt{track_vars}$ include $\texttt{mNrmNow}$, $\texttt{cNrmNow}$, $\texttt{aNrmNow}$, and $\texttt{pLvlNow}$.  Because this model has no idiosyncratic shocks, our simulated data will be quite boring.

### Generating simulated data

Before simulating, the $\texttt{initializeSim}$ method must be invoked.  This resets our instance back to its initial state, drawing a set of initial $\texttt{aNrmNow}$ and $\texttt{pLvlNow}$ values from the specified distributions and storing them in the attributes $\texttt{aNrmNow_init}$ and $\texttt{pLvlNow_init}$.  It also resets this instance's internal random number generator, so that the same initial states will be set every time $\texttt{initializeSim}$ is called.  In models with non-trivial shocks, this also ensures that the same sequence of shocks will be generated on every simulation run.

Finally, the $\texttt{simulate}$ method can be called.

```python
PFexample.track_vars = ["mNrmNow"]
PFexample.initializeSim()
PFexample.simulate()
```

Each simulation variable $\texttt{X}$ named in $\texttt{track_vars}$ will have the *history* of that variable for each agent stored in the attribute $\texttt{X_hist}$ as an array of shape $(\texttt{T_sim},\texttt{AgentCount})$.  To see that the simulation worked as intended, we can plot the mean of $m_t$ in each simulated period:

```python
plt.plot(np.mean(PFexample.history["mNrmNow"], axis=1))
plt.xlabel("Time")
plt.ylabel("Mean normalized market resources")
plt.show()
```

A perfect foresight consumer can borrow against the PDV of his future income-- his human wealth-- and thus as time goes on, our simulated agents approach the (very negative) steady state level of $m_t$ while being steadily replaced with consumers with roughly $m_t=1$.

The slight wiggles in the plotted curve are due to consumers randomly dying and being replaced; their replacement will have an initial state drawn from the distributions specified by the user.  To see the current distribution of ages, we can look at the attribute $\texttt{t_age}$.

```python
N = PFexample.AgentCount
F = np.linspace(0.0, 1.0, N)
plt.plot(np.sort(PFexample.t_age), F)
plt.xlabel("Current age of consumers")
plt.ylabel("Cumulative distribution")
plt.show()
```

The distribution is (discretely) exponential, with a point mass at 120 with consumers who have survived since the beginning of the simulation.

One might wonder why HARK requires users to call $\texttt{initializeSim}$ before calling $\texttt{simulate}$: Why doesn't $\texttt{simulate}$ just call $\texttt{initializeSim}$ as its first step?  We have broken up these two steps so that users can simulate some number of periods, change something in the environment, and then resume the simulation.

When called with no argument, $\texttt{simulate}$ will simulate the model for $\texttt{T_sim}$ periods.  The user can optionally pass an integer specifying the number of periods to simulate (which should not exceed $\texttt{T_sim}$).

In the cell below, we simulate our perfect foresight consumers for 80 periods, then seize a bunch of their assets (dragging their wealth even more negative), then simulate for the reamining 40 periods.

```python
PFexample.initializeSim()
PFexample.simulate(80)
PFexample.aNrmNow += -5.0  # Adjust all simulated consumers' assets downward by 5
PFexample.simulate(40)

plt.plot(np.mean(PFexample.history["mNrmNow"], axis=1))
plt.xlabel("Time")
plt.ylabel("Mean normalized market resources")
plt.show()
```


```python
%matplotlib inline
from sympy import *
init_printing(use_latex=True)
import mpmath
```

```python
P,do,dc,q,mdot,Id,mu,mu0,eps,a,M,Lo,pi = symbols('P,d_o,d_c,q,\dot{m},I_d,\mu,\mu0,\epsilon,a,M,L_o,\pi',real=True,positive=True)
```

```python
a1,b1,c1,d1,e1,f1 = symbols('a1,b1,c1,d1,e1,f1',real=True)
```

```python
PI1 = (P/(mu0*Id**2/(pi**2*do**2)))
PI2 = (do/dc)
PI3 = (do/Lo)
PI4 = ((mdot*q/(M*Id))**2 * (M*do/(mu0*q**2)))
PI5 = (mdot*a/(mu0*Id**2)) * 4*pi
PI6 = (q*eps/(do**2*Lo))*1/(mu0*Id**2/do**2) * (Lo/do) * 4*pi
PI7 = (mdot/(pi*(do/2)**2)*do/mu)

```

```python
#e1 = 1.44712412817975577539
#e2 = -0.11091843726897565592
#e3 = -1.02474104747362382284
#e4 = 2.25572338431441021811
#e5 = -0.35553883591005885911
#e6 = 0.22139953816031118783

### LEAST SQUARES METHOD
#e1 = 0.79190862621816449
#e2 = 0.23399892594775878
#e3 = -0.22954590062564012
#e4 = 0.73078608503690878
#e5 = 0.29823417580302758
#e6= 0.41202315150140373
#Cfit_l = 13255363.148496266

### LEAST SQUARES METHOD
Cfit_l = 10**7.0639102021599545
e1 = 0.78967753163551446
e2 = 0.22577194077376095
e3 = -0.26766913155894489
e4 = 0.81622790719254079
e5 = 0.25441804916181782
e6 = 0.40591950688859579

prod = (mu0*Id**2.0/(pi**2.0*do**2.0))*PI2**e1*PI3**e2*PI4**e3*PI5**e4*PI6**e5*PI7**e6
```

```python
import cathode.constants as cs
import cathode.models.flow as cmf
import numpy as np
```

```python
gam,kb,Tg,Mamu,C_fit = symbols('\gamma,k_B,Tg,M_a,C_f',real=True,positive=True)
prod_noa = C_fit*prod.subs(a,sqrt(gam*kb/M*Tg))
#prod_noa = C_fit*prod.subs(a,sqrt(gam*cs.gas_constant*1e3/(M/cs.atomic_mass)*Tg))
prod_noa.subs([(mu0,4*np.pi*10**(-7)),(q,cs.e),(pi,np.pi),(gam,5/3),(kb,cs.Boltzmann),(C_fit,Cfit_l)])
```

```python
prod_SI = prod_noa.subs([(mu0,4*np.pi*10**(-7)),(q,cs.e),(pi,np.pi),(gam,5/3),(kb,cs.Boltzmann),(C_fit,Cfit_l)])
```

```python
Locm,docm,dccm,mdsccm = symbols('L_{ocm},d_{ocm},d_{ccm},\dot{m}_s')
prod_Torr = prod_SI.subs([(mdot,mdsccm*0.07174496*M/cs.e),(M,Mamu*cs.atomic_mass),(do,docm*1e-2),(dc,dccm*1e-2),(Lo,Locm*1e-2)])*760/101325
```

```python
prod_Torr.subs([(Id,10),(Tg,4443.663),(mdsccm,10),(Locm,0.074),(Mamu,131.293),(eps,12.1298),(mu,1.6538e-4),(dccm,0.38),(docm,0.104)])
```

```python
prod_Torr = prod_SI.subs([(mdot,mdsccm*0.07174496*M/cs.e),(M,Mamu*cs.atomic_mass),(do,docm*1e-2),(dc,dccm*1e-2),(Lo,Locm*1e-2)])*760/101325
prod_Torr
```

```python
mu_n = cmf.viscosity(3000,'Xe',units='Pa-s')
prod_Torr.subs([(Id,22),(Tg,3000),(mdsccm,5.5),(Locm,0.074),(Mamu,131.293),(eps,12.128),(mu,mu_n),(dccm,1.27),(docm,0.25)])
```

```python
mu_n
```

```python
mdot_sccm = 5.5
M_n = 131.293*cs.atomic_mass
mdot_n = mdot_sccm *M_n/cs.e * cs.sccm2eqA
prod_SI.subs([(Id,22),(Tg,3000),(mdot,mdot_n),(Lo,0.74e-3),(M,M_n),(eps,12.128),(mu,mu_n),(dc,1.27e-2),(do,2.5e-3)])*760/101325
```

```python
prod_SI
```

```python
TgK = 4000
mu_n = cmf.viscosity(TgK,'Xe',units='Pa-s')
Idvec = np.arange(8.0,27.0,1)
resvec = np.zeros_like(Idvec)
for idx,Idv in enumerate(Idvec):
    resvec[idx] = prod_Torr.subs([(Id,Idv),(Tg,TgK),(mdsccm,5.5),(Locm,0.074),(Mamu,131.293),(eps,12.128),(mu,mu_n),(dccm,1.27),(docm,0.25)])
    print(Idv,resvec[idx])
```

```python
TgK = 4000
mu_n = cmf.viscosity(TgK,'Xe',units='Pa-s')
mdotvec = np.arange(4.0,11.0,1)
resvec = np.zeros_like(mdotvec)
for idx,mdot in enumerate(mdotvec):
    resvec[idx] = prod_Torr.subs([(Id,22),(Tg,TgK),(mdsccm,mdot),(Locm,0.074),(Mamu,131.293),(eps,12.128),(mu,mu_n),(dccm,1.27),(docm,0.25)])
    print(mdot,resvec[idx])
```

```python
mu_n = cmf.viscosity(4000,'Xe',units='Pa-s')
mdotvec = np.arange(4.0,12.0,1)
resvec = np.zeros_like(mdotvec)
for idx,mdot in enumerate(mdotvec):
    resvec[idx] = prod_Torr.subs([(Id,22),(Tg,4000),(mdsccm,mdot),(Locm,0.074),(Mamu,131.293),(eps,12.128),(mu,mu_n),(dccm,1.27),(docm,0.25)])
    print(mdot,resvec[idx])
```

```python
Cfit_l*1e-7
```

```python
# NSTAR at 4000 K
TgK = 4000
P_outlet = 0.0
mu_n = cmf.viscosity(TgK,'Xe',units='Pa-s')
Lo = 0.74 # mm
do = 1.02 # mm
dc = 3.8 # mm
Lc = 25.4 # mm
mdotvec = np.array([2.47,2.47,2.81,3.7])
for mdot in mdotvec:
    Po = cmf.poiseuille_flow(Lo*1e-3, do*1e-3, mdot, TgK, P_outlet, species='Xe')
    Pu = cmf.poiseuille_flow(Lc*1e-3, dc*1e-3, mdot, TgK, Po, species='Xe')
    Pud = cmf.modified_poiseuille_flow(Lo*1e-3,do*1e-3,mdot,TgK)
    print(Pu,Pud)

```

```python
# PLHC @ 2000 K
TgK = 3000
mu_n = cmf.viscosity(TgK,'Ar',units='Pa-s')
Idvec = np.arange(100,301,25,dtype=np.float64)
resvec = np.zeros_like(Idvec)
for idx,Idv in enumerate(Idvec):
    resvec[idx] = prod_Torr.subs([(Id,Idv),(Tg,TgK),(mdsccm,109),(Locm,0.15),(Mamu,39.948),(eps,15.7596),(mu,mu_n),(dccm,2.715),(docm,0.56)])
    print(Idv,resvec[idx])

Lo = 1.5e-3
do = 5.6e-3
mdot = 109
Ppois = cmf.poiseuille_flow(Lo, do, mdot, TgK, 0, species='Ar')
Pdom = cmf.modified_poiseuille_flow(Lo,do,mdot,TgK)
print(Ppois,Pdom)
```

    100.0 1.12661022928
    125.0 1.2301575237
    150.0 1.32178801031
    175.0 1.40456533679
    200.0 1.48044903534
    225.0 1.55077905277
    250.0 1.61651782661
    275.0 1.67838324273
    300.0 1.73692705244
    1.19679963595 2.6726162526

```python
mu_n
```

```python

```


# Corrugated plate

## Init symbols for *sympy*

```python
from sympy import *
from sympy.vector import CoordSys3D
N = CoordSys3D('N')
x1, x2, x3 = symbols("x_1 x_2 x_3")
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3")
L, ga, gv, c = symbols("L g_a g_v c")
init_printing()
```

## Coordinates

```python
x = alpha1
y = alpha2
a1=gv *2*pi/L
z = ga * cos(c * alpha1)

r = x*N.i + y*N.j + z*N.k
```

### Mid-surface coordinates is defined with the following vector $\vec{r}=\vec{r}(\alpha_1, \alpha_2)$

```python
r
```

#### Tangent to curve

```python
r1 = r.diff(alpha1)
r2 = r.diff(alpha2)
k1 = trigsimp(r1.magnitude())
k2 = trigsimp(r2.magnitude())
r1 = r1/k1
r2 = r2/k2
```

```python
r1
```

```python
r2
```

#### Normal to curve

```python
n = r1.cross(r2)
n = trigsimp(n.normalize())
n
```

#### Curvature

```python
dr1=r1.diff(alpha1)
k1 = trigsimp(r1.cross(dr1).magnitude()/k1**3)
k1
```

```python
dr2=r2.diff(alpha2)
k2 = trigsimp(r2.cross(dr2).magnitude()/k2**3)
k2
```

#### Derivative of base vectors
Let's find
$\frac { d\vec{n} } { d\alpha_1}$
$\frac { d\vec{v} } { d\alpha_1}$
$\frac { d\vec{n} } { d\alpha_2}$
$\frac { d\vec{v} } { d\alpha_2}$

```python
n.diff(alpha1)
```

$ \frac { d\vec{n} } { d\alpha_1} = -\frac {1}{R} \vec{v} = -k \vec{v} $

```python
r1.diff(alpha1)
```

$ \frac { d\vec{v} } { d\alpha_1} = \frac {1}{R} \vec{n} = k \vec{n} $

#### Derivative of vectors

$ \vec{u} = u_v \vec{v} + u_n\vec{n} $

$ \frac { d\vec{u} } { d\alpha_1} =  \frac { d(u_v\vec{v}) } { d\alpha_1} + \frac { d(u_n\vec{n}) } { d\alpha_1} =
 \frac { du_n } { d\alpha_1} \vec{n} + u_n \frac { d\vec{n} } { d\alpha_1} + \frac { du_v } { d\alpha_1} \vec{v} + u_v \frac { d\vec{v} } { d\alpha_1} = \frac { du_n } { d\alpha_1} \vec{n} - u_n k \vec{v} + \frac { du_v } { d\alpha_1} \vec{v} + u_v k \vec{n}$

Then
$ \frac { d\vec{u} } { d\alpha_1} = \left( \frac { du_v } { d\alpha_1} - u_n k \right) \vec{v} + \left( \frac { du_n } { d\alpha_1} + u_v k \right) \vec{n}$

$ \frac { d\vec{u} } { d\alpha_2} = \frac { d(u_n\vec{n}) } { d\alpha_2} + \frac { d(u_v\vec{v}) } { d\alpha_2} =
 \frac { du_n } { d\alpha_2} \vec{n} + u_n \frac { d\vec{n} } { d\alpha_2} + \frac { du_v } { d\alpha_2} \vec{v} + u_v \frac { d\vec{v} } { d\alpha_2} = \frac { du_n } { d\alpha_2} \vec{n} + \frac { du_v } { d\alpha_2} \vec{v} $

### Base Vectors $\vec{R}_1, \vec{R}_2, \vec{R}_3$

```python
R_alpha=r+alpha3*n
R_alpha
```

```python
R1=R_alpha.diff(alpha1)
R2=R_alpha.diff(alpha2)
R3=R_alpha.diff(alpha3)
```

```python
R1
```

```python
R1x = R1.dot(N.i)
```

```python
simplify(R1x)
```

```python
R1z = R1.dot(N.k)
simplify(R1z)
```

```python
R2
```

```python
R3
```

#### Base Vectors $\vec{R}^1, \vec{R}^2, \vec{R}^3$

```python
eps=trigsimp(R1.dot(R2.cross(R3)))
R_1=simplify(trigsimp(R2.cross(R3)/eps))
R_2=simplify(trigsimp(R3.cross(R1)/eps))
R_3=simplify(trigsimp(R1.cross(R2)/eps))
```

```python
R_1
```

```python
R_2
```

```python
R_3
```

#### Jacobi matrix:

$ A = \left(
\begin{array}{ccc}
\frac{\partial x_1}{\partial \alpha_1} & \frac{\partial x_1}{\partial \alpha_2} & \frac{\partial x_1}{\partial \alpha_3} \\
\frac{\partial x_2}{\partial \alpha_1} & \frac{\partial x_2}{\partial \alpha_2} & \frac{\partial x_3}{\partial \alpha_3} \\
\frac{\partial x_3}{\partial \alpha_1} & \frac{\partial x_3}{\partial \alpha_2} & \frac{\partial x_3}{\partial \alpha_3} \\
\end{array}
\right)$

$ \left[
\begin{array}{ccc}
\vec{R}_1 & \vec{R}_2 & \vec{R}_3
\end{array}
\right] = \left[
\begin{array}{ccc}
\vec{e}_1 & \vec{e}_2 & \vec{e}_3
\end{array}
\right] \cdot  \left(
\begin{array}{ccc}
\frac{\partial x_1}{\partial \alpha_1} & \frac{\partial x_1}{\partial \alpha_2} & \frac{\partial x_1}{\partial \alpha_3} \\
\frac{\partial x_2}{\partial \alpha_1} & \frac{\partial x_2}{\partial \alpha_2} & \frac{\partial x_3}{\partial \alpha_3} \\
\frac{\partial x_3}{\partial \alpha_1} & \frac{\partial x_3}{\partial \alpha_2} & \frac{\partial x_3}{\partial \alpha_3} \\
\end{array}
\right) = \left[
\begin{array}{ccc}
\vec{e}_1 & \vec{e}_2 & \vec{e}_3
\end{array}
\right] \cdot A$

$ \left[
\begin{array}{ccc}
\vec{e}_1 & \vec{e}_2 & \vec{e}_3
\end{array}
\right] =\left[
\begin{array}{ccc}
\vec{R}_1 & \vec{R}_2 & \vec{R}_3
\end{array}
\right] \cdot A^{-1}$

```python
dx1da1=R1.dot(N.i)
dx1da2=R2.dot(N.i)
dx1da3=R3.dot(N.i)

dx2da1=R1.dot(N.j)
dx2da2=R2.dot(N.j)
dx2da3=R3.dot(N.j)

dx3da1=R1.dot(N.k)
dx3da2=R2.dot(N.k)
dx3da3=R3.dot(N.k)

A=Matrix([[dx1da1, dx1da2, dx1da3], [dx2da1, dx2da2, dx2da3], [dx3da1, dx3da2, dx3da3]])
simplify(A)
```

```python
A_inv = trigsimp(A**-1)
simplify(trigsimp(A_inv))
```

```python
trigsimp(A.det())
```

### Metric tensor

${\displaystyle \hat{G}=\sum_{i,j} g^{ij}\vec{R}_i\vec{R}_j}$

```python
g11=R1.dot(R1)
g12=R1.dot(R2)
g13=R1.dot(R3)

g21=R2.dot(R1)
g22=R2.dot(R2)
g23=R2.dot(R3)

g31=R3.dot(R1)
g32=R3.dot(R2)
g33=R3.dot(R3)

G=Matrix([[g11, g12, g13],[g21, g22, g23], [g31, g32, g33]])
G=trigsimp(G)
G
```

```python
simplify(G)
```

${\displaystyle \hat{G}=\sum_{i,j} g_{ij}\vec{R}^i\vec{R}^j}$

```python
g_11=R_1.dot(R_1)
g_12=R_1.dot(R_2)
g_13=R_1.dot(R_3)

g_21=R_2.dot(R_1)
g_22=R_2.dot(R_2)
g_23=R_2.dot(R_3)

g_31=R_3.dot(R_1)
g_32=R_3.dot(R_2)
g_33=R_3.dot(R_3)

G_con=Matrix([[g_11, g_12, g_13],[g_21, g_22, g_23], [g_31, g_32, g_33]])
G_con=trigsimp(G_con)
G_con
```

```python
simplify(G_con)
```

```python
G_inv = G**-1
G_inv
```

### Derivatives of vectors

#### Derivative of base vectors

```python
dR1dalpha1 = trigsimp(R1.diff(alpha1))
dR1dalpha1
```

$ \frac { d\vec{R_1} } { d\alpha_1} = -\frac {1}{R} \left( 1+\frac{\alpha_3}{R} \right) \vec{R_3} $

```python
dR1dalpha2 = trigsimp(R1.diff(alpha2))
dR1dalpha2
```

```python
dR1dalpha3 = trigsimp(R1.diff(alpha3))
dR1dalpha3
```

$ \frac { d\vec{R_1} } { d\alpha_3} = \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}  \vec{R_1} $

```python
dR2dalpha1 = trigsimp(R2.diff(alpha1))
dR2dalpha1
```

```python
dR2dalpha2 = trigsimp(R2.diff(alpha2))
dR2dalpha2
```

```python
dR2dalpha3 = trigsimp(R2.diff(alpha3))
dR2dalpha3
```

```python
dR3dalpha1 = trigsimp(R3.diff(alpha1))
dR3dalpha1
```

$ \frac { d\vec{R_3} } { d\alpha_1} = \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}  \vec{R_1} $

```python
dR3dalpha2 = trigsimp(R3.diff(alpha2))
dR3dalpha2
```

```python
dR3dalpha3 = trigsimp(R3.diff(alpha3))
dR3dalpha3
```

$ \frac { d\vec{R_3} } { d\alpha_3} = \vec{0} $

#### Derivative of vectors

$ \vec{u} = u^1 \vec{R_1} + u^2\vec{R_2} + u^3\vec{R_3} $

$ \frac { d\vec{u} } { d\alpha_1} =  \frac { d(u^1\vec{R_1}) } { d\alpha_1} + \frac { d(u^2\vec{R_2}) } { d\alpha_1}+ \frac { d(u^3\vec{R_3}) } { d\alpha_1} = \frac { du^1 } { d\alpha_1} \vec{R_1} + u^1 \frac { d\vec{R_1} } { d\alpha_1} + \frac { du^2 } { d\alpha_1} \vec{R_2} + u^2 \frac { d\vec{R_2} } { d\alpha_1} + \frac { du^3 } { d\alpha_1} \vec{R_3} + u^3 \frac { d\vec{R_3} } { d\alpha_1} = \frac { du^1 } { d\alpha_1} \vec{R_1} - u^1 \frac {1}{R} \left( 1+\frac{\alpha_3}{R} \right) \vec{R_3} + \frac { du^2 } { d\alpha_1} \vec{R_2}+ \frac { du^3 } { d\alpha_1} \vec{R_3} + u^3 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}  \vec{R_1}$

Then
$ \frac { d\vec{u} } { d\alpha_1} = \left( \frac { du^1 } { d\alpha_1} + u^3 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}  \right) \vec{R_1} + \frac { du^2 } { d\alpha_1} \vec{R_2} + \left( \frac { du^3 } { d\alpha_1} - u^1 \frac {1}{R} \left( 1+\frac{\alpha_3}{R} \right) \right) \vec{R_3}$

$ \frac { d\vec{u} } { d\alpha_2} =  \frac { d(u^1\vec{R_1}) } { d\alpha_2} + \frac { d(u^2\vec{R_2}) } { d\alpha_2}+ \frac { d(u^3\vec{R_3}) } { d\alpha_2} =  \frac { du^1 } { d\alpha_2} \vec{R_1} + \frac { du^2 } { d\alpha_2} \vec{R_2} + \frac { du^3 } { d\alpha_2} \vec{R_3} $

Then
$ \frac { d\vec{u} } { d\alpha_2} = \frac { du^1 } { d\alpha_2} \vec{R_1} + \frac { du^2 } { d\alpha_2} \vec{R_2} + \frac { du^3 } { d\alpha_2} \vec{R_3} $

$ \frac { d\vec{u} } { d\alpha_3} = \frac { d(u^1\vec{R_1}) } { d\alpha_3} + \frac { d(u^2\vec{R_2}) } { d\alpha_3}+ \frac { d(u^3\vec{R_3}) } { d\alpha_3} =
 \frac { du^1 } { d\alpha_3} \vec{R_1} + u^1 \frac { d\vec{R_1} } { d\alpha_3} + \frac { du^2 } { d\alpha_3} \vec{R_2} + u^2 \frac { d\vec{R_2} } { d\alpha_3} + \frac { du^3 } { d\alpha_3} \vec{R_3} + u^3 \frac { d\vec{R_3} } { d\alpha_3} = \frac { du^1 } { d\alpha_3} \vec{R_1} + u^1 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}  \vec{R_1} + \frac { du^2 } { d\alpha_3} \vec{R_2}+ \frac { du^3 } { d\alpha_3} \vec{R_3} $

 Then
$ \frac { d\vec{u} } { d\alpha_3} = \left( \frac { du^1 } { d\alpha_3}  + u^1 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}} \right) \vec{R_1} + \frac { du^2 } { d\alpha_3} \vec{R_2}+ \frac { du^3 } { d\alpha_3} \vec{R_3}$

### Gradient of vector

$\nabla_1 u^1 = \frac { \partial u^1 } { \partial \alpha_1} + u^3 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}$

$\nabla_1 u^2 = \frac { \partial u^2 } { \partial \alpha_1} $

$\nabla_1 u^3 = \frac { \partial u^3 } { \partial \alpha_1} - u^1 \frac {1}{R} \left( 1+\frac{\alpha_3}{R} \right) $

$\nabla_2 u^1 = \frac { \partial u^1 } { \partial \alpha_2}$

$\nabla_2 u^2 = \frac { \partial u^2 } { \partial \alpha_2}$

$\nabla_2 u^3 = \frac { \partial u^3 } { \partial \alpha_2}$

$\nabla_3 u^1 = \frac { \partial u^1 } { \partial \alpha_3} + u^1 \frac {1}{R} \frac {1}{1+\frac{\alpha_3}{R}}$

$\nabla_3 u^2 = \frac { \partial u^2 } { \partial \alpha_3} $

$\nabla_3 u^3 = \frac { \partial u^3 } { \partial \alpha_3}$

$ \nabla \vec{u} = \left(
\begin{array}{ccc}
\nabla_1 u^1 & \nabla_1 u^2 & \nabla_1 u^3 \\
\nabla_2 u^1 & \nabla_2 u^2 & \nabla_2 u^3 \\
\nabla_3 u^1 & \nabla_3 u^2 & \nabla_3 u^3 \\
\end{array}
\right)$

```python
u1=Function('u^1')
u2=Function('u^2')
u3=Function('u^3')
q=Function('q') # q(alpha3) = 1+alpha3/R
K = Symbol('K') # K = 1/R

u1_nabla1 = u1(alpha1, alpha2, alpha3).diff(alpha1) + u3(alpha1, alpha2, alpha3) * K / q(alpha3)
u2_nabla1 = u2(alpha1, alpha2, alpha3).diff(alpha1)
u3_nabla1 = u3(alpha1, alpha2, alpha3).diff(alpha1) - u1(alpha1, alpha2, alpha3) * K * q(alpha3)

u1_nabla2 = u1(alpha1, alpha2, alpha3).diff(alpha2)
u2_nabla2 = u2(alpha1, alpha2, alpha3).diff(alpha2)
u3_nabla2 = u3(alpha1, alpha2, alpha3).diff(alpha2)

u1_nabla3 = u1(alpha1, alpha2, alpha3).diff(alpha3) + u1(alpha1, alpha2, alpha3) * K / q(alpha3)
u2_nabla3 = u2(alpha1, alpha2, alpha3).diff(alpha3)
u3_nabla3 = u3(alpha1, alpha2, alpha3).diff(alpha3)
# $\nabla_2 u^2 = \frac { \partial u^2 } { \partial \alpha_2}$

grad_u = Matrix([[u1_nabla1, u2_nabla1, u3_nabla1],[u1_nabla2, u2_nabla2, u3_nabla2], [u1_nabla3, u2_nabla3, u3_nabla3]])
grad_u
```

```python
G_s = Matrix([[q(alpha3)**2, 0, 0],[0, 1, 0], [0, 0, 1]])
grad_u_down=grad_u*G_s
expand(simplify(grad_u_down))
```

$
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)
=
\left(
\begin{array}{c}
\left( 1+\frac{\alpha_2}{R} \right)^2  \frac { \partial u^1 } { \partial \alpha_1} + u^3 \frac {\left( 1+\frac{\alpha_3}{R} \right)}{R} \\
\left( 1+\frac{\alpha_2}{R} \right)^2 \frac { \partial u^1 } { \partial \alpha_2} \\
\left( 1+\frac{\alpha_3}{R} \right)^2  \frac { \partial u^1 } { \partial \alpha_3} + u^1 \frac {\left( 1+\frac{\alpha_3}{R} \right)}{R} \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
\frac { \partial u^3 } { \partial \alpha_1} - u^1 \frac {\left( 1+\frac{\alpha_3}{R} \right)}{R} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right)
$

$
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)
=
B \cdot
\left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right)
$

```python
B = zeros(9, 12)
B[0,1] = (1+alpha3/R)**2
B[0,8] = (1+alpha3/R)/R

B[1,2] = (1+alpha3/R)**2

B[2,0] = (1+alpha3/R)/R
B[2,3] = (1+alpha3/R)**2

B[3,5] = S(1)
B[4,6] = S(1)
B[5,7] = S(1)

B[6,9] = S(1)
B[6,0] = -(1+alpha3/R)/R
B[7,10] = S(1)
B[8,11] = S(1)

B
```

### Deformations tensor

```python
E=zeros(6,9)
E[0,0]=1
E[1,4]=1
E[2,8]=1
E[3,1]=1
E[3,3]=1
E[4,2]=1
E[4,6]=1
E[5,5]=1
E[5,7]=1
E

```

```python
Q=E*B
Q=simplify(Q)
Q
```

### Tymoshenko theory

$u^1 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u\left( \alpha_1 \right)+\alpha_3\gamma \left( \alpha_1 \right) $

$u^2 \left( \alpha_1, \alpha_2, \alpha_3 \right)=0 $

$u^3 \left( \alpha_1, \alpha_2, \alpha_3 \right)=w\left( \alpha_1 \right) $

$ \left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right) = T \cdot
\left(
\begin{array}{c}
u \\
\frac { \partial u } { \partial \alpha_1} \\
\gamma \\
\frac { \partial \gamma } { \partial \alpha_1} \\
w \\
\frac { \partial w } { \partial \alpha_1} \\
\end{array}
\right) $

```python
T=zeros(12,6)
T[0,0]=1
T[0,2]=alpha3
T[1,1]=1
T[1,3]=alpha3
T[3,2]=1

T[8,4]=1
T[9,5]=1
T
```

```python
Q=E*B*T
Q=simplify(Q)
Q
```

### Elasticity tensor(stiffness tensor)

#### General form

```python
from sympy import MutableDenseNDimArray
C_x = MutableDenseNDimArray.zeros(3, 3, 3, 3)

for i in range(3):
    for j in range(3):
        for k in range(3):
            for l in range(3):
                elem_index = 'C^{{{}{}{}{}}}'.format(i+1, j+1, k+1, l+1)
                el = Symbol(elem_index)
                C_x[i,j,k,l] = el

C_x
```

#### Include symmetry

```python
C_x_symmetry = MutableDenseNDimArray.zeros(3, 3, 3, 3)

def getCIndecies(index):
    if (index == 0):
        return 0, 0
    elif (index == 1):
        return 1, 1
    elif (index == 2):
        return 2, 2
    elif (index == 3):
        return 0, 1
    elif (index == 4):
        return 0, 2
    elif (index == 5):
        return 1, 2

for s in range(6):
    for t in range(s, 6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        elem_index = 'C^{{{}{}{}{}}}'.format(i+1, j+1, k+1, l+1)
        el = Symbol(elem_index)
        C_x_symmetry[i,j,k,l] = el
        C_x_symmetry[i,j,l,k] = el
        C_x_symmetry[j,i,k,l] = el
        C_x_symmetry[j,i,l,k] = el
        C_x_symmetry[k,l,i,j] = el
        C_x_symmetry[k,l,j,i] = el
        C_x_symmetry[l,k,i,j] = el
        C_x_symmetry[l,k,j,i] = el

C_x_symmetry
```

#### Isotropic material

```python
C_isotropic = MutableDenseNDimArray.zeros(3, 3, 3, 3)

C_isotropic_matrix = zeros(6)

mu = Symbol('mu')
la = Symbol('lambda')

for s in range(6):
    for t in range(s, 6):
        if (s < 3 and t < 3):
            if(t != s):
                C_isotropic_matrix[s,t] = la
                C_isotropic_matrix[t,s] = la
            else:
                C_isotropic_matrix[s,t] = 2*mu+la
                C_isotropic_matrix[t,s] = 2*mu+la
        elif (s == t):
            C_isotropic_matrix[s,t] = mu
            C_isotropic_matrix[t,s] = mu

for s in range(6):
    for t in range(s, 6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        el = C_isotropic_matrix[s, t]
        C_isotropic[i,j,k,l] = el
        C_isotropic[i,j,l,k] = el
        C_isotropic[j,i,k,l] = el
        C_isotropic[j,i,l,k] = el
        C_isotropic[k,l,i,j] = el
        C_isotropic[k,l,j,i] = el
        C_isotropic[l,k,i,j] = el
        C_isotropic[l,k,j,i] = el

C_isotropic
```

```python
def getCalpha(C, A, q, p, s, t):
    res = S(0)
    for i in range(3):
        for j in range(3):
            for k in range(3):
                for l in range(3):
                    res += C[i,j,k,l]*A[q,i]*A[p,j]*A[s,k]*A[t,l]
    return simplify(trigsimp(res))

C_isotropic_alpha = MutableDenseNDimArray.zeros(3, 3, 3, 3)

for i in range(3):
    for j in range(3):
        for k in range(3):
            for l in range(3):
                c = getCalpha(C_isotropic, A_inv, i, j, k, l)
                C_isotropic_alpha[i,j,k,l] = c

C_isotropic_alpha[0,0,0,0]
```

```python
C_isotropic_matrix_alpha = zeros(6)

for s in range(6):
    for t in range(6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        C_isotropic_matrix_alpha[s,t] = C_isotropic_alpha[i,j,k,l]

C_isotropic_matrix_alpha
```

#### Orthotropic material

```python
C_orthotropic = MutableDenseNDimArray.zeros(3, 3, 3, 3)

C_orthotropic_matrix = zeros(6)

for s in range(6):
    for t in range(s, 6):
        elem_index = 'C^{{{}{}}}'.format(s+1, t+1)
        el = Symbol(elem_index)
        if ((s < 3 and t < 3) or t == s):
            C_orthotropic_matrix[s,t] = el
            C_orthotropic_matrix[t,s] = el

for s in range(6):
    for t in range(s, 6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        el = C_orthotropic_matrix[s, t]
        C_orthotropic[i,j,k,l] = el
        C_orthotropic[i,j,l,k] = el
        C_orthotropic[j,i,k,l] = el
        C_orthotropic[j,i,l,k] = el
        C_orthotropic[k,l,i,j] = el
        C_orthotropic[k,l,j,i] = el
        C_orthotropic[l,k,i,j] = el
        C_orthotropic[l,k,j,i] = el

C_orthotropic
```

#### Orthotropic material in shell coordinates

```python
def getCalpha(C, A, q, p, s, t):
    res = S(0)
    for i in range(3):
        for j in range(3):
            for k in range(3):
                for l in range(3):
                    res += C[i,j,k,l]*A[q,i]*A[p,j]*A[s,k]*A[t,l]
    return simplify(trigsimp(res))

C_orthotropic_alpha = MutableDenseNDimArray.zeros(3, 3, 3, 3)

for i in range(3):
    for j in range(3):
        for k in range(3):
            for l in range(3):
                c = getCalpha(C_orthotropic, A_inv, i, j, k, l)
                C_orthotropic_alpha[i,j,k,l] = c

C_orthotropic_alpha[0,0,0,0]
```

```python
C_orthotropic_matrix_alpha = zeros(6)

for s in range(6):
    for t in range(6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        C_orthotropic_matrix_alpha[s,t] = C_orthotropic_alpha[i,j,k,l]

C_orthotropic_matrix_alpha
```

### Physical coordinates

$u^1=\frac{u_{[1]}}{1+\frac{\alpha_3}{R}}$

$\frac{\partial u^1} {\partial \alpha_3}=\frac{1}{1+\frac{\alpha_3}{R}} \frac{\partial u_{[1]}} {\partial \alpha_3} + u_{[1]} \frac{\partial} {\partial \alpha_3} \left( \frac{1}{1+\frac{\alpha_3}{R}} \right) = =\frac{1}{1+\frac{\alpha_3}{R}} \frac{\partial u_{[1]}} {\partial \alpha_3} - u_{[1]} \frac{1}{R \left( 1+\frac{\alpha_3}{R} \right)^2} $

```python
P=eye(12,12)
P[0,0]=1/(1+alpha3/R)
P[1,1]=1/(1+alpha3/R)
P[2,2]=1/(1+alpha3/R)
P[3,0]=-1/(R*(1+alpha3/R)**2)
P[3,3]=1/(1+alpha3/R)
P
```

```python
Def=simplify(E*B*P)
Def
```

```python
rows, cols = Def.shape
D_p=zeros(rows, cols)
q = 1+alpha3/R
for i in range(rows):
    ratio = 1
    if (i==0):
        ratio = q*q
    elif (i==3 or i == 4):
        ratio = q

    for j in range(cols):
        D_p[i,j] = Def[i,j] / ratio

D_p = simplify(D_p)
D_p
```

#### Stiffness tensor

```python
C_isotropic_alpha_p = MutableDenseNDimArray.zeros(3, 3, 3, 3)
q=1+alpha3/R
for i in range(3):
    for j in range(3):
        for k in range(3):
            for l in range(3):
                fact = 1
                if (i==0):
                    fact = fact*q
                if (j==0):
                    fact = fact*q
                if (k==0):
                    fact = fact*q
                if (l==0):
                    fact = fact*q
                C_isotropic_alpha_p[i,j,k,l] = simplify(C_isotropic_alpha[i,j,k,l]*fact)

C_isotropic_matrix_alpha_p = zeros(6)

for s in range(6):
    for t in range(6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        C_isotropic_matrix_alpha_p[s,t] = C_isotropic_alpha_p[i,j,k,l]

C_isotropic_matrix_alpha_p
```

```python
C_orthotropic_alpha_p = MutableDenseNDimArray.zeros(3, 3, 3, 3)
q=1+alpha3/R
for i in range(3):
    for j in range(3):
        for k in range(3):
            for l in range(3):
                fact = 1
                if (i==0):
                    fact = fact*q
                if (j==0):
                    fact = fact*q
                if (k==0):
                    fact = fact*q
                if (l==0):
                    fact = fact*q
                C_orthotropic_alpha_p[i,j,k,l] = simplify(C_orthotropic_alpha[i,j,k,l]*fact)

C_orthotropic_matrix_alpha_p = zeros(6)

for s in range(6):
    for t in range(6):
        i,j = getCIndecies(s)
        k,l = getCIndecies(t)
        C_orthotropic_matrix_alpha_p[s,t] = C_orthotropic_alpha_p[i,j,k,l]

C_orthotropic_matrix_alpha_p
```

#### Tymoshenko

```python
D_p_T = D_p*T
K = Symbol('K')
D_p_T = D_p_T.subs(R, 1/K)
simplify(D_p_T)
```

## Square of segment

$A=\frac {\theta}{2} \left( R + h_2 \right)^2-\frac {\theta}{2} \left( R + h_1 \right)^2$

```python
theta, h1, h2=symbols('theta h_1 h_2')
square_geom=theta/2*(R+h2)**2-theta/2*(R+h1)**2
expand(simplify(square_geom))
```

${\displaystyle A=\int_{0}^{L}\int_{h_1}^{h_2} \left( 1+\frac{\alpha_3}{R} \right) d \alpha_1 d \alpha_3}, L=R \theta$

```python
square_int=integrate(integrate(1+alpha3/R, (alpha3, h1, h2)), (alpha1, 0, theta*R))
expand(simplify(square_int))
```

## Virtual work

### Isotropic material physical coordinates

```python
simplify(D_p.T*C_isotropic_matrix_alpha_p*D_p)
```

### Isotropic material physical coordinates - Tymoshenko

```python
W = simplify(D_p_T.T*C_isotropic_matrix_alpha_p*D_p_T*(1+alpha3*K)**2)
W
```

```python
h=Symbol('h')
E=Symbol('E')
v=Symbol('nu')
W_a3 = integrate(W, (alpha3, -h/2, h/2))
W_a3 = simplify(W_a3)
W_a3.subs(la, E*v/((1+v)*(1-2*v))).subs(mu, E/((1+v)*2))
```

```python
A_M = zeros(3)
A_M[0,0] = E*h/(1-v**2)
A_M[1,1] = 5*E*h/(12*(1+v))
A_M[2,2] = E*h**3/(12*(1-v**2))
Q_M = zeros(3,6)
Q_M[0,1] = 1
Q_M[0,4] = K
Q_M[1,0] = -K
Q_M[1,2] = 1
Q_M[1,5] = 1
Q_M[2,3] = 1

W_M=Q_M.T*A_M*Q_M
W_M
```


```python
import numpy as np
import pandas as pd
import scipy as sp
import math
import time
import sklearn as sl
import seaborn as sns; sns.set()
import matplotlib as mpl
from matplotlib import pyplot as plt
from sympy import integrate, init_printing
from sympy.abc import x
from mpl_toolkits.mplot3d import axes3d
from matplotlib import cm

%matplotlib inline
```

# Tarea 4

Con base a los métodos vistos en clase resuelva las siguientes dos preguntas

## (A) Integrales

* $\int_{0}^{1}x^{-1/2}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.01
x0 = 1
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return (x)**(-1/2)
```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
Integral=((vectF.T*V_i*I)[0,0])
```

```python
Integral=round(Integral)
```

```python
print ("El valor de la integral es",Integral)
```

    El valor de la integral es 2

-Metdodo del Trapecio

```python
def f(x):
    return (x)**(-1/2)

a=0.001
b=1
n=100

x=np.linspace(a,b,n)
dx=(b-a)/n
```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
    a=a+dx
r=round(I)
print ("El valor de la integral es",r)
```

    El valor de la integral es 2

* $\int_{0}^{\infty}e^{-x}\ln{x}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.1
x0 = 1
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return np.exp(-x)*np.log(x)
```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
print ("El valor de la integral es", (vectF.T*V_i*I)[0,0])
```

    El valor de la integral es -0.4800999190068662

-Metodo del trapecio

```python
def f(x):
    return np.exp(-x)*np.log(x)

a=0.1
b=1/2.5
n=10

x=np.linspace(a,b,n)
dx=(b-a)/n
```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
print ("El valor de la integral es",I)

```

    El valor de la integral es -0.5812460606926427

* $\int_{0}^{\infty}\frac{\sin{x}}{x}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.000000001
x0 = 10-2
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return np.sin(x)/x

```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
print ("El valor de la integral es", (vectF.T*V_i*I)[0,0])
```

    El valor de la integral es 1.574368090025942

-Metodo del trapecio

```python
def f(x):
    return np.sin(x)/x

a=0.00001
b=1.5
n=101

x=np.linspace(a,b,n)
dx=(b-a)/n

```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
print ("El valor de la integral es",I)

```

    El valor de la integral es 1.499962392876672

## (B) Fourier

Calcule la transformada rápida de Fourier para la función de la **Tarea 3 (D)** en el intervalo $[0,4]$ ($k$ máximo $2\pi n/L$ para $n=25$). Ajuste la transformada de Fourier para los datos de la **Tarea 3** usando el método de regresión exacto de la **Tarea 3 (C)** y compare con el anterior resultado. Para ambos ejercicios haga una interpolación y grafique para comparar.

La funcion es:
$$y=\frac{a}{\left[(x-b)^2+c\right]^\gamma}.$$
Halle $a$, $b$, $c$ y $\gamma$.

En la tarea 3 se encontraron los valores de $a$, $b$, $c$ y $\gamma$.
Los cuales son:
* $a$ =0.68740987
* $b$ =1.39239372
* $c$ =0.49566194
* $\gamma$ =0.93666128

```python
df = pd.read_pickle('ex1.gz')
sns.scatterplot(x='x',y='y',data=df)
plt.show()
df
```

```python
a=0.68740987
b=1.39239372
c=0.49566194
g=0.93666128
```

```python
X=df["x"]
y=df["y"]
def f(x):
    return (a)/((x-b)**2 + c)**g

x=f(X)
```

```python

```

```python
plt.plot(x,y)
```

```python
Nf = 25
a = np.min(x)
b = np.max(x)
def a_j(j):
    global a, b, x, y
    # integra la funcion entre 0 y 10
    L = b - a
    k_j = 2*np.pi*j/4
    new_y = y*np.cos(k_j*x)/L
    if j > 0:
        new_y = new_y * 2
    return sp.integrate.simpson(new_y, x)
def b_j(j):
    global a, b, x, y
    # integra la funcion entre 0 y 10
    L = b - a
    k_j = 2*np.pi*j/4
    new_y = y*np.sin(k_j*x)/L
    if j > 0:
        new_y = new_y * 2
    return sp.integrate.simpson(new_y, x)
J = np.array(range(Nf))
A_j = np.array([a_j(j) for j in range(Nf)])
B_j = np.array([b_j(j) for j in range(Nf)])
```

```python
x_tilde = np.linspace(a,b-a,20000)
k_j = np.array([2*np.pi*j/4 for j in range(Nf)])
y_tilde = np.sum([
    A_j[j] * np.cos(k_j[j]*x_tilde) + B_j[j] * np.sin(k_j[j]*x_tilde) for j in range(Nf)
], axis=0)
```

```python
plt.plot(x,y)
plt.plot(x_tilde, y_tilde)
```

```python

```


# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Dimensionality Reduction: Geometric view of data

__Content creators:__ Alex Cayco Gajic, John Murray

__Content reviewers:__ Roozbeh Farhoudi, Matt Krause, Spiros Chavlis, Richard Gao, Michael Waskom

---
# Tutorial Objectives

In this notebook we'll explore how multivariate data can be represented in different orthonormal bases. This will help us build intuition that will be helpful in understanding PCA in the following tutorial.

Overview:
 - Generate correlated multivariate data.
 - Define an arbitrary orthonormal basis.
 - Project the data onto the new basis.

```python
# @title Video 1: Geometric view of data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="THu9yHnpq9I", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=THu9yHnpq9I

---
# Setup

```python
# Import
import numpy as np
import matplotlib.pyplot as plt
```

```python
# @title Figure Settings
import ipywidgets as widgets  # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian.

  Note that samples are sorted in ascending order for the first random variable

  Args:
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with each
                              column corresponding to a different random
                              variable
  """

  mean = np.array([0, 0])
  X = np.random.multivariate_normal(mean, cov_matrix, size=1000)
  indices_for_sorting = np.argsort(X[:, 0])
  X = X[indices_for_sorting, :]

  return X

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats) :   Data matrix each column corresponds to a
                                  different random variable

  Returns:
    Nothing.
  """

  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(X[:, 0], color='k')
  plt.ylabel('Neuron 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(X[:, 0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(X[:, 1], color='k')
  plt.xlabel('Sample Number')
  plt.ylabel('Neuron 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(X[:, 1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:, 0], X[:, 1], '.', markerfacecolor=[.5, .5, .5],
           markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:, 0], X[:, 1])[0, 1]))
  plt.show()

def plot_basis_vectors(X, W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats) :   Data matrix each column corresponds to a
                                  different random variable
    W (numpy array of floats) :   Square matrix representing new orthonormal
                                  basis each column represents a basis vector

  Returns:
    Nothing.
  """

  plt.figure(figsize=[4, 4])
  plt.plot(X[:, 0], X[:, 1], '.', color=[.5, .5, .5], label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0, W[0, 0]], [0, W[1, 0]], color='r', linewidth=3,
           label='Basis vector 1')
  plt.plot([0, W[0, 1]], [0, W[1, 1]], color='b', linewidth=3,
           label='Basis vector 2')
  plt.legend()
  plt.show()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases.
  Similar to plot_data but with colors corresponding to projections onto
  basis 1 (red) and basis 2 (blue). The title indicates the sample correlation
  calculated from the data.

  Note that samples are re-sorted in ascending order for the first
  random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis each column
                                 corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8, 4])
  gs = fig.add_gridspec(2, 2)
  ax1 = fig.add_subplot(gs[0, 0])
  ax1.plot(Y[:, 0], 'r')
  plt.xlabel
  plt.ylabel('Projection \n basis vector 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(Y[:, 0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1, 0])
  ax2.plot(Y[:, 1], 'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(Y[:, 1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:, 0], Y[:, 1], '.', color=[.5, .5, .5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:, 0], Y[:, 1])[0, 1]))
  plt.show()
```

---
# Section 1: Generate correlated multivariate data

```python
# @title Video 2: Multivariate data
video = YouTubeVideo(id="jcTq2PgU5Vw", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=jcTq2PgU5Vw

To gain intuition, we will first use a simple model to generate multivariate data. Specifically, we will draw random samples from a *bivariate normal distribution*. This is an extension of the one-dimensional normal distribution to two dimensions, in which each $x_i$ is marginally normal with mean $\mu_i$ and variance $\sigma_i^2$:

\begin{align}
x_i \sim \mathcal{N}(\mu_i,\sigma_i^2).
\end{align}

Additionally, the joint distribution for $x_1$ and $x_2$ has a specified correlation coefficient $\rho$. Recall that the correlation coefficient is a normalized version of the covariance, and ranges between -1 and +1:

\begin{align}
\rho = \frac{\text{cov}(x_1,x_2)}{\sqrt{\sigma_1^2 \sigma_2^2}}.
\end{align}

For simplicity, we will assume that the mean of each variable has already been subtracted, so that $\mu_i=0$. The remaining parameters can be summarized in the covariance matrix, which for two dimensions has the following form:

\begin{equation*}
{\bf \Sigma} =
\begin{pmatrix}
 \text{var}(x_1) & \text{cov}(x_1,x_2) \\
 \text{cov}(x_1,x_2) &\text{var}(x_2)
\end{pmatrix}.
\end{equation*}

In general, $\bf \Sigma$ is a symmetric matrix with the variances $\text{var}(x_i) = \sigma_i^2$ on the diagonal, and the covariances on the off-diagonal. Later, we will see that the covariance matrix plays a key role in PCA.

## Exercise 1: Draw samples from a distribution

We have provided code to draw random samples from a zero-mean bivariate normal distribution. Throughout this tutorial, we'll imagine these samples represent the activity (firing rates) of two recorded neurons on different trials. Fill in the function below to calculate the covariance matrix given the desired variances and correlation coefficient. The covariance can be found by rearranging the equation above:

\begin{align}
\text{cov}(x_1,x_2) = \rho \sqrt{\sigma_1^2 \sigma_2^2}.
\end{align}

Use these functions to generate and plot data while varying the parameters. You should get a feel for how changing the correlation coefficient affects the geometry of the simulated data.

**Steps**
* Fill in the function `calculate_cov_matrix` to calculate the desired covariance.
* Generate and plot the data for $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$. Try plotting the data for different values of the correlation coefficent: $\rho = -1, -.5, 0, .5, 1$.

```python
help(plot_data)
help(get_data)
```

    Help on function plot_data in module __main__:

    plot_data(X)
        Plots bivariate data. Includes a plot of each random variable, and a scatter
        plot of their joint activity. The title indicates the sample correlation
        calculated from the data.

        Args:
          X (numpy array of floats) :   Data matrix each column corresponds to a
                                        different random variable

        Returns:
          Nothing.

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian.

        Note that samples are sorted in ascending order for the first random variable

        Args:
          cov_matrix (numpy array of floats): desired covariance matrix

        Returns:
          (numpy array of floats) : samples from the bivariate Gaussian, with each
                                    column corresponding to a different random
                                    variable

```python
def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation
  coefficient.

  Args:
    var_1 (scalar)          : variance of the first random variable
    var_2 (scalar)          : variance of the second random variable
    corr_coef (scalar)      : correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """

  #################################################
  ## TODO for students: calculate the covariance matrix
  # Fill out function and remove
#   raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  #################################################

  # Calculate the covariance from the variances and correlation
  cov = corr_coef*np.sqrt(var_1*var)

  cov_matrix = np.array([[var_1, cov], [cov, var_2]])

  return cov_matrix

###################################################################
## TO DO for students: generate and plot bivariate Gaussian data with variances of 1
## and a correlation coefficients of: 0.8
## repeat while varying the correlation coefficient from -1 to 1
###################################################################
np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

# Uncomment to test your code and plot
# cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
# X = get_data(cov_matrix)
# plot_data(X)
```

```python
# to_remove solution
def calculate_cov_matrix(var_1, var_2, corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation
  coefficient.

  Args:
    var_1 (scalar)          : variance of the first random variable
    var_2 (scalar)          : variance of the second random variable
    corr_coef (scalar)      : correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """

  # Calculate the covariance from the variances and correlation
  cov = corr_coef * np.sqrt(var_1 * var_2)

  cov_matrix = np.array([[var_1, cov], [cov, var_2]])

  return cov_matrix

np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

# Uncomment to test your code and plot
cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X = get_data(cov_matrix)
with plt.xkcd():
  plot_data(X)
```

---
# Section 2: Define a new orthonormal basis

```python
# @title Video 3: Orthonormal bases
video = YouTubeVideo(id="PC1RZELnrIg", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=PC1RZELnrIg

Next, we will define a new orthonormal basis of vectors ${\bf u} = [u_1,u_2]$ and ${\bf w} = [w_1,w_2]$. As we learned in the video, two vectors are orthonormal if:

1.   They are orthogonal (i.e., their dot product is zero):
\begin{equation}
{\bf u\cdot w} = u_1 w_1 + u_2 w_2 = 0
\end{equation}
2.   They have unit length:
\begin{equation}
||{\bf u} || = ||{\bf w} || = 1
\end{equation}

In two dimensions, it is easy to make an arbitrary orthonormal basis. All we need is a random vector ${\bf u}$, which we have normalized. If we now define the second basis vector to be ${\bf w} = [-u_2,u_1]$, we can check that both conditions are satisfied:
\begin{equation}
{\bf u\cdot w} = - u_1 u_2 + u_2 u_1 = 0
\end{equation}
and
\begin{equation}
{|| {\bf w} ||} = \sqrt{(-u_2)^2 + u_1^2} = \sqrt{u_1^2 + u_2^2} = 1,
\end{equation}
where we used the fact that ${\bf u}$ is normalized. So, with an arbitrary input vector, we can define an orthonormal basis, which we will write in matrix by stacking the basis vectors horizontally:

\begin{equation}
{{\bf W} } =
\begin{pmatrix}
 u_1 & w_1 \\
 u_2 & w_2
\end{pmatrix}.
\end{equation}

## Exercise 2: Find an orthonormal basis

In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary 2-dimensional vector as an input.

**Steps**
* Modify the function `define_orthonormal_basis` to first normalize the first basis vector $\bf u$.
* Then complete the function by finding a basis vector $\bf w$ that is orthogonal to $\bf u$.
* Test the function using initial basis vector ${\bf u} = [3,1]$. Plot the resulting basis vectors on top of the data scatter plot using the function `plot_basis_vectors`. (For the data, use  $\sigma_1^2 =1$, $\sigma_2^2 =1$, and $\rho = .8$).

```python
help(plot_basis_vectors)
```

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats) :   Data matrix each column corresponds to a
                                        different random variable
          W (numpy array of floats) :   Square matrix representing new orthonormal
                                        basis each column represents a basis vector

        Returns:
          Nothing.

```python
def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2-dimensional vector used for new
                                basis

  Returns:
    (numpy array of floats)   : new orthonormal basis
                                columns correspond to basis vectors
  """

  #################################################
  ## TODO for students: calculate the orthonormal basis
  # Fill out function and remove
  raise NotImplementedError("Student excercise: implement the orthonormal basis function")
  #################################################

  # normalize vector u
  u = ...
  # calculate vector w that is orthogonal to w
  w = ...

  W = np.column_stack([u, w])

  return W

np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X = get_data(cov_matrix)
u = np.array([3, 1])

# Uncomment and run below to plot the basis vectors
# W = define_orthonormal_basis(u)
# plot_basis_vectors(X, W)
```

```python
# to_remove solution
def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats) : arbitrary 2-dimensional vector used for new
                                basis

  Returns:
    (numpy array of floats)   : new orthonormal basis
                                columns correspond to basis vectors
  """

  # normalize vector u
  u = u / np.sqrt(u[0] ** 2 + u[1] ** 2)
  # calculate vector w that is orthogonal to w
  w = np.array([-u[1], u[0]])

  W = np.column_stack((u, w))

  return W

np.random.seed(2020)  # set random seed
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

cov_matrix = calculate_cov_matrix(variance_1, variance_2, corr_coef)
X = get_data(cov_matrix)
u = np.array([3, 1])

# Uncomment and run below to plot the basis vectors
W = define_orthonormal_basis(u)
with plt.xkcd():
  plot_basis_vectors(X, W)
```

---
# Section 3: Project data onto new basis

```python
# @title Video 4: Change of basis
video = YouTubeVideo(id="Mj6BRQPKKUc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=Mj6BRQPKKUc

Finally, we will express our data in the new basis that we have just found. Since $\bf W$ is orthonormal, we can project the data into our new basis using simple matrix multiplication :

\begin{equation}
{\bf Y = X W}.
\end{equation}

We will explore the geometry of the transformed data $\bf Y$ as we vary the choice of basis.

## Exercise 3: Define an orthonormal basis
In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary vector as an input.

**Steps**
* Complete the function `change_of_basis` to project the data onto the new basis.
* Plot the projected data using the function `plot_data_new_basis`.
* What happens to the correlation coefficient in the new basis? Does it increase or decrease?
* What happens to variance?

```python
def change_of_basis(X, W):
  """
  Projects data onto new basis W.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)    : Data matrix expressed in new basis
  """

  #################################################
  ## TODO for students: project the data onto o new basis W
  # Fill out function and remove
  raise NotImplementedError("Student excercise: implement change of basis")
  #################################################

  # project data onto new basis described by W
  Y = ...

  return Y

# Unomment below to transform the data by projecting it into the new basis
# Y = change_of_basis(X, W)
# plot_data_new_basis(Y)
```

```python
# to_remove solution
def change_of_basis(X, W):
  """
  Projects data onto new basis W.

  Args:
    X (numpy array of floats) : Data matrix each column corresponding to a
                                different random variable
    W (numpy array of floats) : new orthonormal basis columns correspond to
                                basis vectors

  Returns:
    (numpy array of floats)    : Data matrix expressed in new basis
  """

  # project data onto new basis described by W
  Y = np.matmul(X, W)

  return Y

# Unomment below to transform the data by projecting it into the new basis
Y = change_of_basis(X, W)
with plt.xkcd():
  plot_data_new_basis(Y)
```

## Interactive Demo: Play with the basis vectors
To see what happens to the correlation as we change the basis vectors, run the cell below. The parameter $\theta$ controls the angle of $\bf u$ in degrees. Use the slider to rotate the basis vectors.

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!

def refresh(theta=0):
  u = [1, np.tan(theta * np.pi / 180)]
  W = define_orthonormal_basis(u)
  Y = change_of_basis(X, W)
  plot_basis_vectors(X, W)
  plot_data_new_basis(Y)

_ = widgets.interact(refresh, theta=(0, 90, 5))
```

## Questions

* What happens to the projected data as you rotate the basis?
* How does the correlation coefficient change? How does the variance of the projection onto each basis vector change?
* Are you able to find a basis in which the projected data is **uncorrelated**?

---
# Summary

- In this tutorial, we learned that multivariate data can be visualized as a cloud of points in a high-dimensional vector space. The geometry of this cloud is shaped by the covariance matrix.

- Multivariate data can be represented in a new orthonormal basis using the dot product. These new basis vectors correspond to specific mixtures of the original variables - for example, in neuroscience, they could represent different ratios of activation  across a population of neurons.

- The projected data (after transforming into the new basis) will generally have a different geometry from the original data. In particular, taking basis vectors that are aligned with the spread of cloud of points decorrelates the data.

* These concepts - covariance, projections, and orthonormal bases - are key for understanding PCA, which we be our focus in the next tutorial.


## 7.2 나이브베이즈 분류모형

### 조건부독립

확률변수 A, B가 독립이면 A, B의 결합확률은 주변확률의 곱과 같다.

$$ P(A, B) = P(A)P(B) $$

조건부독립(conditional independence)은 일반적인 독립과 달리 조건이 되는 별개의 확률변수 C가 존재해야 한다.
조건이 되는 확률변수 C에 대한 A, B의 결합조건부확률이 C에 대한 A, B의 조건부확률의 곱과 같으면 A와 B가 C에 대해 조건부독립이라고 한다.

$$ P(A, B|C) = P(A|C) P(B|C) $$

기호로는 다음과 같이 표기한다.

$$ A \text{⫫} B \;\vert\; C $$

조건부독립과 비교하여 일반적인 독립은 무조건부독립이라고 한다. 무조건부독립은 다음과 같이 표기하도 한다.

$$ A \text{⫫} B \;\vert\; \emptyset $$

A, B가 C에 대해 조건부독립이면 다음도 만족한다.

$$ P(A|B,C) = P(A|C) $$

$$ P(B|A,C) = P(B|C) $$

주의할 점은 조건부독립과 무조건부독립은 관계가 없다는 점이다. 즉, 두 확률변수가 독립이라고 항상 조건부독립이 되는 것도 아니고 조건부독립이라고 꼭 독립이 되는 것도 아니다.

$$ P(A,B) = P(A)P(B) \;\; \bcancel{\implies} \;\; P(A,B|C) = P(A|C)P(B|C) $$

$$  P(A,B|C) = P(A|C)P(B|C) \;\; \bcancel{\implies} \;\; P(A,B) = P(A)P(B) $$

예를 들어 어떤 동물의 어미의 몸무게가 $x$kg일 때 새끼의 몸무게는 $x$를 기댓값으로 하고 5kg 표준편차를 가지는 정규분포라고 가정하자. 이 동물의 새끼 중 2마리의 몸무게를 각각 A, B라고 하고 어미의 몸무게를 C라고 한다. 시물레이션을 통해 어미 표본과 각각의 어미에 대해 2마리의 새끼 표본을 만들자.

```python
np.random.seed(0)
C = np.random.normal(100, 15, 2000)
A = C + np.random.normal(0, 5, 2000)
B = C + np.random.normal(0, 5, 2000)
```

스캐터플롯을 보면 B와 C 자체는 상관관계가 있음을 알 수 있다. 하지만 어미의 몸무게가 고정되어 있으면 두 새끼의 몸무게는 서로 독립인 것을 볼 수 있다.

```python
plt.figure(figsize=(8, 4))

plt.subplot(121)
plt.scatter(A, B)
plt.xlabel("A")
plt.ylabel("B")
plt.xlim(30, 180)
plt.ylim(30, 180)
plt.title("B와 C의 무조건부 상관관계")

plt.subplot(122)
idx1 = (118 < C) & (C < 122)
idx2 = (78 < C) & (C < 82)
plt.scatter(A[idx1], B[idx1], label="C=120")
plt.scatter(A[idx2], B[idx2], label="C=80")
plt.xlabel("A")
plt.ylabel("B")
plt.xlim(30, 180)
plt.ylim(30, 180)
plt.legend()
plt.title("B와 C의 조건부 상관관계")

plt.tight_layout()
plt.show()
```

### 나이브 가정

독립변수 $x$가 $D$차원이라고 가정하자.

$$ x = (x_1, \ldots, x_D) $$

가능도함수는 $x_1, \ldots, x_D$의 결합확률이 된다.

$$ P(x \mid y = k) = P(x_1, \ldots, x_D \mid y = k) $$

원리상으로는 $y=k$인 데이터만 모아서 이 가능도함수의 모양을 추정할 수 있다. 하지만 차원 $D$가 커지면 가능도함수의 추정이 현실적으로 어려워진다.

따라서 나이즈베이즈 분류모형(Naive Bayes classification model)에서는 모든 차원의 개별 독립변수가 서로 조건부독립(conditional independent)이라는 가정을 사용한다. 이러한 가정을 나이브 가정(naive assumption)이라고 한다.

나이브 가정으로 사용하면 벡터 $x$의 결합확률분포함수는 개별 스칼라 원소 $x_d$의 확률분포함수의 곱이 된다.

$$
P(x_1, \ldots, x_D \mid y = k) = \prod_{d=1}^D P(x_d \mid y = k)
$$

스칼라 원소 $x_d$의 확률분포함수는 결합확률분포함수보다 추정하기 훨씬 쉽다.

가능도함수를 추정한 후에는 베이즈정리를 사용하여 조건부확률을 계산할 수 있다.

$$
\begin{align}
P(y = k \mid x)
&= \dfrac{ P(x_1, \ldots, x_D \mid y = k) P(y = k) }{P(x)} \\
&= \dfrac{ \left( \prod_{d=1}^D P(x_{d} \mid y = k) \right) P(y = k) }{P(x)}
\end{align}
$$

### 정규분포 가능도 모형

$x$ 벡터의 원소가 모두 실수이고 클래스마다 특정한 값 주변에서 발생한다고 하면 가능도 분포로 정규분포를 사용한다.
각 독립변수 $x_d$마다, 그리고 클래스 $k$마다 정규 분포의 기댓값 $\mu_{d,k}$, 표준 편차 $\sigma_{d,k}^2$가 달라진다. QDA 모형과는 달리 모든 독립변수들이 서로 조건부독립이라고 가정한다.

$$ P(x_d \mid y = k) = \dfrac{1}{\sqrt{2\pi\sigma_{d,k}^2}} \exp \left(-\dfrac{(x_d-\mu_{d,k})^2}{2\sigma_{d,k}^2}\right) $$

### 베르누이분포 가능도 모형

베르누이분포 가능도 모형에서는 각각의 $x = (x_1,\ldots, x_D)$의 각 원소 $x_d$가 0 또는 1이라는 값만을 가질 수 있다. 즉 독립변수는 $D$개의 독립적인 베르누이 확률변수, 즉 동전으로 구성된 동전 세트로 표현할 수 있다. 이 동전들의 모수 $\mu_{d}$는 동전 $d$마다 다르다.

그런데 클래스 $y = k$ $(k=1,\ldots,K)$마다도 $x_d$가 1이 될 확률이 다르다. 즉, 동전의 모수 $\mu_{d,k}$는 동전 $d$마다 다르고 클래스 $k$마다도 다르다. 즉, 전체 $D \times K$의 동전이 존재하며 같은 클래스에 속하는 $D$개의 동전이 하나의 동전 세트를 구성하고 이러한 동전 세트가 $K$개 있다고 생각할 수 있다.

$$ P(x_d \mid y = k) = \mu_{d,k}^{x_d} (1-\mu_{d,k})^{(1-x_d)} $$

$$
P(x_1, \ldots, x_D \mid y = k)
= \prod_{d=1}^D \mu_{d,k}^{x_d} (1-\mu_{d,k})^{(1-x_d)}
$$

이러한 동전 세트마다 확률 특성이 다르므로 베르누이분포 가능도 모형을 기반으로 하는 나이브베이즈 모형은 동전 세트를 $N$번 던진 결과로부터 $1,\ldots,K$ 중 어느 동전 세트를 던졌는지를 찾아내는 모형이라고 할 수 있다.

### 다항분포 가능도 모형

다항분포 모형에서는 $x$ 벡터가 다항분포의 표본이라고 가정한다. 즉, $D$개의 면을 가지는 주사위를 $\sum_{d=1}^D x_d$번 던져서 나온 결과로 본다.
예를 들어 $x$가 다음과 같다면,

$$ x = (1, 4, 0, 5) $$

4면체 주사위를 10번 던져서 1인 면이 1번, 2인 면이 4번, 4인 면이 5번 나온 결과로 해석한다.

각 클래스마다 주사위가 다르다고 가정하므로 $K$개의 클래스를 구분하는 문제에서는 $D$개의 면을 가진 주사위가  $K$개 있다고 본다.

$$
P(x_1, \ldots, x_D \mid y = k)
\;\; \propto \;\; \prod_{d=1}^D \mu_{d,k}^{x_{d,k}}
$$

$$ \sum_{d=1}^{D} \mu_{d,k} = 1 $$

따라서 다항분포 가능도 모형을 기반으로 하는 나이브베이즈 모형은 주사위를 던진 결과로부터  $1,\ldots,K$ 중 어느 주사위를 던졌는지를 찾아내는 모형이라고 할 수 있다.

### 사이킷런에서 제공하는 나이브베이즈 모형

사이킷런의 naive_bayes 서브패키지에서는 다음과 같은 세가지 나이브베이즈 모형 클래스를 제공한다.

* [`GaussianNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html):	정규분포 나이브베이즈
* [`BernoulliNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html): 베르누이분포 나이브베이즈
* [`MultinomialNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html): 다항분포 나이브베이즈

이 클래스들은 다양한 속성값 및 메서드를 가진다. 우선 사전 확률과 관련된 속성은 다음과 같다.

* `classes_`

 * 종속변수 Y의 클래스(라벨)

* `class_count_`

 * 종속변수 Y의 값이 특정한 클래스인 표본 데이터의 수

* `class_prior_`

 * 종속변수 Y의 무조건부 확률분포 $ P(Y) $ (정규분포의 경우에만)

* `class_log_prior_`

 * 종속변수 Y의 무조건부 확률분포의 로그 $ \log P(Y) $ (베르누이분포나 다항분포의 경우에만)

### 정규분포 나이브베이즈 모형

가우시안 나이브베이즈 모형 `GaussianNB`은 가능도 추정과 관련하여 다음과 같은 속성을 가진다.

* `theta_`: 정규분포의 기댓값 $\mu$
* `sigma_`: 정규분포의 분산 $\sigma^2$

#### 예제

실수인 두 개의 독립변수 $x_1, x_2$와 두 종류의 클래스 $y=0,1$을 가지는 분류문제가 있다.

두 독립변수의 분포는 정규분포이고 $y$의 클래스에 따라 다음처럼 모수가 달라진다.

$$
\mu_0 = \begin{bmatrix} -2 \\ -2 \end{bmatrix},
\;\;
\Sigma_0 = \begin{bmatrix} 1 & 0.9 \\ 0.9 & 2 \end{bmatrix}
$$
$$
\mu_1 = \begin{bmatrix} 2 \\ 2 \end{bmatrix},
\;\;
\Sigma_1 = \begin{bmatrix} 1.2 & -0.8 \\ -0.8 & 2 \end{bmatrix}
$$

데이터는 $y=0$인 데이터가 40개, $y=1$인 데이터가 60개 주어졌다. 이 데이터를 시각화하면 다음과 같다.

```python
np.random.seed(0)
rv0 = sp.stats.multivariate_normal([-2, -2], [[1, 0.9], [0.9, 2]])
rv1 = sp.stats.multivariate_normal([2, 2], [[1.2, -0.8], [-0.8, 2]])
X0 = rv0.rvs(40)
X1 = rv1.rvs(60)
X = np.vstack([X0, X1])
y = np.hstack([np.zeros(40), np.ones(60)])

xx1 = np.linspace(-5, 5, 100)
xx2 = np.linspace(-5, 5, 100)
XX1, XX2 = np.meshgrid(xx1, xx2)
plt.grid(False)
plt.contour(XX1, XX2, rv0.pdf(np.dstack([XX1, XX2])), cmap=mpl.cm.cool)
plt.contour(XX1, XX2, rv1.pdf(np.dstack([XX1, XX2])), cmap=mpl.cm.hot)
plt.scatter(X0[:, 0], X0[:, 1], marker="o", c='b', label="y=0")
plt.scatter(X1[:, 0], X1[:, 1], marker="s", c='r', label="y=1")
plt.legend()
plt.title("데이터의 확률분포")
plt.axis("equal")
plt.show()
```

이 데이터를 가우시안 나이브베이즈 모형으로 다음처럼 풀 수 있다.

```python
from sklearn.naive_bayes import GaussianNB
model_norm = GaussianNB().fit(X, y)
```

y 클래스의 종류와 각 클래스에 속하는 표본의 수, 그리고 그 값으로부터 구한 사전 확률

$$ p(y=0), p(y=1) $$

의 값은 다음과 같다.

```python
model_norm.classes_
```

    array([0., 1.])

```python
model_norm.class_count_
```

    array([40., 60.])

```python
model_norm.class_prior_
```

    array([0.4, 0.6])

각 클래스에 따라 $x$가 이루는 확률분포의 모수를 계산하면 다음과 같다. 나이브 가정에 따라 $x_1, x_2$는 독립이므로 상관관계를 구하지 않았다.

```python
model_norm.theta_[0], model_norm.sigma_[0]
```

    (array([-1.96197643, -2.00597903]), array([1.02398854, 2.31390497]))

```python
model_norm.theta_[1], model_norm.sigma_[1]
```

    (array([2.19130701, 2.12626716]), array([1.25429371, 1.93742544]))

이렇게 구한 데이터의 확률분포를 시각화하면 다음과 같다.

```python
rv0 = sp.stats.multivariate_normal(model_norm.theta_[0], model_norm.sigma_[0])
rv1 = sp.stats.multivariate_normal(model_norm.theta_[1], model_norm.sigma_[1])

xx1 = np.linspace(-5, 5, 100)
xx2 = np.linspace(-5, 5, 100)
XX1, XX2 = np.meshgrid(xx1, xx2)
plt.grid(False)
plt.contour(XX1, XX2, rv0.pdf(np.dstack([XX1, XX2])), cmap=mpl.cm.cool)
plt.contour(XX1, XX2, rv1.pdf(np.dstack([XX1, XX2])), cmap=mpl.cm.hot)
plt.scatter(X0[:, 0], X0[:, 1], marker="o", c='b', label="y=0")
plt.scatter(X1[:, 0], X1[:, 1], marker="s", c='r', label="y=1")

x_new = [0, 0]
plt.scatter(x_new[0], x_new[1], c="g", marker="x", s=150, linewidth=5)
plt.legend()
plt.title("나이브베이즈로 추정한 데이터의 확률분포")
plt.axis("equal")
plt.show()
```

이 모형을 사용하여 $x_{\text{new}} = (0, 0)$인 데이터의 $y$값을 예측하자. 각 클래스값이 나올 확률은 `predict_proba` 메서드로 구할 수 있다. 결과는 y=0일 확률이 0.48, y=1일 확률이 0.52이다.

```python
model_norm.predict_proba([x_new])
```

    array([[0.48475244, 0.51524756]])

이 값이 나오게 된 중간 과정을 살펴보자. 우선 추정된 독립변수의 모수와 정규 분포의 확률 밀도 함수를 사용하여 가능도를 구할 수 있다.
나이브베이즈 가정에 따라 두 입력 변수의 곱을 결합 확률로 계산한다.

$$ p(x_1, x_2 | y) \propto p(x_1) p(x_2) $$

```python
likelihood = [
    (sp.stats.norm(model_norm.theta_[0][0], np.sqrt(model_norm.sigma_[0][0])).pdf(x_new[0]) * \
     sp.stats.norm(model_norm.theta_[0][1], np.sqrt(model_norm.sigma_[0][1])).pdf(x_new[1])),
    (sp.stats.norm(model_norm.theta_[1][0], np.sqrt(model_norm.sigma_[1][0])).pdf(x_new[0]) * \
     sp.stats.norm(model_norm.theta_[1][1], np.sqrt(model_norm.sigma_[1][1])).pdf(x_new[1])),
]
likelihood
```

    [0.006615760017637298, 0.00468796559514829]

여기에 사전 확률을 곱하면 사후 확률에 비례하는 값이 나온다.

$$ p(y|x_1, x_2) \propto p(x_1, x_2|y) p(y) $$

아직 정규화 상수 $p(x)$로 나누어주지 않았으므로 두 값의 합이 1이 아니다. 즉, 확률이라고 부를 수는 없다. 하지만 크기를 비교하면 y=0일 확률이 y=1일 확률보다 훨씬 크다는 것을 알 수 있다.

```python
posterior = likelihood * model_norm.class_prior_
posterior
```

    array([0.0026463 , 0.00281278])

이 값을 정규화하면 `predict_proba` 메서드로 구한 것과 같은 값이 나온다.

```python
posterior / posterior.sum()
```

    array([0.48475244, 0.51524756])

#### 연습 문제 1

붓꽃 분류문제를 가우시안 나이브베이즈 모형을 사용하여 풀어보자.

(1) 각각의 종이 선택될 사전확률을 구하라.

(2) 각각의 종에 대해 꽃받침의 길이, 꽃받침의 폭, 꽃잎의 길이, 꽃잎의 폭의 평균과 분산을 구하라.

(3) 학습용 데이터를 사용하여 분류문제를 풀고 다음을 계산하라.

* 분류결과표
* 분류보고서

```python
from sklearn.datasets import load_iris

iris = load_iris()
X1 = iris.data
y1 = iris.target

from sklearn.naive_bayes import GaussianNB

model1 = GaussianNB().fit(X1, y1)
model1.class_prior_
```

    array([0.33333333, 0.33333333, 0.33333333])

```python
model1.theta_
```

    array([[5.006, 3.428, 1.462, 0.246],
           [5.936, 2.77 , 4.26 , 1.326],
           [6.588, 2.974, 5.552, 2.026]])

```python
model1.sigma_
```

    array([[0.121764, 0.140816, 0.029556, 0.010884],
           [0.261104, 0.0965  , 0.2164  , 0.038324],
           [0.396256, 0.101924, 0.298496, 0.073924]])

```python
y1_pred = model1.predict(X1)

from sklearn.metrics import confusion_matrix

confusion_matrix(y1, y1_pred)
```

    array([[50,  0,  0],
           [ 0, 47,  3],
           [ 0,  3, 47]])

```python
from sklearn.metrics import classification_report

print(classification_report(y1, y1_pred))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00        50
               1       0.94      0.94      0.94        50
               2       0.94      0.94      0.94        50

        accuracy                           0.96       150
       macro avg       0.96      0.96      0.96       150
    weighted avg       0.96      0.96      0.96       150

### 베르누이분포 나이브베이즈 모형

전자우편과 같은 문서 내에 특정한 단어가 포함되어 있는지의 여부는 베르누이 확률변수로 모형화할 수 있다. 이렇게 독립변수가 0 또는 1의 값을 가지면 베르누이 나이브베이즈 모형을 사용한다.

베르누이분포 나이브베이즈 모형 클래스 `BernoulliNB`는 가능도 추정과 관련하여 다음 속성을 가진다.

* `feature_count_`: 각 클래스 $k$에 대해 $d$번째 동전이 앞면이 나온 횟수 $N_{d,k}$
* `feature_log_prob_`: 베르누이분포 모수의 로그

$$ \log \mu_k = (\log \mu_{1,k}, \ldots, \log \mu_{D, k}) = \left( \log \dfrac{N_{1,k}}{N_k}, \ldots, \log \dfrac{N_{D,k}}{N_k} \right)$$

여기에서 $N_k$은 클래스 $k$에 대해 동전을 던진 총 횟수이다.
표본 데이터의 수가 적은 경우에는 모수에 대해 다음처럼 스무딩(smoothing)을 할 수도 있다.

### 스무딩

표본 데이터의 수가 적은 경우에는 베르누이 모수가 0 또는 1이라는 극단적인 모수 추정값이 나올 수도 있다. 하지만 현실적으로는 실제 모수값이 이런 극단적인 값이 나올 가능성이 적다. 따라서 베르누이 모수가 0.5인 가장 일반적인 경우를 가정하여 0이 나오는 경우와 1이 나오는 경우, 두 개의 가상 표본 데이터를 추가한다. 그러면 0이나 1과 같은 극단적인 추정값이 0.5에 가까운 다음과 값으로 변한다. 이를 라플라스 스무딩(Laplace smoothing) 또는 애드원(Add-One) 스무딩이라고 한다.

$$ \hat{\mu}_{d,k} = \frac{ N_{d,k} + \alpha}{N_k + 2 \alpha} $$

가중치 $\alpha$를 사용하여 스무딩의 정도를 조절할 수도 있다. 가중치 $\alpha$는 정수가 아니라도 괜찮다. 가중치가 1인 경우는 무정보 사전확률을 사용한 베이즈 모수추정의 결과와 같다.

```python
X = np.array([
    [0, 1, 1, 0],
    [1, 1, 1, 1],
    [1, 1, 1, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1],
    [0, 1, 1, 0],
    [0, 1, 1, 1],
    [1, 0, 1, 0],
    [1, 0, 1, 1],
    [0, 1, 1, 0]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
```

이 데이터는 4개의 키워드를 사용하여 정상 메일 4개와 스팸 메일 6개를 BOW 인코딩한 행렬이다. 예를 들어 첫번째 메일은 정상 메일이고 1번, 4번 키워드는 포함하지 않지만 2번, 3번 키워드를 포함한다고 볼 수 있다.

이 데이터를 베르누이 나이브베이즈 모형으로 예측해 보자.

```python
from sklearn.naive_bayes import BernoulliNB
model_bern = BernoulliNB().fit(X, y)
```

y 클래스의 종류와 각 클래스에 속하는 표본의 수, 그리고 그 값으로부터 구한 사전 확률의 값은 다음과 같다.

```python
model_bern.classes_
```

    array([0, 1])

```python
model_bern.class_count_
```

    array([4., 6.])

```python
np.exp(model_bern.class_log_prior_)
```

    array([0.4, 0.6])

각 클래스 $k$ 별로, 그리고 각 독립변수 $d$ 별로, 각각 다른 베르누이 확률변수라고 가정하여 모두 8개의 베르누이 확률변수의 모수를 구하면 다음과 같다.

```python
fc = model_bern.feature_count_
fc
```

    array([[2., 4., 3., 1.],
           [2., 3., 5., 3.]])

```python
fc / np.repeat(model_bern.class_count_[:, np.newaxis], 4, axis=1)
```

    array([[0.5       , 1.        , 0.75      , 0.25      ],
           [0.33333333, 0.5       , 0.83333333, 0.5       ]])

그런데 이 값은 모형 내에서 구한 값과 다르다. 모형 내에서 스무딩(smoothing)이 이루어지기 때문이다. 스무딩은 동전의 각 면 즉, 0과 1이 나오는 가상의 데이터를 추가함으로서 추정한 모수의 값이 좀 더 0.5에 가까워지도록 하는 방법이다. 이 때 사용한 스무딩 가중치 값은 다음처럼 확인할 수 있다.

```python
model_bern.alpha
```

    1.0

스무딩이 적용된 베르누이 모수값은 다음과 같다.

```python
theta = np.exp(model_bern.feature_log_prob_)
theta
```

    array([[0.5       , 0.83333333, 0.66666667, 0.33333333],
           [0.375     , 0.5       , 0.75      , 0.5       ]])

이에 모형이 완성되었으니 테스트 데이터를 사용하여 예측을 해 본다. 예를 들어 1번, 2번 키워드를 포함한 메일이 정상 메일인지 스팸 메일인지 알아보자.

```python
x_new = np.array([0, 1, 1, 1])
```

```python
model_bern.predict_proba([x_new])
```

    array([[0.34501348, 0.65498652]])

위 결과에서 정상 메일일 가능성이 약 3배임을 알 수 있다. 이 값은 다음처럼 구할 수도 있다.

```python
p = ((theta ** x_new) * (1 - theta) ** (1 - x_new)).prod(axis=1)\
    * np.exp(model_bern.class_log_prior_)
p / p.sum()
```

    array([0.34501348, 0.65498652])

반대로 3번, 4번 키워드가 포함된 메일은 스팸일 가능성이 약 90%이다.

```python
x_new = np.array([0, 0, 1, 1])
```

```python
model_bern.predict_proba([x_new])
```

    array([[0.09530901, 0.90469099]])

```python
p = ((theta ** x_new) * (1 - theta) ** (1 - x_new)).prod(axis=1)\
    * np.exp(model_bern.class_log_prior_)
p / p.sum()
```

    array([0.09530901, 0.90469099])

#### 연습 문제 2

(1) MNIST 숫자 분류문제에서 `sklearn.preprocessing.Binarizer`로 x값을 0, 1로 바꾼다(값이 8 이상이면 1, 8 미만이면 0). 즉 흰색과 검은색 픽셀로만 구성된 이미지로 만든다(다음 코드 참조)
```
from sklearn.datasets import load_digits
digits = load_digits()
X2 = digits.data
y2 = digits.target
from sklearn.preprocessing import Binarizer
X2 = Binarizer(7).fit_transform(X2)
```

이 이미지에 대해 베르누이 나이브베이즈 모형을 적용하자. 분류 결과를 분류보고서 형식으로 나타내라.

(2) `BernoulliNB` 클래스의 `binarize` 인수를 사용하여 같은 문제를 풀어본다.

(3) 계산된 모형의 모수 벡터 값을 각 클래스별로 8x8 이미지의 형태로 나타내라. 이 이미지는 무엇을 뜻하는가?

```python
from sklearn.datasets import load_digits

digits = load_digits()
X2 = digits.data
y2 = digits.target

plt.imshow(X2[0, :].reshape((8, 8)), cmap=plt.cm.binary)
plt.grid(False)
plt.show()
```

```python
from sklearn.preprocessing import Binarizer

X2 = Binarizer(7).fit_transform(X2)

plt.imshow(X2[0, :].reshape((8, 8)), cmap=plt.cm.binary)
plt.grid(False)
plt.show()
```

```python
from sklearn.naive_bayes import BernoulliNB

model = BernoulliNB().fit(X2, y2)
y2_pred = model.predict(X2)

from sklearn.metrics import classification_report

print(classification_report(y2, y2_pred))
```

                  precision    recall  f1-score   support

               0       0.99      0.97      0.98       178
               1       0.80      0.80      0.80       182
               2       0.91      0.90      0.91       177
               3       0.93      0.85      0.89       183
               4       0.96      0.94      0.95       181
               5       0.92      0.88      0.90       182
               6       0.97      0.96      0.97       181
               7       0.91      0.99      0.95       179
               8       0.80      0.82      0.81       174
               9       0.80      0.87      0.83       180

        accuracy                           0.90      1797
       macro avg       0.90      0.90      0.90      1797
    weighted avg       0.90      0.90      0.90      1797

```python
X2 = digits.data
y2 = digits.target

model = BernoulliNB(binarize=7).fit(X2, y2)
y2_pred = model.predict(X2)

from sklearn.metrics import classification_report

print(classification_report(y2, y2_pred))
```

                  precision    recall  f1-score   support

               0       0.99      0.97      0.98       178
               1       0.80      0.80      0.80       182
               2       0.91      0.90      0.91       177
               3       0.93      0.85      0.89       183
               4       0.96      0.94      0.95       181
               5       0.92      0.88      0.90       182
               6       0.97      0.96      0.97       181
               7       0.91      0.99      0.95       179
               8       0.80      0.82      0.81       174
               9       0.80      0.87      0.83       180

        accuracy                           0.90      1797
       macro avg       0.90      0.90      0.90      1797
    weighted avg       0.90      0.90      0.90      1797

```python
plt.imshow(np.exp(model.feature_log_prob_)[2].reshape((8, 8)), cmap=plt.cm.binary)
plt.grid(False)
plt.show()
```

### 다항분포 나이브베이즈 모형

다항분포 나이브베이즈 모형 클래스 `MultinomialNB`는 가능도 추정과 관련하여 다음 속성을 가진다.

* `feature_count_`: 각 클래스 $k$에서 $d$번째 면이 나온 횟수 $N_{d,k}$
* `feature_log_prob_`: 다항분포의 모수의 로그

$$ \log \mu_k = (\log \mu_{1,k}, \ldots, \log \mu_{D, k}) = \left( \log \dfrac{N_{1,k}}{N_k}, \ldots, \log \dfrac{N_{D,k}}{N_k} \right)$$

여기에서 $N_k$은 클래스 $k$에 대해 주사위를 던진 총 횟수를 뜻한다.

스무딩 공식은

$$ \hat{\mu}_{d,k} = \frac{ N_{d,k} + \alpha}{N_k + D \alpha} $$

이다.

이번에도 스팸 메일 필터링을 예로 들어보다. 다만 BOW 인코딩을 할 때, 각 키워드가 출현한 빈도를 직접 입력 변수로 사용한다.

```python
X = np.array([
    [3, 4, 1, 2],
    [3, 5, 1, 1],
    [3, 3, 0, 4],
    [3, 4, 1, 2],
    [1, 2, 1, 4],
    [0, 0, 5, 3],
    [1, 2, 4, 1],
    [1, 1, 4, 2],
    [0, 1, 2, 5],
    [2, 1, 2, 3]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
```

```python
from sklearn.naive_bayes import MultinomialNB
model_mult = MultinomialNB().fit(X, y)
```

사전 확률은 다음과 같이 구한다.

```python
model_mult.classes_
```

    array([0, 1])

```python
model_mult.class_count_
```

    array([4., 6.])

```python
np.exp(model_mult.class_log_prior_)
```

    array([0.4, 0.6])

다음으로 각 클래스에 대한 가능도 확률분포를 구한다. 다항분포 모형을 사용하므로 각 클래스틑 4개의 면을 가진 주사위로 생각할 수 있다. 그리고 각 면이 나올 확률은 각 면이 나온 횟수를 주사위를 던진 전체 횟수로 나누면 된다. 우선 각 클래스 별로 각각의 면이 나온 횟수는 다음과 같다.

```python
fc = model_mult.feature_count_
fc
```

    array([[12., 16.,  3.,  9.],
           [ 5.,  7., 18., 18.]])

이 데이터에서 클래스 Y=0인 주사위를 던진 횟수는 첫번째 행의 값의 합인 40이므로 클래스 Y=0인 주사위를 던져 1이라는 면이 나올 확률은 다음처럼 계산할 수 있다.

$$ \mu_{1,Y=0} = \dfrac{12}{40} = 0.3 $$

```python
fc / np.repeat(fc.sum(axis=1)[:, np.newaxis], 4, axis=1)
```

    array([[0.3       , 0.4       , 0.075     , 0.225     ],
           [0.10416667, 0.14583333, 0.375     , 0.375     ]])

실제로는 극단적인 추정을 피하기 위해 이 값을 가중치 1인 스무딩을 한 추정값을 사용한다.

```python
model_mult.alpha
```

    1.0

```python
(fc + model_mult.alpha) / \
    (np.repeat(fc.sum(axis=1)[:, np.newaxis],
               4, axis=1) + model_mult.alpha * X.shape[1])
```

    array([[0.29545455, 0.38636364, 0.09090909, 0.22727273],
           [0.11538462, 0.15384615, 0.36538462, 0.36538462]])

이렇게 구한 모수 추정치는 다음과 같다.

```python
theta = np.exp(model_mult.feature_log_prob_)
theta
```

    array([[0.29545455, 0.38636364, 0.09090909, 0.22727273],
           [0.11538462, 0.15384615, 0.36538462, 0.36538462]])

이제 이 값을 사용하여 예측을 해 보자. 만약 어떤 메일에 1번부터 4번까지의 키워드가 각각 10번씩 나왔다면 다음처럼 확률을 구할 수 있다. 구해진 확률로부터 이 메일이 스팸임을 알 수 있다.

```python
x_new = np.array([10, 10, 10, 10])
model_mult.predict_proba([x_new])
```

    array([[0.38848858, 0.61151142]])

다항분포의 확률질량함수을 사용하면 다음처럼 직접 확률을 구할 수도 있다.

```python
p = (theta ** x_new).prod(axis=1)*np.exp(model_bern.class_log_prior_)
p / p.sum()
```

    array([0.38848858, 0.61151142])

#### 연습 문제 3

MNIST 숫자 분류문제를 다항분포 나이브베이즈 모형을 사용하여 풀고 이진화(Binarizing)를 하여 베르누이 나이브베이즈 모형을 적용했을 경우와 성능을 비교하라.

```python
from sklearn.datasets import load_digits

digits = load_digits()
X2 = digits.data
y2 = digits.target

from sklearn.naive_bayes import GaussianNB

model = GaussianNB().fit(X2, y2)
y2_pred = model.predict(X2)

from sklearn.metrics import classification_report

print(classification_report(y2, y2_pred))
```

                  precision    recall  f1-score   support

               0       0.99      0.99      0.99       178
               1       0.83      0.85      0.84       182
               2       0.98      0.64      0.77       177
               3       0.94      0.79      0.86       183
               4       0.98      0.84      0.90       181
               5       0.91      0.93      0.92       182
               6       0.96      0.99      0.98       181
               7       0.72      0.99      0.83       179
               8       0.58      0.86      0.69       174
               9       0.94      0.71      0.81       180

        accuracy                           0.86      1797
       macro avg       0.88      0.86      0.86      1797
    weighted avg       0.89      0.86      0.86      1797

#### 연습 문제 4

텍스트 분석에서 TF-IDF 인코딩을 하면 단어의 빈도수가 정수가 아닌 실수값이 된다. 이런 경우에도  다항분포 모형을 적용할 수 있는가?

```python
X4 = np.array([
    [3.1, 4.0, 1.0, 2.0],
    [3.0, 5.0, 1.0, 1.0],
    [3.0, 3.5, 0.0, 4.0],
    [3.0, 4.0, 1.0, 2.8],
    [1.0, 2.1, 1.0, 4.0],
    [0.0, 0.0, 5.0, 3.0],
    [1.0, 2.0, 4.0, 1.0],
    [1.0, 1.0, 4.1, 2.0],
    [0.0, 1.0, 2.0, 5.0],
    [2.9, 1.0, 2.0, 3.0]])
y4 = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

from sklearn.naive_bayes import MultinomialNB

model_mult4 = MultinomialNB().fit(X4, y4)
y4_pred = model_mult4.predict(X4)

from sklearn.metrics import classification_report

print(classification_report(y4, y4_pred))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00         4
               1       1.00      1.00      1.00         6

        accuracy                           1.00        10
       macro avg       1.00      1.00      1.00        10
    weighted avg       1.00      1.00      1.00        10

### 뉴스그룹 분류

다음은 뉴스그룹 데이터에 대해 나이브베이즈 분류모형을 적용한 결과이다.

```python
from sklearn.datasets import fetch_20newsgroups

news = fetch_20newsgroups(subset="all")
X = news.data
y = news.target

from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

model1 = Pipeline([
    ('vect', CountVectorizer()),
    ('model', MultinomialNB()),
])
model2 = Pipeline([
    ('vect', TfidfVectorizer()),
    ('model', MultinomialNB()),
])
model3 = Pipeline([
    ('vect', TfidfVectorizer(stop_words="english")),
    ('model', MultinomialNB()),
])
model4 = Pipeline([
    ('vect', TfidfVectorizer(stop_words="english",
                             token_pattern=r"\b[a-z0-9_\-\.]+[a-z][a-z0-9_\-\.]+\b")),
    ('model', MultinomialNB()),
])
```

```python
%%time
from sklearn.model_selection import cross_val_score, KFold

for i, model in enumerate([model1, model2, model3, model4]):
    scores = cross_val_score(model, X, y, cv=5)
    print(("Model{0:d}: Mean score: {1:.3f}").format(i + 1, np.mean(scores)))
```

    Model1: Mean score: 0.855
    Model2: Mean score: 0.856
    Model3: Mean score: 0.883
    Model4: Mean score: 0.888
    CPU times: user 5min 52s, sys: 8.16 s, total: 6min
    Wall time: 6min 5s

#### 연습 문제 5

(1) 만약 독립변수로 실수 변수, 0 또는 1 값을 가지는 변수, 자연수 값을 가지는 변수가 섞여있다면 사이킷런에서 제공하는 나이브베이즈 클래스를 사용하여 풀 수 있는가?

(2) 사이킷런에서 제공하는 분류문제 예제 중 숲의 수종을 예측하는 covtype 분류문제는 연속확률분포 특징과 베르누이확률분포 특징이 섞여있다. 이 문제를 사이킷런에서 제공하는 나이브베이즈 클래스를 사용하여 풀어라.

```python
from sklearn.datasets import fetch_covtype

covtype = fetch_covtype()
X = covtype.data
y = covtype.target
X1 = X[:, :10]
X2 = X[:, 10:]
X1[0], X2[0]
```

    (array([2.596e+03, 5.100e+01, 3.000e+00, 2.580e+02, 0.000e+00, 5.100e+02,
            2.210e+02, 2.320e+02, 1.480e+02, 6.279e+03]),
     array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]))

```python
from sklearn.naive_bayes import GaussianNB, BernoulliNB

model1 = GaussianNB().fit(X1, y)
model2 = BernoulliNB().fit(X2, y)

prob1 = model1.predict_proba(X1)
prob2 = model2.predict_proba(X2)

likeli1 = prob1 / model1.class_prior_
likeli2 = prob2 / model1.class_prior_
prob = likeli1 * likeli2 * model1.class_prior_
y_pred = np.argmax(prob, axis=1) + 1
```

```python
confusion_matrix(y, y_pred)
```

    array([[140348,  53648,    157,      0,   2071,    881,  14735],
           [ 65031, 188638,   5849,     19,  12541,   7752,   3471],
           [     0,   2307,  23675,   3094,   1319,   5359,      0],
           [     0,      0,    551,   1941,      0,    255,      0],
           [   245,   5134,    275,      0,   3513,    326,      0],
           [     0,   1910,   6230,    621,    464,   8142,      0],
           [  6371,    183,      0,      0,     22,      0,  13934]])

```python
print(classification_report(y, y_pred))
```

                  precision    recall  f1-score   support

               1       0.66      0.66      0.66    211840
               2       0.75      0.67      0.71    283301
               3       0.64      0.66      0.65     35754
               4       0.34      0.71      0.46      2747
               5       0.18      0.37      0.24      9493
               6       0.36      0.47      0.41     17367
               7       0.43      0.68      0.53     20510

        accuracy                           0.65    581012
       macro avg       0.48      0.60      0.52    581012
    weighted avg       0.68      0.65      0.66    581012

```python
model = BernoulliNB().fit(X, y)
y_pred = model.predict(X)
print(classification_report(y, y_pred))
```

                  precision    recall  f1-score   support

               1       0.65      0.48      0.55    211840
               2       0.65      0.76      0.70    283301
               3       0.60      0.87      0.71     35754
               4       0.55      0.43      0.48      2747
               5       0.22      0.06      0.10      9493
               6       0.24      0.23      0.23     17367
               7       0.63      0.61      0.62     20510

        accuracy                           0.63    581012
       macro avg       0.51      0.49      0.49    581012
    weighted avg       0.63      0.63      0.62    581012