# Theoretical Foundations of Buffer Stock Saving

Generator: BufferStockTheory-make/notebooks_byname

For the following badges: GitHub does not allow click-through redirects; right-click to get the link, then paste into navigation bar

[](https://colab.research.google.com/github/econ-ark/REMARK/blob/master/REMARKs/BufferStockTheory/BufferStockTheory.ipynb)

[This notebook](https://github.com/econ-ark/REMARK/blob/master/REMARKs/BufferStockTheory/BufferStockTheory.ipynb) uses the [Econ-ARK/HARK](https://github.com/econ-ark/hark) toolkit to describe the main results and reproduce the figures in the paper [Theoretical Foundations of Buffer Stock Saving](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory)

If you are not familiar with the HARK toolkit, you may wish to browse the ["Gentle Introduction to HARK"](https://mybinder.org/v2/gh/econ-ark/DemARK/master?filepath=Gentle-Intro-To-HARK.ipynb) before continuing (since you are viewing this document, you presumably know a bit about [Jupyter Notebooks](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/)).

For instructions on how to install the [Econ-ARK/HARK](https://github.com/econ-ark/hark) toolkit on your computer, please refer to the [QUICK START GUIDE](https://github.com/econ-ark/HARK/blob/master/README.md).

The main HARK tool used here is $\texttt{ConsIndShockModel.py}$, in which agents have CRRA utility and face idiosyncratic shocks to permanent and transitory income.  For an introduction to this module, see the [ConsIndShockModel.ipynb](https://econ-ark.org/notebooks) notebook at the [Econ-ARK](https://econ-ark.org) website.

```python
# This cell does some setup; please be patient, it may take 3-5 minutes

# The tools for navigating the filesystem
import sys
import os

# Determine the platform so we can do things specific to each
import platform
pform = ''
pform = platform.platform().lower()
if 'darwin' in pform:
    pf = 'darwin' # MacOS
if 'debian'in pform:
    pf = 'debian' # Probably cloud (MyBinder, CoLab, ...)
if 'ubuntu'in pform:
    pf = 'debian' # Probably cloud (MyBinder, CoLab, ...)
if 'win' in pform:
    pf = 'win'

# Test whether latex is installed (some of the figures require it)
from distutils.spawn import find_executable

iflatexExists=False

if find_executable('latex'):
    iflatexExists=True

# if not iflatexExists:
#     print('Some of the figures below require a full installation of LaTeX')

#     # If running on Mac or Win, user can be assumed to be able to install
#     # any missing packages in response to error messages; but not on cloud
#     # so load LaTeX by hand (painfully slowly)
#     if 'debian' in pf: # CoLab and MyBinder are both ubuntu
#         print('Installing LaTeX now; please wait 3-5 minutes')
#         from IPython.utils import io

#         with io.capture_output() as captured: # Hide hideously long output
#             os.system('apt-get update')
#             os.system('apt-get install texlive texlive-latex-extra texlive-xetex dvipng')
#             iflatexExists=True
#     else:
#         print('Please install a full distributon of LaTeX on your computer then rerun.')
#         print('A full distribution means textlive, texlive-latex-extras, texlive-xetex, dvipng, and ghostscript')
#         sys.exit()

# This is a jupytext paired notebook that autogenerates BufferStockTheory.py
# which can be executed from a terminal command line via "ipython BufferStockTheory.py"
# But a terminal does not permit inline figures, so we need to test jupyter vs terminal
# Google "how can I check if code is executed in the ipython notebook"

from IPython import get_ipython # In case it was run from python instead of ipython

# If the ipython process contains 'terminal' assume not in a notebook
def in_ipynb():
    try:
        if 'terminal' in str(type(get_ipython())):
            return False
        else:
            return True
    except NameError:
        return False

if in_ipynb():
    # Now install stuff aside from LaTeX (if not already installed)
    os.system('pip install econ-ark==0.10.0.dev3')
    os.system('pip install matplotlib')
    os.system('pip install numpy')
    os.system('pip install scipy')
    os.system('pip install ipywidgets')
    os.system('pip install jupyter_contrib_nbextensions')
    os.system('jupyter contrib nbextension install --user')
    os.system('jupyter nbextension enable codefolding/main')
    os.system('jupyter nbextension enable latex_envs/latex_envs')
    os.system('jupyter nbextension enable navigation-hotkeys')
    os.system('pip install cite2c')
    os.system('python -m cite2c.install')
else:
    print('In batch mode')

# Import related generic python packages
import numpy as np
from time import clock
mystr = lambda number : "{:.4f}".format(number)

import matplotlib
import matplotlib.pyplot as plt
from matplotlib.pyplot import plot, draw, show

# In order to use LaTeX to manage all text layout in our figures,
# we import rc settings from matplotlib.
from matplotlib import rc

plt.rc('font', family='serif')
plt.rc('text', usetex=iflatexExists)

# The warnings package allows us to ignore some harmless but alarming warning messages
import warnings
warnings.filterwarnings("ignore")

from copy import copy, deepcopy

# Determine whether to make the figures inline (for spyder or jupyter)
# vs whatever is the automatic setting that will apply if run from the terminal
if in_ipynb():
    # %matplotlib inline generates a syntax error when run from the shell
    # so do this instead
    get_ipython().run_line_magic('matplotlib', 'inline')
else:
    get_ipython().run_line_magic('matplotlib', 'auto')

# Code to allow a master "Generator" and derived "Generated" versions
Generator=False # Is this notebook the master or is it generated?

# Define (and create, if necessary) the figures directory "Figures"
if Generator:
    my_file_path = os.path.dirname(os.path.abspath("BufferStockTheory.ipynb")) # Find pathname to this file:
    Figures_HARK_dir = os.path.join(my_file_path,"Figures/") # LaTeX document assumes figures will be here
    Figures_HARK_dir = os.path.join(my_file_path,"/tmp/Figures/") # Uncomment to make figures outside of git path
    if not os.path.exists(Figures_HARK_dir):
        os.makedirs(Figures_HARK_dir)

if not in_ipynb(): # running in batch mode
    print('You appear to be running from a terminal')
    print('By default, figures will appear one by one')
```

```python
# Import HARK tools needed

from HARK.ConsumptionSaving.ConsIndShockModel import IndShockConsumerType
from HARK.utilities import plotFuncsDer, plotFuncs
```

## [The Problem](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#The-Problem)

The paper defines and calibrates a small set of parameters:

| Parameter | Description | Code | Value |
|:---:| ---         | ---  | :---: |
| $\Gamma$ | Permanent Income Growth Factor | $\texttt{PermGroFac}$ | 1.03 |
| $\mathsf{R}$ | Interest Factor | $\texttt{Rfree}$ | 1.04 |
| $\beta$ | Time Preference Factor | $\texttt{DiscFac}$ | 0.96 |
| $\rho$ | Coeﬃcient of Relative Risk Aversion| $\texttt{CRRA}$ | 2 |
| $\wp$ | Probability of Unemployment | $\texttt{UnempPrb}$ | 0.005 |
| $\mu$ | Income when Unemployed | $\texttt{IncUnemp}$ | 0. |
| $\sigma_\psi$ | Std Dev of Log Permanent Shock| $\texttt{PermShkStd}$ | 0.1 |
| $\sigma_\theta$ | Std Dev of Log Transitory Shock| $\texttt{TranShkStd}$ | 0.1 |

For a microeconomic consumer with 'Market Resources' (net worth plus current income) $M_{t}$, end-of-period assets $A_{t}$ will be the amount remaining after consumption of $C_{t}$.
\begin{eqnarray}
A_{t}   &=&M_{t}-C_{t}
\end{eqnarray}

The consumer's permanent noncapital income $P$ grows by a predictable factor $\Gamma$ and is subject to an unpredictable lognormally distributed multiplicative shock $\mathbb{E}_{t}[\psi_{t+1}]=1$,
\begin{eqnarray}
P_{t+1} & = & P_{t} \Gamma \psi_{t+1}
\end{eqnarray}

and actual income is permanent income multiplied by a logormal multiplicative transitory shock, $\mathbb{E}_{t}[\theta_{t+1}]=1$, so that next period's market resources are
\begin{eqnarray}
%M_{t+1} &=& B_{t+1} +P_{t+1}\theta_{t+1},  \notag
M_{t+1} &=& A_{t}\mathsf{R} +P_{t+1}\theta_{t+1}.  \notag
\end{eqnarray}

When the consumer has a CRRA utility function $u(c)=\frac{c^{1-\rho}}{1-\rho}$, the paper shows that the problem can be written in terms of ratios of money variables to permanent income, e.g. $m_{t} \equiv M_{t}/P_{t}$, and the Bellman form of [the problem reduces to](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#The-Related-Problem):

\begin{eqnarray*}
v_t(m_t) &=& \max_{c_t}~~ u(c_t) + \beta~\mathbb{E}_{t} [(\Gamma\psi_{t+1})^{1-\rho} v_{t+1}(m_{t+1}) ] \\
& s.t. & \\
a_t &=& m_t - c_t \\
m_{t+1} &=& R/(\Gamma \psi_{t+1}) a_t + \theta_{t+1} \\
\end{eqnarray*}

```python
# Define a parameter dictionary with baseline parameter values

# Set the baseline parameter values
PermGroFac = 1.03
Rfree      = 1.04
DiscFac    = 0.96
CRRA       = 2.00
UnempPrb   = 0.005
IncUnemp   = 0.0
PermShkStd = 0.1
TranShkStd = 0.1
# Import default parameter values
import HARK.ConsumptionSaving.ConsumerParameters as Params

# Make a dictionary containing all parameters needed to solve the model
base_params = Params.init_idiosyncratic_shocks

# Set the parameters for the baseline results in the paper
# using the variable values defined in the cell above
base_params['PermGroFac'] = [PermGroFac]   # Permanent income growth factor
base_params['Rfree']      = Rfree          # Interest factor on assets
base_params['DiscFac']    = DiscFac        # Time Preference Factor
base_params['CRRA']       = CRRA           # Coefficient of relative risk aversion
base_params['UnempPrb']   = UnempPrb       # Probability of unemployment (e.g. Probability of Zero Income in the paper)
base_params['IncUnemp']   = IncUnemp       # Induces natural borrowing constraint
base_params['PermShkStd'] = [PermShkStd]   # Standard deviation of log permanent income shocks
base_params['TranShkStd'] = [TranShkStd]   # Standard deviation of log transitory income shocks

# Some technical settings that are not interesting for our purposes
base_params['LivPrb']       = [1.0]   # 100 percent probability of living to next period
base_params['CubicBool']    = True    # Use cubic spline interpolation
base_params['T_cycle']      = 1       # No 'seasonal' cycles
base_params['BoroCnstArt']  = None    # No artificial borrowing constraint
```

## Convergence of the Consumption Rules

Under the given parameter values, [the paper's first figure](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Convergence-of-the-Consumption-Rules) depicts the successive consumption rules that apply in the last period of life $(c_{T}(m))$, the second-to-last period, and earlier periods $(c_{T-n})$.  $c(m)$ is the consumption function to which these converge as

\[
c(m) = \lim_{n \uparrow \infty} c_{T-n}(m)
\]

```python
# Create a buffer stock consumer instance by passing the dictionary to the class.
baseEx = IndShockConsumerType(**base_params)
baseEx.cycles = 100   # Make this type have a finite horizon (Set T = 100)

baseEx.solve()        # Solve the model
baseEx.unpackcFunc()  # Make the consumption function easily accessible

```

```python
# Plot the different periods' consumption rules.

m1 = np.linspace(0,9.5,1000) # Set the plot range of m
m2 = np.linspace(0,6.5,500)
c_m  = baseEx.cFunc[0](m1)   # c_m can be used to define the limiting inﬁnite-horizon consumption rule here
c_t1 = baseEx.cFunc[-2](m1) # c_t1 defines the second-to-last period consumption rule
c_t5 = baseEx.cFunc[-6](m1) # c_t5 defines the T-5 period consumption rule
c_t10 = baseEx.cFunc[-11](m1)  # c_t10 defines the T-10 period consumption rule
c_t0 = m2                            # c_t0 defines the last period consumption rule
plt.figure(figsize = (12,9))
plt.plot(m1,c_m,color="black")
plt.plot(m1,c_t1,color="black")
plt.plot(m1,c_t5,color="black")
plt.plot(m1,c_t10,color="black")
plt.plot(m2,c_t0,color="black")
plt.xlim(0,11)
plt.ylim(0,7)
plt.text(7,6,r'$c_{T}(m) = 45$ degree line',fontsize = 22,fontweight='bold')
plt.text(9.6,5.3,r'$c_{T-1}(m)$',fontsize = 22,fontweight='bold')
plt.text(9.6,2.6,r'$c_{T-5}(m)$',fontsize = 22,fontweight='bold')
plt.text(9.6,2.1,r'$c_{T-10}(m)$',fontsize = 22,fontweight='bold')
plt.text(9.6,1.7,r'$c(m)$',fontsize = 22,fontweight='bold')
plt.arrow(6.9,6.05,-0.6,0,head_width= 0.1,width=0.001,facecolor='black',length_includes_head='True')
plt.tick_params(labelbottom=False, labelleft=False,left='off',right='off',bottom='off',top='off')
plt.text(0,7.05,"$c$",fontsize = 26)
plt.text(11.1,0,"$m$",fontsize = 26)
# Save the figures in several formats
if Generator:
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncsConverge.png'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncsConverge.jpg'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncsConverge.pdf'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncsConverge.svg'))
if not in_ipynb():
    plt.ioff()
    plt.draw()
#    plt.show(block=False)
    plt.pause(1)
else:
     plt.show(block=True) # Change to False if you want to run uninterrupted

```

## Factors and Conditions

### [The Finite Human Wealth Condition](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Human-Wealth)

Human wealth for a perfect foresight consumer is defined as the present discounted value of future income:

\begin{eqnarray}
H_{t} & = & \mathbb{E}_{t}[P_{t} + \mathsf{R}^{-1} P_{t+1} + \mathsf{R}^{2} P_{t+2} ... ] \\
      & = & P_{t} \left(1 + (\Gamma/\mathsf{R}) + (\Gamma/\mathsf{R})^{2} ... \right)
\end{eqnarray}
which is an infinite number if $\Gamma/\mathsf{R} \geq 1$.  We say that the 'Finite Human Wealth Condition' (FHWC) holds if
$0 \leq (\Gamma/\mathsf{R}) < 1$.

### [Absolute Patience and the AIC](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#AIC)

The paper defines the Absolute Patience Factor as being equal to the ratio of $C_{t+1}/C_{t}$ for a perfect foresight consumer.  The Old English character "Þ" is used for this object in the paper, but "Þ" cannot currently be rendered conveniently in Jupyter notebooks, so we will substitute $\Phi$ here:

\begin{equation}
\Phi = (\mathsf{R} \beta)^{1/\rho}
\end{equation}

If $\Phi = 1$, a perfect foresight consumer will spend exactly the amount that can be sustained perpetually (given their current and future resources).  If $\Phi < 1$ (the consumer is 'absolutely impatient'; or, 'the absolute impatience condition holds'), the consumer is consuming more than the sustainable amount, so consumption will fall, and if the consumer is 'absolutely patient' with $\Phi > 1$ consumption will grow over time.

### [Growth Patience and the GIC](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#GIC)

For a [perfect foresight consumer](http://econ.jhu.edu/people/ccarroll/public/lecturenotes/consumption/PerfForesightCRRA), whether the ratio of consumption to the permanent component of income $P$ is rising, constant, or falling depends on the relative growth rates of consumption and permanent income, which is measured by the "Perfect Foresight Growth Patience Factor":

\begin{eqnarray}
\Phi_{\Gamma} & = & \Phi/\Gamma
\end{eqnarray}
and whether the ratio is falling or rising over time depends on whether $\Phi_{\Gamma}$ is below or above 1.

An analogous condition can be defined when there is uncertainty about permanent income.  Defining $\tilde{\Gamma} = (\mathbb{E}[\psi^{-1}])^{-1}\Gamma$, the 'Growth Impatience Condition' (GIC) is that
\begin{eqnarray}
  \Phi/\tilde{\Gamma} & < & 1
\end{eqnarray}

### [The Finite Value of Autarky Condition (FVAC)](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Autarky-Value)

The paper [shows](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Autarky-Value) that a consumer who planned to spend his permanent income $\{ p_{t}, p_{t+1}, ...\} $ in every period would have value defined by

\begin{equation}
v_{t}^{\text{autarky}} = u(p_{t})\left(\frac{1}{1-\beta \Gamma^{1-\rho} \mathbb{E}[\psi^{1-\rho}]}\right)
\end{equation}

and defines the 'Finite Value of Autarky Condition' as the requirement that the denominator of this expression be a positive finite number:

\begin{equation}
\beta \Gamma^{1-\rho} \mathbb{E}[\psi^{1-\rho}] < 1
\end{equation}

### [The Weak Return Impatience Condition (WRIC)](http://www.econ2.jhu.edu/people/ccarroll/papers/BufferStockTheory/#WRIC)

The 'Return Impatience Condition' $\Phi/\mathsf{R} < 1$ has long been understood to be required for the perfect foresight model to have a nondegenerate solution (when $\rho=1$, this reduces to $\beta < R$).  If the RIC does not hold, the consumer is so patient that the optimal consumption function approaches zero as the horizon extends.

When the probability of unemployment is $\wp$, the paper articulates an analogous (but weaker) condition:

\begin{eqnarray}
 \wp^{1/\rho} \Phi/\mathsf{R} & < & 1
\end{eqnarray}

# Key Results

## [Nondegenerate Solution Requires FVAC and WRIC](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Sufficient-Conditions-For-Nondegenerate-Solution)

A main result of the paper is that the conditions required for the model to have a nondegenerate solution ($0 < c(m) < \infty$ for feasible $m$) are that the Finite Value of Autarky (FVAC) and Weak Return Impatience Condition (WRAC) hold.

## [Natural Borrowing Constraint limits to Artificial Borrowing Constraint](http://www.econ2.jhu.edu/people/ccarroll/papers/BufferStockTheory/#The-Liquidity-Constrained-Solution-as-a-Limit)

Defining $\chi(\wp)$ as the consumption function associated with any particular value of $\wp$, and defining $\hat{\chi}$ as the consumption function that would apply in the absence of the zero-income shocks but in the presence of an 'artificial' borrowing constraint requiring $a \geq 0$, a la Deaton (1991), the paper shows that

\begin{eqnarray}
\lim_{\wp \downarrow 0}~\chi(\wp) & = & \hat{\chi}
\end{eqnarray}

That is, as $\wp$ approaches zero the problem with uncertainty becomes identical to the problem that instead has constraints.  (See [Precautionary Saving and Liquidity Constraints](http://econ.jhu.edu/people/ccarroll/papers/LiqConstr) for a full treatment of the relationship between precautionary saving and liquidity constraints).

## [$c(m)$ is Finite Even When Human Wealth Is Infinite](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#When-The-GIC-Fails)

In the perfect foresight model, if $\mathsf{R} < \Gamma$ the present discounted value of future labor income is infinite and so the limiting consumption function is $c(m) = \infty$ for all $m$.  Many models have no well-defined solution in this case.

The presence of uncertainty changes this: The limiting consumption function is finite for all values of $m$.

This is because uncertainty imposes a "natural borrowing constraint" that deters the consumer from borrowing against their unbounded future labor income.

A [table](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Sufficient-Conditions-For-Nondegenerate-Solution) puts this result in the context of implications of other conditions and restrictions.

## [If the GIC Holds, $\exists$ a finite 'target' $m$](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#onetarget)

Section [There Is Exactly One Target $m$ Ratio, Which Is Stable](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#onetarget) shows that, under parameter values for which the limiting consumption function exists, if the GIC holds then there will be a value $\check{m}$ such that:

\begin{eqnarray}
\mathbb{E}[m_{t+1}] & > & m_{t}~\text{if $m_{t} < \check{m}$} \\
\mathbb{E}[m_{t+1}] & < & m_{t}~\text{if $m_{t} > \check{m}$} \\
\mathbb{E}[m_{t+1}] & = & m_{t}~\text{if $m_{t} = \check{m}$}
\end{eqnarray}

## [If the GIC Fails, Target Wealth is Infinite ](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#The-GIC)

[A figure](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#FVACnotGIC) depicts a solution when the **FVAC** (Finite Value of Autarky Condition) and **WRIC** hold (so that the model has a solution) but the **GIC** (Growth Impatience Condition) fails.  In this case the target wealth ratio is infinity.

The parameter values in this specific example are:

| Param | Description | Code | Value |
| :---: | ---         | ---  | :---: |
| $\Gamma$ | Permanent Income Growth Factor | $\texttt{PermGroFac}$ | 1.00 |
| $\mathrm{\mathsf{R}}$ | Interest Factor | $\texttt{Rfree}$ | 1.08 |

The figure is reproduced below.

```python
# Construct the "GIC fails" example.

GIC_fail_dictionary = dict(base_params)
GIC_fail_dictionary['Rfree']      = 1.08
GIC_fail_dictionary['PermGroFac'] = [1.00]

GICFailExample = IndShockConsumerType(
    cycles=0, # cycles=0 makes this an infinite horizon consumer
    **GIC_fail_dictionary)
```

    The given type violates the absolute impatience condition with the supplied parameter values; the AIF is 1.01823
    The given parameter values violate the growth impatience condition for this consumer type; the GIF is: 1.0088

The $\mathtt{IndShockConsumerType}$ tool automatically checks various parametric conditions, and will give a warning as well as the values of the factors if any conditions fail to be met.

We can also directly check the conditions, in which case results will be a little more verbose by default.

```python
# The checkConditions method does what it sounds like it would
GICFailExample.checkConditions(verbose=True)
```

    The given type violates the absolute impatience condition with the supplied parameter values; the AIF is 1.01823
        Therefore, the absolute amount of consumption is expected to grow over time
    The given parameter values violate the growth impatience condition for this consumer type; the GIF is: 1.0088
        Therefore, a target level of wealth does not exist.
    The weak return impatience factor value for the supplied parameter values satisfies the weak return impatience condition.
    The finite value of autarky factor value for the supplied parameter values satisfies the finite value of autarky condition.

    [!] For more information on the conditions, see Table 3 in "Theoretical Foundations of Buffer Stock Saving" at http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/

Next we define the function $\mathrm{\mathbb{E}}_{t}[\Delta m_{t+1}]$ that shows the ‘sustainable’ level of spending at which $m$ is expected to remain unchanged.

```python
# Calculate "Sustainable" consumption that leaves expected m unchanged
# In the perfect foresight case, this is just permanent income plus interest income
# A small adjustment is required to take account of the consequences of uncertainty
InvEpShInvAct = np.dot(GICFailExample.PermShkDstn[0][0], GICFailExample.PermShkDstn[0][1]**(-1))
InvInvEpShInvAct = (InvEpShInvAct) ** (-1)
PermGroFacAct = GICFailExample.PermGroFac[0] * InvInvEpShInvAct
ER = GICFailExample.Rfree / PermGroFacAct
Er = ER - 1
mSSfunc = lambda m : 1 + (m-1)*(Er/ER)
```

```python
# Plot GICFailExample consumption function against the sustainable level of consumption

GICFailExample.solve() # Above, we set up the problem but did not solve it
GICFailExample.unpackcFunc()  # Make the consumption function easily accessible for plotting
m = np.linspace(0,5,1000)
c_m = GICFailExample.cFunc[0](m)
E_m = mSSfunc(m)
plt.figure(figsize = (12,8))
plt.plot(m,c_m,color="black")
plt.plot(m,E_m,color="black")
plt.xlim(0,5.5)
plt.ylim(0,1.6)
plt.text(0,1.63,"$c$",fontsize = 26)
plt.text(5.55,0,"$m$",fontsize = 26)
plt.tick_params(labelbottom=False, labelleft=False,left='off',right='off',bottom='off',top='off')
plt.text(1,0.6,"$c(m_{t})$",fontsize = 18)
plt.text(1.5,1.2,"$\mathsf{E}_{t}[\Delta m_{t+1}] = 0$",fontsize = 18)
plt.arrow(0.98,0.62,-0.2,0,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(2.2,1.2,0.3,-0.05,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
if Generator:
    plt.savefig(os.path.join(Figures_HARK_dir, 'FVACnotGIC.png'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'FVACnotGIC.jpg'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'FVACnotGIC.pdf'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'FVACnotGIC.svg'))

# This figure reproduces the figure shown in the paper.
# The gap between the two functions actually increases with $m$ in the limit.
if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True) # Change to False if you want to run uninterrupted
```

As a foundation for the remaining figures, we define another instance of the class $\texttt{IndShockConsumerType}$, which has the same parameter values as the instance $\texttt{baseEx}$ defined previously but is solved to convergence (our definition of an infinite horizon agent type)

```python
# cycles=0 tells the solver to find the infinite horizon solution
baseEx_inf = IndShockConsumerType(cycles=0,**base_params)

baseEx_inf.solve()
baseEx_inf.unpackcFunc()
```

### [Target $m$, Expected Consumption Growth, and Permanent Income Growth](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#AnalysisoftheConvergedConsumptionFunction)

The next figure is shown in  [Analysis of the Converged Consumption Function](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#cGroTargetFig), which shows the expected consumption growth factor $\mathrm{\mathbb{E}}_{t}[c_{t+1}/c_{t}]$ for a consumer behaving according to the converged consumption rule.

```python
# Define a function to calculate expected consumption
def exp_consumption(a):
    '''
    Taking end-of-period assets as input, return expectation of next period's consumption
    Inputs:
       a: end-of-period assets
    Returns:
       expconsump: next period's expected consumption
    '''
    GrowFactp1 = baseEx_inf.PermGroFac[0]* baseEx_inf.PermShkDstn[0][1]
    Rnrmtp1 = baseEx_inf.Rfree / GrowFactp1
    # end-of-period assets plus normalized returns
    btp1 = Rnrmtp1*a
    # expand dims of btp1 and use broadcasted sum of a column and a row vector
    # to obtain a matrix of possible beginning-of-period assets next period
    mtp1 = np.expand_dims(btp1, axis=1) + baseEx_inf.TranShkDstn[0][1]
    part_expconsumption = GrowFactp1*baseEx_inf.cFunc[0](mtp1).T
    # finish expectation over permanent income shocks by right multiplying with
    # the weights
    part_expconsumption = np.dot(part_expconsumption, baseEx_inf.PermShkDstn[0][0])
    # finish expectation over transitory income shocks by right multiplying with
    # weights
    expconsumption = np.dot(part_expconsumption, baseEx_inf.TranShkDstn[0][0])
    # return expected consumption
    return expconsumption
```

```python
# Calculate the expected consumption growth factor
m1 = np.linspace(1,baseEx_inf.solution[0].mNrmSS,50) # m1 defines the plot range on the left of target m value (e.g. m <= target m)
c_m1 = baseEx_inf.cFunc[0](m1)
a1 = m1-c_m1
exp_consumption_l1 = []
for i in range(len(a1)):
    exp_consumption_tp1 = exp_consumption(a1[i])
    exp_consumption_l1.append(exp_consumption_tp1)

# growth1 defines the values of expected consumption growth factor when m is less than target m
growth1 = np.array(exp_consumption_l1)/c_m1

# m2 defines the plot range on the right of target m value (e.g. m >= target m)
m2 = np.linspace(baseEx_inf.solution[0].mNrmSS,1.9,50)

c_m2 = baseEx_inf.cFunc[0](m2)
a2 = m2-c_m2
exp_consumption_l2 = []
for i in range(len(a2)):
    exp_consumption_tp1 = exp_consumption(a2[i])
    exp_consumption_l2.append(exp_consumption_tp1)

# growth 2 defines the values of expected consumption growth factor when m is bigger than target m
growth2 = np.array(exp_consumption_l2)/c_m2
```

```python
# Define a function to construct the arrows on the consumption growth rate function
def arrowplot(axes, x, y, narrs=15, dspace=0.5, direc='neg',
              hl=0.01, hw=3, c='black'):
    '''
    The function is used to plot arrows given the data x and y.

    Input:
        narrs  :  Number of arrows that will be drawn along the curve

        dspace :  Shift the position of the arrows along the curve.
                  Should be between 0. and 1.

        direc  :  can be 'pos' or 'neg' to select direction of the arrows

        hl     :  length of the arrow head

        hw     :  width of the arrow head

        c      :  color of the edge and face of the arrow head
    '''

    # r is the distance spanned between pairs of points
    r = np.sqrt(np.diff(x)**2+np.diff(y)**2)
    r = np.insert(r, 0, 0.0)

    # rtot is a cumulative sum of r, it's used to save time
    rtot = np.cumsum(r)

    # based on narrs set the arrow spacing
    aspace = r.sum() / narrs

    if direc is 'neg':
        dspace = -1.*abs(dspace)
    else:
        dspace = abs(dspace)

    arrowData = [] # will hold tuples of x,y,theta for each arrow
    arrowPos = aspace*(dspace) # current point on walk along data
                                 # could set arrowPos to 0 if you want
                                 # an arrow at the beginning of the curve

    ndrawn = 0
    rcount = 1
    while arrowPos < r.sum() and ndrawn < narrs:
        x1,x2 = x[rcount-1],x[rcount]
        y1,y2 = y[rcount-1],y[rcount]
        da = arrowPos-rtot[rcount]
        theta = np.arctan2((x2-x1),(y2-y1))
        ax = np.sin(theta)*da+x1
        ay = np.cos(theta)*da+y1
        arrowData.append((ax,ay,theta))
        ndrawn += 1
        arrowPos+=aspace
        while arrowPos > rtot[rcount+1]:
            rcount+=1
            if arrowPos > rtot[-1]:
                break

    for ax,ay,theta in arrowData:
        # use aspace as a guide for size and length of things
        # scaling factors were chosen by experimenting a bit

        dx0 = np.sin(theta)*hl/2.0 + ax
        dy0 = np.cos(theta)*hl/2.0 + ay
        dx1 = -1.*np.sin(theta)*hl/2.0 + ax
        dy1 = -1.*np.cos(theta)*hl/2.0 + ay

        if direc is 'neg' :
            ax0 = dx0
            ay0 = dy0
            ax1 = dx1
            ay1 = dy1
        else:
            ax0 = dx1
            ay0 = dy1
            ax1 = dx0
            ay1 = dy0

        axes.annotate('', xy=(ax0, ay0), xycoords='data',
                xytext=(ax1, ay1), textcoords='data',
                arrowprops=dict( headwidth=hw, frac=1., ec=c, fc=c))
```

```python
# Plot consumption growth as a function of market resources
# Calculate Absolute Patience Factor Phi = lower bound of consumption growth factor
AbsPatientFac = (baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)

fig = plt.figure(figsize = (12,8))
ax = fig.add_subplot(111)
# Plot the Absolute Patience Factor line
ax.plot([0,1.9],[AbsPatientFac,AbsPatientFac],color="black")

# Plot the Permanent Income Growth Factor line
ax.plot([0,1.9],[baseEx_inf.PermGroFac[0],baseEx_inf.PermGroFac[0]],color="black")

# Plot the expected consumption growth factor on the left side of target m
ax.plot(m1,growth1,color="black")

# Plot the expected consumption growth factor on the right side of target m
ax.plot(m2,growth2,color="black")

# Plot the arrows
arrowplot(ax, m1,growth1)
arrowplot(ax, m2,growth2, direc='pos')

# Plot the target m
ax.plot([baseEx_inf.solution[0].mNrmSS,baseEx_inf.solution[0].mNrmSS],[0,1.4],color="black",linestyle="--")
ax.set_xlim(1,2.05)
ax.set_ylim(0.98,1.08)
ax.text(1,1.082,"Growth Rate",fontsize = 26,fontweight='bold')
ax.text(2.055,0.98,"$m_{t}$",fontsize = 26,fontweight='bold')
ax.text(1.9,1.01,"$\mathsf{E}_{t}[c_{t+1}/c_{t}]$",fontsize = 22,fontweight='bold')
ax.text(baseEx_inf.solution[0].mNrmSS,0.975, r'$\check{m}$', fontsize = 26,fontweight='bold')
ax.tick_params(labelbottom=False, labelleft=False,left='off',right='off',bottom='off',top='off')
ax.text(1.9,0.998,r'$\Phi = (\mathrm{\mathsf{R}}\beta)^{1/\rho}$',fontsize = 22,fontweight='bold')
ax.text(1.9,1.03, r'$\Gamma$',fontsize = 22,fontweight='bold')
if Generator:
    fig.savefig(os.path.join(Figures_HARK_dir, 'cGroTargetFig.png'))
    fig.savefig(os.path.join(Figures_HARK_dir, 'cGroTargetFig.jpg'))
    fig.savefig(os.path.join(Figures_HARK_dir, 'cGroTargetFig.pdf'))
    fig.savefig(os.path.join(Figures_HARK_dir, 'cGroTargetFig.svg'))
if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True) # Change to False if you want to run uninterrupted
```

### [Consumption Function Bounds](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#AnalysisOfTheConvergedConsumptionFunction)
[The next figure](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#cFuncBounds)
illustrates theoretical bounds for the consumption function.

We define two useful variables: lower bound of $\kappa$ (marginal propensity to consume) and limit of $h$ (Human wealth), along with some functions such as limiting perfect foresight consumption functions ($\bar{c}(m)$), $\bar{\bar c}(m)$ and $\underline{c}(m)$.

```python
# Define k_lower, h_inf and perfect foresight consumption function, upper bound of consumption function and lower
# bound of consumption function.
k_lower = 1.0-(baseEx_inf.Rfree**(-1.0))*(baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)
h_inf = (1.0/(1.0-baseEx_inf.PermGroFac[0]/baseEx_inf.Rfree))
conFunc_PF = lambda m: (h_inf -1)* k_lower + k_lower*m
conFunc_upper = lambda m: (1 - baseEx_inf.UnempPrb ** (1.0/baseEx_inf.CRRA)*(baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)/baseEx_inf.Rfree)*m
conFunc_lower = lambda m: (1 -(baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)/baseEx_inf.Rfree) * m
intersect_m = ((h_inf-1)* k_lower)/((1 - baseEx_inf.UnempPrb
            **(1.0/baseEx_inf.CRRA)*(baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)/baseEx_inf.Rfree)-k_lower)
```

```python
# Plot the consumption function and its bounds

x1 = np.linspace(0,25,1000)
x3 = np.linspace(0,intersect_m,300)
x4 = np.linspace(intersect_m,25,700)
cfunc_m = baseEx_inf.cFunc[0](x1)
cfunc_PF_1 = conFunc_PF(x3)
cfunc_PF_2 = conFunc_PF(x4)
cfunc_upper_1 = conFunc_upper(x3)
cfunc_upper_2 = conFunc_upper(x4)
cfunc_lower = conFunc_lower(x1)
plt.figure(figsize = (12,8))
plt.plot(x1,cfunc_m, color="black")
plt.plot(x1,cfunc_lower, color="black",linewidth=2.5)
plt.plot(x3,cfunc_upper_1, color="black",linewidth=2.5)
plt.plot(x4,cfunc_PF_2 , color="black",linewidth=2.5)
plt.plot(x4,cfunc_upper_2 , color="black",linestyle="--")
plt.plot(x3,cfunc_PF_1 , color="black",linestyle="--")
plt.tick_params(labelbottom=False, labelleft=False,left='off',right='off',bottom='off',top='off')
plt.xlim(0,25)
plt.ylim(0,1.12*conFunc_PF(25))
plt.text(0,1.12*conFunc_PF(25)+0.05,"$c$",fontsize = 22)
plt.text(25+0.1,0,"$m$",fontsize = 22)
plt.text(2.5,1,r'$c(m)$',fontsize = 22,fontweight='bold')
plt.text(6,5,r'$\overline{\overline{c}}(m)= \overline{\kappa}m = (1-\wp^{1/\rho}\Phi_{R})m$',fontsize = 22,fontweight='bold')
plt.text(2.2,3.8, r'$\overline{c}(m) = (m-1+h)\underline{\kappa}$',fontsize = 22,fontweight='bold')
plt.text(9,4.1,r'Upper Bound $ = $ Min $[\overline{\overline{c}}(m),\overline{c}(m)]$',fontsize = 22,fontweight='bold')
plt.text(7,0.7,r'$\underline{c}(m)= (1-\Phi_{R})m = \underline{\kappa}m$',fontsize = 22,fontweight='bold')
plt.arrow(2.45,1.05,-0.5,0.02,head_width= 0.05,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(2.15,3.88,-0.5,0.1,head_width= 0.05,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(8.95,4.15,-0.8,0.05,head_width= 0.05,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(5.95,5.05,-0.4,0,head_width= 0.05,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(14,0.70,0.5,-0.1,head_width= 0.05,width=0.001,facecolor='black',length_includes_head='True')
if Generator:
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncBounds.png'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncBounds.jpg'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncBounds.pdf'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cFuncBounds.svg'))
if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True) # Change to False if you want to run uninterrupted
```

### [The Consumption Function and Target $m$](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#cFuncBounds)

This figure shows the $\mathrm{\mathbb{E}}_{t}[\Delta m_{t+1}]$ and consumption function $c(m_{t})$, along with the intrsection of these two functions, which defines the target value of $m$

```python
# This just plots objects that have already been constructed

m1 = np.linspace(0,4,1000)
cfunc_m = baseEx_inf.cFunc[0](m1)
mSSfunc = lambda m:(baseEx_inf.PermGroFac[0]/baseEx_inf.Rfree)+(1.0-baseEx_inf.PermGroFac[0]/baseEx_inf.Rfree)*m
mss = mSSfunc(m1)
plt.figure(figsize = (12,8))
plt.plot(m1,cfunc_m, color="black")
plt.plot(m1,mss, color="black")
plt.xlim(0,3)
plt.ylim(0,1.45)
plt.plot([baseEx_inf.solution[0].mNrmSS, baseEx_inf.solution[0].mNrmSS],[0,2.5],color="black",linestyle="--")
plt.tick_params(labelbottom=False, labelleft=False,left='off',right='off',bottom='off',top='off')
plt.text(0,1.47,r"$c$",fontsize = 26)
plt.text(3.02,0,r"$m$",fontsize = 26)
plt.text(2.3,0.95,r'$\mathsf{E}[\Delta m_{t+1}] = 0$',fontsize = 22,fontweight='bold')
plt.text(2.3,1.1,r"$c(m_{t})$",fontsize = 22,fontweight='bold')
plt.text(baseEx_inf.solution[0].mNrmSS,-0.05, r"$\check{m}$",fontsize = 26)
plt.arrow(2.28,1.12,-0.1,0.03,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(2.28,0.97,-0.1,0.02,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
if Generator:
    plt.savefig(os.path.join(Figures_HARK_dir, 'cRatTargetFig.png'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cRatTargetFig.jpg'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cRatTargetFig.pdf'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'cRatTargetFig.svg'))
if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True)
```

### [Upper and Lower Limits of the Marginal Propensity to Consume](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#MPCLimits)

The paper shows that as $m_{t}~\uparrow~\infty$ the consumption function in the presence of risk gets arbitrarily close to the perfect foresight consumption function.  Defining $\underline{\kappa}$ as the perfect foresight model's MPC, this implies that $\lim_{m_{t}~\uparrow~\infty} c^{\prime}(m) = \underline{\kappa}$.

The paper also derives an analytical limit $\bar{\kappa}$ for the MPC as $m$ approaches 0., its bounding value.  Strict concavity of the consumption function implies that the consumption function will be everywhere below a function $\bar{\kappa}m$, and strictly declining everywhere.  The last figure plots the MPC between these two limits.

```python
# The last figure shows the upper and lower limits of the MPC
plt.figure(figsize = (12,8))
# Set the plot range of m
m = np.linspace(0.001,8,1000)

# Use the HARK method derivative to get the derivative of cFunc, and the values are just the MPC
MPC = baseEx_inf.cFunc[0].derivative(m)

# Define the upper bound of MPC
MPCUpper = (1 - baseEx_inf.UnempPrb ** (1.0/baseEx_inf.CRRA)*(baseEx_inf.Rfree*baseEx_inf.DiscFac)**(1.0/baseEx_inf.CRRA)/baseEx_inf.Rfree)

# Define the lower bound of MPC
MPCLower = k_lower

plt.plot(m,MPC,color = 'black')
plt.plot([0,8],[MPCUpper,MPCUpper],color = 'black')
plt.plot([0,8],[MPCLower,MPCLower],color = 'black')
plt.xlim(0,8)
plt.ylim(0,1)
plt.text(1.5,0.6,r'$\kappa(m) \equiv c^{\prime}(m)$',fontsize = 26,fontweight='bold')
plt.text(6,0.87,r'$(1-\wp^{1/\rho}\Phi_{R})\equiv \overline{\kappa}$',fontsize = 26,fontweight='bold')
plt.text(0.5,0.07,r'$\underline{\kappa}\equiv(1-\Phi_{R})$',fontsize = 26,fontweight='bold')
plt.text(8.05,0,"$m$",fontsize = 26)
plt.arrow(1.45,0.61,-0.4,0,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(1.7,0.07,0.2,-0.01,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
plt.arrow(5.95,0.875,-0.2,0.03,head_width= 0.02,width=0.001,facecolor='black',length_includes_head='True')
if Generator:
    plt.savefig(os.path.join(Figures_HARK_dir, 'MPCLimits.png'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'MPCLimits.jpg'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'MPCLimits.pdf'))
    plt.savefig(os.path.join(Figures_HARK_dir, 'MPCLimits.svg'))
if not in_ipynb():
    plt.show(block=False)
    plt.pause(1)
else:
    plt.show(block=True) # Change to False if you want to run uninterrupted
```

# Summary

[Two tables in the paper](https://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/#Sufficient-Conditions-For-Nondegenerate-Solution) summarize the various definitions, and then articulate conditions required for the problem to have a nondegenerate solution.

The main other contribution of the paper is to show that, under parametric combinations where the solution is nondegenerate, if the Growth Impatience Condition holds there will be a target level of wealth.


Lambda School Data Science

*Unit 2, Sprint 3, Module 1*

---

# Define ML problems

You will use your portfolio project dataset for all assignments this sprint.

## Assignment

Complete these tasks for your project, and document your decisions.

- [ ] Choose your target. Which column in your tabular dataset will you predict?
- [ ] Is your problem regression or classification?
- [ ] How is your target distributed?
    - Classification: How many classes? Are the classes imbalanced?
    - Regression: Is the target right-skewed? If so, you may want to log transform the target.
- [ ] Choose which observations you will use to train, validate, and test your model.
    - Are some observations outliers? Will you exclude them?
    - Will you do a random split or a time-based split?
- [ ] Choose your evaluation metric(s).
    - Classification: Is your majority class frequency > 50% and < 70% ? If so, you can just use accuracy if you want. Outside that range, accuracy could be misleading. What evaluation metric will you choose, in addition to or instead of accuracy?
- [ ] Begin to clean and explore your data.
- [ ] Begin to choose which features, if any, to exclude. Would some features "leak" future information?

```python
import pandas as pd
adoption_url = 'https://data.austintexas.gov/resource/9t4d-g238.csv?$limit=100000'
adoption = pd.read_csv(adoption_url)
# in order to see all of the columns:
pd.options.display.max_columns = 100
```

# Target

```python
adoption.head()
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      outcome_type
      outcome_subtype
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color

      0
      A808320
      Kyha
      2019-11-14T15:37:00.000
      2019-11-14T15:37:00.000
      2017-11-07T00:00:00.000
      Rto-Adopt
      NaN
      Dog
      Spayed Female
      2 years
      German Shepherd Mix
      Sable

      1
      A781697
      Pookie
      2019-11-14T15:33:00.000
      2019-11-14T15:33:00.000
      2017-10-05T00:00:00.000
      Adoption
      NaN
      Dog
      Spayed Female
      2 years
      Cairn Terrier
      White/Brown

      2
      A808382
      NaN
      2019-11-14T14:57:00.000
      2019-11-14T14:57:00.000
      2014-11-08T00:00:00.000
      Transfer
      Partner
      Cat
      Intact Male
      5 years
      Domestic Shorthair
      Orange Tabby

      3
      A806701
      *Emerald
      2019-11-14T14:41:00.000
      2019-11-14T14:41:00.000
      2019-09-02T00:00:00.000
      Adoption
      Foster
      Cat
      Neutered Male
      2 months
      Domestic Shorthair
      Blue Tabby/White

      4
      A804553
      *Wendy
      2019-11-14T14:36:00.000
      2019-11-14T14:36:00.000
      2019-08-09T00:00:00.000
      Adoption
      Foster
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico

```python
adoption.shape
```

    (100000, 12)

The target in this project will be to predict whether or not an animal will be adopted or not (transferred to another shelter or, sadly, euthanized) so that perhaps animal shelters, though overwhelmed, can give some extra love or use unique methods to get those animals that may not have the best odds forever homes.

```python
adoption['outcome_type'].value_counts(dropna=False)
```

    Adoption           44390
    Transfer           29861
    Return to Owner    17510
    Euthanasia          6294
    Died                 969
    Rto-Adopt            508
    Disposal             387
    Missing               61
    Relocate              17
    NaN                    3
    Name: outcome_type, dtype: int64

```python
# this might be a feature that can create leakage, will come back to it.
adoption['outcome_subtype'].value_counts(dropna=False)
```

    NaN                    54837
    Partner                24943
    Foster                  7953
    Rabies Risk             2789
    SCRP                    2636
    Suffering               2503
    Snr                     2279
    In Kennel                508
    Aggressive               358
    Offsite                  281
    Medical                  254
    In Foster                243
    At Vet                   171
    Behavior                  82
    Enroute                   65
    Underage                  29
    Court/Investigation       21
    In Surgery                19
    Possible Theft            15
    Field                      8
    Barn                       4
    Customer S                 1
    Prc                        1
    Name: outcome_subtype, dtype: int64

# Classification or Regression?

There are 9 classes of outcomes but for this project I'd like to focus on the animals that were adopted or not. There are a large number of animals that were returned to owners but that would just be due to them getting out etc but they do have a home so I will not include those in my project. "Rto-adopt" or return to owner adoption will also be included with "return to owner."

For animals that were adopted I will consider that to be:
-adoption
-rto-adopt (return to owner through adoption)

I will combine the following for not adopted:
-transfer
-euthanasia
-relocate
-missing (animals that went missing from the shelter--still unsuccessful in getting them homes)
"Died" and "disposal" are animals that may have died while at the shelter or were brought in that needed to be properly disposed of so I will not include these either as they may have been very ill when brought in.

# How is the target distributed?
## Are the classes imbalanced?

```python
adoption['outcome_type'].value_counts(normalize=True)
```

    Adoption           0.443913
    Transfer           0.298619
    Return to Owner    0.175105
    Euthanasia         0.062942
    Died               0.009690
    Rto-Adopt          0.005080
    Disposal           0.003870
    Missing            0.000610
    Relocate           0.000170
    Name: outcome_type, dtype: float64

will need to drop the rows where the outcome type are the ones listed above to be excluded:
-Return to owner
-Rto-adopt
-Died
-Disposal

```python
# adoption updated to drop the outcomes we are excluding:
adoption_upd = adoption[~adoption['outcome_type'].isin(['Return to Owner',
                                                        'Rto-Adopt',
                                                        'Died',
                                                        'Disposal'])]
```

```python
adoption_upd['outcome_type'].value_counts(dropna=False)
```

    Adoption      44390
    Transfer      29861
    Euthanasia     6294
    Missing          61
    Relocate         17
    NaN               3
    Name: outcome_type, dtype: int64

```python
# am going to drop all animals except cats and dogs, 2 of the unknown
# outcome types are dogs and 1 also has a lot of other missing info so
# will drop these rows.
adoption_upd = adoption_upd.dropna(subset = ['outcome_type'])

```

```python
# need to redefine classes as binary. Adoption as 'adopted' and
# the rest as 'not adopted'.
def new_status(outcome):
    if outcome == 'Transfer' or outcome == 'Euthanasia' or outcome == 'Missing' or outcome == 'Relocate':
      return 'Not adopted'
    else:
      return 'Adopted'

```

```python
adoption_upd = adoption_upd.copy()
adoption_upd['new_outcome_type'] = adoption_upd['outcome_type'].apply(new_status)
```

```python
adoption_upd['new_outcome_type'].value_counts(normalize=True)
```

    Adopted        0.550587
    Not adopted    0.449413
    Name: new_outcome_type, dtype: float64

The classes now are combined into a binary classification and the classes are not imbalanced.

```python
# drop the original 'Outcome_Type' column:
adoption_upd = adoption_upd.drop(columns='outcome_type')
```

# Choose Observations

As mentioned above, since the focus of my project is predicting if animals that are in need of forever homes will be adopted or not, I have already excluded the following observations from my model:
-Return to Owner
-Rto-Adopt
-Died
-Disposal

There are 3 missing values for the outcome ,so we can try to look at the outcome subtype to see if we can determine what happened to them but the remaining animals have been categorized into 'adopted' or 'not adopted.'

# How to Split Data:

```python
adoption_upd.dtypes
```

    animal_id           object
    name                object
    datetime            object
    monthyear           object
    date_of_birth       object
    outcome_subtype     object
    animal_type         object
    sex_upon_outcome    object
    age_upon_outcome    object
    breed               object
    color               object
    new_outcome_type    object
    dtype: object

```python
adoption_upd['datetime'] = pd.to_datetime(adoption_upd['datetime'], infer_datetime_format=True)
```

```python
adoption_upd['datetime'].dt.year.value_counts()
```

    2015    14792
    2019    14373
    2016    14119
    2017    14058
    2018    13361
    2014     9920
    Name: datetime, dtype: int64

The description of the data set said that Austin is becoming a more pet-friendly city so there may be more animals going in and out of shelters in the more recent data vs the earlier data. I will therefore split the data based on time with the most recent data being the test set and then create a test and validation set with the remaining data.

test = adoption_upd[(adoption_upd['datetime'].dt.year == 2019)]
val =  adoption_upd[(adoption_upd['datetime'].dt.year == 2018)]
train = adoption_upd[(adoption_upd['datetime'].dt.year < 2018)]

```python
# how big to make test set? 2019:
# 14292 observations
```

# Evaluation Metrics

since the classes aren't imbalanced I can use accuracy but will also explore the precision and recall for this problem.

precision positive: correctly predict all the animals that were adopted.

\begin{align}
precision = \frac{accurately \ predicted \ adopted}{total\ predicted \ adopted}
\end{align}

recall positive: of all the animals that were adopted, how many were we able to identify?
\begin{align}
recall = \frac{accurately \ predicted \ adopted}{actually \ adopted}
\end{align}

# Begin to clean data and feature selection

```python
adoption_upd.isnull().sum()
```

    animal_id               0
    name                30056
    datetime                0
    monthyear               0
    date_of_birth           0
    outcome_subtype     36351
    animal_type             0
    sex_upon_outcome        2
    age_upon_outcome       24
    breed                   0
    color                   0
    new_outcome_type        0
    dtype: int64

```python
# the name column has a lot of missing values, will change NaN's to Unknown
adoption_upd['name'].fillna("Unknown", inplace = True)
```

```python
# the outcome subtype is missing 36353 values, almost half of all of our
# data, since we will know all of the outcome types and this may cause
# leakage into the test set because certain outcomes can be deduced from
# the outcome subtype, I will drop that entire column.

adoption_upd = adoption_upd.drop(columns='outcome_subtype')
adoption_upd.shape

```

    (80623, 11)

```python
# the sex_upon_outcome column has 2 missing values but noticed an 'unknown'
# value so check to see if that's common:
adoption_upd['sex_upon_outcome'].value_counts()
# will one hot encode this column.
```

    Neutered Male    27561
    Spayed Female    26019
    Intact Female    10098
    Intact Male       9372
    Unknown           7571
    Name: sex_upon_outcome, dtype: int64

```python
# age_upon_outcome has 25 missing values but date_of_birth has none, lets
# look at how many times 'unknown' shows up in the data:
adoption_upd.isin(['Unknown']).sum()
# the name already has 18 unknowns, so will stick to changing NaN's to unknown.
```

    animal_id               0
    name                30074
    datetime                0
    monthyear               0
    date_of_birth           0
    animal_type             0
    sex_upon_outcome     7571
    age_upon_outcome        0
    breed                   0
    color                   0
    new_outcome_type        0
    dtype: int64

```python
# to see if 'Other is a common entry as well'
adoption_upd.isin(['Other']).sum()
```

    animal_id              0
    name                   0
    datetime               0
    monthyear              0
    date_of_birth          0
    animal_type         4450
    sex_upon_outcome       0
    age_upon_outcome       0
    breed                  0
    color                  0
    new_outcome_type       0
    dtype: int64

```python
# to see what kind of animals come in to the shelter
adoption_upd['animal_type'].value_counts()
```

    Dog          39758
    Cat          35974
    Other         4450
    Bird           432
    Livestock        9
    Name: animal_type, dtype: int64

```python
# initially thinking of only using cats and dogs.
adoption_upd = adoption_upd[~adoption_upd['animal_type'].isin(['Bird',
                                                        'Other',
                                                        'Livestock'])]
```

```python
# check for missing values now:
adoption_upd.isnull().sum()
```

    animal_id           0
    name                0
    datetime            0
    monthyear           0
    date_of_birth       0
    animal_type         0
    sex_upon_outcome    2
    age_upon_outcome    8
    breed               0
    color               0
    new_outcome_type    0
    dtype: int64

```python
adoption_upd['date_of_birth'] = pd.to_datetime(adoption_upd['date_of_birth'], infer_datetime_format=True)
```

```python
adoption_upd.loc[adoption_upd['age_upon_outcome'].isnull()]
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type

      59
      A808636
      Unknown
      2019-11-13 13:46:00
      2019-11-13T13:46:00.000
      2004-11-11
      Cat
      Intact Female
      NaN
      Siamese
      Seal Point
      Not adopted

      96
      A808702
      Keepers
      2019-11-12 15:20:00
      2019-11-12T15:20:00.000
      2009-11-12
      Dog
      Neutered Male
      NaN
      Golden Retriever
      Gold
      Not adopted

      131
      A808649
      Unknown
      2019-11-11 17:49:00
      2019-11-11T17:49:00.000
      2019-10-11
      Cat
      Intact Male
      NaN
      Domestic Shorthair
      Brown Tabby
      Not adopted

      132
      A738697
      Boots
      2019-11-11 17:48:00
      2019-11-11T17:48:00.000
      2007-11-17
      Dog
      NaN
      NaN
      Miniature Schnauzer Mix
      Black
      Not adopted

      161
      A808626
      Unknown
      2019-11-11 13:57:00
      2019-11-11T13:57:00.000
      2019-09-11
      Cat
      Intact Female
      NaN
      Domestic Shorthair
      Brown Tabby
      Not adopted

      214
      A808466
      Unknown
      2019-11-10 09:35:00
      2019-11-10T09:35:00.000
      2019-09-09
      Cat
      Intact Male
      NaN
      Domestic Shorthair
      Brown/Black
      Not adopted

      347
      A808352
      Unknown
      2019-11-07 16:15:00
      2019-11-07T16:15:00.000
      2011-11-07
      Dog
      Intact Male
      NaN
      Dachshund Mix
      Tan
      Not adopted

      6649
      A752967
      Gray
      2019-07-24 11:42:00
      2019-07-24T11:42:00.000
      2015-06-29
      Dog
      NaN
      NaN
      Pit Bull Mix
      Blue/White
      Not adopted

```python
# 2 of the rows that are missing the age are also missing the sex so I will
# drop those:
adoption_upd = adoption_upd.dropna(subset = ['sex_upon_outcome'])
```

```python
# since there are no missing or unknown values for DOB which still seems strange
# especially for stray animals that were found but maybe they approximated. We can
# create a column where we subtract 2019 from the born on year to get the age and see
# if there are a lot of differences.

now = pd.Timestamp('now')
# first, get DOB year column and DOB month columns:
adoption_upd['DOB_month'] = adoption_upd['date_of_birth'].dt.month
# now, subtract current year from DOB year:
adoption_upd['DOB_year'] = now.year - adoption_upd['date_of_birth'].dt.year
# now turn DOB_year into months by multiplying by 12
adoption_upd['DOB_year'] = adoption_upd['DOB_year'] * 12
# add DOB_year which is now in months to DOB month and divide by 12 to get years
adoption_upd['calculated_age'] = adoption_upd['DOB_year'] + adoption_upd['DOB_month']
def calculate_age(age):
        if age >= 12:
            return(f'{age // 12} years')
        else:
            return(f'{age} months')
adoption_upd['calculated_age'] = adoption_upd['calculated_age'].apply(calculate_age)
# fill NaN's in age upon outcome column with calculated age
adoption_upd['age_upon_outcome'].fillna(adoption_upd['calculated_age'], inplace=True)
# drop DOB month, DOB year, calculated age:
adoption_upd.drop(columns =['DOB_month', 'DOB_year', 'calculated_age'], inplace=True)
```

```python
adoption_upd.isnull().sum()
# no more missing values!
```

    animal_id           0
    name                0
    datetime            0
    monthyear           0
    date_of_birth       0
    animal_type         0
    sex_upon_outcome    0
    age_upon_outcome    0
    breed               0
    color               0
    new_outcome_type    0
    dtype: int64

```python
adoption_upd.head()
```

      animal_id
      name
      datetime
      monthyear
      date_of_birth
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type

      1
      A781697
      Pookie
      2019-11-14 15:33:00
      2019-11-14T15:33:00.000
      2017-10-05
      Dog
      Spayed Female
      2 years
      Cairn Terrier
      White/Brown
      Adopted

      2
      A808382
      Unknown
      2019-11-14 14:57:00
      2019-11-14T14:57:00.000
      2014-11-08
      Cat
      Intact Male
      5 years
      Domestic Shorthair
      Orange Tabby
      Not adopted

      3
      A806701
      *Emerald
      2019-11-14 14:41:00
      2019-11-14T14:41:00.000
      2019-09-02
      Cat
      Neutered Male
      2 months
      Domestic Shorthair
      Blue Tabby/White
      Adopted

      4
      A804553
      *Wendy
      2019-11-14 14:36:00
      2019-11-14T14:36:00.000
      2019-08-09
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted

      5
      A804552
      *Tinkerbell
      2019-11-14 14:35:00
      2019-11-14T14:35:00.000
      2019-08-09
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted

```python
# looking at redundant columns, date of birth can be dropped since the ages are all accounted for.
# also datetime and monthyear are the same, I will get rid of datetime for now.
adoption_upd.drop(columns=['date_of_birth'], inplace=True)
```

```python
# going to create a new column for the season the animal came into the shelter to see if certain
# seasons have higher adoption rates:
adoption_upd['monthyear'] = pd.to_datetime(adoption_upd['monthyear'], infer_datetime_format=True)
adoption_upd['month_arrived'] = adoption_upd['monthyear'].dt.month
adoption_upd.head()

```

      animal_id
      name
      datetime
      monthyear
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type
      month_arrived

      1
      A781697
      Pookie
      2019-11-14 15:33:00
      2019-11-14 15:33:00
      Dog
      Spayed Female
      2 years
      Cairn Terrier
      White/Brown
      Adopted
      11

      2
      A808382
      Unknown
      2019-11-14 14:57:00
      2019-11-14 14:57:00
      Cat
      Intact Male
      5 years
      Domestic Shorthair
      Orange Tabby
      Not adopted
      11

      3
      A806701
      *Emerald
      2019-11-14 14:41:00
      2019-11-14 14:41:00
      Cat
      Neutered Male
      2 months
      Domestic Shorthair
      Blue Tabby/White
      Adopted
      11

      4
      A804553
      *Wendy
      2019-11-14 14:36:00
      2019-11-14 14:36:00
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted
      11

      5
      A804552
      *Tinkerbell
      2019-11-14 14:35:00
      2019-11-14 14:35:00
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted
      11

```python
def season(arrival_month):
    if arrival_month == 1 or arrival_month == 2 or arrival_month == 3:
        return 'Winter'
    if arrival_month == 3 or arrival_month == 4 or arrival_month ==5:
        return 'Spring'
    if arrival_month == 6 or arrival_month == 7 or arrival_month ==8:
        return 'Summer'
    if arrival_month == 9 or arrival_month == 10 or arrival_month ==11:
        return 'Fall'
adoption_upd['season_arrived'] = adoption_upd['month_arrived'].apply(season)
```

```python
# want to change the datetime column to year only since we have month_arrived now:
adoption_upd['year_arrived'] = adoption_upd['datetime'].dt.year
```

```python
# going to drop datetime and monthyear now:
adoption_upd.drop(columns=['datetime', 'monthyear'], inplace=True)
```

```python
adoption_upd.head(10)
```

      animal_id
      name
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type
      month_arrived
      season_arrived
      year_arrived

      1
      A781697
      Pookie
      Dog
      Spayed Female
      2 years
      Cairn Terrier
      White/Brown
      Adopted
      11
      Fall
      2019

      2
      A808382
      Unknown
      Cat
      Intact Male
      5 years
      Domestic Shorthair
      Orange Tabby
      Not adopted
      11
      Fall
      2019

      3
      A806701
      *Emerald
      Cat
      Neutered Male
      2 months
      Domestic Shorthair
      Blue Tabby/White
      Adopted
      11
      Fall
      2019

      4
      A804553
      *Wendy
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted
      11
      Fall
      2019

      5
      A804552
      *Tinkerbell
      Cat
      Spayed Female
      3 months
      Domestic Shorthair
      Calico
      Adopted
      11
      Fall
      2019

      6
      A808132
      Daryl
      Cat
      Neutered Male
      5 years
      Domestic Shorthair
      Brown Tabby/White
      Adopted
      11
      Fall
      2019

      7
      A805655
      *Fezzik
      Cat
      Neutered Male
      2 months
      Domestic Shorthair
      Orange Tabby
      Adopted
      11
      Fall
      2019

      9
      A808526
      Jugez
      Dog
      Spayed Female
      13 years
      Shih Tzu
      Brown/Cream
      Not adopted
      11
      Fall
      2019

      10
      A787448
      Watermelon
      Dog
      Spayed Female
      3 years
      Labrador Retriever Mix
      Brown Tiger/White
      Adopted
      11
      Fall
      2019

      11
      A808522
      Unknown
      Cat
      Intact Female
      4 years
      Domestic Shorthair
      Brown Tabby
      Not adopted
      11
      Fall
      2019

```python
# how many different breeds are there:
adoption_upd['breed'].value_counts()
# 2042...very high cardinality
```

    Domestic Shorthair Mix                         26063
    Pit Bull Mix                                    4720
    Labrador Retriever Mix                          4372
    Chihuahua Shorthair Mix                         3974
    Domestic Shorthair                              3566
    Domestic Medium Hair Mix                        2548
    German Shepherd Mix                             1816
    Domestic Longhair Mix                           1184
    Siamese Mix                                     1019
    Australian Cattle Dog Mix                        969
    Dachshund Mix                                    625
    Border Collie Mix                                594
    Boxer Mix                                        572
    Domestic Medium Hair                             478
    Miniature Poodle Mix                             453
    Catahoula Mix                                    432
    Labrador Retriever                               403
    Staffordshire Mix                                401
    Chihuahua Shorthair                              401
    Pit Bull                                         384
    Australian Shepherd Mix                          381
    Great Pyrenees Mix                               367
    Pointer Mix                                      367
    Rat Terrier Mix                                  364
    Beagle Mix                                       358
    German Shepherd                                  348
    Yorkshire Terrier Mix                            346
    Siberian Husky Mix                               344
    Jack Russell Terrier Mix                         338
    Cairn Terrier Mix                                335
                                                   ...
    West Highland/Lhasa Apso                           1
    Vizsla/Greyhound                                   1
    Pembroke Welsh Corgi/Collie Smooth                 1
    Whippet/Catahoula                                  1
    Yorkshire Terrier/Dachshund Longhair               1
    Staffordshire/Border Collie                        1
    French Bulldog/Chihuahua Shorthair                 1
    Dachshund Longhair/Cairn Terrier                   1
    Standard Poodle/Whippet                            1
    Bluetick Hound/Great Pyrenees                      1
    Labrador Retriever/Collie Rough                    1
    Pbgv/Pit Bull                                      1
    English Bulldog/Boxer                              1
    Chinese Sharpei/Chow Chow                          1
    Domestic Shorthair/Manx                            1
    Tibetan Spaniel/Dachshund Longhair                 1
    Tibetan Spaniel                                    1
    Collie Rough/Chow Chow                             1
    Vizsla/Dachshund                                   1
    Snowshoe/Ragdoll                                   1
    Jack Russell Terrier/Australian Shepherd           1
    Golden Retriever/Akita                             1
    Chinese Crested/Papillon                           1
    Labrador Retriever/English Springer Spaniel        1
    English Bulldog/Mastiff                            1
    Rhod Ridgeback/Pointer                             1
    Cocker Spaniel/Dachshund Longhair                  1
    Akita/Pit Bull                                     1
    Chihuahua Shorthair/Cavalier Span                  1
    Black Mouth Cur/Belgian Malinois                   1
    Name: breed, Length: 1867, dtype: int64

```python
# create subsets for dogs and cats:
dogs = adoption_upd[adoption_upd.animal_type=='Dog']
cats = adoption_upd[adoption_upd.animal_type=='Cat']
```

```python
# Reduce cardinality for breed feature ...
dogs =dogs.copy()
# Get a list of the top 75 breeds
top75 = dogs['breed'].value_counts()[:75].index
# Breeds that are NOT in the top 100,
# replace the breed with 'OTHER'
dogs.loc[~dogs['breed'].isin(top75), 'breed'] = 'OTHER'
dogs['breed'].value_counts()
```

    OTHER                                       9368
    Pit Bull Mix                                4720
    Labrador Retriever Mix                      4372
    Chihuahua Shorthair Mix                     3974
    German Shepherd Mix                         1816
    Australian Cattle Dog Mix                    969
    Dachshund Mix                                625
    Border Collie Mix                            594
    Boxer Mix                                    572
    Miniature Poodle Mix                         453
    Catahoula Mix                                432
    Labrador Retriever                           403
    Chihuahua Shorthair                          401
    Staffordshire Mix                            401
    Pit Bull                                     384
    Australian Shepherd Mix                      381
    Great Pyrenees Mix                           367
    Pointer Mix                                  367
    Rat Terrier Mix                              364
    Beagle Mix                                   358
    German Shepherd                              348
    Yorkshire Terrier Mix                        346
    Siberian Husky Mix                           344
    Jack Russell Terrier Mix                     338
    Cairn Terrier Mix                            335
    Chihuahua Longhair Mix                       329
    Miniature Schnauzer Mix                      315
    Plott Hound Mix                              300
    Anatol Shepherd Mix                          298
    Black Mouth Cur Mix                          267
                                                ...
    Golden Retriever Mix                         122
    Queensland Heeler Mix                        120
    Maltese Mix                                  120
    Dachshund                                     97
    Shih Tzu                                      95
    Black/Tan Hound Mix                           90
    Labrador Retriever/Border Collie              88
    Manchester Terrier Mix                        88
    Cardigan Welsh Corgi Mix                      88
    Lhasa Apso Mix                                88
    Basset Hound Mix                              87
    Doberman Pinsch Mix                           87
    Boxer                                         83
    Chow Chow Mix                                 83
    Pit Bull/Labrador Retriever                   83
    Collie Smooth Mix                             79
    Labrador Retriever/Great Pyrenees             79
    Dachshund Longhair Mix                        78
    Cocker Spaniel Mix                            78
    Rottweiler                                    76
    Dachshund Wirehair Mix                        72
    Labrador Retriever/Australian Cattle Dog      69
    Pug Mix                                       69
    Mastiff Mix                                   68
    Pomeranian Mix                                68
    Great Pyrenees                                67
    Cairn Terrier                                 66
    Border Collie/Labrador Retriever              66
    Flat Coat Retriever Mix                       66
    Chinese Sharpei Mix                           66
    Name: breed, Length: 76, dtype: int64

```python
# Reduce cardinality for breed feature ...cats don't have as many
# breeds so will only use top 30
cats =cats.copy()
# Get a list of the top 30 breeds
top30 = cats['breed'].value_counts()[:30].index
# Breeds that are NOT in the top 30,
# replace the breed with 'OTHER'
cats.loc[~cats['breed'].isin(top30), 'breed'] = 'OTHER'
cats['breed'].value_counts()
```

    Domestic Shorthair Mix         26063
    Domestic Shorthair              3566
    Domestic Medium Hair Mix        2548
    Domestic Longhair Mix           1184
    Siamese Mix                     1019
    Domestic Medium Hair             478
    American Shorthair Mix           201
    Snowshoe Mix                     143
    Siamese                          130
    OTHER                            125
    Domestic Longhair                103
    Maine Coon Mix                    81
    Manx Mix                          75
    Russian Blue Mix                  46
    Ragdoll Mix                       34
    American Shorthair                29
    Himalayan Mix                     27
    Persian Mix                       16
    Balinese Mix                      14
    American Curl Shorthair Mix       12
    Japanese Bobtail Mix               9
    Persian                            9
    Russian Blue                       8
    Tonkinese Mix                      8
    Turkish Van Mix                    7
    Siamese/Domestic Shorthair         7
    Manx                               7
    Himalayan                          7
    Cymric Mix                         6
    Abyssinian                         6
    Snowshoe                           6
    Name: breed, dtype: int64

```python
# concatenate the cats and dogs subsets, matches shape of adoption_upd
adoption_final = pd.concat([dogs, cats])
adoption_final.shape
```

    (75730, 11)

```python
# lets do something similar to color:
adoption_final['color'].value_counts()
```

    Black/White                  8297
    Black                        6815
    Brown Tabby                  5511
    Brown Tabby/White            2809
    Orange Tabby                 2642
    White                        2320
    Tan/White                    2258
    Brown/White                  2151
    Blue/White                   2113
    White/Black                  2067
    Tan                          1869
    Tortie                       1664
    Calico                       1636
    Brown                        1620
    Tricolor                     1612
    Blue                         1540
    Black/Tan                    1514
    Blue Tabby                   1381
    Black/Brown                  1377
    White/Brown                  1302
    Orange Tabby/White           1289
    Brown Brindle/White          1196
    White/Tan                    1106
    Torbie                       1035
    Brown/Black                  1025
    Red                           748
    Red/White                     704
    Blue Tabby/White              675
    Tan/Black                     675
    Brown Brindle                 657
                                 ...
    Brown/Liver                     1
    Gray/Blue Merle                 1
    Tricolor/Orange                 1
    Blue Smoke/Gray                 1
    Tricolor/Red Tick               1
    White/Liver Tick                1
    Red Tick/Tricolor               1
    Black Tabby/Black               1
    Red/Brown Brindle               1
    Calico/Orange Tabby             1
    Brown Brindle/Blue              1
    Lynx Point/Blue                 1
    Gold/Yellow                     1
    Gray Tabby/Orange               1
    Lilac Point/Gray                1
    Red/Silver                      1
    White/Yellow Brindle            1
    Blue/Calico                     1
    Tortie/Tortie                   1
    Red Merle/Black                 1
    Buff/Yellow                     1
    Black Smoke/Black Tiger         1
    Brown Tabby/Black Brindle       1
    Fawn/Chocolate                  1
    Brown Merle/Chocolate           1
    Cream Tabby/Cream Tabby         1
    Orange/Orange Tabby             1
    Red/Red Merle                   1
    Orange Tabby/Tortie Point       1
    Gray/Buff                       1
    Name: color, Length: 477, dtype: int64

```python
adoption_final = adoption_final.copy()
# Get a list of the top 75 colors
top75 = adoption_final['color'].value_counts()[:75].index
# colors that are NOT in the top 75,
# replace the color with 'OTHER'
adoption_final.loc[~adoption_final['color'].isin(top75), 'color'] = 'OTHER'
```

```python
adoption_final.head()
```

      animal_id
      name
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type
      month_arrived
      season_arrived
      year_arrived

      1
      A781697
      Pookie
      Dog
      Spayed Female
      2 years
      Cairn Terrier
      White/Brown
      Adopted
      11
      Fall
      2019

      9
      A808526
      Jugez
      Dog
      Spayed Female
      13 years
      Shih Tzu
      OTHER
      Not adopted
      11
      Fall
      2019

      10
      A787448
      Watermelon
      Dog
      Spayed Female
      3 years
      Labrador Retriever Mix
      OTHER
      Adopted
      11
      Fall
      2019

      17
      A808589
      Lola
      Dog
      Spayed Female
      5 years
      OTHER
      Brown Brindle
      Not adopted
      11
      Fall
      2019

      18
      A808590
      Cheyenne
      Dog
      Spayed Female
      7 years
      Australian Shepherd Mix
      Brown/White
      Not adopted
      11
      Fall
      2019

```python
# # change animal type to numerical:
# # 1 for dog, 2 for cat
# def change_type(animal):
#     if animal == 'dog':
#         return 1
#     else:
#         return 2

```

```python
# adoption_final['animal_type'] = adoption_final['animal_type'].apply(change_type).astype(int)
```

```python
adoption_final['sex_upon_outcome'].value_counts()
```

    Neutered Male    27406
    Spayed Female    25892
    Intact Female     9822
    Intact Male       8973
    Unknown           3637
    Name: sex_upon_outcome, dtype: int64

```python
# do the same thing with sex_upon_outcome:
def change_sex(animal):
    if animal == 'Neutered Male':
        return 1
    if animal == 'Spayed Female':
        return 2
    if animal == 'Intact Male':
        return 3
    if animal == 'Intact Female':
        return 4
    if animal == 'Unknown':
        return 5
adoption_final['sex_upon_outcome'] = adoption_final['sex_upon_outcome'].apply(change_sex).astype(int)
```

```python
#need to bin ages:
adoption_final['age_upon_outcome'].value_counts()
```

    1 year       12232
    2 months     11998
    2 years       9739
    3 months      4480
    1 month       4321
    3 years       3684
    4 months      2890
    4 years       2126
    5 years       2050
    5 months      1979
    6 months      1818
    3 weeks       1756
    2 weeks       1664
    8 months      1321
    6 years       1302
    8 years       1191
    4 weeks       1180
    10 months     1158
    7 years       1064
    7 months      1000
    10 years       955
    9 months       794
    1 weeks        656
    9 years        582
    1 week         547
    12 years       466
    11 months      461
    11 years       315
    2 days         270
    3 days         262
    13 years       251
    1 day          184
    6 days         182
    4 days         157
    14 years       156
    15 years       137
    5 days         108
    0 years        103
    5 weeks         92
    16 years        39
    17 years        28
    18 years        15
    19 years         8
    20 years         7
    -1 years         1
    22 years         1
    Name: age_upon_outcome, dtype: int64

```python
# This is difficult because of the age units, there are days, weeks, months
# and years. Best way I could think of it is to bin the ages that are under 1 year
# and then do the rest of the data in years since that's the majority and it's numerical data.

def bin_ages(age):
    if age == '11 months' or age == '12 months' or age == '1 year':
        return '1 years'
    if age == '8 months' or age == '9 months' or age == '10 months':
        return '.75 years'
    if age == '6 months' or age == '7 months' or age =='5 months' or age == '4 months':
        return '.5 years'
    if age == '1 month' or age == '3 weeks' or age == '2 weeks' or age == '4 weeks' or age == '1 weeks' or age == '1 week' or age == '2 days' or age == '3 days' or age == '1 day' or age == '6 days' or age == '4 days' or age == '5 days' or age == '0 years' or age == '5 weeks' or age == '2 months' or age == '3 months' or age == '2 month' or age == '-1 years':
        return '.25 years'
    else:
        return age
adoption_final['age_upon_outcome'] = adoption_final['age_upon_outcome'].apply(bin_ages)
```

```python
adoption_final['age_upon_outcome'].value_counts()
```

    .25 years    27961
    1 years      12693
    2 years       9739
    .5 years      7687
    3 years       3684
    .75 years     3273
    4 years       2126
    5 years       2050
    6 years       1302
    8 years       1191
    7 years       1064
    10 years       955
    9 years        582
    12 years       466
    11 years       315
    13 years       251
    14 years       156
    15 years       137
    16 years        39
    17 years        28
    18 years        15
    19 years         8
    20 years         7
    22 years         1
    Name: age_upon_outcome, dtype: int64

```python
# now for the rest of the ages, need to strip the 'years' from the values:
# also going to drop the row that says -1 because not sure what that means.
def age_to_int(age):
  return float(age.strip('years'))

adoption_final['age_upon_outcome'] = adoption_final['age_upon_outcome'].apply(age_to_int)
```

```python
adoption_final.dtypes
```

    animal_id            object
    name                 object
    animal_type          object
    sex_upon_outcome      int64
    age_upon_outcome    float64
    breed                object
    color                object
    new_outcome_type     object
    month_arrived         int64
    season_arrived       object
    year_arrived          int64
    dtype: object

```python
adoption_final.head()
```

      animal_id
      name
      animal_type
      sex_upon_outcome
      age_upon_outcome
      breed
      color
      new_outcome_type
      month_arrived
      season_arrived
      year_arrived

      1
      A781697
      Pookie
      Dog
      2
      2.0
      Cairn Terrier
      White/Brown
      Adopted
      11
      Fall
      2019

      9
      A808526
      Jugez
      Dog
      2
      13.0
      Shih Tzu
      OTHER
      Not adopted
      11
      Fall
      2019

      10
      A787448
      Watermelon
      Dog
      2
      3.0
      Labrador Retriever Mix
      OTHER
      Adopted
      11
      Fall
      2019

      17
      A808589
      Lola
      Dog
      2
      5.0
      OTHER
      Brown Brindle
      Not adopted
      11
      Fall
      2019

      18
      A808590
      Cheyenne
      Dog
      2
      7.0
      Australian Shepherd Mix
      Brown/White
      Not adopted
      11
      Fall
      2019

```python
# going to make a copy without the age upon outcome column since it has high cardinality and isn't numeric:
adoption_use = adoption_final.drop(columns=['age_upon_outcome'])
```

```python
adoption_use.head()
```

      animal_id
      name
      animal_type
      sex_upon_outcome
      breed
      color
      new_outcome_type
      month_arrived
      season_arrived
      year_arrived

      1
      A781697
      Pookie
      Dog
      2
      Cairn Terrier
      White/Brown
      Adopted
      11
      Fall
      2019

      9
      A808526
      Jugez
      Dog
      2
      Shih Tzu
      OTHER
      Not adopted
      11
      Fall
      2019

      10
      A787448
      Watermelon
      Dog
      2
      Labrador Retriever Mix
      OTHER
      Adopted
      11
      Fall
      2019

      17
      A808589
      Lola
      Dog
      2
      OTHER
      Brown Brindle
      Not adopted
      11
      Fall
      2019

      18
      A808590
      Cheyenne
      Dog
      2
      Australian Shepherd Mix
      Brown/White
      Not adopted
      11
      Fall
      2019

Think I'm ready to start fitting models, may need to come back to features:

```python
# split data into train, val, test
adoption_use['year_arrived'].value_counts()
```

    2015    13964
    2019    13644
    2016    13141
    2017    13035
    2018    12513
    2014     9433
    Name: year_arrived, dtype: int64

```python
test = adoption_use[(adoption_use['year_arrived'] == 2019)]
val =  adoption_use[(adoption_use['year_arrived']== 2018)]
train = adoption_upd[(adoption_upd['year_arrived'] < 2018)]
```

```python
test.shape, val.shape, train.shape
```

    ((13644, 10), (12513, 10), (49573, 11))

```python
# %matplotlib inline
# import matplotlib.pyplot as plt
# import seaborn as sns

# for col in adoption_upd.columns:
#     if adoption_upd[col].nunique() < 10:
#         try:
#             sns.catplot(x=col, y='new_outcome_type', data=adoption_upd, kind='bar', color='grey')
#             plt.show()
#         except:
#             pass
```

```python
# numeric = train.select_dtypes('number')
# def change_type(animal):
#     if animal == 'Adopted':
#         return 1
#     else:
#         return 2
# train['new_outcome_type'] = train['new_outcome_type'].apply(change_type)
# for col in sorted(numeric.columns):
#     sns.lmplot(x=col, y='new_outcome_type', data=train, scatter_kws=dict(alpha=0.05))
#     plt.show()
```

```python
train.dtypes
```

    animal_id           object
    name                object
    animal_type         object
    sex_upon_outcome    object
    age_upon_outcome    object
    breed               object
    color               object
    new_outcome_type    object
    month_arrived        int64
    season_arrived      object
    year_arrived         int64
    dtype: object

```python

```


```python
#format the book
%matplotlib inline
from __future__ import division, print_function
import sys;sys.path.insert(0,'..')
from  book_format import load_style;load_style('..')
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Vollkorn');
@import url('http://fonts.googleapis.com/css?family=Arimo');
@import url('http://fonts.googleapis.com/css?family=Fira+sans');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        width: 900px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 600;
        font-size: 11pt;
        font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.text_cell_render{
        font-family: 'Vollkorn', verdana,arial,sans-serif;
        line-height: 150%;
        font-size: 130%;
        font-weight: 400;
        text-align:justify;
        text-justify:inter-word;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       font-weight: bold;
       color:#447adb;
    }
    a:visited{
       font-weight: bold;
       color: #1d3b84;
    }
    a:hover{
       font-weight: bold;
       color: #1d3b84;
    }
    a:focus{
       font-weight: bold;
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

# Computing and plotting PDFs of discrete data

So let's investigate how to compute and plot probability distributions.

First, let's make some data according to a normal distribution. We use `numpy.random.normal` for this. The parameters are not well named. `loc` is the mean of the distribution, and `scale` is the standard deviation. We can call this function to create an arbitrary number of data points that are distributed according to that mean and std.

```python
import numpy as np
import numpy.random as random

mean = 3
std = 2

data = random.normal(loc=mean, scale=std, size=50000)
print(len(data))
print(data.mean())
print(data.std())
```

    50000
    2.99685400941
    1.99912142836

As you can see from the print statements we got 5000 points that have a mean very close to 3, and a standard deviation close to 2.

We can plot this Gaussian by using `scipy.stats.norm` to create a frozen function that we will then use to compute the pdf (probability distribution function) of the Gaussian.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_normal(xs, mean, std, **kwargs):
    norm = stats.norm(mean, std)
    plt.plot(xs, norm.pdf(xs), **kwargs)

xs = np.linspace(-5, 15, num=200)
plot_normal(xs, mean, std, color='k')
```

But we really want to plot the PDF of the discrete data, not the idealized function.

There are a couple of ways of doing that. First, we can take advantage of `matplotlib`'s `hist` method, which computes a histogram of a collection of data. Normally `hist` computes the number of points that fall in a bin, like so:

```python
plt.hist(data, bins=200)
plt.show()
```

that is not very useful to us - we want the PDF, not bin counts. Fortunately `hist` includes a `density` parameter which will plot the PDF for us.

```python
plt.hist(data, bins=200, normed=True)
plt.show()
```

I may not want bars, so I can specify the `histtype` as 'step' to get a line.

```python
plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
plt.show()
```

To be sure it is working, let's also plot the idealized Gaussian in black.

```python
plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
norm = stats.norm(mean, std)
plt.plot(xs, norm.pdf(xs), color='k', lw=2)
plt.show()
```

There is another way to get the approximate distribution of a set of data. There is a technique called *kernel density estimate* that uses a kernel to estimate the probability distribution of a set of data. NumPy implements it with the function `gaussian_kde`. Do not be mislead by the name - Gaussian refers to the type of kernel used in the computation. This works for any distribution, not just Gaussians. In this section we have a Gaussian distribution, but soon we will not, and this same function will work.

```python
kde = stats.gaussian_kde(data)

xs = np.linspace(-5, 15, num=200)
plt.plot(xs, kde(xs))
plt.show()
```

## Monte Carlo Simulations

We (well I) want to do this sort of thing because I want to use monte carlo simulations to compute distributions. It is easy to compute Gaussians when they pass through linear functions, but difficult to impossible to compute them analytically when passed through nonlinear functions. Techniques like particle filtering handle this by taking a large sample of points, passing them through a nonlinear function, and then computing statistics on the transformed points. Let's do that.

We will start with the linear function $f(x) = 2x + 12$ just to prove to ourselves that the code is working. I will alter the mean and std of the data we are working with to help ensure the numbers that are output are unique It is easy to be fooled, for example, if the formula multipies x by 2, the mean is 2, and the std is 2. If the output of something is 4, is that due to the multication factor, the mean, the std, or a bug? It's hard to tell.

```python
def f(x):
    return 2*x + 12

mean = 1.
std = 1.4
data = random.normal(loc=mean, scale=std, size=50000)

d_t = f(data) # transform data through f(x)

plt.hist(data, bins=200, normed=True, histtype='step', lw=2)
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)

plt.ylim(0, .35)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

This is what we expected. The input is the Gaussian $\mathcal{N}(\mu=1, \sigma=1.4)$, and the function is $f(x) = 2x+1$. Therefore we expect the mean to be shifted to $f(\mu) = 2*1+12=14$. We can see from the plot and the print statement that this is what happened.

Before I go on, can you explain what happened to the standard deviation? You may have thought that the new $\sigma$ should be passed through $f(x)$ like so $2(1.4) + 12=14.81$. But that is not correct - the standard deviation is only affected by the multiplicative factor, not the shift. If you think about that for a moment you will see it makes sense. We multiply our samples by 2, so they are twice as spread out as before. Standard deviation is a measure of how spread out things are, so it should also double. It doesn't matter if we then shift that distribution 12 places, or 12 million for that matter - the spread is still twice the input data.

## Nonlinear Functions

Now that we believe in our code, lets try it with nonlinear functions.

```python
def f2(x):
    return (np.cos((1.5*x + 2.1))) * np.sin(0.3*x) - 1.6*x

d_t = f2(data)
plt.subplot(121)
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)

plt.subplot(122)
kde = stats.gaussian_kde(d_t)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, kde(xs), 'k')
plot_normal(xs, d_t.mean(), d_t.std(), color='g', lw=3)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

Here I passed the data through the nonlinear function $f(x) = \cos(1.5x+2.1)\sin(\frac{x}{3}) - 1.6x$. That function is quite close to linear, but we can see how much it alters the pdf of the sampled data.

There is a lot of computation going on behind the scenes to transform 50,000 points and then compute their PDF. The Extended Kalman Filter (EKF) gets around this by linearizing the function at the mean and then passing the Gaussian through the linear equation. We saw above how easy it is to pass a Gaussian through a linear function. So lets try that.

We can linearize this by taking the derivative of the function at x. We can use sympy to get the derivative.

```python
import sympy
x = sympy.symbols('x')
f = sympy.cos(1.5*x+2.1) * sympy.sin(x/3) - 1.6*x
dfx = sympy.diff(f, x)
dfx
```

    -1.5*sin(x/3)*sin(1.5*x + 2.1) + cos(x/3)*cos(1.5*x + 2.1)/3 - 1.6

We can now compute the slope of the function by evaluating the derivative at the mean.

```python
m = dfx.subs(x, mean)
m
```

    -1.66528051815545

The equation of a line is $y=mx+b$, so the new standard deviation should be $~1.67$ times the input std. We can compute the new mean by passing it through the original function because the linearized function is just the slope of f(x) evaluated at the mean. The slope is a tangent that touches the function at $x$, so both will return the same result. So, let's plot this and compare it to the results from the monte carlo simulation.

```python
plt.hist(d_t, bins=200, normed=True, histtype='step', lw=2)
plot_normal(xs, f2(mean), abs(float(m)*std), color='k', lw=3, label='EKF')
plot_normal(xs, d_t.mean(), d_t.std(), color='r', lw=3, label='MC')
plt.legend()
plt.show()
```

We can see from this that the estimate from the EKF (in red) is not exact, but it is not a bad approximation either.