# Modeling the Time Evolution of the Annualized Rate of Public Mass Shootings with Gaussian Processes

Nathan Sanders, Victor Lei (Legendary Entertainment)

January, 2017

## Abstract

Much of the public policy debate over gun control and gun rights in the United States hinges on the alarming incidence of public mass shootings, here defined as attacks killing four or more victims.  Several times in recent years, individual, highly salient public mass shooting incidents have galvanized public discussion of reform efforts.  But deliberative legislative action proceeds over a much longer timescale that should be informed by knowledge of the long term evolution of these events.  We have used *Stan* to develop a new model for the annualized rate of public mass shootings in the United States based on a Gaussian process with a time-varying mean function.  This design yields a predictive model with the full non-parametric flexibility of a Gaussian process, while retaining the direct interpretability of a parametric model for long-term evolution of the mass shooting rate.  We apply this model to the Mother Jones database of public mass shootings and explore the posterior consequences of different prior choices and of correlations between hyperparameters.  We reach conclusions about the long term evolution of the rate of public mass shootings in the United States and short-term periods deviating from this trend.

## Background

Tragic, high profile public events over the past few years like the shootings at the Washington Navy Yard; the Emanuel AME Church in Charleston; San Bernadino, CA; and Orlando, FL have raised public awareness of the dangers posed by public mass shooting events and sociological interest in understanding the motivations and occurrence rates of such events.  There is no commonly accepted definition of a public mass shooting, but such an event is generally understood to be the simultaneous homicide of multiple people perpetrated by an individual or coordinated group via firearm.

A particular question facing elevated public, political, and scholarly scrutiny is whether the rate of public mass shootings has increased significantly over recent years.  Lott (2014) responded to a [September, 2013 FBI report](https://www.fbi.gov/news/stories/2014/september/fbi-releases-study-on-active-shooter-incidents/pdfs/a-study-of-active-shooter-incidents-in-the-u.s.-between-2000-and-2013) on public mass shootings by re-evaluating sources of bias, reviewing data consistency, and redefining the period under consideration to conclude that no statistically significant increase is identifiable.  Lott's work has been the subject of persistent controversy (see e.g. Johnson et al. 2012).  In contrast, Cohen et al. (2014) claim that the rate of public mass shootings tripled over the four year period 2011-2014 based on a Statistical Process Control (SPC) analysis of the duration between successive events.

In this study, we present a new statistical approach to evaluating the time evolution of the rate of public mass shootings.  We do not present original data on occurrences in the United States, address the myriad considerations inherent in defining a "mass shooting" event, or seek to resolve the causal issues of why the growth rate may have changed over time.  We do adopt a commonly cited public mass shooting dataset and definition from Mother Jones.

We develop a Gaussian process-based model for the time evolution of the occurrence rate of public mass shootings and demonstrate inference under this model by straightforward application of the probabilistic programming language *Stan*.  We use this case to explore the intersection of parametric and non-parametric models.  We seek to merge a parametric model, with straightforward interpretations of posterior marginalized parameter inferences, with a non-parametric model that captures and permits discovery of unspecified trends.  *Stan's* flexible modeling language permits rapid model design and iteration, while the No-U-Turn sampler allows us to fully explore the model posterior and understand the dependence between the parametric and non-parametric components of our model and the implications of our prior assumptions.

In the following notebook, we describe the Mother Jones dataset on US public mass shootings and lay out our statistical model and inference scheme.  We then discuss the results from this inference, how they depend on choices for the prior distribution, and explore correlations between hyperparameters.  Finally, we discuss the conclusions that can be reached from inspection of the marginal posterior distributions.

```python
## Notebook setup
%matplotlib inline

import pandas as pd
import numpy as np
import pickle, os, copy
import scipy
from matplotlib import pyplot as plt
from matplotlib import cm
from matplotlib.ticker import FixedLocator, MaxNLocator, AutoMinorLocator

## NOTE: We encounter an error with this model using PyStan 2.14,
## so for now we will wrap cmdstan using stanhelper instead.
#import pystan

## See https://github.com/akucukelbir/stanhelper
import stanhelper
import subprocess
cmdstan_path = os.path.expanduser('~/Stan/cmdstan_2.14.0/')

from scipy import stats as sstats
```

### Package versions

```python
%load_ext watermark
%watermark -v -m -p pandas,numpy,scipy,matplotlib,pystan
```

    CPython 2.7.6
    IPython 5.1.0

    pandas 0.18.1
    numpy 1.11.3
    scipy 0.18.1
    matplotlib 1.4.3
    pystan 2.14.0.0

    compiler   : GCC 4.8.4
    system     : Linux
    release    : 3.16.0-38-generic
    machine    : x86_64
    processor  : x86_64
    CPU cores  : 4
    interpreter: 64bit

```python
print subprocess.check_output(cmdstan_path+'bin/stanc --version', shell=1)
```

    stanc version 2.14.0

## Data

For this study, we consider the [database published by Mother Jones](http://www.motherjones.com/politics/2012/12/mass-shootings-mother-jones-full-data) (retrieved for this study on October 16, 2016; as of January 14, 2017, Mother Jones had not added any further events to its database for 2016), compiling incidents of public mass shootings in the United States from 1982 through the end of 2016.  The database includes rich (quantitative and qualitative) metadata on the effects of the incidents, the mental health condition of the perpetrators, weapon type, how the perpetrators obtained their weapons, and more; however, we focus primarily on the dates of incident occurrence.

The definition of a public mass shooting is not universally agreed upon, and even when a firm definition is adopted there can be ambiguity in how to apply it to the complex and uncertain circumstances of these chaotic events.  See Fox & Levin (2015) for a recent discussion.  The criteria for inclusion in the Mother Jones database were described in a [2014 article by Mark Follman](http://www.motherjones.com/politics/2014/10/mass-shootings-rising-harvard):

> [The database] includes attacks in public places with four or more victims killed, a baseline established by the FBI a decade ago. We excluded mass murders in private homes related to domestic violence, as well as shootings tied to gang or other criminal activity.''

Follman discusses their motivations for these criteria and provide some examples of prominent incidents excluded by the criteria, such as the shooting at Ft. Hood in April, 2014.  Note that the federal threshold for investigation of public mass shootings was lowered to three victim fatalities in January of 2013, and the Mother Jones database includes shootings under this more expansive definition starting from that date.  To maintain a consistent definition for public mass shootings throughout the studied time period, we only consider shootings with four or more victim fatalities.

Our primary dataset is the count of incidents reported in this database per calendar year.  We include incidents labeled as both "Mass" or "Spree" by Mother Jones.

```python
## Load data
data = pd.read_excel('MotherJonesData_2016_10_16.xlsx','US mass shootings')

## Stadardize on definition of fatalities at 4.  Mother Jones changed it to 3 in 2013.
data = data[data.Fatalities > 3]

## Prepare data
# Aggregate data anually
data_annual = data.groupby('Year')
# Count cases by year and fill in empty years
cases_resamp = data_annual.count().Case.ix[np.arange(1982,2017)].fillna(0)
# Enumerate years in range
data_years = cases_resamp.index.values
# Enumerate quarters across daterange for later plotting
data_years_samp = np.arange(min(data_years), max(data_years)+10, .25)
# Format for Stan
stan_data = {
	'N1': len(cases_resamp),
	'x1': data_years - min(data_years),
	'z1': cases_resamp.values.astype(int),
	'N2': len(data_years_samp),
	'x2': data_years_samp - min(data_years),
	}
```

```python
## Print the stan model inputs
for key in stan_data:
    print key
    print stan_data[key]
    print '\n'
```

    x2
    [  0.     0.25   0.5    0.75   1.     1.25   1.5    1.75   2.     2.25
       2.5    2.75   3.     3.25   3.5    3.75   4.     4.25   4.5    4.75   5.
       5.25   5.5    5.75   6.     6.25   6.5    6.75   7.     7.25   7.5
       7.75   8.     8.25   8.5    8.75   9.     9.25   9.5    9.75  10.    10.25
      10.5   10.75  11.    11.25  11.5   11.75  12.    12.25  12.5   12.75  13.
      13.25  13.5   13.75  14.    14.25  14.5   14.75  15.    15.25  15.5
      15.75  16.    16.25  16.5   16.75  17.    17.25  17.5   17.75  18.    18.25
      18.5   18.75  19.    19.25  19.5   19.75  20.    20.25  20.5   20.75  21.
      21.25  21.5   21.75  22.    22.25  22.5   22.75  23.    23.25  23.5
      23.75  24.    24.25  24.5   24.75  25.    25.25  25.5   25.75  26.    26.25
      26.5   26.75  27.    27.25  27.5   27.75  28.    28.25  28.5   28.75  29.
      29.25  29.5   29.75  30.    30.25  30.5   30.75  31.    31.25  31.5
      31.75  32.    32.25  32.5   32.75  33.    33.25  33.5   33.75  34.    34.25
      34.5   34.75  35.    35.25  35.5   35.75  36.    36.25  36.5   36.75  37.
      37.25  37.5   37.75  38.    38.25  38.5   38.75  39.    39.25  39.5
      39.75  40.    40.25  40.5   40.75  41.    41.25  41.5   41.75  42.    42.25
      42.5   42.75  43.    43.25  43.5   43.75]

    N1
    35

    N2
    176

    x1
    [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
     25 26 27 28 29 30 31 32 33 34]

    z1
    [1 0 2 0 1 1 1 2 1 3 2 4 1 1 1 2 3 5 1 1 0 1 1 2 3 4 3 4 1 3 7 5 3 4 4]

```python
## Number of years with data
print len(stan_data['x1'])
```

    35

```python
## Number of interpolated points to do prediction for
print len(stan_data['x2'])
```

    176

## Statistical Model

We adopt a univariate Gaussian process model (see e.g. Rasmussen & Williams 2006) as a non-parametric description of the time evolution of the annualized occurrence rate.  The Gaussian process describes deviations from a mean function by a covariance matrix that controls the probability of the deviation as a function of the time differential between points.  Robers et al. (2012) surveyed applications of Gaussian process models to timeseries data, and explored the implications of different choices for the mean and covariance functions.

We adopt the following system of units for the Gaussian Process model.  The time vector $x$ is measured in years since 1982 and the outcome vector $z$ is defined as the number of occurrences per year.

Many applications of Gaussian processes adopt a constant, zero mean function.  In that case, the relationship between the dependent variable(s) and the predictors is described entirely by the non-parametric family of functions generated from the Gaussian process covariance function.

We adopt a linear mean function and a squared-exponential covariance function.  The mean function $\mu(x)$ is simply:

\begin{equation}
\mu(x) = \mu_0 + \mu_b~x
\end{equation}

Note that we use a logarithmic parameterization of the likelihood for the occurence rate (see below), so the linear mean function corresponds to an exponential function for the evolution of the rate of shootings per year.

The familiar squared-exponential covariance function, which generates infinitely-differentiable functions from the Gaussian process, is:

\begin{equation}
k(x)_{i,j} = \eta^2~exp \big( -\rho^2 \sum_{d=1}^{D}(x_{i,d} - x_{j,d})^2 \big) + \delta_{i,j}~\sigma^2
\end{equation}

where the hyperparameter $\eta$ controls the overall strength of covariance, $\rho$ controls the timescale over which functions drawn from the process vary, and $\sigma$ controls the baseline level of variance.

Our likelihood assumes that the occurrence rate is specified by exponentiated draws of the occurrence rate $y$ from the mean and covariance functions, and the observed outcome data is negative binomial-distributed according to the rate.

\begin{align}
y(x) \sim \rm{N}(\mu(x), k(x)^2) \\
z(x) ~ \sim \rm{NB}(exp(y(x)), \phi)
\end{align}

where $\rm{N}$ is the normal (parameterized by the standard deviation rather than the variance, per *Stan* standard syntax) and $\rm{NB}$ is the negative binomial distribution.  We use the "alternative" parameterization of the negative binomial distribution described in the *Stan* manual, where the second parameter directly scales the overdispersion relative to a Poisson distribution.  While we choose the negative binomial to permit overdispersion in the annualized mass shooting rate beyond counting noise, as we will see, the data provide strong evidence for small values of $\phi^{-1}$, consistent with Poisson noise.

The role of each component of the Gaussian process will depend largely on the timescale parameter $\rho$.  When the timescale is short, the model effectively divides the response into a long-term (timescale of the range of the data; in this case, decades) parametric effect and a short-term (timescale of e.g. years) non-parametric effect.  This approach gives us the full flexibility of the Gaussian process for predictive applications, while still allowing us to make interpretable, parametric inferences on the long-term evolution of the system.

We apply the following prior and hyperprior distributions to provide weak information about the scale of the relevant parameters in the adopted unit system:

\begin{align*}
\rho^{-1} \sim \Gamma(\alpha_{\rho}, \beta_{\rho})  \\
\eta^2 \sim \rm{C}(2. 5) \\
\sigma^2 \sim \rm{C}(0, 2.5)  \\
\mu_0 \sim \rm{N}(0, 2)  \\
\mu_b \sim \rm{N}(0, 0.2)  \\
\phi^{-1} \sim C(0, 5)
\end{align*}

where $\Gamma$ is the gamma distribution; $\rm{C}$ is the half-Cauchy distribution; the parameters $\eta^2$, $\sigma^2$, and $\phi^{-1}$ are constrained to be positive; and we apply the constraint $\rho^{-1} > 1$ to enforce timescales $>1$ yr (the spacing of our data).

Below we explore different choices for the $\alpha$ and $\beta$ parameters of the gamma hyperprior on $\rho^{-1}$, labeled as $\alpha_{\rho}$ and $\beta_{\rho}$. In particular, we explore $(\alpha_{\rho},\beta_{\rho}) = (4,1)$ and $(1,1/100)$.  These correspond to prior distributions with standard deviations of $2$ and $100$ years, respectively.  On top of the linear trend in the mean function, the former represents a strong prior expectation that the annualized rate of public mass shootings evolves on a timescale of a few years, and the latter represents a nearly-flat expectation for variations on timescales from a few years to a few centuries.

We implement the Gaussian process model in *Stan*, adapting the logistic classification example in Section 14.5 of the *Stan* manual.  *Stan's* *NUTS* sampler performs full joint Bayesian estimation of all parameters, including the mean function parameters $\mu_0$ and $\mu_b$ and the Gaussian Process hyperparmeters $\eta$, $\rho$, and $\sigma$ and the negative binomial over-dispersion $\phi^{-1}$.  The $\alpha_{\rho}$ and $\beta_{\rho}$ hyperparameters of the $\rho$ hyperprior distribution are fixed.  We use the Cholesky factor transformed implementation of the normal distribution to calculate the likelihood.

We expect these hyperparameters to be at least somewhat correlated and not well-identified, introducing significant curvature in the model posterior, indicating that Hamiltonian Monte Carlo (HMC) would be a particularly effective sampling strategy for this model (Betancourt & Girolami 2013).  We fit the model to the 35 annual observations of the Mother Jones dataset and do model interpolation and prediction over a grid of 176 quarters from 1980 to 2024.

We typically fit 8 independent chains of length 2000 iterations (following an equal number of NUTS warmup samples) in parallel using *Stan* and observe a typical execution time of ~1 min.  For the purposes of this notebook, we obtain a larger number of samples by fitting 20 chains of 4000 samples in order to improve the resolution of 2D posterior histograms.

```python
with open('gp_model_final.stan', 'r') as f:
    stan_code = f.read()
    print stan_code
```

    data {
    	int<lower=1> N1;
    	vector[N1] x1;
    	int z1[N1];
    	int<lower=1> N2;
    	vector[N2] x2;
    	real<lower=0> alpha_rho;
    	real<lower=0> beta_rho;
    }
    transformed data {
    	int<lower=1> N;
    	vector[N1+N2] x;
    	// cov_exp_quad wants real valued inputs
    	real rx[N1+N2];
    	real rx1[N1];
    	real rx2[N2];

    	N = N1 + N2;
    	x = append_row(x1, x2);

    	rx = to_array_1d(x);
    	rx1 = to_array_1d(x1);
    	rx2 = to_array_1d(x2);
    }
    parameters {
    	vector[N1] y_tilde1;
    	real<lower=0> eta_sq;
    	real<lower=1> inv_rho;
    	real<lower=0> sigma_sq;
    	real mu_0;
    	real mu_b;
    	real<lower=0> NB_phi_inv;
    }
    model {
    	vector[N1] mu1;
    	vector[N1] y1;
    	matrix[N1,N1] Sigma1;
    	matrix[N1,N1] L1;

    	// Calculate mean function
    	mu1 = mu_0 + mu_b * x1;

    	// GP hyperpriors
    	eta_sq ~ cauchy(0, 1);
    	sigma_sq ~ cauchy(0, 1);
    	inv_rho ~ gamma(alpha_rho, beta_rho); // Gamma prior with mean of 4 and std of 2

    	// Calculate covariance matrix using new optimized function
    	Sigma1 = cov_exp_quad(rx1, sqrt(eta_sq), sqrt(0.5) * inv_rho);
    	for (n in 1:N1) Sigma1[n,n] = Sigma1[n,n] + sigma_sq;

    	// Decompose
    	L1 = cholesky_decompose(Sigma1);
    	// We're using a the non-centered parameterization, so rescale y_tilde
    	y1 = mu1 + L1 * y_tilde1;

    	// Mean model priors
    	mu_0 ~ normal(0, 2);
    	mu_b ~ normal(0, 0.2);

    	// Negative-binomial prior
    	// For neg_binomial_2, phi^-1 controls the overdispersion.
    	// phi^-1 ~ 0 reduces to the poisson.  phi^-1 = 1 represents variance = mu+mu^2
    	NB_phi_inv ~ cauchy(0, 5);

    	// Generate non-centered parameterization
    	y_tilde1 ~ normal(0, 1);

    	// Likelihood
    	z1 ~ neg_binomial_2_log(y1, inv(NB_phi_inv));
    }

    generated quantities {
    	vector[N1] y1;
    	vector[N2] y2;
    	vector[N] y;
    	int z_rep[N];

    	{
    		// Don't save these parameters
    		matrix[N,N] Sigma;
    		matrix[N,N] L;
    		vector[N] y_tilde;

    		Sigma = cov_exp_quad(rx, sqrt(eta_sq), sqrt(0.5) * inv_rho);
    		for (n in 1:N) Sigma[n,n] = Sigma[n,n] + sigma_sq;

    		for (n in 1:N1) y_tilde[n] = y_tilde1[n];
    		for (n in (N1 + 1):N) y_tilde[n] = normal_rng(0,1);

    		// Decompose
    		L = cholesky_decompose(Sigma);
    		y = mu_0 + mu_b * x + L * y_tilde;

    		for (n in 1:N1) y1[n] = y[n];
    		for (n in 1:N2) y2[n] = y[N1+n];

    		for (n in 1:N) z_rep[n] = neg_binomial_2_log_rng(y[n], inv(NB_phi_inv));
    	}
    }

Note that we use the newly introduced *cov_exp_quad* function to implement the squared exponential covariance function, and we rescale $\rho^{-1}$ by $2^{-1/2}$ to accomodate the difference between this implementation and our definition above.  Moreover, we use a non-centered parameterization (see e.g. Papaspiliopoulos et al. 2003) for the Gaussian process, modeling the latent parameter $\tilde{y}$ as standard normal and then transforming to a sampled value for $y$ by rescaling by the covariance matrix.

## Model fitting

```python
## Compile using pystan
#stan_model_compiled = pystan.StanModel(model_code=stan_code)

### Compile using cmdstan
### Script expects cmdstan installation at cmdstan_path
subprocess.call("mkdir "+cmdstan_path+"user-models", shell=1)
subprocess.call("cp gp_model_final.stan " + cmdstan_path+"user-models/", shell=1)
subprocess.call("make user-models/gp_model_final", cwd=cmdstan_path, shell=1)
```

    0

Below we explore the consequences of different choices for the prior distribution on $\rho^{-1}$.  To facilitate that analysis, here we fit the model twice with two different hyperparameter specifications provided as data.  We will visualize and discuss these hyperprior choices in the next section.  When not explicitly making comparisons between the two models, we focus on the model with the stronger prior on $\rho^{-1}$.

```python
## Sampling parameters
Nchains = 20
Niter = 8000
cdic = {'max_treedepth': 15, 'adapt_delta': 0.95}

```

```python
## Sample with strong prior on rho
stan_data_rho_strong = copy.copy(stan_data)
stan_data_rho_strong['alpha_rho'] = 4
stan_data_rho_strong['beta_rho'] = 1

## Sample with pystan
#stan_model_samp_rho_strong = stan_model_compiled.sampling(
#    data = stan_data_rho_strong, iter=Niter,
#    chains=Nchains, control=cdic, seed=1002
#    )

## Sample with cmdstan
## Delete any old samples first
os.system('rm output_cmdstan_gp_rhostrong_samples*.csv')
stanhelper.stan_rdump(stan_data_rho_strong, 'input_data_rhostrong_final.R')
p = []
for i in range(Nchains):
    cmd = """
{0}user-models/gp_model_final \
data file='input_data_rhostrong_final.R' \
sample num_warmup={2} num_samples={2} \
adapt delta={4} \
algorithm=hmc engine=nuts max_depth={3} \
random seed=1002 id={1} \
output file=output_cmdstan_gp_rhostrong_samples{1}.csv
            """.format(cmdstan_path, i+1, Niter/2, cdic['max_treedepth'], cdic['adapt_delta'])
    p += [subprocess.Popen(cmd, shell=True)]

## Don't move on until sampling is complete.
for i in range(Nchains):
    p[i].wait()

## Write out results if using pystan
#stan_model_ext_rho_strong = stan_model_samp_rho_strong.extract()
#with open('stan_model_ext_rho_strong.p','w') as f: pickle.dump(stan_model_ext_rho_strong,f)
```

```python
## Sample with weak prior on rho
stan_data_rho_weak = copy.copy(stan_data)
stan_data_rho_weak['alpha_rho'] = 1
stan_data_rho_weak['beta_rho'] = 1/100.

## Sample with pystan
#stan_model_samp_rho_weak = stan_model_compiled.sampling(data = stan_data_rho_weak, iter=Niter, chains=Nchains, control=cdic)

## Sample with cmdstan
## Delete any old samples first
os.system('rm output_cmdstan_gp_rhoweak_samples*.csv')
stanhelper.stan_rdump(stan_data_rho_weak, 'input_data_rhoweak_final.R')
p = []
for i in range(Nchains):
    cmd = """
{0}user-models/gp_model_final \
data file='input_data_rhoweak_final.R' \
sample num_warmup={2} num_samples={2} \
adapt delta={4} \
algorithm=hmc engine=nuts max_depth={3} \
random seed=1002 id={1} \
output file=output_cmdstan_gp_rhoweak_samples{1}.csv
            """.format(cmdstan_path, i+1, Niter/2, cdic['max_treedepth'], cdic['adapt_delta'])
    p += [subprocess.Popen(cmd, shell=True)]

## Don't move on until sampling is complete.
for i in range(Nchains):
    p[i].wait()

## Write out results if using pystan
#stan_model_ext_rho_weak = stan_model_samp_rho_weak.extract()
#with open('stan_model_ext_rho_weak.p','w') as f: pickle.dump(stan_model_ext_rho_weak,f)
```

```python
def stan_read_csv_multi(path):
    """
    Wrap the stanhelper.stan_read_csv function to load outputs
    from multiple chains.

    Parameters:
    * path: file path for cmdstan output files including wildcard (*)
    """
    ## Enumerate files
    from glob import glob
    files = glob(path)

    ## Read in each file
    result = {}
    for file in files:
        result[file] = stanhelper.stan_read_csv(file)

    ## Combine dictionaries
    result_out = {}
    keys = result[files[0]]
    for key in keys:
        result_out[key] = result[files[0]][key]
        for f in files:
                result_out[key] = np.append(result_out[key], result[f][key], axis=0)

    ## Remove extraneous dimension
    for key in keys:
        if result_out[key].shape[-1] == 1:
            result_out[key] = np.squeeze(result_out[key], -1)

    return result_out

stan_model_ext_rho_strong = stan_read_csv_multi('output_cmdstan_gp_rhostrong_samples*.csv')
stan_model_ext_rho_weak = stan_read_csv_multi('output_cmdstan_gp_rhoweak_samples*.csv')
```

The MCMC trace illustrates the high independence of samples achieved after the *NUTS* algorithm warm-up period, and the low variance in sampling distributions between chains.

```python
## Traceplot
trace_pars = [('eta_sq','$\\eta^2$'),
	      ('inv_rho','$\\rho^{-1}$'),
	      ('sigma_sq','$\\sigma^2$'),
	      ('mu_0','$\\mu_0$'),
	      ('mu_b','$\\mu_b$'),
	      ('NB_phi_inv','$\\rm{NB}_\\phi^{-1}$')]
fig,axs = plt.subplots(len(trace_pars),2, figsize=(8,8), sharex='all', sharey='row')
exts = [stan_model_ext_rho_strong, stan_model_ext_rho_weak]
exts_names = [r'Strong $\rho$ prior', r'Weak $\rho$ prior']
for j in range(2):
    axs[0,j].set_title(exts_names[j])
    for i,par in enumerate(trace_pars):
        axs[i,j].plot(exts[j][par[0]], color='.5')
        if j==0: axs[i,j].set_ylabel(par[1])
        for k in range(1, Nchains+1):
            axs[i,j].axvline(Niter/2 * k, c='r', zorder=-1)

    axs[len(trace_pars) - 1,j].set_xticks(np.arange(0, (Niter/2)*Nchains+1, Niter*2))

```

We assess MCMC convergence quantitatively using the Gelman-Rubin convergence diagnostic, $\hat{R}$, a comparison of within- to between-chain variance.  We find that $\hat{R} \ll 1.05$ for all parameters, indicating a negligable discrepancy in the sampling distributions between chains.

```python
def read_stansummary(path, cmdstan_path=cmdstan_path):
    """
    Wrapper for the cmdstan program stan_summary to calculate
    sampling summary statistics across multiple MCMC chains.

    Args:
        path (str): Path, with a wildcard (*) for the id number
        of each output chain

        cmdstan_path (str): Path to the stan home directory

    Returns:
        out: A pandas dataframe with the summary statistics provided
        by stan_summary.  Note that each element of array variables
        are provided on separate lines
    """
    from StringIO import StringIO
    summary_string = subprocess.check_output(cmdstan_path + 'bin/stansummary --sig_figs=5 '+path, shell=1)
    out = pd.read_table(StringIO(summary_string), sep='\s+', header=4, skip_footer=6, engine='python')
    return out

## Use cmdstan's stansummary command to calculate rhat
stan_model_sum_rho_strong = read_stansummary('output_cmdstan_gp_rhostrong*.csv')
stan_model_sum_rho_weak = read_stansummary('output_cmdstan_gp_rhoweak*.csv')
```

```python
## Get summary statistics using pystan
#model_summary = stan_model_samp_rho_strong.summary()
#Rhat_vec = model_summary['summary'][:,array(model_summary['summary_colnames'])=='Rhat']
#pars = model_summary['summary_rownames']

## Get summary statistics using cmdstan wrapper
model_summary = stan_model_sum_rho_strong
Rhat_vec = stan_model_sum_rho_strong['R_hat'].values
pars = stan_model_sum_rho_strong.index

## Replace y1, y2 with summaries
sel_pars = ['y1', 'y2', u'eta_sq', u'inv_rho', u'sigma_sq', u'mu_0', u'mu_b', 'NB_phi_inv']
Rhat_dic = {}
for spar in sel_pars:
	if spar in ('y1','y2'):
		sel = np.where([True if p.startswith(spar) else False for p in pars])
		Rhat_dic[spar] = np.percentile(Rhat_vec[sel], [5,50,95])
	else:
		Rhat_dic[spar] = [Rhat_vec[[pars==spar]],]*3

plt.figure(figsize=(5,6))
plt.errorbar(np.array(Rhat_dic.values())[:,1], np.arange(len(sel_pars)), \
	xerr= [np.array(Rhat_dic.values())[:,1] - np.array(Rhat_dic.values())[:,0],\
		np.array(Rhat_dic.values())[:,2] - np.array(Rhat_dic.values())[:,1]],\
	capsize=0, marker='o', color='k', lw=0)
plt.yticks(np.arange(len(sel_pars)), Rhat_dic.keys(), size=11)
plt.xlabel('$\hat{R}$')
plt.axvline(1.0, color='.5', ls='solid', zorder=-2)
plt.axvline(1.05, color='.5', ls='dashed', zorder=-2)
plt.ylim(-.5, len(sel_pars)-.5)
plt.xlim(0.99, 1.06)
```

## Posterior Simulations and Predictive Checks

To assess goodness of fit, we inspect simulated draws of the Gaussian process from the posterior and perform posterior predictive checks.

### Simulated draws

First we perform a posterior predictive check by visualizing the sampled values of $z$, which realizes both a draw from the latent Gaussian process for the public mass shootings rate and the overdispersed counting noise of the negative binomial distribution.

```python
N_samp = Niter / 2
print len(stan_model_ext_rho_strong['z_rep'])
print Niter

fig, axs = plt.subplots(5,5, figsize=(7,7), sharex='all', sharey='all')
po = axs[0,0].plot(data_years, stan_data['z1'], 'o', c='k', mfc='k', label='Observations', zorder=2, lw=1, ms=4)
axs[0,0].legend(numpoints=1, prop={'size':6})
for i in range(1,25):
	draw = np.random.randint(0, N_samp)
	py = stan_model_ext_rho_strong['z_rep'][draw][:stan_data['N1']]
	axs.flatten()[i].plot(data_years, py,  mfc='k', marker='o',
	  lw=.5, mec='none', ms=2, color='.5', label='GP realization')
axs[0,1].legend(numpoints=1, prop={'size':6})
axs[0,0].set_ylim(0,15)
axs[0,0].set_xticks([1980, 1990, 2000, 2010, 2020])
for ax in axs.flatten():
	plt.setp(ax.get_xticklabels(), rotation='vertical', fontsize=9)
	plt.setp(ax.get_yticklabels(), fontsize=9)

axs[2,0].set_ylabel('public mass shootings per year', size=9)

```

Visual inspection suggests that the observations simulated under the model show similar variation over time as the actual observations (first panel).  We note that some realizations have annual counts at the later end of the modeled time range that exceed the largest observed annual count (7 public mass shootings).  Some exceedence is expected given the counting noise, but this posterior predictive check could guide revision of the prior on the over-dispersion parameter or the choice of the negative binomial likelihood.

Because the relative variance in the annualized counting statistics is high (i.e. public mass shootings are generally infrequent on an annual basis), it is also helpful to examine the model for the underlying shooting rate in detail.  Next we plot the posterior distribution of the Gaussian process for the annualized mass shooting rate simulated across a grid of timepoints subsampled between years and extending beyond the current year (2016), effectively interpolating and extrapolating from the observations.  The mean of the posterior predictive distribution of the Gaussian process is shown with the solid blue line, and the shaded region shows the 16 and 84th percentile intervals of the posterior (i.e. the "$1\sigma$ range").

```python
def plot_GP(stan_model_ext):
    y2_sum = np.percentile(np.exp(stan_model_ext['y2']), [16,50,84], axis=0)
    plt.figure(figsize=(7,5))
    pfb = plt.fill_between(data_years_samp, y2_sum[0], y2_sum[2], color='b', alpha=.5)
    pfg = plt.plot(data_years_samp, y2_sum[1], c='b', lw=2, label='GP model', zorder=0)
    po = plt.plot(data_years, stan_data['z1'], 'o', c='k', label='Observations', zorder=2)
    plt.xlabel('Year')
    plt.ylabel('Annual rate of public mass shootings')
    plt.legend(prop={'size':10}, loc=2)
    plt.ylim(0,15)
    plt.gca().xaxis.set_minor_locator(FixedLocator(np.arange(min(data_years_samp), max(data_years_samp))))
    plt.gca().set_xlim(min(data_years_samp) - 1, max(data_years_samp) + 1)
    return pfb, pfg, po

pfb, pfg, po = plot_GP(stan_model_ext_rho_strong)
```

The Gaussian process captures an increase in the mass shooting rate over the decades and some fluctuations against that trend during certain periods, as we will explore in more detail below.  The model does not show any visually apparent deviations from the evolution of the observational time series, although comparison to the data highlights several years with substantially outlying mass shooting totals (e.g. 1993 and 1999).  The extrapolated period ($>2016$) suggests a range of possible future rates of growth from the 2016 level.

We add random draws from the mean function to visualize our inferences on the long-term time evolution of the mass shooting rate.

```python
def plot_GP_mu_draws(stan_model_ext):
    plot_GP(stan_model_ext)
    N_samp = len(stan_model_ext['mu_0'])
    px = np.linspace(min(data_years_samp), max(data_years_samp), 100)
    pfms = []
    for i in range(20):
        draw = np.random.randint(0, N_samp)
        py = np.exp(stan_model_ext['mu_0'][draw] + (px - min(data_years)) * stan_model_ext['mu_b'][draw])
        pfms.append(plt.plot(px, py,  c='r',
          zorder = 1, label = 'Mean function draws' if i==0 else None))
    plt.legend(prop={'size':10}, loc=2)

plot_GP_mu_draws(stan_model_ext_rho_strong)
```

The comparison between draws of the mean functions (red) and the model posterior (blue) suggests that the mean function captures most of the modeled variation in the shooting rate over time.

We can understand the behavior of the Gaussian process covariance function by isolating it from the mean function.  We do so by subtracting the linear component of the mean function from the simulated Gaussian process rates ($y_2$) and plotting against the observations.

```python
y2_gp_rho_strong = np.percentile(np.exp(
        stan_model_ext_rho_strong['y2'] -
        np.dot(stan_model_ext_rho_strong['mu_b'][:,np.newaxis], (data_years_samp[np.newaxis,:] - min(data_years)))
          ), [16,25,50,75,84], axis=0)

fig, axs = plt.subplots(2, figsize=(7,7), sharex='all')
pfb = axs[1].fill_between(data_years_samp, y2_gp_rho_strong[1], y2_gp_rho_strong[3], color='b', alpha=.25)
pfb2 = axs[1].fill_between(data_years_samp, y2_gp_rho_strong[0], y2_gp_rho_strong[4], color='b', alpha=.25)
pfg = axs[1].plot(data_years_samp, y2_gp_rho_strong[2], c='b', lw=2, label='GP model (covariance only)', zorder=0)
po = axs[0].plot(data_years, stan_data['z1'], 'o', c='k', label='Observations', zorder=2)
axs[1].axhline(np.exp(stan_model_ext_rho_strong['mu_0'].mean()), color='orange', label='$\mu_0$')

axs[0].set_ylabel('Annual rate of \npublic mass shootings\n(observations)')
axs[1].legend(prop={'size':8}, loc=2, ncol=2)
axs[1].set_ylabel('Annual rate of \npublic mass shootings\n(model)')

axs[1].set_ylim(0, 2.2)
axs[1].xaxis.set_minor_locator(FixedLocator(np.arange(min(data_years_samp), max(data_years_samp))))
axs[1].set_xlim(min(data_years_samp) - 1, max(data_years_samp) + 1)

```

In this plot, the shaded regions show the interquartile and $[16-84]$th percentile ranges.  The fact that the interquartile contours never cross the mean ($\mu_0$) indicates that there is never $>75\%$ probability that the annualized trend deviates from the linear mean function.  However, there are times when the interquartile range approaches the mean.

Perhaps the most salient feature captured by the covariance function of the Gaussian process is a dip in the annualized rate of public mass shootings in the years from about 2000 to 2005.  The model has no features that would seek to explain the causal origin of this dip, although many readers may be surprised by its juxtoposition with the Columbine High School massacre (1999), which is understood to have spawned dozens of "copycat" attacks over time (see e.g. Follman & Andrews 2015).

The largest positive deviation from the mean function occurs between  about 1988 and 1993.  During that time, the mean function itself is very small (see previous figure), so this does not reresent a large absolute deviation.

### Gaussian process with weak $\rho^{-1}$ prior

For comparison, we visualize the latent Gaussian process under a weak prior for $\rho^{-1}$.

```python
plot_GP(stan_model_ext_rho_weak)
```

It's clear from this visualization that the Gaussian process does not capture significant short-timescale variations when the timescale prior is loosened.  This model also generally expresses lower uncertainty in the annual public mass shootings rate.  Consistent with the reliance on the parametric, linear mean function, the extrapolated predictions do not account for any substantial probability of decrease in the rate of public mass shootings after 2016.

We can see the dominance of the mean function over the covariance function directly by again visualizing the isolated Gaussian process covariance function, which shows virtually no deviation from the mean:

```python
y2_gp_rho_weak = np.percentile(np.exp(
        stan_model_ext_rho_weak['y2'] -
        np.dot(stan_model_ext_rho_weak['mu_b'][:,np.newaxis], (data_years_samp[np.newaxis,:] - min(data_years)))
          ), [16,25,50,75,84], axis=0)

fig, axs = plt.subplots(1, figsize=(7,5), sharex='all')
pfb = axs.fill_between(data_years_samp, y2_gp_rho_weak[1], y2_gp_rho_weak[3], color='b', alpha=.25)
pfb2 = axs.fill_between(data_years_samp, y2_gp_rho_weak[0], y2_gp_rho_weak[4], color='b', alpha=.25)
pfg = axs.plot(data_years_samp, y2_gp_rho_weak[2], c='b', lw=2, label='GP model (covariance only)', zorder=0)
axs.axhline(np.exp(stan_model_ext_rho_weak['mu_0'].mean()), color='orange', label='$\mu_0$')

axs.legend(prop={'size':8}, loc=2, ncol=2)
axs.set_ylabel('Annual rate of \npublic mass shootings\n(model)')
axs.set_title(r'Weak $\rho$ prior')

axs.set_ylim(0, 2.2)
axs.xaxis.set_minor_locator(FixedLocator(np.arange(min(data_years_samp), max(data_years_samp))))
axs.set_xlim(min(data_years_samp) - 1, max(data_years_samp) + 1)

```

## Inspection of posterior correlations

Before we explore the marginalized posterior distributions of the parameters in our model, we take advantage of the fully Bayesian posterior samples generated by the NUTS simulations to understand the correlations between parameters in the posterior distribution.

First we note that the parameters of the linearized mean function are highly correlated:

```python
plt.figure()
pa = plt.hist2d(stan_model_ext_rho_strong['mu_0'],
                stan_model_ext_rho_strong['mu_b'],
                bins=100, cmap=cm.Reds, cmin=4)
plt.xlabel(r'$\mu_0$ (log shootings)')
plt.ylabel(r'$\mu_b$ (log shootings per year)')
plt.axvline(0, color='k', ls='dashed')
plt.axhline(0, color='k', ls='dashed')
plt.axis([-1.5,1.5,-0.05,.1])
cb = plt.colorbar()
cb.set_label('Number of posterior samples')

```

If the mean rate of public mass shootings at the beginning of the time series ($\mu_0$) is inferred to be higher, then the increase in the mean function over time needed to explain the observations ($\mu_b$) would be lower.  However, at all probable values of $\mu_0$, the distribution of $\mu_b$ is predominantly positive.

We can fit a simple linear model to understand more subtle correlations in the multivariate posterior distribution.  Here we fit a model for $\rho^{-1}$ as a function of the other major parameters of the model.  We standardize the predictors so that we can directly compare the coefficients on the linear model.

```python
import statsmodels.api as sm

## Assemble data matrices
y = pd.Series(stan_model_ext_rho_strong['inv_rho']); y.name = 'inv_rho'
X = pd.DataFrame({
	'eta':np.sqrt(stan_model_ext_rho_strong['eta_sq']),
	'mu_0':stan_model_ext_rho_strong['mu_0'],
	'mu_b':stan_model_ext_rho_strong['mu_b'],
	'sigma':np.sqrt(stan_model_ext_rho_strong['sigma_sq']),
	'NB_phi_inv':np.sqrt(stan_model_ext_rho_strong['NB_phi_inv']),
	})
## Standardize
X = X - X.mean()
X = X / X.std()
X = sm.add_constant(X)
y = (y - y.mean()) / y.std()
## Fit linear model using stats models
est = sm.OLS(y, X).fit()
## Print summary
print est.summary2()
```

                       Results: Ordinary least squares
    ====================================================================
    Model:              OLS              Adj. R-squared:     0.054
    Dependent Variable: inv_rho          AIC:                233702.4431
    Date:               2017-01-14 23:07 BIC:                233758.4745
    No. Observations:   84000            Log-Likelihood:     -1.1685e+05
    Df Model:           5                F-statistic:        964.7
    Df Residuals:       83994            Prob (F-statistic): 0.00
    R-squared:          0.054            Scale:              0.94575
    ----------------------------------------------------------------------
                   Coef.    Std.Err.      t      P>|t|     [0.025   0.975]
    ----------------------------------------------------------------------
    const         -0.0000     0.0034   -0.0000   1.0000   -0.0066   0.0066
    NB_phi_inv     0.0170     0.0034    5.0545   0.0000    0.0104   0.0236
    eta            0.2318     0.0034   68.7066   0.0000    0.2252   0.2384
    mu_0           0.0585     0.0062    9.4253   0.0000    0.0463   0.0706
    mu_b           0.0647     0.0062   10.4458   0.0000    0.0525   0.0768
    sigma          0.0172     0.0034    5.0956   0.0000    0.0106   0.0238
    --------------------------------------------------------------------
    Omnibus:             11670.523      Durbin-Watson:         2.016
    Prob(Omnibus):       0.000          Jarque-Bera (JB):      18349.721
    Skew:                0.978          Prob(JB):              0.000
    Kurtosis:            4.189          Condition No.:         3
    ====================================================================

We see that the most significant correlation is between $\rho^{-1}$ and $\eta$.  When we visualize this correlation, we observe that the level of posterior curvature associated with these two variables is small, though significant.

```python
plt.figure()
pa = plt.hist2d(np.sqrt(stan_model_ext_rho_strong['eta_sq']),
                stan_model_ext_rho_strong['inv_rho'],
                bins=40, cmap=cm.Reds, cmin=4,
                range = [[0,1],[1,12]])
plt.xlabel(r'$\eta$ (log shootings per year)')
plt.ylabel(r'$\rho^{-1}$ (years)')
sqrt_eta = np.sqrt(stan_model_ext_rho_strong['eta_sq'])
px = np.linspace(min(sqrt_eta), max(sqrt_eta), 10)
px_std = (px - np.mean(sqrt_eta)) / np.std(sqrt_eta)
plt.plot(px,
         # Constant term
         (est.params[est.model.exog_names.index('const')] +
         # Linear term
         px * est.params[est.model.exog_names.index('eta')]
         # Standardization adjustment
         * stan_model_ext_rho_strong['inv_rho'].std()) + stan_model_ext_rho_strong['inv_rho'].mean())

plt.axis()
cb = plt.colorbar()
cb.set_label('Number of posterior samples')
plt.title(r'Strong prior on $\rho^{-1}$')
```

When we explore the same correlation in the posterior of the model with a weak prior specified on the timescale hyperparameter, we see somewhat different results:

```python
## Assemble data matrices
y = pd.Series(np.log(stan_model_ext_rho_weak['inv_rho'])); y.name = 'inv_rho'
X = pd.DataFrame({
	'eta':np.sqrt(stan_model_ext_rho_weak['eta_sq']),
	'mu_0':stan_model_ext_rho_weak['mu_0'],
	'mu_b':stan_model_ext_rho_weak['mu_b'],
	'sigma':np.sqrt(stan_model_ext_rho_weak['sigma_sq']),
	'NB_phi_inv':np.sqrt(stan_model_ext_rho_weak['NB_phi_inv']),
	})
## Standardize
X = X - X.mean()
X = X / X.std()
X = sm.add_constant(X)
y = (y - y.mean()) / y.std()
## Fit linear model using stats models
est = sm.OLS(y, X).fit()
## Print summary
print est.summary2()

plt.figure()
pa = plt.hist2d(np.sqrt(stan_model_ext_rho_weak['eta_sq']),
                stan_model_ext_rho_weak['inv_rho'],
                bins=40, cmap=cm.Reds, cmin=4,
                range = [[0,4],[1,300]])
plt.xlabel(r'$\eta$ (log shootings per year)')
plt.ylabel(r'$\rho^{-1}$ (years)')
sqrt_eta = np.sqrt(stan_model_ext_rho_weak['eta_sq'])
px = np.linspace(min(sqrt_eta), max(sqrt_eta), 10)
px_std = (px - np.mean(sqrt_eta)) / np.std(sqrt_eta)
plt.plot(px,
         # Constant term
         (est.params[est.model.exog_names.index('const')] +
         # Linear term
         px * est.params[est.model.exog_names.index('eta')]
         # Standardization adjustment
         * stan_model_ext_rho_weak['inv_rho'].std()) + stan_model_ext_rho_weak['inv_rho'].mean())

plt.axis()
cb = plt.colorbar()
cb.set_label('Number of posterior samples')
plt.title(r'Weak prior on $\rho^{-1}$')
```

Again, $\eta$ is the parameter most significantly correlated with $\rho^{-1}$, but now the 2D posterior visualization shows that this correlation is substantially non-linear.  In particular for the model with the weak prior on $\rho$, $\eta$ is constrained to much smaller values when the timescale $\rho^{-1}$ is small. In other words, in models that permit variations from the mean function on timescales smaller than the observational range ($\sim35$ years), the amplitude of those variations is constrained to be very small.  In any scenario, as we have seen, the importance of the covariance function is minimal under this prior.

## Parameter inferences

Below we show the marginalized posterior distributions of the parameters of the Gaussian process under the strong prior on $\rho$.

```python
def gt0(y, x, lbound=0, ubound=np.inf):
	y[(x<lbound) & (x>ubound)] = 0
	return y

def marg_post_plot(stan_model_ext, alpha_rho, beta_rho, Nhist=25):
    hyp_dic = {
        'eta_sq': ('$\\eta$', np.sqrt, 'log shootings per year', lambda x: sstats.cauchy.pdf(x**2, 0, 1)),
        'inv_rho': ('$\\rho^{-1}$', lambda x: x, 'years', lambda x: gt0(sstats.gamma.pdf(x, alpha_rho, scale=beta_rho), x, lbound=1)),
        'sigma_sq': ('$\\sigma$', np.sqrt, 'log shootings per year', lambda x: sstats.cauchy.pdf(x**2, 0, 1)),
        'NB_phi_inv':('$\\rm{NB}_\\phi^{-1}$', lambda x:x, '', lambda x: sstats.cauchy.pdf(x**2, 0, 0.5)),
        }

    meanfunc_dic = {
        'mu_0': ('$\\mu_0$', lambda x: x, 'log shootings per year, '+str(np.min(data_years)), lambda x: sstats.norm.pdf(x, 0,2)),
        'mu_b': ('$\\mu_b$', lambda x: x, 'annual increase in\nlog shootings per year', lambda x: sstats.norm.pdf(x, 0,0.2)),
        }

    for name,pdic in (('hyper', hyp_dic), ('meanfunc', meanfunc_dic)):
        fig,axs = plt.subplots(1,len(pdic), figsize=(2.5*len(pdic), 2.5), sharey='all')
        axs[0].set_ylabel('HMC samples ({} total)'.format(N_samp))
        for i,hyp in enumerate(pdic.keys()):
            samps = pdic[hyp][1](stan_model_ext[hyp])
            hn, hb, hp = axs[i].hist(samps, Nhist, edgecolor='none', facecolor='.5', label='Posterior samples')
            ppx = np.linspace(np.min(samps), np.max(samps), 10000)
            ppy = pdic[hyp][1]( pdic[hyp][3](ppx) )
            ## Normalize
            ppy *= len(samps) / np.sum(ppy) * len(ppy) / len(hn)
            axs[i].plot(ppx, ppy, color='b', zorder=2, label='Hyperprior')
            axs[i].xaxis.set_major_locator(MaxNLocator(3))
            axs[i].xaxis.set_minor_locator(AutoMinorLocator(3))
            axs[i].set_xlabel(pdic[hyp][0] + ' ({})'.format(pdic[hyp][2]), ha='center')
            axs[i].axvline(0, ls='dashed', color='.2')
        axs[-1].legend(prop={'size':9})

print "Strong prior on rho:"
marg_post_plot(stan_model_ext_rho_strong, stan_data_rho_strong['alpha_rho'], 1/stan_data_rho_strong['beta_rho'], Nhist=100)
```

The comparison of the posterior and prior distributions show strong evidence from the data to identify most hyperparameters.  The posterior for $\mu_0$ shows a concentration around a baseline rate of $\exp(-1)\sim0.4$ to $\exp(1)\sim 3$ public mass shootings per year at the start of the dataset, 1982, reflecting a variance, much smaller than the corresponding prior.  The negative binomial overdispersion parameter ($\phi^{-1}$) is concentrated towards very small values $\ll 1$, indicating that the Poisson disitrubtion is a good approximation to the variance in the observations.  The amplitude of the Gaussian process covariance function, $\eta$, is strongly shifted from the mode of the prior distribution, to a mean of $\exp(0.5)\sim1.6$ public mass shootings per year.  The variance of the Gaussian process covariance function, $\sigma$, has a posterior variance much smaller than the prior distribution.

The posterior distribution of $\rho^{-1}$ is a notable exception.  It shows no visual deviation from the prior distribution, indicating that this parameter is not identified by the observations.

Next we explore the same marginalized posteriors under the weak prior on $\rho$.

```python
print "Weak prior on rho:"
marg_post_plot(stan_model_ext_rho_weak, stan_data_rho_weak['alpha_rho'], 1/stan_data_rho_weak['beta_rho'], Nhist=100)
```

With the weak prior on $\rho$, most parameters have posterior distributions nearly identical to their distributions under the strong prior on $\rho$.  In particular, the conclusions about the mean function parameters ($\mu_0$ and $\mu_b$), $\phi$, and $\sigma$ seem robust to the choice of prior.

Importantly, the $\rho$ parameter is again largely non-identified.  Its posterior distribution generally follows the weaker prior, although it shows a posterior probability less than the prior for the very smallest values.  The consequence is that the models sampled from the Gaussian process have very long timescales for their covariance function.  The distribution of the amplitude, $\eta$, is skewed to larger values under the weaker prior, although the amplitude of the mean function has little consequence when the time variation is negligable (as discussed in the previous section).

## Model predictions

We calculate the posterior probability that the annualized rate of public mass shootings has increased in the US since 1982 ($\mu_b > 0$).

```python
print_ext_names = ['...with strong prior on rho: ', '...with weak prior on rho: ']

print 'p(mu_b > 0):'
for i in range(2):
    print print_ext_names[i]+'%0.0f'%(np.mean(exts[i]['mu_b'] > 0)*100)+'%'

```

    p(mu_b > 0):
    ...with strong prior on rho: 97%
    ...with weak prior on rho: 97%

This indicates strong statistical evidence for a long term increase in the annualized rate of public mass shootings over the past three decades, regardless of our choice of prior for the timescale parameter, $\rho$.  In linear terms, the mean percentage increase in the rate of public mass shootings is found to be,

```python
zincreaseraw = {}
for i in range(2):
    zincreaseraw[i] = (np.exp((2016 - np.min(data_years)) * exts[i]['mu_b']) - 1) * 100
    zincrease = np.percentile(zincreaseraw[i], [16,50,84])
    print print_ext_names[i]+'%0.0f'%round(zincrease[1], -1)+'^{+%0.0f'%round(zincrease[2]-zincrease[1], -1)+'}_{-%0.0f'%round(zincrease[1]-zincrease[0], -1)+'}'

```

    ...with strong prior on rho: 350^{+500}_{-230}
    ...with weak prior on rho: 360^{+490}_{-230}

While the uncertainty interval is large, the $1\sigma$ estimate suggests at least a doubling in the annualized rate of public mass shootings over these three decades, and more likely a quadrupling or greater increase.

For comparison, the US population has grown from $\sim231$ million to $318$ million residents according to [worldbook data](http://data.worldbank.org/indicator/SP.POP.TOTL?cid=GPD_1), an increase of $38\%$, over that same period.  The model posterior suggests that the rate of public mass shootings has surpassed the rate of population growth with high confidence:

```python
for i in range(2):
    print print_ext_names[i]+'%0.0f'%(np.mean(zincreaseraw[i] > 38)*100)+'%'
```

    ...with strong prior on rho: 94%
    ...with weak prior on rho: 94%

Cohen et al. (2014) reported a tripling in the rate of mass shootings between 2011 and 2014 on the basis of a SPC methodology.  Our inference on the mean function of the Gaussian process, because it is parameterized as linear over the full time extent of the modeled period, does not directly address this claim.  But the simulated predictions of the Gaussian process, including the covariance component, can generate relevant comparisons.

```python
i1 = np.argmin(abs(data_years_samp - 2011.5))
i2 = np.argmin(abs(data_years_samp - 2014.5))
py = np.exp(stan_model_ext_rho_strong['y2'][:,i2]) / np.exp(stan_model_ext_rho_strong['y2'][:,i1])

plt.figure()
ph = plt.hist(py, 50, edgecolor='none', facecolor='.5', range=[0,8], normed=1)
plt.xlabel('Relative rate of public mass shootings in 2014 versus 2011')
plt.ylabel('Posterior probability')
plt.axvline(1, color='k', label='Unity')
plt.axvline(np.mean(py), color='b', label='Mean posterior estimate', ls='dashed')
plt.axvline(3, color='g', label='Cohen et al. estimate', lw=2, ls='dotted')
plt.legend()

print "Probability that rate increased: ", '%0.0f'%(np.mean(py > 1) * 100), '%'
print "Mean predicted level of increase: ", '%0.1f'%(np.mean(py)), 'X'
print "Probability of increase by at least 3X: ", '%0.2f'%(np.mean(py > 3)), '%'
```

While we have reported that the increase in the rate of public mass shootings over the past three decades is likely to be a factor of several, we find much less evidence for such a dramatic increase over the time period from 2011 to 2014. As reported above, our model predicts better than even odds that there was an increase during that three year period, but the probability that it was as high as a tripling are small. Our model suggests that the increase was more likely to be $\sim30\%$ .

## Conclusions

We have used Stan to implement and estimate a negative binomial regression model for the annualized rate of public mass shootings in the United States based on a Gaussian process with a time-varying mean function.  When combined with a strong prior on the timescale of the Gaussian process covariance function, this design yields a predictive model with the full non-parametric flexibility of a Gaussian process to explain short timescale variation, while retaining the clear interpretability of a parametric model by isolating and jointly modeling long-term (timescale of decades) evolution in the shooting rate.  Applying this model to the Mother Jones database of US public mass shootings, our major conclusions are as follows,

* We use posterior simulations and predictive checks to demonstrate the efficacy of the Gaussian process model in generating and fitting the observations of the annual mass shooting rate from the Mother Jones database.  We explore the effects of prior choices on the posterior and visualize posterior curvature between hyperparameters.
* We use the non-parametric Gaussian process predictive model to identify an apparent dip in the mass shooting rate in the first few years of the new millenium.
* With a 97% probability, we find that the annualized rate of public mass shootings has risen over the past three decade.  This finding is robust to the choice of prior on the timescale parameter.
* The posterior mean estimate for the increase in the shooting rate since 1982 is $\sim300\%$.
* We compare to an independent, 2014 analysis of the increase in the rate of public mass shootings between 2011 and 2014 by Cohen et al.  Our model predicts a smaller rate of increase for the underlying rate of public mass shootings than those authors over this period, closer to 30% over the 4 year period.

## Acknowledgements

The authors would like to thank the anonymouse reviewers, Michael Betancourt, Amy Cohen, Deb Azrael, and Matt Miller for very helpful feedback on this manuscript and the statistical techniques applied.

## References
* Betancourt & Girolami 2013: Michael Betancourt &  Mark Girolami 2013, [Hamiltonian Monte Carlo for Hierarchical Models](https://arxiv.org/abs/1312.0906)
* Cohen et al. 2014: Amy Cohen et al. 2014, [Rate of public mass shootings Has Tripled Since 2011, Harvard Research Shows](http://www.motherjones.com/politics/2014/10/mass-shootings-increasing-harvard-research)
* Follman & Andrews 2015: Mark Follman and Becca Andrews 2015, [How Columbine Spawned Dozens of Copycats](http://www.motherjones.com/politics/2015/10/columbine-effect-mass-shootings-copycat-data)
* Fox & Levin 2015: James Alan Fox and Jack Levin 2015, Mass confusion concerning mass murder, The Criminologist, Vol. 40, No. 1
* Johnson et al. 2012: Johnson et al. 2012, [Who Is Gun Advocate John Lott?](http://mediamatters.org/research/2012/12/17/who-is-gun-advocate-john-lott/191885) http://mediamatters.org/research/2012/12/17/who-is-gun-advocate-john-lott/191885
* Lott 2014: John R. Lott 2014, [The FBI's Misrepresentation of the Change in Mass Public Shootings](http://dx.doi.org/10.2139/ssrn.2524731)
* Papaspiliopoulos et al. 2003: Non-Centered Parameterisations for Hierarchical Models and Data Augmentation in Bayesian Statistics 7, Oxford University Press, p. 307–326.
* Rasmussen & Williams 2006: Carl Edward Rasmussen & CKI Williams 2006, [Gaussian processes for machine learning](http://www.gaussianprocess.org/gpml/)
* Roberts et al. 2012: S. Roberts et al 2012, [Gaussian processes for time-series modelling](http://www.robots.ox.ac.uk/~sjrob/Pubs/Phil.%20Trans.%20R.%20Soc.%20A-2013-Roberts-.pdf)
* *Stan* Manual: [Stan Modeling Language Users Guide and Reference Manual](http://mc-stan.org/), Version 2.8.0

2017/Contributed-Talks/09_sanders/Annualized Rate of Mass Shootings; Sanders & Lei (StanCon2017 - revised).ipynb


# Lecture 7, direct methods for constrained optimization

Direct methods for constrained optimization are also known as *methods of feasible directions*

## Feasible descent directions

Let $S\subset \mathbb R^n$ ($S\neq \emptyset$ closed) and $x^*\in S$.
**Definition:** The set
$$ D = \{d\in \mathbb R^n: d\neq0,x^*+\alpha d\in S \text{ for all } \alpha\in (0,\delta) \text{ for some } \delta>0\}$$
is called the cone of feasible directions of $S$ in $x^*$.

**Definition:** The set
$$ F = \{d\in \mathbb R^n: f(x^*+\alpha d)<f(x^*)\text{ for all } \alpha\in (0,\delta) \text{ for some } \delta>0\}$$
is called the cone of descent directions.

**Definition:** The set $F\cap D$ is called the cone of feasible descent directions.

**(Obvious) Theorem:** Consider an optimization problem
$$
\begin{align}
\min &\  f(x)\\
\text{s.t. }&\ x\in S
\end{align}
$$
and let $x^*\in S$. Now if $x^*$ is a local minimizer **then** the set of feasible descent directions $F\cap D$ is empty.

## Idea for the methods of feasible descent directions

1. Assume a feasible solution $x$.
2. Find a feasible descent direction $d\in D\cap F$.
3. Determine the step length to the direction $d$
4. Update $x$ accordingly.

# Rosen's projected gradient method

Assume a problem with linear equality constraints
$$
\min f(x)\\
\text{s.t. }Ax=b,
$$
where $A$ is a $lxn$ matrix ($l\leq n$) and $b$ is a vector.

Let $x$ be a feasible solution to the above problem.

It holds that:

$d$ is a feasible direction *if and only if* $Ad=0$

Thus, the gradient $\nabla f(x)$ is a feasible descent direction, if
$$ A\nabla f(x)=0.$$

This may or may not be true.

However, we can project the gradient to the set of feasible descent directions
$$ \{d\in \mathbb R^n: Ad=0\},$$
which now is a linear subspace.

### Projection

Let $a\in \mathbb R^n$ be a vector and let $L$ be a linear subspace of $\mathbb R^n$. Now, the following are equivalent
* $a^P$ is the projection of $a$ on $L$,
* $\{a^P\} = \operatorname{argmin}_{l\in L}\|a-l\|$, and
* $a^P\in A$ and $(a-a^P)^Tl=0$ for all $l\in L$.

## Projected gradient

The projection of the gradient $\nabla f(x)$ on the set $\{d\in \mathbb R^n: Ad=0\}$ is denoted by $\nabla f(x)^P$ and called the *projected gradient*.

Now, given some conditions, the projected gradient gives us a feasible descent direction.

## How to compute the projected gradient?

There are different ways, but at this course we can use optimization. Basically, the optimization problem that we have to solve is
$$
\min \|\nabla f(x)-d\|\\
\text{s.t. }Ad=0.
$$

Since it is equivalent to minimize the square of the objective function $\sum_{i=n}\nabla_i f(x)^2+d_i^2-2\nabla_i f(x)d_i$, we can see that the problem is a quadratic problem with inequality constraints,
$$
\min \frac12 d^TId-\nabla f(x)^Td\\
\text{s.t. }Ad=0
$$
which means that we just need to solve the system of equations (see e.g., https://en.wikipedia.org/wiki/Quadratic_programming#Equality_constraints)
$$
\left[
\begin{array}{cc}
I&A^T\\
A&0
\end{array}
\right]
\left[\begin{align}d\\\lambda\end{align}\right]
= \left[
\begin{array}{c}
\nabla f(x)\\
0
\end{array}
\right],
$$
where I is the identity matrix, and $\lambda$ are the KKT multipliers.

### Code in Python

#### A function for projecting a vector to a linear space defined by $Ax=0$.

```python
import numpy as np
def project_vector(A,vector):
    #convert A into a matrix
    A_matrix = np.matrix(A)
    #construct the "first row" of the matrix [[I,A^T],[A,0]]
    left_matrix_first_row = np.concatenate((np.identity(len(vector)),A_matrix.transpose()), axis=1)
    #construct the "second row" of the matrix
    left_matrix_second_row = np.concatenate((A_matrix,np.matrix(np.zeros([len(A),len(vector)+len(A)-len(A[0])]))), axis=1)
    #combine the whole matrix by combining the rows
    left_matrix = np.concatenate((left_matrix_first_row,left_matrix_second_row),axis = 0)
    #Solve the system of linear equalities from the previous page
    return np.linalg.solve(left_matrix, \
                           np.concatenate((np.matrix(vector).transpose(),\
                                           np.zeros([len(A),1])),axis=0))[:len(vector)]
```

```python
A = [[1,0,0],[0,1,0]]
gradient = [1,1,1]
project_vector(A,gradient)
```

    matrix([[ 0.],
            [ 0.],
            [ 1.]])

# Example

Let us study optimization problem
$$
\begin{align}
\min \qquad& x_1^2+x_2^2+x_3^2\\
\text{s.t.}\qquad &x_1+x_2=3\\
    &x_1+x_3=4.
\end{align}
$$
Let us project a gradient from a feasible point $x=(1,2,3)$

Now, the matrix
$$
A = \left[
\begin{array}{ccc}
1& 1 & 0\\
1& 0 & 1
\end{array}
\right]
$$.

```python
import ad
A = [[1,1,0],[1,0,1]]
gradient = ad.gh(lambda x:x[0]**2+x[1]**2+x[2]**2)[0]([1,2,3])
d = project_vector(A,[-i for i in gradient])
print d
```

    [[ 2.66666667]
     [-2.66666667]
     [-2.66666667]]

### d is a feasible direction

```python
np.matrix(A)*d
```

    matrix([[  0.00000000e+00],
            [  4.44089210e-16]])

### d is a descent direction

```python
def f(x):
    return x[0]**2+x[1]**2+x[2]**2
alpha = 0.001
print "Value of f at [1,2,3] is "+str(f([1,2,3]))
x_mod= np.array([1,2,3])+alpha*np.array(d).transpose()[0]
print "Value of f at [1,2,3] +alpha*d is "+str(f(x_mod))
print "Gradient dot product direction (i.e., directional derivative) is " \
+ str(np.matrix(ad.gh(f)[0]([1,2,3])).dot(np.array(d)))
```

    Value of f at [1,2,3] is 14
    Value of f at [1,2,3] +alpha*d is 13.978688
    Gradient dot product direction (i.e., directional derivative) is [[-21.33333333]]

## Finally, the algorithm of the projected gradient

```python
import numpy as np
import ad
def projected_gradient_method(f,A,start,step,precision):
    f_old = float('Inf')
    x = np.array(start)
    steps = []
    f_new = f(x)
    while abs(f_old-f_new)>precision:
        f_old = f_new
        gradient = ad.gh(f)[0](x)
        grad_proj = project_vector(A,[-i for i in gradient])#The only changes to steepest..
        grad_proj = np.array(grad_proj.transpose())[0] #... descent are here!
#        import pdb; pdb.set_trace()
        x = x+grad_proj*step
        f_new = f(x)
        steps.append(list(x))
    return x,f_new,steps
```

```python
f = lambda x:x[0]**2+x[1]**2+x[2]**2
A = [[1,1,0],[1,0,1]]
start = [1,2,3]
(x,f_val,steps) = projected_gradient_method(f,A,start,0.6,0.000001)
```

```python
print x
print f(x)
print f([1,2,3])
print np.matrix(A)*np.matrix(x).transpose()
```

    [ 2.333248  0.666752  1.666752]
    8.66666668851
    14
    [[ 3.]
     [ 4.]]

Lecture 7, Direct methods for constrained optimization.ipynb


# Кластерный анализ
## Метод к-средних

Дана матрица данных $X$ и дано число $k$ предполагаемых кластеров. Цель кластеризации представить данные в виде групп кластеров $C=\{C_1, C_2, \ldots, C_k\}$. Каждый кластер имеет свой центр:

\begin{equation}
\mu_i = \frac{1}{n_i} \sum \limits_{x_j \in C_i} x_j
\end{equation}

где $n_i = |C_i|$ - это количество точек в кластере $C_i$.

Таким образом, мы имеем некоторые кластеры $C=\{C_1, C_2, \ldots, C_k\}$ и необходимо оценить качество разбиения. Для этого будем вычислять сумму квадратических ошибок (SSE):

\begin{equation}
SSE(C) = \sum \limits_{i=1}^{k} \sum \limits_{x_j \in C_i} ||x_j - \mu_i||^2
\end{equation}

Цель найти
\begin{equation}
C^* = arg\min\limits_C \{SSE(C)\}
\end{equation}

### Алгоритм к-средних
На вход алгоритм получает матрицу данных $D$, количество кластеров $k$, и критерий остановки $\epsilon$:
1. t = 0
2. случайным образом инициализируем $k$ центров кластеров: $\mu_1^t, \mu_2^t, \ldots, \mu_k^t \in R^d$;
3. повторять
    4. $t = t + 1$;
    5. $C_j = 0$ для всех $j = 1, \ldots, k$
    6. для каждого $x_j \in D$
        7. $j^* = arg\min\limits_C \{||X_j - \mu_i^{t-1}||^2\}$ \\\ присваиваем $x_j$ к ближайшему центру
        8. $C_{j^*} = C_{j^*} \cup {x_j}$
    9. для каждого i=1 до k
        10. $\mu_i = \frac{1}{|C_i|} \sum_{x_j \in C_i} x_j$
11. пока $\sum_{i=1}^k ||\mu_i^{t} - \mu_i^{t-1}||^2 \leq \epsilon$

## Задание
1. Написать программу, реализующую алгоритм к-средних.
2. Визуализировать сходимость центров кластеров.
3. Оценить $SSE$ для значений $k = 1, \ldots, 10$ и построить график зависимости $SSE$ от количества кластеров.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.spatial import distance
from sklearn.datasets import make_blobs
X, Y = make_blobs(n_samples = 1000, n_features=2, centers=5, cluster_std = 1.2, random_state=17)
plt.scatter(X[:,0], X[:,1])
plt.show()
```

```python
def get_center(cluster):
    return np.array([cluster[:,0].mean(), cluster[:,1].mean()])

def get_dist(point_1, point_2):
    return np.linalg.norm(point_1 - point_2)

def get_clusters(X, centers):
    clusters = dict()
    for point in X:
        best_cluster = min([(index, get_dist(point, center)) for index, center in enumerate(centers)],
                          key=lambda t:t[1])[0]
        try:
            clusters[best_cluster].append(point)
        except KeyError:
            clusters[best_cluster] = [point]
    return {key: np.array(clusters[key]) for key in clusters}

def get_new_value_centers(clusters):
    new_centers = list()
    for key in clusters:
        new_centers.append(np.mean(clusters[key], axis = 0))
    return np.array(new_centers)

def get_cluster_plt(clusters):
    for key in clusters:
        plt.scatter(clusters[key][0:,0], clusters[key][0:,1])
    plt.show()

def k_means(X, k, early_stop, show_clusters=False):
    centers = np.array([get_center(cluster) for cluster in np.array_split(X, k)])
    eps = float("inf")
    while eps >= early_stop:
        clusters = get_clusters(X, centers)
        if show_clusters: get_cluster_plt(clusters)
        new_centers = get_new_value_centers(clusters)
        eps = sum([get_dist(centers[index], new_centers[index]) for index, _ in enumerate(new_centers)])
        centers = new_centers
    return clusters
```

Сходимость центров кластеров

```python
clusters = k_means(X, 4, 0.1, show_clusters=True)
```

Оценка SSE

```python
def k_means_centers(X, k, early_stop, show_clusters=False):
    centers = np.array([get_center(cluster) for cluster in np.array_split(X, k)])
    eps = float("inf")
    while eps >= early_stop:
        clusters = get_clusters(X, centers)
        if show_clusters: get_cluster_plt(clusters)
        new_centers = get_new_value_centers(clusters)
        eps = sum([get_dist(centers[index], new_centers[index]) for index, _ in enumerate(new_centers)])
        centers = new_centers
    return centers
```

```python
def get_sse(data,clusters):
    error = 0
    for j in range(len(data)):
        min_cluster_distance = min([distance.euclidean(clusters[i],data[j]) for i in range(len(clusters))])
        error = error + min_cluster_distance ** 2
    return error

k = list(range(1,11))
k_sse = [get_sse(X,k_means_centers(X,k[i],0.1)) for i in range(len(k))]

plt.plot(k, k_sse)
plt.gcf().set_size_inches(18.5, 10.5)
```

## Реальные данные

используйте метод KMeans из sklearn.clustering

1. Выбрать оптимальное количество кластеров.
2. Построить
2. Произвести анализ получившихся кластеров:
    1. определите средний год автомобилей;
    2. определите средний пробег автомобилей;
    3. определите среднюю мощность;
    4. определите среднюю цену автомобилей;
    5. основные марки автомобилей в кластере;
    6. определите тип топлива;
    7. определите основной тип кузова;
    8. определите основной тип привода;
    9. определите основной тип КПП;
    10. определите количество хозяев автомобиля.
Охарактеризуйте каждый класстер.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv', encoding='cp1251')
df = df.drop(columns=['Модель', 'Цвет'])
df.head()
```

      Марка
      Год
      Состояние
      Пробег
      Объем
      Топливо
      Мощность
      Кузов
      Привод
      КПП
      Руль
      Хозяев в ПТС
      Цена

      0
      Volkswagen
      2013.0
      БУ
      42000.0
      1200.0
      бензин
      105.0
      хэтчбек
      передний
      автомат
      левый
      1 владелец
      689196.0

      1
      Skoda
      2012.0
      БУ
      62000.0
      1800.0
      бензин
      152.0
      кроссовер
      полный
      механика
      левый
      1 владелец
      639196.0

      2
      Renault
      2015.0
      БУ
      4700.0
      1600.0
      бензин
      106.0
      хэтчбек
      передний
      механика
      левый
      1 владелец
      629196.0

      3
      Nissan
      2012.0
      БУ
      70000.0
      1600.0
      бензин
      110.0
      хэтчбек
      передний
      автомат
      левый
      1 владелец
      479196.0

      4
      УАЗ
      2014.0
      БУ
      50000.0
      2700.0
      бензин
      128.0
      внедорожник
      полный
      механика
      левый
      1 владелец
      599196.0

```python
new_df = pd.get_dummies(df)
new_df.head()
```

      Год
      Пробег
      Объем
      Мощность
      Цена
      Марка_Acura
      Марка_Alfa Romeo
      Марка_Audi
      Марка_BMW
      Марка_BYD
      ...
      Привод_полный
      КПП_автомат
      КПП_вариатор
      КПП_механика
      КПП_роботизированная
      Руль_левый
      Руль_правый
      Хозяев в ПТС_1 владелец
      Хозяев в ПТС_2 владельца
      Хозяев в ПТС_3 и более

      0
      2013.0
      42000.0
      1200.0
      105.0
      689196.0
      0
      0
      0
      0
      0
      ...
      0
      1
      0
      0
      0
      1
      0
      1
      0
      0

      1
      2012.0
      62000.0
      1800.0
      152.0
      639196.0
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      1
      0
      1
      0
      1
      0
      0

      2
      2015.0
      4700.0
      1600.0
      106.0
      629196.0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      1
      0
      1
      0
      1
      0
      0

      3
      2012.0
      70000.0
      1600.0
      110.0
      479196.0
      0
      0
      0
      0
      0
      ...
      0
      1
      0
      0
      0
      1
      0
      1
      0
      0

      4
      2014.0
      50000.0
      2700.0
      128.0
      599196.0
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      1
      0
      1
      0
      1
      0
      0

5 rows × 126 columns

```python
from sklearn.preprocessing import StandardScaler
ss = StandardScaler()
new_df[['Год', 'Пробег', 'Объем', 'Мощность', 'Цена']] = ss.fit_transform(new_df[['Год', 'Пробег', 'Объем', 'Мощность', 'Цена']])
new_df.head()
```

      Год
      Пробег
      Объем
      Мощность
      Цена
      Марка_Acura
      Марка_Alfa Romeo
      Марка_Audi
      Марка_BMW
      Марка_BYD
      ...
      Привод_полный
      КПП_автомат
      КПП_вариатор
      КПП_механика
      КПП_роботизированная
      Руль_левый
      Руль_правый
      Хозяев в ПТС_1 владелец
      Хозяев в ПТС_2 владельца
      Хозяев в ПТС_3 и более

      0
      1.055883
      -1.129295
      -1.096928
      -0.458895
      0.420103
      0
      0
      0
      0
      0
      ...
      0
      1
      0
      0
      0
      1
      0
      1
      0
      0

      1
      0.868335
      -0.842782
      -0.125615
      0.440164
      0.315187
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      1
      0
      1
      0
      1
      0
      0

      2
      1.430978
      -1.663641
      -0.449386
      -0.439766
      0.294203
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      1
      0
      1
      0
      1
      0
      0

      3
      0.868335
      -0.728177
      -0.449386
      -0.363250
      -0.020546
      0
      0
      0
      0
      0
      ...
      0
      1
      0
      0
      0
      1
      0
      1
      0
      0

      4
      1.243431
      -1.014690
      1.331355
      -0.018930
      0.231254
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      1
      0
      1
      0
      1
      0
      0

5 rows × 126 columns

```python
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN, AffinityPropagation
inertia = []
for it in np.arange(1,20,1):
    method = KMeans(n_clusters=it)
    method.fit(new_df)
    inertia.append(method.inertia_)
    print(it)
```

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19

```python
for i in range(len(inertia)):
    print(i+1,inertia[i])
```

    1 303666.09226419945
    2 238066.35809271294
    3 203215.59994571476
    4 188096.71848079565
    5 176308.26383170072
    6 166868.51967859885
    7 160779.23246553304
    8 155563.4221062198
    9 150434.37152236237
    10 146538.95858094987
    11 143247.76194929198
    12 140234.2185084517
    13 137206.36955768696
    14 134389.44205518265
    15 132121.6934500015
    16 129743.02831074157
    17 127996.209894227
    18 126181.09337907878
    19 124530.70736049648

Оптимальное количество кластеров 15

```python
def cls_info(df, k):
    print('------ Кластер ', k, ' -------')
    claster =  df[method.labels_ == k]
    print(claster['Год'].mean())
    print(claster['Пробег'].mean())
    print(claster['Объем'].mean())
    print(claster['Мощность'].mean())
    print(claster['Цена'].mean())
    print(claster['Привод'].value_counts().head(1))
    print(claster['Марка'].value_counts().head(2))
    print(claster['Кузов'].value_counts().head(2))
    print(claster['КПП'].value_counts().head(2))
    print(claster['Хозяев в ПТС'].value_counts().head(2))
    print('---------------------------')
```

```python
print('------ Реальность -------')
print(df['Год'].mean())
print(df['Пробег'].mean())
print(df['Объем'].mean())
print(df['Мощность'].mean())
print(df['Цена'].mean())
print(df['Привод'].value_counts().head(1))
print(df['Марка'].value_counts().head(2))
print(df['Кузов'].value_counts().head(2))
print(df['КПП'].value_counts().head(2))
print(df['Хозяев в ПТС'].value_counts().head(2))
print('---------------------------')
```

    ------ Реальность -------
    2007.3700562551758
    120830.33855906794
    1877.595019846369
    128.98960564265113
    488987.6818298638
    передний    23933
    Name: Привод, dtype: int64
    ВАЗ       5497
    Toyota    2700
    Name: Марка, dtype: int64
    седан      14241
    хэтчбек     8462
    Name: Кузов, dtype: int64
    механика    19308
    автомат     13935
    Name: КПП, dtype: int64
    3 и более      12613
    2 владельца    12070
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------

```python
for it in range(16):
    cls_info(df, it)
```

    ------ Кластер  0  -------
    2012.1751925192518
    63387.76347634764
    1542.7117711771177
    104.2871287128713
    432276.9573707371
    передний    3525
    Name: Привод, dtype: int64
    ВАЗ        684
    Renault    345
    Name: Марка, dtype: int64
    седан      1675
    хэтчбек    1113
    Name: Кузов, dtype: int64
    механика            3556
    роботизированная      60
    Name: КПП, dtype: int64
    1 владелец    3519
    3 и более       75
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  1  -------
    1998.067221067221
    72930.8024948025
    1722.3146223146223
    84.61607761607762
    86363.76507276508
    задний    554
    Name: Привод, dtype: int64
    ВАЗ    889
    УАЗ    143
    Name: Марка, dtype: int64
    седан          730
    внедорожник    323
    Name: Кузов, dtype: int64
    механика    1383
    автомат       60
    Name: КПП, dtype: int64
    3 и более      757
    2 владельца    468
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  2  -------
    2012.8614298323037
    60326.43865842895
    2132.8773168578996
    171.48455428067078
    1227472.0706090026
    полный    1404
    Name: Привод, dtype: int64
    Toyota    344
    Audi      203
    Name: Марка, dtype: int64
    кроссовер    1228
    седан         552
    Name: Кузов, dtype: int64
    автомат     1815
    вариатор     229
    Name: КПП, dtype: int64
    1 владелец     1587
    2 владельца     590
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  3  -------
    1999.6247086247085
    237500.89393939395
    2937.5291375291376
    191.7937062937063
    407217.94755244756
    полный    506
    Name: Привод, dtype: int64
    Mercedes-Benz    142
    BMW              111
    Name: Марка, dtype: int64
    седан          326
    внедорожник    279
    Name: Кузов, dtype: int64
    автомат     717
    механика    133
    Name: КПП, dtype: int64
    3 и более      563
    2 владельца    235
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  4  -------
    2008.9737206085754
    98074.15952051636
    1448.6399262332873
    95.61180267404333
    298626.3042876902
    передний    2132
    Name: Привод, dtype: int64
    ВАЗ     520
    Ford    235
    Name: Марка, dtype: int64
    хэтчбек        2138
    внедорожник       7
    Name: Кузов, dtype: int64
    механика    1736
    автомат      294
    Name: КПП, dtype: int64
    2 владельца    2168
    1 владелец        1
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  5  -------
    2008.9607907742998
    108518.84349258649
    1605.7001647446457
    106.36507413509061
    326849.65667215816
    передний    2910
    Name: Привод, dtype: int64
    ВАЗ          495
    Chevrolet    268
    Name: Марка, dtype: int64
    седан        2368
    универсал     238
    Name: Кузов, dtype: int64
    механика    2757
    автомат      194
    Name: КПП, dtype: int64
    2 владельца    3035
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  6  -------
    2002.2536157024792
    193473.40857438016
    1476.9111570247933
    82.2448347107438
    98850.0444214876
    передний    1843
    Name: Привод, dtype: int64
    ВАЗ       1406
    Daewoo      62
    Name: Марка, dtype: int64
    седан      918
    хэтчбек    783
    Name: Кузов, dtype: int64
    механика    1914
    автомат       15
    Name: КПП, dtype: int64
    3 и более      1259
    2 владельца     491
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  7  -------
    2011.2364380757422
    91274.27328556807
    3325.793244626407
    256.89252814738995
    1657216.7727737974
    полный    909
    Name: Привод, dtype: int64
    Toyota    183
    BMW       150
    Name: Марка, dtype: int64
    кроссовер      440
    внедорожник    345
    Name: Кузов, dtype: int64
    автомат     931
    вариатор     24
    Name: КПП, dtype: int64
    2 владельца    431
    1 владелец     430
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  8  -------
    2009.939024390244
    72631.38455284553
    2067.967479674797
    105.95365853658537
    418875.9130081301
    полный    1204
    Name: Привод, dtype: int64
    УАЗ          344
    Chevrolet    289
    Name: Марка, dtype: int64
    внедорожник    1038
    кроссовер        95
    Name: Кузов, dtype: int64
    механика    1215
    автомат       12
    Name: КПП, dtype: int64
    1 владелец     596
    2 владельца    433
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  9  -------
    2007.0385576579793
    136103.9032488397
    1629.9892895394503
    106.0124955373081
    262912.8768297037
    передний    2686
    Name: Привод, dtype: int64
    ВАЗ     370
    Ford    294
    Name: Марка, dtype: int64
    седан        2489
    универсал     116
    Name: Кузов, dtype: int64
    механика    2465
    автомат      249
    Name: КПП, dtype: int64
    3 и более     2475
    1 владелец     326
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  10  -------
    2006.8225130890053
    145801.34764397907
    2130.418848167539
    165.565445026178
    513192.24659685866
    передний    1478
    Name: Привод, dtype: int64
    Toyota    243
    BMW       167
    Name: Марка, dtype: int64
    седан      1502
    хэтчбек     129
    Name: Кузов, dtype: int64
    автомат     1589
    механика     179
    Name: КПП, dtype: int64
    3 и более      1009
    2 владельца     704
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  11  -------
    2007.657043305127
    138630.5719263315
    2279.5420607267297
    162.04828272772522
    671646.3285216526
    полный    1892
    Name: Привод, dtype: int64
    Nissan     256
    Hyundai    249
    Name: Марка, dtype: int64
    кроссовер      1178
    внедорожник     450
    Name: Кузов, dtype: int64
    автомат     1238
    механика     627
    Name: КПП, dtype: int64
    2 владельца    967
    3 и более      709
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  12  -------
    1998.7640871525168
    258800.57325319308
    1972.1262208865514
    123.88354620586026
    209069.97595792636
    передний    854
    Name: Привод, dtype: int64
    Volkswagen       117
    Mercedes-Benz    107
    Name: Марка, dtype: int64
    седан      813
    хэтчбек    132
    Name: Кузов, dtype: int64
    механика    1069
    автомат      245
    Name: КПП, dtype: int64
    3 и более      949
    2 владельца    282
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  13  -------
    2006.2400306748466
    153372.99386503067
    3458.128834355828
    258.6296012269939
    755522.2369631901
    полный    1070
    Name: Привод, dtype: int64
    BMW              201
    Mercedes-Benz    143
    Name: Марка, dtype: int64
    кроссовер    613
    седан        320
    Name: Кузов, dtype: int64
    автомат     1210
    вариатор      68
    Name: КПП, dtype: int64
    3 и более      628
    2 владельца    495
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  14  -------
    1997.5278246205733
    249334.54890387857
    1879.4266441821248
    127.0725126475548
    195677.2639123103
    передний    771
    Name: Привод, dtype: int64
    Toyota    464
    Nissan    236
    Name: Марка, dtype: int64
    седан      585
    минивэн    231
    Name: Кузов, dtype: int64
    автомат     988
    механика    120
    Name: КПП, dtype: int64
    3 и более      829
    2 владельца    235
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------
    ------ Кластер  15  -------
    2007.2490660024907
    118465.71731008717
    1459.194686591947
    92.75425487754255
    233711.92818596927
    передний    2355
    Name: Привод, dtype: int64
    ВАЗ     808
    Ford    284
    Name: Марка, dtype: int64
    хэтчбек      2276
    универсал      66
    Name: Кузов, dtype: int64
    механика    1886
    автомат      363
    Name: КПП, dtype: int64
    3 и более     2108
    1 владелец     301
    Name: Хозяев в ПТС, dtype: int64
    ---------------------------

Для задачи классификации  исходных данных я посчитал оптимальным 15 кластеров. Каждый кластер описывает новые данные об автомобилях, однако начиная с 15 отличия в кластерах менее очевидны (если 4 и 9 отличаются довольно сильно, то сходства между 9 и 15 гораздо больше)

## Работу выполнил
---
Студент группы **РИМ-181226**

Кабанов Евгений Алексеевич


# MathJax for Mathematical Display in Jupyter Notebook

## 公式展示的方式

1. **行内模式**

行内模式用于在行文间渲染公式，使用成对的`$`符号将需要在行内渲染的表达式或字符包裹起来，比如`$x=2$` $x=2$ 是 $(x-2)(x-3)=0$ 的一个解。

这里，$x=2$ 和 $(x-2)(x-3)=0$ 两个公式被插入到了文字之间。

2. **展示模式**

展示模式用于显示整段的公式。

比如，

 ```latex
$$
P(A|B)=\frac{P(AB)}{P(B)} = \frac{P(B|A)P(A)}{P(B)}
$$
```
渲染出单行表达式：

$$
P(A|B)=\frac{P(AB)}{P(B)} = \frac{P(B|A)P(A)}{P(B)}
$$

---

## 上标和下标

直接在公式中使用数学中的次方符号`^`，便可渲染为上标，就像 $2x^2=0$ ，或者$e^{x^2}$

❗在 $e^{x^2}$ 中，指数部分为一个表达式，并非使用括号`（ ）`包裹表达式部分，而是使用`{ }`表示表达式的开始和结束。

下标需要使用下划线`_`表示，就像 $x_2$

#### 上标
```latex
$2x^2=0$
```

$2x^2=0$

#### 下标
```latex
$f_{X}(x)=0$
```
$f_{X}(x)=0$

---

## 表达式

对于特定表达式，例如表示求平方根的$\sqrt{2}$，和表示分式的 $\frac{a}{b}$

有些符号需要参数，例如表示求*n*次方的 $\sqrt[n]{m}$

都需要使用到表示特殊排版的`\`符号

#### 平方根
```latex
$\sqrt{2}$
```

$\sqrt{2}$

#### *n*次方

```latex
$\sqrt[n]{m}$
```

$\sqrt[n]{m}$

---

## 数学符号

数学表达式中经常出现的符号，例如

### 小写希腊字母
$\alpha$
$\beta$
$\gamma$
$\delta$
$\epsilon$
$\zeta$
$\eta$
$\theta$
$\iota$
$\kappa$
$\lambda$
$\mu$
$\nu$
$\xi$
$\omicron$
$\pi$
$\rho$
$\sigma$
$\tau$
$\upsilon$
$\phi$
$\chi$
$\psi$
$\omega$

### 大写希腊字母
$A$
$B$
$\Gamma$
$\Delta$
$E$
$Z$
$H$
$\Theta$
$I$
$K$
$\Lambda$
$M$
$N$
$\Xi$
$O$
$\Pi$
$P$
$\Sigma$
$T$
$Y$
$\Phi$
$X$
$\Psi$
$\Omega$

#### 小写希腊字母
```latex
$\alpha$
$\beta$
$\gamma$
$\delta$
$\epsilon$
$\zeta$
$\eta$
$\theta$
$\iota$
$\kappa$
$\lambda$
$\mu$
$\nu$
$\xi$
$\omicron$
$\pi$
$\rho$
$\sigma$
$\tau$
$\upsilon$
$\phi$
$\chi$
$\psi$
$\omega$
```

#### 大写希腊字母
```latex
$A$
$B$
$\Gamma$
$\Delta$
$E$
$Z$
$H$
$\Theta$
$I$
$K$
$\Lambda$
$M$
$N$
$\Xi$
$O$
$\Pi$
$P$
$\Sigma$
$T$
$Y$
$\Phi$
$X$
$\Psi$
$\Omega$
```

### 运算符号

* 乘: $\times$
* 正负: $\pm$
* 并集: $\cup$
* 交集: $\cap$
* 卷积: $\oplus$

```latex
乘: $\times$
正负: $\pm$
并集: $\cup$
交集: $\cap$
卷积: $\oplus$
```

### 三角函数

$\sin$ $\cos$ $\tan$ $\sinh$ $\arccos$

```latex
$\sin$ $\cos$ $\tan$ $\sinh$ $\arccos$
```

### 关系运算符

$\equiv$ $\leq$ $\geq$ $\approx$ $\neq$

```latex
$\equiv$ $\leq$ $\geq$ $\approx$ $\neq$
```

### 其它符号

$\infty$ $\nabla$ $\partial$

```latex
$\infty$ $\nabla$ $\partial$
```

## 积分运算

$\int_0^\pi \sin{x}\,dx$

❗注意`_`如何表示“下方”的概念，而`^`如何表示“上方”的概念

```latex
$\int_0^\pi \sin{x}\,dx$
```

## 求和运算

$\sum_{i=0}^n x_i$

```latex
$\sum_{i=0}^n x_i$
```

## 乘积运算

$\prod_{i=0}^n x_i$

```latex
$\sum_{i=0}^n x_i$
```

---

## 声调与符号变换
一些符号是普通字符的变换

* hat: $\hat x$
* wide hat: $\widehat{MLE}$
* bar: $\bar x$
* line above: $\overline{x+y}$
* line below: $\underline{x+y}$
* dot: $\dot x$
* double dots: $\ddot x$ or $\ddot{x}$ or $\ddot{xyz}$
* vector: $\vec x$
* arrow: $\overrightarrow{XYZ}$
* 在上方表示加和数量: $\overbrace{1+1+1+\cdots+1}^n$
* 在下方表示加和数量: $\underbrace{1+1+1+\cdots+1}_n$

```latex
* hat: $\hat x$
* wide hat: $\widehat{MLE}$
* bar: $\bar x$
* line above: $\overline{x+y}$
* line below: $\underline{x+y}$
* dot: $\dot x$
* double dots: $\ddot x$ or $\ddot{x}$ or $\ddot{xyz}$
* vector: $\vec x$
* arrow: $\overrightarrow{XYZ}$
* 在上方表示加和数量: $\overbrace{1+1+1+\cdots+1}^n$
* 在下方表示加和数量: $\underbrace{1+1+1+\cdots+1}_n$
```

---

## 括号

### 非弹性括号

括号和其它类型括号都是“非弹性”的，也就是括号具有固定高度，不会随着所包括内容的变化发生变化，例如
$(\frac{-b\pm\sqrt{b^2-4ac}}{2a})^2$

```latex
$(\frac{-b\pm\sqrt{b^2-4ac}}{2a})^2$
```

### 弹性括号

如果需要更好地包裹住表达式，需要使用`\left`和`\right`，例如
$\left(\frac{-b\pm\sqrt{b^2-4ac}}{2a}\right)^2$
❗*注意`\left`和`\right`在表达式中的位置*

```latex
$\left(\frac{-b\pm\sqrt{b^2-4ac}}{2a}\right)^2$
```

### 尖括号

使用`/langle`和`/rangle`作为尖括号开始和结尾的标志，例如
$\langle\frac{-b\pm\sqrt{b^2-4ac}}{2a}\rangle$

```latex
$\langle\frac{-b\pm\sqrt{b^2-4ac}}{2a}\rangle$
```

---

## 字体

罗马文本：$\textrm{abcdefghijklmn123456}$

*斜体*文本：$\mathit{abcdefghijklmn123456}$

Sans serif 字体：$\mathsf{abcdefghijklmn123456}$

Typerwriter 字体：$\mathtt{abcdefghijklmn123456}$

Calligraphic 字体：$\mathcal{abcdefghijklmn123456}$

Blackborad bold 字体：$\mathbb{abcdefghijklmn123456}$

Boldsymbol 字体：$\boldsymbol{A\cdot x}=\lambda\cdot v$

```latex
罗马文本：$\textrm{abcdefghijklmn123456}$

*斜体*文本：$\mathit{abcdefghijklmn123456}$

Sans serif 字体：$\mathsf{abcdefghijklmn123456}$

Typerwriter 字体：$\mathtt{abcdefghijklmn123456}$

Calligraphic 字体：$\mathcal{abcdefghijklmn123456}$

Blackborad bold 字体：$\mathbb{abcdefghijklmn123456}$

Boldsymbol 字体：$\boldsymbol{A\cdot x}=\lambda\cdot v$
```

---

## 空格

表达式中的空格会被自动忽略，例如

$\int f(x) dx$

```latex
$\int f(x) dx$
```

为了输入空格，可以插入`\,`、`\:`或`\;`，例如

`\,`：$\int\,f(x)\,dx$

`\:`：$\int\:f(x)\:dx$

`\;`：$\int\;f(x)\;dx$

`\quad`：$\int\quad f(x)\quad dx$

`\qquad`：$\int\qquad f(x)\qquad dx$

❗注意不同符号所指代的空格长度

```latex
\,：$\int\,f(x)\,dx$

\:：$\int \: f(x) \:dx$

\;：$\int \;f(x)\;dx$

\quad：$\int\quad f(x)\quad dx$

\qquad：$\int\qquad f(x)\qquad dx$
```

---

## 表达式组

### 居中对齐的表达式组

利用`\\`分隔不同表达式，可以让表达式组居中对齐，例如

$$
x+2y=1\\
2x+4y=2\\
3x+8y=3
$$

```latex
$$
x+2y=1 \\
2x+4y=2 \\
3x+8y=3
$$
```

### 符号对齐的表达式组

$$
\begin{align}
x+2y=1\\
2x+4y=2\\
3x+8y=3
\end{align}
$$

❗注意不同对齐方式下，表达式组符号的对齐方式

```
$$
\begin{align}
x+2y=1\\
2x+4y=2\\
3x+8y=3
\end{align}
$$
```

### 对表达式进行编号

在每个表达式后添加`\tag{}`并填入数字，可以对表达式进行编号

$$ \begin{align}
x+2y=1\tag{1}\\
2x+4y=2\tag{2}\\
3x+8y=3\tag{3}
\end{align}
$$

```
$$
\begin{align}
x+2y=1\tag{1}\\
2x+4y=2\tag{2}\\
3x+8y=3\tag{3}
\end{align}
$$
```

## 分段函数

使用`{cases}`作为首尾标志的参数

$$
sign(x)=\begin{cases}
1&x>0 \\ 0&x=0 \cr -1&x<0
\end{cases}
$$

❗
* `&`在此表示缩进
* `\\`和`\cr`表示另起一个条件

```
$$
sign(x)=\begin{cases}
1&x>0 \\ 0&x=0 \cr -1&x<0
\end{cases}
$$
```

supplements/supports/MathJax Examples for Markdown.ipynb