# Measuring Variation in the Empirical Storm Season Over Time

Table of Contents

1  Measuring Variation in the Empirical Storm Season Over Time

2  Introduction

3  Related Research

4  Data Preparation

5  Analysis

5.1  Has the variation in the empirical storm season changed over time?

5.1.1  State Space Model

5.1.1.1  Model Priors

5.2  Is the Annual percentage of storms outside official storm season changing?

5.3  Is the Distribution of the Empirical Storm Season Stationary?

6  Conclusion

7  Acknowledgments

8  References

9  Licenses

# Introduction

This blog post attempts an analysis of the annual variability of dates on which storms formed during the years 1950 to 2016.  Throughout, the word storm will refer to named tropical or subtropical cyclones in the Atlantic Ocean.  The inspiration of this analysis came from a [CSU, Chico](http://www.csuchico.edu/) campus seminar.

During the Fall semester of 2017, CSU, Chico's [Dr. Ann Bykerk-Kauffman](http://www.csuchico.edu/%7eabykerk-kauffman/) gave one of three talks at the [Department of Geological and Environmental Sciences](http://www.csuchico.edu/geos/index.shtml) seminar titled [2017 Hurricane Talk](http://www.csuchico.edu/geos/stories/17.09.14_HurricaneTalk.shtml).  Part of Dr. Bykerk-Kauffman's presentation included data from the National Oceanic and Atmospheric Administration's National Hurricane Center's (NHC) Data Archive: https://www.nhc.noaa.gov/data/#hurdat [Landsea:2013].  After her talk, Dr. David M. Hassenzahl, Dean of the [College of Natural Sciences](https://www.csuchico.edu/nsci/index.shtml), asked a great question, which I'll try to paraphrase:  Is the variation of the empirical storm season changing over time?

One possible way to interpret the phrase *variation of the empirical storm season* goes like this.  The [official storm season](https://en.wikipedia.org/wiki/Atlantic_hurricane_season) is June 01 to November 30.  The official storms season proves useful as a contrast to an empirical storm season.  The word empirical refers to the actual dates for which the named storms in any given year form.  Just because we define a storm season, does not mean all the storms form within the official storm season.  Some years contain some storms outside the official storm season, and some years contain all storms within the official storm season -- as we'll see later, no year in our data set has all storms outside of the official storm season.  Since there is not a direct correspondence between the official storm season and the empirical storm season, it's useful to separate these two phrases.  Last, the word variation describes the average distance (measured in days) from the middle of the empirical storm season.

There are of course other ways to measure the variation of the empirical storm season.  This blog post attempts to quantify the annual variation of the empirical storm season in three different ways.  The first attempt follows the logic in the last paragraph.  The second attempt defines a percentage of storms that form outside of the official storm season, relative to the total number of storms in each year.  The last attempt hypothesizes a probability distribution that produces storms throughout the year, and then asks if this distribution of storms changes over time.

The rest of this post proceeds as follows.  Section 3 discusses some related research and briefly compares this analysis to previous efforts.  Section 4 walks through the data preparation necessary for our analyses.  Section 5 contains the bulk of the analysis, for which multiple measurements of variation of the storm season are considered as a time series.  Here, we focus on Atlantic storms from the years 1950 to 2016.  However, no qualitative, and only minor quantitative, differences are found by analyzing Pacific Ocean storms simiarly.  The post is concluded in Section 6, and Sections 7 and 8 offer my appreciation to those who directly and indirectly made this analysis possible.

# Related Research

Much research effort is spent on storms of the Atlantic Ocean.  For instance, just a quick glance of [NOAA's media releases](http://www.noaa.gov/media-releases) shows that much of their latest research revolves around the storm season.  With a little more attention to their articles, it's clear that this work focuses on storm intensity and/or number of storms.  Less research is devoted to the variation in the storm season.

In 2008, Kossin tried to answer "Is the North Atlantic hurricane season getting longer?" [Kossin:2008].  Based on the sign of his point estimates, Kossin concluded that the hurricane season is indeed getting longer.  This was the conclusion despite high uncertainty in the point estimates.  For the analysis, Kossin used quantile regression on the dates for which storms formed within each year.  We note that quantile regression inherently treats the observations as independent acorss time.

Despite no statistically significant finding, due to high uncertainty, at least two websites picked up on Kossin's article: [Live Science](https://www.livescience.com/2686-hurricane-season-longer.html) and [Wunderground](https://www.wunderground.com/blog/JeffMasters/is-the-atlantic-hurricane-season-getting-longer.html).  Within each story, the uncertainty is glossed over.  This stands in stark contrast to Karloski's summary of Kossin's work [Karloski:2016].  Karloski replicated Kossin's work and provided stronger wording about the uncertain estimates of an increase in variation.

In 2017, evidence of correlations in the storm seasons across time surfaced in the article "Variation of the Tropical Cyclone Season Start in the Western North Pacific" [Kim:2017].  Specifically, the authors found that a strong El Niño year will delay the start of the storm season in the following year.  This analysis, too, treated the annual measurements of the storm season as independent across time.  Using linear regression applied to a statistic similar to the interquartile range (using percentiles 95% and 5%, instead of 75% and 25%), Kim found a statistically insignificant shrink in the storm season.

Like these previous studies, the following analysis attempts to measure the variation in the storm season by using the dates within the year for which named storms first formed.  Unlike these previous studies, this study fits a model more common to time series data.  We use a semilocal linear trend model to account for potential autocorrelation in the storm season.  Further, we introduce novel measures of variation of the storm season.

# Data Preparation

We begin by setting up the working environment: loading [Python](https://www.python.org) and various packages common to Python's data science community, including [Stan](http://mc-stan.org) the probabilistic programming language which we'll use to fit a Bayesian structural time series model.

The HURDAT2 [Landsea:2013] data used here was processed from the NHC data archive to enable this analysis.  The source code and data are available on [GitHub](https://github.com/roualdes/stormevents).

```python
import statsmodels.api as sm
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from matplotlib import rc
import bplot as bp
import datetime
import pystan
import stan_utility
import requests
%matplotlib inline
rc('text', usetex=True)
```

```python
df = pd.read_csv('https://raw.githubusercontent.com/roualdes/stormevents/master/data.csv')
```

To properly work with dates in Python, we convert dates, from the format YYYMMDD, into formal datetime objects.  We assign numbers to dates by counting days since 0001-01-01 UTC plus 1; see [matplotlib dates](https://matplotlib.org/api/dates_api.html) for technical details.

```python
df['date'] = pd.to_datetime(df['date'], format='%Y%m%d')
df.drop_duplicates('name', inplace=True) # don't double count storms
df['year'] = df.date.dt.year
df = df.loc[df['year'] >= 1950]
df['month'] = df.date.dt.month
df['cnum_dates'] = mdates.date2num([datetime.datetime(2018, d.month, d.day)
                         for _, d in df['date'].dropna().iteritems()])
df['num_dates'] = mdates.date2num([datetime.datetime(d.year, d.month, d.day)
                         for _, d in df['date'].iteritems()])
df = df.loc[df['name'].str.contains('^AL')].copy() # Atlantic
df.head()
```

      date
      time
      record_identifier
      intensity_rating
      name
      year
      month
      cnum_dates
      num_dates

      21879
      1950-08-12
      0.0
      NaN
      TS
      AL011950
      1950
      8
      736918.0
      712081.0

      21930
      1950-08-18
      1200.0
      NaN
      TD
      AL021950
      1950
      8
      736924.0
      712087.0

      21990
      1950-08-21
      1200.0
      NaN
      TS
      AL031950
      1950
      8
      736927.0
      712090.0

      22051
      1950-08-30
      1800.0
      NaN
      HU
      AL041950
      1950
      8
      736936.0
      712099.0

      22125
      1950-09-01
      600.0
      NaN
      TS
      AL051950
      1950
      9
      736938.0
      712101.0

With dates properly set up, we can easily visualize the total number of storms each year. As expected, 2005 stands out as a sensational year for Atlantic Ocean storms.

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(df['year'].unique(),
         df['date'].dt.year.value_counts().sort_index())
plt.xlabel('Year', fontsize=20)
plt.ylabel('Number of storms', fontsize=20)
plt.tight_layout()
```

Or we can sum, within each month, the number storms that formed in the Atlantic Ocean.  September appears to be the most common month for a storm.

```python
df['date'].dt.month.value_counts().sort_index()
```

    1       3
    2       1
    4       5
    5      25
    6      73
    7     113
    8     251
    9     332
    10    163
    11     47
    12     11
    Name: date, dtype: int64

Similar information is displayed with a histogram.  The histogram is nice because we can see how well the official storm season captures the highest density of storms.  Like above, the plot below is aggregated over all the years for which we have data.

```python
storm_season = mdates.date2num([datetime.datetime(2018, 6, 1),
                                datetime.datetime(2018, 11, 30)])

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.histogram(df['cnum_dates'], bins=365//4)
bp.density(df['cnum_dates'], alpha=0.25)
bp.rug(storm_season, markersize=50, color='tab:red')
plt.xlabel('Month', fontsize=20)
plt.ylabel('Density', fontsize=20)
plt.title('Average Density of Storms', fontsize=20)
ax.set_xlim([df['cnum_dates'].min(), df['cnum_dates'].max()])
ax.xaxis.set_major_locator(mdates.AutoDateLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter('%m'))
plt.tight_layout()
```

Averaged across all years, there is about 10% of storms that fall outside of the official storm season.  Below is the calculation of the percentage of storms within the official storm season.

```python
np.sum((df['cnum_dates'] > storm_season[0]) & (df['cnum_dates'] < storm_season[1])) / df.shape[0] * 100
```

    95.5078125

# Analysis

## Has the variation in the empirical storm season changed over time?

In an effort to answer Dean Hassenzahl's question, we attempt to measure the variation of the empirical storm season year over year.  To measure such variation, we will use the [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) (MAD) statistic.  The MAD statistic is an estimator of the population standard deviation σ, which is based on the median instead of the mean making it more robust to potential outliers.

To measure the variation of the emprical storm season over time, first consider a year $Y$.  In year $Y$, there will be a handful of storms that occur at different times.  Recall, we converted dates to numbers as per the discussion above in Section Data Preparation.  Define the median storm date as the date corresponding to the median of the numbers that represent the storms' dates.

```python
df['median'] = df.groupby('year')['num_dates'].transform(lambda x: x.median())
df['median_date'] = mdates.num2date(df['median'])
```

The plot below depicts this visually for the year 2005.  The histogram of storms in 2005 appears in blue, as do the vertical ticks representing each storm.  The median storm date appears as the taller, red tick.

```python
idx_2005 = df['year'] == 2005
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.histogram(df['num_dates'].loc[idx_2005], bins=365//30)
bp.rug(df['num_dates'].loc[idx_2005].values, markersize=20)
bp.rug(df['median'].loc[idx_2005].values[:1], markersize=50, color='tab:red')
plt.xlabel('Month', fontsize=20)
plt.ylabel('Density', fontsize=20)
plt.title('Density of Storms in 2005', fontsize=20)
ax.set_xlim([datetime.datetime(2005, 1, 1),
             datetime.datetime(2005, 12, 1)])
ax.xaxis.set_major_locator(mdates.AutoDateLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter('%m'))
plt.tight_layout()
```

By this definition, across all the years, the median storm dates generaly fall in August or September.  The following table depicts the counts of the median storm date within each numbered month, over all the years for which we have data.

```python
df.drop_duplicates('year', inplace=False)['median_date'].dt.month.value_counts().sort_index()
```

    7      1
    8     23
    9     42
    10     1
    Name: median_date, dtype: int64

The median storm date strictly measures an average storm date within year $Y$. Variation on the other hand measures average distance from the median storm date.  We measure distance from the median storm date in days.  Since some storms form before the median storm date and some storms after, we take the absolute value of the distance of each storm from the median storm date. Our MAD estimator is the median of the absolute values of these distances.  The last step scales our MAD estimator by $\frac{1}{\Phi^{-1}(0.75)} \approx 1.4826$, in a fairly standard assumption of normal data.  In the end, we have MAD estimates of the variation of the empirical storm season for each year in our data set.

To help visualize this calculation, consider the plot above.  For each storm (blue ticks), count the days between each storm and the median storm date (red tick), disregarding any potential negative signs.  From these counts, calculate the median and then scale it by $1.4826$.  This calcuation is done for each year, and we refer to the collection of yearly statistics as the MAD estimates.

```python
df['mad'] = df.apply(lambda x: np.abs((x['num_dates'] - x['median'])), axis=1)
sdf = (df[['year', 'mad']].groupby('year', as_index=False)
            .aggregate(lambda x: 1.4826*np.median(x)))
```

A plot of the MAD estimates across time appears below.  A non-zero slope would indicate a change in the variation of the empirical storm season across time.  Hence, to answer Dean Hassenzahl's question, we seek to quantify statistically the slope of these data.

A standard first attempt would fit simple linear regression to these data.  However, fitting linear regression to these data strictly ignores any possible correlation in storm seasons across time.  In this scenario, each year is treated as independent.  Recall, both Kossin and Kim treated their data as independent across time.

Treating each year as independent can be suspicious and can lead to over confident conclusions.  We are more interested in the semilocal linear trend model mentioned previously.  As a comparison, we fit linear regression and the semilocal linear trend model.

The plot below depicts the MAD estimates of the storm season by year in blue, with the estimate from linear regression overlayed in red.  The subtle negative slope indicates that, if anything, the variation of the emprical storm season is decreasing in time.  The negative sign of this slope agrees qualitatively with Kim, but disagrees with Kossin and Karloski.  Though we remind the reader that we are measureing the variation of the empirical storm season differently than Kossin and Karloski. Nevertheless, here we are most interested in the uncertainty of this slope estimate.

```python
reg = sm.OLS(sdf['mad'], sm.tools.add_constant(sdf['year'])).fit()

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['mad'], alpha=0.5)
bp.line(sdf['year'], reg.fittedvalues, color='tab:red')
plt.xlabel('Year', fontsize=20)
plt.ylabel('MAD', fontsize=20)
plt.title('MAD of Storm Season by Year', fontsize=20)
plt.tight_layout()
```

The plot above looks reasonable and the slope point estimate is reasonable given the uncertainty seen in that point estimate.  We should not focus on the p-value, rather we should focus on the standard error in the estimated coefficient year.

```python
reg.summary()
```

OLS Regression Results

  Dep. Variable:	           mad	         R-squared:         	    0.000

  Model:	                   OLS	         Adj. R-squared:    	   -0.015

  Method:	             Least Squares	    F-statistic:       	 0.009973

  Date:	             Wed, 04 Jul 2018	   Prob (F-statistic):	   0.921

  Time:	                 17:13:23	       Log-Likelihood:    	  -251.32

  No. Observations:	          67	        AIC:               	    506.6

  Df Residuals:	              65	        BIC:               	    511.1

  Df Model:	                   1

  Covariance Type:	      nonrobust

    	       coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  const	    47.9342	   131.006	     0.366	  0.716	  -213.703	   309.571

  year	     -0.0066	     0.066	    -0.100	  0.921	    -0.139	     0.125

  Omnibus:	        1.829	   Durbin-Watson:     	    2.333

  Prob(Omnibus):	  0.401	   Jarque-Bera (JB):  	    1.784

  Skew:	           0.322	   Prob(JB):          	    0.410

  Kurtosis:	       2.527	   Cond. No.          	 2.03e+05

The above regression output provides us a number of important summary statistics.  The slope is estimated to be $-0.007$ with a standard error one order of magnitude larger.  This indicates low confidence in the slope estimate.

The Durbin-Watson statistic is greater than 2, possibly indicating negative autocorrelation in these data, but there's likely to be high uncertainty in this statistic as well.

To build upon this simple regression model, we will allow possible autocorrelation in the data.  We next introduce the semilocal linear trend model.

### State Space Model

A more sophisticated model for these data, such that we do not strictly rule out a correlation of the storm seasons across time, is the following Bayesian structural time series model.  The semilocal linear trend model describes the time series of interest $y_t$ as

$$
\begin{align}
y_t & = \mu_t + \epsilon_t \\
\mu_{t+1} & = \mu_t + \nu_t + \gamma_t \\
\nu_{t+1} & = \eta + \phi(\nu_t - \eta) + \zeta_t \\
\epsilon_t & \sim \mathbb{N}(0, \sigma_y^2) \\
\gamma_t & \sim \mathbb{N}(0, \sigma_{\mu}^2) \\
\zeta_t & \sim \mathbb{N}(0, \sigma_{\nu}^2) \\
\end{align}
$$

where $\epsilon_t, \gamma_t$, and $\zeta_t$ are independent.  The notation $\mathbb{N}(0, \sigma^2)$ stands for a normal random variable with mean $0$ and variance $\sigma^2$.

The variable $\eta$, the long run slope in the time series, is of most interest.  Specifically, we allow deviations from the slope, $\nu_t$, so long asthe deviations $\nu_{t}$ tend to revert back to the long run slope.  For more details on this model, consult [Time Series Analysis by State Space Methods](https://www.amazon.com/Time-Analysis-State-Space-Methods/dp/019964117X/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=BXZQ1XM11ZFFZDP4QVSP) and the [R](https://cran.r-project.org/)  package [bsts](https://cran.r-project.org/web/packages/bsts/index.html)'s help page on their semilocal linear trend model or their blog post about the [semilocal linear trend model](http://www.unofficialgoogledatascience.com/2017/07/fitting-bayesian-structural-time-series.html).

Here, the time series of interest $y_t$ is the yearly MAD estimates of variation of the empirical storm season. Since this model will better handle possible correlations across time in the MAD estimates of variation, we will have an appropriate estimate of the variation in the long run slope.

In Stan, we write this model as follows.  The priors are discussed in the next subsection.

```python
local_linear = requests.get('https://raw.githubusercontent.com/roualdes/stormevents/master/semilocal_linear.stan').text
print(local_linear)
```

    data {
      int <lower=1> T;
      vector[T] x;
      vector[T] y;
    }
    transformed data {
      real sd_y = sd(y);
      real sd_x = sd(x);
    }
    parameters {
      real<lower=0> sigma_y;
      vector[T] gamma;
      real<lower=0> sigma_gamma;
      vector[T] zeta;
      real<lower=0> sigma_zeta;
      real eta;
      real<lower=-1, upper=1> phi;
    }
    transformed parameters {
      vector[T] mu;
      vector[T] nu;

      mu[1] = y[1] + sigma_gamma * gamma[1];
      nu[1] = zeta[1];
      for (t in 2:T) {
        mu[t] = mu[t-1] + nu[t-1] + sigma_gamma * gamma[t];
        nu[t] = eta + phi * (nu[t-1] - eta) + sigma_zeta * zeta[t];
      }
    }
    model {
      // likelihood
      y ~ normal(mu, sigma_y);

      // priors
      sigma_y ~ exponential(1 / sd_y);
      gamma ~ normal(0, 1);
      sigma_gamma ~ gamma(2, 1 / sd_y);
      zeta ~ normal(0, 1);
      sigma_zeta ~ gamma(2, 1 / sd_y);
      eta ~ student_t(3, 0, sd_y / sd_x);
      phi ~ normal(0, 0.5);
    }
    generated quantities {
      vector[T] y_pred;
      for (t in 1:T)
        y_pred[t] = normal_rng(mu[t], sigma_y);
    }

In order to fit this model to the yearly variation in the storm season, we need to compile the above Stan program.

```python
stmod = pystan.StanModel(model_code=local_linear)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_c5a67a1a28bbc98cedc1cecc2429214e NOW.

#### Model Priors

The semilocal linear trend model above is fit as a Bayesian model, hence there are priors are on all parameters.  An effort has been made such that all priors are weakly informative.  We tried to follow a combination of the [guidelines established by the Stan community](https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations) and the default [priors for rstanarm](https://cran.r-project.org/web/packages/rstanarm/vignettes/priors.html) [Stan-Development-Team:2016].

Specifically, the parameters $\sigma_y, \sigma_{\gamma}, \sigma_{\zeta}, \eta$, and $\phi$ have priors.  Let $y = (y_1, \
\ldots, y_T)'$ denote the observations of interest and $x$ be the x-axis values along which the observations $y$ are observed.

$$
\begin{align}
\sigma_y & \sim \text{Exponential}(1 / \text{sd}(y) ) \\
\sigma_{\gamma} & \sim \Gamma(2, 1 / \text{sd}(y) ) \\
\sigma_{\zeta} & \sim \Gamma(2, 1 / \text{sd}(y) ) \\
\eta & \sim \mathbb{t}_3(0, \text{sd}(y) / \text{sd}(x) ) \\
\phi & \sim \mathbb{N}(0, 0.5) \\
\end{align}
$$

The prior on $\sigma_y$ attempts to match the scale of the standard deviation of the observations $y$ by scaling an $\text{Exponential(1)}$ distribution appropriately.  The scale parameters of $\gamma$ and $\zeta$ are given boundary avoiding priors and are scaled the same as $\sigma_y$. The prior on $\eta$ is treated as a regression coefficient in a multiple regression model and thus has a t-distribution on it with degrees of freedom that allow for as wide a distribution as possible with a finite variance.  Since $\eta$ is a slope parameter it is given the ratio of standard deviations of $y$ and $x$ to match the scale of a simple linear regression slope of $y$ on $x$.  The prior on $\phi$ attempts to weakly inform $\phi$ towards 0 with a standard deviation of 0.5.  This leaves room for the data to insist upon a posterior distribution on $\phi$ to be near the extremes $-1$ or $1$.

The above model is fit with Stan, and random samples from the joint posterior distribution over the model parameters are extracted.  We next evaluate the fit of the model and discuss the interpretation of the model in the context of the data.

```python
sdata = {
    'T': sdf.shape[0],
    'x': np.arange(sdf.shape[0]),
    'y': sdf['mad'].values.ravel()
}
stfit = stmod.sampling(data=sdata,
                      control={'adapt_delta': .99})
posterior = stfit.extract()
```

    /Users/ez/py3/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

Stan models, like many other sophisticated, statistical models, are susceptible to fitting problems. The interested reader may refer to Section 34 of the [Stan reference manual](http://mc-stan.org/users/documentation/index.html) for further details.  For now, the checks below help assure us that this model applied to these data offer little to be concerned about.

```python
stan_utility.check_treedepth(stfit)
stan_utility.check_energy(stfit)
stan_utility.check_div(stfit)
stan_utility.check_rhat(stfit)
stan_utility.check_n_eff(stfit)
```

    0 of 4000 iterations saturated the maximum tree depth of 10 (0.0%)
    E-BFMI indicated no pathological behavior
    0.0 of 4000 iterations ended with a divergence (0.0%)
    Rhat looks reasonable for all parameters
    n_eff / iter looks reasonable for all parameters

The plot below superimposes our fitted model over the MAD estimates of the yearly variation of the empirical storm season.

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['mad'], alpha=0.5)
bp.curve(sdf['year'], posterior['y_pred'].mean(0), color='tab:red')
plt.ylabel('MAD', fontsize=20)
plt.xlabel('Year', fontsize=20)
plt.title('MAD of Storm Season by Year', fontsize=20)
plt.tight_layout()
```

The variable $\eta$ defines the long run slope of the variation of the MAD estimates.  The point estimate of this model is similar to the point estimate from simple linear regression.  Here, however, we have an accurate estimate of the variation of the long run slope.

The median of the posterior distrbution of the long run slope, $\eta$, from the semilocal linear trend model is $.03$.  Below, $80$% and $95$% credible intervals provide estimates of error of $\eta$.  The density below, with $2.5, 10, 50, 90$, and $97.5$ percentiles drawn as short, blue ticks, depicts the posterior distribution of $\eta$.

We find evidence of a slight postive slope in MAD estimates across time.  The semilocal linear trend model shows that there is more noise in the estimate of slope than does simple linear regression.  With such high uncertainty, there is little evidence that the variation in the empirical storm season is getting significantly longer.

```python
eta_percentiles = np.percentile(posterior['eta'], [2.5, 10, 50, 90, 97.5])
print("Linear regression slope estimate: {}".format(reg.params[1]))
print(eta_percentiles)

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.density(posterior['eta'])
bp.rug(eta_percentiles)
plt.ylabel('Density', fontsize=20)
plt.title('Posterior Distribution of $\eta$', fontsize=20)
plt.xlabel('$\eta$', fontsize=20)
plt.tight_layout()
```

In short, we have some evidence that the variation of the empirical storm season is not changing in time.

Nonetheless, there are other ways to address the idea of Dr. Hassenzahl's question.  Next, we investigate whether or not the proportion of storms occuring outside of the official storm season is changing in time.

## Is the Annual percentage of storms outside official storm season changing?

The official storm season is between June 01 and November 30.  In some years, all the storms form within the official storm season.  In other years, there are a few storms that form outside the official storm season.  Next, we consider the yearly percentage of storms that form outside of the official storm season.  This percentage is calculated for each year as follows.  Divide the number of storms that form outside the official storm season in a given year by the number of named storms that formed in that year and then multiply by 100.  If this yearly percentage were to trend up or down over time, we might believe that the variation of the empirical storm season were changing.

First, we calculate the percentage of storms forming outside of the official storm season for each year.

```python
def outside_SS(df):
    y = df['year'].values[0]
    early = np.sum(df['date'] < datetime.datetime(y, 6, 1))
    late = np.sum(df['date'] > datetime.datetime(y, 10, 30))
    return 100 * (early + late) / df['year'].shape[0]

sdf['prop'] = df.groupby('year', as_index=False).apply(outside_SS)
```

Plotting these percentages over time shows that there are plenty of years where 0 storms formed outside of the official storm season. In no years were all the storms outside of the official storm season.

```python
reg = sm.OLS(sdf['prop'], sm.tools.add_constant(sdf['year'])).fit()

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['prop'])
bp.line(sdf['year'], reg.fittedvalues, color='tab:red')
plt.xlabel('Year', fontsize=20)
plt.ylabel('Percentage', fontsize=20)
plt.title('Percentage of Storms Outside of Official Storm Season by Year', fontsize=20)
plt.tight_layout()
```

We analyze these percentages using the same semilocal linear trend model as above.    If the slope parameter $\eta$ is significantly non-zero, we'll have found some evidence in a changing proportion of storms occuring outside of the official storm season.

```python
sdata = {
    'T': sdf.shape[0],
    'x': np.arange(sdf.shape[0]),
    'y': sdf['prop']
}
stfit = stmod.sampling(data=sdata,
                      control={'adapt_delta': .99})
posterior = stfit.extract()
```

    /Users/ez/py3/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

```python
stan_utility.check_treedepth(stfit)
stan_utility.check_energy(stfit)
stan_utility.check_div(stfit)
stan_utility.check_rhat(stfit)
stan_utility.check_n_eff(stfit)
```

    0 of 4000 iterations saturated the maximum tree depth of 10 (0.0%)
    E-BFMI indicated no pathological behavior
    0.0 of 4000 iterations ended with a divergence (0.0%)
    Rhat looks reasonable for all parameters
    n_eff / iter looks reasonable for all parameters

The plot below displays the semilocal linear trend estimates of the percentages of storms that formed outside of the official storm season.  The model estimates are drawn in red and are superimposed over the light blue observations.  It appears to have a gentle positive slope.

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['prop'], alpha=0.5)
bp.curve(sdf['year'], posterior['y_pred'].mean(0), color='tab:red')
plt.ylabel('Percentage', fontsize=20)
plt.xlabel('Year', fontsize=20)
plt.title('Percentage of Storms Outside of Official Storm Season by Year', fontsize=20)
plt.tight_layout()
```

The posterior distribution of $\eta$ appears below, alongside multiple percentiles.  While the median of $\eta$ is positive, even the $80$% credible interval includes 0.  While I certainly don't want to make a binary conclusion here, it appears that there is much noise in the estimate of the long term slope parameter $\eta$.

```python
eta_percentiles = np.percentile(posterior['eta'], [2.5, 10, 50, 90, 97.5])
print("Linear regression slope estimate: {}".format(reg.params[1]))
print(eta_percentiles)

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.density(posterior['eta'])
bp.rug(eta_percentiles)
plt.ylabel('Density', fontsize=20)
plt.xlabel('$\eta$', fontsize=20)
plt.title('Posterior Distribution of $\eta$', fontsize=20)
plt.tight_layout()
```

It doesn't appear that the percentage of storms occuring outside of the official storm season is changing over time.

Last, we analyze the stationarity of the empirical storm season.

## Is the Distribution of the Empirical Storm Season Stationary?

Consider the average density [histogram](#average_density) above, where the empirical storm season was averaged over all the years for which we have data.  Let's refer to this histogram as the average storm season.  If the empirical storm season was not changing over time, then this average storm season would represent the stationary distribution from which each year's storm season was randomly drawn.  If such a stationary storm season producing distribution exists, then no year's storm season in our data set should be too different from the average storm season.

To evaluate the stationarity of the storm season, we compute such a histogram for each year's empirical storm season separately.  For instance, consider the [histogram of the storms from 2005](#2005_density).  Now imagine we overlayed the average storm season histogram with the histogram from the 2005 storm season.  We calculate the maximal, absolute difference between these two histograms.  This idea is most easily visualized with histograms, but in fact we do all the calculations with the empirical cumulative distribution functions of the empirical storm seasons.  Intuitively, these ideas are similar.  In fact, this is just the [Kolmogorov-Smirnov](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) (KS) statistic, for each year's storm season relative to the average storm season.

We walk through each step of the calculations involved.  First, we calculate the average storm season.

```python
hist, bins = np.histogram(df['cnum_dates'],
                          bins=df['cnum_dates'].nunique(), density=True)
cdf = np.cumsum(hist)
ecdf = cdf/cdf[-1]
```

Next, we'll replicate this calculation for each year in our data set.

```python
ks = []
for year in np.nditer(df['date'].dt.year.unique()):
    h, _ = np.histogram(df['cnum_dates'][df['date'].dt.year == year],
                        bins=df['cnum_dates'].nunique(), density=True)
    bcdf = np.cumsum(h)
    ks.append(np.max(np.abs(ecdf - bcdf/bcdf[-1])))
sdf['KS'] = np.array(ks)*100
```

The plot below displays the year over year KS statistics.  There appears to be a negative slope over time.  We remind the reader that the yearly KS statistics appear in absolute value.  Hence, a (possible) negative slope is not as indicative as the word negative connotes.  Here, any slope, positive or negative, will simply suggest a non-stationary storm season.

```python
reg = sm.OLS(sdf['KS'], sm.tools.add_constant(sdf['year'])).fit()

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['KS'], alpha=0.5)
bp.line(sdf['year'], reg.fittedvalues, color='tab:red')
plt.xlabel('Year', fontsize=20)
plt.ylabel('KS', fontsize=20)
plt.title('KS Statistics by Year', fontsize=20)
plt.tight_layout()
```

```python
sdata = {
    'T': sdf.shape[0],
    'x': np.arange(sdf.shape[0]),
    'y': sdf['KS']
}
stfit = stmod.sampling(data=sdata,
                       control={'adapt_delta': .99,
                                'max_treedepth': 15})
posterior = stfit.extract()
```

    /Users/ez/py3/lib/python3.6/site-packages/pystan/misc.py:399: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      elif np.issubdtype(np.asarray(v).dtype, float):

```python
stan_utility.check_treedepth(stfit, max_depth=15)
stan_utility.check_energy(stfit)
stan_utility.check_div(stfit)
stan_utility.check_rhat(stfit)
stan_utility.check_n_eff(stfit)
```

    0 of 4000 iterations saturated the maximum tree depth of 15 (0.0%)
    E-BFMI indicated no pathological behavior
    2.0 of 4000 iterations ended with a divergence (0.05%)
      Try running with larger adapt_delta to remove the divergences
    Rhat looks reasonable for all parameters
    n_eff / iter looks reasonable for all parameters

The semilocal linear trend model smoothes out the yearly KS statistics.  Similar to linear regression, there appears to be a negative slope.

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.curve(sdf['year'], sdf['KS'], alpha=0.5)
bp.curve(sdf['year'], posterior['y_pred'].mean(0), color='tab:red')
plt.ylabel('KS', fontsize=20)
plt.xlabel('Year', fontsize=20)
plt.title('KS Statistics by Year', fontsize=20)
plt.tight_layout()
```

The posterior distribution of $\eta$ is noisy, despite the negative median value.

```python
eta_percentiles = np.percentile(posterior['eta'], [2.5, 10, 50, 90, 97.5])
print("Linear regression slope estimate: {}".format(reg.params[1]))
print(eta_percentiles)

fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.cla()
bp.density(posterior['eta'])
bp.rug(eta_percentiles)
plt.ylabel('Density', fontsize=20)
plt.title('Posterior Distribution of $\eta$', fontsize=20)
plt.xlabel('$\eta$', fontsize=20)
plt.tight_layout()
```

We find little evidence that the empirical storm season is non-stationary, since the posterior distribution of $\eta$ is so noisy about $0$.

# Conclusion

This blog post analyzed the variation of the empirical storm season in multiple, novel ways.  First, we considered the MAD estimates of the variation of the empirical storm season.  Second, we analyzed the percentage of storms that occured outside of the official storm season.  Neither of these analyses found evidence of significant changes in the variation of the empirical storm season.

The last analysis considered the stationarity of the empirical storm season by investigating the slope of the yearly maximal discrepancy between the average storm season and each year's storm season.  Here, we calculated one KS statistic for each year in our data set.  Unlike the first two analyses, the KS statistic will identify any changes in distribution, not just the variance.  Yet, even with the yearly KS statistics, we still didn't find any significant changes in the empirical storm season.  Though not displayed, similar results hold for the storms in the Pacific Ocean.  The analysis here is easily adapted to storms from the Pacific Ocean; in fact the starting data set includes storms from both oceans.

A strong word of caution is in order.  None of the work here said anything about the [intensity of storms](https://www.ncdc.noaa.gov/news/tropical-cyclone-%E2%80%9Cmaximum-intensity%E2%80%9D-shifting-toward-poles), nor the length of storms, nor the [speed of storms](https://www.ncei.noaa.gov/news/tropical-cyclone-slowdown), nor changes in anything that may influence storms [Karloski:2016].  In part, our focus on the variation of the empirical storm makes this analysis unique.  However, as we largely didn't discover any broad changes in the empirical storm season, we didn't find much of interest.  Which is to say, there may be a reason so many other people have focused their attention elsewhere; that's where all the interesting things are happening.

# Acknowledgments

This blog post would not have happened without the great minds of Dave Hassenzahl, Ann Bykerk-Kauffman, and the Department of Geological and Environmental Sciences.  Thanks goes out to Rachel Hensler, Robin Donatello, Kristen Kaczynski, Rachel Teasdale, and Cab Esposito for their helpful feedback.  And of course, the communities of Python, Stan, and NOAA's National Hurricane Center deserve much credit for their endless hard work.

# References

[Carpenter:2017] B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017. https://www.jstatsoft.org/article/view/v076i01

[Durbin:2012] J. Durbin and S. J. Koopman. Time Series Analysis by State Space Methods. Oxford University Press, 2 edition, 2012.

[Karloski:2016] J. M. Karloski and C. Evans. Seasonal influences upon and long-term trends in the length of the atlantic hurricane season. Journal of Climate, 29(1):273–292, 2016.  https://journals.ametsoc.org/doi/abs/10.1175/JCLI-D-15-0324.1

[Kim:2017] D. Kim, H.-S. Kim, D.-S. R. Park, and M.-S. Park. Variation of the tropical cyclone season start in the western north pacific. Journal of Climate, 30(9):3297–3302, 2017. https://journals.ametsoc.org/doi/full/10.1175/JCLI-D-16-0888.1

[Kossin:2008] J. P. Kossin. Is the north atlantic hurricane season getting longer? Geophysical Research Letters, 35(23), 2008. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2008GL036012

[Landsea:2013] C. W. Landsea and J. L. Franklin. Atlantic hurricane database uncertainty and presentation of a new database format. Monthly Weather Review, 141(10):3576–3592, 2013. https://journals.ametsoc.org/doi/10.1175/MWR-D-12-00254.1

[R-Core-Team:2018] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2018. https://www.R-project.org

[Stan-Development-Team:2016] Stan Development Team. rstanarm: Bayesian applied regression modeling via Stan., 2016. R package version 2.13.1. http://mc-stan.org/rstanarm/index.html

[Scott:2018] S. L. Scott. bsts: Bayesian Structural Time Series, r package version 0.8.0 edition, 2018. https://CRAN.R-project.org/package=bsts

# Licenses

Code: Copyright (2018) California State University, Chico. Released under the BSD 3-clause license. https://opensource.org/licenses/BSD-3-Clause

Text: Copyright (2018) Edward A. Roualdes. Released under the the CC BY-NC 4.0 license. https://creativecommons.org/licenses/by-nc/4.0/legalcode


# $B_E$ speed dependancy

```python
from src.notebook_to_latex import Equation, equation_dict
from rolldecayestimators import equations_ikeda_naked
```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 463 ('figure.dpi       : 100        ## figure dots per inch')

# Purpose
Investigating the speed dependancy of the eddy damping $B_E$ by looking the the MDL database. Ikeda express the speed dependancy as:

```python
Equation(equations_ikeda_naked.eq_eddy_speed, label='eq:eddy_speed')
```

$\displaystyle
\begin{equation}
\frac{B_{E}}{B_{E0}} = \frac{0.0016 K^{2}}{0.0016 K^{2} + 1}
\label{eq:eddy_speed}
\end{equation}
$

But where does the 0.0016 term come from? Is there a better value, when looking at results from MDL db? It seems that it is a fit to the following results:

# Methodology
Create something similar to the plot above, but based on MDL DB. It can be assumed that for a ship ship without bilge keels the eddy damping $B_E$ represent the nonlinear part of the damping.
Using $B_1$, $B_2$ and $B_3$, $B_E$ should therefore represent what is in the $B_2$ coefficient.

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')
from reports.paper_writing import save_fig

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import src.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables
import shipflowmotionshelpers.shipflowmotionshelpers as helpers
import src.visualization.visualize as visualize

```

```python
df_rolldecay = database.load(limit_score=0.98)
#df_rolldecay['BKL'].fillna(0,inplace=True)
scale_factor = df_rolldecay.scale_factor
df_rolldecay['V'] = df_rolldecay.ship_speed*1.852/3.6/np.sqrt(scale_factor)
```

```python
mask = df_rolldecay['BKL'] == 0
df_bare_hull = df_rolldecay.loc[mask].copy()
exclude = [20147221]
mask = df_bare_hull['project_number'].isin(exclude)
df_bare_hull = df_bare_hull.loc[~mask]
```

```python
ok = [
    20064244,
    20095349,
    20095421,
    20157577,
    20157491,
]  # Visual inspection of pictures of models
```

```python
#mask = df_bare_hull['project_number'].isin(ok)
#df_bare_hull = df_bare_hull.loc[mask].copy()
```

```python
for loading_condition_id, df in df_bare_hull.groupby(by='loading_condition_id'):

    if len(df['ship_speed'].unique()) == 1:
        continue

    mask = df['ship_speed'] == 0
    df_zero_speed = df.loc[mask]

    if len(df_zero_speed) == 0:
        continue

    zero_speed = df_zero_speed.iloc[0].copy()

    factor = df['B_2A'] / zero_speed['B_2A']  ## Assuming that B_E is proportional to B_2

    df_bare_hull.loc[df.index,r'B_E/B_E0'] = factor

```

```python
Equation(equations_ikeda_naked.eq_K, label='eq:K')
```

$\displaystyle
\begin{equation}
K = \frac{L_{pp} \omega}{V}
\label{eq:K}
\end{equation}
$

```python
lambda_K = lambdify(equations_ikeda_naked.eq_K.rhs)
```

```python
lambda_eddy_speed = lambdify(equations_ikeda_naked.eq_eddy_speed_general.rhs)
```

```python
lpp = df_bare_hull.lpp/scale_factor
df_bare_hull['K'] = lambda_K(L_pp=lpp, V=df_bare_hull.V, omega=df_bare_hull.omega0)
df_bare_hull[r'1/K'] = 1/df_bare_hull['K']
```

```python
fig,ax=plt.subplots()

for loading_condition_id, df in df_bare_hull.groupby(by='loading_condition_id'):
    df.plot(x='1/K', y = r'B_E/B_E0', ax=ax, style = '.-')

for _, row in df_bare_hull.iterrows():
    ax.annotate(text='  %s' % row['project_number'], xy=(row['1/K'],row[r'B_E/B_E0']))

one_div_K = np.linspace(0,0.2,100)
k = 1/one_div_K

alphas = np.arange(0.04,0.10,0.02)
for alpha in alphas:
    ax.plot(one_div_K, lambda_eddy_speed(K=k, alpha=alpha), label=r'$\alpha:%0.2f$' % alpha)

ax.legend()
ax.set_xlabel(r'$\frac{1}{K}$')
ax.set_ylabel(r'$\frac{B_E}{B_{E0}}$')
ax.grid()

ax.get_legend().set_visible(False)
```

```python
equations.B_e_equation
```

$\displaystyle B_{e} = B_{1} + \frac{8 B_{2} \omega_{0} \phi_{a}}{3 \pi}$

```python
equations.B_2_hat_equation
```

$\displaystyle B_{2 hat} = \frac{B_{2}}{Disp beam^{2} \rho}$

```python
for p in df_bare_hull['project_path'].unique():

    mask = df_bare_hull['project_path'] == p
    project_number = df_bare_hull.loc[mask].iloc[0]['project_number']

    print('%s:%s' % (project_number,p))
```

    20064244:\\sspa.local\gbg\ProjektArkiv\2006\20064244
    20084942:\\sspa.local\gbg\ProjektArkiv\2008\20084942 Bluefin
    20095421:\\sspa.local\gbg\ProjektArkiv\2009\20095421_STX_Polar_Supply
    20095111:\\sspa.local\gbg\ProjektArkiv\2009\20095111
    20157491:S:\2015\20157491-SDARI---twin-skeg-type-modular-carrier

## Now looking at results from Ikeda experiments

```python
import json

def load_digitalized(file_path:str):

    with open(file_path, mode='r') as file:
        j = json.load(file)

    df = pd.DataFrame()
    for dataset in j['datasetColl']:
        df_ = load_dataset(dataset=dataset)
        df = df.append(df_)

    return df

def load_dataset(dataset):

    name = dataset['name']

    xs = []
    ys = []
    for data in dataset['data']:
        x = float(data['value'][0])
        y = float(data['value'][1])
        xs.append(x)
        ys.append(y)

    df = pd.DataFrame()
    df['x'] = xs
    df['y'] = ys
    df['x'] = df['x'].astype(float)
    df['y'] = df['y'].astype(float)

    df['name'] = name

    return df
```

```python
df_experiment = load_digitalized('../data/external/effect_of_advance_speed_on_eddy_damping.json')
df_experiment.reset_index(inplace=True, drop=True)
df_experiment.rename(columns={
    'x' : '1/K',
    'y' : r'B_E/B_E0',
}, inplace=True)
df_experiment['K'] = 1/df_experiment['1/K']

```

```python
fig,ax=plt.subplots()

alphas = np.arange(0.04,0.10,0.02)
for alpha in alphas:
    ax.plot(one_div_K, lambda_eddy_speed(K=k, alpha=alpha), label=r'$\alpha:%0.2f$' % alpha)

styles = {
    'Series60Cb0.6' : 'o',
    'Series60Cb0.7' : 'd',
    'Series60Cb0.8' : '^',

}

for name, df in df_experiment.groupby(by='name'):
    df.plot(x='1/K', y=r'B_E/B_E0', style=styles[name], label=name, ax=ax)

ax.set_xlabel(r'$\frac{1}{K}$')
ax.set_ylabel(r'$\frac{B_E}{B_{E0}}$')
ax.grid()
ax.legend()
```

```python
from scipy.optimize import curve_fit

coeffs, _ = curve_fit(f=lambda_eddy_speed, xdata=df_experiment['K'], ydata=df_experiment[r'B_E/B_E0'],
         bounds=(0,np.inf))

alpha = coeffs[0]
alpha
#lambda_eddy_speed(K=k, alpha=alpha)
```

    0.04084350453107395

```python
df_experiment['alpha'] = 0
for name, df in df_experiment.groupby(by='name'):

    coeffs, _ = curve_fit(f=lambda_eddy_speed, xdata=df['K'], ydata=df[r'B_E/B_E0'],
             bounds=(0,np.inf))

    df['alpha'] = coeffs[0]
    df_experiment.loc[df.index,'alpha'] = df['alpha']
```

```python
df_experiment
```

      1/K
      B_E/B_E0
      name
      K
      alpha

      0
      0.036552
      0.754287
      Series60Cb0.6
      27.358025
      0.045737

      1
      0.051218
      0.550490
      Series60Cb0.6
      19.524229
      0.045737

      2
      0.045126
      0.420088
      Series60Cb0.6
      22.160000
      0.045737

      3
      0.057310
      0.461544
      Series60Cb0.6
      17.448819
      0.045737

      4
      0.069720
      0.200409
      Series60Cb0.6
      14.343042
      0.045737

      5
      0.081453
      0.047269
      Series60Cb0.6
      12.277008
      0.045737

      6
      0.048060
      0.498789
      Series60Cb0.7
      20.807512
      0.032396

      7
      0.036327
      0.268351
      Series60Cb0.7
      27.527950
      0.032396

      8
      0.049639
      0.205179
      Series60Cb0.7
      20.145455
      0.032396

      9
      0.064982
      0.301710
      Series60Cb0.7
      15.388889
      0.032396

      10
      0.091832
      0.338468
      Series60Cb0.7
      10.889435
      0.032396

      11
      0.083709
      0.010119
      Series60Cb0.7
      11.946092
      0.032396

      12
      0.084612
      0.134966
      Series60Cb0.7
      11.818667
      0.032396

      13
      0.122744
      0.010717
      Series60Cb0.7
      8.147059
      0.032396

      14
      0.052121
      0.448115
      Series60Cb0.8
      19.186147
      0.045143

      15
      0.057536
      0.414296
      Series60Cb0.8
      17.380392
      0.045143

      16
      0.080551
      0.249756
      Series60Cb0.8
      12.414566
      0.045143

      17
      0.103565
      0.232573
      Series60Cb0.8
      9.655773
      0.045143

      18
      0.090478
      0.146135
      Series60Cb0.8
      11.052369
      0.045143

      19
      0.093412
      0.104476
      Series60Cb0.8
      10.705314
      0.045143

      20
      0.104242
      0.099830
      Series60Cb0.8
      9.593074
      0.045143

      21
      0.135830
      0.144398
      Series60Cb0.8
      7.362126
      0.045143

      22
      0.149594
      0.094719
      Series60Cb0.8
      6.684766
      0.045143

```python

```


# Lab 4: Shor's Algorithm

In this lab, you will implement a quantum program to factor the number 15. In order to do this, you will write `Qiskit` code for Shor's algorithm following the material presented in lecture 4.

You might find the following chapters of the Qiskit Textbook useful:
- **Shor's algorithm**: https://qiskit.org/textbook/ch-algorithms/shor.html
- **Quantum phase estimation**: https://qiskit.org/textbook/ch-algorithms/quantum-phase-estimation.html
- **Quantum Fourier transform**: https://qiskit.org/textbook/ch-algorithms/quantum-fourier-transform.html

Remember, to run a cell in Jupyter notebooks, you press `Shift` + `Return/Enter` on your keyboard.

### Installing necessary packages

Before we begin, you will need to install some prerequisites into your environment. Run the cell below to complete these installations. At the end, the cell outputs will be cleared.

```python
!pip install -U -r grading_tools/requirements.txt

from IPython.display import clear_output
clear_output()
```

    Requirement already satisfied, skipping upgrade: marshmallow-polyfield<6,>=5.7 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (5.9)
    Requirement already satisfied, skipping upgrade: psutil>=5 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (5.7.0)
    Requirement already satisfied, skipping upgrade: jsonschema>=2.6 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (3.2.0)
    Requirement already satisfied, skipping upgrade: pybind11>=2.4 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aer==0.5.1->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.5.0)
    Requirement already satisfied, skipping upgrade: cython>=0.27.1 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aer==0.5.1->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.29.20)
    Requirement already satisfied, skipping upgrade: setuptools>=40.1.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-ignis==0.3.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (47.3.1)
    Requirement already satisfied, skipping upgrade: h5py in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.10.0)
    Requirement already satisfied, skipping upgrade: docplex in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.14.186)
    Requirement already satisfied, skipping upgrade: cplex; python_version >= "3.6" and python_version < "3.8" in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (12.10.0.2)
    Requirement already satisfied, skipping upgrade: dlx in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.0.4)
    Requirement already satisfied, skipping upgrade: fastdtw in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.3.4)
    Requirement already satisfied, skipping upgrade: scikit-learn>=0.20.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.23.1)
    Requirement already satisfied, skipping upgrade: quandl in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (3.5.0)
    Requirement already satisfied, skipping upgrade: six>=1.5 in c:\users\codie\appdata\roaming\python\python37\site-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (1.12.0)
    Requirement already satisfied, skipping upgrade: cryptography>=1.3 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (2.8)
    Requirement already satisfied, skipping upgrade: ntlm-auth>=1.0.2 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (1.5.0)
    Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (3.0.4)
    Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (2.9)
    Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (2020.6.20)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from sympy>=1.3->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.1.0)
    Requirement already satisfied, skipping upgrade: decorator>=4.3.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from networkx>=2.2; python_version > "3.5"->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (4.4.2)
    Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < "3.8" in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from jsonschema>=2.6->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.6.0)
    Requirement already satisfied, skipping upgrade: pyrsistent>=0.14.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from jsonschema>=2.6->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.16.0)
    Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from jsonschema>=2.6->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (19.3.0)
    Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.1.0)
    Requirement already satisfied, skipping upgrade: joblib>=0.11 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.15.1)
    Requirement already satisfied, skipping upgrade: pandas>=0.14 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.0.5)
    Requirement already satisfied, skipping upgrade: more-itertools in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (8.4.0)
    Requirement already satisfied, skipping upgrade: inflection>=0.3.1 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.5.0)
    Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (1.14.0)
    Requirement already satisfied, skipping upgrade: zipp>=0.5 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=2.6->qiskit-terra==0.14.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (3.1.0)
    Requirement already satisfied, skipping upgrade: pytz>=2017.2 in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from pandas>=0.14->quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2020.1)
    Requirement already satisfied, skipping upgrade: pycparser in c:\users\codie\appdata\local\programs\python\python37\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7->-r grading_tools/requirements.txt (line 2)) (2.20)
    Installing collected packages: qiskit-terra
      Attempting uninstall: qiskit-terra
        Found existing installation: qiskit-terra 0.14.2
        Uninstalling qiskit-terra-0.14.2:
          Successfully uninstalled qiskit-terra-0.14.2
    Successfully installed qiskit-terra-0.14.0

# Review of Shor's Algorithm

You might recall from lecture 4 that Shor's algorithm can be used to factor numbers $N$ that are products of the form

$$N = pq$$

where $p$ and $q$ are prime numbers. This is done in four main steps, similar to the implementation of quantum phase estimation. In this case, we will use two registers of qubits. The first register will have $n$ qubits, and will contain the measurement qubits. The second register will have $m$ qubits, and will be the eigenstate for quantum phase estimation.

1. First, we begin by initializing our qubits. We create a superposition of all $2^n$ computational basis states on the $n$ measurement qubits by applying a Hadamard ($H$) gate on each qubit starting off in the state $\vert0\rangle^{\otimes n}$. We also initialize the $m$ target qubits in the state $\vert1\rangle$. Here, the exponent $\otimes n$ means that we have a tensor product of the states of $n$ qubits.

2. Second, we apply the unitary operator $U$ with various powers onto the target qubits by controlling it with each of the different measurement qubits. The unitary operator in this case implements modular exponentiation. The schematic above shows the ordering and respective powers.

3. Third, we apply an inverse quantum Fourier transform on the $n$ measurement qubits.

4. Finally, we measure the first $n$ qubits.

After the measurement outcomes are determined, we will need to do additional classical post-processing in order to determine the factors or to decide to run the program again.

# Graded Exercise 1: Implementing Shor's Algorithm

In this lab, we will implement Shor's algorithm and use it to factor 15 into 3 and 5.

### 1. Initializing the qubits

We will need to initialize our qubits as described above by applying a Hadamard gate on each of the $n$ measurement qubits. We will also set the target qubits to $\vert1\rangle$, since that is the eigenstate onto which the unitary operator $U$ will be applied. Here, $\vert1\rangle$ is initialized by applying an $X$ gate on the last qubit.

We have created a function below called `initialize_qubits` which takes in three arguments. The first argument is the quantum circuit onto which the gates will be applied. The second argument, `n`, is the number of measurement qubits. The third argument, `m`, is the number of target qubits for the unitary operator.

```python
def initialize_qubits(given_circuit, n, m):

    ### WRITE YOUR CODE BETWEEN THESE LINES - START
    given_circuit.h(range(n))
    #given_circuit.x(m+n-1)
    #given_circuit.x(m+n-2)
    #given_circuit.x(m+n-3)
    #given_circuit.x(m+n-4)

    ### WRITE YOUR CODE BETWEEN THESE LINES - END
```

### 2. Modular exponentiation

We have created a function called `a_x_mod15` below which takes in two arguments, `a` and `x`, and implements the unitary operator $$U = a^x \text{ mod } 15$$

**You do not need to modify this function.**

```python
from qiskit import QuantumCircuit

def a_x_mod15(a, x):
    if a not in [2,7,8,11,13]:
        raise ValueError("'a' must be 2,7,8,11 or 13")
    U = QuantumCircuit(4)
    for iteration in range(x):
        if a in [2,13]:
            U.swap(0,1)
            U.swap(1,2)
            U.swap(2,3)
        if a in [7,8]:
            U.swap(2,3)
            U.swap(1,2)
            U.swap(0,1)
        if a == 11:
            U.swap(1,3)
            U.swap(0,2)
        if a in [7,11,13]:
            for q in range(4):
                U.x(q)
    U = U.to_gate()
    U.name = "%i^%i mod 15" % (a, x)
    c_U = U.control()
    return c_U
```

Note that the function `a_x_mod15` creates a 4-qubit unitary controlled by an additional fifth qubit. In order to use this gate, you will need to append it to your quantum circuit using `Qiskit`'s `circuit.append()` function by passing in the five qubits in a list containing the control qubit first, followed by the four target qubits.

Below, we have created a function called `modular_exponentiation` which takes in four arguments. The first argument, `given_circuit`, is the circuit onto which modular exponentiation will be applied. The next two arguments, `n` and `m`, are the numbers of measurement and target qubits. The schematic above for Shor's algorithm will be useful here. The last argument, `a`, is the base of the modular exponentiation. You will need to call the function `a_x_mod15` as needed in the function below.

```python
def modular_exponentiation(given_circuit, n, m, a):

    ### WRITE YOUR CODE BETWEEN THESE LINES - START
    for q in range(n):
        given_circuit.append(a_x_mod15(a, 2**(n-q-1)),
             [q] + [i+n for i in range(m)])
    ### WRITE YOUR CODE BETWEEN THESE LINES - END

```

### 3. Implementing the inverse quantum Fourier transform

The last step before measuring the first $n$ qubits is the implementation of the inverse quantum Fourier transform. As with `lab3`, you can either implement it on your own or use `Qiskit`'s circuit library.

The function `apply_iqft` takes two arguments. The first argument, `given_circuit`, contains the qubits onto which the inverse quantum Fourier transform will be applied. The second argument, `measurement_qubits`, contains the list of qubits onto which the inverse quantum Fourier transform will be applied.

```python
from qiskit.circuit.library import QFT
```

```python
def apply_iqft(given_circuit, measurement_qubits):

    ### WRITE YOUR CODE BETWEEN THESE LINES - START
    p = len(measurement_qubits)
    given_circuit.append(QFT(p).inverse(), measurement_qubits)
    ### WRITE YOUR CODE BETWEEN THESE LINES - END
```

### 4. Putting it all together

Finally, we combine the functions to construct the quantum program that implements Shor's algorithm.

The next lines of code put everything together. **You do not need to modify anything below, but you will need to run the cell to submit your solution.**

```python
from qiskit import QuantumCircuit
```

```python
def shor_program(n, m, a):

    # set up quantum circuit
    shor = QuantumCircuit(n+m, n)

    # initialize the qubits
    initialize_qubits(shor, n, m)
    shor.barrier()

    # apply modular exponentiation

    modular_exponentiation(shor, n, m, a)
    shor.barrier()

    # apply inverse QFT
    apply_iqft(shor, range(n))

    # measure the first n qubits
    shor.measure(range(n), range(n))

    return shor

n = 4; m = 4; a = 7

mycircuit = shor_program(n, m, a)
mycircuit.draw(output='text')
```

<pre style="word-wrap: normal;white-space: pre;background: #fff0;line-height: 1.1;font-family: "Courier New",Courier,monospace">     ┌───┐ ░                                                              ░ »
q_0: ┤ H ├─░────────■─────────────────────────────────────────────────────░─»
     ├───┤ ░        │                                                     ░ »
q_1: ┤ H ├─░────────┼──────────────■──────────────────────────────────────░─»
     ├───┤ ░        │              │                                      ░ »
q_2: ┤ H ├─░────────┼──────────────┼──────────────■───────────────────────░─»
     ├───┤ ░        │              │              │                       ░ »
q_3: ┤ H ├─░────────┼──────────────┼──────────────┼──────────────■────────░─»
     └───┘ ░ ┌─────┴┼──────┐┌─────┴┼──────┐┌─────┴┼──────┐┌─────┴┼──────┐ ░ »
q_4: ──────░─┤0     │      ├┤0     │      ├┤0     │      ├┤0     │      ├─░─»
           ░ │             ││             ││             ││             │ ░ »
q_5: ──────░─┤1            ├┤1            ├┤1            ├┤1            ├─░─»
           ░ │  7^8 mod 15 ││  7^4 mod 15 ││  7^2 mod 15 ││  7^1 mod 15 │ ░ »
q_6: ──────░─┤2            ├┤2            ├┤2            ├┤2            ├─░─»
           ░ │             ││             ││             ││             │ ░ »
q_7: ──────░─┤3            ├┤3            ├┤3            ├┤3            ├─░─»
           ░ └─────────────┘└─────────────┘└─────────────┘└─────────────┘ ░ »
c_0: ═══════════════════════════════════════════════════════════════════════»
                                                                            »
c_1: ═══════════════════════════════════════════════════════════════════════»
                                                                            »
c_2: ═══════════════════════════════════════════════════════════════════════»
                                                                            »
c_3: ═══════════════════════════════════════════════════════════════════════»
                                                                            »
«     ┌───────┐┌─┐
«q_0: ┤0      ├┤M├─────────
«     │       │└╥┘┌─┐
«q_1: ┤1      ├─╫─┤M├──────
«     │  iqft │ ║ └╥┘┌─┐
«q_2: ┤2      ├─╫──╫─┤M├───
«     │       │ ║  ║ └╥┘┌─┐
«q_3: ┤3      ├─╫──╫──╫─┤M├
«     └───────┘ ║  ║  ║ └╥┘
«q_4: ──────────╫──╫──╫──╫─
«               ║  ║  ║  ║
«q_5: ──────────╫──╫──╫──╫─
«               ║  ║  ║  ║
«q_6: ──────────╫──╫──╫──╫─
«               ║  ║  ║  ║
«q_7: ──────────╫──╫──╫──╫─
«               ║  ║  ║  ║
«c_0: ══════════╩══╬══╬══╬═
«                  ║  ║  ║
«c_1: ═════════════╩══╬══╬═
«                     ║  ║
«c_2: ════════════════╩══╬═
«                        ║
«c_3: ═══════════════════╩═
«                          </pre>

That's it! In order to run your quantum circuit and get the measurement outcomes, you simply need to run `Qiskit`'s `execute` function as follows.

```python
from qiskit import Aer, execute
simulator = Aer.get_backend('qasm_simulator')
counts = execute(mycircuit, backend=simulator, shots=1000).result().get_counts(mycircuit)
from qiskit.visualization import plot_histogram
plot_histogram(counts)
```

```python
for measured_value in counts:
    print(f"Measured {int(measured_value[::-1], 2)}")
```

    Measured 8
    Measured 4
    Measured 12
    Measured 0

You can then follow the classical post-processing details described in lecture 4 to obtain the factors from the measurement outcomes. If you did everything correctly, you should have only measured 0, 4, 8 and 12.

#### Classical post-processing

Recall that in order to factor the number $N = pq$, we used a number $a$ coprime with $N$ and computed the period of $a^x \text{ mod } N$.

We will use the sequence of steps outlined in lecture 4 to guess the factors of $N$ using the measurement outcomes.

```python
from math import gcd

for measured_value in counts:
    measured_value_decimal = int(measured_value[::-1], 2)
    print(f"Measured {measured_value_decimal}")

    if measured_value_decimal % 2 != 0:
        print("Failed. Measured value is not an even number")
        continue
    x = int((a ** (measured_value_decimal/2)) % 15)
    if (x + 1) % 15 == 0:
        print("Failed. x + 1 = 0 (mod N) where x = a^(r/2) (mod N)")
        continue
    guesses = gcd(x + 1, 15), gcd(x - 1, 15)
    print(guesses)
```

    Measured 8
    (1, 15)
    Measured 4
    (5, 3)
    Measured 12
    (5, 3)
    Measured 0
    (1, 15)

#### That's it!

You can grade your solution by running the cell below after filling in your name and email address. **Always provide the same name and email as the one you used during registration to ensure consistency.**

```python
name = 'Rohit Prasad'
email = 'rohitprasad231@gmail.com'

### Do not change the lines below
from grading_tools import grade
from qiskit import transpile
mycircuit_t = transpile(mycircuit, basis_gates = ['u1', 'u2', 'u3', 'cx'])
grade(answer=mycircuit_t, name=name, email=email, labid='lab4', exerciseid='ex1')
```

# Additional reading

- The first experimental demonstration of Shor's algorithm was completed by researchers at IBM and Stanford in 2001 using an experimental platform called nuclear magnetic resonance. You can find the paper here: https://www.nature.com/articles/414883a
- For additional details on the method of continued fractions, you may refer to this page (https://riliu.math.ncsu.edu/437/notes3se4.html) or any standard reference such as Mermin's Quantum Computer Science text.