## Analyze A/B Test Results

You may either submit your notebook through the workspace here, or you may work from your local machine and submit through the next page.  Either way assure that your code passes the project [RUBRIC](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric).  **Please save regularly

This project will assure you have mastered the subjects covered in the statistics lessons.  The hope is to have this project be as comprehensive of these topics as possible.  Good luck!

## Table of Contents
- [Introduction](#intro)
- [Part I - Probability](#probability)
- [Part II - A/B Test](#ab_test)
- [Part III - Regression](#regression)

### Introduction

A/B tests are very commonly performed by data analysts and data scientists.  It is important that you get some practice working with the difficulties of these

For this project, you will be working to understand the results of an A/B test run by an e-commerce website.  Your goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.

**As you work through this notebook, follow along in the classroom and answer the corresponding quiz questions associated with each question.** The labels for each classroom concept are provided for each question.  This will assure you are on the right track as you work through the project, and you can feel more confident in your final submission meeting the criteria.  As a final check, assure you meet all the criteria on the [RUBRIC](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric).

#### Part I - Probability

To get started, let's import our libraries.

```python
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline

random.seed(42)
```

`1.` Now, read in the `ab_data.csv` data. Store it in `df`.

a. Read in the dataset and take a look at the top few rows here:

```python
df = pd.read_csv('ab_data.csv')
df.head()
```

      user_id
      timestamp
      group
      landing_page
      converted

      0
      851104
      2017-01-21 22:11:48.556739
      control
      old_page
      0

      1
      804228
      2017-01-12 08:01:45.159739
      control
      old_page
      0

      2
      661590
      2017-01-11 16:55:06.154213
      treatment
      new_page
      0

      3
      853541
      2017-01-08 18:28:03.143765
      treatment
      new_page
      0

      4
      864975
      2017-01-21 01:52:26.210827
      control
      old_page
      1

b. Use the below cell to find the number of rows in the dataset.

```python
df.shape[0]
```

    294478

> The total number of rows in dataset are **294478**

c. The number of unique users in the dataset.

```python
df.user_id.nunique()
```

    290584

d. The proportion of users converted.

```python
df.groupby('converted')['user_id'].nunique()[1]/df.groupby('converted')['user_id'].nunique().sum()
```

    0.12068362109192721

e. The number of times the `new_page` and `treatment` don't line up.

```python
df.query('(group=="treatment" and landing_page!="new_page") or (group!="treatment" and landing_page=="new_page")').count()
```

    user_id         3893
    timestamp       3893
    group           3893
    landing_page    3893
    converted       3893
    dtype: int64

f. Do any of the rows have missing values?

```python
df.isnull().sum()
```

    user_id         0
    timestamp       0
    group           0
    landing_page    0
    converted       0
    dtype: int64

> No rows missing values. i.e. There is no NaN in the dataset.

`2.` For the rows where **treatment** is not aligned with **new_page** or **control** is not aligned with **old_page**, we cannot be sure if this row truly received the new or old page.  Use **Quiz 2** in the classroom to provide how we should handle these rows.

a. Now use the answer to the quiz to create a new dataset.  Store your new dataframe in **df2**.

```python
index_of_wrong_mapped_cols = df.query('(group=="treatment" and landing_page!="new_page") or (group=="control" and landing_page!="old_page")').index
```

```python
df2 = df.drop(index_of_wrong_mapped_cols, axis=0)
```

```python
df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]
```

    0

`3.` Use **df2** and the cells below to answer questions for **Quiz3** in the classroom.

a. How many unique **user_id**s are in **df2**?

```python
df2.user_id.nunique()
```

    290584

b. There is one **user_id** repeated in **df2**.  What is it?

```python
df2[df2.user_id.duplicated()].user_id
```

    2893    773192
    Name: user_id, dtype: int64

> User id of user having duplicated entry is  **773192**

c. What is the row information for the repeat **user_id**?

```python
df2[df2.user_id.duplicated()]
```

      user_id
      timestamp
      group
      landing_page
      converted

      2893
      773192
      2017-01-14 02:55:59.590927
      treatment
      new_page
      0

d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**.

```python
df2.drop_duplicates(subset='user_id', inplace=True)
```

`4.` Use **df2** in the below cells to answer the quiz questions related to **Quiz 4** in the classroom.

a. What is the probability of an individual converting regardless of the page they receive?

```python
df2.converted.mean()
```

    0.11959708724499628

> Total proportion of converted: **0.1196**

b. Given that an individual was in the `control` group, what is the probability they converted?

```python
df2[df2['group'] == 'control'].converted.mean()
```

    0.1203863045004612

> Proportion of converted for **control** group : **0.12**

c. Given that an individual was in the `treatment` group, what is the probability they converted?

```python
df2[df2['group'] == 'treatment'].converted.mean()
```

    0.11880806551510564

> Proportion of converted for **treatment** group : **0.1188**

d. What is the probability that an individual received the new page?

```python
df2[df2.landing_page == 'new_page'].shape[0]/df2.shape[0]
```

    0.5000619442226688

```python
df2[df2.landing_page == 'old_page'].shape[0]/df2.shape[0]
```

    0.4999380557773312

> Probability of individual receiving either page is **0.5**

e. Use the results in the previous two portions of this question to suggest if you think there is evidence that one page leads to more conversions?  Write your response below.

> The probability of individual receiving either page is equal. The proportion of receiving a **new page** and getting converted is **0.1188** and proportion of receiving an **old page** and getting converted is **0.12**. So, from the given data, **old page led to more conversions**.

### Part II - A/B Test

Notice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.

However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do you run to render a decision that neither page is better than another?

These questions are the difficult parts associated with A/B tests in general.

`1.` For now, consider you need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should your null and alternative hypotheses be?  You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the converted rates for the old and new pages.

**Null hypothesis**        : The new page is either worst or same as that of the old page. i.e. The difference between the converted rates of new page and old page is less than or equal to zero.

**Alternative Hypothesis** : The new page is actually batter than the old page. i.e. The difference between the converted rates of new page and old page is greater than zero.

**Mathematically**:
\begin{align}
\\
H_0 : p_{new} \le p_{old} \; or \; p_{new} - p_{old} \le 0 \\
H_a : p_{new} \gt p_{old} \; or \; p_{new} - p_{old} \gt 0
\end{align}

`2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have "true" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page.

Use a sample size for each page equal to the ones in **ab_data.csv**.

Perform the sampling distribution for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.

Use the cells below to provide the necessary parts of this simulation.  If this doesn't make complete sense right now, don't worry - you are going to work through the problems below to complete this problem.

a. What is the **convert rate** for $p_{new}$ under the null?

```python
p_new = df2['converted'].mean()
p_new
```

    0.11959708724499628

> The value of convert rate for **$p_{new}$** under null is **0.1196** which is the convert rate of whole data.

b. What is the **convert rate** for $p_{old}$ under the null?

```python
p_old = p_new
```

> The convert rate of **$p_{old}$** under null will be equal to **$p_{new}$** i.e. **0.1196** as both are equal under null hypothesis.

c. What is $n_{new}$?

```python
n_new = df2.query('group == "treatment"')['converted'].count()
n_new
```

    145310

> The value of number of rows (**$n_{new}$**) with **new page** as landing page is **145310**.

d. What is $n_{old}$?

```python
n_old = df2.query('group == "control"')['converted'].count()
n_old
```

    145274

> The value of number of rows (**$n_{old}$**) with **old page** as landing page is **145274**.

e. Simulate $n_{new}$ transactions with a convert rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**.

```python
new_page_converted = np.random.binomial(1, p_new, n_new)
```

f. Simulate $n_{old}$ transactions with a convert rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**.

```python
old_page_converted = np.random.binomial(1, p_old, n_old)
```

g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f).

```python
new_page_converted.mean()-old_page_converted.mean()
```

    -0.00014670566009628017

h. Simulate 10,000 $p_{new}$ - $p_{old}$ values using this same process similarly to the one you calculated in parts **a. through g.** above.  Store all 10,000 values in **p_diffs**.

```python
new_simulation = np.random.binomial(n_new, p_new, 10000)/n_new
old_simulation = np.random.binomial(n_old, p_old, 10000)/n_old
p_diffs = new_simulation - old_simulation
```

i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here.

```python
plt.hist(p_diffs);
```

> We have calculated the mean difference of the sampling dustribution (with bootstraping) of convert rate for new and old page. The data follows the **CLT (Central Limit Theorem)** and is in normal distribution shape.

j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?

```python
obs_diff = df2.query('group=="treatment"')['converted'].mean() - df2.query('group=="control"')['converted'].mean()
obs_diff
```

    -0.0015782389853555567

> The **Observed Difference** is **-0.001578**

```python
plt.hist(p_diffs)
plt.axvline(obs_diff, c='r');
```

> As we can see from the histogram above that a large portion of values are above the observed value. It means we **fail to reject the null hypothesis**.

```python
(p_diffs > obs_diff).mean()
```

    0.90169999999999995

> The **p-value** for distribution under null hypothesis is **0.902** which is way larger than **$\alpha$** of 0.05.

k. In words, explain what you just computed in part **j.**.  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages?

> In **j** part, we computed the observed difference between the **$p_{new}$** and **$p_{old}$**. This value is used to calculate the **p-value** of our statistic by simulating the distribution under the **null hypothesis** and finding the probability that statistic came from that distribution.

l. We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively.

```python
import statsmodels.api as sm

convert_old = df2.query('group=="control" and converted == 1')['converted'].count()
convert_new = df2.query('group=="treatment" and converted == 1')['converted'].count()
n_old = df2.query('group=="control"')['group'].count()
n_new = df2.query('group=="treatment"')['group'].count()
```

    /opt/conda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

m. Now use `stats.proportions_ztest` to compute your test statistic and p-value.  [Here](http://knowledgetack.com/python/statsmodels/proportions_ztest/) is a helpful link on using the built in.

```python
sm.stats.proportions_ztest([convert_new, convert_old], [n_new, n_old], alternative='larger')
```

    (-1.3109241984234394, 0.90505831275902449)

n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?

> The **z score** tell us about how many standard deviations below or above from the population mean, our observed value is. Since we got a z-score of **-1.31**, our value is 1.31 standard deviations below the population mean.

> The **p-value** is **0.905**, which is more than the **Type I** threshold value of **0.05**. It is similar to the value we calculated by simulating sample distribution from the null. So, on the basis of this, we **fail to reject null hypothesis**.

### Part III - A regression approach

`1.` In this final part, you will see that the result you acheived in the previous A/B test can also be acheived by performing regression.

a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case?

> Since, each row is either conversion or no conversion. We have only two discrete values of **0** or **1**. Therefore, we will use Logistic Regression a.k.a. Categorical Regression.

b. The goal is to use **statsmodels** to fit the regression model you specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives.  However, you first need to create a colun for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**.

```python
df2['intercept'] = 1
```

```python
df2[['ab_page', 'old_page']] = pd.get_dummies(df2['landing_page'])
```

```python
df2.head()
```

      user_id
      timestamp
      group
      landing_page
      converted
      intercept
      ab_page
      old_page

      0
      851104
      2017-01-21 22:11:48.556739
      control
      old_page
      0
      1
      0
      1

      1
      804228
      2017-01-12 08:01:45.159739
      control
      old_page
      0
      1
      0
      1

      2
      661590
      2017-01-11 16:55:06.154213
      treatment
      new_page
      0
      1
      1
      0

      3
      853541
      2017-01-08 18:28:03.143765
      treatment
      new_page
      0
      1
      1
      0

      4
      864975
      2017-01-21 01:52:26.210827
      control
      old_page
      1
      1
      0
      1

c. Use **statsmodels** to import your regression model.  Instantiate the model, and fit the model using the two columns you created in part **b.** to predict whether or not an individual converts.

```python
model = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.366118
             Iterations 6

d. Provide the summary of your model below, and use it as necessary to answer the following questions.

```python
model.summary()
```

Logit Regression Results

  Dep. Variable:	     converted	      No. Observations:  	   290584

  Model:	               Logit	        Df Residuals:      	   290582

  Method:	               MLE	         Df Model:          	        1

  Date:	          Sun, 13 May 2018	   Pseudo R-squ.:     	  8.077e-06

  Time:	              14:42:33	       Log-Likelihood:    	 -1.0639e+05

  converged:	           True	         LL-Null:           	 -1.0639e+05

   	                       	          LLR p-value:       	   0.1899

      	         coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  intercept	    -1.9888	     0.008	  -246.669	  0.000	    -2.005	    -1.973

  ab_page	      -0.0150	     0.011	    -1.311	  0.190	    -0.037	     0.007

e. What is the p-value associated with **ab_page**? Why does it differ from the value you found in the **Part II**?

> The  **p-value** associated with the **ab_page** is **0.190** but the **p-value** calculated in the **Part II** was around **0.9**. This is happening because in case of simulating a sampling distribution and z-test, we tested the hypothesis for **one-sided *t-test***. But regression model tested the hypothesis for a **two-sided *t-test***.

Since the **p-value** from Regression **(0.19)** is greater than the alpha threshold value of 0.05, **We fail to reject the Null Hypothesis**. It means that the new page is not performing better than the old page.

f. Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?

It is a good idea to consider other factors to add to regression model because there are chances that other factors contribute to check whether an individual converts or not.

There are some disadvantages though. Like, Two or more features can be interdependent on each other i.e **multicollinearity**. Thus, can make the estimate inaccurate.

g. Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives. You will need to read in the **countries.csv** dataset and merge together your datasets on the approporiate rows.  [Here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) are the docs for joining tables.

Does it appear that country had an impact on conversion?  Don't forget to create dummy variables for these country columns. Provide the statistical output as well as a written response to answer this question.

```python
countries_df = pd.read_csv('countries.csv')
df_new = countries_df.set_index('user_id').join(df2.set_index('user_id'), how='inner')
```

```python
df_new.head()
```

      country
      timestamp
      group
      landing_page
      converted
      intercept
      ab_page
      old_page

      user_id

      834778
      UK
      2017-01-14 23:08:43.304998
      control
      old_page
      0
      1
      0
      1

      928468
      US
      2017-01-23 14:44:16.387854
      treatment
      new_page
      0
      1
      1
      0

      822059
      UK
      2017-01-16 14:04:14.719771
      treatment
      new_page
      1
      1
      1
      0

      711597
      UK
      2017-01-22 03:14:24.763511
      control
      old_page
      0
      1
      0
      1

      710616
      UK
      2017-01-16 13:14:44.000513
      treatment
      new_page
      0
      1
      1
      0

```python
df_new.country.value_counts()
```

    US    203619
    UK     72466
    CA     14499
    Name: country, dtype: int64

```python
df_new[['CA', 'UK', 'US']] = pd.get_dummies(df_new['country'])
```

```python
model = sm.Logit(df_new['converted'], df_new[['intercept', 'ab_page', 'CA', 'UK']]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.366113
             Iterations 6

```python
model.summary()
```

Logit Regression Results

  Dep. Variable:	     converted	      No. Observations:  	   290584

  Model:	               Logit	        Df Residuals:      	   290580

  Method:	               MLE	         Df Model:          	        3

  Date:	          Sun, 13 May 2018	   Pseudo R-squ.:     	  2.323e-05

  Time:	              14:42:35	       Log-Likelihood:    	 -1.0639e+05

  converged:	           True	         LL-Null:           	 -1.0639e+05

   	                       	          LLR p-value:       	   0.1760

      	         coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  intercept	    -1.9893	     0.009	  -223.763	  0.000	    -2.007	    -1.972

  ab_page	      -0.0149	     0.011	    -1.307	  0.191	    -0.037	     0.007

  CA	           -0.0408	     0.027	    -1.516	  0.130	    -0.093	     0.012

  UK	            0.0099	     0.013	     0.743	  0.457	    -0.016	     0.036

```python
print('UK is having {:0.4f} times more converted users than US, all other features held constant.'.format(np.exp(0.0099)))
```

    UK is having 1.0099 times more converted users than US, all other features held constant.

```python
print('US is having {:0.4f} times more converted users than UK, all other features held constant.'.format(1/np.exp(-0.0408)))
```

    US is having 1.0416 times more converted users than UK, all other features held constant.

h. Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.

Provide the summary results, and your conclusions based on the results.

```python
df_new['interaction_ab_CA'] = df_new['ab_page']*df_new['CA']
df_new['interaction_ab_UK'] = df_new['ab_page']*df_new['UK']
```

```python
model = sm.Logit(df_new['converted'], df_new[['intercept', 'ab_page', 'CA', 'UK', 'interaction_ab_CA', 'interaction_ab_UK']]).fit()
```

    Optimization terminated successfully.
             Current function value: 0.366109
             Iterations 6

```python
model.summary()
```

Logit Regression Results

  Dep. Variable:	     converted	      No. Observations:  	   290584

  Model:	               Logit	        Df Residuals:      	   290578

  Method:	               MLE	         Df Model:          	        5

  Date:	          Sun, 13 May 2018	   Pseudo R-squ.:     	  3.482e-05

  Time:	              14:42:36	       Log-Likelihood:    	 -1.0639e+05

  converged:	           True	         LL-Null:           	 -1.0639e+05

   	                       	          LLR p-value:       	   0.1920

          	             coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  intercept	            -1.9865	     0.010	  -206.344	  0.000	    -2.005	    -1.968

  ab_page	              -0.0206	     0.014	    -1.505	  0.132	    -0.047	     0.006

  CA	                   -0.0175	     0.038	    -0.465	  0.642	    -0.091	     0.056

  UK	                   -0.0057	     0.019	    -0.306	  0.760	    -0.043	     0.031

  interaction_ab_CA	    -0.0469	     0.054	    -0.872	  0.383	    -0.152	     0.059

  interaction_ab_UK	     0.0314	     0.027	     1.181	  0.238	    -0.021	     0.084

> The **p_value** of all the estimators is greater than the **$\alpha$** threshold value of **0.05**, which provides same result as before despite the fact that we have added more features. The conclusion based on the given data: we **fail to reject the Null Hypothesis**.

## Finishing Up

> Congratulations!  You have reached the end of the A/B Test Results project!  This is the final project in Term 1.  You should be very proud of all you have accomplished!

## Directions to Submit

> Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).

> Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.

> Once you've done this, you can submit your project by clicking on the "Submit Project" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!

```python
from subprocess import call
call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])
```

    0


最初に必要なライブラリを読み込みます。

```python
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import Qubit, QubitBra, measure_all, measure_all_oneshot
from sympy.physics.quantum.gate import H,X,Y,Z,S,T,CPHASE,CNOT,SWAP,UGate,CGateS,gate_simp
from sympy.physics.quantum.gate import IdentityGate as _I
from sympy.physics.quantum.qft import *

from sympy.printing.dot import dotprint
init_printing()

%matplotlib inline
import matplotlib.pyplot as plt
from sympy.physics.quantum.circuitplot import CircuitPlot,labeller, Mz,CreateOneQubitGate

```

## （狭義の）量子プログラミングの手順

1. 計算に必要な量子ビット（量子レジスタ）を準備して、その値を初期化する

2. 量子計算をユニタリ行列（ゲート演算子）で記述する

3. ユニタリ行列を量子ビットに作用する

4. 測定する

#### （１の例）計算に必要な量子ビット（量子レジスタ）を準備して、その値を初期化する

```python
# 全て 0 の３量子ビットを準備
Qubit('000')
```

#### （２の例）量子計算をユニタリ行列（ゲート演算子）で記述する

```python
# 基本的なユニタリ演算子
pprint(represent(X(0),nqubits=1))
pprint(represent(Y(0),nqubits=1))
pprint(represent(Z(0),nqubits=1))
pprint(represent(H(0),nqubits=1))
pprint(represent(S(0),nqubits=1))
pprint(represent(S(0)**(-1),nqubits=1))
pprint(represent(T(0),nqubits=1))
pprint(represent(T(0)**(-1),nqubits=1))
pprint(represent(CNOT(1,0),nqubits=2))
```

    ⎡0  1⎤
    ⎢    ⎥
    ⎣1  0⎦
    ⎡0  -ⅈ⎤
    ⎢     ⎥
    ⎣ⅈ  0 ⎦
    ⎡1  0 ⎤
    ⎢     ⎥
    ⎣0  -1⎦
    ⎡1    1  ⎤
    ⎢──   ── ⎥
    ⎢√2   √2 ⎥
    ⎢        ⎥
    ⎢1   -√2 ⎥
    ⎢──  ────⎥
    ⎣√2   2  ⎦
    ⎡1  0⎤
    ⎢    ⎥
    ⎣0  ⅈ⎦
    ⎡1  0 ⎤
    ⎢     ⎥
    ⎣0  -ⅈ⎦
    ⎡1   0  ⎤
    ⎢       ⎥
    ⎢    ⅈ⋅π⎥
    ⎢    ───⎥
    ⎢     4 ⎥
    ⎣0  ℯ   ⎦
    ⎡1    0   ⎤
    ⎢         ⎥
    ⎢    -ⅈ⋅π ⎥
    ⎢    ─────⎥
    ⎢      4  ⎥
    ⎣0  ℯ     ⎦
    ⎡1  0  0  0⎤
    ⎢          ⎥
    ⎢0  1  0  0⎥
    ⎢          ⎥
    ⎢0  0  0  1⎥
    ⎢          ⎥
    ⎣0  0  1  0⎦

#### （３の例）ユニタリ行列を量子ビットに作用する

```python
# ユニタリ行列を量子ビットに作用するには、qapply() を使います。
hadamard3 = H(2)*H(1)*H(0)
qapply(hadamard3*Qubit('000'))
```

#### （４の例）測定する

```python
# 測定は、qapply() した量子状態に対して、measure_all_oneshot() で確率的な結果を得ます。
for i in range(10):
    pprint(measure_all_oneshot(qapply(hadamard3*Qubit('000'))))
```

    ❘111⟩
    ❘000⟩
    ❘100⟩
    ❘100⟩
    ❘110⟩
    ❘001⟩
    ❘001⟩
    ❘100⟩
    ❘010⟩
    ❘100⟩

```python
# SymPyの量子シミュレーターでは、内部で量子状態を厳密に計算して、すべての状態を保持しています。
# そのため。measure_all() では、全ての量子状態の確率を得ることができます。
measure_all(qapply(hadamard3*Qubit('000')))
```

## 【練習問題】いつもの説明資料の量子回路をプログラミング手順にそって計算しましょう。

```python
### 1. 計算に必要な量子ビット（量子レジスタ）を準備して、その値を初期化する
## ２量子ビットを 0 で初期化してください。
Qubit('00')
```

```python
### 2. 量子計算をユニタリ行列（ゲート演算子）で記述する
## Hadamard のテンソル積 の行列表現を表示してください。

```

```python
## CNOT を Hadamard で挟んだゲート操作 の行列表現を表示してください。

```

```python
### 3. ユニタリ行列を量子ビットに作用する
## Hadamard のテンソル積 を `Qubit('00')` に作用してください。

```

```python
## 次に、CNOT を Hadamard で挟んだゲート操作 を 前の状態に作用してください。

```

```python
### 4. 測定する
## measure_all() を使って、それぞれの状態が測定される確率を表示してください。

```

## 【課題１】グローバーのアルゴリズム

問１）
１. 次の「問１の初期状態」 quest_state を入力として、この量子状態に $\lvert 111 \rangle $ が含まれるか
　グローバーのアルゴリズムを使って調べてください。

２. 上の条件で、この量子状態に $\lvert 101 \rangle $ が含まれるかをグローバーのアルゴリズムを
　使って調べる考察をします。（うまくいかない例を見ます）

　・プログラムを作り、実際は、$\lvert 101 \rangle $ が高確率で検出されることを調べてください。
　・なぜ、初期状態に含まれていない状態が検出されるか理由を考えましょう。（解答は口頭でよい）

問２）
１. 下の「問２の初期状態」quest2_state を入力として、問１と同様、
　$\lvert 111 \rangle $ と $\lvert 101 \rangle $ の状態にの検知について  グローバーのアルゴリズムを適用して、
　その状況を考察してください。

**以降、【課題１】問１−１）の回答欄：**

```python
# 問１の初期状態
quest_state = CNOT(1,0)*CNOT(2,1)*H(2)*H(0)*Qubit('000')
CircuitPlot(quest_state,nqubits=3)
```

```python
# 計算した初期状態を init_state とする
init_state = qapply(quest_state)
init_state
```

```python
# 以降で役立ちそうな関数を定義します。
def CCX(c1,c2,t):  return CGateS((c1,c2),X(t))
def hadamard(s,n):
    h = H(s)
    for i in range(s+1,n+s):
       h = H(i)*h
    return h
def CCZ(c1,c2,t): return (H(t)*CCX(c1,c2,t)*H(t)) # ＣＣＺ演算子を定義します。
def DOp(n): return (Qubit('0'*n)*QubitBra('0'*n)*2-_I(0))     # ゲート操作で計算するには、上記コメントのような演算になります。
h_3 = hadamard(0,3)
d_3 = h_3 * DOp(3) * h_3    # 平均値周りの反転操作
# represent(d_3,nqubits=3)
```

```python
# | 111 > の検索する量子回路を作成する。

```

```python
# 上で作った量子回路を初期状態と作用させて measure_all_oneshot() で何回か試行して、結果をみる。

```

**以降、【課題１】問１−２）の回答欄：**

```python
# | 101 > の検索する量子回路を作成する。

```

```python
# 上で作った量子回路を初期状態と作用させて measure_all() でかく状態の確率をみて、考察する。

```

**以降、【課題１】問２−１）の回答欄：**

```python
# 問２の初期状態
quest2_state = CNOT(2,1)*H(2)*X(2)*CNOT(2,1)*CNOT(2,0)*H(2)*X(2)*Qubit('000')
CircuitPlot(quest2_state,nqubits=3)
```

```python
# 問２の回答欄（１）

```

```python
# 問２の回答欄（２）

```

```python
# 問２の回答欄（３）

```

## 【課題２】量子フーリエ変換

問１）
１. ３量子ビットを対象にした、量子フーリエ変換を行います。
　|000>, |001>, ..., |110>, |111> の全ての状態のそれぞれの QFT の結果を出してください。

　　　ヒント）sympy.physics.quantum.qft の QFT 関数を使います。

２. QFT(0,3) の量子回路図を CircuitPlot() で作図してください。

問２）
１. ３量子ビットを対象にした、量子フーリエ変換を基本的な量子ゲートだけで表してください。
　  $\sqrt{T}$ゲートである Rk(n,4) は利用してもよい。

　・演算をテンソル積で表してください。
　・（この場合の量子回路図は、うまく描けません。）

**以降、【課題２】問１−１）の回答欄：**

```python
## QFT(0,3) の行列表現を表示してください。

```

```python
# |000> を量子フーリエ変換してください。

```

```python
# |001> を量子フーリエ変換してください。

```

```python
# |010> を量子フーリエ変換してください。

```

```python
# |011> を量子フーリエ変換してください。

```

```python
# |100> を量子フーリエ変換してください。

```

```python
# |101> を量子フーリエ変換してください。

```

```python
# |110> を量子フーリエ変換してください。

```

```python
# |111> を量子フーリエ変換してください。

```

**以降、【課題２】問１−２）の回答欄：**

```python
### QFT(0,3) は、SymPy ではひと塊りのまとまったオペレータとして定義されています。
### 基本ゲートを知るためには、decompose() を使います。
QFT(0,3).decompose()
```

```python
# QFT(0,3) の量子回路図を CircuitPlot() で作図してください。

```

```python
# decompose() した上記の回路を改めて、定義しなおします。
qft3_decomp = SWAP(0,2)*H(0)*CGateS((0,),S(1))*H(1)*CGateS((0,),T(2))*CGateS((1,),S(2))*H(2)
qft3_decomp
```

```python
# 上記で定義しなおした QFT の量子回路図を CircuitPlot() で作図します。
# QFT(0,3).decompose() の量子回路図と比較してください。
CircuitPlot(qft3_decomp,nqubits=3)
```

**以降、【課題２】問２−１）の解答欄：**
（ヒント）$c_{g}$ をグローバル位相として、Z軸回転
$ R_{z\theta} = c_{g} X \cdot R_{z\theta/2}^{\dagger} \cdot X \cdot R_{z\theta/2} $
と表せることを使います。

```python
# S = c・X・T†・X・T であることを示します。
pprint(represent(S(0),nqubits=1))
represent(exp(I*pi/4)*X(0)*T(0)**(-1)*X(0)*T(0),nqubits=1)
```

```python
# T = c・X・sqrt(T)†・X・sqrt(T) であることを示します。
pprint(represent(T(0),nqubits=1))
represent(exp(I*pi/8)*X(0)*Rk(0,4)**(-1)*X(0)*Rk(0,4),nqubits=1)
```

```python
# qft3_decomp = SWAP(0,2)*H(0)*CGateS((0,),S(1))*H(1)*CGateS((0,),T(2))*CGateS((1,),S(2))*H(2)
# qft3_decomp を見ながら、制御Ｓゲートを置き換えて、qft3_decomp2 へ代入します。

```

```python
# qft3_decomp2 = SWAP(0,2)*H(0)*CNOT(0,1)*T(1)**(-1)*CNOT(0,1)*T(1)*H(1)*CGateS((0,),T(2))*CNOT(1,2)*T(2)**(-1)*CNOT(1,2)*T(2)*H(2)
# qft3_decomp を見ながら、制御Ｔゲートを置き換えて、qft3_decomp3 へ代入します。

```

```python
# |000> の量子フーリエ変換の結果をみます。
### ゲート操作が少し複雑になるため、SymPyがうまく判断できません。
### represent()で計算します。解答例では、結果が縦ベクトルで行数が長くなるのを嫌い、transpose()します。

# （解答例）transpose(represent(qft3_decomp2*Qubit('000'), nqubits=3))

```

```python
# |001> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/4) をかけると同じになります。

```

```python
# |010> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/4) をかけると同じになります。

```

```python
# |011> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/2) をかけると同じになります。

```

```python
# |100> の量子フーリエ変換の結果をみます。

```

```python
# |101> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/4) をかけると同じになります。

```

```python
# |110> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/4) をかけると同じになります。

```

```python
# |111> の量子フーリエ変換の結果をみます。
### グローバル位相 exp(I*pi/2) をかけると同じになります。

```

```python

```


# Hierarchical Forecasting: Single-level Approaches

In this notebook, we go through the more traditional single-level approach to forecasting hierarchical or grouped time series. The concept is simple: either an aggregation or a disaggregation step is conducted on a single level (base forecast) to obtain coherent forecasts for the rest of the structure.

```python
%load_ext autoreload
```

```python
%autoreload 2
import M5_competition.phdinds2024_entry.PATHS as PATHS
import M5_competition.phdinds2024_entry.utils as utils

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")
```

```python
train = pd.read_csv(PATHS.train_path)
train.head(5)
```

      item_id
      dept_id
      cat_id
      store_id
      state_id
      d_1
      d_2
      d_3
      d_4
      d_5
      ...
      d_1932
      d_1933
      d_1934
      d_1935
      d_1936
      d_1937
      d_1938
      d_1939
      d_1940
      d_1941

      0
      HOBBIES_1_001
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      ...
      2
      4
      0
      0
      0
      0
      3
      3
      0
      1

      1
      HOBBIES_1_002
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      ...
      0
      1
      2
      1
      1
      0
      0
      0
      0
      0

      2
      HOBBIES_1_003
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      ...
      1
      0
      2
      0
      0
      0
      2
      3
      0
      1

      3
      HOBBIES_1_004
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      ...
      1
      1
      0
      4
      0
      1
      3
      0
      2
      6

      4
      HOBBIES_1_005
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      2
      1
      0
      0
      2
      1
      0

5 rows × 1946 columns

Suppose we are forecasting at `level_id` 9. We shall construct a function that performs top-down and bottom-up approaches to obtain forecasts at other levels.

```python
# reference dictionary for variable grouping
utils.level_indexes
```

    {'Level1': None,
     'Level2': 'state_id',
     'Level3': 'store_id',
     'Level4': 'cat_id',
     'Level5': 'dept_id',
     'Level6': ['state_id', 'cat_id'],
     'Level7': ['state_id', 'dept_id'],
     'Level8': ['store_id', 'cat_id'],
     'Level9': ['store_id', 'dept_id'],
     'Level10': ['item_id'],
     'Level11': ['item_id', 'state_id'],
     'Level12': ['item_id', 'store_id']}

```python
level9_forecasts = train.groupby(utils.level_indexes['Level9']).sum().iloc[:, -28:]
level9_forecasts.head(5)
```

      d_1914
      d_1915
      d_1916
      d_1917
      d_1918
      d_1919
      d_1920
      d_1921
      d_1922
      d_1923
      ...
      d_1932
      d_1933
      d_1934
      d_1935
      d_1936
      d_1937
      d_1938
      d_1939
      d_1940
      d_1941

      store_id
      dept_id

      CA_1
      FOODS_1
      294
      259
      268
      328
      387
      455
      403
      262
      332
      272
      ...
      412
      435
      415
      277
      302
      397
      330
      310
      427
      334

      FOODS_2
      458
      500
      412
      400
      463
      653
      720
      620
      525
      531
      ...
      545
      611
      781
      555
      476
      427
      411
      434
      627
      719

      FOODS_3
      2141
      1759
      1787
      1718
      2182
      2842
      3131
      2396
      2140
      2037
      ...
      2408
      3000
      3521
      2366
      2016
      1992
      2113
      2249
      2729
      3274

      HOBBIES_1
      546
      395
      418
      385
      550
      574
      625
      478
      582
      522
      ...
      546
      732
      631
      389
      347
      353
      354
      494
      656
      588

      HOBBIES_2
      43
      31
      33
      47
      34
      62
      59
      51
      46
      43
      ...
      42
      74
      56
      28
      46
      28
      52
      45
      63
      58

5 rows × 28 columns

## Bottom Up

Bottom up approach is straightforward. Forecasts at finer levels are aggregated to get the forecasts at other levels. In this case we forecast `level_id`s 1 to 8 using a bottom up approach from `level_id` 9. We can observe that `level_id`s 1 to 8 are aggregations that can be obtained from the `store_id` and `dept_id`.

**Advantage**
* No loss of information due to aggregation

**Disadvantage**
* Forecasting bottom-level data can be challenging given that these are usually quite noisy.

```python
# reconstructing ids at less granular aggregations (level 1 to 8)
state_id = [i[0].split('_')[0] for i in level9_forecasts.index]
cat_id = [i[1].split('_')[0] for i in level9_forecasts.index]

level9_forecasts = level9_forecasts.reset_index()
level9_forecasts['state_id'] = state_id
level9_forecasts['cat_id'] = cat_id
level9_forecasts.set_index(['state_id', 'store_id', 'cat_id', 'dept_id'], inplace=True)
level9_forecasts.head()
```

      d_1914
      d_1915
      d_1916
      d_1917
      d_1918
      d_1919
      d_1920
      d_1921
      d_1922
      d_1923
      ...
      d_1932
      d_1933
      d_1934
      d_1935
      d_1936
      d_1937
      d_1938
      d_1939
      d_1940
      d_1941

      state_id
      store_id
      cat_id
      dept_id

      CA
      CA_1
      FOODS
      FOODS_1
      294
      259
      268
      328
      387
      455
      403
      262
      332
      272
      ...
      412
      435
      415
      277
      302
      397
      330
      310
      427
      334

      FOODS_2
      458
      500
      412
      400
      463
      653
      720
      620
      525
      531
      ...
      545
      611
      781
      555
      476
      427
      411
      434
      627
      719

      FOODS_3
      2141
      1759
      1787
      1718
      2182
      2842
      3131
      2396
      2140
      2037
      ...
      2408
      3000
      3521
      2366
      2016
      1992
      2113
      2249
      2729
      3274

      HOBBIES
      HOBBIES_1
      546
      395
      418
      385
      550
      574
      625
      478
      582
      522
      ...
      546
      732
      631
      389
      347
      353
      354
      494
      656
      588

      HOBBIES_2
      43
      31
      33
      47
      34
      62
      59
      51
      46
      43
      ...
      42
      74
      56
      28
      46
      28
      52
      45
      63
      58

5 rows × 28 columns

```python
def generate_bu(level_id, forecast_df, level_indexes):
    """Return df of forecasts for level_id using bottom up approach from a
    reference forecast dataframe of a bottom/more granular aggregation level
    """
    index = level_indexes[level_id]
    if index:
        level_forecasts = forecast_df.groupby(index).sum()
    else:
        level_forecasts = forecast_df.sum().to_frame().T
    return level_forecasts

# test : level 8 forecast, function also in utils.py file
utils.generate_bu('Level8', level9_forecasts, utils.level_indexes).head()
```

      d_1914
      d_1915
      d_1916
      d_1917
      d_1918
      d_1919
      d_1920
      d_1921
      d_1922
      d_1923
      ...
      d_1932
      d_1933
      d_1934
      d_1935
      d_1936
      d_1937
      d_1938
      d_1939
      d_1940
      d_1941

      store_id
      cat_id

      CA_1
      FOODS
      2893
      2518
      2467
      2446
      3032
      3950
      4254
      3278
      2997
      2840
      ...
      3365
      4046
      4717
      3198
      2794
      2816
      2854
      2993
      3783
      4327

      HOBBIES
      589
      426
      451
      432
      584
      636
      684
      529
      628
      565
      ...
      588
      806
      687
      417
      393
      381
      406
      539
      719
      646

      HOUSEHOLD
      990
      759
      797
      740
      957
      1312
      1459
      987
      969
      829
      ...
      998
      1393
      1303
      953
      762
      798
      876
      901
      1262
      1316

      CA_2
      FOODS
      2654
      2306
      2224
      2457
      2861
      4041
      4017
      2995
      2624
      3015
      ...
      3255
      3972
      4285
      2902
      2649
      2561
      2725
      3179
      4502
      4198

      HOBBIES
      361
      336
      360
      400
      438
      594
      579
      360
      377
      350
      ...
      370
      687
      588
      326
      385
      334
      357
      434
      762
      660

5 rows × 28 columns

We iterate the following procedure on aggregation levels higher than `level_id` 9, `level_id`s 1 to 8.

## Top Down

Top down approach on the contrary is a disaggregation step which may be done using three common methods:
* Proportions of historical averages
* Average historical proporions
* Forecast proportions

In the step below we construct top-down functions which use the first two methods. Unlike the third, the first two are based on historical proportions. As such we choose the a period `T` for which we base the proportions of the finer levels on.

As a demonstration, we make use of the last `T=28` days prior to the time interval we supposed as foreasts (last 28 days of the entire data in `sales_train_evaluation.csv`). Additionally, we explore getting the historical proportions using `level_id` 10--product (`item_id`) aggregation level.

**Advantage**
* Forecasts a top levels can be simpler, especially for time series of counts, less intermittent series values*. Top-level values tend to have sufficiently large counts (rarely, do we get zero counts).
* Reliable forecasts on the top levels. Can also be reliable at lower levels for low count data.

**Disadvantage**
* All top-down approaches tend to be biased even if the base forecasts (in our example, `level_id` 9) is unbiased. (Hyndman et al., 2011).
* Loss of information due to aggregation.
* Does not take advantage of bottom-level characteristics (e.g., time dynamics, special events, differnet seasonal patterns, etc.)

\**Note: For time series of counts, we may get series values of zero--like in the case of the early stages of new products in the supermarket. In dealing with such, we may need to employ more elaborate models suitable for a sample space of non-negative integers. (e.g. Croston's method--which constructs two series, one which is now the periods between zero values and non-zero values.)*

```python
# construct data used to get historical proportions : prop_reference
# prop_reference is a dataframe containing the last 28 days of unit sales
# from known/training data
T = 28 # period
use_index = ['item_id', 'dept_id', 'cat_id', 'store_id', 'state_id']
prop_reference = train.set_index(use_index).iloc[:, -T*2:-T]
prop_reference.to_csv(PATHS.proportions_reference)
prop_reference.head(3)
```

      d_1886
      d_1887
      d_1888
      d_1889
      d_1890
      d_1891
      d_1892
      d_1893
      d_1894
      d_1895
      ...
      d_1904
      d_1905
      d_1906
      d_1907
      d_1908
      d_1909
      d_1910
      d_1911
      d_1912
      d_1913

      item_id
      dept_id
      cat_id
      store_id
      state_id

      HOBBIES_1_001
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      1
      0
      0
      0
      0
      0
      1
      0
      4
      2
      ...
      1
      3
      0
      1
      1
      1
      3
      0
      1
      1

      HOBBIES_1_002
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0

      HOBBIES_1_003
      HOBBIES_1
      HOBBIES
      CA_1
      CA
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      ...
      2
      1
      2
      1
      1
      1
      0
      1
      1
      1

3 rows × 28 columns

### Proportions of historical averages

\begin{equation}
p_{j}=\sum_{t=1}^{T} \frac{y_{j, t}}{T} / \sum_{t=1}^{T} \frac{y_{t}}{T}
\end{equation}
> for $j = 1, ..., m$. Each proportion $p_j$ reflects the average of the historical proportions of the bottom-level series $y_{j,t}$ relative to the average value of the total aggregate $y_t$. This is implemented below as `get_prop_averages`.

```python
# proportion_averages, constructing function
yts = prop_reference.groupby('item_id').sum().sum()
yjts = prop_reference.groupby('item_id').sum()
pjs = ((yjts.sum(axis=1)/T) / (yts/T).sum())

# check if weights/proportions sum to 1 : good enough
print(pjs.sum())
```

    0.9999999999999999

```python
def get_prop_averages(level_id, level_indexes, prop_reference, T):
    """Returns the proportions of historical averages across a period T"""
    index = level_indexes[level_id]
    if index:
        yjts = prop_reference.groupby(index).sum()
        yts = yjts.sum()
        pjs = ((yjts.sum(axis=1)/T) / (yts/T).sum())
        return pjs
    else:
        raise "Inputted the top-most level. Try again."
        return None
```

```python
# test function, also in utils.py file
# check if weights/proportions sum to 1 : good enough
utils.get_prop_averages('Level10', utils.level_indexes, prop_reference, T).sum()
```

    0.9999999999999999

### Average historical proportions

\begin{equation}
p_{j}=\frac{1}{T} \sum_{t=1}^{T} \frac{y_{j, t}}{y_{t}}
\end{equation}

> for $j = 1, ..., m$. Each proportion $p_j$ reflects the average of the historical proportions of the bottom-level series $y_{j,t}$ over the period $t=1, ..., T$ relative to the total aggregate $y_t$. This is implemented below as `get_ave_proportions`.

```python
# average_proportions
yjts = prop_reference.groupby('item_id').sum()
yts = yjts.sum()
pjs = yjts.apply(lambda row: (row / yts).sum()/T, axis=1)

# check if weights/proportions sum to 1 : good enough
pjs.sum()
```

    1.0

```python
def get_ave_proportions(level_id, level_indexes, prop_reference, T):
    """Returns the average historical proportions across a period T"""
    index = level_indexes[level_id]
    if index:
        yjts = prop_reference.groupby(index).sum()
        yts = yjts.sum()
        pjs = yjts.apply(lambda row: (row / yts).sum()/T, axis=1)
        return pjs
    else:
        raise "Inputted the top-most level. Try again."
        return None
```

```python
# test function, also in utils.py file
# check if weights/proportions sum to 1 : good enough
utils.get_ave_proportions('Level10', utils.level_indexes,
                          prop_reference, T).sum()
```

    1.0

### Forecast Proportions

This is currently not implemented althought literature (Hyndman & Athansopoulos, 2018) states that often, using forecast proportions works better than other top-down methods.

The method is detailed in Athanasopoulos, et al. (2009). *See references below.*

### TD implementation

```python
def generate_td(level_id, forecast_df, method, **kwargs):
    """Return df of forecasts for level_id using top_down approach from a
    reference/base forecast dataframe at a less granualar aggregation level
    """
    pjs = method(level_id, **kwargs)
    yts = forecast_df.sum()
    index = kwargs['level_indexes'][level_id]
    level_forecasts = yts.apply(lambda x: x * pjs).round().astype(int).T
    return level_forecasts
```

```python
# test function, also available in utils.py
utils.generate_td('Level12', level9_forecasts, get_ave_proportions,
                  level_indexes=utils.level_indexes,
                  prop_reference=prop_reference, T=28)
```

      d_1914
      d_1915
      d_1916
      d_1917
      d_1918
      d_1919
      d_1920
      d_1921
      d_1922
      d_1923
      ...
      d_1932
      d_1933
      d_1934
      d_1935
      d_1936
      d_1937
      d_1938
      d_1939
      d_1940
      d_1941

      item_id
      store_id

      FOODS_1_001
      CA_1
      1
      1
      1
      1
      1
      1
      2
      1
      1
      1
      ...
      1
      2
      2
      1
      1
      1
      1
      1
      1
      2

      CA_2
      1
      1
      1
      1
      1
      1
      1
      1
      1
      1
      ...
      1
      1
      2
      1
      1
      1
      1
      1
      1
      1

      CA_3
      1
      1
      1
      1
      1
      1
      1
      1
      1
      1
      ...
      1
      1
      1
      1
      1
      1
      1
      1
      1
      1

      CA_4
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      TX_1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      HOUSEHOLD_2_516
      TX_2
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      TX_3
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      WI_1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      WI_2
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      WI_3
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

30490 rows × 28 columns

## Middle-out Approach

Middle-out simply employs both top-down and bottom-up approaches when base forecasts are done on the middle aggregation levels of a hierarchical data. This is already illustrated through out the notebook, given that `level_id` 9 is a mid-level base forecast.

## References

Hyndman, R. J., & Athanasopoulos, G. (2018). Forecasting: principles and practice. OTexts.

Hyndman, R. J., Ahmed, R. A., Athanasopoulos, G., & Shang, H. L. (2011). Optimal combination forecasts for hierarchical time series. Computational Statistics and Data Analysis, 55(9), 2579–2589.

Athanasopoulos, G., Ahmed, R. A., & Hyndman, R. J. (2009). Hierarchical forecasts for Australian domestic tourism. International Journal of Forecasting, 25, 146–166.


# 14. Decision Trees

[](https://colab.research.google.com/github/rhennig/EMA6938/blob/main/Notebooks/14.DecisionTrees.ipynb)

Slides and notebook based on https://www.datacamp.com/community/tutorials/decision-tree-classification-python, https://nanohub.org/tools/mseml/, https://machinelearningmastery.com/information-gain-and-mutual-information/.

We will use the Decision Tree Classifier Building in Scikit-learn.

Let's first load the required libraries.

```python
# Install the mendeleev and poymatgen packages using pip in the current Jupyter kernel
# To use them, you may need to restart the kernel
import sys
!{sys.executable} -m pip install mendeleev
!{sys.executable} -m pip install pymatgen

import pymatgen as pymat
from pymatgen.core.periodic_table import Element
import mendeleev as mendel
import pandas as pd
import numpy as np
import random

import matplotlib.pyplot as plt
```

    Requirement already satisfied: mendeleev in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (0.9.0)
    Requirement already satisfied: colorama<0.5.0,>=0.4.4 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (0.4.4)
    Requirement already satisfied: Pygments<3.0.0,>=2.8.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (2.10.0)
    Requirement already satisfied: six<2.0.0,>=1.15.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (1.15.0)
    Requirement already satisfied: pandas>=0.25.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (1.3.5)
    Requirement already satisfied: pyfiglet<0.9,>=0.8.post1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (0.8.post1)
    Requirement already satisfied: numpy<2.0.0,>=1.19.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (1.22.2)
    Requirement already satisfied: SQLAlchemy<2.0.0,>=1.3.23 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from mendeleev) (1.4.31)
    Requirement already satisfied: python-dateutil>=2.7.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pandas>=0.25.0->mendeleev) (2.8.2)
    Requirement already satisfied: pytz>=2017.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pandas>=0.25.0->mendeleev) (2021.3)
    Requirement already satisfied: greenlet!=0.4.17 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from SQLAlchemy<2.0.0,>=1.3.23->mendeleev) (1.1.2)
    Requirement already satisfied: pymatgen in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (2022.1.9)
    Requirement already satisfied: requests in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (2.27.1)
    Requirement already satisfied: monty>=3.0.2 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (2022.1.12.1)
    Requirement already satisfied: tqdm in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (4.62.3)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (0.17.20)
    Requirement already satisfied: palettable>=3.1.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (3.3.0)
    Requirement already satisfied: numpy>=1.20.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (1.22.2)
    Requirement already satisfied: sympy in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (1.9)
    Requirement already satisfied: tabulate in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (0.8.9)
    Requirement already satisfied: pandas in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (1.3.5)
    Requirement already satisfied: Cython>=0.29.23 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (0.29.26)
    Requirement already satisfied: plotly>=4.5.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (5.5.0)
    Requirement already satisfied: matplotlib>=1.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (3.4.3)
    Requirement already satisfied: pybtex in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (0.24.0)
    Requirement already satisfied: uncertainties>=3.1.4 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (3.1.6)
    Requirement already satisfied: scipy>=1.5.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (1.7.3)
    Requirement already satisfied: networkx>=2.2 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (2.6.3)
    Requirement already satisfied: spglib>=1.9.9.44 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pymatgen) (1.16.3)
    Requirement already satisfied: kiwisolver>=1.0.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen) (1.3.2)
    Requirement already satisfied: python-dateutil>=2.7 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen) (2.8.2)
    Requirement already satisfied: cycler>=0.10 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen) (0.11.0)
    Requirement already satisfied: pillow>=6.2.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen) (8.3.2)
    Requirement already satisfied: pyparsing>=2.2.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from matplotlib>=1.5->pymatgen) (2.4.7)
    Requirement already satisfied: tenacity>=6.2.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from plotly>=4.5.0->pymatgen) (8.0.1)
    Requirement already satisfied: six in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from plotly>=4.5.0->pymatgen) (1.15.0)
    Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from ruamel.yaml>=0.15.6->pymatgen) (0.2.6)
    Requirement already satisfied: future in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from uncertainties>=3.1.4->pymatgen) (0.18.2)
    Requirement already satisfied: pytz>=2017.3 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pandas->pymatgen) (2021.3)
    Requirement already satisfied: latexcodec>=1.0.4 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pybtex->pymatgen) (2.0.1)
    Requirement already satisfied: PyYAML>=3.01 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from pybtex->pymatgen) (5.4.1)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests->pymatgen) (2021.10.8)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests->pymatgen) (2.0.9)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests->pymatgen) (1.26.7)
    Requirement already satisfied: idna<4,>=2.5 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from requests->pymatgen) (3.1)
    Requirement already satisfied: mpmath>=0.19 in /Users/rhennig/opt/anaconda3/envs/tf2/lib/python3.8/site-packages (from sympy->pymatgen) (1.2.1)

### Getting the dataset

We select 47 elements that occur in the fcc, hcp, and bcc structure. The elements listed were chosen because querying them for these properties yields a dataset with no unknown values, and because they represent the three most common crystallographic structures.

We then query both Pymatgen and Mendeleev to get a complete set of properties per element. We will use this data to create the features from which the model will train and test.

```python
fcc_elements = ["Ag", "Al", "Au", "Cu", "Ir", "Ni", "Pb", "Pd", "Pt", "Rh", "Th", "Yb"]
bcc_elements = ["Ba", "Ca", "Cr", "Cs", "Eu", "Fe", "Li", "Mn", "Mo", "Na", "Nb", "Rb", "Ta", "V", "W" ]
hcp_elements = ["Be", "Cd", "Co", "Dy", "Er", "Gd", "Hf", "Ho", "Lu", "Mg", "Re",
                "Ru", "Sc", "Tb", "Ti", "Tl", "Tm", "Y", "Zn", "Zr"]

elements = fcc_elements + bcc_elements + hcp_elements

random.Random(1).shuffle(elements)

querable_mendeleev = ["atomic_number", "atomic_volume", "boiling_point", "en_ghosh",  "evaporation_heat", "heat_of_formation",
                     "lattice_constant", "melting_point", "specific_heat"]
querable_pymatgen = ["atomic_mass", "atomic_radius", "electrical_resistivity","molar_volume", "bulk_modulus", "youngs_modulus",
                     "average_ionic_radius", "density_of_solid", "coefficient_of_linear_thermal_expansion"]
querable_values = querable_mendeleev + querable_pymatgen
```

We will use the database queries to populate a pandas dataframe.

```python
all_values = [] # Values for Attributes
all_labels = [] # Crystal structure labels (0 = fcc, 1 = bcc, 2 = hcp)

for item in elements:
    element_values = []

    # This section queries Mendeleev
    element_object = mendel.element(item)
    for i in querable_mendeleev:
        element_values.append(getattr(element_object,i))

    # This section queries Pymatgen
    element_object = Element(item)
    for i in querable_pymatgen:
        element_values.append(getattr(element_object,i))

    all_values.append(element_values) # All lists are appended to another list, creating a List of Lists

    if (item in fcc_elements):
        all_labels.append(0) # The crystal structure labels are assigned here
    elif (item in bcc_elements):
        all_labels.append(1) # The crystal structure labels are assigned here
    elif (item in hcp_elements):
        all_labels.append(2) # The crystal structure labels are assigned here

# Pandas Dataframe
df = pd.DataFrame(all_values, columns=querable_values)

# We will patch some of the values that are not available in the datasets.

# Value for the CTE of Cesium
index_Cs = df.index[df['atomic_number'] == 55]
df.iloc[index_Cs, df.columns.get_loc("coefficient_of_linear_thermal_expansion")] = 0.000097
# Value from: David R. Lide (ed), CRC Handbook of Chemistry and Physics, 84th Edition. CRC Press. Boca Raton, Florida, 2003

# Value for the CTE of Rubidium
index_Rb = df.index[df['atomic_number'] == 37]
df.iloc[index_Rb, df.columns.get_loc("coefficient_of_linear_thermal_expansion")] = 0.000090
# Value from: https://www.azom.com/article.aspx?ArticleID=1834

# Value for the Evaporation Heat of Ruthenium
index_Ru = df.index[df['atomic_number'] == 44]
df.iloc[index_Ru, df.columns.get_loc("evaporation_heat")] = 595 # kJ/mol
# Value from: https://www.webelements.com/ruthenium/thermochemistry.html

# Value for the Bulk Modulus of Zirconium
index_Zr = df.index[df['atomic_number'] == 40]
df.iloc[index_Zr, df.columns.get_loc("bulk_modulus")] = 94 # GPa
# Value from: https://materialsproject.org/materials/mp-131/

df.head(n=10)
```

      atomic_number
      atomic_volume
      boiling_point
      en_ghosh
      evaporation_heat
      heat_of_formation
      lattice_constant
      melting_point
      specific_heat
      atomic_mass
      atomic_radius
      electrical_resistivity
      molar_volume
      bulk_modulus
      youngs_modulus
      average_ionic_radius
      density_of_solid
      coefficient_of_linear_thermal_expansion

      0
      27
      6.70
      3143.0
      0.143236
      389.1
      426.7
      2.51
      1768.00
      0.456
      58.933195
      1.35
      6.000000e-08
      6.67
      180.0
      209.0
      0.768333
      8900.0
      0.000013

      1
      69
      18.10
      2220.0
      0.216724
      232.0
      232.2
      3.54
      1818.00
      0.160
      168.934210
      1.75
      6.760000e-07
      19.10
      45.0
      74.0
      1.095000
      9321.0
      0.000013

      2
      39
      19.80
      3611.0
      0.121699
      367.0
      424.7
      3.65
      1795.00
      0.284
      88.905850
      1.80
      6.000000e-07
      19.88
      41.0
      64.0
      1.040000
      4472.0
      0.000011

      3
      75
      8.85
      5900.0
      0.243516
      704.0
      774.0
      2.76
      3453.00
      0.138
      186.207000
      1.35
      1.800000e-07
      8.86
      370.0
      463.0
      0.712500
      21020.0
      0.000006

      4
      28
      6.60
      3005.0
      0.147207
      378.6
      430.1
      3.52
      1726.00
      0.443
      58.693400
      1.35
      7.200000e-08
      6.59
      180.0
      200.0
      0.740000
      8908.0
      0.000013

      5
      67
      18.70
      2968.0
      0.207795
      301.0
      300.6
      3.58
      1747.00
      0.164
      164.930320
      1.75
      8.140000e-07
      18.74
      40.0
      65.0
      1.041000
      8795.0
      0.000011

      6
      79
      10.20
      3080.0
      0.261370
      340.0
      368.2
      4.08
      1337.58
      0.129
      196.966569
      1.35
      2.200000e-08
      10.21
      220.0
      78.0
      1.070000
      19300.0
      0.000014

      7
      21
      15.00
      3104.0
      0.119383
      332.7
      377.8
      3.31
      1814.00
      0.556
      44.955912
      1.60
      5.500000e-07
      15.00
      57.0
      74.0
      0.885000
      2985.0
      0.000010

      8
      45
      8.30
      4000.0
      0.140838
      494.0
      556.0
      3.80
      2239.00
      0.244
      102.905500
      1.35
      4.300000e-08
      8.28
      380.0
      275.0
      0.745000
      12450.0
      0.000008

      9
      74
      9.53
      5930.0
      0.239050
      824.0
      851.0
      3.16
      3680.00
      0.133
      183.840000
      1.35
      5.400000e-08
      9.47
      310.0
      411.0
      0.766667
      19250.0
      0.000005

### Processing and Organizing Data

We normalize the data and randomly split it into training and testing sets.

##### SETS

We have 47 elements for which the crystal structure is known and we will use 40 of these as a training set and the remaining 7 as testing set.

##### NORMALIZATION

We will again use the Standard Score Normalization, which subtracts the mean of the feature and divide by its standard deviation.
$$
\frac{X - µ}{σ}
$$
While our model might converge without feature normalization, the resultant model would be difficult to train and would be dependent on the choice of units used in the input.

```python
# SETS
all_values = [list(df.iloc[x]) for x in range(len(all_values))]

# List of lists are turned into Numpy arrays to facilitate calculations in steps to follow
# (Normalization).
all_values = np.array(all_values, dtype = float)
print("Shape of Values:", all_values.shape)
all_labels = np.array(all_labels, dtype = int)
print("Shape of Labels:", all_labels.shape)

# Training Set
train_values = all_values[:40, :]
train_labels = all_labels[:40]

# Testing Set
test_values = all_values[-7:, :]
test_labels = all_labels[-7:]

# NORMALIZATION

mean = np.nanmean(train_values, axis = 0) # mean
std = np.nanstd(train_values, axis = 0) # standard deviation

train_values = (train_values - mean) / std # input scaling
test_values = (test_values - mean) / std # input scaling

print(train_values[0]) # print a sample entry from the training set
print(train_labels[0])
```

    Shape of Values: (47, 18)
    Shape of Labels: (47,)
    [-0.80084167 -0.75983551 -0.00894162 -0.40732945  0.15599373  0.16654528
     -1.09549525  0.09455406  0.02631292 -0.82400017 -0.80570946 -0.67799461
     -0.75661221  0.70972845  0.6516648  -0.77257498  0.11409173 -0.3075323 ]
    2

### Creating the Decision Tree Model

For this classification, we will use a simple decision tree.

```python
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn import tree

# Create Decision Tree classifer object
model = DecisionTreeClassifier()

# Train Decision Tree Classifer
model.fit(train_values, train_labels)
```

    DecisionTreeClassifier()

### Validation

We calculate the accuracy score on the training and the testing sets.

```python
#Predict the response for training and testing dataset
train_pred = model.predict(train_values)
test_pred = model.predict(test_values)

# Model Accuracy for training and testing set, how often is the classifier correct?

print('Training accuracy = %.3f ' % accuracy_score(train_labels, train_pred))
print('Testing accuracy  = %.3f ' % accuracy_score(test_labels, test_pred))
```

    Training accuracy = 1.000
    Testing accuracy  = 0.857

### Visualize the decision tree

```python
label_names = ('fcc', 'bcc', 'hcp')
fig = plt.figure(figsize=(25,20))
_ = tree.plot_tree(model, feature_names=querable_values, class_names = label_names, filled=True)
```

```python
train_predictions = model.predict(train_values)
test_predictions = model.predict(test_values)

print("train_labels = ", train_labels)
print("test_labels = ", test_labels)

all_labels = np.hstack((train_labels, test_labels))
all_predictions = np.hstack((train_predictions, test_predictions))

predicted_labels = []
true_labels = []

for i in range(all_predictions.shape[0]):
    if (all_predictions[i] == 0):
        predicted_labels.append("FCC")
    if (all_labels[i] == 0):
        true_labels.append("FCC")
    if (all_predictions[i] == 1):
        predicted_labels.append("BCC")
    if (all_labels[i] == 1):
        true_labels.append("BCC")
    if (all_predictions[i] == 2):
        predicted_labels.append("HCP")
    if (all_labels[i] == 2):
        true_labels.append("HCP")

predicted_labels = np.array(predicted_labels).reshape((-1, 1))
true_labels = np.array(true_labels).reshape((-1, 1))
headings = ["Atomic number", "True crystal structure", "Predicted crystal structure"]

atomic_number_array = np.array(df.iloc[:, 0]).reshape((-1, 1))
plot_table = np.concatenate((atomic_number_array, true_labels, predicted_labels), axis=1)

plot_df = pd.DataFrame(plot_table, columns=headings)
```

    train_labels =  [2 2 2 2 0 2 0 2 0 1 2 2 2 0 1 2 1 1 1 1 1 2 0 0 1 2 1 1 2 1 1 0 2 2 0 2 0
     1 1 2]
    test_labels =  [2 2 0 1 0 2 0]

```python
plot_df
```

      Atomic number
      True crystal structure
      Predicted crystal structure

      0
      27
      HCP
      HCP

      1
      69
      HCP
      HCP

      2
      39
      HCP
      HCP

      3
      75
      HCP
      HCP

      4
      28
      FCC
      FCC

      5
      67
      HCP
      HCP

      6
      79
      FCC
      FCC

      7
      21
      HCP
      HCP

      8
      45
      FCC
      FCC

      9
      74
      BCC
      BCC

      10
      64
      HCP
      HCP

      11
      65
      HCP
      HCP

      12
      72
      HCP
      HCP

      13
      70
      FCC
      FCC

      14
      55
      BCC
      BCC

      15
      30
      HCP
      HCP

      16
      56
      BCC
      BCC

      17
      25
      BCC
      BCC

      18
      26
      BCC
      BCC

      19
      42
      BCC
      BCC

      20
      11
      BCC
      BCC

      21
      71
      HCP
      HCP

      22
      90
      FCC
      FCC

      23
      29
      FCC
      FCC

      24
      3
      BCC
      BCC

      25
      81
      HCP
      HCP

      26
      23
      BCC
      BCC

      27
      37
      BCC
      BCC

      28
      40
      HCP
      HCP

      29
      24
      BCC
      BCC

      30
      41
      BCC
      BCC

      31
      47
      FCC
      FCC

      32
      4
      HCP
      HCP

      33
      44
      HCP
      HCP

      34
      13
      FCC
      FCC

      35
      22
      HCP
      HCP

      36
      82
      FCC
      FCC

      37
      20
      BCC
      BCC

      38
      73
      BCC
      BCC

      39
      66
      HCP
      HCP

      40
      48
      HCP
      HCP

      41
      68
      HCP
      HCP

      42
      46
      FCC
      FCC

      43
      63
      BCC
      FCC

      44
      77
      FCC
      FCC

      45
      12
      HCP
      HCP

      46
      78
      FCC
      FCC

### Optimization of the decision tree

The default Attribute Selection Measure to determine the quality of a split for a decision node is Gini. We can try the Entropy measure and also see if we can reduce the depths of the tree.

```python
# Create Decision Tree classifer object
model = DecisionTreeClassifier(criterion="entropy", max_depth=2)

# Train Decision Tree Classifer
model.fit(train_values, train_labels)

#Predict the response for training and testing dataset
train_pred = model.predict(train_values)
test_pred = model.predict(test_values)

# Model Accuracy for training and testing set, how often is the classifier correct?

print('Training accuracy = %.3f ' % accuracy_score(train_labels, train_pred))
print('Testing accuracy  = %.3f ' % accuracy_score(test_labels, test_pred))
```

    Training accuracy = 0.825
    Testing accuracy  = 1.000

```python
label_names = ('fcc', 'bcc', 'hcp')
fig = plt.figure(figsize=(25,20))
_ = tree.plot_tree(model, feature_names=querable_values, class_names = label_names, filled=True)
```

### Questions

#### Hyperparameter optimization

We can select different methods for the attribute selection measure (ASM) and also chose the maximum number of layers. Modify the maximum number of layers and determine what an optimal value is using the training and testing error.

#### Learning from the machine

1. How many parameters are optimized in the decision trees for each node?

2. Which features is most strongly correlated with the crystal structure?

3. What is the physical reason that this feature determines the crystal structure?

3. Should this feature be included in the model?

4. How often does the decision tree identify the right structures if you remove this feature from the dataset?
Remove the appropriate column using the numpy delete command `np.delete(train_values, X-1, 1)`, which removes the column number X from the train_values array.

### Comparison with other machine learning methods

A tutorial on nanoHUB applies a neural network machine learning model to the same dataset for crystal structure predition, which we study here, see https://nanohub.org/tools/mseml/. That model optimizes a total of 350 parameters to achieve an accuracy:

`Training Set Accuracy: 1.000000`

`Testing Set Accuracy: 0.857143`

As we will see when we discuss neural networks, it is difficult to identify the importance of different features in a model.

Question: Based on this information, what are some of the advantages of a decision tree model for this specific dataset?

```python
# Set the column for the feature to be removed from the dataset
Remove_column = 7
train_values_del    = np.delete(train_values, Remove_column-1, 1)
test_values_del     = np.delete(test_values, Remove_column-1, 1)
querable_values_del = np.delete(querable_values, Remove_column-1, 0)

# Create Decision Tree classifer object
model = DecisionTreeClassifier(criterion="gini", max_depth=4)

# Train Decision Tree Classifer
model.fit(train_values_del, train_labels)

#Predict the response for training and testing dataset
train_pred_del = model.predict(train_values_del)
test_pred_del = model.predict(test_values_del)

# Model Accuracy for training and testing set, how often is the classifier correct?

print('Training accuracy = %.3f ' % accuracy_score(train_labels, train_pred_del))
print('Testing accuracy  = %.3f ' % accuracy_score(test_labels, test_pred_del))

# Plot the resulting decision tree
fig = plt.figure(figsize=(25,20))
_ = tree.plot_tree(model, feature_names=querable_values, class_names = label_names, filled=True)
```

```python

```