# Data 512 Final Project
## China's Wage Premium: A Comparison of Urban Areas and Rural Areas
Xiaolu Qian
12/14/2020

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

### Introduction

Traditionally, people living in the urban areas tend to earn more compared to those who live in the rural areas. Such bias in income exists for multiply possible reasons. The main research question I have is: What makes workers in urban area makes more money than those living in the rural area capitals? Previous theories for the explanation includes ability bias of workers living in urban areas, higher living costs in urban areas, and urban agglomeration effect. After controlling for all potential confounding variables, there does not exist urban wage premium; higher wages in urban areas is likely to explain by the higher cost of living, high-skilled jobs, being male in large cities.

### Background

According to United Nations’ statistics, more than half of the population in the world are living in urban regions, and urbanization keeps spreading throughout the world. Regional opportunity structures attract a lot of investigations of the urban labor market at a micro level. High-income level and fast income growth of workers in populated regions are in contrast with the wage level of workers in the rural area and less dense places. A vast literature studies the urban wage premium, particularly in the U.S. context.

China led the global urbanization of the past few decades and according to the World Bank is expected to have 70% of its population living in urban areas by the year 2030. Although China has been going through such a dramatic scale of urbanization, currently there is not a lot of literature on urban wage premium in China.

The goal of this project is to investigate China's urban wage premium. My goal is to determine if there exist urban agglomeration effects in China that contribute to the wage premium accounting for the cost of living and the unobservable ability bias of workers.

Wages of workers living in urban areas may differ because people with higher skills self-select to reside in cities (perhaps with a preference for provincial capitals), generating an ability bias in standard OLS regression of wages on the place of residence. My study will build on mainstream explanations for the urban wage premium to investigate the specific case of China's urban wage premium and provincial capital wage premium. Peng (2016) used China’s Comprehensive Social Survey (CGSS) in 2010 and applied an instrumental variable method. To control for ability bias, he instrumented urban location with urban scale data from 10 years ago. He discovered that there exists a significant urban scale wage premium in larger cities with a population of more than 5 million for both high-skilled and low-skilled workers. Huang’s (2008) study focuses on China’s provincial capitals. Without controlling for the cost of living, he finds that there exists about a 14% wage premium in provincial capital cities. He also finds that there is a 7% provincial capital premium for workers because of the agglomeration economic effects using Mundlak-Chamberlain's random-effect model to control for the cost of living. However, he uses the average monthly expenses on childcare in cities to control for the cost of living. Building on his work, I will use the average housing price per unit to account for different levels of costs of living in different locations.

### Research Question and hypothesis

Main Question:

What makes workers in urban area makes more money than those living in the rural area?

Sub Questions:
- Does the urban wage premium only reflect that more able workers choose to live in cities?
- Are urban workers having higher wages compensated for the higher cost of living in cities?
- Is it actually true that urban areas make workers more productive and lead to higher wages?

Hypothesis:

After controlling for all potential confounding variables, there still exist urban wage premium which is likely to explain by the agglomeration effect of large cities.

### Data selected for analysis

I will be using data from the China Health and Nutrition Survey (CHNS) https://www.cpc.unc.edu/projects/china, a nationally representative sample of Chinese population running from 1997 to 2015. While there is a lot of restricted access to Chinese government population statistics, the CHNS covers 15 representative provinces that vary substantially in demography, economics, and public resources. It used a multistage, random cluster process in drawing sample statistics from each province. The overall survey contains about 7,200 households and covers roughly 30,000 individuals with residential location types classified as urban neighborhoods, rural villages, county town neighborhoods, and suburban villages. In urban areas, the provincial capital and a lower income city were selected randomly if possible. In rural areas, counties were stratified by income, and a weighted scheme was used to randomly select four counties within each province. This includes detailed household and individual economic and social information.

This survey data is gathered by UNC and is publicly available to be used. According to the survey website, “the survey was conducted by an international team of researchers whose backgrounds include nutrition, public health, economics, sociology, Chinese studies, and demography. The survey took place over a 7-day period using a multistage, random cluster process to draw a sample of about 7,200 households with over 30,000 individuals in 15 provinces and municipal cities that vary substantially in geography, economic development, public resources, and health indicators. In addition, detailed community data were collected in surveys of food markets, health facilities, family planning officials, and other social services and community leaders.” There is no need to worry about the ethical considerations to use this dataset.

```python
# data ingestion
wage_12_2015 = pd.io.stata.read_stata('wage.dta')
jobs_12_2015 = pd.io.stata.read_stata('jobs.dta')
jobs_12_2015 = jobs_12_2015.drop(columns=['t2'])
age_2015 = pd.io.stata.read_stata('age.dta')
age_2015 = age_2015.drop(columns=['t2'])
education_2015 = pd.io.stata.read_stata('education.dta')
education_2015 = education_2015.drop(columns=['t2'])
gender = pd.io.stata.read_stata('gender.dta')
marital_status_2015 = pd.io.stata.read_stata('marital_status.dta')
marital_status_2015 = marital_status_2015.drop(columns=['t2'])
asset = pd.io.stata.read_stata('asset.dta')
asset = asset.drop(columns=['t2'])
```

All the data encoding are listed in this questionnaire: https://www.cpc.unc.edu/projects/china/data/questionnaires/C15Individual_Eng.pdf. Here we perform some simple data manipulation to plot the distribution of individuals monthly wages for those who live in rural area and urban areas. c8 in the survey data  stands for monthly wage. t2 indicates the status of where the individual lives in rural area or urban area. t1 = 1 is for urban area and t1 = 2 is for rural areas.

```python
new = wage.merge(jobs, on = 'idind',how='inner')
new = new[new['c8'] != 999999]
new = new[new['c8'] < 100000]
df3 = new[new.t2 == 1]
df4 = new[new.t2 == 2]
```

We see that there is a clear difference of the monthly wage distribution between urban area and  rural area. People in urban area makes almost  1.5 times more than those who lives in the rural area.

```python
sns.distplot(df3['c8'],  kde=False, label='urban')
sns.distplot(df4['c8'],  kde=False, label='rural')
plt.legend(prop={'size': 12})
plt.title('Histogram of Monthly Wage Distribution')
plt.xlabel('Yuan')
plt.ylabel('Count')
```

### Methodology

- OLS

I started my analysis by running a standard mincerian regression using OLS. OLS is a powerful tool that will help us understand the correlation/ causality between wage and all the confounding variables. OLS is also used a lot in other researches related to this topic.

\begin{align}
log(W) = \sum_{i=1}^n \beta_iX_i + \gamma * urban_i + \alpha_i + \epsilon_i
\end{align}

where W is the hourly wage for an individual i, and $X_i$ an individual characteristic that includes gender, years of schooling, potential experience (years of schooling minus age) and its square, a white-collar job occupation dummy, log of the housing price per unit, and an interaction term between white-collar dummy and urban or capital dummy.Urbankis a dummy variable describing whether the individual lives in an urban area.

The following are some data cleaning process in which outliers are eliminated and all wanted features are renamed and joined together.

```python
# join all the datasets together
urban = wage.merge(jobs, on = 'idind',how='inner')
urban = urban.merge(age, on = 'idind',how='inner').merge(education, on = 'idind',how='inner')
urban = urban.merge(gender, on = 'idind',how='inner').merge(marital_status, on = 'idind',how='inner')
urban = urban.merge(asset, on = 'hhid',how='inner')
```

```python
# create housing price per unit to account for the coat of living
urban =  urban[urban['l18'] > 0]
urban["price_per_unit"] = np.log(urban["l18"] * 10000/urban['l16'])
urban =  urban[urban['price_per_unit'] < 150000]
urban["p_housing"] = np.log(urban["price_per_unit"])
```

```python
# create dummy variable for gender
dummy = pd.get_dummies(urban['gender'])
dummy = dummy.rename(columns={1.0: 'male', 2.0: 'female'})
urban = urban.merge(dummy, left_index = True, right_index = True)
```

```python
# Create variables for high skilled and lower-skilled workers
occupation = []
for val in urban.b4:
    if val == 1 or val == 3 or val == 8 or val == 12 or val == 2 or val == 4 or val == 6 or val == 10:
        occupation.append(1)

    else:
        occupation.append(0)
urban.insert(len(urban.columns),'occupation',occupation)
```

```python
# filter marital status to only keep those who are single or married
urban['a8'] =  urban[urban['a8'] != 3]
urban['a8'] =  urban[urban['a8'] != 4]
urban['a8'] =  urban[urban['a8'] != 5]

marital_status = []
for val in urban.a8:
    if val != 2:
        marital_status.append(1)

    else:
        marital_status.append(0)
urban.insert(len(urban.columns),'marital_status',marital_status)
```

```python
# create and rename dummy variable for urban and rural
urb = []
for val in urban.t2:
    if val == 1:
        urb.append(1)
    else:
        urb.append(0)

urban.insert(len(urban.columns),'urb',urb)
```

```python
# create daily wage variable from monthly wages and days worked in the months
urban = urban[urban['c8'] != 999999]
urban["wage"] = np.log(urban['c8']/4/urban['c5']/urban['c6'])
```

```python
# Create variable for years of education
urban = urban[urban['a11'] != -9]
urban = urban.rename(columns={'a11': 'education'})

year_of_education = []
for val in urban.education:
    if val == 11:
        year_of_education.append(1)
    elif val == 12:
        year_of_education.append(2)
    elif val == 13:
        year_of_education.append(3)
    elif val == 14:
        year_of_education.append(4)
    elif val == 15:
        year_of_education.append(5)
    elif val == 16:
        year_of_education.append(6)
    elif val == 21:
        year_of_education.append(7)
    elif val == 22:
        year_of_education.append(8)
    elif val == 23:
        year_of_education.append(9)
    elif val == 24:
        year_of_education.append(10)
    elif val == 25:
        year_of_education.append(11)
    elif val == 26:
        year_of_education.append(12)
    elif val == 27:
        year_of_education.append(13)
    elif val == 28:
        year_of_education.append(14)
    elif val == 29:
        year_of_education.append(15)
    elif val == 30:
        year_of_education.append(16)
    elif val == 31:
        year_of_education.append(17)
    elif val == 32:
        year_of_education.append(18)
    elif val == 33:
        year_of_education.append(19)
    elif val == 34:
        year_of_education.append(20)
    elif val == 35:
        year_of_education.append(21)
    elif val == 36:
        year_of_education.append(22)
    else:
        year_of_education.append(0)
urban.insert(len(urban.columns),'year_of_education',year_of_education)
```

```python
# Create variable for years of work experience which is deduced from age - year of education
urban["potential_experience"] = urban['age'] - urban['year_of_education']
```

```python
# Create variable for housing price squared
urban["p_housing2"] = urban["p_housing"] * urban["p_housing"]
```

```python
# Create interaction between housing price and urban indicator
urban["price_urban"] = urban['p_housing'] * urban['urb']
```

```python
# Create interaction between occupation and urban  indicator
urban["occupation_urban"] = urban['occupation'] * urban['urb']
```

### Findings

After data preprocessing, I conducted 6 OLS regression models to test for urban wage premium and other confounding variables such as cost of living, years of education, gender, type of occupation and etc. that contributes to the wage premium.

####  Regression 1
Features: urban, male, price of housing

```python
X = np.array(urban[['urb','male','p_housing']])

#For the dependent variable Y, we have the data for mortality within one year.
y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.083
    Model:                            OLS   Adj. R-squared:                  0.082
    Method:                 Least Squares   F-statistic:                     81.36
    Date:                Mon, 14 Dec 2020   Prob (F-statistic):           2.15e-50
    Time:                        18:50:06   Log-Likelihood:                -3117.4
    No. Observations:                2709   AIC:                             6243.
    Df Residuals:                    2705   BIC:                             6267.
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.0231      0.055     36.749      0.000       1.915       2.131
    x1            -0.0330      0.033     -1.011      0.312      -0.097       0.031
    x2             0.2378      0.030      7.989      0.000       0.179       0.296
    x3             0.0096      0.001     12.885      0.000       0.008       0.011
    ==============================================================================
    Omnibus:                      456.342   Durbin-Watson:                   1.622
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             6659.203
    Skew:                           0.324   Prob(JB):                         0.00
    Kurtosis:                      10.653   Cond. No.                         290.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Regression 2
Features: urban, male, price of housing, years of education

```python
X = np.array(urban[['urb','male','p_housing', 'year_of_education']])

y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.122
    Model:                            OLS   Adj. R-squared:                  0.120
    Method:                 Least Squares   F-statistic:                     93.52
    Date:                Mon, 14 Dec 2020   Prob (F-statistic):           1.38e-74
    Time:                        19:15:25   Log-Likelihood:                -3058.3
    No. Observations:                2708   AIC:                             6127.
    Df Residuals:                    2703   BIC:                             6156.
    Df Model:                           4
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          1.6555      0.064     26.057      0.000       1.531       1.780
    x1            -0.0662      0.032     -2.066      0.039      -0.129      -0.003
    x2             0.2360      0.029      8.095      0.000       0.179       0.293
    x3             0.0075      0.001      9.913      0.000       0.006       0.009
    x4             0.0456      0.004     10.926      0.000       0.037       0.054
    ==============================================================================
    Omnibus:                      511.646   Durbin-Watson:                   1.643
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             9178.980
    Skew:                           0.369   Prob(JB):                         0.00
    Kurtosis:                      11.989   Cond. No.                         342.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Regression 3
Features: urban, male, price of housing, years of education, occupation

```python
X = np.array(urban[['urb','male','p_housing', 'year_of_education', 'occupation']])

y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.131
    Model:                            OLS   Adj. R-squared:                  0.129
    Method:                 Least Squares   F-statistic:                     81.39
    Date:                Mon, 14 Dec 2020   Prob (F-statistic):           8.65e-80
    Time:                        19:18:59   Log-Likelihood:                -3043.8
    No. Observations:                2708   AIC:                             6100.
    Df Residuals:                    2702   BIC:                             6135.
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          1.7130      0.064     26.723      0.000       1.587       1.839
    x1            -0.0597      0.032     -1.872      0.061      -0.122       0.003
    x2             0.2225      0.029      7.644      0.000       0.165       0.280
    x3             0.0072      0.001      9.600      0.000       0.006       0.009
    x4             0.0343      0.005      7.353      0.000       0.025       0.043
    x5             0.1782      0.033      5.386      0.000       0.113       0.243
    ==============================================================================
    Omnibus:                      530.387   Durbin-Watson:                   1.645
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):            10167.424
    Skew:                           0.385   Prob(JB):                         0.00
    Kurtosis:                      12.461   Cond. No.                         349.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Regression 4
Features: urban, male, price of housing, years of education, occupation, marital status

```python
X = np.array(urban[['urb','male','p_housing', 'year_of_education', 'occupation', 'marital_status']])

y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.130
    Model:                            OLS   Adj. R-squared:                  0.128
    Method:                 Least Squares   F-statistic:                     67.18
    Date:                Mon, 14 Dec 2020   Prob (F-statistic):           4.19e-78
    Time:                        19:20:19   Log-Likelihood:                -3037.0
    No. Observations:                2702   AIC:                             6088.
    Df Residuals:                    2695   BIC:                             6129.
    Df Model:                           6
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          1.6760      0.090     18.661      0.000       1.500       1.852
    x1            -0.0584      0.032     -1.827      0.068      -0.121       0.004
    x2             0.2219      0.029      7.612      0.000       0.165       0.279
    x3             0.0073      0.001      9.617      0.000       0.006       0.009
    x4             0.0344      0.005      7.309      0.000       0.025       0.044
    x5             0.1765      0.033      5.326      0.000       0.112       0.241
    x6             0.0358      0.056      0.639      0.523      -0.074       0.146
    ==============================================================================
    Omnibus:                      529.467   Durbin-Watson:                   1.649
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):            10194.349
    Skew:                           0.384   Prob(JB):                         0.00
    Kurtosis:                      12.485   Cond. No.                         531.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Regression 5
Features: urban, male, price of housing, years of education, occupation, marital status, potential experience

```python
X = np.array(urban[['urb','male','p_housing', 'year_of_education', 'occupation', 'marital_status', 'potential_experience']])

y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.132
    Model:                            OLS   Adj. R-squared:                  0.130
    Method:                 Least Squares   F-statistic:                     58.67
    Date:                Thu, 10 Dec 2020   Prob (F-statistic):           1.27e-78
    Time:                        15:35:07   Log-Likelihood:                -3033.7
    No. Observations:                2702   AIC:                             6083.
    Df Residuals:                    2694   BIC:                             6131.
    Df Model:                           7
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          1.8162      0.105     17.353      0.000       1.611       2.021
    x1            -0.0519      0.032     -1.622      0.105      -0.115       0.011
    x2             0.2390      0.030      8.006      0.000       0.180       0.298
    x3             0.0074      0.001      9.837      0.000       0.006       0.009
    x4             0.0261      0.006      4.606      0.000       0.015       0.037
    x5             0.1836      0.033      5.526      0.000       0.118       0.249
    x6             0.0984      0.061      1.615      0.106      -0.021       0.218
    x7            -0.0042      0.002     -2.601      0.009      -0.007      -0.001
    ==============================================================================
    Omnibus:                      532.576   Durbin-Watson:                   1.646
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):            10134.436
    Skew:                           0.397   Prob(JB):                         0.00
    Kurtosis:                      12.454   Cond. No.                         618.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Regression 6
Features: urban, male, price of housing, years of education, occupation, marital status, potential experience, price of housing squared, price_urban, occupation_urban

```python
X = np.array(urban[['urb','male','p_housing','potential_experience','occupation', 'marital_status',\
                            'p_housing2', 'price_urban', 'occupation_urban', 'year_of_education']])

y = np.array(urban['lny'])

#Add a constant to the regression model
X = sm.add_constant(X)

#Fit an OLS model
results = sm.OLS(y, X, missing='drop').fit()

#Inspect the results
print(results.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.138
    Model:                            OLS   Adj. R-squared:                  0.135
    Method:                 Least Squares   F-statistic:                     43.22
    Date:                Thu, 10 Dec 2020   Prob (F-statistic):           4.82e-80
    Time:                        14:25:29   Log-Likelihood:                -3024.1
    No. Observations:                2702   AIC:                             6070.
    Df Residuals:                    2691   BIC:                             6135.
    Df Model:                          10
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          1.7974      0.194      9.282      0.000       1.418       2.177
    x1            -0.4447      0.134     -3.319      0.001      -0.707      -0.182
    x2             0.2411      0.030      8.080      0.000       0.183       0.300
    x3             0.0111      0.005      2.255      0.024       0.001       0.021
    x4            -0.0043      0.002     -2.690      0.007      -0.007      -0.001
    x5             0.0986      0.042      2.371      0.018       0.017       0.180
    x6             0.1014      0.061      1.668      0.095      -0.018       0.221
    x7         -3.702e-05   3.26e-05     -1.134      0.257      -0.000     2.7e-05
    x8             0.0039      0.002      2.127      0.034       0.000       0.007
    x9             0.2041      0.059      3.436      0.001       0.088       0.321
    x10            0.0256      0.006      4.486      0.000       0.014       0.037
    ==============================================================================
    Omnibus:                      521.969   Durbin-Watson:                   1.656
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):            10075.415
    Skew:                           0.364   Prob(JB):                         0.00
    Kurtosis:                      12.432   Cond. No.                     9.32e+04
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 9.32e+04. This might indicate that there are
    strong multicollinearity or other numerical problems.

### Discussion

From regression 1 to 6, we see that the coefficient for x1 which is the indicator variable for urban is negative and not significant across all th regressions. This means that we do not observe an existing urban wage premium after controlling for the confounding variables. If we look at regression 1, for example, we see that the cost of living and being male for variable x2, and x3 have significant coefficients. Therefore, cost of living is an important component of the wage premium in China when we take the average price of housing per unit into account. In addition to cost of living, it is consistent across all the six regressions that year of education, occupation type, gender and years of working experience all contribute towards the wage premium.

I find that there is a premium for white-collar workers living in the urban areas from regression 6. The coefficient of x9 which is the interaction between white-collar workers and urban workers is positive and significant.

### Conclusion

Using OLS regression models, I investigate whether there is an urban wage premium in China. The data I use is from the China Health and Nutrition Survey. I account for the cost of living using the average housing price in addition to other observable individual characteristics. In my results, I find that there is no evidence for urban wage premium. Wage premium is entirely caused by other factors such as higher costs of living in urbanized settings, being male, having a white-collar job, having more years of education. I find a negative but not significant urban premium. The urban wage premium is mostly compensated for other factors such as the cost of living.

These results should be interpreted with caution. Regional price indexes are hard to find for China, therefore using housing prices can only approximate differences in the overall costs of living between urban and rural areas. As better data become available, better accounting for individual heterogeneity and regional differences in costs of living may be an interesting area of further research.

### Reference

Peng, Shu-hong. (2016). “Urban Scale and Wage Premium,” Contemporary Finance and Economics , 2016, (3).

Huang, F. and Wu, C.J. (2008) A Study on Wage Premium of Provincial Capital Cities in China—A Space Econometric Analysis Based on Quantile Regression. Journal of Finance and Economics, 34.

This project uses data from China Health and Nutrition Survey (CHNS). We are grateful to research grant funding from the National Institute for Health (NIH), the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) for R01 HD30880, National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) for R01DK104371 and R01HL108427, the NIH Fogarty grant D43 TW009077 for financial support for the CHNS data collection and analysis files since 1989, and the China-Japan Friendship Hospital, Ministry of Health for support for CHNS 2009, Chinese National Human Genome Center at Shanghai since 2009, and Beijing Municipal Center for Disease Prevention and Control since 2011. We thank the National Institute for Nutrition and Health, China Center for Disease Control and Prevention, Beijing Municipal Center for Disease Control and Prevention, and the Chinese National Human Genome Center at Shanghai.


# NMA CN Project

- **Pod**: careful wolfs ⚠🐺
- **Question**: is there a difference in neural activation at the cell level following stimulus presentation in the active vs passive conditions?

```python
# Only for Colab
!python -m pip install --upgrade pip
!pip install allensdk
```

    Collecting pip
      Downloading https://files.pythonhosted.org/packages/47/ca/f0d790b6e18b3a6f3bd5e80c2ee4edbb5807286c21cdd0862ca933f751dd/pip-21.1.3-py3-none-any.whl (1.5MB)
         |████████████████████████████████| 1.6MB 9.5MB/s
    Installing collected packages: pip
      Found existing installation: pip 19.3.1
        Uninstalling pip-19.3.1:
          Successfully uninstalled pip-19.3.1
    Successfully installed pip-21.1.3
    Collecting allensdk
      Downloading allensdk-2.11.2-py3-none-any.whl (1.7 MB)
         |████████████████████████████████| 1.7 MB 8.4 MB/s
    Requirement already satisfied: matplotlib<4.0.0,>=1.4.3 in /usr/local/lib/python3.7/dist-packages (from allensdk) (3.2.2)
    Collecting numpy<1.19.0,>=1.15.4
      Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
         |████████████████████████████████| 20.1 MB 1.5 MB/s
    Requirement already satisfied: scipy<2.0.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (1.4.1)
    Requirement already satisfied: jinja2<2.12.0,>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from allensdk) (2.11.3)
    Collecting hdmf<2.5.0
      Downloading hdmf-2.4.0-py2.py3-none-any.whl (149 kB)
         |████████████████████████████████| 149 kB 72.7 MB/s
    Collecting h5py<3.0.0,>=2.8
      Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
         |████████████████████████████████| 2.9 MB 41.1 MB/s
    Requirement already satisfied: semver in /usr/local/lib/python3.7/dist-packages (from allensdk) (2.13.0)
    Collecting nest-asyncio==1.2.0
      Downloading nest_asyncio-1.2.0-py3-none-any.whl (4.5 kB)
    Collecting marshmallow==3.0.0rc6
      Downloading marshmallow-3.0.0rc6-py2.py3-none-any.whl (42 kB)
         |████████████████████████████████| 42 kB 1.4 MB/s
    Requirement already satisfied: future<1.0.0,>=0.14.3 in /usr/local/lib/python3.7/dist-packages (from allensdk) (0.16.0)
    Collecting requests-toolbelt<1.0.0
      Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
         |████████████████████████████████| 54 kB 2.9 MB/s
    Collecting glymur==0.8.19
      Downloading Glymur-0.8.19.tar.gz (3.4 MB)
         |████████████████████████████████| 3.4 MB 35.0 MB/s
    Collecting tables<4.0.0,>=3.6.0
      Downloading tables-3.6.1-cp37-cp37m-manylinux1_x86_64.whl (4.3 MB)
         |████████████████████████████████| 4.3 MB 35.0 MB/s
    Collecting aiohttp==3.7.4
      Downloading aiohttp-3.7.4-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
         |████████████████████████████████| 1.3 MB 32.6 MB/s
    Collecting ndx-events<=0.2.0
      Downloading ndx_events-0.2.0-py2.py3-none-any.whl (13 kB)
    Collecting pynrrd<1.0.0,>=0.2.1
      Downloading pynrrd-0.4.2-py2.py3-none-any.whl (18 kB)
    Collecting scikit-build<1.0.0
      Downloading scikit_build-0.11.1-py2.py3-none-any.whl (72 kB)
         |████████████████████████████████| 72 kB 954 kB/s
    Collecting xarray<0.16.0
      Downloading xarray-0.15.1-py3-none-any.whl (668 kB)
         |████████████████████████████████| 668 kB 57.4 MB/s
    Collecting simplejson<4.0.0,>=3.10.0
      Downloading simplejson-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (129 kB)
         |████████████████████████████████| 129 kB 58.6 MB/s
    Collecting simpleitk<3.0.0,>=2.0.2
      Downloading SimpleITK-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (47.4 MB)
         |████████████████████████████████| 47.4 MB 32 kB/s
    Collecting pynwb<2.0.0,>=1.3.2
      Downloading pynwb-1.5.1-py2.py3-none-any.whl (98 kB)
         |████████████████████████████████| 98 kB 7.6 MB/s
    Requirement already satisfied: statsmodels<=0.13.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (0.10.2)
    Requirement already satisfied: six<2.0.0,>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (1.15.0)
    Requirement already satisfied: seaborn<1.0.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (0.11.1)
    Collecting pandas<=0.25.3,>=0.25.1
      Downloading pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
         |████████████████████████████████| 10.4 MB 33.1 MB/s
    Collecting argschema<3.0.0
      Downloading argschema-2.0.2.tar.gz (24 kB)
    Requirement already satisfied: requests<3.0.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (2.23.0)
    Collecting boto3==1.17.21
      Downloading boto3-1.17.21-py2.py3-none-any.whl (130 kB)
         |████████████████████████████████| 130 kB 71.3 MB/s
    Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from allensdk) (4.41.1)
    Collecting psycopg2-binary<3.0.0,>=2.7
      Downloading psycopg2_binary-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
         |████████████████████████████████| 3.4 MB 37.3 MB/s
    Requirement already satisfied: scikit-image<0.17.0,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from allensdk) (0.16.2)
    Collecting multidict<7.0,>=4.5
      Downloading multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142 kB)
         |████████████████████████████████| 142 kB 52.4 MB/s
    Requirement already satisfied: chardet<4.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp==3.7.4->allensdk) (3.0.4)
    Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp==3.7.4->allensdk) (21.2.0)
    Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp==3.7.4->allensdk) (3.7.4.3)
    Collecting async-timeout<4.0,>=3.0
      Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
    Collecting yarl<2.0,>=1.0
      Downloading yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294 kB)
         |████████████████████████████████| 294 kB 50.4 MB/s
    Collecting s3transfer<0.4.0,>=0.3.0
      Downloading s3transfer-0.3.7-py2.py3-none-any.whl (73 kB)
         |████████████████████████████████| 73 kB 1.8 MB/s
    Collecting jmespath<1.0.0,>=0.7.1
      Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
    Collecting botocore<1.21.0,>=1.20.21
      Downloading botocore-1.20.112-py2.py3-none-any.whl (7.7 MB)
         |████████████████████████████████| 7.7 MB 28.4 MB/s
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from glymur==0.8.19->allensdk) (57.0.0)
    Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.21.0,>=1.20.21->boto3==1.17.21->allensdk) (2.8.1)
    Collecting urllib3<1.27,>=1.25.4
      Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
         |████████████████████████████████| 138 kB 41.8 MB/s
    Collecting ruamel.yaml<1,>=0.15
      Downloading ruamel.yaml-0.17.10-py3-none-any.whl (108 kB)
         |████████████████████████████████| 108 kB 54.2 MB/s
    Requirement already satisfied: jsonschema<4,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from hdmf<2.5.0->allensdk) (2.6.0)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2<2.12.0,>=2.7.3->allensdk) (2.0.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib<4.0.0,>=1.4.3->allensdk) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib<4.0.0,>=1.4.3->allensdk) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib<4.0.0,>=1.4.3->allensdk) (1.3.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas<=0.25.3,>=0.25.1->allensdk) (2018.9)
    Collecting pynwb<2.0.0,>=1.3.2
      Downloading pynwb-1.5.0-py2.py3-none-any.whl (98 kB)
         |████████████████████████████████| 98 kB 7.5 MB/s
      Downloading pynwb-1.4.0-py2.py3-none-any.whl (94 kB)
         |████████████████████████████████| 94 kB 2.9 MB/s
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0->allensdk) (2.10)
    Collecting urllib3<1.27,>=1.25.4
      Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
         |████████████████████████████████| 127 kB 63.9 MB/s
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0->allensdk) (2021.5.30)
    Collecting ruamel.yaml.clib>=0.1.2
      Downloading ruamel.yaml.clib-0.2.6-cp37-cp37m-manylinux1_x86_64.whl (546 kB)
         |████████████████████████████████| 546 kB 59.1 MB/s
    Collecting distro
      Downloading distro-1.5.0-py2.py3-none-any.whl (18 kB)
    Requirement already satisfied: wheel>=0.29.0 in /usr/local/lib/python3.7/dist-packages (from scikit-build<1.0.0->allensdk) (0.36.2)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from scikit-build<1.0.0->allensdk) (20.9)
    Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image<0.17.0,>=0.14.0->allensdk) (2.5.1)
    Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image<0.17.0,>=0.14.0->allensdk) (2.4.1)
    Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image<0.17.0,>=0.14.0->allensdk) (7.1.2)
    Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image<0.17.0,>=0.14.0->allensdk) (1.1.1)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->scikit-image<0.17.0,>=0.14.0->allensdk) (4.4.2)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels<=0.13.0->allensdk) (0.5.1)
    Requirement already satisfied: numexpr>=2.6.2 in /usr/local/lib/python3.7/dist-packages (from tables<4.0.0,>=3.6.0->allensdk) (2.7.3)
    Building wheels for collected packages: glymur, argschema
      Building wheel for glymur (setup.py) ... done
      Created wheel for glymur: filename=Glymur-0.8.19-py3-none-any.whl size=2722014 sha256=9097c44d95af807085e1cfb1738093a2f047c96a005eae06c28fadc06778cd8b
      Stored in directory: /root/.cache/pip/wheels/36/bc/bd/1786279b44db4cbd6ded18090d02978fc31e78e3e516551bb4
      Building wheel for argschema (setup.py) ... done
      Created wheel for argschema: filename=argschema-2.0.2-py2.py3-none-any.whl size=17795 sha256=c63cf4f097a633e85e4ddbcb5dd61d8b8d497324a03ed3e5634d661d220be444
      Stored in directory: /root/.cache/pip/wheels/ee/97/a8/4dbf0458be0af473d083a6354c0c126490e82447237bb0b907
    Successfully built glymur argschema
    Installing collected packages: ruamel.yaml.clib, numpy, urllib3, ruamel.yaml, pandas, jmespath, h5py, multidict, hdmf, botocore, yarl, s3transfer, pynwb, marshmallow, distro, async-timeout, xarray, tables, simplejson, simpleitk, scikit-build, requests-toolbelt, pynrrd, psycopg2-binary, nest-asyncio, ndx-events, glymur, boto3, argschema, aiohttp, allensdk
      Attempting uninstall: numpy
        Found existing installation: numpy 1.19.5
        Uninstalling numpy-1.19.5:
          Successfully uninstalled numpy-1.19.5
      Attempting uninstall: urllib3
        Found existing installation: urllib3 1.24.3
        Uninstalling urllib3-1.24.3:
          Successfully uninstalled urllib3-1.24.3
      Attempting uninstall: pandas
        Found existing installation: pandas 1.1.5
        Uninstalling pandas-1.1.5:
          Successfully uninstalled pandas-1.1.5
      Attempting uninstall: h5py
        Found existing installation: h5py 3.1.0
        Uninstalling h5py-3.1.0:
          Successfully uninstalled h5py-3.1.0
      Attempting uninstall: xarray
        Found existing installation: xarray 0.18.2
        Uninstalling xarray-0.18.2:
          Successfully uninstalled xarray-0.18.2
      Attempting uninstall: tables
        Found existing installation: tables 3.4.4
        Uninstalling tables-3.4.4:
          Successfully uninstalled tables-3.4.4
      Attempting uninstall: nest-asyncio
        Found existing installation: nest-asyncio 1.5.1
        Uninstalling nest-asyncio-1.5.1:
          Successfully uninstalled nest-asyncio-1.5.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    tensorflow 2.5.0 requires h5py~=3.1.0, but you have h5py 2.10.0 which is incompatible.
    tensorflow 2.5.0 requires numpy~=1.19.2, but you have numpy 1.18.5 which is incompatible.
    nbclient 0.5.3 requires jupyter-client>=6.1.5, but you have jupyter-client 5.3.5 which is incompatible.
    google-colab 1.0.0 requires pandas~=1.1.0; python_version >= "3.0", but you have pandas 0.25.3 which is incompatible.
    fbprophet 0.7.1 requires pandas>=1.0.4, but you have pandas 0.25.3 which is incompatible.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    arviz 0.11.2 requires xarray>=0.16.1, but you have xarray 0.15.1 which is incompatible.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed aiohttp-3.7.4 allensdk-2.11.2 argschema-2.0.2 async-timeout-3.0.1 boto3-1.17.21 botocore-1.20.112 distro-1.5.0 glymur-0.8.19 h5py-2.10.0 hdmf-2.4.0 jmespath-0.10.0 marshmallow-3.0.0rc6 multidict-5.1.0 ndx-events-0.2.0 nest-asyncio-1.2.0 numpy-1.18.5 pandas-0.25.3 psycopg2-binary-2.9.1 pynrrd-0.4.2 pynwb-1.4.0 requests-toolbelt-0.9.1 ruamel.yaml-0.17.10 ruamel.yaml.clib-0.2.6 s3transfer-0.3.7 scikit-build-0.11.1 simpleitk-2.0.2 simplejson-3.17.3 tables-3.6.1 urllib3-1.25.11 xarray-0.15.1 yarl-1.6.3
    WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

## References

- [[AllenNB1]](https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_ophys_data_access.html) Download data using the AllenSDK or directly from our Amazon S3 bucket
- [[AllenNB2]](https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_ophys_dataset_manifest.html) Identify experiments of interest using the dataset manifest
- [[AllenNB3]](https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_load_ophys_data.html) Load and visualize data from a 2-photon imaging experiment
- [[AllenNB4]](https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_mouse_history.html) Examine the full training history of one mouse
- [[AllenNB5]](https://allensdk.readthedocs.io/en/latest/_static/examples/nb/visual_behavior_compare_across_trial_types.html) Compare behavior and neural activity across different trial types in the task

## Imports

Import and setup Python packages. You should not need to touch this section.

```python
from pathlib import Path
from tqdm import tqdm

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from allensdk.brain_observatory.behavior.behavior_project_cache import VisualBehaviorOphysProjectCache

# import mindscope_utilities
# import mindscope_utilities.visual_behavior_ophys as ophys
```

```python
np.random.seed(42)
```

## Setup AllenSDK

Configure AllenSDK to get `cache`, `sessions_df` and `experiments_df`. Data will be stored in `./allensdk_storage` by default.

```python
!mkdir -p allensdk_storage
DATA_STORAGE_DIRECTORY = Path("./allensdk_storage")
```

```python
cache = VisualBehaviorOphysProjectCache.from_s3_cache(cache_dir=DATA_STORAGE_DIRECTORY)
```

    ophys_session_table.csv: 100%|██████████| 165k/165k [00:00<00:00, 4.78MMB/s]
    behavior_session_table.csv: 100%|██████████| 885k/885k [00:00<00:00, 14.1MMB/s]
    ophys_experiment_table.csv: 100%|██████████| 336k/336k [00:00<00:00, 8.79MMB/s]

The data manifest is comprised of three types of tables:

1. `behavior_session_table`
2. `ophys_session_table`
3. `ophys_experiment_table`

The` behavior_session_table` contains metadata for every **behavior session** in the dataset. Some behavior sessions have 2-photon data associated with them, while others took place during training in the behavior facility. The different training stages that mice are progressed through are described by the session_type.

The `ophys_session_table` contains metadata for every 2-photon imaging (aka optical physiology, or ophys) session in the dataset, associated with a unique `ophys_session_id`. An **ophys session** is one continuous recording session under the microscope, and can contain different numbers of imaging planes (aka experiments) depending on which microscope was used. For Scientifica sessions, there will only be one experiment (aka imaging plane) per session. For Multiscope sessions, there can be up to eight imaging planes per session. Quality Control (QC) is performed on each individual imaging plane within a session, so each can fail QC independent of the others. This means that a Multiscope session may not have exactly eight experiments (imaging planes).

The `ophys_experiment_table` contains metadata for every **ophys experiment** in the dataset, which corresponds to a single imaging plane recorded in a single session, and associated with a unique `ophys_experiment_id`. A key part of our experimental design is targeting a given population of neurons, contained in one imaging plane, across multiple `session_types` (further described below) to examine the impact of varying sensory and behavioral conditions on single cell responses. The collection of all imaging sessions for a given imaging plane is referred to as an **ophys container**, associated with a unique `ophys_container_id`. Each ophys container may contain different numbers of sessions, depending on which experiments passed QC, and how many retakes occured (when a given session_type fails QC on the first try, an attempt is made to re-acquire the `session_type` on a different recording day - this is called a retake, also described further below).

*Text copied from [[AllenNB2]](#References)*

---

We will just use the `ophys_experiment_table`.

```python
experiments_df = cache.get_ophys_experiment_table()
```

## Specify Experiment

Get 1 active and 1 passive experiment that meet the following criteria:
- Excitatory cells
- Single-plane imaging

### Cre Line and Reporter Line

 The `cre_line` determines which genetically identified neuron type will be labeled by the reporter_line.

 This dataset have 3 `cre_line`:
 - **Slc17a7-IRES2-Cre**, which labels excitatory neurons across all cortical layers
 - **Sst-IRES-Cre** which labels somatostatin expressing inhibitory interneurons
 - **Vip-IRES-Cre**, which labels vasoactive intestinal peptide expressing inhibitory interneurons

*Text copied from [[AllenNB2]](#References)*

```python
experiments_df["cre_line"].unique()
```

    array(['Sst-IRES-Cre', 'Vip-IRES-Cre', 'Slc17a7-IRES2-Cre'], dtype=object)

There are also 3 `reporter_line`:
 - **Ai93(TITL-GCaMP6f)**, which expresses the genetically encoded calcium indicator GCaMP6f (f is for 'fast', this reporter has fast offset kinetics, but is only moderately sensitive to calcium relative to other sensors) in cre labeled neurons
 - **Ai94(TITL-GCaMP6s)**, which expresses the indicator GCaMP6s (s is for 'slow', this reporter is very sensitive to calcium but has slow offset kinetics), and
 - **Ai148(TIT2L-GC6f-ICL-tTA2)**, which expresses GCaMP6f using a self-enhancing system to achieve higher expression than other reporter lines (which proved necessary to label inhibitory neurons specifically).

```python
experiments_df["reporter_line"].unique()
```

    array(['Ai148(TIT2L-GC6f-ICL-tTA2)', 'Ai94(TITL-GCaMP6s)',
           'Ai93(TITL-GCaMP6f)'], dtype=object)

The specific `indicator` expressed by each `reporter_line` also has its own column in the table.

```python
experiments_df["indicator"].unique()
```

    array(['GCaMP6f', 'GCaMP6s'], dtype=object)

`full_genotype` contains information for both cre line and reporter line.

```python
experiments_df["full_genotype"].unique()
```

    array(['Sst-IRES-Cre/wt;Ai148(TIT2L-GC6f-ICL-tTA2)/wt',
           'Vip-IRES-Cre/wt;Ai148(TIT2L-GC6f-ICL-tTA2)/wt',
           'Slc17a7-IRES2-Cre/wt;Camk2a-tTA/wt;Ai94(TITL-GCaMP6s)/wt',
           'Slc17a7-IRES2-Cre/wt;Camk2a-tTA/wt;Ai93(TITL-GCaMP6f)/wt'],
          dtype=object)

---

We are looking at excitatory cells, so we should use `cre_line` of `Slc17a7-IRES2-Cre`. We did not select a particular `reporter_line`, so we can either.

```python
FULL_GENOTYPE = "Slc17a7-IRES2-Cre/wt;Camk2a-tTA/wt;Ai94(TITL-GCaMP6s)/wt"
```

### Project Code

"The distinct groups of mice are referred to as dataset variants and can be identified using the `project_code` column." [[AllenNB2]](#References)

```python
experiments_df["project_code"].unique()
```

    array(['VisualBehaviorMultiscope', 'VisualBehaviorTask1B',
           'VisualBehavior'], dtype=object)

---

We are interested in single-plane imaging, so either `VisualBehavior` or `VisualBehaviorTask1B` works.

```python
# We are looking at single-plane imaging
# "VisualBehavior" or "VisualBehaviorTask1B"
PROJECT_CODE = "VisualBehavior"
```

### Experiment

(Note that we are looking at single-plane imaging, so there is only one row (container) per mouse.)

#### `MOUSE_ID`

"The mouse_id is a 6-digit unique identifier for each experimental animal in the dataset." [[AllenNB2]](#References)

---

We retrieve all mouse that can be used for our experiment and select one mouse.

```python
experiments_df.query("project_code == @PROJECT_CODE") \
              .query("full_genotype == @FULL_GENOTYPE") \
              ["mouse_id"].unique()
```

    array([449441, 448900, 456564, 479426, 457766, 423606, 442709])

```python
MOUSE_ID = 449441
```

#### `ACTIVE_SESSION`, `PASSIVE_SESSION`

The session_type for each behavior session indicates the behavioral training stage or 2-photon imaging conditions for that particular session. This determines what stimuli were shown and what task parameters were used.

During the 2-photon imaging portion of the experiment, mice perform the task with the same set of images they saw during training (either image set A or B), as well as an additional novel set of images (whichever of A or B that they did not see during training). This allows evaluation of the impact of different sensory contexts on neural activity - familiarity versus novelty.
 - Sessions with **familiar images** include those starting with `OPHYS_0`, `OPHYS_1`, `OPHYS_2`, and `OPHYS_3`.
 - Sessions with **novel images** include those starting with `OPHYS_4`, `OPHYS_5`, and `OPHYS_6`.

Interleaved between **active behavior sessions** are **passive viewing sessions** where mice are given their daily water ahead of the sesssion (and are thus satiated) and view the stimulus with the lick spout retracted so they are unable to earn water rewards. This allows comparison of neural activity in response to stimuli under different behavioral context - active task engagement and passive viewing without reward. There are two passive sessions:
 - `OPHYS_2_images_A_passive`: passive session with familiar images
 - `OPHYS_5_images_A_passive`: passive session with novel images

*Text copied from [[AllenNB2]](#References)*

---

We check which sessions are available for this particular mouse and select one active and one passive session type. Not all sessions may be availble due to QC.

```python
experiments_df.query("project_code == @PROJECT_CODE") \
              .query("full_genotype == @FULL_GENOTYPE") \
              .query("mouse_id == @MOUSE_ID") \
              ["session_type"].unique()
```

    array(['OPHYS_2_images_A_passive', 'OPHYS_1_images_A', 'OPHYS_4_images_B',
           'OPHYS_6_images_B', 'OPHYS_5_images_B_passive'], dtype=object)

```python
ACTIVE_SESSION_TYPE = "OPHYS_1_images_A"
PASSIVE_SESSION_TYPE = "OPHYS_2_images_A_passive"
```

#### `ACTIVE_EXPERIMENT_ID`, `PASSIVE_EXPERIMENT_ID`

We retrieve the `ophys_experiment_id` of each session type we chose. We need this ID to get the experiment data.

```python
experiments_df.query("project_code == @PROJECT_CODE") \
              .query("full_genotype == @FULL_GENOTYPE") \
              .query("mouse_id == @MOUSE_ID") \
              .query("session_type == @ACTIVE_SESSION_TYPE") \
              .index
```

    Int64Index([879332693], dtype='int64', name='ophys_experiment_id')

```python
ACTIVE_EXPERIMENT_ID = 879332693
```

```python
experiments_df.query("project_code == @PROJECT_CODE") \
              .query("full_genotype == @FULL_GENOTYPE") \
              .query("mouse_id == @MOUSE_ID") \
              .query("session_type == @PASSIVE_SESSION_TYPE").index
```

    Int64Index([877696762], dtype='int64', name='ophys_experiment_id')

```python
PASSIVE_EXPERIMENT_ID = 877696762
```

## Explore Experiment

We can now download the experiment. Each experiment will be approximately 2GB in size.

```python
active_experiment = cache.get_behavior_ophys_experiment(ACTIVE_EXPERIMENT_ID)
```

    behavior_ophys_experiment_879332693.nwb: 100%|██████████| 1.93G/1.93G [00:50<00:00, 38.1MMB/s]

```python
active_experiment
```

    <allensdk.brain_observatory.behavior.behavior_ophys_experiment.BehaviorOphysExperiment at 0x7f0d913230d0>

This returns an instance of `BehaviorOphysExperiment`. It contains multiple attributes that we will need to explore.

### What's In Here?

#### `dff_traces`

"`dff_traces` dataframe contains traces for all neurons in this experiment, unaligned to any events in the task." [[AllenNB3]](#References)

```python
active_experiment.dff_traces.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>cell_roi_id</th>
      <th>dff</th>
    </tr>
    <tr>
      <th>cell_specimen_id</th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1086601848</th>
      <td>1080867796</td>
      <td>[0.28201794368039823, 0.1903022438869266, 0.21...</td>
    </tr>
    <tr>
      <th>1086602069</th>
      <td>1080867804</td>
      <td>[0.13884965897445056, 0.3963619942985226, 0.24...</td>
    </tr>
    <tr>
      <th>1086635982</th>
      <td>1080867837</td>
      <td>[0.5498754867600153, 0.26557468920820837, 0.43...</td>
    </tr>
    <tr>
      <th>1086578646</th>
      <td>1080867844</td>
      <td>[0.20886659311964556, 0.08442318243036104, 0.2...</td>
    </tr>
    <tr>
      <th>1086598188</th>
      <td>1080867847</td>
      <td>[0.28967144653172083, 0.49236151316004095, 0.4...</td>
    </tr>
  </tbody>
</table>
</div>

Since `dff` is stored as a list, we need to get timestamps for each of those numbers.

#### `ophys_timestamps`

`ophys_timestamps` contains the timestamps of every record.

```python
active_experiment.ophys_timestamps
```

    array([2.55695000e+00, 2.58926000e+00, 2.62157000e+00, ...,
           4.53777515e+03, 4.53780747e+03, 4.53783979e+03])

Let's do a sanity check by checking the length of both lists.

```python
print(f"dff has length {len(active_experiment.dff_traces.iloc[0]['dff'])}")
print(f"timestamp has length {len(active_experiment.ophys_timestamps)}")
```

    dff has length 140324
    timestamp has length 140324

#### `stimulus_presentations`

We also need timestamps of when stimulus was presented. This information is contained in `stimulus_presentations`.

```python
active_experiment.stimulus_presentations.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>duration</th>
      <th>end_frame</th>
      <th>image_index</th>
      <th>image_name</th>
      <th>image_set</th>
      <th>index</th>
      <th>omitted</th>
      <th>start_frame</th>
      <th>start_time</th>
      <th>stop_time</th>
      <th>is_change</th>
    </tr>
    <tr>
      <th>stimulus_presentations_id</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.25020</td>
      <td>18002.0</td>
      <td>0</td>
      <td>im065</td>
      <td>Natural_Images_Lum_Matched_set_training_2017.0...</td>
      <td>0</td>
      <td>False</td>
      <td>17987</td>
      <td>307.53005</td>
      <td>307.78025</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.25022</td>
      <td>18047.0</td>
      <td>0</td>
      <td>im065</td>
      <td>Natural_Images_Lum_Matched_set_training_2017.0...</td>
      <td>1</td>
      <td>False</td>
      <td>18032</td>
      <td>308.28098</td>
      <td>308.53120</td>
      <td>False</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.24985</td>
      <td>18092.0</td>
      <td>0</td>
      <td>im065</td>
      <td>Natural_Images_Lum_Matched_set_training_2017.0...</td>
      <td>2</td>
      <td>False</td>
      <td>18077</td>
      <td>309.03163</td>
      <td>309.28148</td>
      <td>False</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.25021</td>
      <td>18137.0</td>
      <td>0</td>
      <td>im065</td>
      <td>Natural_Images_Lum_Matched_set_training_2017.0...</td>
      <td>3</td>
      <td>False</td>
      <td>18122</td>
      <td>309.78188</td>
      <td>310.03209</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.25020</td>
      <td>18182.0</td>
      <td>0</td>
      <td>im065</td>
      <td>Natural_Images_Lum_Matched_set_training_2017.0...</td>
      <td>4</td>
      <td>False</td>
      <td>18167</td>
      <td>310.53247</td>
      <td>310.78267</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

During imaging sessions, stimulus presentations (other than the change and pre-change images) are omitted with a 5% probability, resulting in some inter stimlus intervals appearing as an extended gray screen period. [[AllenNB2]](#References)

```python
active_experiment.stimulus_presentations.query("omitted").head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>duration</th>
      <th>end_frame</th>
      <th>image_index</th>
      <th>image_name</th>
      <th>image_set</th>
      <th>index</th>
      <th>omitted</th>
      <th>start_frame</th>
      <th>start_time</th>
      <th>stop_time</th>
      <th>is_change</th>
    </tr>
    <tr>
      <th>stimulus_presentations_id</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>22</th>
      <td>0.25</td>
      <td>NaN</td>
      <td>8</td>
      <td>omitted</td>
      <td>omitted</td>
      <td>0</td>
      <td>True</td>
      <td>18976</td>
      <td>324.02684</td>
      <td>324.27684</td>
      <td>False</td>
    </tr>
    <tr>
      <th>119</th>
      <td>0.25</td>
      <td>NaN</td>
      <td>8</td>
      <td>omitted</td>
      <td>omitted</td>
      <td>1</td>
      <td>True</td>
      <td>23341</td>
      <td>396.83648</td>
      <td>397.08648</td>
      <td>False</td>
    </tr>
    <tr>
      <th>121</th>
      <td>0.25</td>
      <td>NaN</td>
      <td>8</td>
      <td>omitted</td>
      <td>omitted</td>
      <td>2</td>
      <td>True</td>
      <td>23431</td>
      <td>398.33747</td>
      <td>398.58747</td>
      <td>False</td>
    </tr>
    <tr>
      <th>123</th>
      <td>0.25</td>
      <td>NaN</td>
      <td>8</td>
      <td>omitted</td>
      <td>omitted</td>
      <td>3</td>
      <td>True</td>
      <td>23521</td>
      <td>399.83869</td>
      <td>400.08869</td>
      <td>False</td>
    </tr>
    <tr>
      <th>146</th>
      <td>0.25</td>
      <td>NaN</td>
      <td>8</td>
      <td>omitted</td>
      <td>omitted</td>
      <td>4</td>
      <td>True</td>
      <td>24556</td>
      <td>417.10313</td>
      <td>417.35313</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>

### Visualization

#### Plot dF/F Trace

Let's choose some random `cell_specimen_id` (1086601848) and plots its dff trace.

```python
fig, ax = plt.subplots(figsize=(15, 4))

ax.plot(
    active_experiment.ophys_timestamps,
    active_experiment.dff_traces.loc[1086601848]["dff"],
)

ax.set_xlim(400, 450)

fig.show()
```

#### Plot Stimulus

Let's also plot stimulus for a short interval.

*Part of code from [[AllenNB3]](#References)*

```python
# Create a color map for each image
unique_stimuli = [stimulus for stimulus in active_experiment.stimulus_presentations['image_name'].unique()]
colormap = {image_name: sns.color_palette()[image_number] for image_number, image_name in enumerate(np.sort(unique_stimuli))}
# Keep omitted image as white
colormap['omitted'] = (1,1,1)
```

```python
stimulus_presentations_sample = active_experiment.stimulus_presentations.query('stop_time >= 400 and start_time <= 450')
```

```python
fig, ax = plt.subplots(figsize=(15, 4))

for idx, stimulus in stimulus_presentations_sample.iterrows():
    ax.axvspan(stimulus['start_time'], stimulus['stop_time'], color=colormap[stimulus['image_name']], alpha=0.25)

ax.set_xlim(400, 450)

fig.show()
```

#### Both

```python
fig, ax = plt.subplots(figsize=(15, 4))

ax.plot(
    active_experiment.ophys_timestamps,
    active_experiment.dff_traces.loc[1086601848]["dff"],
)

for idx, stimulus in stimulus_presentations_sample.iterrows():
    ax.axvspan(stimulus['start_time'], stimulus['stop_time'], color=colormap[stimulus['image_name']], alpha=0.25)

ax.set_xlim(400, 450)
ax.set_ylim(-0.5, 0.5)
ax.legend(["dff trace"])

fig.show()
```

## Cell Selection

We would like to limit ourselves to looking at cells that respond to a presentation of a visual stimuli.

To understand which cells respond, we check the dF/F traces of cells 250ms before stimulus onset and 250ms after stimulus onset. Then, we choose the cells with the biggest difference in mean dF/F.

```python
TIME_BEFORE_STIMULUS = 0.25
TIME_AFTER_STIMULUS = 0.25
```

### Helper Functions

These are some helper functions for getting the right timestamps and dF/F traces.

```python
def get_cell_specimen_ids(experiment):
    """
    Get all the cell specimen ids recorded in the experiment.
    """
    return experiment.dff_traces.index
```

```python
def get_indices_in_interval(experiment, start, end):
    """
    Get all indices of timestamps where dF/F trace was recorded during a specified interval.
    """
    indices = np.argwhere((experiment.ophys_timestamps >= start) & (experiment.ophys_timestamps <= end))[:, 0]
    return indices
```

```python
def get_timestamps_in_interval(experiment, start, end):
    """
    Get all timestamps when dF/F trace was recorded during a specified interval.
    """
    indices = get_indices_in_interval(experiment, start, end)
    return experiment.ophys_timestamps[indices]
```

```python
def get_dff_in_interval(experiment, cell_specimen_id, start, end):
    """
    Get dF/F trace for one cell during a specified interval.
    """
    indices = get_indices_in_interval(experiment, start, end)
    return experiment.dff_traces.loc[cell_specimen_id]["dff"][indices[0]:indices[-1]+1]
```

### Selection with Single Stimulus

Let's get all the cell specimen IDs.

```python
cell_specimen_ids = get_cell_specimen_ids(active_experiment)
```

We can first choose one stimulus and find the cell with the biggest difference. Let's select the first stimulus.

```python
stimulus_start_time = active_experiment.stimulus_presentations["start_time"][0]
```

Then, we retrieve the dF/F traces before and after the stimulus with `get_dff_in_interval()` helper function. We then take the mean of both and save the difference.

```python
mean_dff_differences = []

for cell_specimen_id in tqdm(cell_specimen_ids):
    before_stimulus_dff = get_dff_in_interval(
        active_experiment,
        cell_specimen_id,
        start=stimulus_start_time - TIME_BEFORE_STIMULUS,
        end=stimulus_start_time,
    )
    after_stimulus_dff = get_dff_in_interval(
        active_experiment,
        cell_specimen_id,
        start=stimulus_start_time,
        end=stimulus_start_time + TIME_BEFORE_STIMULUS,
    )
    mean_dff_difference = np.abs(np.mean(after_stimulus_dff) - np.mean(before_stimulus_dff))
    mean_dff_differences.append(mean_dff_difference)

mean_dff_differences = np.array(mean_dff_differences)
```

    100%|██████████| 415/415 [00:00<00:00, 1201.96it/s]

Let's plot a histogram and a box plot to see what the distribution of mean dF/F differences look like.

```python
fig, axs = plt.subplots(1, 2, figsize=(12, 4))

axs[0].hist(mean_dff_differences)
axs[1].boxplot(mean_dff_differences)

fig.show()
```

Looks like there are a few cells that are outstanding! Let's select the top 4 cells and see what their mean differences look like.

```python
TOP_K = 4
top_k_indices = np.argsort(mean_dff_differences)[::-1][:TOP_K]
selected_cell_specimen_ids = cell_specimen_ids[top_k_indices]
selected_mean_dff_differences = mean_dff_differences[top_k_indices]
print(selected_cell_specimen_ids)
print(selected_mean_dff_differences)
```

    Int64Index([1086604838, 1086578313, 1086623819, 1086637612], dtype='int64', name='cell_specimen_id')
    [0.43820469 0.28274228 0.13674166 0.11737482]

We can also plot their dF/F traces before and after stimulus.

```python
fig, ax = plt.subplots(1, 1, figsize = (12, 4))

for cell_specimen_id, mean_dff_difference in zip(selected_cell_specimen_ids, selected_mean_dff_differences):
    ax.plot(
        get_timestamps_in_interval(
            active_experiment,
            stimulus_start_time - TIME_BEFORE_STIMULUS,
            stimulus_start_time + TIME_AFTER_STIMULUS,
        ),
        get_dff_in_interval(
            active_experiment,
            cell_specimen_id,
            stimulus_start_time - TIME_BEFORE_STIMULUS,
            stimulus_start_time + TIME_AFTER_STIMULUS,
        ),
        label=f"Cell {cell_specimen_id} (diff={mean_dff_difference})"
    )
ax.axvline(x=stimulus_start_time, color="k", linestyle="-")

ax.set_title(f"dF/F trace of cells")
ax.set_ylabel("dF/F")
ax.set_xlabel("time in session (seconds)")
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5), fancybox=True, shadow=True, ncol=1)
fig.show()
```

### Selection with All Stimulus

#### Compute dF/F Differences

TODO: Check if we also exclude changes

We now use all the stimulus (excluding the omitted ones) to select the cells.

```python
# active_stimulus_start_times = active_experiment.stimulus_presentations["start_time"]
active_stimulus_start_times = active_experiment.stimulus_presentations.query("not omitted")["start_time"]
```

Let's get the mean dF/F difference for each cell. Below is bunch of code that could be a bit difficult to understand. Check the **Explanation** section if you want to understand the code.

```python
dff_matrix = np.vstack(active_experiment.dff_traces.dff.values)
```

```python
mean_dff_differences_all_stimulus = []
for stimulus_start_time in tqdm(active_stimulus_start_times):
    before_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time - TIME_BEFORE_STIMULUS,
        end=stimulus_start_time,
    )
    after_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time,
        end=stimulus_start_time + TIME_AFTER_STIMULUS,
    )
    before_mean_dff = np.mean(dff_matrix[:, before_indices], axis=1)
    after_mean_dff = np.mean(dff_matrix[:, after_indices], axis=1)
    mean_dff_differences = np.abs(after_mean_dff - before_mean_dff)
    mean_dff_differences_all_stimulus.append(mean_dff_differences)

difference_matrix = np.stack(mean_dff_differences_all_stimulus, axis=1)
```

    100%|██████████| 4640/4640 [00:02<00:00, 2094.17it/s]

```python
mean_difference_per_cell = np.mean(difference_matrix, axis=1)
```

`mean_difference_per_cell` contains the mean difference dF/F trace for each cell.

```python
mean_difference_per_cell.shape
```

    (415,)

#### Explanation

We will create a 2D matrix of dF/F traces to parallelize computation. Remember how `dff_traces` looked like:

```python
active_experiment.dff_traces.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>cell_roi_id</th>
      <th>dff</th>
    </tr>
    <tr>
      <th>cell_specimen_id</th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1086601848</th>
      <td>1080867796</td>
      <td>[0.28201794368039823, 0.1903022438869266, 0.21...</td>
    </tr>
    <tr>
      <th>1086602069</th>
      <td>1080867804</td>
      <td>[0.13884965897445056, 0.3963619942985226, 0.24...</td>
    </tr>
    <tr>
      <th>1086635982</th>
      <td>1080867837</td>
      <td>[0.5498754867600153, 0.26557468920820837, 0.43...</td>
    </tr>
    <tr>
      <th>1086578646</th>
      <td>1080867844</td>
      <td>[0.20886659311964556, 0.08442318243036104, 0.2...</td>
    </tr>
    <tr>
      <th>1086598188</th>
      <td>1080867847</td>
      <td>[0.28967144653172083, 0.49236151316004095, 0.4...</td>
    </tr>
  </tbody>
</table>
</div>

We just stack these dff vectors vertically.

```python
dff_matrix = np.vstack(active_experiment.dff_traces.dff.values)
```

The `dff_matrix` has 415 rows that correspond to the number of cells, and 140324 columns that correspond to the number of timestamps.

```python
print(f"Shape: {dff_matrix.shape}")
print(f"Number of cells: {len(cell_specimen_ids)}")
print(f"Number of timestamps: {len(active_experiment.ophys_timestamps)}")
```

    Shape: (415, 140324)
    Number of cells: 415
    Number of timestamps: 140324

Each column of this matrix will have all the cell's dF/F value for a particular time. Then, for each stimulus, we can select the columns that correspond to some interval.

```python
before_indices = get_indices_in_interval(
    active_experiment,
    stimulus_start_time - TIME_BEFORE_STIMULUS,
    stimulus_start_time,
)
```

```python
dff_matrix[:, before_indices].shape
```

    (415, 8)

Then we can take the row-wise mean to get the mean dF/F trace for this interval for all the cells for this interval.

```python
before_mean_dff = np.mean(dff_matrix[:, before_indices], axis=1)
```

```python
before_mean_dff.shape
```

    (415,)

We can get two intervals corresponding to the time before and after a stimulus and find the difference.

```python
after_indices = get_indices_in_interval(
    active_experiment,
    stimulus_start_time,
    stimulus_start_time + TIME_BEFORE_STIMULUS,
)
```

```python
after_mean_dff = np.mean(dff_matrix[:, after_indices], axis=1)
```

```python
mean_dff_differences_2 = np.abs(after_mean_dff - before_mean_dff)
```

```python
mean_dff_differences_2.shape
```

    (415,)

Let's do a sanity check with the `mean_dff_differences` that we computed in the previous section.

```python
np.allclose(mean_dff_differences, mean_dff_differences_2)
```

    True

We can now create another matrix `difference_matrix`.

```python
mean_dff_differences_all_stimulus = []
for stimulus_start_time in tqdm(active_stimulus_start_times):
    before_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time - TIME_BEFORE_STIMULUS,
        end=stimulus_start_time,
    )
    after_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time,
        end=stimulus_start_time + TIME_AFTER_STIMULUS,
    )
    before_mean_dff = np.mean(dff_matrix[:, before_indices], axis=1)
    after_mean_dff = np.mean(dff_matrix[:, after_indices], axis=1)
    mean_dff_differences = np.abs(after_mean_dff - before_mean_dff)
    mean_dff_differences_all_stimulus.append(mean_dff_differences)

difference_matrix = np.stack(mean_dff_differences_all_stimulus, axis=1)
```

    100%|██████████| 4640/4640 [00:02<00:00, 2039.63it/s]

The difference_matrix has 415 rows that correspond to the number of cells, and 4807 columns that correspond to the number of stimuli.

```python
print(f"Shape: {difference_matrix.shape}")
print(f"Number of cells: {len(cell_specimen_ids)}")
print(f"Number of stimuli: {len(active_stimulus_start_times)}")
```

    Shape: (415, 4640)
    Number of cells: 415
    Number of stimuli: 4640

We can take the row-wise mean to get the mean difference of all stimuli for each cell.

```python
mean_difference_per_cell = np.mean(difference_matrix, axis=1)
```

#### Select Cells

Let's plot a histogram and a box plot of `mean_difference_per_cell` to look at the distribution.

```python
fig, axs = plt.subplots(1, 2, figsize=(12, 4))

axs[0].hist(mean_difference_per_cell)
axs[1].boxplot(mean_difference_per_cell)

fig.show()
```

Interestingly, looks like there is one cell that stick out, and also quite a few outliers.

```python
TOP_K = 4
top_k_indices = np.argsort(mean_difference_per_cell)[::-1][:TOP_K]
selected_cell_specimen_ids = cell_specimen_ids[top_k_indices]
selected_mean_dff_differences = mean_difference_per_cell[top_k_indices]
print(selected_cell_specimen_ids)
print(selected_mean_dff_differences)
```

    Int64Index([1086579256, 1086592299, 1086578313, 1086637417], dtype='int64', name='cell_specimen_id')
    [0.08101563 0.06412871 0.06329187 0.05767666]

Let's plot the dF/F trace for these top-K cells before and after one stimulus.

```python
stimulus_start_time = active_stimulus_start_times[0]
```

```python
fig, ax = plt.subplots(1, 1, figsize = (12, 4))

for cell_specimen_id, mean_dff_difference in zip(selected_cell_specimen_ids, selected_mean_dff_differences):
    ax.plot(
        get_timestamps_in_interval(
            active_experiment,
            stimulus_start_time - TIME_BEFORE_STIMULUS,
            stimulus_start_time + TIME_AFTER_STIMULUS,
        ),
        get_dff_in_interval(
            active_experiment,
            cell_specimen_id,
            stimulus_start_time - TIME_BEFORE_STIMULUS,
            stimulus_start_time + TIME_AFTER_STIMULUS,
        ),
        label=f"Cell {cell_specimen_id} (diff={mean_dff_difference})"
    )
ax.axvline(x=stimulus_start_time, color="k", linestyle="-")

ax.set_title(f"dF/F trace of cells")
ax.set_ylabel("dF/F")
ax.set_xlabel("time in session (seconds)")
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5), fancybox=True, shadow=True, ncol=1)
fig.show()
```

We can also plot these for the first few stimulus.

```python
fig, ax = plt.subplots(1, 1, figsize = (12, 4))

for stimulus_start_time in active_stimulus_start_times[:5]:
    for cell_specimen_id in selected_cell_specimen_ids:
        ax.plot(
            get_timestamps_in_interval(active_experiment, stimulus_start_time-0.25, stimulus_start_time+0.25),
            get_dff_in_interval(active_experiment, cell_specimen_id, stimulus_start_time-0.25, stimulus_start_time+0.25),
        )
    ax.axvline(x=stimulus_start_time, color="k", linestyle="-")

ax.set_title('dF/F responses')
ax.set_ylabel('dF/F')
ax.set_xlabel('time in session (seconds)')
fig.tight_layout()
fig.show()
```

## Representational Dissimilarity Matrix (RDM)

from https://colab.research.google.com/github/NeuromatchAcademy/course-content/blob/master/tutorials/W2D1_DeepLearning/student/W2D1_Tutorial3.ipynb#scrollTo=hOIuWVZ0Xq_E

To quantify this, we begin by computing the **representational dissimilarity matrix (RDM)** for the mouse V1 data and each model layer. This matrix, which we'll call $\mathbf{M}$, is computed as one minus the correlation coefficients between population responses to each stimulus. We can efficiently compute this by using the $z$-scored responses.

The $z$-scored response of all neurons $\mathbf{r}$ to stimulus $s$ is the response mean-subtracted across neurons $i$ and normalized to standard deviation 1 across neurons $i$ where $N$ is the total number of neurons:
\begin{equation}
  \mathbf{z}^{(s)} = \frac{\mathbf{r}^{(s)} - \mu^{(s)}}
  {\sigma^{(s)}}
\end{equation}
where $\mu^{(s)} = \frac{1}{N}\sum_{i=1}^N r_i^{(s)}$ and
$\sigma^{(s)} = \sqrt{\frac{1}{N}\sum_{i=1}^N \left( r_i^{(s)} - \mu^{(s)} \right)^2}$.

Then the full matrix can be computed as:
\begin{gather}
  \mathbf{M} = 1 - \frac{1}{N} \mathbf{ZZ}^T \\
\end{gather}
where $\mathbf{Z}$ is the z-scored response matrix with rows $\mathbf{r}^{(s)}$ and N is the number of neurons (or units). See [bonus section 3](#b3) for full explanation.

```python
time_after_stimulus= 0.250  # seconds

dff_matrix = np.vstack(active_experiment.dff_traces.dff.values)
```

```python
active_experiment.stimulus_presentations["image_name"].unique()
```

    array(['im065', 'omitted', 'im062', 'im061', 'im063', 'im066', 'im085',
           'im077', 'im069'], dtype=object)

```python
r_lst = []

# pandas
for stimulus_name in active_experiment.stimulus_presentations.query("not omitted")["image_name"].unique():
  presentations = active_experiment.stimulus_presentations.query('image_name == @stimulus_name')
  print("Number of presentations", len(presentations))
  presentation_means = []  # <number of stimulus presentations> x <number of cells>
  for stimulus_onset in tqdm(presentations["start_time"]):
    indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_onset,
        end=stimulus_onset+time_after_stimulus,
    )
    presentation_means.append(np.mean(dff_matrix[:, indices], axis=1))  # mean dff [cell1, cell2, ..., celln]
  cell_means = np.mean(np.vstack(presentation_means), axis=0)  # mean dff over all stim presentations [cell1, cell2, ..., celln]
  print(cell_means.shape)
  r_lst.append(cell_means)
  # make_matrix()  # rows are all stimulus presentations, columns are neurons, values mean dff from stimulus onset to 250ms after

r = np.vstack(r_lst)
print(r.shape)
```

    100%|██████████| 641/641 [00:00<00:00, 4188.69it/s]

    Number of presentations 641
    (415,)

    100%|██████████| 629/629 [00:00<00:00, 3963.62it/s]
      0%|          | 0/597 [00:00<?, ?it/s]

    Number of presentations 629
    (415,)
    Number of presentations 597

    100%|██████████| 597/597 [00:00<00:00, 3245.68it/s]
    100%|██████████| 538/538 [00:00<00:00, 3802.50it/s]
      0%|          | 0/494 [00:00<?, ?it/s]

    (415,)
    Number of presentations 538
    (415,)
    Number of presentations 494

    100%|██████████| 494/494 [00:00<00:00, 3400.16it/s]
    100%|██████████| 569/569 [00:00<00:00, 3944.21it/s]
      0%|          | 0/509 [00:00<?, ?it/s]

    (415,)
    Number of presentations 569
    (415,)
    Number of presentations 509

    100%|██████████| 509/509 [00:00<00:00, 3581.15it/s]
    100%|██████████| 663/663 [00:00<00:00, 3501.85it/s]

    (415,)
    Number of presentations 663
    (415,)
    (8, 415)

```python
print(r.shape)
print(r)
```

    (8, 415)
    [[ 0.0014368  -0.00141839  0.00231166 ...  0.0027447   0.07150208
       0.00516828]
     [-0.00017142  0.05090438  0.00174555 ... -0.0017843   0.00632535
       0.00117901]
     [ 0.00132981  0.00038188  0.0024744  ...  0.00149717  0.00501984
      -0.0002216 ]
     ...
     [ 0.00237082  0.0024943   0.00181042 ...  0.00013242  0.00602932
       0.00099942]
     [ 0.00168079  0.00661778  0.00240998 ...  0.00159245  0.00299351
       0.0023609 ]
     [-0.0004427   0.01225633  0.00307946 ...  0.00197598  0.13105824
       0.01868374]]

Now compute the RDM.

```python
 from scipy.stats import zscore
 zresp = zscore(r, axis=1)
print(zresp.shape)  # according to NMA r is shape <stimulus> x <cell count>
  # Compute RDM
RDM = 1 - (zresp @ zresp.T) / zresp.shape[1]
print(RDM.shape)
```

    (8, 415)
    (8, 8)

```python
ax = sns.heatmap(RDM, cmap="G") # red is very dissimlar, blue is very similar
ax.set_title("RDM active")
None
```

```python
active_experiment
```

```python
mean_dff_differences_all_stimulus = []
for stimulus_start_time in tqdm(active_stimulus_start_times):
    before_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time - TIME_BEFORE_STIMULUS,
        end=stimulus_start_time,
    )
    after_indices = get_indices_in_interval(
        active_experiment,
        start=stimulus_start_time,
        end=stimulus_start_time + TIME_AFTER_STIMULUS,
    )
    before_mean_dff = np.mean(dff_matrix[:, before_indices], axis=1)
    after_mean_dff = np.mean(dff_matrix[:, after_indices], axis=1)
    mean_dff_differences = np.abs(after_mean_dff - before_mean_dff)
    mean_dff_differences_all_stimulus.append(mean_dff_differences)

difference_matrix = np.stack(mean_dff_differences_all_stimulus, axis=1)
```

## Hypothesis Testing

We selected 4 cells in the previous section.

```python
selected_cell_specimen_ids
```

    Int64Index([1086579256, 1086592299, 1086578313, 1086637417], dtype='int64', name='cell_specimen_id')

Let's select one cell to analyze.

```python
selected_cell_specimen_id = selected_cell_specimen_ids[0]
selected_cell_specimen_id
```

    1086579256

To test our hypothesis, we need both active and passive sessions, so let's load the passive session now.

```python
passive_experiment = cache.get_behavior_ophys_experiment(PASSIVE_EXPERIMENT_ID)
```

```python
passive_stimulus_start_times = passive_experiment.stimulus_presentations.query("not omitted")["start_time"]
```

### One stimulus

```python
active_stimulus_start_time = active_stimulus_start_times[0]
passive_stimulus_start_time = passive_stimulus_start_times[0]
```

```python
active_timestamps = get_timestamps_in_interval(active_experiment, active_stimulus_start_time, active_stimulus_start_time+0.250)
active_dff = get_dff_in_interval(active_experiment, selected_cell_specimen_id, active_stimulus_start_time, active_stimulus_start_time+0.250)
```

```python
passive_timestamps = get_timestamps_in_interval(passive_experiment, passive_stimulus_start_time, passive_stimulus_start_time+0.250)
passive_dff = get_dff_in_interval(passive_experiment, selected_cell_specimen_id, passive_stimulus_start_time, passive_stimulus_start_time+0.250)
```

```python
fig, ax = plt.subplots()
ax.plot(active_timestamps - active_timestamps[0], active_dff, label="active")
ax.plot(passive_timestamps - passive_timestamps[0], passive_dff, label="passive")
fig.legend()
fig.show()
```

### Multiple stimulus (WIP)

```python
diff_traces_per_stimulus = []
active_dffs = []
for stimulus_start_time in tqdm(active_stimulus_start_times):
    dff = get_dff_in_interval(active_experiment, selected_cell_specimen_id, stimulus_start_time, stimulus_start_time+0.250)
    active_dffs.append(dff)
```

    100%|██████████| 4640/4640 [00:01<00:00, 2641.66it/s]

```python
from collections import Counter

Counter([len(active_dff) for active_dff in active_dffs])
```

    Counter({8: 3411, 7: 1229})