---
Project for the course in Microeconometrics | Summer 2020, M.Sc. Economics, Bonn University | Julia Wilhelm

# Replication of F. Barrera-Osorio, M. Bertrand, L. L. Linden, F. Perez-Calle  (2011)
---

This notebook contains my replication of the results from the following paper:

> Barrera-Osorio, Felipe, Marianne Bertrand, Leigh L. Linden, and Francisco Perez-Calle (2011). "Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia." American Economic Journal: Applied Economics, 3 (2): 167-95.

The original paper, as well as the data and code provided by the authors can be accessed [here](https://www.aeaweb.org/articles?id=10.1257/app.3.2.167).

Table of Contents

1. Introduction

2. Identification

3. Empirical Strategy

4. Replication of Barrera-Osorio et al. (2011)

4.1. Data & Descriptive Statistics

4.2. Baseline Comparison

4.3. Results

4.3.1. Attendence

4.3.2. Re-enrollment

4.3.3. Survey-Based Outcomes - Graduation and Tertiary Enrollment

4.3.4. Siblings Effects

5. Critical Assessment
6. Extensions

6.1. Check for Balanced Groups across Experiments

6.2. Elimination of Back-door Paths Controlling only for Locality

7. Conclusion

8. References

```python
import pandas as pd
import numpy as np
import pandas.io.formats.style
import statsmodels.api as sm_api
```

```python
from auxiliary.auxiliary_tables import *
```

---
# 1. Introduction
---

Barrera-Osorio et al. (2011) compare three education-based conditional cash transfer designs aimed at incentivizing academic participation. Using data from a pilot study in Bogota, Colombia they examine the effects of a bi-monthly transfer (basic treatment), a bi-monthly transfer combined with a lump-sum payment at the time students are supposed to re-enroll in school (savings treatment) and a bi-monthly transfer combined with a large payment upon graduation (tertiary treatment). The payments are conditional on school attendence of the child and designed to prevent dropout from secondary schools and to encourage matriculation at tertiary institutions. On the one hand, the savings and tertiary treatment impose more binding short-term liquidity constraints on participating families than the basic treatment. The authors examine whether this decreases monthly school attendence. On the other hand, these two treatments might provide stronger incentives for families to re-enroll their children at school or for graduation.

To estimate and compare the causal impact of the three treatments, Barrera-Osorio et al. (2011) apply a difference model to the data from Bogota. The Secretary of Education of the City implemented a pilot study running for one year, where they randomly allocated treatments to children in two localities. They randomize at child-level, generating variation within schools and families. This allows the authors to assess comparability of different groups. Barrera-Osorio et al. (2011) find that all designs significantly increase attendance and that the savings and tertiary treatments increase enrollment rates more strongly than the basic treatment. They conclude, that the structure of the intervention can help targeting resources.

In this notebook, I replicate the results presented in the paper by Barrera-Osorio et al. (2011). Additionally, I critically discuss the quality of the strategy and the results. My analysis supports the findings of Barrera-Osorio et al. (2011).

This notebook is structured as follows. In the next section, I present the identification strategy Barrera-Osorio et al. (2011) use to unravel the causal effects of the conditional cash transfers (section 2). Section 3 briefly discusses the empirical strategy the authors use for estimation. Section 4 shows my replication of the results of the paper, and section 5 is a critical discussion thereof. In section 6 I check the identification assumption across the two experiments and conduct regressions conditioning on one variable that blocks all back-door paths from the causal variable to the outcome variable to identify causal effects. Section 7 presents some conclusions.

---
# 2. Identification
---

Barrera-Osorio et al. (2011) aim to answer the question on how three different education-based conditional cash transfer designs perform in preventing dropout from secondary schools and encouraging matriculation at tertiary institutions.
The different treatments were implemented in two localities in Bogota, San Cristobal and Suba. Eligible children in San Cristobal were randomly assigned between a control group, the basic treatment (bi-monthly transfer) and the savings treatment (bi-monthly transfer combined with a lump-sum payment at the time students are supposed to re-enroll in school). In Suba eligible children were randomly assigned to the tertiary treatment (bi-monthly transfer combined with a large payment upon graduation) and a control group. Since it is impossible to observe treatment effects at the individual level, researchers thus estimate average effects using treatment and control groups. For each individual $i$ we can image a potential outcome where they are treated $Y_i(1)$ and where they are not $Y_i(0)$, but we can never simultaneously observe both outcomes for each individual. The random treatment assignment allows the authors to estimate the causal effects of the three treatments using experiments. While they can compare the effect of the basic and savings treatment directly, comparing those with the tertiary treatment they cannot rely on purely random variation. This is because the tertiary treatment was implemented in another locality and therefore, the comparison to the tertiary treatment occurs across experiments.

Since treatments were assigned randomly within the two localities, potential outcomes are independent of the treatment indicator $D$ and the selection bias is eliminated. The naive estimate which simply compares the observed average outcome of the treatment and control groups, then equals the true average treatment effect:

\begin{align*}
  E[Y\mid D = 1] - E[Y\mid D = 0] & = E[Y^1\mid D = 1] - E[Y^0\mid D = 0] \\
                                  & =E[Y^1\mid D = 1]  - E[Y^0\mid D = 1] + E[Y^0\mid D = 1] - E[Y^0\mid D = 0]  \\
                                  & = \underbrace{E[Y^1 - Y^0\mid D = 1]}_{ATT} + \underbrace{E[Y^0\mid D= 1]- E[Y^0 \mid D = 0]}_{\text{Selection bias}} \\
                                  & =E[Y^1 - Y^0\mid D = 1] \\
                                  & =E[Y^1 - Y^0\mid D = 0] \\
                                  & =E[Y^1 - Y^0]
\end{align*}

The authors here rely on the following two assumptions:
\begin{align*}
E[Y^1\mid D = 1] = E[Y^1\mid D = 0] \\
E[Y^0\mid D = 1] = E[Y^0\mid D = 0] \\
\end{align*}

The causal graphs below illustrate the relationship between the treatments $D_B$, $D_S$, $D_T$ and outcome $Y$ in the two localities. Additionally there may be observables $W$ and unobservables $U$ also affecting $Y$. Due to random treatment assignment within the two localities, treatment is independent of $W$ and $U$ and there is no back-door path which has to be eliminated.

**San Cristobal:**

$D_B$: Basic treatment
$D_S$: Savings treatment
$Y$: Students outcome
$U$: Unobservables
$W$: Observables

**Suba:**

$D_T$: Tertiary treatment
$Y$: Students outcome
$U$: Unobservables
$W$: Observables

The identification assumption to eliminate causal effects is, that randomization is successful within the localities. Barrera-Osorio et al. (2011) account for this checking whether treatment assignment created balanced treatment and control groups using household- and individual-level characteristics. These information were collected prior to the randomization, which suggests that students in each group should, on average, have similar characteristics. The authors make 60 comparisons and find 7 differences that are statistically significant at the 10 percent level, 5 at the 5 percent level and 2 at the 1 percent level. They conclude that randomization of the treatment assignment is successful.

In order to compare the effects of all three treatments the authors consider the experiments in the two localities together. The causal graph has three back-door paths which have to be eliminated. Treatment assignment is not completely random, since it is not random whether a person lives in Suba or San Cristobal. Observable or unobservable factors may affect treatment assignment and the outcome at the same time. The causal graph the authors use looks as follows:

$D_B$: Basic treatment
$D_S$: Savings treatment
$D_T$: Tertiary treatment
$L$: Dummy for locality of household (Suba or San Cristobal)
$Y$: Students outcome
$U$: Unobservables
$W$: Observables

In order to eliminate the back-door paths Barrera-Osorio et al. (2011) control for the locality of the householods and a large set of observable demographic characteristics. Nevertheless, the authors mention that differences between the tertiary treatment and the other treatments could be due to unobserved heterogeneity in treatment effects. I will refer to this problem later again.

---
# 3. Empirical Strategy
---
Barrera-Osorio et al. (2011) examine the impact of the basic, the savings and the tertiary treatment on student outcome. They use a simple difference model that compares different subsets of the sample without controlling for covariates.

For the basic-savings experiment in San Cristobal the specification takes the following form:

\begin{equation}
y_{ij} = \beta_0 + \beta_B Basic_i + \beta_S Savings_i + \epsilon_{ij}
\end{equation}

For the tertiary experiment in Suba the specification takes the following form:

\begin{equation}
y_{ij} = \beta_0 + \beta_T Tertiary_i + \epsilon_{ij}
\end{equation}
* $y_{ij}$ denotes a particular outcome for child $i$ in school $j$,
* $Basic_i$, $Savings_i$ and $Tertiary_i$ are indicator variables for whether or not the child is in the respective treatment group,
* $\epsilon_{ij}$ is the error term, which is allowed to vary up to the school level.

The authors additionally use a difference estimator that controls for socio-demographic and school characteristics.
For the basic-savings experiment the model is specified as follows:

\begin{equation}
y_{ij} = \beta_0 + \beta_B Basic_i + \beta_S Savings_i + \delta X_{ijk} + \phi_{j} + \epsilon_{ij}
\end{equation}

For the tertiary treatment the model is specified as follows:

\begin{equation}
y_{ij} = \beta_0 + \beta_T Tertiary_i + \delta X_{ijk} + \phi_{j} + \epsilon_{ij}
\end{equation}

The variables are defined as before. Additionally,
* $X_{ijk}$ is a vector of socio-demographic controls for child $i$ in school $j$ and family $k$,
* $\phi_{j}$ are school fixed effects.

---
# 4. Replication of Barrera-Osorio et al. (2011)
---

## 4.1. Data & Descriptive Statistics
Barrera-Osorio et al. (2011) restricted their sample of students spread across 251 schools to the 68 school with the largest number of registered students. Additionally, they filter the data by those students who completed a baseline survey they conducted. For the tertiary experiment they drop students in grade 6-8 since those were not eligible for the program.

```python
data = pd.read_stata('data/Public_Data_AEJApp_2010-0132.dta')
data.index.name = "individual"
data['grade_group'] = 'Grades 6-8'
data.loc[data['grade'] > 8, 'grade_group'] = 'Grades 9-10'
data.loc[data['grade'] > 10, 'grade_group'] = 'Grade 11'
data['group'] = 'Control'
data.loc[data['T1_treat'] == 1, 'group'] = 'Basic'
data.loc[data['T2_treat'] == 1, 'group'] = 'Savings'
data.loc[data['T3_treat'] == 1, 'group'] = 'Tertiary'
sample = data.drop(data[(data.suba == 1) & (data.grade < 9)].index)
sample['s_teneviv_int'] = sample['s_teneviv'].cat.codes + 1
sample['s_sexo_int'] = sample['s_sexo'].cat.codes
sample['s_estcivil_int'] = sample['s_estcivil'].cat.codes + 1
sample = sample.join(pd.get_dummies(sample['school_code']))
sample = sample.join(pd.get_dummies(sample['s_teneviv']))
sample = sample.join(pd.get_dummies(sample['s_estcivil']))
sample = sample.join(pd.get_dummies(sample['grade'], prefix='grade'))
sample = sample.join(pd.get_dummies(sample['s_estrato'],  prefix='estrato'))
sample_baselinesurvey =  sample.drop(sample[sample.bl_observed == 0].index)
```

Table 1 summarizes the distribution of children by grade, gender and experimental group. They end up with a sample of 7158 students.

#### Table 1- Distribution of Subjects by Research Groups

```python
create_table1(sample_baselinesurvey)
```

      Experiment
      Basic-Savings
      Tertiary
      Total

      Group
      Control
      Basic
      Savings
      Control
      Tertiary

      Grades 6-8
      1189
      1215
      1166
      0
      0
      3570

      Grades 9-10
      643
      633
      586
      449
      425
      2736

      Grade 11
      179
      188
      177
      160
      148
      852

      Female
      1047
      1022
      1000
      361
      336
      3766

      Male
      964
      1014
      929
      248
      237
      3392

      Total
      2011
      2036
      1929
      609
      573
      7158

## 4.2. Baseline Comparison
Barrera-Osorio et al. (2011) check if the randomization was successful and created balanced research groups. To do so, they compare characteristics of students between research groups. Table 2 shows the averages of 15 different variables for the control groups in the basic-savings experiment (B-S) and the tertiary experiment (T). The 4 other columns show 60 comparisons between the treatment and control groups. The standard errors are in the row below each difference, labeled with "SE". 7 differences are statistically significant at the 10 percent level, 5 at the 5 percent level, and 2 at the 1 percent level. Since the number of statistically significant differences is low, one can conclude that treatment is assigned randomly within the localities, which supports the identification assumption of their strategy.

#### Table 2- Comparison of Students between Research Groups

```python
sancristobal = sample.drop(sample[sample.suba == 1].index)
suba = sample.drop(sample[sample.suba == 0].index)
create_table2(sancristobal, suba)
```

      Control average B-S
      Basic-Control
      Savings-Control
      Basic-Savings
      Control average T
      Tertiary-Control

      Possessions
      1.90
      0.07
      0.04
      0.03
      1.94
      -0.05

      Possessions SE
      1.1
      0.02
      0.02
      0.02
      1.02
      0.04

      Utilities
      4.65
      -0.02
      0.06
      -0.08
      4.85
      0.04

      Utilities SE
      1.42
      0.03
      0.03
      0.03
      1.32
      0.04

      Durable Goods
      1.37
      -0.02
      0.01
      -0.03
      1.63
      0.02

      Durable Goods SE
      0.89
      0.02
      0.02
      0.02
      0.86
      0.03

      Physical Infrastructure
      11.65
      -0.05
      0.04
      -0.09
      12.14
      -0.05

      Physical Infrastructure SE
      1.75
      0.03
      0.03
      0.04
      1.49
      0.06

      Age
      14.38
      0.09
      -0.06
      0.16
      15.67
      -0.06

      Age SE
      5.3
      0.10
      0.14
      0.17
      4.23
      0.19

      Gender
      0.50
      0.00
      -0.00
      0.01
      0.45
      -0.01

      Gender SE
      0.5
      0.01
      0.01
      0.01
      0.5
      0.02

      Years of Education
      5.61
      -0.02
      -0.01
      -0.02
      7.43
      -0.05

      Years of Education SE
      1.86
      0.04
      0.05
      0.04
      1.34
      0.05

      Single Head
      0.30
      0.02
      0.01
      0.01
      0.27
      0.01

      Single Head SE
      0.46
      0.01
      0.01
      0.01
      0.44
      0.02

      Age of Head
      45.92
      -0.07
      0.11
      -0.19
      46.21
      0.21

      Age of Head SE
      10.27
      0.17
      0.23
      0.21
      8.59
      0.30

      Years of ed., head
      5.65
      -0.11
      -0.18
      0.07
      5.94
      -0.13

      Years of ed., head SE
      2.94
      0.08
      0.07
      0.07
      2.94
      0.09

      People in Household
      5.42
      -0.04
      -0.03
      -0.02
      5.16
      -0.01

      People in Household SE
      2.01
      0.05
      0.05
      0.04
      1.78
      0.07

      Member under 18
      2.57
      0.03
      0.01
      0.01
      2.31
      0.05

      Member under 18 SE
      1.35
      0.03
      0.03
      0.03
      1.2
      0.06

      Estrato
      1.44
      -0.01
      0.02
      -0.03
      1.63
      -0.01

      Estrato SE
      0.83
      0.02
      0.02
      0.02
      0.77
      0.03

      SISBEN score
      11.76
      -0.11
      -0.02
      -0.10
      13.44
      0.03

      SISBEN score SE
      4.64
      0.08
      0.11
      0.10
      4.33
      0.18

      Household income (1,000 pesos)
      366.70
      -4.73
      -0.35
      -4.37
      402.03
      3.18

      Household income (1,000 pesos) SE
      241.01
      5.77
      6.17
      6.64
      235.44
      7.67

## 4.3. Results

### 4.3.1 Attendence
First, the authors analyse the effects of the conditional cash transfers on the school attendence rate. They include only individuals who are enrolled in one of the 68 schools selected for surveying. They also restrict the sample to students in grades 6-10, because they exclude students who are in grade 11 for the enrollment effect estimations, since they should graduate rather than re-enroll. In order to replicate their results I run simple regressions of the school attendence rate on the treatment variable, once without control variables, once with demographic controls and once with demographic controls and school fixed effects. The first three columns of table 3 show the results for the basic-savings experiment in Sancristobal, while columns 4 to 6 show the results for the tertiary experiment in Suba. The last column shows results of a regression containing all three treatments, demographic controls and school fixed effects. The estimated treatment effects and their standard errors ("SE") are provided in rows 1-6. The test statistics from comparisons of the relative treatment effects and their p-values are in rows 7-10.

#### Table 3 - Effects on monitored school attendence rates

```python
sancristobal = sample.drop(sample[sample.suba == 1].index)
suba = sample.drop(sample[sample.suba == 0].index)
sancristobal = sancristobal.drop(sancristobal[(sancristobal.survey_selected == 0) | (sancristobal.grade == 11)].index)
suba = suba.drop(suba[(suba.survey_selected == 0) | (suba.grade == 11) | (suba.grade < 9)].index)
sample_survey = sample.drop(sample[(sample.survey_selected == 0) | (sample.grade == 11) | (sample.grade < 9)].index)
create_table34(sancristobal, suba, sample_survey, 'at_msamean')
```

      Basic-Savings
      Basic-Savings with demographics
      Basic-Savings with demographics and school fixed effects
      Tertiary
      Tertiary with demographics
      Tertiary with demographics and school fixed effects
      Both

      Basic treatment
      0.033
      0.032
      0.032

      0.025

      Basic treatment SE
      0.007
      0.008
      0.007

      0.01

      Savings treatment
      0.029
      0.027
      0.027

      0.028

      Savings treatment SE
      0.008
      0.008
      0.007

      0.012

      Tertiary treatment

      0.052
      0.054
      0.056
      0.055

      Tertiary treatment SE

      0.018
      0.016
      0.02
      0.02

      H0: Basic-Savings F-Stat
      0.312
      0.404
      0.481

      0.053

      p-value
      0.581
      0.53
      0.494

      0.819

      H0: Tertiary-Basic F-Stat

      1.863

      p-value

      0.18

      Observations
      5799
      5799
      5799
      930
      930
      930
      2937

      R squared
      0.003
      0.037
      0.089
      0.008
      0.058
      0.269
      0.134

The table shows:
- Basic treatment increases attendence by 3.3 percentage points (significant at the one percent level)
- Savings treatent increases attendence by 2.9 percentage points (significant at the one percent level)
- Tertiary treatment increases attendence by 5.2 percentage points (significant at the one percent level)
- no evidence that the treatments have different effects

My results from the regressions and difference tests are the same as those Barrera-Osorio et al. (2011) estimate in their paper. One can conclude, that although the savings and tertiary treatment impose more binding short-term liquidity constraints on families than the basic treatment, there is no evidence of this hurting monthly attendence.

### 4.3.2 Re-enrollment
Second, the authors analyse the effects of the conditional cash transfers on re-enrollment. Table 4 is designed as table 3, running regressions on the observed re-enrollment rate.

#### Table 4 - Effects on administrative enrollment in following year

```python
sancristobal = sample.drop(sample[(sample.suba == 1) | (sample.grade == 11)].index)
suba = sample.drop(sample[(sample.suba == 0) | (sample.grade == 11) | (sample.grade < 9)].index)
sancristobal = sancristobal[sancristobal['m_enrolled'].notna()]
suba = suba[suba['m_enrolled'].notna()]
sample_grade = sample.drop(sample[(sample.grade == 11) | (sample.grade < 9)].index)
sample_grade = sample_grade[sample_grade['m_enrolled'].notna()]
create_table34(sancristobal, suba, sample_grade, 'm_enrolled')
```

      Basic-Savings
      Basic-Savings with demographics
      Basic-Savings with demographics and school fixed effects
      Tertiary
      Tertiary with demographics
      Tertiary with demographics and school fixed effects
      Both

      Basic treatment
      0.017
      0.016
      0.011

      0

      Basic treatment SE
      0.009
      0.008
      0.01

      0.016

      Savings treatment
      0.045
      0.046
      0.04

      0.03

      Savings treatment SE
      0.016
      0.015
      0.011

      0.017

      Tertiary treatment

      0.042
      0.039
      0.037
      0.042

      Tertiary treatment SE

      0.022
      0.021
      0.02
      0.019

      H0: Basic-Savings F-Stat
      3.99
      3.941
      5.519

      2.271

      p-value
      0.048
      0.049
      0.02

      0.133

      H0: Tertiary-Basic F-Stat

      2.228

      p-value

      0.137

      Observations
      8980
      8980
      8980
      1735
      1735
      1735
      4775

      R squared
      0.002
      0.047
      0.172
      0.002
      0.079
      0.259
      0.208

The table shows:
- Basic treatment increases re-enrollment by 1.7 percentage points (significant at the 10 percent level)
- Savings treatent increases re-enrollment by 4.5 percentage points (significant at the one percent level)
- Tertiary treatment increases re-enrollment by 3.6 percentage points (significant at the 10 percent level)
- difference in magnitude of the basic and savings treatment effects is statistically significant at the 5 percent level
- no evidence that the tertiary and the basic treatment effects are different

Again, my results from the regressions are the same as those Barrera-Osorio et al. (2011) estimate in their paper. One can conclude, that the savings treatment, which consists of a bi-monthly transfer combined with a lump-sum payment at the time students are supposed to re-enroll in school, is more effective in increasing re-enrollment than the basic treatment.

### 4.3.3 Survey-Based Outcomes - Graduation and Tertiary Enrollment
Barrera-Osorio et al. (2011) use data from a follow-up survey which was conducted after the treatments were implemented to analyse the effects of each treatment on self-reported graduation and tertiary enrollment for students who were in grade 11. The sample is restricted to those students, for which the follow-up survey data is provided. In order to replicate the results from Barrera-Osorio et al. (2011), I run regressions of the binary variable for graduation and of the binary variable for tertiary enrollment on the treatment variables with demographic controls and school fixed effects. The first three columns of table 3 show the effects on graduation in the sample of Sancristobal, the sample in Suba and both together. Columns 4 to 6 show the effects on tertiary enrollment, again, in the sample of Sancristobal, the sample in Suba and both together. The estimated treatment effects and their standard errors ("SE") are provided in rows 1-6 and the test statistics from comparisons of the relative treatment effects and their p-values are in rows 7-10.

#### Table 5 - Effects graduation and tertiary enrollment for students in grade 11

```python
create_table5(sample)
```

      Graduation Basic-Savings
      Graduation Tertiary
      Graduation Both
      Tertiary enrollment Basic-Savings
      Tertiary enrollment Tertiary
      Tertiary enrollment Both

      Basic treatment
      0.039

      0.036
      0.043

      0.048

      Basic treatment SE
      0.042

      0.042
      0.036

      0.033

      Savings treatment
      0.04

      0.039
      0.094

      0.094

      Savings treatment SE
      0.033

      0.03
      0.034

      0.033

      Tertiary treatment

      0.047
      0.044

      0.489
      0.487

      Tertiary treatment SE

      0.037
      0.031

      0.04
      0.041

      H0: Basic-Savings F-Stat
      0

      0.006
      1.769

      1.542

      p-value
      0.991

      0.94
      0.199

      0.223

      H0: Tertiary-Basic F-Stat

      0.022

      75.558

      p-value

      0.882

      0

      Observations
      529
      297
      826
      513
      290
      803

      R squared
      0.103
      0.15
      0.098
      0.085
      0.373
      0.208

The table shows:
- None of the estimated effects on graduation are statistically significant
- Savings treatment increases tertiary enrollment by 9.4 percentage points (statistically significant at the 5 percent level)
- Tertiary treatment increases tertiary enrollment by 48.9 percentage points (statistically significant at the 1 percent level)
- The effect for the tertiary treatment is statistically significantly different from the effect of the basic treatment (p-value < 0.0001)

These estimates are of the same size as those provided by Barrera-Osorio et al. (2011). They mention that the tertiary enrollment findings might not be credible due to the extremely large estimate for the tertiary treatment group. One explanation for this result could be that students in the tertiary treatment group are lying in the follow-up survey about being enrolled in tertiary institutions. However, self-reported graduation rates seem to match the estimates based on the administrative enrollment data and verified attendence data.

### 4.3.4 Siblings Effects
Barrera-Osorio et al. (2011) analyse the effect of the treatments on school attendence and re-enrollment for siblings. They use the intra-familiy variation in treatment assignment to provide a reduced form test of whether receiving the transfer changes the allocation of opportunities within the household. In order to extract causal effects they restrict their sample to the subset of siblings that were also registered, since otherwise systematic differences between those families who registered one and those who registered two children might bias the results. They focus on families who registered two children and only choose those for which administrative enrollment data is available. They pool the treatments due to the small sample size. Columns 1 and 2 of Table 6 contain comparisons of untreated children with and without treated siblings. Columns 3 and 4 show comparisons only for girls and columns 5 and 6 for boys.

#### Table 6 - Effects of treatment on siblings using monitored and administrative participation, households with two registered children

```python
create_table6(data)
```

      Attendence
      Enrollment
      Attendence Female
      Enrollment Female
      Attendence Male
      Enrollment Male

      Sibling is treated?
      -0.03
      -0.071
      -0.053
      -0.114
      -0.029
      -0.04

      Sibling is treated? SE
      0.015
      0.026
      0.021
      0.053
      0.032
      0.04

      Observations
      690
      668
      352
      340
      338
      328

      R squared
      0.278
      0.137
      0.332
      0.23
      0.383
      0.234

The table shows:
- Attendence of untreated children with treated siblings is 3 percentage points lower compared to untreated children whose siblings are also untreated (statistically significant at the 10 percent level)
- Enrollment of untreated children with treated siblings is 7.1 percentage points lower compared to untreated children whose siblings are also untreated (statistically significant at the 1 percent level)
- The effect is qualitatively similar for both genders
- The effect is stronger for girls, statistically significant at the 5 percent level for attendence and at the 10 percent level for enrollment

The results in columns 1, 2, 3 and 5 are the same as those provided by Barrera-Osorio et al. (2011). The estimates of the effect on enrollment of female and male  are slightly different to those provided in the paper. However, qualitatively the effects are the same. The table suggest that the additional household resources generated by the program are not used to invest in the education of the untreated children. Instead, families with a treated child seem to take educational input away from the untreated child. The authors conclude, that eligibility rules that cut across children may increase inequality in the educational attainment within the household.

---
# 5. Critical Assessment
---

The following section discusses the quality of the strategy and estimations provided by Barrera-Osorio et al. (2011).
First, one strength of their analysis is, that they use administrative data for attendence and enrollment. This implies a high quality of data, since it is not systematically affected by individuals lying or misreporting as survey data might be.
Second, they check whether treatments are assigned randomly within the localities. Comparing characteristics between control and treatment groups they conclude that they are balanced. Furthermore, the fact that regression estimates do not differ strongly when demographics and school fixed effects are included, implies, that randomization was successful.
Third, in order to rule out spillover effects of the treatment between children through peer networks, Barrera-Osorio et al. (2011) check that children in the treatment and the control group have similar networks. Therefore, any indirect treatment effect would be equally distributed across the groups.
Fourth, the authors make sure that there is no self selection into treatment. Only children from families, who lived in the localities prior to 2004 were eligible to register for the program. This rules out that families moved to take advantage of the treatments.
Fifth, they mention that the SISBEN data, which they use for background characteristics, may underestimate assets and income, since the surveyed families knew that they were surveyed for the purpose of scoring them on a poverty index. However, the bias due to this hawthorne effect is not correlated with the differences investigated in the paper given the timing and purpose of the survey.
Sixth, Barrera-Osorio et al. (2011) check whether their baseline and follow-up survey induce a bias due to attrition. They conclude that attrition occurs similar across treatment and control groups, which implies that the results should not be biased.

One problem regarding their analysis is, that the tertiary treatment ends up being more generous than the basic and savings treatment. This makes it difficult to compare the treatments and the effects of the tertiary treatment might be biased upwards compared to the effect of the other two treatments.
Next, they cannot rely on random treatment assignment for comparisons between the treatments in the two experiments. This might bias the result if there are systematic differences of characteristics which at the same time affect treatment assignment and the outcome variable. For example, the fact that the difference in magnitude of the basic and the savings treatment effects on re-enrollment is statistically significant in the regression, where I only consider the experiment in San Cristobal, and not statistically significant in the regression, where I include both locations, indicates this.
Another problem is, that the administrative enrollment data could not be matched to 9.3% of the students in the experiment from San Cristobal, and 8.5% could not be matched in Suba. If the children for which the data could not be matched have characteristics, which systematically affect their enrollment rate, this creates a bias.
The results for the effects on graduation and tertiary enrollment relies on survey-based data. Since individuals might lie or misreport information, the results could be biased.
Lastly, the sample size for the tertiary treatment is not very large, which reduces reliability of the results and therefore is another weakness of the paper.

---
# 6. Extensions
---

## 6.1. Check for Balanced Groups across Experiments
As mentioned above, to compare the effects of all three treatments, while considering the experiments in the two localities together, the causal graph has three back-door paths which have to be eliminated. Treatment assignment is not completely random, since it is not random whether a person lives in Suba or San Cristobal. Observable or unobservable factors may affect where a family lives and therefore also are correlated with the treatment assignment and the outcome at the same time. This can be seen, when comparing characteristics of all three treatment groups to each other and to the group of not treated individuals in both localities together. Table 7 shows these comparisons. The first column shows the average for all not treated children in the sample. The following columns present the differences between the corresponding groups. For each mean and difference standard errors are provided below, denoted by 'SE'.

#### Table 7 - Comparisons of Students across Experiments

```python
create_table7(sample)
```

      Control average
      Basic-Control
      Savings-Control
      Tertiary-Conrol
      Basic-Savings
      Basic-Tertiary
      Savings-Tertiary

      Possessions
      1.91
      0.06
      0.02
      -0.01
      0.03
      0.07
      0.04

      Possessions SE
      1.08
      0.02
      0.02
      0.04
      0.02
      0.05
      0.05

      Utilities
      4.70
      -0.07
      0.01
      0.19
      -0.08
      -0.25
      -0.18

      Utilities SE
      1.39
      0.03
      0.03
      0.06
      0.03
      0.08
      0.07

      Durable Goods
      1.44
      -0.09
      -0.06
      0.21
      -0.03
      -0.30
      -0.27

      Durable Goods SE
      0.89
      0.02
      0.02
      0.03
      0.02
      0.04
      0.04

      Physical Infrastructure
      11.78
      -0.18
      -0.09
      0.31
      -0.09
      -0.48
      -0.40

      Physical Infrastructure SE
      1.70
      0.04
      0.03
      0.06
      0.04
      0.07
      0.06

      Age
      14.71
      -0.24
      -0.39
      0.90
      0.16
      -1.13
      -1.29

      Age SE
      5.08
      0.13
      0.14
      0.20
      0.17
      0.25
      0.21

      Gender
      0.48
      0.01
      0.01
      -0.04
      0.01
      0.05
      0.04

      Gender SE
      0.50
      0.01
      0.01
      0.02
      0.01
      0.02
      0.02

      Years of Education
      6.08
      -0.49
      -0.47
      1.30
      -0.02
      -1.79
      -1.77

      Years of Education SE
      1.91
      0.10
      0.10
      0.12
      0.04
      0.13
      0.13

      Single Head
      0.29
      0.02
      0.02
      -0.01
      0.01
      0.04
      0.03

      Single Head SE
      0.45
      0.01
      0.01
      0.01
      0.01
      0.02
      0.02

      Age of Head
      45.99
      -0.15
      0.04
      0.43
      -0.19
      -0.58
      -0.39

      Age of Head SE
      9.87
      0.17
      0.21
      0.27
      0.21
      0.31
      0.30

      Years of ed., head
      5.73
      -0.18
      -0.25
      0.08
      0.07
      -0.26
      -0.33

      Years of ed., head SE
      2.94
      0.08
      0.07
      0.10
      0.07
      0.12
      0.11

      People in Household
      5.35
      0.02
      0.04
      -0.20
      -0.02
      0.22
      0.24

      People in Household SE
      1.95
      0.05
      0.05
      0.07
      0.04
      0.08
      0.07

      Member under 18
      2.50
      0.09
      0.08
      -0.14
      0.01
      0.24
      0.22

      Member under 18 SE
      1.32
      0.03
      0.03
      0.05
      0.03
      0.06
      0.05

      Estrato
      1.49
      -0.06
      -0.03
      0.13
      -0.03
      -0.19
      -0.16

      Estrato SE
      0.82
      0.02
      0.02
      0.04
      0.02
      0.05
      0.05

      SISBEN score
      12.19
      -0.55
      -0.45
      1.28
      -0.10
      -1.83
      -1.73

      SISBEN score SE
      4.62
      0.12
      0.13
      0.23
      0.10
      0.26
      0.25

      Household income (1,000 pesos)
      375.73
      -13.76
      -9.39
      29.47
      -4.37
      -43.23
      -38.86

      Household income (1,000 pesos) SE
      240.07
      5.85
      6.10
      7.15
      6.64
      8.41
      8.71

The table provides 90 differences. 45 are statistically significant at the 1 percent level, 57 at the 5 percent level and 59 at the 10 percent level. Between the basic and the savings treatment group only 2 differences are statistically significant (one at the 1 percent level and two at the 5 percent level), which we already have seen in table 2. Out of the remaining 75 differences, 57 are statistically significant at the 10 percent level, 55 at the 5 percent level and 44 at the 1 percent level. One can conclude that the treatment and control groups in both localities together are not similar regarding these characteristics. This suggests, that simple estimates of regressing outcome variables on the treatment might be biased and the back-door paths have to be eliminated.

## 6.2. Elimination of Back-door Paths Controlling only for Locality
In order to eliminate the back-door paths Barrera-Osorio et al. (2011) control for the locality of the households and a large set of observable demographic characteristics. However, all three back-door paths should be eliminated by simply controling for the locality in which the family lives, since within the localities treatment assignment is random. In the following, I estimate the effects of all three treatments together on attendence and enrollment rates only controlling for the locality. Table 7 shows the results of this regression. I compare my results with those presented in the last column of table 3, respectively table 4.

#### Table 8 - Effects on Attendence and Re-enrollment only Controlling for the Locality

```python
create_table8(sample)
```

      Attendence
      Enrollment

      Basic treatment
      0.023
      0.004

      Basic treatment SE
      0.011
      0.014

      Savings treatment
      0.028
      0.024

      Savings treatment SE
      0.011
      0.019

      Tertiary treatment
      0.052
      0.042

      Tertiary treatment SE
      0.017
      0.022

      H0: Basic-Savings F-Stat
      0.089
      0.932

      p-value
      0.767
      0.335

      H0: Tertiary-Basic F-Stat
      1.965
      1.978

      p-value
      0.168
      0.161

      Observations
      2937
      4775

      R squared
      0.004
      0.002

These estimates are very similar to those presented in the last column of table 3, respectively table 4. For example, the tertiary treatment here is estimated to increase the attendence rate by 5.2 percentage points, table 3 shows an effect of 5.5 percentage points. The largest difference occurs for the effect of the savings treatment on re-enrollment, which is an increase of 2.5 percentage points compared to 3 when we control for demographics and include school fixed effects. However, the results are statistically significant at the same significance levels as in table 3 and 4. Concerning the difference tests of the treatment effects the results differ more strongly, while in none of the cases they are statistically significant. One can conclude that including demographic characteristics and school fixed effects in addition to controlling for the locality in which the family lives does not change the results of the regression with all three treatments. This can be explained by the fact that the back-door paths are blocked by adding the locality to the regression.

---
# 7. Conclusion
---

The results in this notebook support the findings reported in the paper by Barrera-Osorio et al. (2011). I reproduced the results precisely for almost all tables except for the third and sixth column of table 6 for which I could only produce similar results. In addition to the replication of the main results from Barrera-Osorio et al. (2011), I critically discuss the quality of their strategy and estimations and evaluate the robustness of their results. Comparing mean characteristics between the research groups across the two experiments, I find that the groups are different. This indicates that their identification assumption, which is that treatments are assigned randomly, is violated and estimates from the regressions, where all three treatments are included, might be biased. Barrera-Osorio et al. (2011) include demographic controls and school fixed effects in the regression in order to rule out potential biases. However, evaluating the causal graph I provide in section 2 one can conclude that including the locality into the regression equation eliminates all three back-door paths. Running this regression I find similar results as Barrera-Osorio et al. (2011). However, the fact that estimates of the effects of the basic and savings treatment are very different when the tertiary treatment is included in the regression or not, indicates a bias. Further analysis of the difference of the basic and savings treatment compared to the tertiary treatment might be helpful to provide more precise policy implications.

Additionally, the study focuses only on the effects of the conditional cash transfers on the children included in the experiment in the two localities. It might be the case that the children who registered and are eligible for the experiment yield higher outcomes than those who did not register or are not eligible. This raises the question whether external validity of the results is given. Further research may address this issue.

Overall, the findings from Barrera-Osorio et al. (2011) offer credible results and policy implications regarding the design of education-based conditional cash transfers.

---
# 8. References
---

* **Barrera-Osorio, Felipe, Marianne Bertrand, Leigh L. Linden, and Francisco Perez-Calle (2011)**. "Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia." *American Economic Journal: Applied Economics*, 3 (2): 167-95.


# O que é machine learning?

Machine learning, ou aprendizado de máquinas, é o desenvolvimento de algoritmos genéricos que conseguem identificar e aprender coisas sobre um determinado monte de dados sem ser explicitamente programado para tal.

Ao invés de escrever decisões lógicas, você alimenta seu algorítmo com dados e ele decide como interpretar aquilo e tomar decisões em cima dos dados.

Por exemplo, você pode alimentar um algoritmo com dados sobre números escritos à mão e pedir que ele identifique novos números. Você pode, também, alimentar esse mesmo algoritmo com dados sobre emails que são spam e não spam e pedir que ele identifique novos emails.

# Tipos de machine learning
Machine learning é um campo vasto. As 3 sub-áreas mais comuns são, porém, **aprendizado supervisionado**, **não supervisionado** e **de reforço**.

Nessa primeira aula vamos focar no primeiro tipo. Na próxima, falaremos de técnicas de otimização e do segundo. O terceiro tipo está fora do escopo dessas aulas.

Esses 3 tipo são definidos mais ou menos assim:

## Aprendizado supervisionado
Para ilustrar o conceito, vamos pensar em um exemplo.

Digamos que estamos nos anos 90 e você trabalha em uma locadora de filmes. Sua locadora é diferenciada porque vocês costumam recomendar filmes para os seus clientes. Contudo, você percebe que tem um grande problema quando vocês contratam alguém novo. Esse novo empregado não conhece todos os filmes da sua locadora e, por isso, não consegue recomendar muitos filmes para os clientes. Você descobre que vocês têm uma tabela com várias informações como gênero, ano, duração e vários outras características dos filmes. Você também sabe que têm uma tabela com os últimos filmes que todos os seus clientes alugaram.

Você decide então escrever um programa que tenta recomendar filmes para os seus clientes com base no que eles costumam assistir.

Vocês então recebem um filme novo, Rei leão, e devem decidir quem seria o cliente que mais gostaria daquele filme:

Isso é aprendizado supervisionado. Você entendeu o padrão e foi capaz de encontrar qual a pessoa que poderia gostar mais desse filme. Nesse caso, talvez o ano ou nome não façam muita diferença, mas em casos mais complexos, como a previsão de preços de casas ou de músicas, seja necessário mais cálculos.

O seu algoritmo tem que ser capaz de ir além desses dados e generalizar as características para decidir.

Em aprendizado supervisionado você deixa o computador descobrir essa lógica para você e, com ela, ele é capaz de fazer previsões e trabalhar com dados novos.

## Aprendizado não supervisionado

Em aprendizado supervisionado nós não temos o que cada filme **é**. Temos apenas dados e temos que extrair informações deles. Em nosso exemplo anterior, nós podemos aplicar algoritmos de aprendizado não supervisionado para identificar mercados nos filmes. Podemos usar as idades dos clientes para descobrir que gêneros de filmes são mais populares com que faixas etárias, descobrir se homens gostam mais de filmes de ação que mulheres e até fazer análises ao longo do tempo pra saber exatamente onde e como devemos promover um filme para obter o máximo possível de clientes interessados.

## Aprendizado de reforço

Aprendizado de reforço é muito menos sobre previsões e muito mais sobre resultados. Usamos aprendizado de reforço quando queremos criar algoritmos novos que atingem o melhor resultado possível.

Digamos que queremos manter um cliente sendo fiel durante muito tempo. Para isso nós mandamos emails de recomendação e enviamos presentes de vez em quando para manter ele interessado em nós. Porém, não queremos que ele sinta que estamos forçando a barra. Como encontrar esse equilibrio?

Primeiro criamos um algoritmo que decide quando enviar um email ou um presente. A partir daí, começamos a mandar emails e presentes e avaliar a resposta dos clientes. Sempre que um cliente sai da nossa lista de email nós dizemos pro algoritmo que ele fez besteira. Sempre que ele recomenda o nosso sistema para alguém, dizemos que ele fez bem. Ao longo do tempo ele começa a aprender quais padrões de presentes e emails rendem mais "elogios" e menos "críticas". Nós usamos essas reflexões para **reforçar comportamentos** que são bons ou ruins e incentivamos os algoritmos a se melhorarem e se adaptarem. Isso é diferente de um algoritmo supervisionado pois o feedback pode demorar meses para vir e o algoritmo tem que ser capaz de refletir sobre suas ações e experimentar coisas novas para **otimizar** seus resultados.

Agora que já vimos um pouco sobre eles, vamos mais a fundo em algortimos supervisionados.

# Introdução ao aprendizado supervisionado

Ok, já vimos a teoria, agora vamos escrever um programa que encontre filmes para a gente!

Se você não soubesse nada sobre aprendizado de máquinas, talvez escrevesse um programa mais ou menos assim:

```Python
def recomendar_filme(filme, ano, gênero, lista_de_filmes):

    # Encontra os filmes que tem o mesmo genero do filme novo
    filmes_com_mesmo_genero = filter(lambda x: x['gênero'] == gênero, lista_de_filmes)

    # Dentro desses, acha os que tem o um ano próximo
    filmes_ano_proximo = filter(lambda x: ano + 5 >= x['ano'] >= ano - 5, filmes_com_mesmo_genero)

    # Desses encontrados, pegar o cliente que alugou um desses filmes e recomendar pra ele
    cliente = filmes_ano_proximo[0]['cliente']
    return cliente
```

Se você ficar mexendo bastante nisso por horas talvez consiga alguma coisa que funcione, mas seu programa nunca será perfeito e sempre terá que alterar alguma coisa pra filmes novos.

O ideal seria simplesmente pedir que o computador fizesse tudo pra você

```Python
def recomendar_filme(filme, ano, gênero, lista_de_filmes):
    cliente = <computador, por favor, faz esses cálculos por mim>

    return cliente
```

O que ele vai fazer na verdade é transformar esses valores que nós criamos, gêneros, anos e tal em **vetores em um hiperespaço** que pode ser representado por números.

No fim o que o computador vai gerar é algo como:

```Python
def recomendar_filme(filme, ano, gênero, lista_de_filmes):
    vetor_gênero = *vetorizador_gênero(gênero) * [63.61124, .053823, 23.23521, 294.43213, .000123, 1.]

    vetor_ano = *vetorizador_ano(ano) * [54.2421543, .00342, 5143532.3415]

    vetor_cliente = 873.5323 + vetor_gênero + vetor_ano

    cliente = distribuição_prob(lista_de_filmes, vetor_cliente)

    return cliente
```

Todos esses valores nas listas são o que chamamos de **pesos**. Eles são gerados **aleatoriamente** pelo computador e testados milhares de vezes até que o valor que obtem o **menor erro** seja encontrado. Como podem ver, são vetores multi dimensionais que podem ter, na verdade, centenas de dimensões. Cada dimensão representa um ano ou gênero de filme e os pesos representam a interação entre eles.

No fim nós podemos usar uma distribuição probabilística para encontrar o cliente que teria a maior probabilidade de gostar do filme que estamos analisando.

É imporante ressaltar que os valores iniciais desses pesos são aleatórios. Para encontrar os valores finais, existem diversas técnicas de otimização que, eventualmente, tendem a reduzir o erro total do sistema a um mínimo.

Para calcular esse erro, podemos usar uma função de **custo** que define a **distância** entre o valor previsto e o valor definido como correto (aí está, novamente, por que estamos usando aprendizado supervisionado).

A **função de custo** pode tomar diversas formas, mas a mais comum e talvez a mais simples é a **soma do erro quadrado**, onde por erro nós definimos a distância probabilística até o valor correto.

\begin{align}
Custo = \frac{\sum_{i=1}^n \left( meu\_chute(i) - valor\_real(i) \right)^2}{2n}
\end{align}

Vamos agora reescrever essa função usando alguns jargões de machine learning que você não tem que se preocupar agora

\begin{align}
J(\theta) = \frac{1}{2m}\sum_{i=0}^m \left( h_\theta(x^{(i)}) - y^{(i)} \right)^2
\end{align}

Onde $\theta$ representa os **pesos** e $J(\theta)$ o **custo para os pesos atuais**

Se nós fizermos um gráfico dos pesos, teremos algo assim:

Onde o eixo vertical representa o valor do custo dados os pesos nos eixos horizontais.

Nesse gráfico, o que queremos fazer é reduzir ao mínimo o nosso custo (erro). Para isso usamos técnicas de otimização, onde fazemos algo assim:

Para isso nós precisamos ajustar nossos pesos de forma que estejamos 'descendo a ladeira' em direção ao mínimo.

Ter o erro mínimo significa estar o mais próximo possível, teoreticamente, da previsão perfeita.

Se você lembra algo de cálculo, deve saber que se nós pegarmos a derivada de uma função em cada variável e somarmos, obtemos uma outra função cujo valor define a direção de crescimento máximo. ([gradiente](https://en.wikipedia.org/wiki/Gradient))

\begin{align}
\nabla f(x, y) = \frac{\partial f}{\partial x} i + \frac{\partial f}{\partial y} j
\end{align}

Esse é um método de otimização simples, mas eficiente, que nos permite rapidamente encontrar os valores de peso mínimo de nossa fução de custo e, logo, o **estado mais otimizado do nosso modelo**.

Isso é apenas uma introdução, então não vamos ir mais fundo que isso, mas sinta-se livre para [aprender mais caso queira](http://hbfs.wordpress.com/2012/04/24/introduction-to-gradient-descent/).

Não se preocupe, porém, em ter tudo isso na cabeça. Quando você usar uma biblioteca de machine learning tudo isso será **feito pela biblioteca, não por você**. Porém é importante ter uma noção do que está acontecendo.

## O que acabamos de aprender?

Isso tudo que nós vimos é o funcionamento de uma **regressão linear multivariacional**. Vamos ver isso novamente mais pra frente do curso, incluindo alguns conceitos como **overfitting**, **tendência, ou bias**, **validação** e outras coisas.

Saiba, porém, que nem sempre será possível usar esses algoritmos para prever essas coisas. Isso acontece porque muitas vezes as relações entre essa variáveis (ano/gênero) não são lineares. Para esses outros casos nós teremos que usar outros **modelos** e até outras técnicas. Veremos tudo isso mais pra frente.

## Aprendizado de máquinas é mágica?
Assim que nós começarmos a aplicar machine learning para solucionar alguns problemas, começaremos a perceber que é muito fácil solucionar problemas que parecem ser muito difíceis de se resolver de outra forma.

É importante notar, porém, que aprendizado de máquinas só funciona para resolver problemas que **podem ser resolvidos**.

Por exemplo, não é possível determinar o tipo de filme que alguém gosta baseado na raça de cachorro que essa pessoa tem. Não importa o quanto você altere seu algoritmo e o quão bons os seus resultados fiquem para o dataset de treino, será impossível ter uma boa validação fora disso. Seu modelo sofrerá de overfitting e você precisará de outras características para resolver esse problema.

Com isso nós concluímos nossa introdução ao ML. Agora vamos começar a programar algumas coisas e fazer ML de verdade!

# Programando Machine Learning

Para programar e usar machine learning, vamos usar uma biblioteca de ML muito famosa chamada SKLearn. Ela contém centenas de modelos e algoritmos de classificação, regressão, clusterização e muito mais.

Você pode ler sobre o projeto no site deles:
[http://scikit-learn.org/stable/](http://scikit-learn.org/stable/)

## Algoritmos e superficies de decisão

Geralmente nós conseguimos plotar nossos dados em gráficos, onde cada eixo representa uma característica dos dados e cada ponto um dado específico. Geralmente as cores representam o tipo, ou **label** de cada dado.

Primeiro vamos definir uma função que gera gráficos para nós. Isso vai ser bem útil no futuro.

```python
import numpy as np
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
```

```python
def generate_data(blue, red, test_size=0.2, func=lambda x: x):

    blue_x = np.vectorize(func)(np.random.randint(*blue[0]))
    blue_y = np.vectorize(func)(np.random.randint(*blue[1]))
    red_x = np.vectorize(func)(np.random.randint(*red[0]))
    red_y = np.vectorize(func)(np.random.randint(*red[1]))

    blue_vals = np.dstack((blue_x, blue_y, ["blue" for _ in blue_x]))[0]
    red_vals = np.dstack((red_x, red_y, ["red" for _ in red_x]))[0]

    X = np.append(blue_vals, red_vals, axis=0)

    Y = [1 for _ in range(len(blue_vals))]
    Y.extend([0 for _ in range(len(red_vals))])

    return train_test_split(X, Y, test_size=test_size)
```

```python
X_train, X_test, y_train, y_test = generate_data([[0, 40, 100], [0, 40, 100]], [[50, 100, 100], [50, 100, 100]])

plt.scatter(X_train[:, 0], X_train[:, 1], c=X_train[:, 2])
plt.show()
```

Como pode ver, nós temos dados separados no que são claramente objetos diferentes. É muito fácil criar uma barreira que separa os dois clusters.

Depois de separar, fica bem fácil assumir que tudo que estiver de um lado dela será azul e do outro vermelho. Essa barreira se chama uma **superfície de decisão** e é um dos conceitos principais quando lidamos com algoritmos de classificação.

Agora vamos ver um algoritmo de classificação e experimentar com ele.

# Naive Bayes
Para o nosso primeiro modelo de aprendizado supervisionado, aprenderemos um modelo de classificação probabilístico chamado Naive Bayes, que vem de um homem religioso chamado Bayes que criou o teorema de Bayes para provar a existência de Deus.

O algoritmo, como o nome sugere, usa o [Teorema de Bayes](https://en.wikipedia.org/wiki/Bayes%27_theorem) para descobrir a probabilidade de um novo elemento se encaixar numa categoria específica.

Usando o SKLearn, nosso código sempre terá um padrão como veremos abaixo.

```python
# Em X_train nós temos os nossos dados brutos, suas coordenadas.
# Em y_train nós temos a categoria (1 para azul e 0 para vermelho)
# Não se preocupe com os _test por enquanto
X_train, X_test, y_train, y_test = generate_data([[0, 40, 100], [0, 40, 100]], [[50, 100, 100], [50, 100, 100]])

# Desses dados, nós podemos importar o classificador de Nayve Bayes
from sklearn.naive_bayes import GaussianNB

# Criar um objeto classficador
clf = GaussianNB()

# E fitar, ou seja, treinar nosso clssificador usando os dados de X_train e y_train
clf.fit(X_train[:, :2].astype(np.int), y_train)
```

    GaussianNB(priors=None)

```python
# Agora nós criamos uma outra função que vai criar um gráfico e desenhar a nossa superfície de decisão
from matplotlib.colors import ListedColormap
def plot_decision_boundary(clf, X):
    h = .02
    cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
    cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])
    # Plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, x_max]x[y_min, y_max].
    x_min, x_max = X[:, 0].astype(np.float).min() - 1, X[:, 0].astype(np.float).max() + 1
    y_min, y_max = X[:, 1].astype(np.float).min() - 1, X[:, 1].astype(np.float).max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    print("predicting")
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    print("predicted")
    print("length", len(Z))

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure()
    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)
    print("colormesh")

    # Plot also the training points
    plt.scatter(X[:, 0].astype(np.float), X[:, 1].astype(np.float), c=X[:, 2])
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    print("showing")
    plt.show()
```

```python
# Finalmente, passamos os argumentos para nossa função
%time plot_decision_boundary(clf, X_train)
```

Pode demorar um tempo, mas no final você deverá ser capaz de ver bem onde exatamente a decisão é feita para separar ambas as cores. Qualquer ponto na zona azul será previsto por nosso algoritmo como azul e o mesmo vale para a zona vermelha.

Nós podemos, por exemplo, testar nosso algoritmo:

```python
# Criamos um novo ponto que queremos testar
novo_ponto = (0, 75)

# Geramos uma previsão para ele. Se sair 1, será azul e 0 será vermelho
print(clf.predict_proba([novo_ponto]))
```

    [[ 0.12264511  0.87735489]]

Excelente! Temos nosso primeiro classificador. Agora algo importante que devemos fazer é testar esse classificador.

## Testando

Para isso, usaremos uma ferramenta muito útil chamada **matriz de confusão**. Essa matriz nos diz exatamente quantos pontos foram corretamente previstos, quantos foram falsos positivos e quantos foram falsos negativos. Para isso, criamos uma função que gera essa matriz para nós:

```python
from sklearn.metrics import classification_report, confusion_matrix
def plot_confusion_matrix(y_true, y_predicted):
    plt.matshow(confusion_matrix(y_true, y_predicted), cmap=plt.cm.binary, interpolation='nearest')
    plt.title('Matriz de previsão')
    plt.colorbar()
    plt.ylabel('Categoria esperada')
    plt.xlabel('Categoria prevista')
    plt.show()
```

```python
# Vamos ver como fica para os dados que usamos para treinar
plot_confusion_matrix(y_train, clf.predict(X_train[:, :2].astype(np.int)))
```

Esse formato que estamos vendo é o ideal. Ele representa que todos os valores de azul foram previstos corretamente e os de vermelho também, sem falsos positivos ou negativos.

Nós também podemos gerar algo chamado **relatório de classificação**, que faz exatamente o que o nome sugere:

```python
print(classification_report(y_train, clf.predict(X_train[:, :2].astype(np.int))))
```

                 precision    recall  f1-score   support

              0       1.00      1.00      1.00        76
              1       1.00      1.00      1.00        84

    avg / total       1.00      1.00      1.00       160

Nesse relatório, temos 4 valores:

- **Precisão**: Relação entre os verdadeiros positivos e o total. Diz o quão preciso foi a classificação dado todos os dados
- **Recall**: Relação entre verdadeiros positivos e positivos. Diz o quão certeiro foi a classificação para encontrar os positivos
- **F1-Score**: Média harmônica entre precisão e recall. Boa forma de avaliar de forma completa a classificação
- **Support**: É o número de ocorrências de cada classe em verdadeiro-positivo

Esses dados são bacanas, mas não são muito interessantes do ponto de vista de previsões futuras. O que nós queremos acima disso é checar se nosso algoritmo é capaz de ser generalizado para dados novos.

Para isso vamos usar aquelas variáveis que criamos antes: `X_test` e `y_test` que não foram fornecidas ao nosso classificador para treinamento:

```python
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.int)))
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.int))))
```

Como pode ver, mesmo assim o nosso algoritmo foi fantástico na classificação. Isso é de se esperar, dado que foram dados bem simples.

Vamos testar algo mais complexo agora

```python
X_train, X_test, y_train, y_test = generate_data([[0, 100, 200], [0, 60, 200]], [[00, 100, 250], [20, 100, 250]])
plt.scatter(X_train[:, 0], X_train[:, 1], c=X_train[:, 2])
plt.show()
```

Como pode ver, esse já é mais difícil pois temos muita intersecção entre os pontos. Como noss algoritmo vai se sair agora?

```python
clf = GaussianNB()
clf.fit(X_train[:, :2].astype(np.int), y_train)
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.int))))
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.int)))
%time plot_decision_boundary(clf, X_train)
```

70% de precisão não é tão bom assim. Dá pra ver por que foi tão ruim.

Nossas features não estão sendo suficientes para separar corretamente os dados. Para separar, nós precisaremos adiconar features novas que, em outras dimensões, separem os nossos dados. Uma visualização desse resultado seria:

```python
from mpl_toolkits.mplot3d import Axes3D
import random

fig = plt.figure()
ax = Axes3D(fig)

sequence_containing_x_vals = list(range(0, 50))
sequence_containing_y_vals = list(range(0, 50))
sequence_containing_z_vals = list(range(50, 100))

random.shuffle(sequence_containing_x_vals)
random.shuffle(sequence_containing_y_vals)
random.shuffle(sequence_containing_z_vals)

ax.scatter(sequence_containing_x_vals, sequence_containing_y_vals, sequence_containing_z_vals)
sequence_containing_x_vals = list(range(50, 100))
sequence_containing_y_vals = list(range(50, 100))
sequence_containing_z_vals = list(range(50, 100))

random.shuffle(sequence_containing_x_vals)
random.shuffle(sequence_containing_y_vals)
random.shuffle(sequence_containing_z_vals)

ax.scatter(sequence_containing_x_vals, sequence_containing_y_vals, sequence_containing_z_vals)

plt.show()
```

Agora vamos sair um pouco de Nayve Bayes e testar outro algoritmo de classificação. Esse agora se chama Support Vector Machines e é melhor em algumas situações em comparação com o Nayve Bayes.

# Support Vector Machines

Para treinar um SVM, fazemos da mesma forma que fazíamos com NB:

```python
from sklearn.svm import SVC
from math import sqrt
```

```python
X_train, X_test, y_train, y_test = generate_data([[0, 40, 100], [0, 40, 100]], [[50, 100, 100], [50, 100, 100]])

clf = SVC(kernel="linear")
clf.fit(X_train[:, :2].astype(np.int), y_train)
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.int))))
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.int)))
%time plot_decision_boundary(clf, X_train)
```

Como podem ver, é bem parecido com o NB, mas é mais linear.

O que acontece quando nossa barreira de classificação não é separavel linearmente?

```python
def generate_circular_data(min_radius, med_radius, max_radius, d1=100, d2=100, test_size=0.2, func=lambda x: x):

    x1 = np.random.uniform(-min_radius, min_radius, d1)
    y1 = []
    for i in x1:
        yy = np.random.uniform(0, sqrt(min_radius**2 - i**2))
        if random.choice([True, False]):
            yy = -yy
        y1.append(yy)

    x2 = np.random.uniform(-max_radius, max_radius, d2)
    y2 = []
    for i in x2:
        low = sqrt(med_radius**2 - i**2) if med_radius**2 - i**2 > 0 else 0
        yy = np.random.uniform(low, sqrt(max_radius**2 - i**2))
        if random.choice([True, False]):
            yy = -yy
        y2.append(yy)

    blue_vals = np.dstack((np.vectorize(func)(x1), np.vectorize(func)(y1), ["blue" for _ in x1]))[0]
    red_vals = np.dstack((np.vectorize(func)(x2), np.vectorize(func)(y2), ["red" for _ in x2]))[0]

    X = np.append(blue_vals, red_vals, axis=0)

    Y = [1 for _ in range(len(blue_vals))]
    Y.extend([0 for _ in range(len(red_vals))])

    return train_test_split(X, Y, test_size=test_size)
```

```python
X_train, X_test, y_train, y_test = generate_circular_data(10, 15, 20, 100, 200)

plt.scatter(X_train[:, 0], X_train[:, 1], c=X_train[:, 2])
plt.show()
```

```python
clf = SVC(kernel="linear")
clf.fit(X_train[:, :2].astype(np.float), y_train)
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.float))))
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.float)))
%time plot_decision_boundary(clf, X_train)
```

É facil perceber que essa barreira será difícil pro nosso SVM classifier. O que podemos fazer então?

Podemos criar features novas a partir das anteriores. Nesse caso, podemos criar uma feature que representa o **quadrado das variáveis**.

```python
square = lambda x: x**2

X_train, X_test, y_train, y_test = generate_circular_data(10, 15, 20, 100, 200, func=square)

plt.scatter(X_train[:, 0], X_train[:, 1], c=X_train[:, 2])
plt.show()
```

E, finalmente, podemos testar nosso classificador usando essas features novas

```python
clf = SVC(kernel="linear")
clf.fit(X_train[:, :2].astype(np.float), y_train)
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.float))))
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.float)))
%time plot_decision_boundary(clf, X_train)
```

De fato, é isso que o SVC é capaz de fazer sozinho. Ao contrário do NB, o SVC é capaz de usar **truques de kernel** para explorar variáveis novas e configurações novas de variáveis. Experimentemos rodar o código para o gráfico circular, mas com a kernel sendo a padrão:

```python
X_train, X_test, y_train, y_test = generate_circular_data(10, 15, 20, 100, 200)

plt.scatter(X_train[:, 0], X_train[:, 1], c=X_train[:, 2])
plt.show()

clf = SVC()
clf.fit(X_train[:, :2].astype(np.float), y_train)
print(classification_report(y_test, clf.predict(X_test[:, :2].astype(np.float))))
plot_confusion_matrix(y_test, clf.predict(X_test[:, :2].astype(np.float)))
%time plot_decision_boundary(clf, X_train)
```

```python

```


 **Chapter 2: [Diffraction](CH2_00-Diffraction.ipynb)**

# Bethe Theory

[Download](https://raw.githubusercontent.com/gduscher/MSE672-Introduction-to-TEM//main/Diffraction/CCH2_D01-Bloch.ipynb)

[](
    https://colab.research.google.com/github/gduscher/MSE672-Introduction-to-TEM/blob/main/Diffraction/CH2_D01-Bloch.ipynb)

part of

 **[MSE672:  Introduction to Transmission Electron Microscopy](../_MSE672_Intro_TEM.ipynb)**

by Gerd Duscher, Spring 2021

Microscopy Facilities

Joint Institute of Advanced Materials

Materials Science & Engineering

The University of Tennessee, Knoxville

Background and methods to analysis and quantification of data acquired with transmission electron microscopes

## Import numerical and plotting python packages
Import the python packages that we will use:

Beside the basic numerical (numpy) and plotting (pylab of matplotlib) libraries,

```python
import sys
if 'google.colab' in sys.modules:
    %pylab --no-import-all inline
else:
    %pylab --no-import-all notebook
```

    Populating the interactive namespace from numpy and matplotlib

## Notation

In the following I will introduce this Bethe diffraction theory in Dirac's  bra--ket notation.
\begin{eqnarray*}
|\psi\rangle &=& \mbox{ket} = \psi_1 |1\rangle + \psi_2 |2\rangle\\
&=& \left( \begin{array}{c} \psi_1 \\\psi_2  \end{array} \right)= \mbox{column vector}\\
\langle \phi| &=& \mbox{bra} = \phi^*_1\langle 1|  + \phi^*_2 \langle 2|\\
&=& \left( \phi^*_1 ,\phi^*_2 \right)= \mbox{row vector}
\end{eqnarray*}

Consequently the Dirac bracket is:
\begin{equation*}
\langle \phi|\psi\rangle = \left( \phi^*_1 ,\phi^*_2 \right) \left( \begin{array}{c} \psi_1 \\\psi_2  \end{array} \right) = \phi^*_1 \psi_1 +\phi^*_2 \psi_2
\end{equation*}

Similarly

\begin{equation*}
|\psi\rangle \langle \phi| = \mbox {a matrix}
\end{equation*}
and this will make it easier to follow the equations of many beams in diffraction, which otherwise can become very messy.

Throughout this chapter we will use the notation for the wave vector common in solid state physics and not the one found in many crystallography books ($k=1/\lambda$):
\begin{equation}\label{vector}
k = \frac{2\pi}{\lambda} = \frac{\sqrt{2meE}}{\hbar}
\end{equation}
with:
- $\lambda$: wavelength
- $m=\gamma_0$: relativistically corrected mass of the electron
- $\gamma = 1+\frac{e^2E^2}{2m_0c^2}$: relativistic correction necessary for $E$ above 2kV
- $e$: electron charge
- $E$: acceleration voltage
- $c$: speed of light in vacuum

This is the short version of quantum mechanics. The Axioms of quantum mechanics are:

1. The state of a system is described by its state vector $|\psi\rangle$.

2. An observables are expressed by hermitic operators $A$
3. The mean value of an observable is given by $\langle A \rangle$
4. The time dependence is given by the time-dependent Schrödinger
    equation: $ H|\psi,t\rangle = i\hbar \frac{\partial}{\partial t} |\psi,t\rangle $
5. If you measure $A$ the system changes to $|n\rangle$ if $a_n$ was measured

Axioms 2. and 3. give:

If a system is in the state $|\psi\rangle = \sum_n c_n |n\rangle$

with $c_n = \langle n|\psi\rangle$, where $|n\rangle$ are the Eigen states of $A$, meaning $A|n\rangle = a_n |n\rangle$. The probability to find the value $a_n$ when measuring $A$ is given by $|c_n|^2$.

## Introduction to Bethe Theory

The dynamic theory calculates the probability of a transition from an initial state $|i\rangle$ to a final state $|f\rangle$. In our case, the initial state $|i\rangle$ is the incoming beam which scatters to a final state the Bragg reflection $|f\rangle$.
What we want to know is the transition probability from the initial to the final state:
\begin{equation}
\omega_{i\rightarrow f} =  \langle i |T|f\rangle
\end{equation}
the initial state is the beam with wave vector $\vec{k}_0$ and the end state is a wave-vector of diffracted beam according to Bragg's law $\vec{q} = \vec{g}$.
The above equation now looks like this:
\begin{equation}
\omega_{0\rightarrow g} = \langle 0 |T|g\rangle
\end{equation}

For the stationary problems the states $|\psi,t\rangle$ can be expressed as:\\
\begin{equation}
|\psi_n,t\rangle = \exp(-E_nt/\hbar) |\psi_n\rangle
\end{equation}
and we get the time-independent Schröinger equation for $|\psi_n\rangle$:
\begin{equation}
H|\psi_n\rangle = E_n |\psi_n \rangle
\end{equation}

For the incoming wave, we get:

\begin{equation}
H_0 = { -\frac{h^2}{8\pi^2} \nabla^2 }
\end{equation}
which expresses the kinetic energy.

Within a crystal the Hamiltonian will change to:
\begin{equation}
H = H_0 + V
\end{equation}
We have  the Schrödinger equation for the incoming wave:
\begin{equation}  \label{IncomingWave}
(H_0 ) |\vec{k}\rangle = E_k |\vec{k}\rangle
\end{equation}

and we want to solve:
\begin{equation}
(H_0 +V) |\psi\rangle = E_k |\psi\rangle
\end{equation}

which we transform with equation \ref{IncomingWave}:

> Effectively, we changed in the integral equation.

\begin{equation*}
|\psi\rangle = |\vec{k}\rangle + \frac{1}{E_k -H_0}V|\psi\rangle
\end{equation*}

### Hamiltonian in Bethe Theory
For our diffraction experiment is often better to use a Hamiltonian that contains the wave vector:
\begin{equation}
H_0 = { -\Delta_\rho -\xi^2 }
\end{equation}

OOps, where is the wavevector $\vec{k}$?

I replaced it by $\xi$ which is the effective deviation from wavevector $\vec{k}$

We also take into account that our electrons are very fast and distort the space, reducing the problem to two dimensions ($\rho = (x,y)$.
The crystal potential V has then to be changed as well to:
\begin{equation}
V = V(\rho,z)
\end{equation}
and the full Hamiltonian is changed to:
\begin{equation}
H = \frac{1}{k_z} (H_0 + V(\rho,z)
\end{equation}

The time- independent wave equation is then:
\begin{equation}
\nabla^2 \psi  + k^2 \psi = 0
\end{equation}
with the plane wave solution:
\begin{equation}
\psi  = \exp(\pm\vec{k}\bullet\vec{r}) = 0
\end{equation}

## Schrödinger  Equation of Bethe Theory

The Bethe theory is based on the (time independent, non relativistic) Schrödinger  equation:

\begin{equation}  \label{Schrodinger}
\Big[ \underbrace{ -\frac{h^2}{8\pi^2} \nabla^2 }_{\mbox{ kinetic energy}} +\underbrace{ \mathcal{V}(\vec{r})}_{\mbox{ pot. energy}} \Big]\, |\psi(\vec{r}) \rangle = \underbrace{\mathcal{E}}_{\mbox{ total energy}} \underbrace{|\psi(\vec{r})\rangle }_{\mbox{ wave function}}
\end{equation}

What does that mean for the TEM?
>
> We have a acceleration voltage (electric field potential) $E$ of 100kV
>
> We have a charge of the electron $q$ with the value $e$.
>
> We have a total Energy $-\mathcal{E} = E\cdot q$ which is just $E$ in the units of $[eV]$.
>
> We have a crystal with the potential $V(\vec{r})$, which we declare positive inside the crystal and zero outside.
>
> We have a potential Energy $\mathcal{V}(\vec{r}) = q\cdot V(\vec{r})$.

Now, that we declared all our variables we can transform the Schrödinger equation \ref{Schrodinger} to:

$$
\nabla^2  |\psi(\vec{r})\rangle = -\frac{8\pi m e}{h^2}\,  [E+V]\, |\psi(\vec{r})\rangle
$$

The left hand part of this equation is the impulse of the electron and the right hand part consists of a total energy part, which is boring and a part which originates from the crystal (interesting!).

## Bloch Waves in Bethe Theory
Well, if the potential is periodic, then the solution (wave function) must be periodic, too.

First we make a substitution in case our wave function is complicated: we define it as a linear combination of other waves. That is a useful trick, which makes the mathematics easier as we'll see in a bit.
\begin{eqnarray} \label{Bloch}
|\psi(\vec{r})\rangle &=& \sum_j |b_j\rangle \\
|b_j\rangle  &=&  = |b(\vec{k}^{(j)}, \vec{r})\rangle
\end{eqnarray}
The $ |b(\vec{k}^{(j)}, \vec{r})\rangle $ are called Bloch waves and are only defined for specific $\vec{k}$--vectors,
because the  $|b(\vec{k}^{(j)}, \vec{r})\rangle$ are plane waves, each traveling in $k^{(j)}$ direction. For the $|\psi(\vec{r})\rangle$ we did not and could not have made any assumption like that.\\

Now, we express the fact that these Bloch waves are indeed plane waves mathematically:

\begin{equation} \label{FourierExpand}
|b^{(j)}(\vec{r})\rangle  = b(\vec{k}^{(j)}, \vec{r})=\mu(\vec{k}^{(j)}, \vec{r})\cdot e^{2\pi i \vec{k}^{(j)} \vec{r}} = \underbrace{\mu^{(j)}(\vec{r})}_{\mbox{ Bloch function}}   e^{2\pi i \vec{k}^{(j)} \vec{r}}
\end{equation}

by dividing it in a plane wave part (the exponential function) and a amplitude part (the Bloch function).
Because of the periodicity which we assume for the solution, we expand the Bloch waves in a into a Fourier series, again (the same as in equation \ref{FourierExpand} for the potential).

\begin{equation}
b^{(j)}(\vec{r}) = \sum_g C_g^{(j)} e^{2\pi i (\vec{k}^{(j)} + \vec{g}) \vec{r}}
\end{equation}
The sum in this equation goes over all excited (aha!) points in the reciprocal lattice, including the incident direction $g_1 = 0$. (  \vec{g} are defined through the Milller indices as (h/a, k/b, l/c), where the (a,b,c) are the real space lattice vectors).

Theoretically, thre are an infinite number of $\vec{g}$ vectors, but only few are allowed and only a few have a small excitation error.

So in practice there are only a few $\vec{g}$ vectors to consider.

## Crystal Potential in Bethe Theory

The crystal potential is periodic and so we also make a Fourier exanion of that potential
$$
V = V(x,y,z) = V(\vec{r}) = \sum_g V_g \exp(2\pi i\vec{g}\cdot\vec{r})
$$

The Fourier component of the crystal potential (in Volts) consists of several atoms $j$ over which we sum:
\begin{eqnarray}
V_g &=& \frac{h^2}{2 \pi m_0 e} \frac{1}{\Omega} \sum_j  f_{e_j}(\vec{g}) \exp(-2\pi i\vec{g}\cdot\vec{r_j})\\
&=& \frac{2 \pi  e a_0}{\Omega} \sum_j  f_{e_j}(\vec{g}) \exp(-2\pi i\vec{g}\cdot\vec{r_j})\\
\end{eqnarray}

where:

- $f_{e_j}$: atomic form (atomic scattering) factor of the $j$th atom
- $e$: charge of electron
- $m_0$: rest mass of electron
- $a_0$: Bohr radius
- $\Omega$: Volume of the unit cell

## Solution of Bethe Theory

Now, so far we haven't done anything, but substitute and expand. Let's put all this into the Schrödinger equation \ref{Schrodinger}
\begin{equation}
4\pi \left[ K^2 - (k_0^{(j)} + g)^2 +\sum_{h \neq 0} U_h e^{2\pi i \vec{h}\vec{r} }  \right] \cdot C_g^{(j)} e^{2\pi i (\vec{k}_o^{(j)} +\vec{g})\cdot \vec{r} }= 0
\end{equation}
This can only be zero, if all coefficients with same exponential function simultaneous become zero; this results in a set of equations, after collecting up terms containing the factor
$e^{2\pi i (\vec{k}_o^{(j)} +\vec{g})\cdot \vec{r} }$:
\begin{equation} \label{setEquat}
\left[ K^2 -(\vec{k}_o^{(j)} +\vec{g})^2 \right] C_g^{(j)} + \sum_{h\neq 0} U_h C_{g-h}^{(j)}=0; \qquad \vec{g}=\vec{g}_1, \vec{g}_2, ..., \vec{g}_n
\end{equation}

I made use of an abbreviation:
\begin{equation}
K=\frac{1}{h}\left[ 2m_0 E (1+\frac{E}{2E_0}) +2m_0 e U_0(1+\frac{E}{E_0}) \right]^{\frac{1}{2}}
\end{equation}
for the wave vector inside the the crystal which are not identical to the magnitude of the wave vectors of the Bloch waves $\vec{k}_g^{(j)}=\vec{k}_o^{(j)} +\vec{g}$.

Please note, that I introduced relativistic corrections (the terms in the round brackets in the equation above), too. It is enough to add this corrections for the energy at this point; it is not necessary to solve the Dirac equation (relativistic  Schrödinger equation).

The set of equations defined in \ref{setEquat} are essential for the  understanding of dynamic diffraction. Let's look at it a little more closely.

We get for each $j$ one equation; and this means we get for each Bloch wave one equation.

The second term in equation \ref{setEquat} (the term with the sum) mixes the Bloch waves ($C_{g-h}^{(j)}$). Effectively, we state that the inner potential $U_h$ mixes the Bloch waves; this is called dynamical coupling.

> **In summation:**
>
> We separated the problem!

## Two Beam Case
We rewrite the matrix expression for the boundary condition \ref{boundaryCond}in the two beam case:
\begin{equation}
\left(
\begin{array}{*{2}{c}}
C_{0}^{(1)} & C_{0}^{(2)} \\
C_{g}^{(1)} & C_{g}^{(2)} \\
\end{array}
\right) \cdot
\left(
\begin{array}{*{1}{c}}
\epsilon^{(1)} \\
\epsilon^{(2)} \\
\end{array}
\right) =
\left(
\begin{array}{*{1}{c}}
\phi_0^{(0)} \\
\phi_g^{(0)} \\
\end{array}
\right) =
\left(
\begin{array}{*{1}{c}}
1 \\
0\\
\end{array}
\right)
\end{equation}

In the kinematic case, the centers M of the various Ewald spheres (for the various incident directions) lay on a sphere of radius $k=\frac{1}{\lambda}$ around the origin of the reciprocal lattice. At some point the intensity in the diffracted beam will be more intense than in the incident beam, and therefore, we have to treat this scattered beam now in the same way we have treated the incident beam before: we need an Ewald sphere of radius $k$ for this direction/ for this reciprocal lattice point $\vec{g}$. Now we have two Ewald spheres, one around the origin $0$ and one around the reciprocal lattice point $\vec{g}$. The two spheres are not allowed to intersect each other, but a smooth kind of complicated surface has to be constructed.

The fundamental equations of the dynamic theory for the two beam case are:

\begin{eqnarray}
-\gamma^{(j)} C_0^{j} +& \frac{U_g}{2K} C_g^{(j)} &= 0\\
\frac{U_g}{2K} C_0^{j} +& (-\gamma^{(j)}+s) C_g^{(j)} &= 0
\end{eqnarray}

Such a homogeneous linear equation system for the $C_g^{(j)}$ has a non-zero solution of and only if the determinant of the coefficients is zero:

\begin{equation}
\left|
\begin{array}{*{2}{c}}
-\gamma^{(j)} & \frac{U_g}{2K}\\
\frac{U_g}{2K}& (-\gamma^{(j)}+s)\\
\end{array}
\right| = {\gamma^{(j)}}^2 -s\gamma^{(j)}-\frac{U_g^2}{4K^2} =0
\end{equation}

Which is the same as the Howie-Whelan equation \ref{HaW3} with $\xi_g = K^2/U_g$,
but now we know that the $\gamma^{(j)}$ are the Eigenvalues of a matrix problem.

Solution:
\begin{eqnarray}
\gamma^{(j)}  &=& \frac{1}{2} \left[ s-(-1)^j \sqrt{(U_g/K)^2 +s^2}   \right]\\
&=& \frac{1}{2} \left[ s-(-1)^j \sqrt{(1/\xi^2 +s^2}   \right]\\
&=& \frac{1}{2\xi_g} \left[ w-(-1)^j \sqrt{(1+w^2}   \right]\\
\end{eqnarray}

We made the substitution $w =s\xi_g$, in which the parameter $w$ characterizes the tilt out of the exact Bragg condition ($w=0$). The excitation error $s$ is zero in the exact Bragg condition, isn't it.

 The separation is $\Delta k_{z,min} = \gamma^{(1)}-\gamma^{(2)}= \frac{U_g}{2K}=\frac{1}{\xi_g}$.

By use of the eigenvalues $\gamma^{(j)}$, the linear systems of equations can be solved for the $C_g^{(j)}$. For the amplitude $\epsilon^{(j)} C_g^{(j)} = C_0^{j} C_g^{(j)}$ of the four Bloch waves with the vector $\vec{k}_0^{(j)}+\vec{g}$ we obtain:
\begin{eqnarray}
C_0^{j} C_0^{(j)} &=& \frac{1}{2}\left[ 1 + (-1)^j \frac{w}{\sqrt{1+w^2}} \right]\\
C_0^{j} C_g^{(j)} &=& -\frac{1}{2}\left[  \frac{(-1)^j}{\sqrt{1+w^2}} \right]
\end{eqnarray}

We put this into the equation for the scattered wave and substitute the thickness $t$ for the $z$ component of the vector $\vec{r}$:
\begin{eqnarray}
\psi_0(t)&=&\sum_{j=1}^2 C_0^{j} C_0^{(j)}e^{2\pi i k_z^{(j)}t}\\
\psi_g(t)&=&\sum_{j=1}^2 C_0^{j} C_g^{(j)}e^{2\pi i k_z^{(j)}t}\, e^{2\pi i k_z gx}
\end{eqnarray}
and we find (omitting common phase factors):
\begin{eqnarray}
\psi_0(t)= \cos (\pi \sqrt{1+w^2}\frac{t}{\xi_g}) & - \frac{iw}{\sqrt{1+w^2}} \sin (\pi \sqrt{1+w^2}\frac{t}{\xi_g})\\
\psi_g(t)=  &  \frac{i}{\sqrt{1+w^2}} \sin (\pi \sqrt{1+w^2}\frac{t}{\xi_g}) \, e^{2\pi i k_z gx}
\end{eqnarray}

The intensities of the transmission $T$ and reflection $R$ become:
\begin{equation}
\underbrace{\psi_g \psi_g^*}_R = \underbrace{1-\psi_0 \psi_0^*}_{1-T} =  \frac{1}{1+w^2} \sin^2 (\pi \sqrt{\sqrt{1+w^2} ( \frac{1}{\xi_g}})_{s_{eff}} \cdot t)
\end{equation}

The solution is the Pendellösung of two coupled oscillators (in mechanics: two pendulums connected with a spring).

Even in exact Bragg condition, the intensity oscillates between primary beam and Bragg reflected beam with increasing film thickness. Look at the plot below, the Pendellösung is shown without absorption.

Normally one would want to add an absorption term to reduce the intensity with thickness. This absorption term is better named a damping term and stems form the inelatic scattering to random angles instead of the  considered (here two)  Bragg angles

These oscillations of the intensities are commonly called ``rocking curve``.

```python
# ------ Input ------
xi_g = 4  # extiction distance (in terms of relative thickness )
omega = 0 # tilt from Bragg condition
damping = 0.6
# --------------------

t = np.linspace(0,8,401)

print(omega)
plt.figure()
plt.plot(t, (1-np.sin(np.pi * np.sqrt(np.sqrt(1.+ omega**2)*1/xi_g)*t)**2) * 1/np.exp(t*damping), label='incident beam')
plt.plot(t, np.sin(np.pi * np.sqrt(np.sqrt(1.+ omega**2)*1/xi_g)*t)**2 * 1/np.exp(t*damping), label='reflected beam')

plt.legend();
```

    0

    <IPython.core.display.Javascript object>

## Summary of Bethe Theory
The solution is the Pendellösung of two coupled oscillators.

The periodicity is the extinction length $\xi$, which tells at which thickness a beam is completely vanished.

Considering some absorption (well it's not a real absorption, but inelastic scattering) then we see that the amplitudes decrease slowly.

## Using Bethe Theory for Thickness Determination

We will do this in a lab and it will be your homework.

- The accurate thickness of the sample is an important but hard to obtain parameter, but it influences the contrast in all imaging modes.
- Be aware that with different techniques you perform different thickness measurement. In any high resolution image and diffraction experiment, you always look at the thickness of the crystalline part of the sample, omitting the contribution of contamination and amorphous surface layer (from sample preparation).
- In the Analytic Section of this class we learn how to the thickness from the  whole sample.

We can observe the above rocking curve in convergent beam electron diffraction patterns (CBED).

But we have to ensure that:
- Excitation error is as small as possible
- We are in two beam condition

### Experimental Considerations
- Choose a convergence angle $\alpha$ so that $\alpha < \theta_B$, to avoid overlapping of disks in the ZOLZ.
- The 000 disk usually contains concentric diffuse fringes, the Kossel-Möllenstedt fringes

- If you move the specimen, then you will see that the number of this fringes changes. In fact the number of each fringes increases by one every time the thickness increases by one extinction length.
- The foil thickness can be measured precisely at the point where you do your other analysis.

Please be aware that dynamic effects also occur for the HOLZ lines in a CBED pattern.

In practice to simplify the interpretation, we don't use zone axis conditions, but tilt to two--beam conditions with only one strongly excited Bragg beam.
- The CBED disks contain then parallel rather than concentric intensity oscillations as shown in the earlier figure.
- In fact, this intensity oscillations are equivalent to the rocking curve intensity oscillations discussed earlier.
- It helps, of you use an energy filter for this method.

### Thickness Determination
Because the oscillations are symmetric in the hkl disk we concentrate the analysis on this disk.
- The middle of the hkl disk is bright and originates from the exact Bragg condition ($\vec{s}=0$).
- We measure the distance between the midle (bright fringe) of the $hkl$ disk and the dark lines.

You obtain a deviation $s_i$ for each fringe from the equation:
\begin{equation}
s_i=\lambda\frac{\Delta\theta_i}{e\theta_B d^2}
\end{equation}
The Bragg angle $\theta_B$ is known from the separation of two disks and the lattice spacing $d$ is known from the sample or can be calculated through the camera length.

If the extinction distance $\xi_g$ is known you can calculate the foil thickness $t$ with:
\begin{equation}
\frac{1}{t^2} = \frac{s_1^2}{n_k^2}+\frac{1}{\xi_g^2n_k^2}
\end{equation}
where $n_k$ is an integer.

### Data Analysis
- assign $n=1$ to the first fringe $s_1$
- assign $n=2$ to the second fringe $s_2$
  and so on for all other fringes
- plot $(s_1/n_k)^2$ versus $(1/n_k)^2$.
- if you get a straight line, then you are finished and you have $k=i+j$, where $j$ is the largest integer $<(t/\xi_g)$.
- if not repeat the same thing with $n=2$ for $s_1$, $n=3$ for $s_2$, etc.
- repeat this increase by one till you get a straight line
- the slope of the line is $1/\xi_g^2$
- the extrapolated value for $1/n_k^2$ is $1/t^2$.

The whole procedure is summarized in the figure below.

## More about Bloch Waves
We can replace the  exponential functions by trigonometric functions and get:
\begin{eqnarray}
A^{(1)} &=& \cos \frac{\beta}{2}\\
A^{(2)} &=& \sin \frac{\beta}{2}
\end{eqnarray}

Some of the Bloch waves are located (have their maxima) between the atoms. These Bloch waves channel and are more or less undisturbed.

Another set is located on the atomic rows and will cause much more inelastic scattering than the other, also they will travel much faster.

The second set is especially important for Z-contrast image, where a small convergent beam is located at the atomic rows. You might consider the atoms like little lenses which keep the beam focused on the column.

## Navigation

-   **Next: [Mutli-Slice Theory](CH2_D02-Multi_Slice.ipynb)**
-   **Chapter 2: [Diffraction](CH2_00-Diffraction.ipynb)**
-   **List of Content: [Front](../_MSE672_Intro_TEM.ipynb)**

```python

```


# Lazy Evaluation & Generators

## Carrot Cake Recipe
- Add 3 eggs
- Add 300g flour
- Add some sugar
- Add 3 carrots cut into small pieces
- Add baking powder
- Mix all ingredients
- Bake for 1 hour
- Pour chocolate on top
- Serve

Python would do this in order and would realise late that preheating the oven is needed which could have been done earlier!

Distributed computing is also a problem!

Would go to the shop and by an ingredient, come back, see it needs flour, goes to shop etc...

## Lazy Evaluation in Python

- Lazy Evaluation reads it all once, and works out the steps!

- It only starts to work when you know what the outcome is

Lazy evaluation with Generators saves time because:
- Data is generated only when it is needed
- Not all data is kept in memory all the time - huge!

```python
def example1():
    '''functions with yield are called generator functions'''
    yield 1
    yield 2
    yield 4
    yield 8
```

```python
g = example1() # Does nothing yet
```

```python
g
```

    <generator object example1 at 0x107730620>

```python
next(g)
```

    1

```python
def example1():
    '''functions with yield are called generator functions'''
    a = 1
    for i in range(15):
        yield a
        a *= 2
```

```python
eg = example1()
eg
```

    <generator object example1 at 0x1077304c0>

```python
next(eg)
```

    1

```python
for i in range(3):
    print(next(eg), next(eg))
    print('----------------')
```

    2 4
    ----------------
    8 16
    ----------------
    32 64
    ----------------

```python
list(eg)
```

    [128, 256, 512, 1024, 2048, 4096, 8192, 16384]

```python
def example1():
    '''functions with yield are called generator functions'''
    a = 1
    while True:
        yield a
        a *= 2
```

#### We won't get endless loop because yield statement ends the loop

```python
h = example1()
```

```python
next(h)
```

    1

```python
[next(h) for x in range(3, 5)]
```

    [2, 4]

```python
e = enumerate(['a','b','c'])
e
```

    <enumerate at 0x107751048>

```python
next(e)
```

    (0, 'a')

```python
z = zip([1,2,3], ['a', 'b', 'c'])
z
```

    <zip at 0x107584848>

```python
next(z)
```

    (1, 'a')

```python
for x in z:
    print(x)
```

    (2, 'b')
    (3, 'c')

## Generators in Python
- enumerate
- zip
- BeautifulSoup.findall
- pd.GroupBy
- open text files
- many functions that iterate over data

### For more info, see: FUNCTIONAL PROGRAMMING IN PYTHON

### When to use Generators?

**Generators are an advanced tool present in Python.**

- Generators are “lazy functions”. They produce results like normal Python functions, but only when they are needed. Generators are executed on demand and return results in small portions, instead of returning everything at once.

- The main purpose of using generators is to save memory and calculation time when processing big datasets.

### Cases where generators can increase efficiency:
- Processing large amounts of data
- Stream processing.
- Piping: stacked generators can be used as pipes, in a manner similar to Unix pipes.
- Concurrency: generators can be used to generate (simulate) concurrency.

### Key concepts
| concept | description |
|---------|-------------|
| yield | returns a value, generator keeps running |
| next( ) | retrieves one value from a generator |
| while True: | is possible in generators |  |
| generator expression | generator equivalent of a list comprehension |
| iterator | generator-like instance, e.g. from enumerate() |

### Examples
A Python generator is defined by the `yield` keyword. The `yield` keyword replaces return. `yield` can be executed mutliple times. You cannot have `yield` and `return` in the same function.

An example:

```python
def simple_generator_function():
    yield 1
    yield 2
    yield 3
```

- Generators can be consumed like any iterable type, e.g. in `for` loops:

```python
for value in simple_generator_function():
    print(value)
```

    1
    2
    3

- Alternatively you can retrieve single elements from a generator with `next()`:

```python
our_generator = simple_generator_function()
```

- Calling the generator does nothing yet. Only when `next()` requests the next value, the generator is executed until the `yield` statement. Then it pauses until the next `yield` and so on.

```python
next(our_generator)
```

    1

```python
next(our_generator)

```

    2

```python
next(our_generator)

```

    3

### Number Generator
Many generators contain a `while` loop:

```python
def numberGenerator(n):
    number = 0
    while number < n:
        yield number
        number += 1
```

```python
myGenerator = numberGenerator(3)
```

```python
next(myGenerator)
```

    0

```python
next(myGenerator)
```

    1

```python
next(myGenerator)
```

    2

When the generator function exits, an Exception is raised:

```python
next(myGenerator)
```

### Iterators
The thing returned by a generator is called an **iterator**. Many functions in Python 3 return iterators (e.g. `range()`, `enumerate()`, `zip()`).

Among the things you can do to iterators are:

- request values with `next`.
- use them in a `for` loop.
- convert them to lists with `list()`.

### Infinite generators
Some generators yield results forever. Of course, you shouldn’t use them in a `for` loop:

```python
def double():
    i = 1
    while True:
        yield i
        i = i * 2
```

```python
next(double())
```

    1

```python
next(double())
```

    1

```python
def take(n, seq):
    """Returns first n values from the given sequence."""
    seq = iter(seq)
    result = []
    try:
        for i in range(n):
            result.append(next(seq))
    except StopIteration:
        pass
    return result
```

- We combine both functions to a stacked pipe:

```python
print(take(5, double()))
```

    [1, 2, 4, 8, 16]

### Generator Expressions
Generator expressions are very similar to list comprehensions, only with lazy evaluation.

```python
gen = (x + 1 for x in double() if x > 10)
```

```python
next(gen), next(gen), next(gen)
```

    (17, 33, 65)

### Exercises
#### Exercise 1: Square numbers
- Write a generator that produces a sequence of square numbers (1, 4, 9..).

- Retrieve the 100th number from the sequence.

```python
def square(n):
    yield n**2
```

```python
next(square(100))
```

    10000

#### Exercise 2: Build a Fibonacci generator
Write a generator that produces a sequence of numbers: 0, 1, 1, 2, 3, 5, 8, 13

\begin{align}
F_n = F_{n - 1} + F_{n - 2}
\end{align}

```python

```

```python

```

#### Exercise 3: Huge generator
Write a generator that endlessly repeats the text:

`"Peter Piper picked a peck of pickled peppers"`

Write that sentence into a text file 10 million times. Track how much memory is consumed while the program is running.

```python

```