# Longitudinal Data Analysis

# Panel Data Analysis II

Table of Contents

1  What we can relax about

2  Defining our statistical model

3  Understanding sources of variation

4  Data exploration

5  Pooled OLS Model

5.1  Conditions where Pooled OLS is suitable

6  Between Effects Model

6.1  Benefits of Between Effects

6.2  Limitations of Between Effects

7  Summary

In this section we estimate our first set of statistical models using panel data: **Pooled OLS** and **Between Effects**. We show some examples of how to estimate and interpret these models, and reflect on the conditions under which the models are appropriate.

## What we can relax about

In the sessions demonstrating how to quantitatively analyse panel data, we will cast aside the following concerns:

Missing data

Weights

Attrition

Multicollinearity

All three issues impinge on the estimation of panel data models but are not necessary to address for the purposes of learning about said models. We encourage you to consult the [reading list](https://github.com/DiarmuidM/longitudinal-data-analysis/blob/master/reading-list) for suggestions of resources that cover these topics.

## Defining our statistical model

Now we arrive at the interesting bit: estimating statistical models.

Let's return to our panel data on charities and define a statistical model for predicting a charity's annual gross income as a function of its age, the scale of its charitable activities, where it is located, what type of charity it is, and the number of sources of income it has, and the share of its income provided by government.

\begin{equation} \text{y}_{it} = \beta_0 + \beta_1x_{1it} + \beta_2x_{2i} + \beta_3x_{3i} + \beta_4x_{4i} + \beta_5x_{5it} + \beta_6x_{6it} + \epsilon_{it} \tag{1.1} \end{equation}

Where:

$\text{y}_{it}$ is log income for charity *i* at time *t*

$\beta_0$ is the constant term, which is our prediction for log income when the values of all other variables in the model are set to 0

$\text{x}_{1it}$ captures the age of charity *i* at time *t*, and $\beta_1$ is the effect of this variable on the outcome

$\text{x}_{2i}$ is a dummy variable identifying charities that operate at a local level

$\text{x}_{3i}$ is a dummy variable identifying charities registered in Westminster

$\text{x}_{4i}$ is a dummy variable identifying general charities

$\text{x}_{5it}$ captures the number of sources of income for charity *i* at time *t*

$\text{x}_{6it}$ captures the share its income charity *i* derives from government sources at time *t*

$\epsilon_{it}$ captures the residual for charity *i* at time *t* ($\text{y}_{it} - \hat{y}_{it}$)

## Understanding sources of variation

Remember to keep in mind the two sources of variation that exist in panel data ([Gould, n.d.](https://www.stata.com/support/faqs/statistics/between-estimator/)):

Cross-section information on differences between units

Time series information on differences over time within units

## Data exploration

Let's spend a little bit of time exploring the key variables in our statistical model.

```stata
use "../data/charity-panel-analysis-2020-09-10.dta", clear
```

    (Contains annual accounts of charities in E&W for financial years 2006-2017)

```stata
sum orgage, detail
```

                      Age of charity - in years
    -------------------------------------------------------------
          Percentiles      Smallest
     1%            4              0
     5%            7              1
    10%           10              1       Obs              23,826
    25%           16              1       Sum of Wgt.      23,826

    50%           27                      Mean           39.20129
                            Largest       Std. Dev.       42.4661
    75%           48            496
    90%           82            497       Variance       1803.369
    95%          112            498       Skewness       4.595531
    99%          180            499       Kurtosis       37.17673

```stata
sum nsources, detail
```

          Number of income sources where income >= £1,000
    -------------------------------------------------------------
          Percentiles      Smallest
     1%            1              0
     5%            2              0
    10%            2              1       Obs              23,826
    25%            3              1       Sum of Wgt.      23,826

    50%            4                      Mean           3.806724
                            Largest       Std. Dev.       1.24789
    75%            5              6
    90%            5              6       Variance       1.557228
    95%            6              6       Skewness      -.1130695
    99%            6              6       Kurtosis       2.425233

```stata
tab1 localc socser
```

    -> tabulation of localc

          Local |
        charity |      Freq.     Percent        Cum.
    ------------+-----------------------------------
              0 |      8,756       36.75       36.75
              1 |     15,070       63.25      100.00
    ------------+-----------------------------------
          Total |     23,826      100.00

    -> tabulation of socser

         Social |
        service |
        charity |      Freq.     Percent        Cum.
    ------------+-----------------------------------
              0 |     20,449       85.83       85.83
              1 |      3,377       14.17      100.00
    ------------+-----------------------------------
          Total |     23,826      100.00

## Pooled OLS Model

The starting point for any statistical modelling of panel data is to estimate a *Pooled OLS* model (fancy way of saying linear regression).

The observations are "pooled", which just means we ignore the nested nature of panel data.

In other words we assume that each observation (i.e., row within a long format data set) is independent of other observations (Gayle and Lambert, 2018).

Fundamental problem of pooling observations (Gayle & Lambert, 2018, p. 58):
> The model does not recognise that there are multiple contributions of data from the same individuals, and therefore, it estimates results as if there are many individuals who shared the same characteristics. This impacts upon the estimate of measures such as variances and standard errors.

```stata
regress linc orgage localc west genchar nsources govern_share
est store pols
```

          Source |       SS           df       MS      Number of obs   =    23,826
    -------------+----------------------------------   F(6, 23819)     =    410.54
           Model |   2225.8864         6  370.981066   Prob > F        =    0.0000
        Residual |  21523.6961    23,819  .903635591   R-squared       =    0.0937
    -------------+----------------------------------   Adj R-squared   =    0.0935
           Total |  23749.5825    23,825  .996834524   Root MSE        =     .9506

    ------------------------------------------------------------------------------
            linc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
          orgage |   .0036028     .00015    24.01   0.000     .0033087    .0038969
          localc |  -.3302434   .0130224   -25.36   0.000    -.3557682   -.3047187
            west |   .1121865   .0253139     4.43   0.000     .0625697    .1618033
         genchar |  -.3170303   .0139082   -22.79   0.000    -.3442913   -.2897693
        nsources |   .1053884   .0050963    20.68   0.000     .0953993    .1153774
    govern_share |    .000644   .0002035     3.16   0.002     .0002451    .0010429
           _cons |   14.96317   .0236406   632.94   0.000     14.91683    15.00951
    ------------------------------------------------------------------------------

### Conditions where Pooled OLS is suitable

Pooled OLS can produce consistent estimates of the explanatory variables if:

The model is correctly specified

The explanatory variables are uncorrelated with the error term (Cameron and Trivedi, 2010)

**TASK:** Do you think our statistical model is correctly specified, and there is no correlation between error term and explanatory variables?

In our statistical model of charity income, it is unlikely that the interpretation of the coefficients would change drastically if we addressed the under-estimation of the standard errors (the sample size is very large).

We'll cover the various tests and checks we can perform to examine whether Pooled OLS model violates the *independence of errors* assumption in a later section.

## Between Effects Model

Once again estimate a cross-sectional model (Pooled OLS).

However this time we transform the data so that there is one observation per unit.

As a result we end up modelling the mean of Y on the mean of our X variables.

```stata
xtreg linc orgage localc west genchar nsources govern_share, be
est store beff
```

    Between regression (regression on group means)  Number of obs     =     23,826
    Group variable: regno                           Number of groups  =      2,166

    R-sq:                                           Obs per group:
         within  = 0.0063                                         min =         11
         between = 0.1042                                         avg =       11.0
         overall = 0.0925                                         max =         11

                                                    F(6,2159)         =      41.86
    sd(u_i + avg(e_i.))=  .9109813                  Prob > F          =     0.0000

    ------------------------------------------------------------------------------
            linc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
          orgage |   .0035048   .0004791     7.32   0.000     .0025652    .0044443
          localc |  -.3282906   .0414364    -7.92   0.000      -.40955   -.2470312
            west |   .1167392   .0805284     1.45   0.147     -.041182    .2746604
         genchar |  -.3210918   .0449127    -7.15   0.000    -.4091685    -.233015
        nsources |   .1384596   .0193749     7.15   0.000     .1004643    .1764549
    govern_share |   .0002749   .0007478     0.37   0.713    -.0011916    .0017415
           _cons |   14.85058   .0835091   177.83   0.000     14.68681    15.01435
    ------------------------------------------------------------------------------

Estimating a Between Effects model is equivalent to collapsing the data and estimating your regression model on the resulting observations:

```stata
preserve
    collapse (mean) linc orgage localc west genchar nsources govern_share, by(regno)
    regress linc orgage localc west genchar nsources govern_share
    est store coll
restore
```

          Source |       SS           df       MS      Number of obs   =     2,166
    -------------+----------------------------------   F(6, 2159)      =     41.86
           Model |  208.427331         6  34.7378885   Prob > F        =    0.0000
        Residual |  1791.72593     2,159  .829886952   R-squared       =    0.1042
    -------------+----------------------------------   Adj R-squared   =    0.1017
           Total |  2000.15326     2,165  .923858319   Root MSE        =    .91098

    ------------------------------------------------------------------------------
            linc |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
          orgage |   .0035048   .0004791     7.32   0.000     .0025652    .0044443
          localc |  -.3282906   .0414364    -7.92   0.000      -.40955   -.2470313
            west |   .1167393   .0805284     1.45   0.147    -.0411819    .2746605
         genchar |  -.3210918   .0449127    -7.15   0.000    -.4091685    -.233015
        nsources |   .1384596   .0193749     7.15   0.000     .1004643     .176455
    govern_share |   .0002749   .0007478     0.37   0.713    -.0011916    .0017415
           _cons |   14.85058   .0835091   177.83   0.000     14.68681    15.01435
    ------------------------------------------------------------------------------

```stata
est table pols beff coll
```

    -----------------------------------------------------
        Variable |    pols         beff         coll
    -------------+---------------------------------------
          orgage |  .00360282    .00350476    .00350476
          localc | -.33024344    -.3282906   -.32829062
            west |  .11218649    .11673923    .11673928
         genchar | -.31703032   -.32109178   -.32109176
        nsources |  .10538836     .1384596    .13845961
    govern_share |  .00064402    .00027494    .00027494
           _cons |  14.963168    14.850581    14.850581
    -----------------------------------------------------

### Benefits of Between Effects

Sidesteps the problem of interdependence of observations in the original panel data.

Smooths the effect of anomalous time periods (e.g., excess deaths calculation).

Controls for omitted variables that change over time but are constant between units (e.g., national policies).

### Limitations of Between Effects

What might the limitations of this approach be?

Cannot estimate observed variables that change over time but are constant between units (e.g., national policies).

Discard a lot of information by examining mean outcomes and inputs e.g., change over time.

Cannot control for unobserved explanatory variables that are fixed within but vary between units e.g., organisational culture.

The limitations of the Between Effects model far outweigh the benefits in most cases, and thus it is not widely used in practice (Mehmetoglu and Jakobsen, 2016).

However it plays a crucial role in the estimation of another panel data model — Random Effects model — and thus it is important to understand how it works and what it offers.

## Summary

Both the Pooled OLS and Between Effects models provide useful information on the association between an outcome *Y* and a set of explanatory variables *X*.

However both can be suboptimal from a substantive perspective (no change over time).

More concerningly, they offer no ability to control for residual heterogeniety in the form of *unobserved time-invariant* explanatory variables.


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

```python
import numpy as np
np.random.seed(123456)
```

The absence of the probability density for the raw data means that we have to
argue about sequences of random variables in a structured way. From basic
calculus, recall the following convergence notation,

$$
x_n \rightarrow x_o
$$
for the real number sequence $x_n$. This means that for any given
$\epsilon>0$,
no matter how small, we can exhibit a $m$ such that for
any $n>m$, we have

$$
\vert x_n-x_o \vert < \epsilon
$$

 Intuitively, this means that once we get
past $m$ in the sequence, we
get as to
within $\epsilon$ of $x_o$. This means
that nothing surprising
happens in the
sequence on the long march to infinity,
which gives a sense of
uniformity to the
convergence process.  When we argue
about convergence for
statistics, we want to
same look-and-feel as we have here,
but because we are
now talking about random
variables, we need other concepts.
There are two
moving parts for random
variables. Recall from our probability
chapter that
random variables are really
functions that map sets into the real
line:
$X:\Omega \mapsto \mathbb{R}$.
Thus, one part is the behavior of the
subsets
of $\Omega$ in terms of
convergence.   The other part is how the
sequences of
real values of the random
variable behave in convergence.

##
Almost Sure Convergence

The most
straightforward extension into statistics of
this convergence concept
is *almost
sure convergence*, which is also known as
*convergence with probability one*,

$$
\begin{equation}
\mathbb{P}\lbrace
\textnormal{for each } \epsilon>0
\textnormal{ there is } n_\epsilon>0
\textnormal{ such that for all }
n>n_\epsilon, \: \vert X_n-X \vert < \epsilon
\rbrace = 1
\label{eq:asconv}
\tag{1}
\end{equation}
$$

 Note the similarity to
the prior notion of convergence for real
numbers.  When
this happens, we write
this as $X_n \overset{as}{\to} X$.  In
this context,
almost sure convergence
means that if we take any particular
$\omega\in\Omega$
and then look at the
sequence of real numbers that are
produced by each of the
random variables,

$$
(X_1(\omega),X_2(\omega),X_3(\omega),\ldots,X_n(\omega))
$$

 then this sequence
is just a real-valued sequence in the
sense of our
convergence on the real line
and converges in the same way. If we
collect all of
the $\omega$ for which this
is true and the measure of that
collection equals
one, then we have almost sure
convergence of the random
variable. Notice how the
convergence idea applies to
both sides of the random
variable: the (domain)
$\Omega$ side and the (co-
domain) real-valued side.

An equivalent and more
compact way of writing this
is the following,

$$
\mathbb{P}\left(\omega\in\Omega
\colon\lim_{n\rightarrow\infty}
X_n(\omega)=X(\omega) \right)=1
$$

**Example.**
To get some feel for the mechanics of this kind of convergence
consider the
following sequence of uniformly distributed random variables on
the
unit
interval, $X_n \sim \mathcal{U}[0,1]$. Now, consider taking
the maximum of
the
set of $n$ such variables as the following,

$$
X_{(n)} = \max \lbrace
X_1,\ldots,X_n \rbrace
$$

 In other words, we scan through a list of $n$
uniformly distributed
random
variables and pick out the maximum over the set.
Intuitively, we should
expect
that $X_{(n)}$ should somehow converge to one.
Let's see if we can make
this
happen almost surely.  We want to exhibit $m$ so
that the following is
true,

$$
\mathbb{P}(\vert 1 - X_{(n)} \vert) < \epsilon
\textnormal{ when } n>m
$$

 Because $X_{(n)}<1$, we can simplify this as the
following,

$$
1-\mathbb{P}(X_{(n)}<\epsilon)=1-(1-\epsilon)^m
\underset{m\rightarrow\infty}{\longrightarrow} 1
$$

 Thus, this sequence
converges almost surely. We can work this
example out in
Python using Scipy to
make it concrete with the following
code,

```python
from scipy import stats
u=stats.uniform()
xn = lambda i: u.rvs(i).max()
xn(5)
```

    0.966717838482003

Thus, the `xn` variable is the same as the $X_{(n)}$ random variable
in our
example. [Figure](#fig:Convergence_001) shows a plot of these random
variables
for different values of $n$ and multiple realizations of each random
variable
(multiple gray lines). The dark horizontal line is at the `0.95`
level. For this
example, suppose we are interested in the convergence of the
random variable to
within `0.05` of one so we are interested in the region
between one and `0.95`.
Thus, in our Equation [1](#eq:asconv), $\epsilon=0.05$.
Now, we have to find
$n_\epsilon$ to get the almost sure convergence. From
[Figure](#fig:Convergence_001), as soon as we get past $n>60$, we can see that
all the realizations start to fit in the region above the `0.95` horizontal
line.  However, there are still some cases where a particular realization will
skip below this line. To get the probability  guarantee of the definition
satisfied, we have to make sure that for whatever $n_\epsilon$ we settle on,
the
probability of this kind of noncompliant behavior should be extremely
small,
say, less than 1%.  Now, we can compute the following to estimate this
probability for $n=60$ over 1000 realizations,

```python
import numpy as np
np.mean([xn(60) > 0.95 for i in range(1000)])
```

    0.961

So, the probability of having a noncompliant case beyond $n>60$ is
pretty good,
but not still what we are after (`0.99`). We can solve for the $m$
in our
analytic proof of convergence by plugging in our factors for $\epsilon$
and our
desired probability constraint,

```python
print (np.log(1-.99)/np.log(.95))
```

    89.78113496070968

Now, rounding this up and re-visiting the same estimate as above,

```python
import numpy as np
np.mean([xn(90) > 0.95 for i in range(1000)])
```

    0.995

which is the result we were looking for. The important thing to
understand from
this example is that we had to choose convergence criteria for
*both* the values
of the random variable (`0.95`) and for the probability of
achieving that level
(`0.99`) in order to compute the $m$.  Informally
speaking, almost sure
convergence means that not only will any particular $X_n$
be close to $X$ for
large $n$, but whole sequence of values will remain close
to $X$ with high
probability.

Almost sure convergence example for
multiple realizations of the limiting sequence.

## Convergence
in Probability

A weaker kind of convergence is *convergence in probability*
which means the
following:

$$
\mathbb{P}(\mid X_n -X\mid > \epsilon)
\rightarrow 0
$$

 as $n \rightarrow \infty$ for each $\epsilon > 0$.

This is
notationally
shown
as $X_n \overset{P}{\to} X$.  For example, let's consider the
following
sequence
of random variables where $X_n = 1/2^n$ with probability
$p_n$ and
where $X_n=c$
with probability $1-p_n$. Then, we have $X_n
\overset{P}{\to} 0$
as $p_n
\rightarrow 1$.  This is allowable under this notion
of convergence
because a
diminishing amount of *non-converging* behavior
(namely, when
$X_n=c$) is
possible. Note that we have said nothing about *how*
$p_n
\rightarrow 1$.
**Example.** To get some sense of the mechanics of this
kind of convergence,
let
$\lbrace X_1,X_2,X_3,\ldots \rbrace$ be the indicators
of the corresponding
intervals,

$$
(0,1],(0,\tfrac{1}{2}],(\tfrac{1}{2},1],(0,\tfrac{1}{3}],(\tfrac{1}{3},\tfrac{2}{3}],(\tfrac{2}{3},1]
$$

 In other words, just keep splitting the unit interval into equal
chunks and
enumerate those chunks with $X_i$. Because each $X_i$ is an
indicator function,
it takes only two values: zero and one.  For example,
for $X_2=1$ if $0<x \le
1/2$ and zero otherwise. Note that $x \sim
\mathcal{U}(0,1)$. This means that
$P(X_2=1)=1/2$. Now, we want to compute
the sequence of $P(X_n>\epsilon)$ for
each $n$ for some $\epsilon\in (0,1)$.
For $X_1$, we  have $P(X_1>\epsilon)=1$
because we already chose $\epsilon$
in the interval covered by $X_1$. For $X_2$,
we have $P(X_2>\epsilon)=1/2$,
for $X_3$, we have $P(X_3>\epsilon)=1/3$, and so
on.  This produces the
following sequence:
$(1,\frac{1}{2},\frac{1}{2},\frac{1}{3},\frac{1}{3},\ldots)$.  The limit
of the
sequence is zero so that $X_n \overset{P}{\to} 0$. However, for
every $x\in
(0,1)$, the sequence of  function values of $X_n(x)$ consists
of infinitely many
zeros and ones (remember that indicator functions can
evaluate to either zero or
one).  Thus, the set of $x$ for which the
sequence $X_n(x)$ converges is empty
because the sequence bounces
between zero and one. This means that almost sure
convergence fails here even though we have convergence in probability.
The key
distinction is that convergence in probability considers the convergence
of a
sequence of probabilities whereas almost sure convergence is
concerned about the
sequence of values of the random variables over
sets of events that *fill out*
the underlying probability space entirely (i.e.,
with probability one).

This is
a good example so let's see if we can make it concrete with some
Python. The
following is a function to compute the different subintervals,

```python
make_interval= lambda n: np.array(list(zip(range(n+1),
                                           range(1,n+1))))/n
```

Now, we can use this function to create a Numpy
array of intervals, as in the
example,

```python
intervals= np.vstack([make_interval(i) for i in range(1,5)])
print (intervals)
```

    [[0.         1.        ]
     [0.         0.5       ]
     [0.5        1.        ]
     [0.         0.33333333]
     [0.33333333 0.66666667]
     [0.66666667 1.        ]
     [0.         0.25      ]
     [0.25       0.5       ]
     [0.5        0.75      ]
     [0.75       1.        ]]

The following function computes the bit string in our example,
$\lbrace
X_1,X_2,\ldots,X_n \rbrace$,

```python
bits= lambda u:((intervals[:,0] < u) & (u<=intervals[:,1])).astype(int)
bits(u.rvs())
```

    array([1, 0, 1, 0, 0, 1, 0, 0, 0, 1])

Now that we have the individual bit strings, to show convergence we
want to
show
that the probability of each entry goes to a limit. For example,
using ten
realizations,

```python
print (np.vstack([bits(u.rvs()) for i in range(10)]))
```

    [[1 1 0 1 0 0 0 1 0 0]
     [1 1 0 1 0 0 0 1 0 0]
     [1 1 0 0 1 0 0 1 0 0]
     [1 0 1 0 0 1 0 0 1 0]
     [1 0 1 0 0 1 0 0 1 0]
     [1 1 0 0 1 0 0 1 0 0]
     [1 1 0 1 0 0 1 0 0 0]
     [1 1 0 0 1 0 0 1 0 0]
     [1 1 0 0 1 0 0 1 0 0]
     [1 1 0 1 0 0 1 0 0 0]]

We want the limiting probability of a one in each column to convert
to a limit.
We can estimate this over 1000  realizations using the following
code,

```python
np.vstack([bits(u.rvs()) for i in range(1000)]).mean(axis=0)
```

    array([1.   , 0.493, 0.507, 0.325, 0.34 , 0.335, 0.253, 0.24 , 0.248,
           0.259])

Note that these entries should approach the
$(1,\frac{1}{2},\frac{1}{2},\frac{1}{3},\frac{1}{3},\ldots)$ sequence we found
earlier. [Figure](#fig:Convergence_002) shows the convergence of these
probabilities for a large number of intervals. Eventually, the probability
shown
on this graph will decrease to zero with large enough $n$. Again, note
that the
individual sequences of zeros and ones do not converge, but the
probabilities of
these sequences converge. This is the key difference between
almost sure
convergence and convergence in probability. Thus, convergence in
probability
does *not* imply  almost sure convergence. Conversely, almost sure
convergence
*does* imply convergence in probability.

Convergence in
probability for the random variable sequence.

The following
notation should help emphasize the difference
between almost sure convergence
and convergence in probability,
respectively,

$$
\begin{align*}
P\left(\lim_{n\rightarrow \infty} \vert X_n-X\vert <
\epsilon\right)&=1
\textnormal{(almost sure convergence)}  \\\
\lim_{n\rightarrow \infty}  P(\vert
X_n-X\vert < \epsilon)&=1
\textnormal{(convergence in probability)}
\end{align*}
$$

## Convergence in Distribution

So far, we have been discussing
convergence in
terms of
sequences of probabilities or sequences of values taken
by
the random
variable.  By contrast,  the next major kind of
convergence is
*convergence in
distribution* where

$$
\lim_{n \to \infty}  F_n(t) = F(t)
$$
for all $t$ for which $F$ is continuous and $F$ is the
cumulative density
function. For this case, convergence is only
concerned with the cumulative
density function, written as $X_n
\overset{d}{\to} X$.

**Example.** To
develop some intuition about this kind of convergence,
consider a sequence of
$X_n$ Bernoulli random variables. Furthermore,
suppose these are all really just
the same random variable $X$.
Trivially, $X_n \overset{d}{\to} X$. Now, suppose
we define $Y=1-X$,
which means that $Y$ has the same distribution as $X$. Thus,
$X_n
\overset{d}{\to} Y$. By contrast, because $\vert X_n - Y\vert=1$ for all
$n$, we can never have almost sure convergence or convergence in
probability.
Thus, convergence in distribution is the weakest
of the three forms of
convergence in the sense that it is implied by
the other two, but implies
neither of the two.

As another striking example, we could have $Y_n
\overset{d}{\to} Z$ where $Z
\sim \mathcal{N}(0,1)$, but we could also have $Y_n
\overset{d}{\to} -Z$.
That is, $Y_n$ could converge in distribution to either
$Z$ or $-Z$. This
may seem ambiguous, but this kind of convergence is
practically very useful
because it allows for complicated distributions to be
approximated by
simpler distributions.

## Limit Theorems

Now that we have all of these notions of
convergence, we can apply them to
different situations and see what kinds of
claims we can construct from them.

**Weak Law of Large Numbers.**  Let $\lbrace
X_1,X_2,\ldots,X_n \rbrace$ be an
iid (independent, identically distributed) set
of random variables with finite
mean $\mathbb{E}(X_k)=\mu$ and finite variance.
Let $\overline{X}_n =
\frac{1}{n}\sum_k X_k$. Then, we have $\overline{X}_n
\overset{P}{\to} \mu$.
This result is important because we frequently estimate
parameters using an
averaging process of some kind. This basically justifies
this in terms of
convergence in probability. Informally, this means that the
distribution of
$\overline{X}_n$ becomes concentrated around $\mu$ as
$n\rightarrow\infty$.

**Strong Law of Large Numbers.**  Let $\lbrace
X_1,X_2,\ldots,\rbrace$ be an
iid set of random variables. Suppose that
$\mu=\mathbb{E}\vert
X_i\vert<\infty$, then $\overline{X}_n \overset{as}{\to}
\mu$. The reason this
is called the strong law is that it implies the weak law
because almost sure
convergence implies convergence in probability. The so-
called  Komogorov
criterion gives the convergence of the following,

$$
\sum_k
\frac{\sigma_k^2}{k^2}
$$

 as a sufficient condition for concluding that the
Strong Law applies
to the
sequence $ \lbrace X_k \rbrace$ with corresponding
$\lbrace \sigma_k^2
\rbrace$.
As an example, consider an infinite sequence of
Bernoulli trials with $X_i=1$
if
the $i^{th}$ trial is successful. Then
$\overline{X}_n$ is the relative
frequency of successes in $n$ trials and
$\mathbb{E}(X_i)$ is the
probability
$p$ of success on the $i^{th}$ trial. With
all that established,
the Weak Law
says only that if we consider a sufficiently
large and fixed
$n$, the
probability that the relative frequency will converge
to $p$ is
guaranteed. The
Strong Law states that if we regard the observation of
all
the infinite $\lbrace
X_i \rbrace$ as one performance of the experiment, the
relative frequency of
successes will almost surely converge to $p$.  The
difference between the Strong
Law and the Weak Law of large numbers is
subtle
and rarely arises in practical
applications of probability theory.

**Central
Limit Theorem.**  Although the
Weak Law of Large Numbers tells us
that the
distribution of $\overline{X}_n$
becomes concentrated around $\mu$, it
does not
tell us what that distribution
is. The Central Limit Theorem (CLT)
says that
$\overline{X}_n$ has a
distribution that is approximately Normal
with mean $\mu$
and variance
$\sigma^2/n$. Amazingly, nothing is assumed
about the distribution
of $X_i$,
except the existence
of the mean and variance. The following is the
Central
Limit Theorem:
Let $\lbrace X_1,X_2,\ldots,X_n \rbrace$ be iid with mean
$\mu$
and
variance $\sigma^2$. Then,

$$
Z_n =
\frac{\sqrt{n}(\overline{X}_n-\mu)}{\sigma}
\overset{P}{\longrightarrow}
Z\sim\mathcal{N}(0,1)
$$

 The loose interpretation of the Central Limit Theorem
is that
$\overline{X}_n$
can be legitimately approximated by a Normal
distribution.
Because we are
talking about convergence in probability here,
claims
about probability are
legitimized, not claims about the random variable
itself. Intuitively, this
shows that normality arises from sums of small,
independent disturbances of
finite variance. Technically, the finite
variance
assumption is essential for
normality. Although the Central Limit
Theorem
provides a powerful, general
approximation, the quality of the
approximation for
a particular situation still
depends on the original
(usually unknown)
distribution.


# Fundamental Concepts

An enormous amount of the work in this class will rely on speed and dexterity with units, the basic physics of particles, and conversion between masses, numbers, and concentrations of isotopes.

## Learning Objectives

By the end of this lesson, you should be able to:

- Differentiate modern physical units
- List the SI units and derived SI units
- Disambiguate SI magnitude prefixes
- Recognize physical units important to nuclear engineering
- Calculate conversions among units
- Derive the electron volt
- Define the atomic mass unit
- Place neutrons, protons, photons, and electrons within the Standard Model
- Decipher the $^A_ZX$ atomic nomenclature
- Define nuclide, isotope, isobar, isotone, isomer
- Calculate relative atomic masses from $\gamma_i$ atomic abundances
- Understand and use Avogadro's number $N_A$ to convert between atoms, moles, and grams
- Calculate atom density, isotope number density, and molecule density from various combinations of $\rho$, $\mathscr{A}$, $\gamma_i$, $N_a$, $w_i$, etc.
- Estimate the diameter of an atom and a nucleus

# Before we get started....

* I'd like to state that my office hours will be held in 118 Talbot Laboratory
* The readings for next week are posted as an announcement in Canvas. The readings for week 3 will be posted on Monday
* Homework 1 and quiz 2 are now active on gradescope

# Units

The seven SI base units and their interdependency.
Clockwise from top left:
- second (time),
- meter (distance),
- Ampere (electric current),
- mole (amount of substance),
- kilogram (mass),
- Kelvin (temperature)
- and candela (luminous intensity)

## Consider : Why do the base units have interdependencies?

While the kilogram is in some sense a very "pure" basic unit, others are derived from it. Moles can't be defined without a notion of mass.

## Derived Units

Check the zoom poll! Let's talk about derived units.

```python
from IPython.display import IFrame
IFrame("https://en.wikipedia.org/wiki/SI_derived_unit#Derived_units_with_special_names", width="1000", height="700", frameBorder="0")
```

# Why bother with a standard?

If we compare the _long ton, short ton_, and _metric ton_ in SI units of _kg_, we can see the logic of the kg.

- The common U.S. unit is the short ton of 2000 lb, 907.185 kg, or 20 short hundredweight (cwt).
- The long ton is 20 long cwt, 1016 kg., 22.4 short cwt, or 2240 lb.
- The metric ton is exactly 1000 kg.

In nuclear engineering, our convention is to use metric tons.

# The electronvolt (eV)
An unit of energy, work, and equivalent mass, used in atomic physics, particle physics and high energy physics.

\begin{align}
1 eV = 1.60217653\times10^{-19} J
\end{align}

An electronvolt is the amount of work done by a single electron moving across an electric potential difference of one volt. Hence, it has a value of one volt, 1 J/C, multiplied by the electron's elementary charge e, 1.6021766208(98)×10−19 C.

Its definition is empirical. That is, its value in SI units must be obtained experimentally.[5]

# Atomic Nomenclature

When we talk about isotopes, this is a very common way you'll see them expressed

\begin{align}
^{A}_{Z}X
\end{align}

You might see a shorthand for this called a ZAID number. Can you guess what it looks like?

# Now, let's talk about the chart of the nuclides:

# Number of Atoms (n) and Number Density (N)
The number of atoms or molecules (n) in a mass (m) of a pure material having atomic or molecular weight (M) is easily computed from the following equation using Avogadro's number.

\begin{align}
n=\frac{m N_A}{\mathscr{A}}
\end{align}

where

\begin{align}
n &=\mbox{number of atoms or molecules in sample}\\
m &= \mbox{mass of the sample}\\
N_A &= \mbox{Avogadro's number, }6.022\times10^{23}\\
\mathscr{A} &= \mbox{atomic weight of the sample }
\end{align}

```python
def NA():
    """Avogadro's number (NA = 6.022x10^23 atoms or molecules per gram-mole)"""
    return 6.022E23

def n(m, a, na):
    """The number of atoms or molecules (n) in a mass (m) of pure
    material with atomic or molecular weight (a).

    Parameters
    -----------
    m : double
        The mass of the material
    a : double
        The atomic or molecular weight of the atom or molecule of interest
    na : double
        Avogadro's number
    """
    return m*na/a
```

```python
# How many atoms of Boron are in 5 grams of boron?

a_boron = 10.811 # atomic weight of elemental boron (g)
m_sample = 5 # grams in sample

n_boron = n(m=m_sample, a=a_boron, na=NA())
print(n_boron)
```

    2.785126260290445e+23

```python
# How many of those are Boron 10?
# The isotopic oabundance of Boron 10 in elemental boron is 19.9%
gamma_boron10 = 0.199
n_boron10 = n_boron*gamma_boron10
print(n_boron10)
```

    5.542401257977986e+22

In some situations, the **atomic number density (N)**, which is the concentration of atoms or molecules per unit volume (V), is an easier quantity to find when the material density ($\rho$) is given.

\begin{align}
N &= \frac{n}{V}\\
  &= \frac{\rho N_A}{\mathscr{A}}
\end{align}

```python
def num_dens_from_vol(n, v):
    """The atomic number density.
    That is, the concentration of atoms or molecules per unit volume (V)

    Parameters
    -----------
    n : double
        Number of atoms in the sample
    v : double
        volume of the sample
    """
    return n/v

def num_dens_from_rho(rho, na, a):
    """The atomic number density.
    That is, the concentration of atoms or molecules per unit volume (V)

    Parameters
    -----------
    rho : double
        material density (in units like g/cm^3 or kg/m^3) of the sample
    na : double
        Avogadro's number
    a : double
        The atomic or molecular weight of the atom or molecule of interest
    """
    return rho*na/a
```

```python
boron_density = 2.35 # g/cm^3
sample_vol = m_sample/boron_density

num_dens_from_vol(n_boron, sample_vol)
```

    1.3090093423365092e+23

```python
num_dens_from_rho(boron_density, NA(), a_boron)
```

    1.3090093423365092e+23

# Number density for compounds
For a chemical compound (mixture) Z, which is composed of elements X and Y, the number (atom) density of the compound is calculated from
\begin{align}
N_Z &= N_{mix}\\
    &= \frac{\rho_{mix}N_A}{\mathscr{A}_{mix}}
\end{align}

In some cases, the desired quantity is the number density of the compound constituents. Specifically, if $Z=X_pY_q$, then there are p atoms of X and q atoms of Y for every molecule of Z. Thus
\begin{align}
N_X &= pN_{Z}\\
N_Y &= qN_{Z}
\end{align}

# Exercise: Find the $^{235}U$ concentration for 3 atom percent $^{235}U$ in $UO_2$.

- We have a sample of uranium oxide and would like to find the number of $^{235}U$ atoms in it.
- We are told it has a $^{235}U$ concentration for 3 atom percent $^{235}U$.

\begin{align}
\gamma_{^{235}U} &= 0.03\\
N_{^{235}U} &= \gamma_{^{235}U}N_U \\
\end{align}

- We have no way of calculating $N_U$ because there's all of this oxygen in the sample, so we can only measure for densities and volumes of $UO_2$. We measure the sample and find $\rho_{UO_2} = 10.5\frac{g}{cm^3}$.
- We also know that there is one uranium atom in each molecule of $UO_2$, so maybe we can get somewhere.

\begin{align}
N_{^{235}U} &= \gamma_{^{235}U}N_U \\
            &= \gamma_{^{235}U}N_{UO_2}\\
            &= \gamma_{^{235}U}\frac{\rho_{^{235}U}N_A}{\mathscr{A}_{UO_2}}\\
\end{align}

- Now,  we need $\mathscr{A}_{UO_2}$, which will rely on our understanding of compounds.
- We determine $\mathscr{A}_{UO_2} = \gamma_{238}\mathscr{A}_{238}
               + \gamma_{235}\mathscr{A}_{235} + 2\mathscr{A}_{O}$

\begin{align}
N_{^{235}U} &= \gamma_{^{235}U}\frac{\rho_{UO_2}N_A}{\mathscr{A}_{UO_2}}\\
            &= \gamma_{^{235}U}\frac{\rho_{UO_2}N_A}{\gamma_{238}\mathscr{A}_{238}
               + \gamma_{235}\mathscr{A}_{235} + 2\mathscr{A}_{O}}\\
            &= 0.03\left[\frac{\mbox{atoms }^{235}U}{\mbox{atoms U}}\right]\frac{10.5 \left[\frac{\mbox{g}}{cm^3}\right]6.022\times10^{23} \left[\frac{\mbox{atoms}}{\mbox{mol}}\right]}{238.05 (0.97)
            \left[\frac{g}{mol}\right]  + 235.04(0.03)\left[\frac{g}{mol}\right] + 2 (15.99994)\left[\frac{g}{mol}\right]}\\
            &= 7.03\times 10^{20}\left[\frac{atoms}{cm^3}\right]
\end{align}

# Estimate the diameter of an atom

The inverse of atom density $\left[\frac{atoms}{cm^3}\right]$ is volume per atom. Thus, one can determine the volume of an atom:

\begin{align}
V&=1/N
\end{align}

where

\begin{align}
N &= \frac{\mbox{atoms}}{cm^3}\\
N &= \frac{cm^3}{\mbox{atom}}
\end{align}

If we assume an atom is approximated by a cubic volume, then their radii are about $\frac{1}{N^{1/3}}$. For a typical atom, the radius is then approximately on the order of an angstrom (Å), or $1\times10^{-10}m$.

# The size of the nucleus

By now you've heard that the volume of the nucleus is much smaller than the volume of an atom. To determine the radius of the nucleus, the formula we use is:

\begin{align}
R &=R_{0}A^{1/3}
\end{align}

where

\begin{align}
R_0 &\simeq 1.2 \times 10^{13}\\
\end{align}

```python

```


```python
%matplotlib inline
```

``torch.autograd`` 에 대한 간단한 소개
-------------------------------------------

``torch.autograd`` 는 신경망 학습을 지원하는 PyTorch의 자동 미분 엔진입니다.
이 단원에서는 autograd가 신경망 학습을 어떻게 돕는지에 대한 개념적 이해를 할 수
있습니다.

배경(Background)
~~~~~~~~~~~~~~~~~~~
신경망(NN; Neural Network)은 어떤 입력 데이터에 대해 실행되는 중첩(nested)된
함수들의 모음(collection)입니다. 이 함수들은 PyTorch에서 Tensor로 저장되는,
(가중치(weight)와 편향(bias)로 구성된) 매개변수들로 정의됩니다.

신경망을 학습하는 것은 2단계로 이루어집니다:

**순전파(Forward Propagation)**: 순전파 단계에서, 신경망은 정답을 맞추기 위해
최선의 추측(best guess)을 합니다. 이렇게 추측을 하기 위해서 입력 데이터를 각
함수들에서 실행합니다.

**역전파(Backward Propagation)**: 역전파 단계에서, 신경망은 추측한 값에서 발생한
오류(error)에 비례하여(proportionate) 매개변수들을 적절히 조절(adjust)합니다.
출력(output)로부터 역방향으로 이동하면서 오류에 대한 함수들의 매개변수들의
미분값( *변화도(gradient)* )을 수집하고, 경사하강법(gradient descent)을 사용하여
매개변수들을 최적화 합니다. 역전파에 대한 자세한 설명은 `3Blue1Brown의 비디오
<https://www.youtube.com/watch?v=tIeHLnjs5U8>`__ 를 참고하세요.

PyTorch에서 사용법
~~~~~~~~~~~~~~~~~~~~

학습 단계를 하나만 살펴보겠습니다. 여기에서는 ``torchvision`` 에서 미리 학습된 resnet18 모델을 불러옵니다.
3채널짜리 높이와 넓이가 64인 이미지 하나를 표현하는 무작위의 데이터 텐서를 생성하고, 이에 상응하는 ``label(정답)`` 을
무작위 값으로 초기화합니다.

```python
import torch, torchvision
model = torchvision.models.resnet18(pretrained=True)
data = torch.rand(1, 3, 64, 64)
labels = torch.rand(1, 1000)
```

다음으로, 입력(input) 데이터를 모델의 각 층(layer)에 통과시켜 예측값(prediction)을 생성해보겠습니다.
이것이 **순전파 단계** 입니다.

```python
prediction = model(data) # 순전파 단계(forward pass)
```

모델의 예측값과 그에 해당하는 정답(label)을 사용하여 오차(error, ``손실(loss)`` )를 계산합니다.
다음 단계는 신경망을 통해 이 에러를 역전파하는 것입니다.
오차 텐서(error tensor)에 ``.backward()`` 를 호출하면 역전파가 시작됩니다.
그 다음 Autograd가 매개변수(parameter)의 ``.grad`` 속성(attribute)에, 모델의
각 매개변수에 대한 변화도(gradient)를 계산하고 저장합니다.

```python
loss = (prediction - labels).sum()
loss.backward() # 역전파 단계(backward pass)
```

다음으로, 옵티마이저(optimizer)를 불러옵니다.
이 예제에서는 학습율(learning rate) 0.1과 모멘텀(momentum) 0.9를 갖는 SGD입니다.
옵티마이저(optimizer)에 모델의 모든 매개변수를 등록합니다.

```python
optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
```

마지막으로 ``.step`` 을 호출하여 경사하강법(gradient descent)을 시작합니다.
옵티마이저는 ``.grad`` 에 저장된 변화도에 따라 각 매개변수를 조정(adjust)합니다.

```python
optim.step() # 경사하강법(gradient descent)
```

지금까지 신경망 학습에 필요한 모든 것을 알아보았습니다.
아래는 autograd의 동작에 대한 세부적인 내용을 설명하므로, 건너뛰어도 됩니다.

--------------

Autograd에서 미분(differentiation)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``autograd`` 가 어떻게 변화도(gradient)를 수집하는지 살펴보겠습니다.
``requires_grad=True`` 를 갖는 2개의 텐서(tensor) ``a`` 와 ``b`` 를 만듭니다.
``requires_grad=True`` 는 ``autograd`` 에 모든 연산(operation)들을 추적해야 한다고 알려줍니다.

```python
import torch

a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
```

이제 ``a`` 와 ``b`` 로부터 새로운 텐서 ``Q`` 를 만듭니다
We create another tensor ``Q`` from ``a`` and ``b``.

\begin{align}Q = 3a^3 - b^2\end{align}

```python
Q = 3*a**3 - b**2
```

이제 ``a`` 와 ``b`` 가 모두 신경망(NN)의 매개변수이고, ``Q`` 가
오차(error)라고 가정해보겠습니다. 신경망을 학습할 때, 아래와 같이
매개변수들에 대한 오차의 변화도(gradient)를 구해야 합니다. 즉,

\begin{align}\frac{\partial Q}{\partial a} = 9a^2\end{align}

\begin{align}\frac{\partial Q}{\partial b} = -2b\end{align}

``Q`` 에 대해서 ``.backward()`` 를 호출할 때, autograd는 이러한 변화도들을 계산하고
이를 각 텐서의 ``.grad`` 속성(attribute)에 저장합니다.

``Q`` 는 벡터(vector)이므로 ``Q.backward()`` 에 ``gradient`` 인자(argument)를 명시적으로
전달해야 합니다. ``gradient`` 는 ``Q`` 와 같은 모양(shape)의 텐서로, ``Q`` 자기 자신에 대한
변화도(gradient)를 나타냅니다. 즉

\begin{align}\frac{dQ}{dQ} = 1\end{align}

동일하게, ``Q.sum().backward()`` 와 같이 ``Q`` 를 스칼라(scalar) 값으로 집계(aggregate)한 뒤 암시적으로
``.backward()`` 를 호출할 수도 있습니다.

```python
external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)
```

이제 변화도는 ``a.grad`` 와 ``b.grad`` 에 저장됩니다.

```python
# 수집된 변화도가 올바른지 확인합니다.
print(9*a**2 == a.grad)
print(-2*b == b.grad)
```

선택적으로 읽기(Optional Reading) - ``autograd`` 를 사용한 벡터 미적분(calculus)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

수학적으로, 벡터 함수 $\vec{y}=f(\vec{x})$ 에서 $\vec{x}$ 에
대한 $\vec{y}$ 의 변화도는 야코비안 행렬(Jacobian Matrix) $J$: 입니다:

\begin{align}J
     =
      \left(\begin{array}{cc}
      \frac{\partial \bf{y}}{\partial x_{1}} &
      ... &
      \frac{\partial \bf{y}}{\partial x_{n}}
      \end{array}\right)
     =
     \left(\begin{array}{ccc}
      \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\
      \vdots & \ddots & \vdots\\
      \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
      \end{array}\right)\end{align}

일반적으로, ``torch.autograd`` 는 벡터-야코비안 곱을 계산하는 엔진입니다. 이는, 주어진
어떤 벡터 $\vec{v}$ 에 대해 $J^{T}\cdot \vec{v}$ 을 연산합니다.

만약 $\vec{v}$ 가 스칼라 함수 $l=g\left(\vec{y}\right)$ 의 변화도(gradient)인 경우:

\begin{align}\vec{v}
   =
   \left(\begin{array}{ccc}\frac{\partial l}{\partial y_{1}} & \cdots & \frac{\partial l}{\partial y_{m}}\end{array}\right)^{T}\end{align}

이며, 연쇄 법칙(chain rule)에 따라, 벡터-야코비안 곱은 $\vec{x}$ 에 대한
$l$ 의 변화도(gradient)가 됩니다:

\begin{align}J^{T}\cdot \vec{v}=\left(\begin{array}{ccc}
      \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{1}}\\
      \vdots & \ddots & \vdots\\
      \frac{\partial y_{1}}{\partial x_{n}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
      \end{array}\right)\left(\begin{array}{c}
      \frac{\partial l}{\partial y_{1}}\\
      \vdots\\
      \frac{\partial l}{\partial y_{m}}
      \end{array}\right)=\left(\begin{array}{c}
      \frac{\partial l}{\partial x_{1}}\\
      \vdots\\
      \frac{\partial l}{\partial x_{n}}
      \end{array}\right)\end{align}

위 예제에서 벡터-야코비안 곱의 이러한 특성을 사용했습니다;
``external_grad`` 가 $\vec{v}$ 를 뜻합니다.

연산 그래프(Computational Graph)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

개념적으로, autograd는 데이터(텐서)의 및 실행된 모든 연산들(및 연산 결과가 새로운 텐서인 경우도 포함하여)의
기록을 `Function <https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function>`__ 객체로
구성된 방향성 비순환 그래프(DAG; Directed Acyclic Graph)에 저장(keep)합니다.
이 방향성 비순환 그래프(DAG)의 잎(leave)은 입력 텐서이고, 뿌리(root)는 결과 텐서입니다.
이 그래프를 뿌리에서부터 잎까지 추적하면 연쇄 법칙(chain rule)에 따라 변화도를 자동으로 계산할 수 있습니다.

순전파 단계에서, autograd는 다음 두 가지 작업을 동시에 수행합니다:

- 요청된 연산을 수행하여 결과 텐서를 계산하고,
- DAG에 연산의 *변화도 기능(gradient function)* 를 유지(maintain)합니다.

역전파 단계는 DAG 뿌리(root)에서 ``.backward()`` 가 호출될 때 시작됩니다. ``autograd`` 는 이 때:

- 각 ``.grad_fn`` 으로부터 변화도를 계산하고,
- 각 텐서의 ``.grad`` 속성에 계산 결과를 쌓고(accumulate),
- 연쇄 법칙을 사용하여, 모든 잎(leaf) 텐서들까지 전파(propagate)합니다.

다음은 지금까지 살펴본 예제의 DAG를 시각적으로 표현한 것입니다. 그래프에서 화살표는 순전파 단계의 방향을 나타냅니다.
노드(node)들은 순전파 단계에서의 각 연산들에 대한 역전파 함수들을 나타냅니다. 파란색 잎(leaf) 노드는 잎 텐서 ``a`` 와 ``b`` 를 나타냅니다.

.. figure:: /_static/img/dag_autograd.png

Note

**PyTorch에서 DAG들은 동적(dynamic)입니다.**
  주목해야 할 중요한 점은 그래프가 처음부터(from scratch) 다시 생성된다는 것입니다; 매번 ``.bachward()`` 가
  호출되고 나면, autograd는 새로운 그래프를 채우기(populate) 시작합니다. 이러한 점 덕분에 모델에서
  흐름 제어(control flow) 구문들을 사용할 수 있게 되는 것입니다; 매번 반복(iteration)할 때마다 필요하면
  모양(shape)이나 크기(size), 연산(operation)을 바꿀 수 있습니다.

DAG에서 제외하기
^^^^^^^^^^^^^^^^^^^^^^

``torch.autograd`` 는 ``requires_grad`` 플래그(flag)가 ``True`` 로 설정된 모든 텐서에 대한
연산들을 추적합니다. 따라서 변화도가 필요하지 않은 텐서들에 대해서는 이 속성을 ``False`` 로 설정하여
DAG 변화도 계산에서 제외합니다.

입력 텐서 중 단 하나라도 ``requres_grad=True`` 를 갖는 경우, 연산의 결과 텐서도 변화도를 갖게 됩니다.

```python
x = torch.rand(5, 5)
y = torch.rand(5, 5)
z = torch.rand((5, 5), requires_grad=True)

a = x + y
print(f"Does `a` require gradients? : {a.requires_grad}")
b = x + z
print(f"Does `b` require gradients?: {b.requires_grad}")
```

신경망에서, 변화도를 계산하지 않는 매개변수를 일반적으로 **고정된 매개변수(frozen parameter)** 이라고 부릅니다.
이러한 매개변수의 변화도가 필요하지 않다는 것을 미리 알고 있으면, 신경망 모델의 일부를 "고정(freeze)"하는 것이 유용합니다.
(이렇게 하면 autograd 연산량을 줄임으로써 성능 상의 이득을 제공합니다.)

DAG에서 제외하는 것이 중요한 또 다른 일반적인 사례(usecase)는
`미리 학습된 모델을 미세조정 <https://tutorials.pytorch.kr/beginner/finetuning_torchvision_models_tutorial.html>`__
하는 경우입니다.

미세조정(finetuning)을 하는 과정에서, 새로운 정답(label)을 예측할 수 있도록 모델의 대부분을 고정한 뒤 일반적으로 분류 계층(classifier layer)만 변경합니다.
이를 설명하기 위해 간단한 예제를 살펴보겠습니다. 이전과 마찬가지로 이미 학습된 resnet18 모델을 불러온 뒤 모든 매개변수를 고정합니다.

```python
from torch import nn, optim

model = torchvision.models.resnet18(pretrained=True)

# 신경망의 모든 매개변수를 고정합니다
for param in model.parameters():
    param.requires_grad = False
```

10개의 정답(label)을 갖는 새로운 데이터셋으로 모델을 미세조정하는 상황을 가정해보겠습니다.
resnet에서 분류기(classifier)는 마지막 선형 계층(linear layer)인 ``model.fc`` 입니다.
이를 새로운 분류기로 동작할 (고정되지 않은) 새로운 선형 계층으로 간단히 대체하겠습니다.

```python
model.fc = nn.Linear(512, 10)
```

이제 ``model.fc`` 를 제외한 모델의 모든 매개변수들이 고정되었습니다.
변화도를 계산하는 유일한 매개변수는 ``model.fc`` 의 가중치(weight)와 편향(bias)뿐입니다.

```python
# 분류기만 최적화합니다.
optimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)
```

옵티마이저(optimizer)에 모든 매개변수를 등록하더라도,
변화도를 계산(하고 경사하강법으로 갱신)할 수 있는 매개변수들은 분류기의 가중치와 편향뿐입니다.

컨텍스트 매니저(context manager)에 `torch.no_grad() <https://pytorch.org/docs/stable/generated/torch.no_grad.html>`__
로도 똑같이 제외하는 기능을 사용할 수 있습니다.

--------------

더 읽어보기:
~~~~~~~~~~~~~~~~~~~

-  `In-place operations & Multithreaded Autograd <https://pytorch.org/docs/stable/notes/autograd.html>`__
-  `Example implementation of reverse-mode autodiff <https://colab.research.google.com/drive/1VpeE6UvEPRz9HmsHh1KS0XxXjYu533EC>`__