# Westeros Tutorial - Introducing reporting

‘Reporting’ is the term used in the MESSAGEix ecosystem to refer to any calculations performed _after_ the MESSAGE mathematical optimization problem has been solved.

This tutorial introduces the reporting features provided by the ``ixmp`` and ``message_ix`` packages.
It was developed by Paul Natsuo Kishimoto ([@khaeru](https://github.com/khaeru)) for a MESSAGEix training workshop held at IIASA in October 2019.
Participants in the MESSAGEix workshops of June and September 2020 contributed feedback.
<!-- Add a line here if you revise the tutorial! -->

**Pre-requisites**
- You have the *MESSAGEix* framework installed and working.
- Complete tutorial Part 1 (``westeros_baseline.ipynb``)
  - Understand the following MESSAGEix terms: ‘variable’, ‘parameter’.
- Open the [‘Reporting’ page in the MESSAGEix documentation](https://docs.messageix.org/en/stable/reporting.html); bookmark it or keep it open in a tab.
  Some text in this tutorial is drawn from that page, and it provides a concise reference for concepts explained below.

## Introduction
### What does ‘reporting’ include?

Individual modelers will make different distinctions between—on one hand—the internals of an optimization model and—on the other—reporting, ‘post-processing’, ‘analysis’, and other tasks.
Doing valid research using models like MESSAGE requires that we understand these differences clearly, as well as how we choose to communicate them.

For example, we might say: “The MESSAGE model shows that total secondary energy (electricity) output in Westeros in the year 720 is 9 GWa.”

But, if we are using the model from ``westeros_baseline.ipynb``:
1. The raw data from the ``Scenario``, after ``.solve()`` has been called, **only** tells us the ``ACT`` variable has certain values.
2. To get the 9 GWa figure, we must:
   1. Compute the product of activity (``ACT``, which is dimensionless) and output efficiency (``output`` in GWa/year), then
   2. Sum across the ``technology`` dimension, and finally
   3. Select the single value for the ``year`` 720.

In this example, steps A, B, and C under #2 are part of ‘reporting’.
Even a intuitive concept like “total secondary energy” is not a *direct* output of the model, but must be reported.

Next, we may want to create a plot of electricity output by year.
Some modelers consider this part of ‘reporting’; for others, ‘reporting’ is complete when the values needed for the plot are written to a file, which they can then use with their favourite plotting tool.

## Reporting features in MESSAGEix

The reporting features in ``ixmp`` and ``message_ix`` are developed to support the complicated reporting and multiple workflows required by the IIASA ECE program for research projects involving large, detailed models such as the [MESSAGEix-GLOBIOM global model](https://docs.messageix.org/global/).
While powerful enough for this purpose, they are also intended to be user-friendly, flexible, and customizable.

The core class used for reporting is ``message_ix.Reporter``, which extends the class ``ixmp.Reporter``.
A reporting workflow has two steps:

1. Describe all computations the Reporter may handle, using ``Reporter.add()`` and other helper methods.
2. Triggering the computation of one or more quantities or reports, using ``Reporter.get()``.

This two-step process allows the Reporter to deliver good performance, by excluding unneeded computations and avoiding re-computing intermediate results that are used in multiple places.

## Concepts: The Graph

``ixmp.Reporter`` is built around a **graph** of *nodes* and *edges*; specifically, a *directed, acyclic graph*.
This means:
- Every edge has a direction; *from* one node *to* another.
- There are no recursive loops in the graph; i.e. no node is its own ancestor.

In the reporting graph, every **node** represents a numerical *calculation*—or, more generally, a *computation* (including other actions like manipulating data formats, writing files, etc.).
The node is labeled with the name of the quantity it produces, which is called a **key**.

A node's computation may depend on certain inputs.
These are represented by the **edges** of the graph.

For example, the following equation:

> $C = A + B$

…is represented by:
- A node named 'A' that provides the raw value of A.
- A node named 'B' that provides the raw value of B.
- A node named 'C' that computes a sum of its inputs.
- An edge from 'A' to 'C', indicating that the value of A is an input to C.
- An edge from 'B' to 'C'.

We use the Reporter to describe this equation (step 1):

```python
from message_ix.reporting import Reporter

# Create a new Reporter object
rep = Reporter()

# Add two nodes
# These have no outputs; they only return a literal value.
rep.add('A', 1)
rep.add('B', 2)

# Add one node and two edges
rep.add('C', (lambda *inputs: sum(inputs), 'A', 'B'))
```

Here is a detailed explanation of what we just did:
- We use the ``add()`` method of a Reporter object to build the graph.
(Remember: you can type ``Reporter.add?`` or ``rep.add?`` in a new cell to use Jupyter's help features; or look at the documentation page linked above.)
- The first argument to ``add()`` is the key of the node.
- The second argument describes the computation to be performed:
  - For nodes ‘A’ and ‘B’, these are simply the raw or literal value to be produced by the node.
  - For node ‘C’, it is a Python tuple with 3 items: ``(lambda *inputs: sum(inputs), 'A', 'B')``.

    Let's break that down further:
    - The first item, ``lambda *inputs: sum(inputs)``, is an anonymous function ([read more](https://doc.python.org/3/tutorial/controlflow.html#lambda-expressions)) that computes the sum of its inputs.
    - All the remaining items, ``'A', B'``, are keys for other nodes in the graph.

Next, let's trigger the calculation of ‘C’ (step 1), which gives the expected value:

```python
rep.get('C')
```

We can use ``Reporter.describe()`` to see steps used in this calculation.
The graph is printed out as a hierarchical list:

```python
print(rep.describe('C'))
```

This description shows how the Reporter traverses the graph in order to calculate the quantity we asked for:

1. The desired value is from node ‘C’, which computes a function of some arguments.
2. The first argument is ‘A’.
3. ‘A’ is the name of another node.
4. Node ‘A’ gives a literal value `int(1)`, which is stored.
5. The Reporter returns to ‘C’ and moves on to the next argument, ‘B’.
6. Steps 3 and 4 are repeated for ‘B’, giving `int(2)`.
7. All of the arguments to ‘C’ have been processed.
8. The computation function for ‘C’ is called.

   As arguments, instead of the (string) keys 'A' and 'B', this function receives the computed `int` values from steps 4 and 6 respectively.
9. The result is returned.

In this example, 'A' and 'B' are, at most, 1 step away from the node we requested, and are each used once.

In more realistic examples, the graph can have:
- Long chains of calculations, each depending on the output of its ancestors, and/or
- Multiple connection, so that 'A' is used by more than one child calculations.

However, the Reporter still follows the same procedure to traverse the graph and calculate the results.

```python
# Store *rep* for the solutions at the bottom of the notebook
rep1 = rep
```

### Exercise 1

Add a node ‘X’ to the graph that returns the literal value 42.

```python
# Add your code here
```

After adding ‘X’, what do you think will be the result when you run the following cell?
Why?

Write down your answer before trying the code.

(Answers and code blocks that solve all exercises are listed at the bottom of the tutorial—don't peek!)

```python
print(rep.describe('C'))
```

### Exercise 2
Extend the `Reporter` to describe the following equation:

> $E = A + D \times \frac{A}{A + B} = A + D \times \frac{A}{C}; \qquad D = 12$

```python
# Some helper functions you can use
def sum_calc(*inputs):
    return sum(inputs)

def product(a, b):
    return a * b

def ratio(a, b):
    return a / b

# Replace 'C' with a reference to sum_calc (instead of an anonymous function)
rep.add('C', (sum_calc, 'A', 'B'))

# Add your code here:
```

## Concepts: Quantities, Keys, and data formats

In the last section, $A$, $B$, and so on were *scalar* variables with a single value.
In energy systems modeling, including with MESSAGE*ix*, we usually deal with scientific **quantities** that are sparse, multi-dimensional arrays with associated units.

That is, they have:
- 1 or more **dimensions**, with *labels* along those dimensions (e.g. specific years; the names of specific technologies);
- *sparse* coverage or “missingness,” i.e. there is not necessarily a value for each combination of labels; and
- associated *units*.

Mathematically, we can say the following:

$$
\begin{align}
A^{ij} & = \left[a_{i,j} \right] \\
i & \in I \\
j & \in J \\
a_{i,j} & \in \left\{ \mathbb{R}, \text{NaN} \right\} \\
a_{i,j} & [=]\, \text{units of X}
\end{align}
$$
…where ‘NaN’ means “not a number,” i.e. a missing value.

(Note: The ``westeros_baseline.ipynb`` distinguishes fixed *parameters*, also called “input data”, from the decision *variables* that the optimization algorithm changes to find the model solution, or “output data”.
While reporting the solution is finished and this distinction is not important; so we refer to everything as a *quantity*.)

### Dimensionality of quantities

Some of the quantities in the MESSAGE mathematical formulation have many dimensions.
For some calculations, we may not care about some of these dimensions.

For instance, $\text{output}$ has ten dimensions: $\text{output}^{n^Lty^Vymnclh^Ah}$.
But we may be interested in the total output in a given year ($y$), but not concerned about different vintages of a technology ($y^V$).
In this case, we don't really want the 10-dimensional quantity—but its **partial sum** over all values of $y^V$.

**Notation.**
Consider a quantity with three dimensions, $A^{ijk}$, and another with two, $B^{kl}$, and a scalar $C$.
We define partial sums over every possible combination of dimensions:

$$
\begin{array}
AA^{ij} = \left[ a_{i,j} \right],
  & a_{i,j} = \sum_{k}{a_{i,j,k}} \ \forall \ i, j
  & \text{similarly } A^{ik}, A^{jk} \\
A^{i} = \left[ a_i \right],
  & a_i = \sum_j\sum_{k}{a_{i,j,k}} \ \forall\  i
  & \text{similarly } A^j, A^k \\
& \qquad A = \sum_i\sum_j\sum_k{a_{i,j,k}}
  & \text{(a scalar)}
\end{array}
$$

Note that $A$ and $B$ share one dimension, $k$, but the other dimensions are distinct.
We specify that simple arithmetic operations result in a quantity whose dimensions are the union of the dimensions of the operands. In other words:

$$
\begin{array}
CC + A^{i} = X^{i} = \left[ x_{i} \right],
  & x_{i} = C + a_{i} \ \forall \ i \\
A^{jk} \times B^{kl} = Y^{jkl} = \left[ y_{j,k,l} \right],
  & y_{j,k,l} = a_{j,k} \times b_{k,l} \ \forall \ j, k, l \\
A^{j} - B^{j} = Z^{j} = \left[ z_{j} \right],
  & z_{j} = a_{j} - b_{j} \ \forall \ j \\
\end{array}
$$

As a result of this rule:
- The difference $Z^j$ has the same dimensionality as *both* of its operands.
- The sum $X^i$ has the same dimensionality as *one* of its operands.
- The product $Y^{jkl}$ has a different dimensonality from each of its operands.

These operations are called **broadcasting** and **alignment**: The scalar value $C$ is *broadcast* across all labels on the dimension $i$ that it lacks, in order to calculate $x_i$.
$A^{jk}$ and $B^{kl}$ are *aligned* on matching values of $k$, but *broadcast* over dimensions $j$ and $l$, respectively.

### Keys

In the first code example, `'C'` was the node label or key that we used to refer to the output of a certain calculation—even before it was been computed.
Likewise, the Python string `'A'` is a key.
When computed, node ‘A’ returns a Python `int(1)`—an object representing its actual *value*.

In step 1 of the reporting workflow, computations are described using *only* keys.
No *values* are created until step 2—and *only* the values needed to provide the result of `get()`.

For multi-dimensional calculations, we need keys that distinguish $A^i$—the partial sum of $A^{ijk}$ used in the calculation of $X^i$—from $A^{jk}$—a *different* partial sum used in the calculation of $Y^{jkl}$.
It is not sufficient to refer to both as `'A'`, since this is ambiguous about what calculation we want to perform.

For this purpose, message_ix (or ixmp) provides the `Key` class.

A Key has a name, zero or more dimensions, and an optional tag:

```python
from message_ix.reporting import Key
# from ixmp.reporting import Key  # Same class

# Quantity named 'a' at two different dimensionalities
k1 = Key('a', ['i'])
k2 = Key('a', ['j', 'k'])

(k1, k2)
```

### Quantity values

To represent the **values** of quantities from a model or produced by reporting calculations, ixmp and message_ix use the `Quantity` class.
Quantity is derived from [`xarray.DataArray`](http://xarray.pydata.org/en/stable/data-structures.html#dataarray)—a labeled, multi-dimensional array, with attributes.

The combination of Key and Quantity lets the Reporter (and you!) handle multi-dimensional data, while automatically handling alignment and broadcasting.

## Automated reporting

A `message_ix.Reporter` for a specific `Scenario` is created using the `.from_scenario()` method.
This method automatically adds many nodes to the graph based on (a) the contents of the Scenario and (b) the known mathematical formulation of MESSAGE.

### Demonstration

```python
from ixmp import Platform
from ixmp.reporting import configure
from message_ix.testing import make_westeros

mp = Platform()
scen = make_westeros(mp, emissions=True, solve=True)
```

```python
# Create a reporter from the existing Scenario
rep = Reporter.from_scenario(scen)

# Reporter uses the Python pint to handle units. '-', used in the Westeros
# tutorial, is not a defined SI unit. We tell the Reporter to replace it with
# '' (unitless) everywhere it appears.
configure(units={'replace': {'-': ''}})
```

What is in `rep`?

```python
len(rep.graph)
```

Almost 8000 nodes!

Remember: `rep` simply *describes* these operations; none of them is executed until or unless you `get()` them.

Let's look at some of the automatically populated content of the graph:

```python
# Return the full-dimensionality Key for the MESSAGE parameter 'output'
output = rep.full_key('output')
output
```

```python
# Return the full-dimensionality Key for the MESSAGE variable 'ACT'
ACT = rep.full_key('ACT')
ACT
```

What would happen if we were to `get()` this key?

```python
print(rep.describe(ACT))
```

We can see:
- The Reporter will call a function named `data_for_quantity()`.

  This (and all built-in computations) are [described in the MESSAGEix documentation](https://docs.messageix.org/en/stable/reporting.html#ixmp.reporting.utils.data_for_quantity).
- The function gets some direct arguments: `'var', 'ACT', 'lvl'`.

  From the documentation, we can see this indicates the level (rather than marginal) of an ixmp `'var'`iable (rather than parameter) named `'ACT'`.
- The next argument is ‘scenario’, another node in the graph.
- This node returns the same Scenario object we passed to `Reporter.from_scenario()`.

In short, if we run this cell, the Reporter will extract a 6-dimensional quantity from the Scenario object and return it.
The other >12,000 nodes will not be computed.

Let's try:

```python
rep.get(ACT)
```

### More automated contents

As mentioned, because `Reporter.from_scenario()` knows that `scen` follows the MESSAGE mathematical formulation, it can automatically populate the graph with useful derived quantities.

For example: the $\text{ACT}$ for various technologies $t$ is a dimensionless quantity.
The specific commodities produced by $t$, with units, are given by the product $\text{ACT} \times \text{output}$.
This product is given the name ‘out’ (the documentation contains [the names for all automatic quantities](https://docs.messageix.org/en/latest/reporting.html#message_ix.reporting.Reporter.from_scenario)):

```python
out = rep.full_key('out')
out
```

```python
# Show what would be done
print(rep.describe(out))
```

```python
rep.get(out)
```

In this case, the mode ($m$), time ($h$) and time_dest ($h^D$) dimensions don't contain useful information.
We also have a single-region model, so we don't need node_loc ($n^L$) or node_dest ($n^L$) either.
We can instead ask for a partial sum.

**Exercise:** review the notation above and satisfy yourself that for $A^{ijk}$, where $i \in I$ and $\|I\| = 1$—that is, when there is only one label along the dimension $I$—then $a_{j,k} = a_{i,j,k} \,\forall\, j, k$. That is, a partial sum over dimension $i$ is the same as ‘dropping’ the dimension $i$.

`Key.drop()` lets us derive its key from the one we already have.
This doesn't perform any calculation; simply returns a new Key with fewer dimensions:

```python
out2 = out.drop('h', 'hd', 'm', 'nd', 'nl')
out2
```

This partial sum is already described in the Reporter:

```python
print(rep.describe(out2))
```

```python
rep.get(out2)
```

### File output

As noted above, the labeled, multi-dimensional Quantity is used so that values passing between reporting calculations are in a consistent, easy-to-manipulate format.

For research purposes, we often want to transform data into other, particular formats or write it to file, in order to feed it into other tools such as existing analysis or plotting codes; both our own, and collaborators'. Reporter provides multiple ways to do this.

For instance, we can `get()` a Quantity and write it directly to a file in a single step:

```python
rep.write(out2, 'output.csv')
```

The file appears in the same directory where we started the Jupyter notebook.

**Exercise:** Try using an .xlsx file name in the above.

We can also define a conversion to a different data format.
This is described in the next section.

## Describing additional computations

The previous section showed how to find and retrieve the results of computations for keys automatically added by `Reporter.from_scenario()`.
Reporter also provides many helper methods to describe additional computations in step 1 of the workflow.

After using these methods, we can continue to describe further calculations using them as input (step 1); or we can `get()` them (step 2).

### Converting to pyam representation

Here, we'll use the [`Reporter.convert_pyam()`](https://docs.messageix.org/en/latest/reporting.html#message_ix.reporting.Reporter.convert_pyam) method.

This adds a node that converts data from a Quantity object to the `IamDataFrame` class from the [`pyam`](https://pyam-iamc.readthedocs.io) package.
`pyam` is built around the [data file format](https://data.ene.iiasa.ac.at/database/) used by the [Integrated Assessment Modeling Consortium](http://www.globalchange.umd.edu/iamc/) (IAMC), and offers plotting and further calculation features.

```python
# The IAMC format does not have 'level', 'technology', or 'commodity'
# columns; only a catch-all 'Variable' column.
def format_variable(df):
    """Callback function to fill the IAMC 'variable' column."""
    df['variable'] = df['l'] + ' energy|' + df['t'] + '|' + df['c']
    return df.drop(['c', 'l', 't'], axis=1)

# Add node(s) that convert data to pyam.IamDataFrame objects
new_key = rep.convert_pyam(
    # Quantity or quantities to convert
    quantities=out.drop('h', 'hd', 'm', 'nd', 'yv'),
    # Dimensions to use for the 'Region' and 'Year' IAMC columns
    rename=dict(nl="region", ya="year"),
    # Use this function to collapse the 'l', 't', and 'c' dimensions
    # into the 'Variable' IAMC column
    collapse=format_variable
)

new_key
```

Note that nothing was computed. (We're still in step 1 of the reporting workflow!)
However, the method did return a new key for the node added to the graph.
This key has the **tag** `':iamc'` added at the end.

We describe the added computation, then execute it to get a `pyam.IamDataFrame`:

```python
print(rep.describe(new_key))
```

```python
df = rep.get(new_key)
df
```

(Note that, unlike a pandas.DataFrame, the contents of a pyam.IamDataFrame are not displayed by default.)

After we have retrieved the `pyam` object, we can use its built-in methods to filter and plot the data:

```python
%matplotlib inline

(
    df.filter(
        model='Westeros Electrified',
        scenario='baseline',
        region='Westeros'
    )
    .plot()
)
```

### Custom computations

Thus far we've described reporting calculations using simple, atomic computations, including those automatically added by `Reporter.from_scenario()`.

However—just as in the first, introductory example—computations are merely Python functions.
This means they can be *any* function, no matter how complex.
Thus, it is easy to insert any existing analysis codes into the graph.

To demonstrate this, we add several nodes, each using a custom function.
- `as_tidy_data()` operates on the internal Quantity value to coerce it into a `pandas.DataFrame` in a specific format.
- `my_plot()` uses a different Python plotting package named [`plotnine`](https://plotnine.readthedocs.io) that implements a “grammar of graphics,” similar to R's `ggplot` package. It returns a plot object without drawing it.
- `save_plot()` saves the plot to file.
- `draw_plot()` outputs the drawn plot directly.

Finally, we define a computation `'do both'` that simply computes two different nodes and returns their outputs in a list.

```python
import plotnine as p9

def as_tidy_data(qty):
    """Convert *qty* to a tidy data frame, as expected by plotnine."""
    return qty.to_series().rename('value').reset_index()

def my_plot(data):
    """Computation that returns a plotnine plot object."""
    # 'Aes'thetic mappings between column names and parts of the plot
    aes = p9.aes(x='ya', y='value', color="t + ' ' + c", shape='l')

    # Set up the plot but don't draw it
    plot = (
        p9.ggplot(data, aes)  # Create the plot
        + p9.geom_line()      # Add a line
        + p9.geom_point()     # Add points
        + p9.labs(            # Label axes & legend
            x='Year',
            y='Energy output',
            color='Tech & commodity',
            shape='Level',
        )
    )

    print('Only computed once.')
    return plot

def save_plot(obj):
    obj.save('westeros_report.pdf', verbose=False)
    return 'Saved to westeros_report.pdf'

def draw_plot(obj):
    obj.draw()
    return 'Drawn in notebook'

# Add nodes to the graph
rep.add('tidy', (as_tidy_data, out2.drop('yv')))
rep.add('plot', (my_plot, 'tidy'))
rep.add('save', (save_plot, 'plot'))
rep.add('draw', (draw_plot, 'plot'))
rep.add('do both', ['save', 'draw'])
```

```python
print(rep.describe('do both'))
```

Note that Reporter avoids calling `my_plot()` repeatedly.
Instead, it stores the resulting object just once.
When the ‘save’ and ‘draw’ nodes are requested, the same object is passed to each of `save_plot()` and `draw_plot()` in turn.

```python
rep.get('do both')
```

In a real-world reporting workflow, a key like `'do both'` could refer to many plots.
The Reporter would compute all the data necessary for these plots, generate, and save them, in a single step.

## Wrapping up

The message_ix reporting code offers other features not covered by this tutorial.
See the [documentation](https://docs.messageix.org/en/stable/reporting.html) to learn how to:
- Add exogenous (non-model) data to be used in other calculations, with `Reporter.add_file()`.
- Use a function to add many nodes at once, with `Reporter.apply()`.
- Generate a visual representation of the graph, with `Reporter.visualize()`.

We would greatly appreciate:
- Reports of your experience using the reporting features in your work, and
- Pull requests to extend the feature set.

## Solutions to exercises

### Exercise 1

The result does not change, because ‘X’ is not needed to calculate ‘C’.

### Exercise 2

One solution involves adding some intermediate nodes—call them ‘foo1’ and ‘foo2’:

```python
# Restore the saved value rep1
rep = rep1

rep.add('D', 12)
rep.add('foo1', (ratio, 'A', 'C'))
rep.add('foo2', (product, 'D', 'foo1'))
rep.add('E', (sum_calc, 'A', 'foo2'))
rep.get('E')
```

Another solution is to define a new anonymous function that computes E in a single step:

```python
rep.add('D', 12)
rep.add('E', (lambda a, c, d: a + d * (a / c), 'A', 'C', 'D'))
rep.get('E')
```


# Problem Set 2, Spring 2020, Villas-Boas

Due <u>Tuesday, February 25 at 9:30am</u>

Submit materials (Jupyter notebook with all code cells run) as one pdf on [Gradescope](https://www.gradescope.com/courses/85265).

# Exercise 1: Food expenditures, food at home, away from home, in US, by region

## Guidelines

This exercise should be completed using R . Remember, when you want output to show in the notebook, you need to *explicitly* call the object in your R code. For example, if I want to show the mean of food at home expenditures per capita (*FAHE*), I can’t just type `MeanFAHE <- mean(FAHE)` because that will just save the output to `MeanFAHE`. Instead, I need to then type `MeanFAHE` on its own so it displays the output. __Answers that do not display any output will be graded as incorrect.__

To write comments in your script (text that will display and will not be read as commands), type a `#` at the beginning of the line you want to be a comment. Use these as notes to keep track of which question you are trying to answer, the purpose of each command, etc.

```R
# Here is an example of a comment in a code cell. Note that running this cell (shift + enter) does not do anything
# because these lines are commented, even if there is a normal command in them. For instance,
# 4 + 4
# or
# library(tidyverse)
```

**To get started:**

* This time you are opening a STATA-formatted ".dta" file, so you will use the `read_dta()` function instead of the import function that was applied to a ".csv" spreadsheet in Problem Set 1. You will need to load the `haven` package to do this. Your command should look like this:

`DataName <- read_dta("file_name")`

* Exercise 1 requires both R work and written answers. Exercise 2 does not require any coding.

**hint:** the function `group_by()` in the `tidyverse` package might be helpful. Section 2 Notes will give some examples of how to use the function.) The function takes two arguments: first the name of the data, second the variable whose values we want to group on: `group_by(data, varname)`. Grouping the data doesn't modify the data itself, instead it changes how the data behaves when we pass it to other `tidyverse` functions. If when we use `summarise()` on our data after using `group_by()`, we will produce separate sets of summary statistics for each level of the grouping variable.

For example, if we were working with the `sleep75` dataset with information on the average time slept per night, $sleep$, and each person's gender, $male$, we could first group on gender by running `sleep75 <- group_by(gender)`. If we then wanted to look at the average sleep per night and number of observations *split by gender*, we could run `summarise(sleep75, avg_sleep = mean(sleep), count = n())` which will yield a table that has two rows of summary statistics: one row for females (`male = 0`) and one row for males (`male =1`)

| male	|	avg_sleep |	 count |
| :----------- | :----------------------- |:------|
| 0 | 7.54 | 306 |
| 1 | 7.67 | 400 |

When you are done with the grouped commands and want to make sure you perform calculations on the entire dataset, you can use the `ungroup()` function to ungroup the data.

## R Tips

* See the Coding Bootcamp Parts 1 and 2 for some basic operations (i.e. counting observations, scatterplots, generating variables). See also all the Lecture R command files to replicate lectures where we did most of what you are asked to do here : e.g., Lecture4.R, Lecture5.R
* The command `table()` lists all values a variable takes in the sample and the number of times it takes each value.
* To summarize data for a specified subset of the observations, you can use `filter()` to subset the data, and then either `summary()` for simple summary statistics or `summarise()` in __tidyverse__ to generate more detailed summary statistics (as we saw in Coding Bootcamp and Problem Set 1).

## Data Description

The data for this exercise come from the Bureau of Labor Statistics (BLS) Consumer Expenditure Survey (CES). With special permission researchers can have access to very dissaggregate data, but anyone can access their representative consumer spending data by U.S. census region (there are nine Census Regions). In this problem set we delve into that data to provide insights into how food spending varies across the nine Census regions and how food is consumed by region (at home or away from home), and then relate food expenditures to regional characteristics.

The *dataPset2.dta* file includes the following variables:

|Variable Name	|	Description |
| :----------- | :----------------------- |
| _region_   | U.S. census region  |
| _ncu_	|	Number of consumer units (in thousands)	|
| _incgross_ | Income before taxes in 2018 |
| _incnet_	| Income after taxes in 2018 |
| _n_	| Number of people per consumer unit |
| _ncars_	| Number of Vehicles in consumer unit in 2018 |
| _exp_	| Average annual expenditures of consumer unit in 2018 Dollars |
| _fahe_	| Food at home expenditures in 2018 |
| _fawaye_	| Food away from home expenditures in 2018 |

## Preamble

Use the below code cell to load all your packages (we'll use functions in both the `haven` and `tidyverse` packages). You can also load your data here.

```R

```

## Question 1

First we would like you to become familiar with your data.

(a) Please detail the following: How many US regions are in the data set? How many regions  have food away from home expenditures larger than food at home expenditures?  What is the average number of people per consumer unit in the data set?   What is the range for the variable _Ncars_ in the data?

```R
# Write your code for part (a) here
```

Write your answer for part (a) here

(b) Construct a variable `totexp_dlr_pc` equal to total expenditures __per capita in USD__ which is equal to total expenditures by consumer unit divided by Number of people in the consumer unit. You will need to create this new variable in R. Plot a histogram of this constructed variable. What is the range of total expenditures per capita in the US in 2018?

```R
# Write your code for part (b) here
```

Write your answer for part (b) here

(c) Calculate the proportion of household expenditures spent on food. You will need to create this new variable. What is the mean? What is the median?

```R
# Write your code for part (c) here
```

Write your answer for part (c) here

(d) Calculate the proportion of household net income spent on food. Note that you need to first create Total Food expenditures in 2018 given the available data (call this new variable $foodExp$), and then with that create the share of net income spent on food. You will need to create this new variable in R. What is the mean? What is the median?

```R
# Write your code for part (d) here
```

Write your answer for part (d) here

## Question 2

Consider the following two models of food expenditures and net income (where $INC$ is the net income variable in our data):

\begin{align}
FAWAYE &= \beta_0 + \beta_1 INC + u~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~(1)
\end{align}

\begin{align}
~~~~~~~~FAHE &= \beta_0 + \beta_1 INC + u ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (2)~~
\end{align}

(a) Estimate the two models in R with the `lm()` command. Interpret your $\hat \beta_1$ coefficients for both models, remembering the triplet **S**(ign), **S**(ignificance), **S**(ize), though you don't need to comment on significance in this problem set.

**Make sure you run all code cells and the corresponding output appears in your notebook before submitting online.**

```R
# Write you code for part (a) here.
```

Write your answer for part (a) here.

(b) How well does total net income predict (i) food at home expenditures, and (ii) food away from home expenditures?

```R
# Write your code for part (b) here.
```

Write your answer for part (b) here.

(c) What are the predicted levels of Food Away From Home expenditures for a consumer unit with total annual  income of 100,000 dollars?

```R
# Write your code for part (c) here.
```

Write your answer for part (c) here.

## Question 3

You already created Total Food expenditures in 2018 ($foodExp$) given the available data. Consider the following model of food expenditures, where *INC* is net income in the data:

\begin{align}
\log(foodExp)= \beta_0 +\beta_1\log(INC) +u ~~~~~~~~~~~~~~(3)
\end{align}

(a) Estimate the model, and interpret your $\hat \beta_1$ coefficient.

```R
# Write your code for part (a) here.
```

Write your answer for part (a) here.

(b) Using the results from estimating equation (3), how would you expect food expenditure to change if total income increases by 25%?

```R
# Write your code for part (b) here.
```

Write your answer for part (b) here.

## Question 4

We will now explore the role of consumer unit size in total food expenditures.

\begin{align}
\log(FoodExp)&=\beta_0 +\beta_1\log(INC) +\beta_2 \log (N) +u  ~~~~(4)
\end{align}

(a) Estimate equation (4), and interpret your $\hat \beta_0$, $\hat \beta_1$ and $\hat \beta_2$ coefficients.

```R
# Write your code for part (a) here.
```

Write your answer for part (a) here.

(b) How did your estimate $\hat\beta_1$ change between equation (3) and equation (4)? Without performing any
calculations, what information does this give you about the correlation between net income and
consumer unit size? (Explain your reasoning in no more than 4 sentences.)

Write your answer for part (b) here.

(c) Predict the expected value of food expenditure for a consumer unit with 3 members and total net
income of \$90,000 using your estimates from equation (4).

```R
# Write your code for part (c) here.
```

Write your answer for part (c) here.

# Exercise 2. Demand for Multivitamins

Many researchers have attempted to estimate important determinants of demand for vitamins. Suppose
a researcher was interested in the effect of price of a bottle of vitamins on quantity purchased of vitamins.
One could estimate such a regression as follows:

\begin{align}
~~~~~~~~~~~~~~~~ Q_j = \beta_0 + \beta_1 Price_j + \beta_2 Education_j + u_{ij}~~~~~~~~~~~~~~~~(1)
\end{align}

where $Q_j$ corresponds to the quantity of vitamins sold in region $j$, $Education_j$ is the level of education in
region $j$, and $Price_j$ is the price of vitamins in region $j$.

(a) What do you expect the sign of $\beta_1$ to be in equation (1)? Why?

Write your answer for part (a) here.

(b) List three other factors that could influence whether the quantity sold of vitamins increases.

Write your answer for part (b) here.

(c) What would happen to $\beta_1$ if you omit education from the estimation? Explain (very briefly) why.

Write your answer for part (c) here.

(d) Give an example of one factor that would induce $\beta_1$ to be biased. State the direction of the bias
and how you determined that direction.

Write your answer for part (d) here.

(e) What are the four conditions that must be satisfied for $\beta_1$ to be unbiased? Explain whether you
believe each assumption is satisfied, and why or why not (suppose we used supermarket data on the quantities and prices of multivitamins across regions to estimate $\beta_1$)

Write your answer for part (e) here.

### Submitting

Save a completed version of your notebook as a pdf by
going to **File > Download As > PDF Via Chrome** in the menu. You will use **PDF Via Chrome** regardless of the web browser you are on).

**Important:** Make sure that all coding cells are run before submission. You can easily do this by going to **Cell > Run All** and then checking to make sure the correct output is displayed.


# Лабораторая работа 4.10
# Поляризация света. Законы Малюса и Брюстера

## Выполнил: Коняхин Всеволод Владимирович, M32051

## Краткие теоретические сведения
Поперечные волны обладают особым, присущим только им, свойством, известным под
названием поляризация.

Если при распространении световой волны направление колебаний электрического вектора
𝐸⃗ бессистемно, хаотически изменяется с равной амплитудой и, следовательно, любое его
направление в плоскости, перпендикулярной распространению волны, равновероятно, то такой свет
называют неполяризованным, или естественным. Если колебания электрического вектора
фиксированы строго в одном направлении, свет называется линейно- или плоско-поляризованным.
В этом случае плоскость, образованная направлением распространения электромагнитной волны и
направлением колебаний вектора напряженности электрического поля, называется плоскостью
поляризации электромагнитной волны.

**Закон Малюса:**

$E_2 = E_1 \cdot \cos \phi$

$I_2 = I_1 \cdot \cos^2 \phi$

Одной из количественных характеристик поляризации является **степень поляризации $P$.**
Для ее определения измеряется интенсивность прошедшего света при вращении поляризатора
вокруг направления светового пучка. Определяются максимальная $I_{max}$ и минимальная $I_{min}$
интенсивности (соответствующие двум ортогональным ориентациям поляризатора) и вычисляется
величина P по формуле:

$P = \frac{I_{max} - I_{min}}{I_{max} + I_{min}}$

**Закон Брюстера:**

Соответствующую зависимость в 1815 г. установил шотландец Дэвид Брюстер. Как
показали опыты, отраженный луч оказывается полностью поляризованым (колебания вектора в нем
перпендикулярны плоскости падения) в случае, когда угол между отраженным и преломленным
лучом равен 90°. Прошедший луч поляризован частично и содержит преимущественно
параллельную составляющую вектора 𝐸⃗ .

$R^{\parallel} = \left( \frac{E_{refl}^{\parallel}}{E_{fal}^{\parallel}} \right)^2 = \frac{\tan^2 (\phi - \psi)}{\tan^2 (\phi + \psi)}$

$R^{\bot} = \left( \frac{E_{refl}^{\bot}}{E_{fal}^{\bot}} \right)^2 = \frac{\sin^2 (\phi - \psi)}{\sin^2 (\phi + \psi)}$

$\frac{n_2}{n_1} = \frac{\sin \phi}{\sin \psi} = \frac{\sin \phi}{\sin(\frac{\pi}{2} - \phi)} = \frac{\sin \phi}{\cos \phi} = \tan \phi$

$\tan \phi_{br} = \frac{n_2}{n_1} = n_{21}$

Соответствующий угол падения называют углом Брюстера.

Степень поляризации преломленной волны при угле падения, равном углу Брюстера,
достигает максимального значения, однако эта волна остается лишь частично поляризованной со
степенью поляризации:

$P = \frac{(n^2 - 1)^2}{2 (n^2 + 1)^2 - (n^2 - 1)^2}$

Для границы воздух–стекло степень поляризации прошедшего света всего 8%. Повысить ее
можно путем ряда последовательных отражений и преломлений. Это осуществляют с помощью так
называемой стопы Столетова, состоящей из нескольких одинаковых и параллельных друг другу
пластинок, установленных под углом Брюстера к падающему свету

## Цель работы
Исследование характера поляризации лазерного излучения и
экспериментальная проверка законов Малюса и Брюстера.

## Рабочие формулы и исходные данные

### Формулы

$P = \frac{I_{max} - I_{min}}{I_{max} + I_{min}}$

$I_{отн} = \frac{I}{I_{max}}$

$K_{\parallel} = \frac{I_{max}}{I_п}$

$K_{\bot} = \frac{I_{min}}{I_п}$

$\tan \phi_{br} = \frac{n_2}{n_1} = n_{21}$

$P = \frac{(n^2 - 1)^2}{2 (n^2 + 1)^2 - (n^2 - 1)^2}$

### Исходные данные

$I_{o} = 1.505$ - относительная интенсивность лазера $I_{п}$, не ослабленная поляризатором

$I^\prime_{o} = 1.505$ - относительная интенсивность источника белого света, не ослабленная поляризатором

$I^\prime = 0.450$ - относительная интенсивность света, прошедшего через поляризатор

$I_{max} = 0.568, I_{min} = 0.448$ - максимальное и минимальное значение интенсивности отраженного луча

$\phi_{br} = 59^{\circ}$ - угол Брюстера

## Схема установки

1.	Верхняя пластина

2.	Стойка с фильтрами

3.	Защитный экран

4.	Поляризатор

5.	Поляризатор (аналог 4)

6.	Блок для измерения угла Брюстера

7.	Анализатор (аналог 4)

8.	Стойка

9.	Вертикальная шкала

10.	Основание установки

11.	Электронный блок

12.	Индикатор измерений блока амперметра-вольтметра

13.	Индикатор режима измерений блока амперметра-вольтметра

14.	Индикатор включенного источника

15.	Регулятор накала белого осветителя

16.	Кнопка переключения режима измерений блока амперметр-вольтметр

17.	Кнопка включения лазера

18.	Ручка установки относительной интенсивности «J/J0»

19.	Кнопка переключения фотоприёмников

20.	Индикатор относительной интенсивности излучения;

21.	Индикатор включенного фотоприёмника

22.	Кнопка “Сеть’

23.	Окно фотоприёмников белого осветителя

24.	Окно фотоприёмника лазерного излучения

```python
import sympy
import scipy
import numpy as np
import pandas as pd
from scipy.signal import argrelextrema
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (10,5)
%matplotlib inline
```

## Результаты измерений и расчеты
**Упражнение 1. Проверка Закона Малюса**

$I_{o} = 1.505$ - относительная интенсивность лазера $I_{п}$, не ослабленная поляризатором

$I^\prime_{o} = 1.505$ - относительная интенсивность источника белого света, не ослабленная поляризатором

$I^\prime = 0.450$ - относительная интенсивность света, прошедшего через поляризатор

**Излучение лазера**

```python
I_o = 1.505
I_o_prime = 1.505
I_prime = 0.450

lazer_df = pd.DataFrame({
    '$\phi$, градусы': [i * 10 for i in range(16)],
    '$I_1$': [0.826, 0.776, 0.671, 0.604, 0.469, 0.321, 0.206, 0.107,
              0.036, 0.003, 0.018, 0.069, 0.176, 0.274, 0.405, 0.495],
    '$I_2$': [0.835, 0.768, 0.703, 0.6, 0.479, 0.316, 0.192, 0.11,
              0.029, 0.004, 0.011, 0.076, 0.145, 0.263, 0.353, 0.513],
})

lazer_df['$I_{mean}$'] = (lazer_df['$I_1$'] + lazer_df['$I_2$']) / 2

lazer_df
```

      $\phi$, градусы
      $I_1$
      $I_2$
      $I_{mean}$

      0
      0
      0.826
      0.835
      0.8305

      1
      10
      0.776
      0.768
      0.7720

      2
      20
      0.671
      0.703
      0.6870

      3
      30
      0.604
      0.600
      0.6020

      4
      40
      0.469
      0.479
      0.4740

      5
      50
      0.321
      0.316
      0.3185

      6
      60
      0.206
      0.192
      0.1990

      7
      70
      0.107
      0.110
      0.1085

      8
      80
      0.036
      0.029
      0.0325

      9
      90
      0.003
      0.004
      0.0035

      10
      100
      0.018
      0.011
      0.0145

      11
      110
      0.069
      0.076
      0.0725

      12
      120
      0.176
      0.145
      0.1605

      13
      130
      0.274
      0.263
      0.2685

      14
      140
      0.405
      0.353
      0.3790

      15
      150
      0.495
      0.513
      0.5040

гду $\alpha$ - угол поворота анализатора, $I_1$ и $I_2$ - интенсивности луча, прошедшего
через анализатор для двух экспериментов.
$I_{mean}$ - среднее значение двух экспериментов.

```python
def polarization_degree(i_max, i_min):
    degree = (i_max - i_min) / (i_max + i_min)
    return degree
```

```python
print('Степень поляризации лазерного излучения: {:.3f}'.format(polarization_degree(
    max(lazer_df['$I_{mean}$']), min(lazer_df['$I_{mean}$']))))
```

    Степень поляризации лазерного излучения: 0.992

**Найдем максимальную интенсивнность, чтобы в дальнейшем получить относителные интенсивности.**

```python
I_max = max(max(lazer_df['$I_1$']), max(lazer_df['$I_2$']))
print('I max: {}'.format(I_max))

I_min = min(min(lazer_df['$I_1$']), min(lazer_df['$I_2$']))
print('I min: {}'.format(I_min))

phi_max = 0
print('Phi max: {}'.format(phi_max))
```

    I max: 0.835
    I min: 0.003
    Phi max: 0

**Построим таблицу с относительными значениями:**

```python
lazer_rel_df = pd.DataFrame({
    '$\phi$, градусы': [i * 10 for i in range(16)]
})

lazer_rel_df['$I_{rel1}$'] = lazer_df['$I_1$'] / I_max
lazer_rel_df['$I_{rel2}$'] = lazer_df['$I_2$'] / I_max
lazer_rel_df['$I_{rel}$'] = (lazer_rel_df['$I_{rel1}$'] + lazer_rel_df['$I_{rel2}$']) / 2

lazer_rel_df
```

      $\phi$, градусы
      $I_{rel1}$
      $I_{rel2}$
      $I_{rel}$

      0
      0
      0.989222
      1.000000
      0.994611

      1
      10
      0.929341
      0.919760
      0.924551

      2
      20
      0.803593
      0.841916
      0.822754

      3
      30
      0.723353
      0.718563
      0.720958

      4
      40
      0.561677
      0.573653
      0.567665

      5
      50
      0.384431
      0.378443
      0.381437

      6
      60
      0.246707
      0.229940
      0.238323

      7
      70
      0.128144
      0.131737
      0.129940

      8
      80
      0.043114
      0.034731
      0.038922

      9
      90
      0.003593
      0.004790
      0.004192

      10
      100
      0.021557
      0.013174
      0.017365

      11
      110
      0.082635
      0.091018
      0.086826

      12
      120
      0.210778
      0.173653
      0.192216

      13
      130
      0.328144
      0.314970
      0.321557

      14
      140
      0.485030
      0.422754
      0.453892

      15
      150
      0.592814
      0.614371
      0.603593

**Построение графика зависимости нормированной интенсивности $I_{отн}$ от угла $\phi$ поворота
поляроида в полярных координатах; графика зависимости $\cos^2(\phi - \phi_m)$ от угла $\phi$ поворота поляроида**

```python
fig, ax = plt.subplots(figsize=(20, 5))
ax.set_title('Зависимость нормированной интенсивности от угла поворота')

ax.scatter(lazer_rel_df['$\phi$, градусы'], lazer_rel_df['$I_{rel}$'], c='r')
ax.plot(lazer_rel_df['$\phi$, градусы'], lazer_rel_df['$I_{rel}$'], 'r--', label='$I_{отн}$')

ax.scatter(lazer_rel_df['$\phi$, градусы'], (np.cos((lazer_rel_df['$\phi$, градусы'] - phi_max)* np.pi / 180)) ** 2, c='b')
ax.plot(lazer_rel_df['$\phi$, градусы'], (np.cos((lazer_rel_df['$\phi$, градусы'] - phi_max)* np.pi / 180)) ** 2, 'b--', label='$\cos^2(\phi - \phi_m)$')

ax.legend()

plt.show()
```

**Исходя из графика выше, поскольку относительная интенсивность лазерного излучения очень похожа на график  $\cos^2(\phi - \phi_m)$, можно утверждать, что лазерное излучение обладает линейным видом поляризации**

**Найдем коэффициенты пропускания использованного поляроида для параллельной и
перпендикулярной ориентации его плоскости пропускания**

```python
K_parallel = I_max / I_o
K_bot = I_min / I_o

print('Коэффициенты: K parallel: {:.5f}, K_bot: {:.5f}'.format(K_parallel, K_bot))
```

    Коэффициенты: K parallel: 0.55482, K_bot: 0.00199

**Излучение белого света**

```python
light_df = pd.DataFrame({
    '$\phi$, градусы': [i * 10 for i in range(16)],
    '$I_1$': [0.45, 0.465, 0.478, 0.493, 0.486, 0.56, 0.49, 0.485,
              0.483, 0.475, 0.473, 0.483, 0.49, 0.51, 0.522, 0.528],
    '$I_2$': [0.491, 0.48, 0.48, 0.482, 0.486, 0.465, 0.475, 0.466,
              0.464, 0.46, 0.468, 0.456, 0.474, 0.491, 0.514, 0.503],
})

light_df['$I_{mean}$'] = (light_df['$I_1$'] + light_df['$I_2$']) / 2

light_df
```

      $\phi$, градусы
      $I_1$
      $I_2$
      $I_{mean}$

      0
      0
      0.450
      0.491
      0.4705

      1
      10
      0.465
      0.480
      0.4725

      2
      20
      0.478
      0.480
      0.4790

      3
      30
      0.493
      0.482
      0.4875

      4
      40
      0.486
      0.486
      0.4860

      5
      50
      0.560
      0.465
      0.5125

      6
      60
      0.490
      0.475
      0.4825

      7
      70
      0.485
      0.466
      0.4755

      8
      80
      0.483
      0.464
      0.4735

      9
      90
      0.475
      0.460
      0.4675

      10
      100
      0.473
      0.468
      0.4705

      11
      110
      0.483
      0.456
      0.4695

      12
      120
      0.490
      0.474
      0.4820

      13
      130
      0.510
      0.491
      0.5005

      14
      140
      0.522
      0.514
      0.5180

      15
      150
      0.528
      0.503
      0.5155

гду $\alpha$ - угол поворота анализатора, $I_1$ и $I_2$ - интенсивности луча белого света, прошедшего через анализатор для двух экспериментов. $I_{mean}$ - среднее значение двух экспериментов.

```python
print('Степень поляризации излучения белого света: {:.3f}'.format(polarization_degree(
    max(light_df['$I_{mean}$']), min(light_df['$I_{mean}$']))))
```

    Степень поляризации излучения белого света: 0.051

**Построение зависимости $\frac{I}{I^{\prime}}(\phi)$ для белого света**

```python
fig, ax = plt.subplots(figsize=(20, 5))
ax.set_title('Зависимость нормированной интенсивности от угла поворота')

ax.scatter(lazer_rel_df['$\phi$, градусы'], lazer_rel_df['$I_{rel}$'], c='r')
ax.plot(lazer_rel_df['$\phi$, градусы'], lazer_rel_df['$I_{rel}$'], 'r--', label='$I_{отн}$')

ax.scatter(light_df['$\phi$, градусы'], light_df['$I_{mean}$'] / I_prime, c='g')
ax.plot(light_df['$\phi$, градусы'], light_df['$I_{mean}$'] / I_prime, 'g--', label='$I / I^{\prime} (\phi)$')

ax.scatter(lazer_rel_df['$\phi$, градусы'], (np.cos((lazer_rel_df['$\phi$, градусы'] - phi_max)* np.pi / 180)) ** 2, c='b')
ax.plot(lazer_rel_df['$\phi$, градусы'], (np.cos((lazer_rel_df['$\phi$, градусы'] - phi_max)* np.pi / 180)) ** 2, 'b--', label='$\cos^2(\phi - \phi_m)$')

ax.legend()

plt.show()
```

**Упражнение 2. Проверка Закона Брюстера**

$I_{max} = 0.568, I_{min} = 0.448$ - максимальное и минимальное значение интенсивности отраженного луча

$\phi_{br} = 59^{\circ}$ - угол Брюстера

**Экспериментальные значения:**

```python
df2 = pd.DataFrame({
    '$\phi$, градусы': [i for i in range(30, 65, 2)],
    '$I_1$': [1.076, 1.073, 1.071, 1.066, 1.063, 1.061, 1.059, 1.055, 1.051,
              1.046, 1.038, 1.028, 1.018, 1.006, 0.988, 0.972, 0.955, 0.926],
    '$I_2$': [1.069, 1.063, 1.061, 1.058, 1.057, 1.054, 1.054, 1.051, 1.041,
          1.043, 1.04, 1.028, 1.014, 0.999, 0.986, 0.976, 0.958, 0.926]
})

phi_br = 59
i_max = 0.568
i_min = 0.448

df2
```

      $\phi$, градусы
      $I_1$
      $I_2$

      0
      30
      1.076
      1.069

      1
      32
      1.073
      1.063

      2
      34
      1.071
      1.061

      3
      36
      1.066
      1.058

      4
      38
      1.063
      1.057

      5
      40
      1.061
      1.054

      6
      42
      1.059
      1.054

      7
      44
      1.055
      1.051

      8
      46
      1.051
      1.041

      9
      48
      1.046
      1.043

      10
      50
      1.038
      1.040

      11
      52
      1.028
      1.028

      12
      54
      1.018
      1.014

      13
      56
      1.006
      0.999

      14
      58
      0.988
      0.986

      15
      60
      0.972
      0.976

      16
      62
      0.955
      0.958

      17
      64
      0.926
      0.926

гду $\phi$ - угол наклона стеклянной пластинки, $I_1$ и $I_2$ - интенсивность прошедшего света в прямом и обратном направлениях, соответственно.

```python
n_21 = np.tan(phi_br * np.pi / 180)
print('Показатель преломления второй среды относительно первой: {:.3f}'.format(n_21))
```

    Показатель преломления второй среды относительно первой: 1.664

```python
p1 = polarization_degree(max(max(df2['$I_1$']), max(df2['$I_2$'])), min(min(df2['$I_1$']), min(df2['$I_2$'])))
print('Cтепень поляризации света после прохождения его через пластинку: {:.3f}'.format(p1))
```

    Cтепень поляризации света после прохождения его через пластинку: 0.075

```python
def partial_polarization(n):
    degree = (n ** 2 - 1) ** 2 / (2 * (n ** 2 + 1) ** 2 - (n ** 2 - 1) ** 2)
    return degree
```

```python
print('Cтепень поляризации света после прохождения его через пластинку теоретическая: {:.3f}'.format(partial_polarization(n_21)))
```

    Cтепень поляризации света после прохождения его через пластинку теоретическая: 0.124

```python
p3 = polarization_degree(i_max, i_min)
print('Cтепень поляризации белого света после прохождения его через пластинку: {:.3f}'.format(p3))
```

    Cтепень поляризации белого света после прохождения его через пластинку: 0.118

## Выводы и анализ результатов работы
В данной лабораторной работе были рассмотрены поляризация света, а также законы Малюса и Брюстера.

В первой части работы было предложено проверить закон Малюса для лазера и белого света. Поскольку графики квадрата косинуса и нормированная интенсивность лазерного излучения совпали, то можно утверждать, что лазерное излучение обладает линейной поляризацией. Были найдены коэффициенты пропускания использованного поляроида для параллельной и перпендикулярной ориентации его плоскости пропускания. Также было рассмотрено и излучение белого света: его степень поляризованности получилась меньше, поскольку изначально свет неполяризован; однако и для белого света верно, что пройдя через поляризатор, он стал линейно поляризован.

Во второй части работы был вычислен показатель преломления среды по найденному углу Брюстера $n_{21} = 1.664, \phi_{br} = 59^{\circ}$. Были получены три числа для степени поляризации белого света через пластинкку: два экспериментальных и одно теоретическое. Эти три числа соотносятся друг с другом в пределах погрешности.


```python
%matplotlib inline  #It will allow visualization created in matplotlib inside the notebook itsels.
```

```python
import matplotlib.pyplot as plt
plt.plot(range(100))
```

```python
%time x = range(10000)  #It measure execution time
```

    CPU times: user 6 µs, sys: 6 µs, total: 12 µs
    Wall time: 13.8 µs

```python
%timeit x = range(10000)   #timeit function measure time for every loop and returns the average at the end.
max(x)
```

    226 ns ± 2.94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

    9999

#### writefile will create a file and write the given content on it. If file already exists then it will overwrite.

```python
%%writefile test.txt
this is written from a jupyter
%notebook
```

    Writing test.txt

```python
%ls
```

    MyFirstNotebook.ipynb        machinelearning/
    Untitled.ipynb               test.txt
    juypyterMagicFunction.ipynb

#### html will let you insert images and videso to the notebook.

```python
%%HTML
<i> Image in jupyter notebook </i>
</img>
```

 Image in jupyter notebook

```latex
%%latex       #this lets you write formulas in the standard format
\begin{align}
Gradient : \nabla J = - 2H^T (Y-HW)
\end{align}
```

\begin{align}
Gradient : \nabla J = - 2H^T (Y-HW)
\end{align}

##### load_ext  this can be used to load other extension, we can use it to plug the sql code in this notebook.

```python
!pip install ipython-sql       #install the sql
```

    Collecting ipython-sql
      Downloading https://files.pythonhosted.org/packages/ab/df/427e7cf05ffc67e78672ad57dce2436c1e825129033effe6fcaf804d0c60/ipython_sql-0.3.9-py2.py3-none-any.whl
    Requirement already satisfied: ipython>=1.0 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython-sql) (7.8.0)
    Requirement already satisfied: ipython-genutils>=0.1.0 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython-sql) (0.2.0)
    Requirement already satisfied: six in /Applications/anaconda3/lib/python3.7/site-packages (from ipython-sql) (1.12.0)
    Collecting prettytable (from ipython-sql)
      Downloading https://files.pythonhosted.org/packages/ef/30/4b0746848746ed5941f052479e7c23d2b56d174b82f4fd34a25e389831f5/prettytable-0.7.2.tar.bz2
    Requirement already satisfied: sqlalchemy>=0.6.7 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython-sql) (1.3.9)
    Collecting sqlparse (from ipython-sql)
      Downloading https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
    Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (2.0.10)
    Requirement already satisfied: pickleshare in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (0.7.5)
    Requirement already satisfied: jedi>=0.10 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (0.15.1)
    Requirement already satisfied: traitlets>=4.2 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (4.3.3)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (4.7.0)
    Requirement already satisfied: decorator in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (4.4.0)
    Requirement already satisfied: appnope; sys_platform == "darwin" in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (0.1.0)
    Requirement already satisfied: backcall in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (0.1.0)
    Requirement already satisfied: pygments in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (2.4.2)
    Requirement already satisfied: setuptools>=18.5 in /Applications/anaconda3/lib/python3.7/site-packages (from ipython>=1.0->ipython-sql) (41.4.0)
    Requirement already satisfied: wcwidth in /Applications/anaconda3/lib/python3.7/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=1.0->ipython-sql) (0.1.7)
    Requirement already satisfied: parso>=0.5.0 in /Applications/anaconda3/lib/python3.7/site-packages (from jedi>=0.10->ipython>=1.0->ipython-sql) (0.5.1)
    Requirement already satisfied: ptyprocess>=0.5 in /Applications/anaconda3/lib/python3.7/site-packages (from pexpect; sys_platform != "win32"->ipython>=1.0->ipython-sql) (0.6.0)
    Building wheels for collected packages: prettytable
      Building wheel for prettytable (setup.py) ... done
      Created wheel for prettytable: filename=prettytable-0.7.2-cp37-none-any.whl size=13700 sha256=d6cb5a7706a7cf36bdc687bf18c854221a0461fcd983d0526efa9b992252312b
      Stored in directory: /Users/neerajsharma/Library/Caches/pip/wheels/80/34/1c/3967380d9676d162cb59513bd9dc862d0584e045a162095606
    Successfully built prettytable
    Installing collected packages: prettytable, sqlparse, ipython-sql
    Successfully installed ipython-sql-0.3.9 prettytable-0.7.2 sqlparse-0.3.0

```python
%load_ext sql
```

    The sql extension is already loaded. To reload it, use:
      %reload_ext sql

##### sql sqlite: is used to connect to any database

```python
%sql sqlite://
```

    'Connected: @None'

```sql
%%sql
CREATE TABLE classroom(name, age, totalmarks);
INSERT INTO classroom VALUES ('Neeraj', 25, 99);
INSERT INTO classroom VALUES ('SWATI', 22, 100);
```

     * sqlite://
    Done.
    1 rows affected.
    1 rows affected.

    []

```python
%sql SELECT * from classroom;
```

     * sqlite://
    Done.

        name
        age
        totalmarks

        Neeraj
        25
        99

        SWATI
        22
        100


Paul Maria Scheikl - Health Robotics and Automation (HERA) - Karlsruhe Institute of Technology (KIT)

# Real World Example: Classifying Pigmented Lesions
To build an example that is more relevant to the medical field, we will now build a complete training pipeline for classifying pigment leasions in the [Skin Cancer MNIST dataset](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000).

The dataset consists of 10015 images that show cropped images of pigmented lesions and are annotated with a diagnostic case label.

"*Cases include a representative collection of all important diagnostic categories in the realm of pigmented lesions: Actinic keratoses and intraepithelial carcinoma / Bowen's disease (akiec), basal cell carcinoma (bcc), benign keratosis-like lesions (solar lentigines / seborrheic keratoses and lichen-planus like keratoses, bkl), dermatofibroma (df), melanoma (mel), melanocytic nevi (nv) and vascular lesions (angiomas, angiokeratomas, pyogenic granulomas and hemorrhage, vasc).*" [\[ref\]](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000)

## Data preparation
We have already prepared a function that reads the image file paths from a folder, as well as the labels that are stored in the csv file. The function returns a [DataLoader](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) which acts as in iterator, so we can use it to generate mini batches of data for our training loop. The images used for training are further scaled down to half resolution.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

from livelossplot import PlotLosses
from pathlib import Path
from tqdm.notebook import tqdm
from copy import deepcopy
from matplotlib import rcParams
from torchsummary import summary

from utils.data import load_dataset
from utils.data import CaseTypes

image_dir = Path("data") / Path("skin") / Path("images")
metadata_csv = Path("data") / Path("skin") / Path("metadata.csv")

# Network training using the GPU -> cuda or CPU -> cpu
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Whether to save the trained model as a file
save_model = False
model_save_path = Path("models") / Path("skin_cancer_prediction_model.pkl")
```

In contrast to our simple sine function example, we will not hold the complete data for training in memory and predict the model outputs of the complete data (called batch gradient descent).
In this case we will use mini batch gradient descent and calculate the gradients on mini batches of size *32* (one input to the model consists of 32 examples).
The gradient with regard to our loss function will be not as accurate, compared to using the complete dataset, but computation will be much faster.

Also note, that we split up the data into three separate datasets, that do not share data points.
In our sine example, we wanted the model to match the data as best as possible, basically memorizing the data.
In this case, however, we do not want the model to memorize the datapoints, otherwise we could not use it on new data. This problem of memorizing data is called [**overfitting**](https://en.wikipedia.org/wiki/Overfitting) and happens when the model has so much capacity, that it begins to interpolate the data as training goes on, rather than learning the underlying distribution of the data.

[ref]

We address this problem by reserving a certain amount of data for training, another set of data points for evaluating how well the model generalizes on unseen data to stop the training before overfitting occurs, and finally the rest of the data to test the final performance of the model.

* **Training Dataset**: The sample of data used to fit the model.
* **Validation Dataset**: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters.
* **Test Dataset**: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset. [ref](https://machinelearningmastery.com/difference-test-validation-datasets/)

Question: Why do we need separate evaluation and testing data?

Because the evaluation results of the validation dataset will be biased towards our hyperparameter choices, because we used this data to select them.

## Data Normalization
Another step to prepare our image dataset for the machine learning part is normalizing the data.

*Data normalization is an important step which ensures that each input parameter (pixel, in this case) has a similar data distribution. This makes convergence faster while training the network. Data normalization is done by subtracting the mean from each pixel and then dividing the result by the standard deviation. The distribution of such data would resemble a Gaussian curve centered at zero.* [ref](https://becominghuman.ai/image-data-pre-processing-for-neural-networks-498289068258)

We could calculate the mean and standard deviation on a single image, but this would lead to very different data distributions on a per image basis. Imaging our dataset contains images that are either completely black or completely white. Normalizing each image independently will lead to images that are all the same.
This is why we have to calculate our values for mean and standard deviation of the complete dataset.
For RGB images, it is common to calculate mean and standard deviation for each color channel independently, so we will get three mean values and three standard deviations.

First, lets look at some of the images in the dataset.

```python
# create a data loader
image_shape = (225, 300)
data_loader = load_dataset(
    image_dir,
    metadata_csv,
    batch_size=100,
    resize_image_shape=image_shape,
    do_shuffle=False,
    single_loader=True,
    num_workers=8,
)

indices = np.random.choice(list(range(len(data_loader.dataset))), 8)
images = [np.moveaxis(deepcopy(data_loader.dataset[i][0].numpy()), 0, -1) for i in indices]
labels = [data_loader.dataset[i][1] for i in indices]
titles = []
for label in labels:
    for case in CaseTypes:
        if case.id == label:
            titles.append(case.description)
        else:
            pass

%matplotlib inline

# figure size in inches optional
rcParams['figure.figsize'] = 18 ,10

# display images
fig, axes = plt.subplots(4,2)
axes = axes.flatten()
for i in range(8):
    axes[i].imshow(images[i]);
    axes[i].set_title(titles[i], wrap=True)
    axes[i].set_xticks([])
    axes[i].set_yticks([])
```

As you can see, there is little contrast in the images and they are all in all mostly red.

```python
# placeholders for temporary values we need to calculate mead and standard deviation
pixel_sum = torch.tensor([0.0, 0.0, 0.0])
pixel_sum_squared = torch.tensor([0.0, 0.0, 0.0])

# loop through the image batches in the data loader
for image_batch, _ in tqdm(data_loader, desc="Calculating..."):
    pixel_sum += image_batch.sum(axis=[0, 2, 3])
    pixel_sum_squared += (image_batch ** 2).sum(axis=[0, 2, 3])

# number of pixel in the dataset
num_images = len(data_loader.dataset)
count = num_images * image_shape[0] * image_shape[1]

# mean and std
mean = pixel_sum / count
var = (pixel_sum_squared / count) - (mean ** 2)
std = torch.sqrt(var)

print(f"Mean: {mean.numpy()}\nStandard Deviation: {std.numpy()}")
```

    Calculating...:   0%|          | 0/101 [00:00<?, ?it/s]

    Mean: [0.7635213  0.546128   0.57053053]
    Standard Deviation: [0.1408046  0.15247564 0.1698052 ]

We can see, that the images are indeed leaning towards the red colors (first value of the mean), and the standard deviations are very small (little contrast) compared to the normal Gaussian distribution with a standard deviation of 1 that we are aiming for.

We can now load the dataset with the normalization parameters into the three different splits for training, validating, and testing.

```python
train_data_loader, val_data_loader, test_data_loader = load_dataset(
    image_dir,
    metadata_csv,
    batch_size=32,
    resize_image_shape=image_shape,
    normalize_images=True,
    normalize_mean=mean.numpy(),
    normalize_std=std.numpy(),
    do_shuffle=True,
    split_ratios=(0.7, 0.15, 0.15),
    single_loader=False,
    num_workers=8
)
```

    [HERA INFO]: Created skin cancer dataset from image paths and labels for 10015 data points.
    [HERA INFO]: Target key is dx for targets of type <enum 'CaseTypes'>.
    [HERA INFO]: Images will be normalized with:
     mean: [0.7635213  0.546128   0.57053053] and
     std: [0.1408046  0.15247564 0.1698052 ].
    [HERA INFO]: Created three DataLoaders from DataSet with batch size 32.
    [HERA INFO]: There are 7010, 1502, and 1503 data points for training, validating, and testing, respectively.
    [HERA INFO]: The image shape is (225, 300)
    [HERA INFO]: Datapoints will be shuffled after a full pass.

## Convolutional Neural Network
As we have seen in the introduction, MLPs are not suited to work on images.
We will use a CNN as the model to classify the images.
For this, we will create a new class called CNN, based on a pytorch nn.Module, that consists of 4 convolutional layers, each with ReLU activation functions, a max pooling layer to reduce the feature size without introducing additional learnable parameters, and finally two additional fully connected layers.
The output of the last layer will be passed to a LogSoftmax function that generates log-likelihoods for each of the seven possible cases labels in the dataset.
$$
\text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right)
$$

```python
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, 3, 2)
        self.conv2 = nn.Conv2d(16, 32, 3, 2)
        self.conv3 = nn.Conv2d(32, 64, 3, 2)
        self.conv4 = nn.Conv2d(64, 128, 3, 2)
        self.fc1 = nn.Linear(6144, 128)
        self.fc2 = nn.Linear(128, 7)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

cnn_model = CNN().to(device)
summary(cnn_model, (3, 225, 300))
```

    ----------------------------------------------------------------
            Layer (type)               Output Shape         Param #
    ================================================================
                Conv2d-1         [-1, 16, 112, 149]             448
                Conv2d-2           [-1, 32, 55, 74]           4,640
                Conv2d-3           [-1, 64, 27, 36]          18,496
                Conv2d-4          [-1, 128, 13, 17]          73,856
                Linear-5                  [-1, 128]         786,560
                Linear-6                    [-1, 7]             903
    ================================================================
    Total params: 884,903
    Trainable params: 884,903
    Non-trainable params: 0
    ----------------------------------------------------------------
    Input size (MB): 0.77
    Forward/backward pass size (MB): 3.72
    Params size (MB): 3.38
    Estimated Total Size (MB): 7.87
    ----------------------------------------------------------------

As you can see, the majority of trainable parameters is part of the fully connected layers.
We could prevent this by introducing more convolutional layers or pooling layers to reduce the dimensionality.
For now, this will do.

## Hyperparameters for the Training
The training loop for this example will also be a bit more complex that the initial sine example.

### Loss Function
The output of our network is a list with seven elements.
The value of element n represents the log-likelihood of the image belonging to class n.
So if the image belongs to class 3 (beginning from 0), we want the neural network's output list to have its maximum value at index 3.
The loss function that quantifies the distance between the output of the neural network and the ground truth list of log-likelihoods in a classification problem with multiple classes is the negative log-likelihood loss (NLLLOSS).

Negative log-likelihood loss is the same as cross entropy loss (aka logistic loss or multinomial logistic loss), with the difference, that our model already outputs the log probalities for each class.

$$
\text{CrossEntropyLoss}(x, label) = -\log\left(\frac{\exp(x[label])}{\sum_j \exp(x[j])}\right)
$$

So the negative log-likelihood loss reduces to
$$
\text{NLLLOSS}(prediction, label) = -prediction[label]
$$
where prediction is the list of log-likelihoods, produced by the neural network.

```python
loss_function = nn.NLLLoss().to(device)
```

```python
with torch.no_grad():
    images, labels = next(iter(data_loader))
    prediction = cnn_model(images.to(device))
    print("Log-likelihoods for one image: ", prediction[0].cpu())
    print("Ground truth label: ", labels[0])
    print("NLLLoss: ", loss_function(prediction[0].unsqueeze(0), labels[0].unsqueeze(0).to(device)).cpu())
```

    Log-likelihoods for one image:  tensor([-1.9646, -1.8996, -1.9267, -1.9468, -1.9996, -1.9540, -1.9331])
    Ground truth label:  tensor(4)
    NLLLoss:  tensor(1.9996)

### Optimizer
At this point, we have a neural network and a loss function.
The next step is defining an optimizer that adapts the neural network parameters to minimize the loss over our training data.

In the last notebook, we used the SGD optimizer. In this notebook, we will use a more sofisticated optimizer: Adaptive Moment Estimation (Adam).
In contrast to SGD, Adam *remembers* previous gradients for each parameter, and adapts the parameter specific learning rates accordingly.

The update Rule from SGD
$$
\begin{align}
\begin{split}
\theta_{t+1} = \theta_{t}-\gamma\nabla L(\theta_{t})
 = \theta_{t}-\gamma g_{t}
\end{split}
\end{align}
$$

changes to Adam's update rule
$$
\begin{align}
\begin{split}
\theta_{t+1} = \theta_{t} - \dfrac{\gamma}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
\end{split}
\end{align}
$$

$\hat{m}_t$ and $\hat{v}_t$ are bias-corrected first and second moment estimates that scale the learning rate.

The bias-corrected first and second moment are estimated through

$$
\begin{align}
\begin{split}
\hat{m}_t &= \dfrac{m_t}{1 - \beta^t_1} \\
\hat{v}_t &= \dfrac{v_t}{1 - \beta^t_2} \end{split}
\end{align}
$$

with

$$
\begin{align}
\begin{split}
m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t \\
v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
\end{split}
\end{align}
$$

$\beta_1$, $\beta_2$, and $\epsilon$ are hyperparameters of the optimizer.

This leads to a dynamic behavior of the optimizer that resembles a heavy ball, rolling down a curved slope under friction.

[ref]

```python
# Learning rate for the optimizer
learning_rate = 0.00001

optimizer = torch.optim.Adam(cnn_model.parameters(), lr=learning_rate)
# test what happens when you use a different optimizer such as
# optimizer = torch.optim.SGD(cnn_model.parameters(), lr=learning_rate*100, momentum=0.9)
```

### Early Stopping
Instead of training a fixed amount of epoch, we will use a technique called early stopping, to interrrupt the training at the right time.
We set the maximum number of epochs to a high value and also specify an early stopping patience.
The early stopping patients will interrupt the training, once the validation loss has not decreased in the last n epochs. This way, we can pinpoint the epoch where our model is likely to have learned the most, without overfitting too much.

[ref]

```python
# How many complete dataset passes to train
epochs = 200

# How many epochs of not decreasing validation loss to wait before stopping the training
early_stop_patience = 20
```

### Learning Rate Scheduler
If the learning rate of our scheduler is too large, we are likely to *miss* the optimum in parameter space by taking steps that are too big. If the learning rate is too small, however, training may take too long.

[ref]

A learning rate scheduler will take an initial learning rate and adapt the learning rate during training, based on a performance metric. In this example, we will start with a *high* learning rate, and use a scheduler, that decreases the learning rate, when the training loss plateaus. The scheduler uses a patience parameter to decide whether it is time to decrease the learning rate, similar to early stopping.

Question: But isn't Adam taking care of the learning rate?

    Correct! In practice, though, reducing the fixed part of the learning rate sometimes helps to descend faster into a minimum.

```python
# Parameters for the learning rate scheduler:

# How many epochs of not decreasing training loss to wait, before decreasing the learning rate
lr_scheduler_patience = 8

# Factor by which the learning rate will be reduced. new_lr : lr * factor.
lr_factor = 0.1

lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    verbose=True,
    factor=lr_factor,
    patience=lr_scheduler_patience
)
```

### Initial Performance
Let's test the initial performance of our model without any training.

```python
import ipywidgets as widgets
from ipywidgets import Layout
from utils.training import evaluate, get_lr

start_train_bar_widget = widgets.IntProgress(
    value=0,
    min=0,
    max=len(train_data_loader),
    description='Calculating initial training loss:',
    bar_style='info', # 'success', 'info', 'warning', 'danger' or ''
    style={'bar_color': 'green', 'description_width': 'initial'},
    orientation='horizontal'
)

start_val_bar_widget = widgets.IntProgress(
    value=0,
    min=0,
    max=len(val_data_loader),
    description='Calculating initial validation loss and accuracy:',
    bar_style='info', # 'success', 'info', 'warning', 'danger' or ''
    style={'bar_color': 'green', 'description_width': 'initial'},
    orientation='horizontal'
)
initial_hbox = widgets.HBox(children=(start_train_bar_widget,start_val_bar_widget))

display(initial_hbox)
initial_train_loss, _ = evaluate(
        cnn_model=cnn_model,
        device=device,
        iterator=train_data_loader,
        criterion=loss_function,
        progress_bar=start_train_bar_widget,
    )

initial_val_loss, initial_val_acc = evaluate(
        cnn_model=cnn_model,
        device=device,
        iterator=val_data_loader,
        criterion=loss_function,
        progress_bar=start_val_bar_widget,
    )

print("Initial training loss: ", initial_train_loss)
print("Initial validation loss: ", initial_val_loss)
print("Initial accuracy: ", initial_val_acc)
```

    HBox(children=(IntProgress(value=0, bar_style='info', description='Calculating initial training loss:', max=22…

    Initial training loss:  0.06110729373652827
    Initial validation loss:  0.061130413111854104
    Initial accuracy:  0.05452127659574468

```python
style = {'description_width': 'initial'}

epoch_widget = widgets.Text(
    value=f'{0}/{epochs}',
    description='Epoch:',
    style=style,
    disabled=True
)

early_stopping_widget = widgets.Text(
    value=f'{0}/{early_stop_patience}',
    description='Early Stopping Patience:',
    style=style,
    disabled=True,
)

learning_rate_widget = widgets.FloatText(
    value=learning_rate,
    description='Learning Rate:',
    style=style,
    disabled=True,
)

learning_rate_patience_widget = widgets.Text(
    value=f'{0}/{lr_scheduler_patience}',
    description='Learning Rate Patience:',
    style=style,
    disabled=True,
)

train_loss_widget = widgets.FloatText(
    value=initial_train_loss,
    description='Training Loss:',
    style=style,
    disabled=True,
)
val_loss_widget = widgets.FloatText(
    value=initial_val_loss,
    description='Validation Loss:',
    style=style,
    disabled=True
)
val_acc_widget = widgets.FloatText(
    value=initial_val_acc,
    description='Validation Accuracy:',
    style=style,
    disabled=True
)

train_bar_widget = widgets.IntProgress(
    value=0,
    min=0,
    max=len(train_data_loader),
    description='Training one epoch:',
    bar_style='info', # 'success', 'info', 'warning', 'danger' or ''
    style={'bar_color': 'green', 'description_width': 'initial'},
    orientation='horizontal'
)

val_bar_widget = widgets.IntProgress(
    value=0,
    min=0,
    max=len(val_data_loader),
    description='Validation:',
    bar_style='info', # 'success', 'info', 'warning', 'danger' or ''
    style={'bar_color': 'green', 'description_width': 'initial'},
    orientation='horizontal'
)

progress_hbox = widgets.HBox(children=(epoch_widget,early_stopping_widget))
learning_rate_hbox = widgets.HBox(children=(learning_rate_widget, learning_rate_patience_widget))
loss_hbox = widgets.HBox(children=(train_loss_widget, val_loss_widget,val_acc_widget))
bar_hbox = widgets.HBox(children=(train_bar_widget, val_bar_widget))

vbox = widgets.VBox(children=(progress_hbox, learning_rate_hbox, loss_hbox, bar_hbox))
```

## Actual Training of the CNN
The training loop in this notebook is pretty much the same as in the initial notebook.

### Train
1. Compute the outputs of the neural network for each data point in the training set.
2. Compute the loss between predicted outputs and ground truth.
3. Clean up the buffer variables inside the optimizer.
4. Compute the gradients from the loss function.
5. Perform one step in the optimizer.

### Validation
1. Compute the outputs of the neural network for each data point in the validation set.
2. Compute the loss between predicted outputs and ground truth.

### Rest
1. Adapt the learning rate with the learning rate scheduler, if necessary.
2. Stop the training, if the early stopping rule applies.

```python
# def train(cnn_model, device, iterator, optimizer, criterion, progress_bar):
```

```python
from utils.training import train

# Initialization of the train loss and the validation loss which allow to memorize the best loss obtained.
# best_valid_loss allows the early stopping function to work.
best_train_loss = float('Inf')
best_val_loss = float('Inf')

# Initialization of the counter for early stopping.
early_stopping = 0

liveloss = PlotLosses()
logs = {}
logs['loss'] = initial_train_loss
logs['val_loss'] = initial_val_loss
logs['accuracy'] = initial_val_acc
liveloss.update(logs)
liveloss.send()

for epoch in range(epochs):

    train_loss = train(
        cnn_model=cnn_model,
        device=device,
        iterator=train_data_loader,
        optimizer=optimizer,
        criterion=loss_function,
        progress_bar=train_bar_widget,
    )

    val_loss, val_acc = evaluate(
        cnn_model=cnn_model,
        device=device,
        iterator=val_data_loader,
        criterion=loss_function,
        progress_bar=val_bar_widget,
    )

    # Pass the validation loss to the learning rate scheduler to determine,
    # whether the learning rate should be reduced
    lr_scheduler.step(val_loss)

    logs['loss'] = train_loss
    logs['val_loss'] = val_loss
    logs['accuracy'] = val_acc

    liveloss.update(logs)
    liveloss.send()

    epoch_widget.value = f"{epoch+1}/{epochs}"
    train_loss_widget.value = train_loss
    val_loss_widget.value = val_loss
    val_acc_widget.value = val_acc

    # Backup of the best train loss achieved
    if train_loss < best_train_loss:
        best_train_loss = train_loss

    # Backup of the best validation loss achieved and save the model if the valid_loss has
    # decreased (valid_loss < best_valid_loss). If not decreased, +1 to early stopping counter.
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        if save_model:
            torch.save(cnn_model, model_save_path)
        early_stopping = 0
        best_model = deepcopy(cnn_model)
    else:
        early_stopping += 1

    early_stopping_widget.value = f"{early_stopping}/{early_stop_patience}"
    learning_rate_patience_widget.value = f"{lr_scheduler.num_bad_epochs}/{lr_scheduler_patience}"

    # Lr scheduler check if training loss has decreased and if
    # it hasn't after LR_scheduler_patience of epoch, lr = lr*lr_factor
    lr_scheduler.step(train_loss)

    learning_rate_widget.value = get_lr(optimizer)

    # Early stopping :
    if early_stopping > early_stop_patience:
        print("Early stopping")
        break

```

```python
display(vbox)
```

    VBox(children=(HBox(children=(Text(value='0/200', description='Epoch:', disabled=True, style=DescriptionStyle(…

Question: Why is the validation loss lower than the training loss?

    Because the training loss is measured during each epoch while validation loss is measured after each epoch. So during validation, the model will already have  new  (updated) parameter values.

Question: What can we do?

    Switch the order of validation and training.
    But remember, that it does not really matter.
    It is often easier to think "what is the validation loss after my nth training epoch?".

Question: How can it be, that the accuracy does not change, even though the loss decreases?

    Ignoring the log function (only taking the Softmax), the output of the model will be a list of probabilities (values between 0 and 1, that sum to 1).
    The accuracy is determined, by comparing the ground truth label {0..6} with the predicted label by taking the argmax of this list (index of the maximum value in the list).
    The loss, on the other hand, will be the distance between each value of the list and the ground truth value.

    For a classification task with two classes:
    With a ground truth label of 1, so a value list of [0.0, 1.0], a change in predicted probabilities from [0.49, 0.51] to [0.1, 0.9] drastically decreases the loss, it will not decrease the predicted class.

## Testing the Final Model Performance
To get the final model performance, we need to calculate the loss and accuracy on the test data set.

```python
test_bar_widget = widgets.IntProgress(
    value=0,
    min=0,
    max=len(test_data_loader),
    description='Testing:',
    bar_style='info', # 'success', 'info', 'warning', 'danger' or ''
    style={'bar_color': 'green', 'description_width': 'initial'},
    orientation='horizontal'
)
display(test_bar_widget)

test_loss, test_acc = evaluate(
    cnn_model=best_model,
    device=device,
    iterator=test_data_loader,
    criterion=loss_function,
    progress_bar=test_bar_widget,
)

print("Final testing loss: ", test_loss)
print("Final testing accuracy: ", test_acc)
```

    IntProgress(value=0, bar_style='info', description='Testing:', max=47, style=ProgressStyle(bar_color='green', …

    Final testing loss:  0.021463156618336414
    Final testing accuracy:  0.7420212765957447

## What's next?
* Instead of importing the training function (`from utils.training import train`), try to implement it yourself.
* Change the CNN model and see how the training results change. You can find a list of possible additional building blocks [here](https://pytorch.org/docs/stable/nn.html#convolution-layers).
* Also try to change the other hyper parameters to maximize the validation accuracy.
* Try to convert the CNN into a fully convolutional architecture, by following the [conv layer formulas](https://cs231n.github.io/convolutional-networks/#conv)

Volume of size \(W_1 \times H_1 \times D_1\)

Requires four hyperparameters:

Number of filters \(K\),

their spatial extent \(F\),

the stride \(S\),

the amount of zero padding \(P\).

Produces a volume of size \(W_2 \times H_2 \times D_2\) where:

\(W_2 = (W_1 - F + 2P)/S + 1\)

\(H_2 = (H_1 - F + 2P)/S + 1\) (i.e. width and height are computed equally by symmetry)

\(D_2 = K\)

    and the [pytorch documentation](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d).