# Basic Functionality

**Prerequisites**

- [pandas Intro](https://datascience.quantecon.org/intro.html)

**Outcomes**

- Be familiar with `datetime`
- Use built-in aggregation functions and be able to create your own and
  apply them using `agg`
- Use built-in Series transformation functions and be able to create your
  own and apply them using `apply`
- Use built-in scalar transformation functions and be able to create your
  own and apply them using `applymap`
- Be able to select subsets of the DataFrame using boolean selection
- Know what the “want operator” is and how to apply it

**Data**

- US state unemployment data from Bureau of Labor Statistics

## Outline

- [Basic Functionality](#Basic-Functionality)
  - [State Unemployment Data](#State-Unemployment-Data)
  - [Dates in pandas](#Dates-in-pandas)
  - [DataFrame Aggregations](#DataFrame-Aggregations)
  - [Transforms](#Transforms)
  - [Boolean Selection](#Boolean-Selection)
  - [Exercises](#Exercises)

```python
# Uncomment following line to install on colab
! pip install qeds
```

    Requirement already satisfied: qeds in c:\users\asus\anaconda3\lib\site-packages (0.6.2)
    Requirement already satisfied: quandl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.5.0)
    Requirement already satisfied: pandas in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.25.1)
    Requirement already satisfied: pyarrow in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.16.0)
    Requirement already satisfied: statsmodels in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.10.1)
    Requirement already satisfied: requests in c:\users\asus\anaconda3\lib\site-packages (from qeds) (2.22.0)
    Requirement already satisfied: plotly in c:\users\asus\anaconda3\lib\site-packages (from qeds) (4.5.4)
    Requirement already satisfied: scipy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.3.1)
    Requirement already satisfied: matplotlib in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.1.1)
    Requirement already satisfied: scikit-learn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.21.3)
    Requirement already satisfied: seaborn in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.9.0)
    Requirement already satisfied: numpy in c:\users\asus\anaconda3\lib\site-packages (from qeds) (1.16.5)
    Requirement already satisfied: quantecon in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.4.6)
    Requirement already satisfied: openpyxl in c:\users\asus\anaconda3\lib\site-packages (from qeds) (3.0.0)
    Requirement already satisfied: pandas-datareader in c:\users\asus\anaconda3\lib\site-packages (from qeds) (0.8.1)
    Requirement already satisfied: inflection>=0.3.1 in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (0.3.1)
    Requirement already satisfied: more-itertools in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (7.2.0)
    Requirement already satisfied: python-dateutil in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (2.8.0)
    Requirement already satisfied: six in c:\users\asus\anaconda3\lib\site-packages (from quandl->qeds) (1.12.0)
    Requirement already satisfied: pytz>=2017.2 in c:\users\asus\anaconda3\lib\site-packages (from pandas->qeds) (2019.3)
    Requirement already satisfied: patsy>=0.4.0 in c:\users\asus\anaconda3\lib\site-packages (from statsmodels->qeds) (0.5.1)
    Requirement already satisfied: certifi>=2017.4.17 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2019.9.11)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (3.0.4)
    Requirement already satisfied: idna<2.9,>=2.5 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (2.8)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\asus\anaconda3\lib\site-packages (from requests->qeds) (1.24.2)
    Requirement already satisfied: retrying>=1.3.3 in c:\users\asus\anaconda3\lib\site-packages (from plotly->qeds) (1.3.3)
    Requirement already satisfied: cycler>=0.10 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\asus\anaconda3\lib\site-packages (from matplotlib->qeds) (2.4.2)
    Requirement already satisfied: joblib>=0.11 in c:\users\asus\anaconda3\lib\site-packages (from scikit-learn->qeds) (0.13.2)
    Requirement already satisfied: numba>=0.38 in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (0.45.1)
    Requirement already satisfied: sympy in c:\users\asus\anaconda3\lib\site-packages (from quantecon->qeds) (1.4)
    Requirement already satisfied: jdcal in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.4.1)
    Requirement already satisfied: et-xmlfile in c:\users\asus\anaconda3\lib\site-packages (from openpyxl->qeds) (1.0.1)
    Requirement already satisfied: lxml in c:\users\asus\anaconda3\lib\site-packages (from pandas-datareader->qeds) (4.4.1)
    Requirement already satisfied: setuptools in c:\users\asus\anaconda3\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->qeds) (41.4.0)
    Requirement already satisfied: llvmlite>=0.29.0dev0 in c:\users\asus\anaconda3\lib\site-packages (from numba>=0.38->quantecon->qeds) (0.29.0)
    Requirement already satisfied: mpmath>=0.19 in c:\users\asus\anaconda3\lib\site-packages (from sympy->quantecon->qeds) (1.1.0)

## State Unemployment Data

In this lecture, we will use unemployment data by state at a monthly
frequency.

```python
import pandas as pd

%matplotlib inline
# activate plot theme
import qeds
qeds.themes.mpl_style();

pd.__version__
```

    '0.25.1'

First, we will download the data directly from a url and read it into a pandas DataFrame.

```python
## Load up the data -- this will take a couple seconds
url = "https://datascience.quantecon.org/assets/data/state_unemployment.csv"
unemp_raw = pd.read_csv(url, parse_dates=["Date"])
```

The pandas `read_csv` will determine most datatypes of the underlying columns.  The
exception here is that we need to give pandas a hint so it can load up the `Date` column as a Python datetime type: the `parse_dates=["Date"]`.

We can see the basic structure of the downloaded data by getting the first 5 rows, which directly matches
the underlying CSV file.

```python
unemp_raw.head()
```

      Date
      state
      LaborForce
      UnemploymentRate

      0
      2000-01-01
      Alabama
      2142945.0
      4.7

      1
      2000-01-01
      Alaska
      319059.0
      6.3

      2
      2000-01-01
      Arizona
      2499980.0
      4.1

      3
      2000-01-01
      Arkansas
      1264619.0
      4.4

      4
      2000-01-01
      California
      16680246.0
      5.0

Note that a row has a date, state, labor force size, and unemployment rate.

For our analysis, we want to look at the unemployment rate across different states over time, which
requires a transformation of the data similar to an Excel pivot-table.

```python
# Don't worry about the details here quite yet
unemp_all = (
    unemp_raw
    .reset_index()
    .pivot_table(index="Date", columns="state", values="UnemploymentRate")
)
unemp_all.head()
```

      state
      Alabama
      Alaska
      Arizona
      Arkansas
      California
      Colorado
      Connecticut
      Delaware
      Florida
      Georgia
      ...
      South Dakota
      Tennessee
      Texas
      Utah
      Vermont
      Virginia
      Washington
      West Virginia
      Wisconsin
      Wyoming

      Date

      2000-01-01
      4.7
      6.3
      4.1
      4.4
      5.0
      2.8
      2.8
      3.5
      3.7
      3.7
      ...
      2.4
      3.7
      4.6
      3.1
      2.7
      2.6
      4.9
      5.8
      3.2
      4.1

      2000-02-01
      4.7
      6.3
      4.1
      4.3
      5.0
      2.8
      2.7
      3.6
      3.7
      3.6
      ...
      2.4
      3.7
      4.6
      3.1
      2.6
      2.5
      4.9
      5.6
      3.2
      3.9

      2000-03-01
      4.6
      6.3
      4.0
      4.3
      5.0
      2.7
      2.6
      3.6
      3.7
      3.6
      ...
      2.4
      3.8
      4.5
      3.1
      2.6
      2.4
      5.0
      5.5
      3.3
      3.9

      2000-04-01
      4.6
      6.3
      4.0
      4.3
      5.1
      2.7
      2.5
      3.7
      3.7
      3.7
      ...
      2.4
      3.8
      4.4
      3.1
      2.7
      2.4
      5.0
      5.4
      3.4
      3.8

      2000-05-01
      4.5
      6.3
      4.0
      4.2
      5.1
      2.7
      2.4
      3.7
      3.7
      3.7
      ...
      2.4
      3.9
      4.3
      3.2
      2.7
      2.3
      5.1
      5.4
      3.5
      3.8

5 rows × 50 columns

Finally, we can filter it to look at a subset of the columns (i.e. “state” in this case).

```python
states = [
    "Arizona", "California", "Florida", "Illinois",
    "Michigan", "New York", "Texas"
]
unemp = unemp_all[states]
unemp.head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

When plotting, a DataFrame knows the column and index names.

```python
unemp.plot(figsize=(8, 6))
```

## Dates in pandas

You might have noticed that our index now has a nice format for the
dates (`YYYY-MM-DD`) rather than just a year.

This is because the `dtype` of the index is a variant of `datetime`.

```python
unemp.index
```

    DatetimeIndex(['2000-01-01', '2000-02-01', '2000-03-01', '2000-04-01',
                   '2000-05-01', '2000-06-01', '2000-07-01', '2000-08-01',
                   '2000-09-01', '2000-10-01',
                   ...
                   '2017-03-01', '2017-04-01', '2017-05-01', '2017-06-01',
                   '2017-07-01', '2017-08-01', '2017-09-01', '2017-10-01',
                   '2017-11-01', '2017-12-01'],
                  dtype='datetime64[ns]', name='Date', length=216, freq=None)

We can index into a DataFrame with a `DatetimeIndex` using string
representations of dates.

For example

```python
# Data corresponding to a single date
unemp.loc["01/01/2000", :]
```

    state
    Arizona       4.1
    California    5.0
    Florida       3.7
    Illinois      4.2
    Michigan      3.3
    New York      4.7
    Texas         4.6
    Name: 2000-01-01 00:00:00, dtype: float64

```python
# Data for all days between New Years Day and June first in the year 2000
unemp.loc["01/01/2000":"06/01/2000", :]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

      2000-06-01
      4.0
      5.1
      3.8
      4.3
      3.7
      4.6
      4.3

We will learn more about what pandas can do with dates and times in an
upcoming lecture on time series data.

## DataFrame Aggregations

Let’s talk about *aggregations*.

Loosely speaking, an aggregation is an operation that combines multiple
values into a single value.

For example, computing the mean of three numbers (for example
`[0, 1, 2]`) returns a single number (1).

We will use aggregations extensively as we analyze and manipulate our data.

Thankfully, pandas makes this easy!

### Built-in Aggregations

pandas already has some of the most frequently used aggregations.

For example:

- Mean  (`mean`)
- Variance (`var`)
- Standard deviation (`std`)
- Minimum (`min`)
- Median (`median`)
- Maximum (`max`)
- etc…

>**Note**
>
>When looking for common operations, using “tab completion” goes a long way.

```python
unemp.mean()
```

    state
    Arizona       6.301389
    California    7.299074
    Florida       6.048611
    Illinois      6.822685
    Michigan      7.492593
    New York      6.102315
    Texas         5.695370
    dtype: float64

As seen above, the aggregation’s default is to aggregate each column.

However, by using the `axis` keyword argument, you can do aggregations by
row as well.

```python
unemp.var(axis=1).head()
```

    Date
    2000-01-01    0.352381
    2000-02-01    0.384762
    2000-03-01    0.364762
    2000-04-01    0.353333
    2000-05-01    0.294762
    dtype: float64

### Writing Your Own Aggregation

The built-in aggregations will get us pretty far in our analysis, but
sometimes we need more flexibility.

We can have pandas perform custom aggregations by following these two
steps:

1. Write a Python function that takes a `Series` as an input and
  outputs a single value.
1. Call the `agg` method with our new function as an argument.

For example, below, we will classify states as “low unemployment” or
“high unemployment” based on whether their mean unemployment level is
above or below 6.5.

```python
#
# Step 1: We write the (aggregation) function that we'd like to use
#
def high_or_low(s):
    """
    This function takes a pandas Series object and returns high
    if the mean is above 6.5 and low if the mean is below 6.5
    """
    if s.mean() < 6.5:
        out = "Low"
    else:
        out = "High"

    return out
```

```python
#
# Step 2: Apply it via the agg method.
#
unemp.agg(high_or_low)
```

    state
    Arizona        Low
    California    High
    Florida        Low
    Illinois      High
    Michigan      High
    New York       Low
    Texas          Low
    dtype: object

```python
# How does this differ from unemp.agg(high_or_low)?
unemp.agg(high_or_low, axis=1).head()
```

    Date
    2000-01-01    Low
    2000-02-01    Low
    2000-03-01    Low
    2000-04-01    Low
    2000-05-01    Low
    dtype: object

Notice that `agg` can also accept multiple functions at once.

```python
unemp.agg([min, max, high_or_low])
```

      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      min
      3.6
      4.5
      3.1
      4.2
      3.2
      4.2
      3.9

      max
      10.9
      12.3
      11.3
      11.3
      14.6
      8.9
      8.3

      high_or_low
      Low
      High
      Low
      High
      High
      Low
      Low

## Transforms

Many analytical operations do not necessarily involve an aggregation.

The output of a function applied to a Series might need to be a new
Series.

Some examples:

- Compute the percentage change in unemployment from month to month.
- Calculate the cumulative sum of elements in each column.

### Built-in Transforms

pandas comes with many transform functions including:

- Cumulative sum/max/min/product (`cum(sum|min|max|prod)`)
- Difference  (`diff`)
- Elementwise addition/subtraction/multiplication/division (`+`, `-`, `*`, `/`)
- Percent change (`pct_change`)
- Number of occurrences of each distinct value (`value_counts`)
- Absolute value (`abs`)

Again, tab completion is helpful when trying to find these functions.

```python
unemp.head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

```python
unemp.pct_change().head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      2000-02-01
      0.00000
      0.00
      0.0
      0.00000
      -0.030303
      0.000000
      0.000000

      2000-03-01
      -0.02439
      0.00
      0.0
      0.02381
      0.000000
      -0.021277
      -0.021739

      2000-04-01
      0.00000
      0.02
      0.0
      0.00000
      0.031250
      0.000000
      -0.022222

      2000-05-01
      0.00000
      0.00
      0.0
      0.00000
      0.060606
      0.000000
      -0.022727

```python
unemp.diff().head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      2000-02-01
      0.0
      0.0
      0.0
      0.0
      -0.1
      0.0
      0.0

      2000-03-01
      -0.1
      0.0
      0.0
      0.1
      0.0
      -0.1
      -0.1

      2000-04-01
      0.0
      0.1
      0.0
      0.0
      0.1
      0.0
      -0.1

      2000-05-01
      0.0
      0.0
      0.0
      0.0
      0.2
      0.0
      -0.1

Transforms can be split into to several main categories:

1. *Series transforms*: functions that take in one Series and produce another Series. The index of the input and output does not need to be the same.
1. *Scalar transforms*: functions that take a single value and produce a single value. An example is the `abs` method, or adding a constant to each value of a Series.

### Custom Series Transforms

pandas also simplifies applying custom Series transforms to a Series or the
columns of a DataFrame. The steps are:

1. Write a Python function that takes a Series and outputs a new Series.
1. Pass our new function as an argument to the `apply` method (alternatively, the `transform` method).

As an example, we will standardize our unemployment data to have mean 0
and standard deviation 1.

After doing this, we can use an aggregation to determine at which date the
unemployment rate is most different from “normal times” for each state.

```python
#
# Step 1: We write the Series transform function that we'd like to use
#
def standardize_data(x):
    """
    Changes the data in a Series to become mean 0 with standard deviation 1
    """
    mu = x.mean()
    std = x.std()

    return (x - mu)/std
```

```python
#
# Step 2: Apply our function via the apply method.
#
std_unemp = unemp.apply(standardize_data)
std_unemp.head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      -1.076861
      -0.935545
      -0.976846
      -1.337203
      -1.605740
      -0.925962
      -0.849345

      2000-02-01
      -1.076861
      -0.935545
      -0.976846
      -1.337203
      -1.644039
      -0.925962
      -0.849345

      2000-03-01
      -1.125778
      -0.935545
      -0.976846
      -1.286217
      -1.644039
      -0.991993
      -0.926885

      2000-04-01
      -1.125778
      -0.894853
      -0.976846
      -1.286217
      -1.605740
      -0.991993
      -1.004424

      2000-05-01
      -1.125778
      -0.894853
      -0.976846
      -1.286217
      -1.529141
      -0.991993
      -1.081964

```python
# Takes the absolute value of all elements of a function
abs_std_unemp = std_unemp.abs()

abs_std_unemp.head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      1.076861
      0.935545
      0.976846
      1.337203
      1.605740
      0.925962
      0.849345

      2000-02-01
      1.076861
      0.935545
      0.976846
      1.337203
      1.644039
      0.925962
      0.849345

      2000-03-01
      1.125778
      0.935545
      0.976846
      1.286217
      1.644039
      0.991993
      0.926885

      2000-04-01
      1.125778
      0.894853
      0.976846
      1.286217
      1.605740
      0.991993
      1.004424

      2000-05-01
      1.125778
      0.894853
      0.976846
      1.286217
      1.529141
      0.991993
      1.081964

```python
# find the date when unemployment was "most different from normal" for each State
def idxmax(x):
    # idxmax of Series will return index of maximal value
    return x.idxmax()

abs_std_unemp.agg(idxmax)
```

    state
    Arizona      2009-11-01
    California   2010-03-01
    Florida      2010-01-01
    Illinois     2009-12-01
    Michigan     2009-06-01
    New York     2009-11-01
    Texas        2009-08-01
    dtype: datetime64[ns]

### Custom Scalar Transforms

As you may have predicted, we can also apply custom scalar transforms to our
pandas data.

To do this, we use the following pattern:

1. Define a Python function that takes in a scalar and produces a scalar.
1. Pass this function as an argument to the `applymap` Series or DataFrame method.

## Boolean Selection

We have seen how we can select subsets of data by referring to the index
or column names.

However, we often want to select based on conditions met by
the data itself.

Some examples are:

- Restrict analysis to all individuals older than 18.
- Look at data that corresponds to particular time periods.
- Analyze only data that corresponds to a recession.
- Obtain data for a specific product or customer ID.

We will be able to do this by using a Series or list of boolean values
to index into a Series or DataFrame.

Let’s look at some examples.

```python
unemp_small = unemp.head()  # Create smaller data so we can see what's happening
unemp_small
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

```python
# list of booleans selects rows
unemp_small.loc[[True, True, True, False, False]]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

```python
# second argument selects columns, the  ``:``  means "all".
# here we use it to select all columns
unemp_small.loc[[True, False, True, False, True], :]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

```python
# can use booleans to select both rows and columns
unemp_small.loc[[True, True, True, False, False], [True, False, False, False, False, True, True]]
```

      state
      Arizona
      New York
      Texas

      Date

      2000-01-01
      4.1
      4.7
      4.6

      2000-02-01
      4.1
      4.7
      4.6

      2000-03-01
      4.0
      4.6
      4.5

### Creating Boolean DataFrames/Series

We can use [conditional statements](https://datascience.quantecon.org/../python_fundamentals/control_flow.html) to
construct Series of booleans from our data.

```python
unemp_small["Texas"] < 4.5
```

    Date
    2000-01-01    False
    2000-02-01    False
    2000-03-01    False
    2000-04-01     True
    2000-05-01     True
    Name: Texas, dtype: bool

Once we have our Series of bools, we can use it to extract subsets of
rows from our DataFrame.

```python
unemp_small.loc[unemp_small["Texas"] < 4.5]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

```python
unemp_small["New York"] > unemp_small["Texas"]
```

    Date
    2000-01-01    True
    2000-02-01    True
    2000-03-01    True
    2000-04-01    True
    2000-05-01    True
    dtype: bool

```python
big_NY = unemp_small["New York"] > unemp_small["Texas"]
unemp_small.loc[big_NY]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

#### Multiple Conditions

In the boolean section of the [basics lecture](https://datascience.quantecon.org/../python_fundamentals/basics.html), we saw
that we can use the words `and` and `or` to combine multiple booleans into
a single bool.

Recall:

- `True and False -> False`
- `True and True -> True`
- `False and False -> False`
- `True or False -> True`
- `True or True -> True`
- `False or False -> False`

We can do something similar in pandas, but instead of
`bool1 and bool2` we write:

```python
(bool_series1) & (bool_series2)
```

Likewise, instead of `bool1 or bool2` we write:

```python
(bool_series1) | (bool_series2)
```

```python
small_NYTX = (unemp_small["Texas"] < 4.7) & (unemp_small["New York"] < 4.7)
small_NYTX
```

    Date
    2000-01-01    False
    2000-02-01    False
    2000-03-01     True
    2000-04-01     True
    2000-05-01     True
    dtype: bool

```python
unemp_small[small_NYTX]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

      2000-05-01
      4.0
      5.1
      3.7
      4.3
      3.5
      4.6
      4.3

#### `isin`

Sometimes, we will want to check whether a data point takes on one of a
several fixed values.

We could do this by writing `(df["x"] == val_1) | (df["x"] == val_2)`
(like we did above), but there is a better way: the `.isin` method

```python
unemp_small["Michigan"].isin([3.3, 3.2])
```

    Date
    2000-01-01     True
    2000-02-01     True
    2000-03-01     True
    2000-04-01     True
    2000-05-01    False
    Name: Michigan, dtype: bool

```python
# now select full rows where this Series is True
unemp_small.loc[unemp_small["Michigan"].isin([3.3, 3.2])]
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      4.1
      5.0
      3.7
      4.2
      3.3
      4.7
      4.6

      2000-02-01
      4.1
      5.0
      3.7
      4.2
      3.2
      4.7
      4.6

      2000-03-01
      4.0
      5.0
      3.7
      4.3
      3.2
      4.6
      4.5

      2000-04-01
      4.0
      5.1
      3.7
      4.3
      3.3
      4.6
      4.4

#### `.any` and `.all`

Recall from the boolean section of the [basics lecture](https://datascience.quantecon.org/../python_fundamentals/basics.html)
that the Python functions `any` and `all` are aggregation functions that
take a collection of booleans and return a single boolean.

`any` returns True whenever at least one of the inputs are True while
`all` is True only when all the inputs are `True`.

Series and DataFrames with `dtype` bool have `.any` and `.all`
methods that apply this logic to pandas objects.

Let’s use these methods to count how many months all the states in our
sample had high unemployment.

As we work through this example, consider the [“want
operator”](http://albertjmenkveld.com/2014/07/07/endogeneous-price-dispersion/), a helpful
concept from Nobel Laureate [Tom
Sargent](http://www.tomsargent.com) for clearly stating the goal of our analysis and
determining the steps necessary to reach the goal.

We always begin by writing `Want:` followed by what we want to
accomplish.

In this case, we would write:

> Want: Count the number of months in which all states in our sample
had unemployment above 6.5%

After identifying the **want**, we work *backwards* to identify the
steps necessary to accomplish our goal.

So, starting from the result, we have:

1. Sum the number of `True` values in a Series indicating dates for
  which all states had high unemployment.
1. Build the Series used in the last step by using the `.all` method
  on a DataFrame containing booleans indicating whether each state had
  high unemployment at each date.
1. Build the DataFrame used in the previous step using a `>`
  comparison.

Now that we have a clear plan, let’s follow through and *apply* the want
operator:

```python
# Step 3: construct the DataFrame of bools
high = unemp > 6.5
high.head()
```

      state
      Arizona
      California
      Florida
      Illinois
      Michigan
      New York
      Texas

      Date

      2000-01-01
      False
      False
      False
      False
      False
      False
      False

      2000-02-01
      False
      False
      False
      False
      False
      False
      False

      2000-03-01
      False
      False
      False
      False
      False
      False
      False

      2000-04-01
      False
      False
      False
      False
      False
      False
      False

      2000-05-01
      False
      False
      False
      False
      False
      False
      False

```python
# Step 2: use the .all method on axis=1 to get the dates where all states have a True
all_high = high.all(axis=1)
all_high.head()
```

    Date
    2000-01-01    False
    2000-02-01    False
    2000-03-01    False
    2000-04-01    False
    2000-05-01    False
    dtype: bool

```python
# Step 1: Call .sum to add up the number of True values in `all_high`
#         (note that True == 1 and False == 0 in Python, so .sum will count Trues)
msg = "Out of {} months, {} had high unemployment across all states"
print(msg.format(len(all_high), all_high.sum()))
```

    Out of 216 months, 41 had high unemployment across all states


```javascript
%%javascript
    MathJax.Hub.Config({
      TeX: { equationNumbers: { autoNumber: "AMS" } }
    });
```

    <IPython.core.display.Javascript object>

```python
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Code Toggle"></form>''')
```

<form action="javascript:code_toggle()"><input type="submit" value="Code Toggle"></form>

```python
from IPython.display import HTML

HTML('''
<a href="{{ site.links.github }}/raw/nist-pages/benchmarks/benchmark7.ipynb"
   download>
<button type="submit">Download Notebook</button>
</a>
''')
```

<a href="{{ site.links.github }}/raw/nist-pages/benchmarks/benchmark7.ipynb"
   download>
<button type="submit">Download Notebook</button>
</a>

# Benchmark Problem 7: MMS Allen-Cahn

```python
from IPython.display import HTML

HTML('''{% include jupyter_benchmark_table.html num="[7]" revision=0 %}''')
```

{% include jupyter_benchmark_table.html num="[7]" revision=0 %}

* [Overview](#Overview)
* [Governing equation and manufactured solution](#Governing-equation-and-manufactured-solution)
* [Domain geometry, boundary conditions, initial conditions, and stopping condition](#Domain-geometry,-boundary-conditions,-initial-conditions,-and-stopping-condition)
* [Parameter values](#Parameter-values)
* [Benchmark simulation instructions](#Benchmark-simulation-instructions)
    * [Part (a)](#Part-%28a%29)
    * [Part (b)](#Part-%28b%29)
    * [Part (c)](#Part-%28c%29)
* [Results](#Results)
* [Feedback](#Feedback)
* [Appendix](#Appendix)
    * [Computer algebra systems](#Computer-algebra-systems)
    * [Source equation](#Source-equation)
    * [Code](#Code)

See the journal publication entitled ["Benchmark problems for numerical implementations of phase field models"][benchmark_paper] for more details about the benchmark problems. Furthermore, read [the extended essay][benchmarks] for a discussion about the need for benchmark problems.

[benchmarks]: ../
[benchmark_paper]: http://dx.doi.org/10.1016/j.commatsci.2016.09.022

# Overview

The Method of Manufactured Solutions (MMS) is a powerful technique for verifying the accuracy of a simulation code. In the MMS, one picks a desired solution to the problem at the outset, the "manufactured solution", and then determines the governing equation that will result in that solution. With the exact analytical form of the solution in hand, when the governing equation is solved using a particular simulation code, the deviation from the expected solution can be determined exactly. This deviation can be converted into an error metric to rigously quantify the error for a calculation. This error can be used to determine the order of accuracy of the simulation results to verify simulation codes. It can also be used to compare the computational efficiency of different codes or different approaches for a particular code at a certain level of error. Furthermore, the spatial/temporal distribution can give insight into the conditions resulting in the largest error (high gradients, changes in mesh resolution, etc.).

After choosing a manufactured solution, the governing equation must be modified to force the solution to equal the manufactured solution. This is accomplished by taking the nominal equation that is to be solved (e.g.  Allen-Cahn equation, Cahn-Hilliard equation, Fick's second law, Laplace equation) and adding a source term. This source term is determined by plugging the manufactured solution into the nominal governing equation and setting the source term equal to the residual. Thus, the manufactured solution satisfies the MMS governing equation (the nominal governing equation plus the source term). A more detailed discussion of MMS can be found in [the report by Salari and Knupp][mms_report].

In this benchmark problem, the objective is to use the MMS to rigorously verify phase field simulation codes and then provide a basis of comparison for the computational performance between codes and for various settings for a single code, as discussed above. To this end, the benchmark problem was chosen as a balance between two factors: simplicity, to minimize the development effort required to solve the benchmark, and transferability to a real phase field system of physical interest.

[mms_report]: http://prod.sandia.gov/techlib/access-control.cgi/2000/001444.pdf

# Governing equation and manufactured solution
For this benchmark problem, we use a simple Allen-Cahn equation as the governing equation

$$\begin{equation}
\frac{\partial \eta}{\partial t} = - \left[ 4 \eta \left(\eta - 1 \right) \left(\eta-\frac{1}{2} \right) - \kappa \nabla^2 \eta \right] + S(x,y,t)
\end{equation}$$

where $S(x,y,t)$ is the MMS source term and $\kappa$ is a constant parameter (the gradient energy coefficient).

The manufactured solution, $\eta_{sol}$ is a hyperbolic tangent function, shifted to vary between 0 and 1, with the $x$ position of the middle of the interface ($\eta_{sol}=0.5$) given by the function $\alpha(x,t)$:

$$\begin{equation}
\eta_{sol}(x,y,t) = \frac{1}{2}\left[ 1 - \tanh\left( \frac{y-\alpha(x,t)}{\sqrt{2 \kappa}} \right) \right]
\end{equation}$$

$$\begin{equation}
\alpha(x,t) = \frac{1}{4} + A_1 t \sin\left(B_1 x \right) + A_2 \sin \left(B_2  x + C_2 t \right)
\end{equation}$$

where $A_1$, $B_1$, $A_2$, $B_2$, and $C_2$ are constant parameters.

This manufactured solution is an equilbrium solution of the governing equation, when $S(x,y,t)=0$ and $\alpha(x,t)$ is constant. The closeness of this manufactured solution to a solution of the nominal governing equation increases the likihood that the behavior of simulation codes when solving this benchmark problem is representive of the solution of the regular Allen-Cahn equation (i.e. without the source term). The form of $\alpha(x,t)$ was chosen to yield complex behavior while still retaining a (somewhat) simple functional form. The two spatial sinusoidal terms introduce two controllable length scales to the interfacial shape. Summing them gives a  "beat" pattern with a period longer than the period of either individual term, permitting a domain size that is larger than the wavelength of the sinusoids without a repeating pattern. The temporal sinusoidal term introduces a controllable time scale to the interfacial shape in addition to the phase transformation time scale, while the linear temporal dependence of the other term ensures that the sinusoidal term can go through multiple periods without $\eta_{sol}$ repeating itself.

Inserting the manufactured solution into the governing equation and solving for $S(x,y,t)$ yields:

$$\begin{equation}
S(x,y,t) = \frac{\text{sech}^2 \left[ \frac{y-\alpha(x,t)}{\sqrt{2 \kappa}} \right]}{4 \sqrt{\kappa}} \left[-2\sqrt{\kappa} \tanh \left[\frac{y-\alpha(x,t)}{\sqrt{2 \kappa}} \right] \left(\frac{\partial \alpha(x,t)}{\partial x} \right)^2+\sqrt{2} \left[ \frac{\partial \alpha(x,t)}{\partial t}-\kappa \frac{\partial^2 \alpha(x,t)}{\partial x^2} \right] \right]
\end{equation}$$

where $\alpha(x,t)$ is given above and where:

$$\begin{equation}
\frac{\partial \alpha(x,t)}{\partial x} = A_1 B_1 t \cos\left(B_1 x\right) + A_2 B_2 \cos \left(B_2  x + C_2 t \right)
\end{equation}$$

$$\begin{equation}
\frac{\partial^2 \alpha(x,t)}{\partial x^2} = -A_1 B_1^2 t \sin\left(B_1 x\right) - A_2 B_2^2 \sin \left(B_2  x + C_2 t \right)
\end{equation}$$

$$\begin{equation}
\frac{\partial \alpha(x,t)}{\partial t} = A_1 \sin\left(B_1 x\right) + A_2 C_2 \cos \left(B_2  x + C_2 t \right)
\end{equation}$$

#### *N.B.*: Don't transcribe these equations. Please download the appropriate files from the [Appendix](#Appendix).

# Domain geometry, boundary conditions, initial conditions, and stopping condition
The domain geometry is a rectangle that spans [0, 1] in $x$ and [0, 0.5] in $y$. This elongated domain was chosen to allow multiple peaks and valleys in $\eta_{sol}$ without stretching the interface too much in the $y$ direction (which causes the thickness of the interface to change) or having large regions where $\eta_{sol}$  never deviates from 0 or 1. Periodic boundary conditions are applied along the $x = 0$ and the $x = 1$ boundaries to accomodate the periodicity of $\alpha(x,t)$. Dirichlet boundary conditions of $\eta$ = 0 and $\eta$ = 1 are applied along the $y = 0$ and the $y = 0.5$ boundaries, respectively. These boundary conditions are chosen to be consistent with $\eta_{sol}(x,y,t)$. The initial condition is the manufactured solution at $t = 0$:

$$
\begin{equation}
\eta_{sol}(x,y,0) = \frac{1}{2}\left[ 1 - \tanh\left( \frac{y-\left(\frac{1}{4}+A_2 \sin(B_2 x) \right)}{\sqrt{2 \kappa}} \right) \right]
\end{equation}
$$

The stopping condition for all calculations is when t = 8 time units, which was chosen to let $\alpha(x,t)$ evolve substantially, while still being slower than the characteristic time for the phase evolution (determined by the CFL condition for a uniform mesh with a reasonable level of resolution of $\eta_{sol}$).

# Parameter values
The nominal parameter values for the governing equation and manufactured solution are given below. The value of $\kappa$ will change in Part (b) in the following section and the values of $\kappa$ and $C_2$ will change in Part (c).

| Parameter | Value |
|-----------|-------|
| $\kappa$  | 0.0004|
| $A_1$     | 0.0075|
| $B_1$     | 0.03  |
| $A_2$     | 8.0   |
| $B_2$     | 22.0  |
| $C_2$     | 0.0625|

# Benchmark simulation instructions
This section describes three sets of tests to conduct using the MMS problem specified above. The primary purpose of the first test is provide a computationally inexpensive problem to verify a simulation code. The second and third tests are more computationally demanding and are primarily designed to serve as a basis for performance comparisons.

## Part (a)
The objective of this test is to verify the accuracy of your simulation code in both time and space. Here, we make use of convergence tests, where either the mesh size (or grid point spacing) or the time step size is systematically changed to determine the response of the error to these quantities. Once a convergence test is completed the order of accuracy can be calculated from the result. The order of accuracy can be compared to the theoretical order of accuracy for the numerical method employed in the simulation. If the two match (to a reasonable degree), then one can be confident that the simulation code is working as expected. The remainder of this subsection will give instructions for convergence tests for this MMS problem.

Implement the MMS problem specified above using the simulation code of your choice. Perform a spatial convergence test by running the simulation for a variety of mesh sizes. For each simulation, determine the discrete $L_2$ norm of the error at $t=8$:

$$\begin{equation}
    L_2 = \sqrt{\sum\limits_{x,y}\left(\eta^{t=8}_{x,y} - \eta_{sol}(x,y,8)\right)^2 \Delta x \Delta y}
\end{equation}$$

For all of these simulations, verify that the time step is small enough that any temporal error is much smaller that the total error. This can be accomplished by decreasing the time step until it has minimal effect on the error. Ensure that at least three simulation results have $L_2$ errors in the range $[5\times10^{-3}, 1\times10^{-4}]$, attempting to cover as much of that range as possible/practical. This maximum and minimum errors in the range roughly represent a poorly resolved simulation and a very well-resolved simulation.

For at least three simulations that have $L_2$ errors in the range $[5\times10^{-3}, 1\times10^{-4}]$, save the effective mesh size and $L_2$ error in a CSV or JSON file. Upload this file to the PFHub website as a 2D data set with the effective mesh size as the x-axis column and the $L_2$ error as the y-axis column. Calculate the effective element size as the square root of the area of the finest part of the mesh for nonuniform meshes. For irregular meshes with continous distributions of element sizes, approximate the effective mesh size as the average of the square root of the area of the smallest 5% of the elements.

Next, confirm that the observed order of accuracy is approximately equal to the expected value. Calculate the order of accuracy, $p$, with a least squares fit of the following function:

$$\begin{equation}
    \log(E)=p \log(R) + b
\end{equation}$$

where $E$ is the $L_2$ error, $R$ is the effective element size, and b is an intercept. Deviations of ±0.2 or more from the theoretical value are to be expected (depending on the range of errors considered and other factors).

Finally, perform a similar convergence test, but for the time step, systematically changing the time step and recording the $L_2$ error. Use a time step that does not vary over the course of any single simulation. Verify that the spatial discretization error is small enough that it does not substantially contribute to the total error. Once again, ensure that at least three simulations have $L_2$ errors in the range $[5\times10^{-3}, 1\times10^{-4}]$, attempting to cover as much of that range as possible/practical. Save the effective mesh size and $L_2$ error for each individual simulation in a CSV or JSON file. [Upload this file to the PFHub website](https://pages.nist.gov/pfhub/simulations/upload_form/) as a 2D data set with the time step size as the x-axis column and the $L_2$ error as the y-axis column. Confirm that the observed order of accuracy is approximately equal to the expected value.

## Part (b)
Now that your code has been verified in (a), the objective of this part is to determine the computational performance of your code at various levels of error. These results can then be used to objectively compare the performance between codes or settings within the same code. To make the problem more computationally demanding and stress solvers more than in (a), decrease $\kappa$ by a factor of $256$ to $1.5625\times10^{-6}$. This change will reduce the interfacial thickness by a factor of $16$.

Run a series of simulations, attempting to optimize solver parameters (mesh, time step, tolerances, etc.) to minimize the required computational resources for at least three levels of $L_2$ error in range  $[5\times10^{-3}, 1\times10^{-5}]$. Use the same CPU and processor type for all simulations. For the best of these simulations, save the wall time, number of computing cores, maximum memory usage, and $L_2$ error for each individual simulation in a CSV or JSON file. [Upload this to the PFHub website](https://pages.nist.gov/pfhub/simulations/upload_form/) as a 3D data set with the wall time as the x-axis column, the number of computing cores as the y-axis column, and the $L_2$ error as the z-axis column. (The PFHub upload system is currently limited to three columns of data. Once this constraint is relaxed, the maximum memory usage data will be incorporated as well.)

<!---For the best of these simulations, submit the wall time, number of computing cores, processor speed, maximum memory usage, and $L_2$ error at $t=8$ to the CHiMaD website.--->

## Part (c)
This final part is designed to stress time integrators even further by increasing the rate of change of $\alpha(x,t)$. Increase $C_2$ to $0.5$. Keep $\kappa= 1.5625\times10^{-6}$ from (b).

Repeat the process from (b), uploading the wall time, number of computing cores, processor speed, maximum memory usage, and $L_2$ error at $t=8$ to the PFHub website.

# Results
Results from this benchmark problem are displayed on the [simulation result page]({{ site.baseurl }}/simulations) for different codes.

# Feedback
Feedback on this benchmark problem is appreciated. If you have questions, comments, or seek clarification, please contact the [CHiMaD phase field community](https://pages.nist.gov/chimad-phase-field/community/) through the [Gitter chat channel](https://gitter.im/usnistgov/chimad-phase-field) or by [email](https://pages.nist.gov/chimad-phase-field/mailing_list/). If you found an error, please file an [issue on GitHub](https://github.com/usnistgov/chimad-phase-field/issues/new).

# Appendix

## Computer algebra systems
Rigorous verification of software frameworks using MMS requires posing the equation and manufacturing the solution with as much complexity as possible. This can be straight-forward, but interesting equations produce complicated source terms. To streamline the MMS workflow, it is strongly recommended that you use a CAS such as SymPy, Maple, or Mathematica to generate source equations and turn it into executable code automatically. For accessibility, we will use [SymPy](http://www.sympy.org/), but so long as vector calculus is supported, and CAS will do.

## Source term

```python
from sympy import Symbol, symbols, simplify
from sympy import Eq, sin, cos, cosh, sinh, tanh, sqrt
from sympy.physics.vector import divergence, gradient, ReferenceFrame, time_derivative
from sympy.printing import pprint
from sympy.abc import kappa, S, t, x, y

# Spatial coordinates: x=R[0], y=R[1], z=R[2]
R = ReferenceFrame('R')

# sinusoid amplitudes
A1, A2 = symbols('A1 A2')
B1, B2 = symbols('B1 B2')
C1, C2 = symbols('C1 C2')

# Define interface offset (alpha)
alpha = (1/4 + A1 * t * sin(B1 * R[0])
             + A2 * sin(B2 * R[0] + C2 * t)
        ).subs({R[0]: x, R[1]: y})

# Define the solution equation (eta)
eta = (1/2 * (1 - tanh((R[1] - alpha) /
                        sqrt(2*kappa)))
      ).subs({R[0]: x, R[1]: y})

# Compute the initial condition
eta0 = eta.subs({t: 0, R[0]: x, R[1]: y})

# Compute the source term from the equation of motion
S = simplify(time_derivative(eta, R)
             + 4 * eta * (eta - 1) * (eta - 1/2)
             - divergence(kappa * gradient(eta, R), R)
            ).subs({R[0]: x, R[1]: y})
```

```python
pprint(Eq(symbols('alpha'), alpha))
```

    α = A₁⋅t⋅sin(B₁⋅x) + A₂⋅sin(B₂⋅x + C₂⋅t) + 0.25

```python
pprint(Eq(symbols('eta'), eta))
```

                  ⎛√2⋅(-A₁⋅t⋅sin(B₁⋅x) - A₂⋅sin(B₂⋅x + C₂⋅t) + y - 0.25)⎞
    η = - 0.5⋅tanh⎜─────────────────────────────────────────────────────⎟ + 0.5
                  ⎝                         2⋅√κ                        ⎠

```python
pprint(Eq(symbols('eta0'), eta0))
```

                   ⎛√2⋅(-A₂⋅sin(B₂⋅x) + y - 0.25)⎞
    η₀ = - 0.5⋅tanh⎜─────────────────────────────⎟ + 0.5
                   ⎝             2⋅√κ            ⎠

```python
pprint(Eq(symbols('S'), S))
```

        ⎛           ⎛√2⋅(A₁⋅t⋅sin(B₁⋅x) + A₂⋅sin(B₂⋅x + C₂⋅t) - y + 0.25)⎞
        ⎜0.5⋅√κ⋅tanh⎜────────────────────────────────────────────────────⎟ - 0.25⋅
        ⎝           ⎝                        2⋅√κ                        ⎠
    S = ──────────────────────────────────────────────────────────────────────────

                                              ⎞ ⎛    2⎛√2⋅(A₁⋅t⋅sin(B₁⋅x) + A₂⋅sin
    √2⋅(A₁⋅sin(B₁⋅x) + A₂⋅C₂⋅cos(B₂⋅x + C₂⋅t))⎟⋅⎜tanh ⎜───────────────────────────
                                              ⎠ ⎝     ⎝                        2⋅√
    ──────────────────────────────────────────────────────────────────────────────
                     √κ

    (B₂⋅x + C₂⋅t) - y + 0.25)⎞    ⎞
    ─────────────────────────⎟ - 1⎟
    κ                        ⎠    ⎠
    ───────────────────────────────

## Code

### Python

Copy the first cell under Source Term directly into your program.
For a performance boost, convert the expressions into lambda functions:
```python
from sympy.utilities.lambdify import lambdify

apy = lambdify([x, y], alpha, modules='sympy')
epy = lambdify([x, y], eta,   modules='sympy')
ipy = lambdify([x, y], eta0,  modules='sympy')
Spy = lambdify([x, y], S,     modules='sympy')
```
#### *N.B.*: You may need to add coefficients to the variables list.

### C

```python
from sympy.utilities.codegen import codegen

[(c_name, c_code), (h_name, c_header)] = codegen([('alpha', alpha),
                                                  ('eta', eta),
                                                  ('eta0', eta0),
                                                  ('S', S)],
                                                 language='C', prefix='MMS', project='PFHub')
print(c_code)
```

    /******************************************************************************
     *                      Code generated with sympy 1.1.1                       *
     *                                                                            *
     *              See http://www.sympy.org/ for more information.               *
     *                                                                            *
     *                        This file is part of 'PFHub'                        *
     ******************************************************************************/
    #include "MMS.h"
    #include <math.h>

    double alpha(double A1, double A2, double B1, double B2, double C2, double t, double x) {

       double alpha_result;
       alpha_result = A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) + 0.25;
       return alpha_result;

    }

    double eta(double A1, double A2, double B1, double B2, double C2, double kappa, double t, double x, double y) {

       double eta_result;
       eta_result = -0.5*tanh((1.0L/2.0L)*sqrt(2)*(-A1*t*sin(B1*x) - A2*sin(B2*x + C2*t) + y - 0.25)/sqrt(kappa)) + 0.5;
       return eta_result;

    }

    double eta0(double A2, double B2, double kappa, double x, double y) {

       double eta0_result;
       eta0_result = -0.5*tanh((1.0L/2.0L)*sqrt(2)*(-A2*sin(B2*x) + y - 0.25)/sqrt(kappa)) + 0.5;
       return eta0_result;

    }

    double S(double A1, double A2, double B1, double B2, double C2, double kappa, double t, double x, double y) {

       double S_result;
       S_result = (0.5*sqrt(kappa)*tanh((1.0L/2.0L)*sqrt(2)*(A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) - y + 0.25)/sqrt(kappa)) - 0.25*sqrt(2)*(A1*sin(B1*x) + A2*C2*cos(B2*x + C2*t)))*(pow(tanh((1.0L/2.0L)*sqrt(2)*(A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) - y + 0.25)/sqrt(kappa)), 2) - 1)/sqrt(kappa);
       return S_result;

    }

### C++

```python
from sympy.printing.cxxcode import cxxcode
```

```python
print("α:")
cxxcode(alpha)
```

    α:

    'A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) + 0.25'

```python
print("η:")
cxxcode(eta)
```

    η:

    '-0.5*tanh((1.0L/2.0L)*std::sqrt(2)*(-A1*t*sin(B1*x) - A2*sin(B2*x + C2*t) + y - 0.25)/std::sqrt(kappa)) + 0.5'

```python
print("η₀:")
cxxcode(eta0)
```

    η₀:

    '-0.5*tanh((1.0L/2.0L)*std::sqrt(2)*(-A2*sin(B2*x) + y - 0.25)/std::sqrt(kappa)) + 0.5'

```python
print("S:")
cxxcode(S)
```

    S:

    '(0.5*std::sqrt(kappa)*tanh((1.0L/2.0L)*std::sqrt(2)*(A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) - y + 0.25)/std::sqrt(kappa)) - 0.25*std::sqrt(2)*(A1*sin(B1*x) + A2*C2*cos(B2*x + C2*t)))*(std::pow(tanh((1.0L/2.0L)*std::sqrt(2)*(A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) - y + 0.25)/std::sqrt(kappa)), 2) - 1)/std::sqrt(kappa)'

### Fortran

```python
from sympy.printing import fcode
```

```python
print("α:")
fcode(alpha)
```

    α:

    '      A1*t*sin(B1*x) + A2*sin(B2*x + C2*t) + 0.25d0'

```python
print("η:")
fcode(eta)
```

    η:

    '      -0.5d0*tanh(0.707106781186548d0*kappa**(-0.5d0)*(-A1*t*sin(B1*x) -\n     @ A2*sin(B2*x + C2*t) + y - 0.25d0)) + 0.5d0'

```python
print("η₀:")
fcode(eta0)
```

    η₀:

    '      -0.5d0*tanh(0.707106781186548d0*kappa**(-0.5d0)*(-A2*sin(B2*x) + y\n     @ - 0.25d0)) + 0.5d0'

```python
print("S:")
fcode(S)
```

    S:

    '      (0.5d0*sqrt(kappa)*tanh(0.707106781186548d0*kappa**(-0.5d0)*(A1*t*\n     @ sin(B1*x) + A2*sin(B2*x + C2*t) - y + 0.25d0)) - 0.25d0*sqrt(\n     @ 2.0d0)*(A1*sin(B1*x) + A2*C2*cos(B2*x + C2*t)))*(tanh(\n     @ 0.707106781186548d0*kappa**(-0.5d0)*(A1*t*sin(B1*x) + A2*sin(B2*x\n     @ + C2*t) - y + 0.25d0))**2 - 1)/sqrt(kappa)'

### Julia

```python
from sympy.printing import julia_code
```

```python
print("α:")
julia_code(alpha)
```

    α:

    'A1.*t.*sin(B1.*x) + A2.*sin(B2.*x + C2.*t) + 0.25'

```python
print("η:")
julia_code(eta)
```

    η:

    '-0.5*tanh(sqrt(2)*(-A1.*t.*sin(B1.*x) - A2.*sin(B2.*x + C2.*t) + y - 0.25)./(2*sqrt(kappa))) + 0.5'

```python
print("η₀:")
julia_code(eta0)
```

    η₀:

    '-0.5*tanh(sqrt(2)*(-A2.*sin(B2.*x) + y - 0.25)./(2*sqrt(kappa))) + 0.5'

```python
print("S:")
julia_code(S)
```

    S:

    '(0.5*sqrt(kappa).*tanh(sqrt(2)*(A1.*t.*sin(B1.*x) + A2.*sin(B2.*x + C2.*t) - y + 0.25)./(2*sqrt(kappa))) - 0.25*sqrt(2)*(A1.*sin(B1.*x) + A2.*C2.*cos(B2.*x + C2.*t))).*(tanh(sqrt(2)*(A1.*t.*sin(B1.*x) + A2.*sin(B2.*x + C2.*t) - y + 0.25)./(2*sqrt(kappa))).^2 - 1)./sqrt(kappa)'

### Mathematica

```python
from sympy.printing import mathematica_code
```

```python
print("α:")
mathematica_code(alpha)
```

    α:

    'A1*t*Sin[B1*x] + A2*Sin[B2*x + C2*t] + 0.25'

```python
print("η:")
mathematica_code(eta)
```

    η:

    '-0.5*Tanh[(1/2)*2^(1/2)*(-A1*t*Sin[B1*x] - A2*Sin[B2*x + C2*t] + y - 0.25)/kappa^(1/2)] + 0.5'

```python
print("η₀:")
mathematica_code(eta0)
```

    η₀:

    '-0.5*Tanh[(1/2)*2^(1/2)*(-A2*Sin[B2*x] + y - 0.25)/kappa^(1/2)] + 0.5'

```python
print("S:")
mathematica_code(S)
```

    S:

    '(0.5*kappa^(1/2)*Tanh[(1/2)*2^(1/2)*(A1*t*Sin[B1*x] + A2*Sin[B2*x + C2*t] - y + 0.25)/kappa^(1/2)] - 0.25*2^(1/2)*(A1*Sin[B1*x] + A2*C2*Cos[B2*x + C2*t]))*(Tanh[(1/2)*2^(1/2)*(A1*t*Sin[B1*x] + A2*Sin[B2*x + C2*t] - y + 0.25)/kappa^(1/2)]^2 - 1)/kappa^(1/2)'

```python

```


# **Save this file as studentid1_studentid2_lab2.ipynb**, please check this suffix when you upload your lab, especially when you have multiple copy's in the same folder!
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab2.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All). Note, that **you are not allowed to use Google Colab**.

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Philipp Lintl"
NAME2 = "Anca Diana Vicol"
EMAIL = "lintl.philipp@gmail.com"
EMAIL2 = "ancadianavicol@gmail.com"

```

# Lab 2: Classification

### Machine Learning 1, November 2018

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Use the provided test cells to check if your answers are correct
* **Make sure your output and plots are correct before handing in your assignment with Kernel -> Restart & Run All**

* **If possible, all your implementations should be vectorized and rely on loops as little as possible. Therefore for some questions, we give you a maximum number of loops that are necessary for an efficient implementation. This number refers to the loops in this particular function and does not count the ones in functions that are called from the function. You should not go above this number for the maximum number of points.**

$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bm}{\mathbf{m}}$
$\newcommand{\bb}{\mathbf{b}}$
$\newcommand{\bS}{\mathbf{S}}$
$\newcommand{\ba}{\mathbf{a}}$
$\newcommand{\bz}{\mathbf{z}}$
$\newcommand{\bv}{\mathbf{v}}$
$\newcommand{\bq}{\mathbf{q}}$
$\newcommand{\bp}{\mathbf{p}}$
$\newcommand{\bh}{\mathbf{h}}$
$\newcommand{\bI}{\mathbf{I}}$
$\newcommand{\bX}{\mathbf{X}}$
$\newcommand{\bT}{\mathbf{T}}$
$\newcommand{\bPhi}{\mathbf{\Phi}}$
$\newcommand{\bW}{\mathbf{W}}$
$\newcommand{\bV}{\mathbf{V}}$

```python
%pylab inline
plt.rcParams["figure.figsize"] = [9,5]

import time
start = time.time()
```

    Populating the interactive namespace from numpy and matplotlib

    /home/anca/miniconda3/envs/ml1labs/lib/python3.6/site-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['indices', 'tanh']
    `%matplotlib` prevents importing * from pylab and numpy
      "\n`%matplotlib` prevents importing * from pylab and numpy"

```python
# This cell makes sure that you have all the necessary libraries installed

import sys
import platform
from importlib.util import find_spec, module_from_spec

def check_newer_version(version_inst, version_nec):
    version_inst_split = version_inst.split('.')
    version_nec_split = version_nec.split('.')
    for i in range(min(len(version_inst_split), len(version_nec_split))):
        if int(version_nec_split[i]) > int(version_inst_split[i]):
            return False
        elif int(version_nec_split[i]) < int(version_inst_split[i]):
            return True
    return True

module_list = [('jupyter', '1.0.0'),
               ('matplotlib', '2.0.2'),
               ('numpy', '1.13.1'),
               ('python', '3.6.2'),
               ('sklearn', '0.19.0'),
               ('scipy', '0.19.1'),
               ('nb_conda', '2.2.1')]

packages_correct = True
packages_errors = []

for module_name, version in module_list:
    if module_name == 'scikit-learn':
        module_name = 'sklearn'
    if module_name == 'pyyaml':
        module_name = 'yaml'
    if 'python' in module_name:
        python_version = platform.python_version()
        if not check_newer_version(python_version, version):
            packages_correct = False
            error = f'Update {module_name} to version {version}. Current version is {python_version}.'
            packages_errors.append(error)
            print(error)
    else:
        spec = find_spec(module_name)
        if spec is None:
            packages_correct = False
            error = f'Install {module_name} with version {version} or newer, it is required for this assignment!'
            packages_errors.append(error)
            print(error)
        else:
            x =__import__(module_name)
            if hasattr(x, '__version__') and not check_newer_version(x.__version__, version):
                packages_correct = False
                error = f'Update {module_name} to version {version}. Current version is {x.__version__}.'
                packages_errors.append(error)
                print(error)

try:
    from google.colab import drive
    packages_correct = False
    error = """Please, don't use google colab!
It will make it much more complicated for us to check your homework as it merges all the cells into one."""
    packages_errors.append(error)
    print(error)
except:
    pass

packages_errors = '\n'.join(packages_errors)
```

# Part 1. Multiclass logistic regression

Scenario: you have a friend with one big problem: she's completely blind. You decided to help her: she has a special smartphone for blind people, and you are going to develop a mobile phone app that can do _machine vision_ using the mobile camera: converting a picture (from the camera) to the meaning of the image. You decide to start with an app that can read handwritten digits, i.e. convert an image of handwritten digits to text (e.g. it would enable her to read precious handwritten phone numbers).

A key building block for such an app would be a function `predict_digit(x)` that returns the digit class of an image patch $\bx$. Since hand-coding this function is highly non-trivial, you decide to solve this problem using machine learning, such that the internal parameters of this function are automatically learned using machine learning techniques.

The dataset you're going to use for this is the MNIST handwritten digits dataset (`http://yann.lecun.com/exdb/mnist/`). You can download the data with scikit learn, and load it as follows:

```python
from sklearn.datasets import fetch_mldata
import os
# Fetch the data
try:
    mnist = fetch_mldata('MNIST original', data_home='.')
except Exception:
    raise FileNotFoundError('Please download mnist-original.mat from Canvas and put it in %s/mldata' % os.getcwd())
data, target = mnist.data, mnist.target.astype('int')
# Shuffle
indices = np.arange(len(data))
np.random.seed(123)
np.random.shuffle(indices)
data, target = data[indices].astype('float32'), target[indices]

# Normalize the data between 0.0 and 1.0:
data /= 255.

# Split
x_train, x_valid, x_test = data[:50000], data[50000:60000], data[60000: 70000]
t_train, t_valid, t_test = target[:50000], target[50000:60000], target[60000: 70000]
```

MNIST consists of small 28 by 28 pixel images of written digits (0-9). We split the dataset into a training, validation and testing arrays. The variables `x_train`, `x_valid` and `x_test` are $N \times M$ matrices, where $N$ is the number of datapoints in the respective set, and $M = 28^2 = 784$ is the dimensionality of the data. The second set of variables `t_train`, `t_valid` and `t_test` contain the corresponding $N$-dimensional vector of integers, containing the true class labels.

Here's a visualisation of the first 8 digits of the trainingset:

```python
def plot_digits(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='Greys')
        if targets is not None:
            plt.title(int(targets[i]))
        plt.colorbar()
        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits(x_train[0:40000:5000], num_cols=4, targets=t_train[0:40000:5000])
```

In _multiclass_ logistic regression, the conditional probability of class label $j$ given the image $\bx$ for some datapoint is given by:

$ \log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $\log q_j = \bw_j^T \bx + b_j$ (the log of the unnormalized probability of the class $j$), and $Z = \sum_k q_k$ is the normalizing factor. $\bw_j$ is the $j$-th column of $\bW$ (a matrix of size $784 \times 10$) corresponding to the class label, $b_j$ is the $j$-th element of $\bb$.

Given an input image, the multiclass logistic regression model first computes the intermediate vector $\log \bq$ (of size $10 \times 1$), using $\log q_j = \bw_j^T \bx + b_j$, containing the unnormalized log-probabilities per class.

The unnormalized probabilities are then normalized by $Z$ such that $\sum_j p_j = \sum_j \exp(\log p_j) = 1$. This is done by $\log p_j = \log q_j - \log Z$ where $Z = \sum_i \exp(\log q_i)$. This is known as the _softmax_ transformation, and is also used as a last layer of many classifcation neural network models, to ensure that the output of the network is a normalized distribution, regardless of the values of second-to-last layer ($\log \bq$)

**Warning**: when computing $\log Z$, you are likely to encounter numerical problems. Save yourself countless hours of debugging and learn the [log-sum-exp trick](https://www.xarg.org/2016/06/the-log-sum-exp-trick-in-machine-learning/ "Title").

The network's output $\log \bp$ of size $10 \times 1$ then contains the conditional log-probabilities $\log p(t = j \;|\; \bx, \bb, \bW)$ for each digit class $j$. In summary, the computations are done in this order:

$\bx \rightarrow \log \bq \rightarrow Z \rightarrow \log \bp$

Given some dataset with $N$ independent, identically distributed datapoints, the log-likelihood is given by:

$ \mathcal{L}(\bb, \bW) = \sum_{n=1}^N \mathcal{L}^{(n)}$

where we use $\mathcal{L}^{(n)}$ to denote the partial log-likelihood evaluated over a single datapoint. It is important to see that the log-probability of the class label $t^{(n)}$ given the image, is given by the $t^{(n)}$-th element of the network's output $\log \bp$, denoted by $\log p_{t^{(n)}}$:

$\mathcal{L}^{(n)} = \log p(t = t^{(n)} \;|\; \bx = \bx^{(n)}, \bb, \bW) = \log p_{t^{(n)}} = \log q_{t^{(n)}} - \log Z^{(n)}$

where $\bx^{(n)}$ and $t^{(n)}$ are the input (image) and class label (integer) of the $n$-th datapoint, and $Z^{(n)}$ is the normalizing constant for the distribution over $t^{(n)}$.

## 1.1 Gradient-based stochastic optimization
### 1.1.1 Derive gradient equations (20 points)

Derive the equations for computing the (first) partial derivatives of the log-likelihood w.r.t. all the parameters, evaluated at a _single_ datapoint $n$.

You should start deriving the equations for $\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$ for each $j$. For clarity, we'll use the shorthand $\delta^q_j = \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$.

For $j = t^{(n)}$:
$$
\delta^q_j
= \frac{\partial \log q_{t^{(n)}}}{\partial \log q_j}
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
= 1
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$$

For $j \neq t^{(n)}$:
$$
\delta^q_j
= \frac{\partial \log q_{t^{(n)}}}{\partial \log q_j}
-
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
=0 - \frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$$

Complete the above derivations for $\delta^q_j$ by furtherly developing $\frac{\partial \log Z}{\partial Z}$ and $\frac{\partial Z}{\partial \log q_j}$. Both are quite simple. For these it doesn't matter whether $j = t^{(n)}$ or not.

For $j = t^{(n)}$:
\begin{align}
\delta^q_j
&=1-\frac{exp(log(q_j))}{\sum_iexp(log(q_i))}=1-\frac{q_j}{\sum_iq_i}
\end{align}
For $j \neq t^{(n)}$:
\begin{align}
\delta^q_j
&= 0-\frac{exp(log(q_j))}{\sum_iexp(log(q_i))}=-\frac{q_j}{\sum_iq_i},
\end{align}

as $\frac{\partial \log Z}{\partial Z}=\frac{1}{Z}$ and
$\frac{\partial Z}{\partial \log q_j}=\frac{\partial \sum_iexp(log(q_i))}{\partial \log q_j}=exp(log(q_j))$

Given your equations for computing the gradients $\delta^q_j$ it should be quite straightforward to derive the equations for the gradients of the parameters of the model, $\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}}$ and $\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}$. The gradients for the biases $\bb$ are given by:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}
= \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial b_j}
= \delta^q_j
\cdot 1
= \delta^q_j
$

The equation above gives the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element of $\bb$, so the vector $\nabla_\bb \mathcal{L}^{(n)}$ with all derivatives of $\mathcal{L}^{(n)}$ w.r.t. the bias parameters $\bb$ is:

$
\nabla_\bb \mathcal{L}^{(n)} = \mathbf{\delta}^q
$

where $\mathbf{\delta}^q$ denotes the vector of size $10 \times 1$ with elements $\mathbf{\delta}_j^q$.

The (not fully developed) equation for computing the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element $W_{ij}$ of $\bW$ is:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q
\frac{\partial \log q_j}{\partial W_{ij}}
$

What is $\frac{\partial \log q_j}{\partial W_{ij}}$? Complete the equation above.

If you want, you can give the resulting equation in vector format ($\nabla_{\bw_j} \mathcal{L}^{(n)} = ...$), like we did for $\nabla_\bb \mathcal{L}^{(n)}$.

It has to be noted that the gradients shapes given have to be obtained by the gradient convention to get column vectors. This is contrary to the entire course so far and caused confusion, which is why we adapted the column convention throughout our formula derivations.

With $\log q_j=\mathbf{w}_j^T\mathbf{x}+b_j=\sum_{i=1}w_{ij}\cdot x_i +b_j$, that's why $\frac{\partial \log q_j}{\partial W_{ij}}=x_i$.

For the entire $\mathbf{w}_j$ vector:
$\frac{\partial \log q_j}{\partial \mathbf{w}_{j}}=\frac{\partial (\mathbf{w}_j^T\mathbf{x}^{(n)}+b_j)}{\partial \mathbf{w}_{j}}
= \mathbf{x}^{(n)},\quad \in \mathbb{R}^{784\times 1}$

So, for $j=t^{(n)}$:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q\cdot x_i^{(n)}\,\,\Rightarrow \frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{w_{j}}}=\nabla_{w_j}\mathcal{L}^{(n)}=\delta_j^{q}\cdot\mathbf{x}^{(n)}
$

and for $j\neq t^{(n)}$:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial W_{ij}}
= (1-\mathbf{\delta}_j^q)\cdot x_i^{(n)}\,\,\Rightarrow \frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{w_{j}}}=(1-\delta_j^{q})\cdot\mathbf{x}^{(n)}
$

If we aggregate these columns for all j's, we end up with a $\mathbb{R}^{784\times 10}$ matrix, which has the $\frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{w}_{j}}$ as columns and can be obtained by matrix multiplication.

### 1.1.2 Implement gradient computations (15 points)

Implement the gradient calculations you derived in the previous question. Write a function `logreg_gradient(x, t, w, b)` that returns the gradients $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (for each $j$) and $\nabla_{\bb} \mathcal{L}^{(n)}$, i.e. the first partial derivatives of the log-likelihood w.r.t. the parameters $\bW$ and $\bb$, evaluated at a single datapoint (`x`, `t`).
The computation will contain roughly the following intermediate variables:

$
\log \bq \rightarrow Z \rightarrow \log \bp\,,\, \mathbf{\delta}^q
$

followed by computation of the gradient vectors $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (contained in a $784 \times 10$ matrix) and $\nabla_{\bb} \mathcal{L}^{(n)}$ (a $10 \times 1$ vector).

For maximum points, ensure the function is numerically stable.

```python
# 1.1.2 Compute gradient of log p(t|x;w,b) wrt w and b
def log_probability(x, t, w, b):
    #
    logq = numpy.matmul(x,w) + b.reshape(1,10)
    # ensures numerical stability
    a = numpy.max(logq)
    logZ = a + numpy.log(numpy.sum(numpy.exp(logq - a)))
    Z = numpy.exp(logZ)
    logp = logq - logZ
    #print(logZ.shape, Z.shape, logp.shape)
    return logq, logp, Z

def logreg_gradient(x, t, w, b):
    logq, logp, Z = log_probability(x, t, w, b)

    # Use one-hot-ecoding for the target.
    t_vec = numpy.zeros(w.shape[1])
    t_vec[t] = 1
    dL_dlogq = t_vec.reshape(1,10) - (1/Z) * numpy.exp(logq)

    dL_dw = numpy.matmul(numpy.transpose(x), dL_dlogq)
    dL_db = dL_dlogq.reshape(10,1)

    # here the statement contains logp[:,t] where logp is meant tas a matrix of shape 1x10
    return logp[:,t].squeeze(), dL_dw, dL_db.squeeze()
```

```python
# Hidden tests for efficiency
```

```python
np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

# test gradients, train on 1 sample
logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)

print("Test gradient on one point")
print("Log Likelihood:\t", logpt)
print("\nGrad_W_ij\t",grad_w.shape,"matrix")
print("Grad_W_ij[0,152:158]=\t", grad_w[152:158,0])
print("\nGrad_B_i shape\t",grad_b.shape,"vector")
print("Grad_B_i=\t", grad_b.T)
print("i in {0,...,9}; j in M")

assert logpt.shape == (), logpt.shape
assert grad_w.shape == (784, 10), grad_w.shape
assert grad_b.shape == (10,), grad_b.shape

```

    Test gradient on one point
    Log Likelihood:	 -2.2959726720744777

    Grad_W_ij	 (784, 10) matrix
    Grad_W_ij[0,152:158]=	 [-0.04518971 -0.06758809 -0.07819784 -0.09077237 -0.07584012 -0.06365855]

    Grad_B_i shape	 (10,) vector
    Grad_B_i=	 [-0.10020327 -0.09977827 -0.1003198   0.89933657 -0.10037941 -0.10072863
     -0.09982729 -0.09928672 -0.09949324 -0.09931994]
    i in {0,...,9}; j in M

```python
# It's always good to check your gradient implementations with finite difference checking:
# Scipy provides the check_grad function, which requires flat input variables.
# So we write two helper functions that provide the gradient and output with 'flat' weights:
from scipy.optimize import check_grad

np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

def func(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return logpt
def grad(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return grad_w.flatten()
finite_diff_error = check_grad(func, grad, w.flatten())
print('Finite difference error grad_w:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for w seems off'

def func(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return logpt
def grad(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return grad_b.flatten()
finite_diff_error = check_grad(func, grad, b)
print('Finite difference error grad_b:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for b seems off'

```

    Finite difference error grad_w: 6.411502515218292e-07
    Finite difference error grad_b: 5.235117487930228e-08

```python
# DO NOT REMOVE THIS CELL!
# It contains hidden tests

```

### 1.1.3 Stochastic gradient descent (15 points)

Write a function `sgd_iter(x_train, t_train, w, b)` that performs one iteration of stochastic gradient descent (SGD), and returns the new weights. It should go through the trainingset once in randomized order, call `logreg_gradient(x, t, w, b)` for each datapoint to get the gradients, and update the parameters **using a small learning rate of `1e-6`**. Note that in this case we're maximizing the likelihood function, so we should actually performing gradient ___ascent___... For more information about SGD, see Bishop 5.2.4 or an online source (i.e. https://en.wikipedia.org/wiki/Stochastic_gradient_descent)

```python
def sgd_iter(x_train, t_train, W, b):
    (N, D) = x_train.shape
    learning_rate = 10**(-4)
    random_iterator = list(range(len(x_train)))
    random.shuffle(random_iterator)
    logp_train = []

    for index in random_iterator:
        logp, grad_w, grad_b = logreg_gradient(x_train[index].reshape(1, D), t_train[index], W, b)
        W = W + learning_rate * grad_w
        b = b + learning_rate * grad_b
        logp_train += [logp]
    return logp_train, W, b

# helper function to calculate logps of validation set
def valid_logp(x_valid, t_valid, W, b):
    logp_valid = []
    true_log_p = []
    for index in range(len(x_valid)):
        logq, logp_val, Z = log_probability(x_valid[index].reshape(1,784), t_valid[index], W, b)
        logp_valid += [logp_val]
        true_log_p += [logp_val[:,t_valid[index]]]
    return logp_valid, true_log_p

```

```python
# Hidden tests for efficiency
```

```python
# Sanity check:
np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)

logp_train, W, b = sgd_iter(x_train[:5], t_train[:5], w, b)

```

## 1.2. Train

### 1.2.1 Train (12 points)
Perform SGD on the training set. Plot (in one graph) the conditional log-probability of the training set and validation set after each iteration. (6 points)

Instead of running SGD for a fixed number of steps, run it until convergence. Think of a reasonable criterion for determining convergence. As a reference: choose a criterion such that the algorithm terminates in less than 15 iterations over the training set. (2 points)

Make sure your implementation (in particular, the output of the conditional log-probability of the training set and validation set) is independent of the size of the dataset. (2 points)

```python
# epsilon chosen thus it converges ~ 14 iterations (still < 15) and yields
# 'clearer' weights
def test_sgd(x_train, t_train, x_valid, t_valid, w, b):
    epsilon = 0.005
    log_p = -2
    last_log_p = -3
    log_prob_train = []
    log_prob_valid = []
    log_p_valid = []
    counter = 0
    while numpy.abs(log_p - last_log_p) > epsilon:
        last_log_p = log_p #otherwise convergence criterion does not apply
        log_p, w, b = sgd_iter(x_train, t_train, w, b)
        log_p = mean(log_p)
        log_prob_train += [log_p]
        # same for valid set
        not_imp, log_p_valid = valid_logp(x_valid, t_valid, w, b)
        log_p_valid = mean(log_p_valid)
        log_prob_valid += [log_p_valid]

        counter += 1
        if counter > 20:
            break
    plt.plot(range(counter), log_prob_train, label='training data')
    plt.plot(range(counter), log_prob_valid, label='test data')
    plt.legend()
    plt.title("average log-likelihood per iteration for both training and test data")
    plt.show()
    return w, b

np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)
w,b = test_sgd(x_train, t_train, x_valid, t_valid, w, b)

```

```python
# Hidden tests for efficiency
```

### 1.2.2 Visualize weights (10 points)
Visualize the resulting parameters $\bW$ after a few iterations through the training set, by treating each column of $\bW$ as an image. If you want, you can use or edit the `plot_digits(...)` above.

```python
def plot_digits_indiv(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='jet')
        if targets is not None:
            plt.title(int(targets[i]))
        plt.colorbar()
        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits_indiv(numpy.transpose(w), 5, targets=None, shape=(28,28))

```

**Describe in less than 100 words why these weights minimize the loss**

The weights are clearly showing the shape of the digits.
There seems to be high contrast in the center, the weights directly linked to the current digit's pixels are quite high while the weights around it are very low (reaching negative values). These low weights are used to discriminate between the different digits.

The outer part of all the weights are around zero meaning that the model is not concerned with the pixels there, which makes sense considering we are expecting those to be simply white background.

### 1.2.3. Visualize the 8 hardest and 8 easiest digits (10 points)
Visualize the 8 digits in the validation set with the highest probability of the true class label under the model.
Also plot the 8 digits that were assigned the lowest probability.

```python
# gets logp of each class for each dataoṕoint of the validation set
log_p, not_imp = valid_logp(x_valid, t_valid, w, b)

# gets logp of the true class (t_valid)
prob_true_class = [(log_p[i][0][t_valid[i]], i) for i in range(len(log_p))]
# Sorts this list and keeps indexnumber to match to original datapoint
sorted_probabilities = sorted(prob_true_class, key=lambda pair: pair[0])

# take highest 8
easiest_examples = numpy.array([x_valid[i] for (_, i) in sorted_probabilities[-8:]])
easiest_targets = numpy.array([t_valid[i] for (_, i) in sorted_probabilities[-8:]])
# take lowest 8
hardest_examples = numpy.array([x_valid[i] for (_, i) in sorted_probabilities[:8]])
hardest_targets = numpy.array([t_valid[i] for (_, i) in sorted_probabilities[:8]])

easiest_examples_targets = numpy.array([t_valid[i] for (_, i) in sorted_probabilities[-8:]])
hardest_examples_targets = numpy.array([t_valid[i] for (_, i) in sorted_probabilities[:8]])

print("Easiest digits in validation set.")
plot_digits_indiv(easiest_examples, 4, targets=easiest_examples_targets, shape=(28,28))
print("Hardest digits in validation set.")
plot_digits_indiv(hardest_examples, 4, targets=hardest_examples_targets, shape=(28,28))
```

Ask yourself if these results make sense. Explain in no more then two sentences what it means that a digit is hard to classify.

A digit will be hard to clasify if it is either different from the other examples of its kind and/or similar to examples of other digits. The results make sense.

# Part 2. Multilayer perceptron

You discover that the predictions by the logistic regression classifier are not good enough for your application: the model is too simple. You want to increase the accuracy of your predictions by using a better model. For this purpose, you're going to use a multilayer perceptron (MLP), a simple kind of neural network. The perceptron will have a single hidden layer $\bh$ with $L$ elements. The parameters of the model are $\bV$ (connections between input $\bx$ and hidden layer $\bh$), $\ba$ (the biases/intercepts of $\bh$), $\bW$ (connections between $\bh$ and $\log q$) and $\bb$ (the biases/intercepts of $\log q$).

The conditional probability of the class label $j$ is given by:

$\log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $q_j$ are again the unnormalized probabilities per class, and $Z = \sum_j q_j$ is again the probability normalizing factor. Each $q_j$ is computed using:

$\log q_j = \bw_j^T \bh + b_j$

where $\bh$ is a $L \times 1$ vector with the hidden layer activations (of a hidden layer with size $L$), and $\bw_j$ is the $j$-th column of $\bW$ (a $L \times 10$ matrix). Each element of the hidden layer is computed from the input vector $\bx$ using:

$h_j = \sigma(\bv_j^T \bx + a_j)$

where $\bv_j$ is the $j$-th column of $\bV$ (a $784 \times L$ matrix), $a_j$ is the $j$-th element of $\ba$, and $\sigma(.)$ is the so-called sigmoid activation function, defined by:

$\sigma(x) = \frac{1}{1 + \exp(-x)}$

Note that this model is almost equal to the multiclass logistic regression model, but with an extra 'hidden layer' $\bh$. The activations of this hidden layer can be viewed as features computed from the input, where the feature transformation ($\bV$ and $\ba$) is learned.

## 2.1 Derive gradient equations (20 points)

State (shortly) why $\nabla_{\bb} \mathcal{L}^{(n)}$ is equal to the earlier (multiclass logistic regression) case, and why $\nabla_{\bw_j} \mathcal{L}^{(n)}$ is almost equal to the earlier case.

Like in multiclass logistic regression, you should use intermediate variables $\mathbf{\delta}_j^q$. In addition, you should use intermediate variables $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial h_j}$.

Given an input image, roughly the following intermediate variables should be computed:

$
\log \bq \rightarrow Z \rightarrow \log \bp \rightarrow \mathbf{\delta}^q \rightarrow \mathbf{\delta}^h
$

where $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial \bh_j}$.

Give the equations for computing $\mathbf{\delta}^h$, and for computing the derivatives of $\mathcal{L}^{(n)}$ w.r.t. $\bW$, $\bb$, $\bV$ and $\ba$.

You can use the convenient fact that $\frac{\partial}{\partial x} \sigma(x) = \sigma(x) (1 - \sigma(x))$.

1.) $\nabla_\mathbf{b}\mathcal{L}^{(n)}$ stays the same, as $\log q_j=\mathbf{w}_j^T\mathbf{h}+b_j$ and thus $\frac{\partial \log q_j}{\partial b_j}=1$, as the only change to the case from before is the term $\mathbf{h}$, which is independent of $b_j$. The therm $\frac{\partial\mathcal{L}^{(n)}}{\partial \log q_j}$ stays unchanged equal to $\delta_j^q$. So, again $\frac{\partial\mathcal{L}^{(n)}}{\partial b_j}=\delta_j^q\cdot 1$ holds, implying for the gradient of all classes: $\nabla_\mathbf{b}\mathcal{L}^{(n)}=\delta^q\cdot 1\in\mathbb{R}^{10\times 1}$

2.) For $ \nabla_{\mathbf{w}_j}\mathcal{L}^{(n)}$, we look at $\frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{w}_j}$:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{w}_j}=\frac{\partial\mathcal{L}^{(n)}}{\partial \log q_j}\cdot \frac{\partial \log q_j}{\partial \mathbf{w}_j}=\delta_j^q\cdot \mathbf{h}^{(n)}\quad \in\mathbb{R}^{L\times 1}
$, meaning that the gradient stays the same with the replacement of $\mathbf{x}$ with $\mathbf{h}$, the activation values of the hiddenlayer and a different shape, as $\mathbf{h}\in\mathbb{R}^{L\times 1}$

3.) Equations for computing $\delta^h$:

$\delta_j^h=\frac{\partial \mathcal{L}^{(n)}}{\partial \mathbf{h}_j}=\sum_i\frac{\partial\mathcal{L}^{(n)}}{\partial \log q_i}\cdot \frac{\partial \log q_i}{\partial \mathbf{h}_j}=\sum_i\delta_i^q\cdot w_{ji}=\mathbf{w}_j\cdot\delta^q, \quad \in\mathbb{R}^{1\times 10}\cdot\mathbb{R}^{10\times 1}=\mathbb{R}^{1\times 1}$,
where $\mathbf{w}_j$ is the j-th row of $\mathbf{W}$ as a row-vector.

Then, $\forall j \in\{1,\ldots,L\}\quad\Rightarrow \quad \delta^h$ emerges as:

$\delta^h=\mathbf{W}\cdot\delta^q, \quad \in\mathbb{R}^{L\times 10}\cdot\mathbb{R}^{10\times 1}=\mathbb{R}^{L\times 1}$, which is exactly the given shape of $\mathbf{h}$.

4.) Derivative of $\frac{\partial\mathcal{L}^{(n)}}{\partial \mathbf{V}}$:

For a single $j \in\{1,\ldots,L\}$:

$
\nabla_{\mathbf{v}_j}\mathcal{L^{(n)}}= \sum_i\frac{\partial\mathcal{L}^{(n)}}{\partial \log q_i}\cdot \frac{\partial \log q_i}{\partial h_j}\cdot\frac{\partial h_j}{\partial\mathbf{v}_j}=\delta_j^h\cdot\frac{\partial \sigma(\mathbf{v}_j^T\mathbf{x}+a_j)}{\partial \mathbf{v}_j}=\delta_j^h\cdot h_j(1-h_j)\frac{\mathbf{v}_j^T\mathbf{x}+a_j}{\partial \mathbf{v}_j}=\delta_j^h\cdot h_j(1-h_j)\cdot\mathbf{x}^{(n)}
$

5.) Derivative of $\frac{\partial\mathcal{L}^{(n)}}{\partial \mathbf{a}}$:

$
\frac{\partial\mathcal{L}^{(n)}}{\partial a_j}=\delta_j^h\cdot\frac{\partial \sigma(\mathbf{v}_j^T\mathbf{x}+a_j)}{\partial a_j}=\delta_j^h\cdot h_j(1-h_j)\cdot 1 \quad \Rightarrow \quad \frac{\partial\mathcal{L}^{(n)}}{\partial \mathbf{a}}\in\mathbb{R}^{L\times 1}
$

## 2.2 MAP optimization (10 points)

You derived equations for finding the _maximum likelihood_ solution of the parameters. Explain, in a few sentences, how you could extend this approach so that it optimizes towards a _maximum a posteriori_ (MAP) solution of the parameters, with a Gaussian prior on the parameters.

To extend the approach for MAP, we would need to add the logarithm of the prior distribution to the logarithm of the likelihood. After deriving with respect to w and v, we get the loss functions that these parameters should minimize. The loss functions will contain the parameters themselsves, thus leading to minimizing parameters as well. This is regularization.

## 2.3. Implement and train a MLP (15 points)

Implement an MLP model with a single hidden layer of **20 neurons**.
Train the model for **10 epochs**.
Test your implementation for learning rates of 1e-2, 1e-3 and 1e-4 and plot (in one graph) the conditional log-probability of the trainingset and validation set.

For the best model plot the weights of the first layer for in epoch 0,4 and 9.

- 10 points: Working MLP that learns with plots
- +5 points: Fast, numerically stable, vectorized implementation

```python
# Write all helper functions here
def sigmoid(x):
    return 1.0 / (1.0 + numpy.exp(-x))

def iter_mlp(x_train, t_train, learning_rate, V, W, a, b):
    (N, D) = x_train.shape
    random_iterator = list(range(len(x_train)))
    random.shuffle(random_iterator)

    logp_train = []

    for index in random_iterator:
        x = x_train[index]
        h = sigmoid(numpy.matmul(x, V) + a)
        h_input = numpy.transpose(h.reshape((-1, 1)))
        logp, grad_w, grad_b = logreg_gradient(h_input, t_train[index], W, b)

        delta_h =  numpy.matmul(grad_b, numpy.transpose(W))
        partial_grad_h = numpy.multiply(delta_h, \
                                        numpy.multiply(h, (1 - h))).reshape((1, -1))
        grad_v = numpy.matmul(x.reshape((-1, 1)), partial_grad_h)
        grad_a = numpy.multiply(delta_h, \
                                numpy.multiply(h, (1 - h)))

        W = W + learning_rate * grad_w
        b = b + learning_rate * grad_b

        V = V + learning_rate * grad_v
        a = a + learning_rate * grad_a

        logp_train += [logp]
    return mean(logp_train), V, W, a, b

def test_mlp(x, t, V, W, a, b):
    h = sigmoid(numpy.matmul(x, V) + a)
    logp_true_class = []
    for i in range(len(x)):
        logq, logp, Z = log_probability(h[i], t, W, b)
        logp_true_class += [logp[0][t[i]]]
    return numpy.average(logp_true_class)
```

```python
# Hidden tests for efficiency
```

```python
def train_mlp(learning_rate):
    np.random.seed(1243)
    V = np.random.normal(size=(28*28, 20), scale=0.01)
    a = np.random.normal(size=(20, ), scale=0.01)
    W = np.random.normal(size=(20, 10), scale=0.01)
    b = np.random.normal(size=(10, ), scale=0.01)

    logps_train = []
    logps_valid = []
    for epoch in range(10):
        logp_train, V, W, a, b = iter_mlp(x_train, t_train, learning_rate, V, W, a, b)
        logps_train += [logp_train]
        logps_valid += [test_mlp(x_valid, t_valid, V, W, a, b)]
    return logps_train, logps_valid, V, W, a, b

```

```python
# plot the train and validation logp for all three learning rates in one figure
def train_and_plot(learning_rate, color):
        logps_train, logps_valid, V, W, a, b = train_mlp(learning_rate)
        plt.plot(range(10), logps_train, color=color, label="Training; lr=" + str(learning_rate))
        plt.plot(range(10), logps_valid, color=color, dashes=[6, 2], label="Validation; lr=" + str(learning_rate))
        return V, W, a, b

models = {}
for (learning_rate, color) in [(10**(-2), 'b'), (10**(-3), 'r'), (10**(-4), 'g')]:
    V, W, a, b = train_and_plot(learning_rate, color)
    models[learning_rate] = (V, W, a, b)

plt.xlabel("Epoch")
plt.ylabel("Log Likelihood")
plt.tight_layout()
plt.legend()
plt.show()
```

### 2.3.1. Explain the learning curves (5 points)
In less than 80 words, explain the observed behaviour for the different learning rates.

From the graph we conclude that the largest learning rate is the most appropriate.

For the first 2 learning rates, the model reaches close to the optimum point, but than the smaller one is not strong enough to move it as close as the larger one.

With the smallest learning rate the steps are so small that the model does not even come close to the optimum in the giving epochs.

### 2.3.2. Explain the weights (5 points)
In less than 80 words, explain how and why the weights of the hidden layer of the MLP differ from the logistic regression model, and relate this to the stronger performance of the MLP.

```python
# Plot the weights of the first layer for the best model

plot_digits_indiv(numpy.transpose(models[10**(-2)][0]), 5, targets=None, shape=(28,28))
```

Because the model is using more than one layer, and the hidden layer has more than 10 neurons, the weights do not resemble the digits anymore.

This makes sense as with more neurons the model can learn more specific features, like a horizontal line in the middle of the image. These are then used to predict the actual digits in the following layer.

### 2.3.2. Different activation functions (10 points)
In the task above we use a sigmoid as an activation function.
Two other popular choices for activation functions are tanh and the rectified linear unit (ReLU). The ReLU is defined as:

$$f(x) = \max(0.,x)$$

You already derived the derivative of the softmax function above. Here, write down the derivative for both the tanh and the ReLU function. Furthermore, for all three, plot the function and its derivative in a range $x\in[-3,3]$

Write down the derivative of ReLU and tanh w.r.t. their respective argument:

ReLu:

$\frac{\partial f(x)}{\partial x} = \{0$ for $x < 0$; 1 for $x>0$; undefined for $x=0\}$

tanh:

$\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\quad \Rightarrow \quad\frac{\partial \tanh(x)}{\partial x}=\frac{(e^x+e^{-x})(e^x+e^{-x})-(e^x-e^{-x})(e^x-e^{-x})}{(e^x+e^{-x})^2}=1-\frac{(e^x-e^{-x})^2}{(e^x+e^{-x})^2}=1-\tanh^2(x)$

Name two properties that you would like your activation function to have (one sentence each). Why are they important?

1) Non Linearity, as linear activation functions would resemble a simple regression model, which is limited in its power and mostly it is not sufficient to classify linearly especially in complex data situations such as texts or images.

2) Differentiability, as the parameters need to be 'learned', in particaular with a backpropagation step that requires the existence of a gradient and thus the computed activation functions need to be differentiable.

```python
# plot the function and the derivative for the activations sigmoid, tanh and ReLU.

def sigmoid(x):
    return(1/(1+numpy.exp(-x)))

def derivative_sigmoid(x):
    return sigmoid(x)*(1-sigmoid(x))

def tanh(x):
    return((numpy.exp(x)-numpy.exp(-x))/(numpy.exp(x)+numpy.exp(-x)))

def derivative_tanh(x):
    return(1-tanh(x)**2)

def ReLu(x):
    if(x>0):
        return(x)
    else:
        return(0)

def derivative_ReLu(x):
    if(x==0):
        return
    if(x > 0):
        return 1
    else:
        return 0

x = [i / 1000 for i in range(-3000, 3000, 1)]

plt.plot(x, [sigmoid(i) for i in x], label='sigmoid')
plt.plot(x, [tanh(i) for i in x], label='tanh')
plt.plot(x, [ReLu(i) for i in x], label='ReLu')
plt.legend()
plt.title("Activation functions")
plt.show()

plt.plot(x, [derivative_sigmoid(i) for i in x], label='sigmoid')
plt.plot(x, [derivative_tanh(i) for i in x], label='tanh')
plt.plot(x, [derivative_ReLu(i) for i in x], label='ReLu')
plt.legend()
plt.title("Derivatives of activation functions")
plt.show()
```

Now that you plotted the activations and derivatives, which activation do you think is the best? Why would you choose this activation function? For your answer consider what you named as essential properties for an activation function above. Keep your answer short at no more then 3 sentences.

Considering non-linearity and differentiability, clearly Relu is neither and thus in this case ruled out (though very popular for deep neural nets as it combats very small gradients which lead to a stop in updates -the Vanishing Gradient Problem-). Choosing between sigmoid and tanh, we would argue for tanh, as its range incorporates negative values as well and the derivatives range is also larger, thus alowing larger steps to be made.

```python
print('Notebook ran in {:2.3} minutes.'.format((time.time()-start)/60))
```

    Notebook ran in 8.5e+02 minutes.