# Tutorial: optimal piecewise binning with binary target

## Basic

To get us started, let's load a well-known dataset from the UCI repository and transform the data into a ``pandas.DataFrame``.

```python
import numpy as np
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
```

```python
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
```

We choose a variable to discretize and the binary target.

```python
variable = "mean radius"
x = df[variable].values
y = data.target
```

Import and instantiate an ``OptimalPWBinning`` object class and we pass the variable name. The ``OptimalPWBinning`` can **ONLY** handle numerical variables. This differs from the ``OptimalBinning`` object class.

```python
from optbinning import OptimalPWBinning
```

```python
optb = OptimalPWBinning(name=variable)
```

We fit the optimal binning object with arrays ``x`` and ``y``. We set a minimum (lower bound) and maximum (upper bound) probability using arguments ``lb`` and ``ub``. Note that these bounds are optional, but guarantee an event rate between $[0, 1]$ as any probability distribution. As shown later, settings these bounds will increase the problem size thus leading to higher solution times. An alternative approach to bypass these bounds is implemented when transforming to event rate or WoE using functions ``fit`` and ``fit_transform``, where values outside the interval $[0, 1]$ are clipped.

Note that the Weight-of-Evidence is not defined for event rate equal 0 or 1. Therefore, for small datasets, we recommend modifying the interval $[0, 1]$ with a small value, thus using $[\epsilon, 1 - \epsilon]$.

```python
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean radius')

You can check if an optimal solution has been found via the ``status`` attribute:

```python
optb.status
```

    'OPTIMAL'

You can also retrieve the optimal split points via the ``splits`` attribute:

```python
optb.splits
```

    array([11.42500019, 12.32999992, 13.09499979, 13.70499992, 15.04500008,
           16.92500019])

#### The binning table

The optimal binning algorithms return a binning table; a binning table displays the binned data and several metrics for each bin. Class ``OptimalPWBinning`` returns an object ``PWBinningTable`` via the ``binning_table`` attribute.

```python
binning_table = optb.binning_table
```

```python
type(binning_table)
```

    optbinning.binning.piecewise.binning_statistics.PWBinningTable

The `binning_table` is instantiated, but not built. Therefore, the first step is to call the method `build`, which returns a ``pandas.DataFrame``.

```python
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 11.43)
      118
      0.207381
      3
      115
      1.032648
      -0.004820

      1
      [11.43, 12.33)
      79
      0.138840
      3
      76
      1.644837
      -0.058403

      2
      [12.33, 13.09)
      68
      0.119508
      7
      61
      2.140569
      -0.098609

      3
      [13.09, 13.70)
      49
      0.086116
      10
      39
      2.914961
      -0.157745

      4
      [13.70, 15.05)
      83
      0.145870
      28
      55
      4.105273
      -0.244597

      5
      [15.05, 16.93)
      54
      0.094903
      44
      10
      3.562494
      -0.208520

      6
      [16.93, inf)
      118
      0.207381
      117
      1
      0.082142
      -0.002887

      7
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      8
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

Let's describe the columns of this binning table:

- Bin: the intervals delimited by the optimal split points.
- Count: the number of records for each bin.
- Count (%): the percentage of records for each bin.
- Non-event: the number of non-event records $(y = 0)$ for each bin.
- Event: the number of event records $(y = 1)$ for each bin.
- $c_0$: the first coefficient of the event rate polynomial.
- $c_1$: the second coefficient of the event rate polynomial.

The event rate for bin $i$ is defined as $ER_i = c_0 + c_1 x_i$, where $x_i \in \text{Bin}_{i}$. In general,
\begin{equation}
ER_i = \sum_{j=0}^d c_j x_i^j,
\end{equation}
where $d$ is the degree of the event rate polynomial.

The last row shows the total number of records, non-event records, event records.

You can use the method ``plot`` to visualize the number of non-event and event records by bin, and WoE or event rate curve.

```python
binning_table.plot(metric="woe")
```

```python
binning_table.plot(metric="event_rate")
```

##### Event rate / WoE transformation

Now that we have checked the binned data, we can transform our original data into WoE or event rate values.

```python
x_transform_woe = optb.transform(x, metric="woe")
```

```python
x_transform_event_rate = optb.transform(x, metric="event_rate")
```

## Advanced

#### Optimal binning Information

The ``OptimalPWBinning`` can print overview information about the options settings, problem statistics, and the solution of the computation. By default, ``print_level=1``.

```python
optb = OptimalPWBinning(name=variable)
optb.fit(x, y)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean radius')

If ``print_level=0``, a minimal output including the header, variable name, status, and total time are printed.

```python
optb.information(print_level=0)
```

    optbinning (Version 0.13.1)
    Copyright (c) 2019-2022 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean radius
      Status  : OPTIMAL

      Time    : 0.1114  sec

If ``print_level>=1``, statistics on the pre-binning phase and the solver are printed. More detailed timing statistics are also included.

```python
optb.information(print_level=1)
```

    optbinning (Version 0.13.1)
    Copyright (c) 2019-2022 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean radius
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         7

      Solver statistics
        Type                                auto
        Number of variables                   14
        Number of constraints                 20

      Timing
        Total time                          0.11 sec
        Pre-processing                      0.00 sec   (  0.73%)
        Estimator                           0.01 sec   ( 10.63%)
        Pre-binning                         0.05 sec   ( 40.58%)
        Solver                              0.04 sec   ( 35.02%)
        Post-processing                     0.01 sec   ( 12.37%)

If ``print_level=2``, the list of all options of the ``OptimalBinning`` are displayed.

```python
optb.information(print_level=2)
```

    optbinning (Version 0.13.1)
    Copyright (c) 2019-2022 Guillermo Navas-Palencia, Apache License 2.0

      Begin options
        name                         mean radius   * U
        estimator                            yes   * U
        objective                             l2   * d
        degree                                 1   * d
        continuous                          True   * d
        prebinning_method                   cart   * d
        max_n_prebins                         20   * d
        min_prebin_size                     0.05   * d
        min_n_bins                            no   * d
        max_n_bins                            no   * d
        min_bin_size                          no   * d
        max_bin_size                          no   * d
        monotonic_trend                     auto   * d
        n_subsamples                          no   * d
        max_pvalue                            no   * d
        max_pvalue_policy            consecutive   * d
        outlier_detector                      no   * d
        outlier_params                        no   * d
        user_splits                           no   * d
        special_codes                         no   * d
        split_digits                          no   * d
        solver                              auto   * d
        h_epsilon                           1.35   * d
        quantile                             0.5   * d
        regularization                        no   * d
        reg_l1                               1.0   * d
        reg_l2                               1.0   * d
        random_state                          no   * d
        verbose                            False   * d
      End options

      Name    : mean radius
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         7

      Solver statistics
        Type                                auto
        Number of variables                   14
        Number of constraints                 20

      Timing
        Total time                          0.11 sec
        Pre-processing                      0.00 sec   (  0.73%)
        Estimator                           0.01 sec   ( 10.63%)
        Pre-binning                         0.05 sec   ( 40.58%)
        Solver                              0.04 sec   ( 35.02%)
        Post-processing                     0.01 sec   ( 12.37%)

#### Binning table statistical analysis

The ``analysis`` method performs a statistical analysis of the binning table, computing the statistics Gini index, Information Value (IV), Jensen-Shannon divergence, and the quality score. Additionally, several statistical significance tests between consecutive bins of the contingency table are performed: a frequentist test using the Chi-square test or the Fisher's exact test, and a Bayesian A/B test using the beta distribution as a conjugate prior of the Bernoulli distribution. The piecewise binning also includes two performance metrics: the average precision and the Brier score.

```python
binning_table.analysis(pvalue_test="chi2")
```

    ---------------------------------------------
    OptimalBinning: Binary Binning Table Analysis
    ---------------------------------------------

      General metrics

        Gini index               0.87503303
        IV (Jeffrey)             4.47808263
        JS (Jensen-Shannon)      0.37566002
        Hellinger                0.44267970
        Triangular               1.20862637
        KS                       0.72390466
        Avg precision            0.95577179
        Brier score              0.08956351
        HHI                      0.15727342
        HHI (normalized)         0.05193260
        Cramer's V               0.80066760
        Quality score            0.00000000

      Significance tests

        Bin A  Bin B  t-statistic      p-value  P[A > B]     P[B > A]
            0      1     0.289789 5.903557e-01  0.697563 3.024368e-01
            1      2     2.285355 1.306002e-01  0.941469 5.853092e-02
            2      3     2.317546 1.279217e-01  0.936661 6.333919e-02
            3      4     2.539025 1.110634e-01  0.950869 4.913141e-02
            4      5    29.418844 5.830790e-08  1.000000 1.898159e-11
            5      6    18.624518 1.591604e-05  0.999997 3.106835e-06

```python
binning_table.analysis(pvalue_test="fisher")
```

    ---------------------------------------------
    OptimalBinning: Binary Binning Table Analysis
    ---------------------------------------------

      General metrics

        Gini index               0.87503303
        IV (Jeffrey)             4.47808263
        JS (Jensen-Shannon)      0.37566002
        Hellinger                0.44267970
        Triangular               1.20862637
        KS                       0.72390466
        Avg precision            0.95577179
        Brier score              0.08956351
        HHI                      0.15727342
        HHI (normalized)         0.05193260
        Cramer's V               0.80066760
        Quality score            0.00000000

      Significance tests

        Bin A  Bin B  odd ratio      p-value  P[A > B]     P[B > A]
            0      1   0.660870 6.855009e-01  0.697563 3.024368e-01
            1      2   0.343985 1.879769e-01  0.941469 5.853092e-02
            2      3   0.447541 1.829850e-01  0.936661 6.333919e-02
            3      4   0.503663 1.153088e-01  0.950869 4.913141e-02
            4      5   0.115702 3.464724e-08  1.000000 1.898148e-11
            5      6   0.037607 4.144258e-05  0.999997 3.106835e-06

#### Event rate / WoE monotonicity

The ``monotonic_trend`` option permits forcing a monotonic trend to the event rate curve. The default setting "auto" should be the preferred option, however, some business constraints might require to impose different trends. The default setting "auto" chooses the monotonic trend most likely to maximize the information value from the options "ascending", "descending", "peak" and "valley" using a machine-learning-based classifier. For this example, we choose Gradient Boosting as an estimator.

```python
variable = "mean texture"
x = df[variable].values
y = data.target
```

```python
optb = OptimalPWBinning(name=variable, estimator=GradientBoostingClassifier())
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=GradientBoostingClassifier(), name='mean texture')

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 15.05)
      92
      0.161687
      4
      88
      0.931559
      -0.000000

      1
      [15.05, 16.39)
      61
      0.107206
      8
      53
      2.075863
      -0.076059

      2
      [16.39, 17.03)
      29
      0.050967
      6
      23
      2.149342
      -0.080541

      3
      [17.03, 18.46)
      79
      0.138840
      17
      62
      1.113793
      -0.019751

      4
      [18.46, 19.47)
      55
      0.096661
      20
      35
      4.139964
      -0.183682

      5
      [19.47, 20.20)
      36
      0.063269
      18
      18
      4.417902
      -0.197957

      6
      [20.20, 21.71)
      72
      0.126538
      43
      29
      0.933414
      -0.025415

      7
      [21.71, 22.74)
      40
      0.070299
      27
      13
      2.611206
      -0.102697

      8
      [22.74, 24.00)
      29
      0.050967
      24
      5
      0.275868
      -0.000000

      9
      [24.00, 26.98)
      43
      0.075571
      30
      13
      -1.111300
      0.057799

      10
      [26.98, inf)
      33
      0.057996
      15
      18
      0.448108
      0.000000

      11
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      12
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

```python
binning_table.plot(metric="event_rate")
```

For example, we can force the variable mean texture to be convex with respect to the probability of having breast cancer.

```python
optb = OptimalPWBinning(name=variable, estimator=GradientBoostingClassifier(),
                        monotonic_trend="convex")
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=GradientBoostingClassifier(),
                     monotonic_trend='convex', name='mean texture')

```python
binning_table = optb.binning_table
binning_table.build()
binning_table.plot(metric="event_rate")
```

#### User-defined split points

In some situations, we have defined split points or bins required to satisfy a priori belief,  knowledge or business constraint. The ``OptimalPWBinning`` permits to pass user-defined split points. The supplied information is used as a pre-binning, disallowing any pre-binning method set by the user. Furthermore, version 0.5.0 introduces ``user_splits_fixed`` parameter, to allow the user to fix some user-defined splits, so these must appear in the solution.

Example numerical variable:

```python
user_splits =       [  14,    15,    16,    17,    20,    21,    22,    27]
user_splits_fixed = [False, True,  True, False, False, False, False, False]
```

```python
optb = OptimalPWBinning(name=variable, user_splits=user_splits,
                        user_splits_fixed=user_splits_fixed)

optb.fit(x, y)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean texture',
                     user_splits=[14, 15, 16, 17, 20, 21, 22, 27],
                     user_splits_fixed=[False, True, True, False, False, False,
                                        False, False])

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 14.00)
      54
      0.094903
      2
      52
      1.153974
      -0.020565

      1
      [14.00, 15.00)
      37
      0.065026
      2
      35
      1.354515
      -0.034890

      2
      [15.00, 16.00)
      43
      0.075571
      7
      36
      1.244455
      -0.027552

      3
      [16.00, 20.00)
      210
      0.369069
      59
      151
      1.570900
      -0.047955

      4
      [20.00, 21.00)
      45
      0.079086
      26
      19
      1.874636
      -0.063142

      5
      [21.00, 22.00)
      49
      0.086116
      30
      19
      1.788369
      -0.059034

      6
      [22.00, 27.00)
      99
      0.173989
      71
      28
      1.708444
      -0.055401

      7
      [27.00, inf)
      32
      0.056239
      15
      17
      0.789092
      -0.021351

      8
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      9
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

```python
optb.information()
```

    optbinning (Version 0.13.1)
    Copyright (c) 2019-2022 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean texture
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         8

      Solver statistics
        Type                                auto
        Number of variables                   16
        Number of constraints                 23

      Timing
        Total time                          0.15 sec
        Pre-processing                      0.00 sec   (  0.47%)
        Estimator                           0.02 sec   ( 11.16%)
        Pre-binning                         0.10 sec   ( 67.15%)
        Solver                              0.02 sec   ( 15.12%)
        Post-processing                     0.01 sec   (  5.40%)

#### Performance: choosing a solver

``OptimalPWBinning`` uses the RoPWR library to solving the piecewise regression problem. See https://github.com/guillermo-navas-palencia/ropwr for details.

#### Robustness: choosing an objective function and regularization

``OptimalPWBinning`` uses the RoPWR library to solving the piecewise regression problem. See https://github.com/guillermo-navas-palencia/ropwr for details.

#### Missing data and special codes

For this example, let's load data from the FICO Explainable Machine Learning Challenge: https://community.fico.com/s/explainable-machine-learning-challenge

```python
df = pd.read_csv("data/FICO_challenge/heloc_dataset_v1.csv", sep=",")
```

The data dictionary of this challenge includes three special values/codes:

* -9 No Bureau Record or No Investigation
* -8 No Usable/Valid Trades or Inquiries
* -7 Condition not Met (e.g. No Inquiries, No Delinquencies)

```python
special_codes = [-9, -8, -7]
```

```python
variable = "AverageMInFile"
x = df[variable].values
y = df.RiskPerformance.values
```

```python
df.RiskPerformance.unique()
```

    array(['Bad', 'Good'], dtype=object)

Target is a categorical dichotomic variable, which can be easily transform into numerical.

```python
mask = y == "Bad"
y[mask] = 1
y[~mask] = 0
y = y.astype(int)
```

For the sake of completeness, we include a few missing values

```python
idx = np.random.randint(0, len(x), 500)
x = x.astype(float)
x[idx] = np.nan
```

```python
optb = OptimalPWBinning(name=variable, special_codes=special_codes, solver="osqp")

optb.fit(x, y, lb=0, ub=1)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='AverageMInFile',
                     solver='osqp', special_codes=[-9, -8, -7])

```python
optb.information(print_level=1)
```

    optbinning (Version 0.13.1)
    Copyright (c) 2019-2022 Guillermo Navas-Palencia, Apache License 2.0

      Name    : AverageMInFile
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                        13

      Solver statistics
        Type                                osqp
        Number of variables                   26
        Number of constraints              18852

      Timing
        Total time                          0.46 sec
        Pre-processing                      0.00 sec   (  0.50%)
        Estimator                           0.04 sec   (  8.24%)
        Pre-binning                         0.13 sec   ( 28.37%)
        Solver                              0.27 sec   ( 58.64%)
        Post-processing                     0.02 sec   (  3.86%)

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 30.50)
      558
      0.053351
      98
      460
      0.791741
      -0.003083

      1
      [30.50, 41.50)
      551
      0.052682
      148
      403
      0.803858
      -0.003480

      2
      [41.50, 48.50)
      502
      0.047997
      141
      361
      0.807896
      -0.003577

      3
      [48.50, 54.50)
      526
      0.050292
      186
      340
      0.813452
      -0.003692

      4
      [54.50, 59.50)
      486
      0.046467
      197
      289
      0.817065
      -0.003758

      5
      [59.50, 64.50)
      595
      0.056889
      257
      338
      0.820498
      -0.003816

      6
      [64.50, 70.50)
      782
      0.074768
      359
      423
      0.822974
      -0.003854

      7
      [70.50, 75.50)
      676
      0.064633
      336
      340
      0.828829
      -0.003937

      8
      [75.50, 81.50)
      766
      0.073238
      405
      361
      0.818921
      -0.003806

      9
      [81.50, 95.50)
      1533
      0.146572
      872
      661
      0.840311
      -0.004068

      10
      [95.50, 101.50)
      471
      0.045033
      270
      201
      0.614140
      -0.001700

      11
      [101.50, 116.50)
      869
      0.083086
      539
      330
      1.157173
      -0.007050

      12
      [116.50, inf)
      1092
      0.104408
      709
      383
      0.482622
      -0.001260

      13
      Special
      562
      0.053734
      251
      311
      0.553381
      0.000000

      14
      Missing
      490
      0.046850
      232
      258
      0.526531
      0.000000

      Totals

      10459
      1.000000
      5000
      5459
      -
      -

Note the dashed bins 10 and 11, corresponding to the special codes bin and the missing bin, respectively.

```python
binning_table.plot(metric="event_rate")
```

#### Verbosity option

For debugging purposes, we can print information on each step of the computation by triggering the ``verbose`` option.

```python
optb = OptimalPWBinning(name=variable, solver="osqp", verbose=True)
optb.fit(x, y, lb=0, ub=1)
```

    2022-02-18 11:13:05,394 | INFO : Optimal piecewise binning started.
    2022-02-18 11:13:05,396 | INFO : Options: check parameters.
    2022-02-18 11:13:05,397 | INFO : Pre-processing started.
    2022-02-18 11:13:05,398 | INFO : Pre-processing: number of samples: 10459
    2022-02-18 11:13:05,402 | INFO : Pre-processing: number of clean samples: 9969
    2022-02-18 11:13:05,403 | INFO : Pre-processing: number of missing samples: 490
    2022-02-18 11:13:05,404 | INFO : Pre-processing: number of special samples: 0
    2022-02-18 11:13:05,405 | INFO : Pre-processing terminated. Time: 0.0009s
    2022-02-18 11:13:05,406 | INFO : Pre-binning: set logistic regression as an estimator.
    2022-02-18 11:13:05,408 | INFO : Pre-binning: estimator fitting started.
    2022-02-18 11:13:05,452 | INFO : Pre-binning: estimator terminated. Time 0.0466s.
    2022-02-18 11:13:05,454 | INFO : Pre-binning: optimal binning started.
    2022-02-18 11:13:05,641 | INFO : Pre-binning: number of splits: 9.
    2022-02-18 11:13:05,643 | INFO : Pre-binning: no need for subsamples.
    2022-02-18 11:13:05,644 | INFO : Pre-binning: optimal binning terminated. Time 0.1893s.
    2022-02-18 11:13:05,645 | INFO : Optimizer started.
    2022-02-18 11:13:05,649 | INFO : Optimizer: descending monotonic trend.
    ===============================================================================
                                         CVXPY
                                        v1.1.12
    ===============================================================================
    (CVXPY) Feb 18 11:13:05 AM: Your problem has 20 variables, 4 constraints, and 0 parameters.
    (CVXPY) Feb 18 11:13:05 AM: It is compliant with the following grammars: DCP, DQCP
    (CVXPY) Feb 18 11:13:05 AM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
    (CVXPY) Feb 18 11:13:05 AM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
    -------------------------------------------------------------------------------
                                      Compilation
    -------------------------------------------------------------------------------
    (CVXPY) Feb 18 11:13:05 AM: Compiling problem (target solver=OSQP).
    (CVXPY) Feb 18 11:13:05 AM: Reduction chain: CvxAttr2Constr -> Qp2SymbolicQp -> QpMatrixStuffing -> OSQP
    (CVXPY) Feb 18 11:13:05 AM: Applying reduction CvxAttr2Constr
    (CVXPY) Feb 18 11:13:05 AM: Applying reduction Qp2SymbolicQp
    (CVXPY) Feb 18 11:13:05 AM: Applying reduction QpMatrixStuffing
    (CVXPY) Feb 18 11:13:05 AM: Applying reduction OSQP
    (CVXPY) Feb 18 11:13:05 AM: Finished problem compilation (took 9.970e-02 seconds).
    -------------------------------------------------------------------------------
                                    Numerical solver
    -------------------------------------------------------------------------------
    (CVXPY) Feb 18 11:13:05 AM: Invoking solver OSQP  to obtain a solution.
    -----------------------------------------------------------------
               OSQP v0.6.0  -  Operator Splitting QP Solver
                  (c) Bartolomeo Stellato,  Goran Banjac
            University of Oxford  -  Stanford University 2019
    -----------------------------------------------------------------
    problem:  variables n = 20, constraints m = 19967
              nnz(P) + nnz(A) = 39952
    settings: linear system solver = qdldl,
              eps_abs = 1.0e-05, eps_rel = 1.0e-05,
              eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
              rho = 1.00e-01 (adaptive),
              sigma = 1.00e-06, alpha = 1.60, max_iter = 10000
              check_termination: on (interval 25),
              scaling: on, scaled_termination: off
              warm start: on, polish: on, time_limit: off

    iter   objective    pri res    dua res    rho        time
       1  -4.3045e+01   1.33e-03   1.77e+05   1.00e-01   2.15e-02s
     200  -2.4215e+03   7.80e-03   1.68e+04   1.22e-02   1.16e-01s
     400  -2.7207e+03   1.03e-02   6.09e+03   1.22e-02   2.04e-01s
     600  -2.8224e+03   7.81e-06   2.84e-03   4.69e-05   3.01e-01s
    plsh  -2.8224e+03   1.41e-16   7.67e-11   --------   3.02e-01s

    status:               solved
    solution polish:      successful
    number of iterations: 600
    optimal objective:    -2822.3989
    run time:             3.02e-01s
    optimal rho estimate: 2.81e-04

    -------------------------------------------------------------------------------
                                        Summary
    -------------------------------------------------------------------------------
    (CVXPY) Feb 18 11:13:06 AM: Problem status: optimal
    (CVXPY) Feb 18 11:13:06 AM: Optimal value: -2.822e+03
    (CVXPY) Feb 18 11:13:06 AM: Compilation took 9.970e-02 seconds
    (CVXPY) Feb 18 11:13:06 AM: Solver (including time spent in interface) took 3.100e-01 seconds
    2022-02-18 11:13:06,113 | INFO : Optimizer terminated. Time: 0.4628s
    2022-02-18 11:13:06,115 | INFO : Post-processing started.
    2022-02-18 11:13:06,116 | INFO : Post-processing: compute binning information.
    2022-02-18 11:13:06,123 | INFO : Post-processing: compute performance metrics.
    2022-02-18 11:13:06,153 | INFO : Post-processing terminated. Time: 0.0362s
    2022-02-18 11:13:06,155 | INFO : Optimal piecewise binning terminated. Status: OPTIMAL. Time: 0.7606s

    OptimalPWBinning(estimator=LogisticRegression(), name='AverageMInFile',
                     solver='osqp', verbose=True)


```python
%%html

```

# The shearing sheet

## Equations of motion to first order

We have is a patch of the disk centered on some $(R_0,\phi_0)$ that is shearing at a rate $\Omega(R)$. We derive the dynamics of stars in this sheet in coordinates $x = R-R_0$ and $y = R_0(\phi-\Omega_0\,t+\phi_0)$. We can always easily define the coordinate system such that $\phi_0 = 0$, so we will assume that from now on.
To derive the equations of motion, we write down the Lagrangian (see [here](http://astro.utoronto.ca/~bovy/AST1420/notes-2019/chapters/I-02.-Elements-of-Classical-Mechanics.html#Lagrangian-formulation-of-classical-mechanics)). In polar coordinates, the Lagrangian is

\begin{equation}\label{eq-lagrangian-polar}
    \mathcal{L} = \frac{1}{2}\,\left(\dot{R}^2 + R^2\,\dot{\phi}^2\right) - \Phi(R,\phi)\,.
\end{equation}

Writing this in terms of $(x,y)$, we first of all need $(\dot{x},\dot{y})$:

\begin{align}
    \dot{x} & = {\mathrm{d}\over \mathrm{d} t}\,\left(R-R_0\right) = \dot{R}\,,\\
    \dot{y} & = {\mathrm{d}\over \mathrm{d} t}\,\left(R_0\,[\phi-\phi_0]\right) = R_0\,\dot{\phi}- R_0\,\Omega_0\,,
\end{align}

Thus we have that

\begin{align}
    R & = R_0 + x\,,\\
    \phi & = {y + R_0\,\Omega_0 t \over R_0}\,,\\
    \dot{R} & = \dot{x}\,,\\
    \dot{\phi} & = {1 \over R_0}\,\left(\dot{y} +R_0\,\Omega_0\right)
\end{align}

Plugging these into Equation \eqref{eq-lagrangian-polar}, we get the Lagrangian as a function of $(x,y,\dot{x},\dot{y})$

\begin{equation}\label{eq-lagrangian-shearing}
    \mathcal{L} = \frac{1}{2}\,\left(\dot{x}^2 + {(R_0+x)^2 \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]^2\right) - \Phi\left(R_0+x,{y + R_0\,\Omega_0 t \over R_0}\right)\,.
\end{equation}

The generalized momenta are then

\begin{align}
    p_x & = {\partial \mathcal{L}  \over \partial \dot{x}} = \dot{x}\,,\\
    p_y & = {\partial \mathcal{L}  \over \partial \dot{y}} = {(R_0+x)^2 \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]\,.
\end{align}

The main purpose of using the shearing sheet for our purposes is that we will assume that both $|x|$ and $|y| \ll R$, such that we can keep terms only to first order in $x$, $y$, $\dot{x}$, and $\dot{y}$. The generalized momenta in this approximation are

\begin{align}
    p_x & = {\partial \mathcal{L}  \over \partial \dot{x}} = \dot{x}\,,\\
    p_y & = {\partial \mathcal{L}  \over \partial \dot{y}} = R_0\,\Omega_0 + \dot{y} + 2\,x\,\Omega_0\,.
\end{align}

Using Lagrange's equations, we can then derive the equations of motion, again only keeping terms up to first order in the small quantities:

\begin{align}\label{eq-shearing-eom-1}
    \ddot{x} & =  R_0\,\Omega_0^2 +2\,\dot{y}\,\Omega_0 + x\,\Omega_0^2 - {\partial \Phi \over \partial x}\,,\\
    \ddot{y} + 2\,\dot{x}\,\Omega_0 & = - {\partial \Phi \over \partial y}\,.
\end{align}

We will write the gravitational potential as the sum of an axisymmetric contribution $\Phi_\mathrm{axi} \equiv \Phi_\mathrm{axi}(R) = \Phi_\mathrm{axi}(x)$ and a perturbation $\tilde{\Phi}$ to this: $\Phi = \Phi_\mathrm{axi}+ \tilde{\Phi}$. For the axisymmetric part, we have that

\begin{align}
    {\partial \Phi_\mathrm{axi} \over \partial x} & =     {\partial \Phi_\mathrm{axi} \over \partial R} \\
    & = R\,\Omega^2\\
    & = (R_0+x)\,(\Omega_0 + \Omega_0' x)^2\\
    & \approx R_0\,\Omega_0^2 + 2\,R_0\,\Omega_0\,\Omega_0'\,x + x\,\Omega_0^2\,,
\end{align}

where $\Omega \approx \Omega_0 + \Omega_0'\,x$ for small $|x|$ and $\Omega_0' = \mathrm{d} \Omega / \mathrm{d} R$ evaluated at $R_0$. Plugging this into Equation \eqref{eq-shearing-eom-1}, we find that

\begin{align}
    \ddot{x} & =  2\,\dot{y}\,\Omega_0 - 2\,R_0\,\Omega_0\,\Omega_0'\,x -{\partial \tilde{\Phi} \over \partial x}\,.
\end{align}

Using the Oort constant $A = -R\,\mathrm{d} \Omega /\mathrm{d} \mathrm{R} / 2$ evaluated at $R_0$, we can write the full set of equations of motion as

\begin{align}
    \ddot{x} -2\,\Omega_0\,\dot{y} - 4\,A\,\Omega_0\,x & = -{\partial \tilde{\Phi} \over \partial x}\,,\\
    \ddot{y} + 2\,\Omega_0\,\dot{x} & = - {\partial \tilde{\Phi} \over \partial y}\,,
\end{align}

which are the basic equations of motion in the shearing sheet.

## Equations of motion to second order

To get a more precise solution, we can use the equations of motion that keep terms up to second order in the small quantities. The generalized momenta are then

\begin{align}
    p_x & = \dot{x}\,,\\
    p_y & = R_0\,\Omega_0 + \dot{y} + 2\,x\,\Omega_0 + x^2\,{\Omega_0 \over R_0} + 2\,{x\,\dot{y} \over R_0}\,
\end{align}

Using Lagrange's equations, we again derive the equations of motion, but now keeping terms to second order

\begin{align}\label{eq-shearing-eom-2nd-1}
    \ddot{x} = R_0\,\Omega_0^2 + 2\,\dot{y}\,\Omega_0 + x\,\Omega_0^2 + 2\,{x\over R_0}\,\dot{y}\,\Omega_0+{\dot{y}^2\over R_0} & - {\partial \Phi \over \partial x}\,,\\
    \left[1+2\,{x \over R_0}\right]\,\ddot{y} + 2\,\left[1+{x\over R_0}\right]\,\dot{x}\,\Omega_0  + 2\,{\dot{x}\,\dot{y} \over R_0} & = - {\partial \Phi \over \partial y}\,.\label{eq-shearing-eom-2nd-2}
\end{align}

We can again split the potential into an axisymmetric and non-axisymmetric part and write the axisymmetric derivative as

\begin{align}
    {\partial \Phi_\mathrm{axi} \over \partial x} & =     {\partial \Phi_\mathrm{axi} \over \partial R} \\
    & = R\,\Omega^2\\
    & = (R_0+x)\,(\Omega_0 + \Omega_0' x + \Omega_0''\,x^2/2)^2\\
    & \approx R_0\,\Omega_0^2 + 2\,R_0\,\Omega_0\,\Omega_0'\,x + x\,\Omega_0^2 + 2\,x^2\,\Omega_0\,\Omega_0' + R_0\,\Omega_0\,\Omega_0''\,x^2+R_0\,\Omega_0'^2 x^2\,,\label{eq-axipot-2nd}
\end{align}

where $\Omega \approx \Omega_0 + \Omega_0'\,x + \Omega_0''\,x^2/2$ now for small $|x|$ and $\Omega_0'' = \mathrm{d}^2 \Omega / \mathrm{d}^2 R$ evaluated at $R_0$. Plugging this into Equation \eqref{eq-shearing-eom-2nd-1} and \eqref{eq-shearing-eom-2nd-2}, we find that

\begin{align}
    \ddot{x} -2\,\dot{y}\,\Omega_0 -4\,\left[1+{x\over R_0}\right]\,\Omega_0\,A\,x- 2\,{x\over R_0}\,\dot{y}\,\Omega_0-{\dot{y}^2\over R_0}
       + R_0\,\Omega_0\,\Omega_0''\,x^2 +4\,A^2\,{x^2\over R_0} & =
    - {\partial \Phi \over \partial x}\,,\\
    \left[1+2\,{x \over R_0}\right]\,\ddot{y} + 2\,\left[1+{x\over R_0}\right]\,\dot{x}\,\Omega_0  + 2\,{\dot{x}\,\dot{y} \over R_0} & = - {\partial \Phi \over \partial y}\,.
\end{align}

## Full equations of motion

We can also work out the full, non-linear equations of motion. For this we use the full momenta

\begin{align}
    p_x & = {\partial \mathcal{L}  \over \partial \dot{x}} = \dot{x}\,,\\
    p_y & = {\partial \mathcal{L}  \over \partial \dot{y}} = {(R_0+x)^2 \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]\,.
\end{align}

and their derivatives

\begin{align}
    \dot{p}_x & = \ddot{x}\,,\\
    \dot{p}_y & = 2\,{(R_0+x) \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]\,\dot{x} + {(R_0+x)^2 \over R_0^2}\,\ddot{y}\,.
\end{align}

Lagrange's equations then give

\begin{align}
    \ddot{x} & = {(R_0+x) \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]^2 -{\partial  \Phi \over \partial x}\,,\\
    {(R_0+x)^2 \over R_0^2}\,\ddot{y} & =
    -2\,{(R_0+x) \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]\,\dot{x}-{\partial  \Phi \over \partial y}\,.
\end{align}

If we now work out the axisymmetric force to second order as in Equation \eqref{eq-axipot-2nd}, we get

\begin{align}
    \ddot{x} & = {(R_0+x) \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]^2
    - R_0\,\Omega_0^2 - (\Omega_0^2-4\,\Omega_0\,A)\,x - (R_0\,\Omega_0\,\Omega_0''+4\,{A^2 \over R_0}-{4\,\Omega_0\,A\over R_0} )\, x^2 -{\partial  \tilde{\Phi} \over \partial x}\,,\\
    {(R_0+x)^2 \over R_0^2}\,\ddot{y} & =
    -2\,{(R_0+x) \over R_0^2}\,\left[\dot{y} +R_0\,\Omega_0\right]\,\dot{x}-{\partial  \tilde{\Phi} \over \partial y}\,.
\end{align}


# Método de Welch

Neste notebook avaliamos o método de Welch, uma ferramenta para análise estatística do espectro. Basicamente tomamos transformadas de Fourier de pequenas janelas temporais que vão se deslocando com o tempo e, então, o espectro médio.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
from scipy import signal
import IPython.display as ipd # to play signalsa
import sounddevice as sd
import soundfile as sf
```

# Intuição

Vejamos uma intuição sobre o método de Welch.

# Os parâmetros do  auto-espectro e espectro cruzado

Dê uma olhada na documentação do scipy:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.csd.html

- Os sinais, $x(t)$ e $y(t)$ no tempo
- A frequência de amostragem
- window: A janela temporal desejada
- Overlap: Número de pontos em que teremos uma sobreposição das janelas.
- nfft: Número de pontos da fft (zero padding). Se não for fornecido, é o número de pontos na janela temporal.

# Exemplo 1: O ruído de uma bomba d'agua

```python
##### x(t) water pump #################
xg, Fs = sf.read('waterpump.wav')
print("O sinal tem {:.2f} [s]".format((len(xg)-1)/Fs))

ti = 0
tp = 5
xt = xg[int(ti*Fs):int(tp*Fs),0]
time = np.linspace(0, (len(xt)-1)/Fs, len(xt))

#plot
plt.figure(figsize=(12,3))
plt.plot(time, xt, '-b',linewidth = 1, alpha = 0.7)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, time[-1]))

ipd.Audio(xt, rate=Fs) # load a NumPy array
```

# Cálculos dos espectros

Vamos calcular:

- Via uma FFT direta no sinal
- Via método de Welch

```python
# Método direto
Xw = np.fft.fft(xt)
Xw2 = (np.abs(Xw)/len(Xw))**2
freq = np.linspace(0, (len(Xw)-1)*Fs/len(Xw), len(Xw))

# Método de Welch
win_size = 2*8192
win = signal.windows.hann(win_size)
noverlap = win_size/2
nfft = 8*win_size
f, Sxx = signal.csd(xt, xt, Fs, window = win, noverlap = noverlap, nfft = nfft, scaling = 'density')

# plots
plt.figure(figsize=(12,3))
plt.semilogx(freq, 10*np.log10(Xw2), '-b',linewidth = 1, alpha = 0.7, label = 'direto')
plt.semilogx(f, 10*np.log10(Sxx), '-r',linewidth = 2,  label = 'Welch')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((10, Fs/2))
plt.ylim((-100, -20));
```

# Exemplo 2 - O ruído branco

Vimos nas aulas passadas que o ruído branco tem uma auto-correlação do tipo impulsiva.

```python
fs = 44100
time = np.arange(0, 2, 1/fs)

# sinal aleatório completo
xt = np.random.normal(loc = 0, scale = 1, size = len(time))

# plot signal
plt.figure(figsize = (10, 3))
plt.plot(time, xt, linewidth = 1, color = 'b', alpha = 0.7)
plt.grid(linestyle = '--', which='both')
plt.ylabel(r'$x(t)$ [Pa]')
plt.xlim((0, time[-1]))
plt.ylim((-4, 4))
plt.xlabel('Tempo [s]')
plt.tight_layout()

# Calculemos a auto-correlação
Rxx = np.correlate(xt, xt, mode = 'same')
tau = np.linspace(-len(Rxx)/fs, len(Rxx)/fs, len(Rxx))

# plot autocorrelação
plt.figure(figsize = (10, 3))
plt.plot(tau, Rxx/fs, linewidth = 1, color = 'b', alpha = 0.7)
plt.grid(linestyle = '--', which='both')
plt.ylabel(r'$R_{xx}(\tau)$ [Pa$^2$]')
plt.xlim((tau[0], tau[-1]))
plt.ylim((-0.5, 1.2*max(np.abs(Rxx/fs))))
plt.xlabel(r'$\tau$ [s]')
plt.tight_layout()
```

```python
# Método direto
Xw = np.fft.fft(xt)
Xw2 = (np.abs(Xw)/len(Xw))**2
freq = np.linspace(0, (len(Xw)-1)*Fs/len(Xw), len(Xw))

# Método de Welch
win_size = 4096
win = signal.windows.hann(win_size)
noverlap = win_size/2
nfft = win_size
f, Sxx = signal.csd(xt, xt, Fs, window = win, noverlap = noverlap, nfft = nfft, scaling = 'density')

# plots
plt.figure(figsize=(12,3))
plt.semilogx(freq, 10*np.log10(Xw2), '-b',linewidth = 1, alpha = 0.7, label = 'direto')
plt.semilogx(f, 10*np.log10(Sxx), '-r',linewidth = 2,  label = 'Welch')
plt.legend(loc = 'lower left')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((10, Fs/2))
plt.ylim((-100, -20));
```

# Exemplo 3 - Seno contaminado por ruído

Consideremos um sinal senoidal contaminado por ruído. Temos que

\begin{equation}
x(t) = s(t) + n(t),
\end{equation}
com

\begin{equation}
s(t) = A \mathrm{sin}(2\pi f t + \phi)
\end{equation}

e $n(t)$ um sinal aleatório com distribuição normal. A autocorrelação de $x(t)$ é dada por
\begin{equation}
R_{xx}(\tau) = R_{ss}(\tau) + R_{nn}(\tau)
\end{equation}

Note que $R_{nn}(\tau)$ é um sinal impulsivo como do exemplo anterior e que

\begin{equation}
R_{ss}(\tau) = \frac{A^2}{2}\mathrm{cos}(2\pi f \tau),
\end{equation}
uma função cossenoidal desprovida da informação de fase de $s(t)$. Vamos tomar o auto-espectro do sinal contaminado e comparar com o espectro calculado de maneira direta. Note que a ***TF*** da auto-correlação vai conter informações sobre o cosseno e sobre o ruído.

\begin{equation}
\mathcal{F}[R_{xx}(\tau)] = \mathcal{F}[R_{ss}(\tau)] + \mathcal{F}[R_{nn}(\tau)]
\end{equation}

```python
fs = 2000
time = np.arange(0, 20, 1/fs)

# sinal aleatório completo
st = np.sin(2*np.pi*100*time + np.pi/3)
nt = np.random.normal(loc = 0, scale = 4, size = len(time))
xt = st + nt

# Vamos tomar a fft de maneira direta
Xw = np.fft.fft(xt)
Xw2 = (np.abs(Xw)/len(Xw))**2
freq = np.linspace(0, (len(Xw)-1)*fs/len(Xw), len(Xw))

# Vamos calcular o auto-espectro usando o Método de Welch
win_size = 4096
win = signal.windows.hann(win_size)
noverlap = win_size/2
nfft = win_size
f, Sxx = signal.csd(xt, xt, fs, window = win, noverlap = noverlap, nfft = nfft, scaling = 'density')

# plots
plt.figure(figsize=(12,3))
plt.semilogx(f, 10*np.log10(Sxx), '-r',linewidth = 2,  label = 'Welch')
plt.semilogx(freq, 10*np.log10(Xw2), '-b',linewidth = 2, alpha = 0.7, label = 'direto')
plt.legend(loc = 'lower left')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((10, fs/2))
plt.ylim((-100, 0));
```


```python
%matplotlib inline
```

A Complete Guide to Matrix Notation and Linear Regression
=========================================================

Let’s really understand matrix notation in context of linear regression,
from the ground up.

Linear Regression finds the best line, or *hyperplane* $\hat{y}$
in higher dimension, or generally a function $f$:

\begin{align}\hat{y} = f(x) = wx\end{align}

that fits the whole data. This is just a dot product between vector
$w$ and a data point $x$ in $d$ dimension:

\begin{align}\hat{y} = w_0 + w_1x_1 + w_2x_2 + ... + w_dx_d\end{align}

Notice that we use $w_0$ as an intercept term, and thus we need to
add a dummy dimension with value of “1” ($x_0$) for all data
points $x$. Thus, $x$ here is on $d+1$ dimension.
Think of it as the y-intercept term $c$ in 2-dimension
($y = mx + c$).

Another way to look at this is that $f(x)$ transforms a data point
$x$ on $d+1$ dimension into a predicted scalar value
$\hat{y}$ that is close to target $y$:

\begin{align}\begin{bmatrix}
   x_0 \\
   x_1 \\
   \vdots \\
   x_d
   \end{bmatrix}
   \xrightarrow{f}
   \hat{y}
   \approx y\end{align}

The Sum of Squared Error Loss
-----------------------------

The best way to solve this is to find $w$ that minimizes the **sum
of squared errors (SSE)**, or the “error” between all of predicted value
$\hat{y}^i$ and the target $y^i$ of $i^{th}$ data
point for $i = 1$ to $n$, writing this as a loss function
$L(w)$:

\begin{align}L(w) = \sum_{i=1}^{n} \left( y^i - \hat{y}^i \right)^2 = \sum_{i=1}^{n} \left( y^i - wx^i \right)^2\end{align}

From now on we refer to a data point (d+1 vector) as $x^i$ and its
corresponding target (scalar) as $y^i$. I know it’s confusing for
the first time, but you’ll get used to using superscript for indexing
data points.

Surprisingly, the SSE loss is not from someone’s intuition, but it’s
from the assumption that there is **Gaussian noise in our observation**
of the underlying linear relationship. We will show how this leads to
SSE loss later, but first let’s visualize what we’re trying to do.

.. raw:: html

   <center>

| |linear regression|

.. raw:: html

   </center>

1. There is a true line, the true linear relationship that we want to
   discover (blue line).
2. The data points are then observed through noise **deviating from that
   line, with Gaussian distribution**.
3. Suppose we predict a random line (red), not the best one yet.
4. We calculate the distance or difference between the predicted points
   (along the line) and the actual data points. This is then **squared
   and sum up to get sum of squared error**.

..

   Linear regression is the method to get the line that fits the given
   data with the minimum sum of squared error.

.. |linear regression| image:: imgs/img_lr_objective.png

How to Find the Optimal Solution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An optimal solution ($w$) for this equation can be found either
using *closed-form solution* or via iterative methods like gradient
descent.

**A closed-form solution** means we figure out the formula for
$w = ...$. Implementing that formula in a program directly solves
the problem. The thing is you have to come up with the correct one
yourself, by hand.

Do you remember how to find a minimum (or maximum) value for a function?
We take the derivative of the function above with respect to $w$,
set it to zero, and solve for the $w$ in terms of other
parameters. This is like taking a single jump to the optimal value. We
do all the hard work for computers.

Luckily we can do this for linear regression, but not all loss functions
be solved this way, actually, only a few. In those cases, we use
**iterative methods like gradient descent** to search for the solution.
In contrast to closed-form solution, we do not jump directly to the
optimal answer, instead, we take many steps that lead us near to where
the optimal answer lives.

Next let’s derive the closed-form solution for linear regression. In
order to do that efficiently, we need some matrix notations.

Going into Matrix Notation
--------------------------

Writing things down in matrix notation makes things much faster in
NumPy. **But it’s not easy to read matrix notation, especially if you
study machine learning on your own.** There’re things like dot product,
matrix multiplication, transpose and stuff that you need to keep track
of in your head. If you’re starting out, then please write them on
papers, drawing figures as needed to make you understand. It really pays
off.

On top of that, these few **key standards** will make our lives with
linear algebra easier:

1. Always a *column* vector
~~~~~~~~~~~~~~~~~~~~~~~~~~~

When you see standalone vectors in a matrix notation formula, assumes
it’s a column vector. E.g.,

\begin{align}x =
   \begin{bmatrix}
   1 \\
   2 \\
   3 \\
   \end{bmatrix}\end{align}

and so its transpose is a row vector,

\begin{align}x^T =
   \begin{bmatrix}
   1 & 2 & 3
   \end{bmatrix}\end{align}

Likewise, you should try to make the final result of matrix operation to
be a column vector.

Note that the NumPy vector created by ``np.zeros``, ``np.arange``, etc.,
is not really a column vector. It has only one dimension ``(N,)``. So,
you cannot transpose it directly (``x.T`` still gives you ``x``.) To
convert it to a column vector, we use ``x.reshape(N,1)`` or
``x[:, None]``.

2. Feature matrix $X$ is rows of data points
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Our data points $x^i$ are on $d+1$ dimension, and there is
$n$ of them, we store them all in a 2-d matrix $X$:

\begin{align}X = \begin{align}
   \underset{n\times d}
   {\begin{bmatrix}
   \longleftarrow & x^1 & \longrightarrow \\
   \longleftarrow & x^2 & \longrightarrow \\
   & \vdots & \\
   \longleftarrow & x^n & \longrightarrow \\
   \end{bmatrix}}
   \end{align}\end{align}

Each row in $X$ is a row vector for each data point. Also note
that we use uppercase letter for matrix.

3. Again, $w$ is a column vector
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Like the first point, our $w$ will be $d+1$ dimension column
vector with w_0 as an intercept term:

\begin{align}w =
   \begin{bmatrix}
   w_0 \\
   w_1 \\
   \vdots \\
   w_d
   \end{bmatrix}\end{align}

4. Dot products of rows in matrix $X$ with vector $w$ is $Xw$
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes we want the dot product of each row in matrix with a vector:

\begin{align}Xw = \underset{n\times (d+1)}
   {
       \begin{bmatrix}
       \longleftarrow & x^1 & \longrightarrow \\
       \longleftarrow & x^2 & \longrightarrow \\
       & \vdots & \\
       \longleftarrow & x^n & \longrightarrow \\
       \end{bmatrix}
   }
   \underset{(d+1) \times 1}
   {
       \begin{bmatrix}
       \uparrow \\
       w \\
       \downarrow
       \end{bmatrix}
   }
   =
   \begin{bmatrix}
   x^1w \\
   x^2w \\
   \vdots \\
   x^nw
   \end{bmatrix}\end{align}

given that $X$ contains rows of vectors we want to dot product
with.

Interestingly, this gives us a column vector of our predictions
$\hat{y}$:

\begin{align}\begin{bmatrix}
   x^1w \\
   x^2w \\
   \vdots \\
   x^nw
   \end{bmatrix} =
   \begin{bmatrix}
   \hat{y}^1 \\
   \hat{y}^2 \\
   \vdots \\
   \hat{y}^n \\
   \end{bmatrix} = \hat{y}\end{align}

It’s also good to remind yourself that it sums along dimension of
$x^i$ and $w$:

\begin{align}Xw =
   \begin{bmatrix}
   \sum_{j=0}^{d} x_j^1w_j \\
   \sum_{j=0}^{d} x_j^2w_j \\
   \vdots \\
   \sum_{j=0}^{d} x_j^nw_j \\
   \end{bmatrix}\end{align}

5. Sum of Squared is $x^Tx$
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is a useful pattern to memorize. Sometimes we want the sum of
squared of each element in arbitrary d-dimension vector $x$:

\begin{align}\sum_{j=1}^{d} x_i^2\end{align}

which is simply $x^Tx$:

\begin{align}x^Tx =
   \begin{bmatrix}
   x_1 & ... & x_d
   \end{bmatrix}
   \begin{bmatrix}
   x_1 \\
   \vdots \\
   x_d
   \end{bmatrix}
   = \sum_{j=1}^{d} x_i^2\end{align}

Notice that the result of $x^Tx$ is scalar, e.g., a number.

In fancy term,
${\left\lVert x \right\rVert} = \sqrt{\sum_{j=1}^{d} x_i^2}$ is
L2-norm (or Euclidean norm) of $x$. So we can write sum of squared
as ${\left\lVert x \right\rVert}^2 = \sum_{j=1}^{d} x_i^2$. For
now, let’s not care what norm actually means.

Writing SSE Loss in Matrix Notation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now you’re ready, let’s write the above SSE loss function in matrix
notation. If you look at $L(w)$ closely, it’s a sum of squared of
vector $y - \hat{y}$. This means we can kick-off by applying our
fifth trick:

\begin{align}L(w) = {\left\lVert y - \hat{y} \right\rVert}^2\end{align}

Next we just have to find $y - \hat{y}$. First we encode target
$y$ in a long column vector of shape ``[n, 1]``:

\begin{align}y =
   \begin{bmatrix}
   y^1 \\
   y^2 \\
   \vdots \\
   y^n
   \end{bmatrix}\end{align}

(Remember that we use superscript for indexing the $i^{th}$ data
point.)

Next we encode each of our predicted values ($\hat{y}^i$) in a
column vector $\hat{y}$. Since $\hat{y}^i$ is a dot product
between $w$ and each of $x^i$, we can apply
`4 <#4.-Dot-products-of-rows-in-matrix-$X$-with-vector-$w$-is-$Xw$>`__:

\begin{align}\begin{align}
   y - \hat{y} &=
   \begin{bmatrix}
   y^1 \\
   y^2 \\
   \vdots \\
   y^n
   \end{bmatrix}
   - \begin{bmatrix}
   \hat{y}^1 \\
   \hat{y}^2 \\
   \vdots \\
   \hat{y}^n
   \end{bmatrix} && \text{(Error between target and predicted)} \\ &=
   \begin{bmatrix}
   y^1 \\
   y^2 \\
   \vdots \\
   y^n
   \end{bmatrix}
   - \begin{bmatrix}
   x^1w \\
   x^2w \\
   \vdots \\
   x^nw
   \end{bmatrix}
   && \text{(Predicted is a dot product of $w$ and each of data point $x^i$)} \\ &=
   \underset{n\times 1}
   {
       \begin{bmatrix}
       y^1 \\
       y^2 \\
       \vdots \\
       y^n
       \end{bmatrix}
   }
   - \underset{n\times (d+1)}
   {
       \begin{bmatrix}
       \longleftarrow & x^1 & \longrightarrow \\
       \longleftarrow & x^2 & \longrightarrow \\
       & \vdots & \\
       \longleftarrow & x^n & \longrightarrow \\
       \end{bmatrix}
   }
   \underset{(d+1)\times 1}
   {
       \begin{bmatrix}
       \uparrow \\
       w \\
       \downarrow
       \end{bmatrix}
   } && \text{(Separate them out)} \\ &=
   y - Xw && \text{(Encode in matrix/vector form)}
   \end{align}\end{align}

Putting all together we get our loss function for linear regression:

\begin{align}L(w) = {\left\lVert y - Xw \right\rVert}^2\end{align}

In NumPy code, we can compute $L(w) = (y - Xw)^T(y - Xw)$.

There’s no intuitive way to come up with this nice formula the first
time you saw it. You have to work it out and put things together
yourself. Then you’ll start to memorize the pattern and it’ll become
easier.

Deriving a Closed-form Solution
-------------------------------

To do that, we’ll take derivative of $L(w)$ with respect to
$w$, set to zero and solve for $w$.

Writing matrix notation is already hard, taking derivative of it is even
harder. I recommend writing out partial derivatives to see what happens.
For $L(w) = L_w$, we have to take derivative with respect to each
dimension of $w$:

\begin{align}\nabla L_w =
   \begin{bmatrix}
   \frac{\partial L}{\partial w_0} \\
   \frac{\partial L}{\partial w_1} \\
   \vdots \\
   \frac{\partial L}{\partial w_d} \\
   \end{bmatrix}
   =
   \begin{bmatrix}
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_0} \\
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_1} \\
   \vdots \\
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_d}
   \end{bmatrix}
   =
   \begin{bmatrix}
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_0} \\
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_1} \\
   \vdots \\
   \frac{\partial \sum_{i=1}^{n} \left( y^i - wx^i \right)^2}{\partial w_d}
   \end{bmatrix}
   =
   \underset{(d+1) \times 1}
   {
   \begin{bmatrix}
   -2\sum_{i=1}^{n} x^i_0 \left( y^i - wx^i \right) \\
   -2\sum_{i=1}^{n} x^i_1 \left( y^i - wx^i \right) \\
   \vdots \\
   -2\sum_{i=1}^{n} x^i_d \left( y^i - wx^i \right) \\
   \end{bmatrix}
   }\end{align}

Looks like we might be able to apply our fourth point ($Xw$, but
in this case $w$ is $(y - Xw)$. But unlike our fourth point,
we now sum along data points ($n$) instead of dimensions
($d$). For this, we want each row of $X$ to be one given
dimension along all data points instead of one data point with all
dimensions, and thus we use $X^T$ instead of $X$. Finally,
here’s the full derivative in matrix notation:

\begin{align}\nabla L_w = -2X^T(y-Xw)\end{align}

Setting to zero and solve:

\begin{align}\begin{align}
   0 &= -2X^T(y-Xw) \\
   &= X^T(y-Xw)     \\
   &= X^Ty - X^TXw
   \end{align}\end{align}

Move $X^TX$ to other side and we get a closed-form solution:

\begin{align}\begin{align}
   X^TXw &= X^Ty    \\
   w &= (X^TX)^{-1}X^Ty
   \end{align}\end{align}

In NumPy, this is:

.. code:: python

   w = np.linalg.inv(X.T @ X) @ X @ y

A NumPy Example
---------------

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)
```

We will create a fake dataset from the underlying equation
$y = 2x + 7$:

```python
def true_target(x):
  return 2*x + 7
```

In practical settings, there is no way we know this exact equation. We
only get **observed** targets, and there’s some **noise** on it. The
reason is that it’s impossible to measure any data out there in the
world perfectly:

```python
def observed_target(x):
  """Underlying data with Gaussian noise added"""
  normal_noise = np.random.normal() * 3
  return true_target(x) + normal_noise
```

Creating data points
~~~~~~~~~~~~~~~~~~~~

Next, make 50 data points, observations and targets:

```python
N = 50

# Features, X is [1,50]
X = np.random.rand(N).reshape(N, 1) * 10

# Observed targets
y = np.array([observed_target(x) for x in X]).reshape(N, 1)
```

Adding dummy dimension term to each $x^i$:

```python
# Append 1 for intercept term later
X = np.hstack([np.ones((N, 1)), X])
```

Note that it **doesn’t matter** here whether we add it to the front or
back, it will simply reflect correspondingly in our solution $w$.

Visualize our data points with respect to the true line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
# For plotting
features = X[:,1:] # exclude the intercept for plotting
target = y
true_targets = true_target(X[:,1:])

plt.scatter(features, target, s=10, label='Observed data points')
plt.plot(features, true_targets, c='blue', label='True target line y = 2x + 7', alpha=0.3)

plt.xlabel('Feature')
plt.ylabel('Target')
plt.legend(loc='best')
plt.title('True and observed data points')
plt.show()
```

Compute a closed-form solution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Our goal is to get the line that is closest to that true target (blue)
line as possible, without the knowledge of its existence. For this we
use linear regression to fit observed data points by following the
formula from the previous section:

```python
w = np.linalg.inv(X.T @ X) @ X.T @ y
```

To predict, we compute $\hat{y} = xw$ for each data point
$x^i$. Here we predict the training set (``X``) itself:

```python
predicted = X @ w # y_hat
```

To predict a set of new points, you just make it the same format as
``X``, e.g., rows of data points.

Visualize best fit line vs. true target line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
plt.scatter(features, target, s=10, label='Data points')
plt.plot(features, true_targets, c='blue', label='True target line', alpha=0.3)
plt.plot(features, predicted, c='red', label='Best fit line')

plt.xlabel('Feature')
plt.ylabel('Target')
plt.legend(loc='best')
plt.show()
```

That’s pretty close.

Understanding the result
~~~~~~~~~~~~~~~~~~~~~~~~

And our $w$ is:

```python
print(w)
```

Since we append ones in front of each data point $x$, ``w[0]``
will be the intercept term and ``w[1]`` will be the slope. So our
predicted line will be in the format of ``y = w[1]*x + w[0]``. Recall
the *true* equation $y = 2x + 7$, you can see that we almost got
the true slope (2):

```python
print("Our slope is", w[1][0])
```

The intercept seems a little off, but that’s okay because our data is in
a big range ($x \in [0, 50], y \in [7, 107]$). If we normalize the
data into $[0, 1]$ range, expect it to be much closer.

Below is our sum of squared error for the best fit line. Note that the
number doesn’t mean anything much, apart from that this is the least
possible loss we would get from any lines that try to fit the data:

```python
diff = (y - X @ w)
loss = diff.T @ diff
print(loss)
```

If you don’t want intermediate variable, you can use ``np.linalg.norm``,
but to get the sum of squared loss, you have to square that after:

```python
loss = np.linalg.norm(y - X @ w, ord=2) ** 2
print(loss)
```

Visualize the loss surface
~~~~~~~~~~~~~~~~~~~~~~~~~~

Let’s confirm that our solution is really the one with lowest loss by
seeing the loss surface.

Our loss function $L(w)$ depends on two dimensions of $w$,
e.g., ``w[0]`` and ``w[1]``. If we plot $L(w)$ over possible
values of ``w[0]`` and ``w[1]``, the minimum of $L(w)$ should be
near ``w = [5.17,2.066]``, which is our solution.

To plot that out, first we have to create all possible values of w[0]
and w[1] in a grid.

```python
from mpl_toolkits.mplot3d import Axes3D

# Ranges of w0 and w1 to see, centering at the true line
spanning_radius = 10
w0range = np.arange(7-spanning_radius, 7+spanning_radius, 0.05)
w1range = np.arange(2-spanning_radius, 2+spanning_radius, 0.05)
w0grid, w1grid = np.meshgrid(w0range, w1range)

range_len = len(w0range)
print("Number of values in each axis:", range_len)
```

This means we’ll look into a total of 400*400 = 160,000 values of ``w``.
We have to calculate loss for each pair of ``w0, w1``:

```python
# Make [w0, w1] in (2, 14400) shape
all_w0w1_values = np.hstack([w0grid.flatten()[:,None], w1grid.flatten()[:,None]]).T

# Compute all losses, reshape back to grid format
all_losses = (np.linalg.norm(y - (X @ all_w0w1_values), axis=0, ord=2) ** 2).reshape((range_len, range_len))
```

Then, we can plot the loss surface (with minimum at the red point):

```python
fig = plt.figure(figsize=(10,6))
ax = fig.gca(projection='3d')

ax.plot_surface(w0grid, w1grid, all_losses, alpha=0.5, cmap='RdBu')
ax.contour(w0grid, w1grid, all_losses, offset=0, alpha=1, cmap='RdBu')
ax.scatter(w[0], w[1], loss, lw=3, c='red', s=100, label="Minimum point (5.9,2.2)")

ax.legend(loc='best')
ax.set_xlabel('w[0]')
ax.set_ylabel('w[1]')
ax.set_zlabel('L(w)')
ax.set_xticks(np.arange(7-spanning_radius, 7+spanning_radius, 2))
ax.set_yticks(np.arange(2-spanning_radius, 2+spanning_radius, 2))
ax.set_zticks([loss])
plt.show()
```

You can notice the bowl **centers** at the solution.

Using sklearn
-------------

Using ``sklearn`` for linear regression is very simple (if you already
understand all the concepts above).

```python
from sklearn.linear_model import LinearRegression
```

First we create the classifier ``clf``. If ``fit_intercept`` is ``True``
(default), then it adds the dummy ‘1’ to the ``X``. But we already did
that manually, so set it to ``False`` here.

```python
clf = LinearRegression(fit_intercept=False)
```

Then fit the data:

```python
clf.fit(X,y)
```

Check the $w$ learned, it’s the same as ours:

```python
print(clf.coef_)
```

Previously we use ``X @ w`` to predict data. For ``sklearn`` we can use
``clf.predict``:

```python
predicted = clf.predict(X)
```

And the result is the same:

```python
plt.figure()
plt.scatter(X[:,1:], y, s=10, label='Data points')
plt.plot(X[:,1:], true_targets, c='blue', label='True line', alpha=0.3)
plt.plot(X[:,1:], predicted, c='red', label='Best fit line')
plt.legend(loc='best')
plt.show()
```


# Solution to the preparatory lab
In this notebook, we use the following modules `numpy` and `maze`. The latter is a home made module, where all the solutions to the questions are implemented. We will refer to it at each answer, and we encourage you to read it.

```python
import numpy as np
import maze as mz
import pandas as pd
```

# Problem 1: Shortest path in the maze

The objective of problem 1 is to solve the shortest path problem in a maze. We start first by describing the maze as a numpy array.

```python
# Description of the maze as a numpy array
maze = np.array([
    [0, 0, 1, 0, 0, 0, 0],
    [0, 0, 1, 0, 0, 0, 0],
    [0, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0],
    [0, 1, 1, 1, 1, 1, 0],
    [0, 0, 0, 0, 0, 2, 0]
])
# with the convention
# 0 = empty cell
# 1 = obstacle
# 2 = exit of the Maze
```

The method `maze.draw_maze()` helps us draw the maze given its numpy array discription.

```python
mz.draw_maze(maze)
```

## MDP formulation

We propose the following MDP formulation:

#### State space $\mathcal{S}$
We model the state space as the set of all possible positions of the player in the maze. Note that we exclude the obstacles' position since these are impossible states to be in for the player. Formally, the state space is

$$\mathcal{S} = \big\lbrace (i,j):\textrm{such that the cell
} (i,j) \textrm{ is not an obstacle}\big\rbrace.$$
> **Note:** The choice of state space is not unique. For instance one could consider $\mathcal{S}$ to be the set of all positions in the  maze regardless of whether they correspond to an obstacle or not. But note that, this will increase the size $\vert \mathcal{S} \vert $. This is fine for small mazes, but it leads to many redundant states as the maze dimension increases.

#### Action space $\mathcal{A}$
We allow the player to chose to either move `left`, `right`, `down`, `up` or not move at all (`stay`). Note that sometimes the player cannot move in a certain direction because of an obstacle or a wall, yet we permit this to be action. We will see that this is not an issue as long as we define our transition probabilities and rewards appropriately.
Formally, the action space is

$$\mathcal{A} = \lbrace \textrm{up}, \textrm{ down}, \textrm{ left}, \textrm{ right}, \textrm{ stay} \rbrace.$$
> **Note:** Once again, the choice of the action space is not unique. For instance one could remove the action `stay` from $\mathcal{A}$, but then one should modify the transition probabilities accordingly as well as the rewards.

#### Transition probabilities $\mathcal{P}$
Note that there is no randomness involved upon taking an action by the player. As a consequence, the transition probabilities are deterministic. More precisely,
- If at state (or position) $s$ taking action (or move) $a$ does not lead to a wall or an obstacle but to another state (or position) $s'$, then $\mathbb{P}(s' \vert s, a) = 1$.
- If at state (or position)  $s$ taking action (or move) $a$ leads to a wall or an obstacle, the player remains in his state (or position) $s$, then $\mathbb{P}(s \vert s, a) = 1$.

> **Note**: Recall that for a fixed $s \in \mathcal{S}$ and $a \in \mathcal{A}$ we have $\sum_{s' \in \mathcal{S}} \mathbb{P}(s' \vert s, a) = 1$, thus if for some $s' \in \mathcal{S}$  we have $\mathbb{P}(s' \vert s, a) = 1$, then for all $s'' \in \mathcal{S} \backslash \lbrace s'\rbrace$ we have $\mathbb{P}(s'' \vert s, a) = 0$,

#### Rewards $\mathcal{R}$
The objective of the player is to find the exit of the maze while avoiding the obstacles.
   - If at state $s$, taking action $a$, leads to a wall or an obstacle then $r(s,a) = -\infty$
   - If at state $s$, taking action $a$, leads to some other position in the maze that is not the exit nor a wall nor an obstacle, then $r(s, a) = -1$.
   - If at state $s$, taking action $a$, leads to the exit then $r(s ,a) = 0$.
> **Note**: Here the rewards are independent of time (i.e. $r_t(.,.) = r(.,.)$).

### Implementation
The above MDP formulation is implemented as a class ``maze.Maze`` in the file [maze.py](./maze.py) which given a matrix description of the maze instanciates the state space, action space, transition probabilities and rewards.

> **Note:** In the class `maze.Maze` each state $s = (i,j)$ is given a unique identifier $s_{id} \in \lbrace 0, , \dots, \vert S \vert -1 \rbrace$. In other words, the state space from an implementation perspective is viewed as the set of integers $\lbrace 0, , \dots, \vert S \vert -1 \rbrace$. This mapping is done via the dictionary `self.map` and its inverse mapping via the dictionary `self.states`.

```python
# Create an environment maze
env = mz.Maze(maze)
# env.show()
```

##  Dynamic Programming

Before solving the MDP problem, recall that the finite horizon objective function is
$$
    \mathbb{E} \Big[ \sum_{t=0}^T r(s_t, a_t) \Big],
$$
where $T$ is the horizon.
Recall the Bellman equation
\begin{equation}
\forall s \in \mathcal{S} \qquad  V(s) = \max_{a \in \mathcal{A}} \Big\lbrace r(s,a) + \sum_{s' \in \mathcal{S}} \mathbb{P}(s'\vert s,a) V(s') \Big\rbrace
\end{equation}
The dynamic programming solution for the finite horizon MDP problem consists of solving the above backward recursion. The method `maze.dynamic_programming` achieves this.
> **Note:** To find the optimal path, it is enough to set the time horizon $T = 10$. Indeed, looking at the maze one can see that the player needs at least 10 steps to attain the exit $B$, if her starting position is at $A$. In fact if you set the time horizon less than 10, you will see that you do not find the optimal path.

```python
# Finite horizon
horizon = 10
# Solve the MDP problem with dynamic programming
V, policy= mz.dynamic_programming(env,horizon);
```

```python
# Simulate the shortest path starting from position A
method = 'DynProg';
start  = (0,0);
path = env.simulate(start, policy, method);
```

```python
# Show the shortest path
mz.animate_solution(maze, path)
```

## Value Iteration

Here we solve the discounted infinite-horizon MDP problem using value iteration, the objective here is to find a stationary policy $\pi$ that minimizes the infinite horizon objective with a discount factor $\gamma$
$$
    \mathbb{E} \Big[\sum_{t=0}^\infty \gamma^t r\big(s_t, \pi(s_t)\big) \Big].
$$
Recall the Bellman equation in the case of a stationary policy $\pi$
\begin{equation}
\forall s \in \mathcal{S} \qquad  V^*(s) = \max_{\pi} \Big\lbrace r(s,\pi(s)) + \gamma \sum_{s' \in \mathcal{S}} \mathbb{P}(s'\vert s,\pi(s)) V^*(s') \Big\rbrace
\end{equation}
or equivalently in terms of the Bellman operator $\mathcal{L}$
\begin{equation}
V^* =  \mathcal{L}(V^*)
\end{equation}
where
\begin{equation}
   \forall s \in \mathcal{S} \qquad  \mathcal{L}(V)(s) = \max_{\pi} \Big\lbrace r(s,\pi(s)) + \gamma \sum_{s' \in \mathcal{S}} \mathbb{P}(s'\vert s,\pi(s)) V(s') \Big\rbrace.
\end{equation}
Value iteration solves the Bellman equation described above. This method is implemented as `maze.value_iteration` in the file [maze.py]().

> **Note:** Recall that the mapping $\mathcal{L}$ is a contraction, therefore value iteration converges. To achieve an $\varepsilon>0$ approximation (i.e. $\Vert V^* - V_{n+1} \Vert \le \varepsilon$),
 the stopping criterion of value iteration is $\Vert V - \mathcal{L}(V) \Vert < \frac{1-\gamma}{\gamma}\varepsilon$.

```python
# Discount Factor
gamma   = 0.95;
# Accuracy treshold
epsilon = 0.0001;
V, policy = mz.value_iteration(env, gamma, epsilon)
```

```python
method = 'ValIter';
start  = (0,0);
path = env.simulate(start, policy, method)
```

```python
# Show the shortest path
mz.animate_solution(maze, path)
```

## Random rewards

### The new MDP formulation
As stated in the problem statement, we only modify the rewards $\mathcal{R}$ to be random. In fact we will only need to modify the rewards corresponding to the state action pair $(s,a)$ that lead to either the cell R1 or R2.
#### Rewards $\mathcal{R}$
The objective of the player is to find the exit of the maze while avoiding the obstacles.
   - If at state $s$, taking action $a$, leads to the cell R1 then the reward is random according to the following     $$ R(s,a) = \begin{cases}
            -7 \quad \textrm{ w.p. } 0.5 \\
            -1 \quad \textrm{ w.p. } 0.5
            \end{cases}
     $$
   - If at state $s$, taking action $a$, leads to the cell R2 then the reward is random according to the following
     $$ R(s,a) = \begin{cases}
            -2 \quad \textrm{ w.p. } 0.5 \\
            -1 \quad \textrm{ w.p. } 0.5
            \end{cases}
     $$
   - The remaining rewards remain deterministic and with the same values as in the previous formulation.

> **Note**: The fact that you stay in a cell for a number of rounds $n$ means that you are forced to incur the reward of ending up in that state for an additional $n$ times. Thus, instead of modifying the transition probabilities, we can modify the reward of ending up at that round by multiplying it by $n + 1$.

### Solving the new MDP
As mentioned in the appendix [random_rewards.pdf]() (see in canvas), when solving the problem we will only have to look at the average rewards instead of the realization of the rewards, and the methods implemented for the previous case remain unchanged.

> **Note**: In the implementation, the only change will be the rewards. In addition, the policies we obtain remain deterministic. However, when running a policy the accumulated reward is random, but its average over multiple repetitions should converge to the value function.

```python
# Description of the maze as a numpy array
maze = np.array([
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 0, 0, 0, 0, -1],
    [ 0, 1, 1, 1, 1, 1,  0],
    [-6, 0, 0, 0, 0, 2,  0]
])
# with the convention
#  0 = empty cell
#  1 = obstacle
#  2 = exit of the Maze
# -n = trapped cell with probability 0.5. If the cell is trapped the player must stay there for n times.
```

```python
mz.draw_maze(maze);
```

### Dynamic programming

Run the following python code to obtain the optimal solution of the newly formulated MDP with dynamic programming.

```python
# Create an environment maze
env = mz.Maze(maze, random_rewards=True)
# env.show()
```

```python
# Finite horizon
horizon = 15
# Solve the MDP problem with dynamic programming
V, policy= mz.dynamic_programming(env,horizon);
# Simulate the shortest path starting from position A
method = 'DynProg';
start  = (0,0);
path = env.simulate(start, policy, method);
# Show the shortest path
mz.animate_solution(maze, path)
```

> **Note:** The animation does not illustrate the event where the player is trapped as it assumes average rewards. Nonetheless, the shown policy is the optimal one.

### Value iteration

Run the following python code to obtain the optimal solution of the newly formulated MDP with value iteration.

```python
# Discount Factor
gamma   = 0.95;
# Accuracy treshold
epsilon = 0.0001;
V, policy = mz.value_iteration(env, gamma, epsilon)

method = 'ValIter';
start  = (0,0);
path = env.simulate(start, policy, method)
# Show the shortest path
mz.animate_solution(maze, path)
```

# Problem 2 : Plucking berries

### The new MDP formulation

In this problem, the introduction of weights is translated in our previous MDP formulation by a modification of the rewards $\mathcal{R}$. This is done by simply setting $r(s,a)$ to $w_{ij}$ if being in state $s$ and taking action $a$ leads to being in th new state $s'=(i,j)$.

```python
# Description of the maze as a numpy array
maze = np.array([
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 1, 0, 0, 0,  0],
    [ 0, 0, 0, 0, 0, 0,  0],
    [ 0, 1, 1, 1, 1, 1,  0],
    [ 0, 0, 0, 0, 0, 2,  0]
])

# Description of the weight matrix as a numpy array
w = np.array([
    [0,    1, -100,   10,   10,   10, 10],
    [0,    1, -100,   10,    0,    0, 10],
    [0,    1, -100,   10,    0,    0, 10],
    [0,    1,    1,    1,    0,    0, 10],
    [0, -100, -100, -100, -100, -100, 10],
    [0,    0,    0,    0,    0,   11, 10]
])
```

```python
# Create an environment maze
env = mz.Maze(maze, weights=w)
# env.show()
```

### Dynamic programming

Run the following python code to obtain the optimal solution of the newly formulated MDP with dynamic programming.

```python
# Finite horizon
horizon = 20
# Solve the MDP problem with dynamic programming
V, policy= mz.dynamic_programming(env,horizon);
# Simulate the shortest path starting from position A
method = 'DynProg';
start  = (0,0);
path = env.simulate(start, policy, method);
# Show the shortest path
mz.animate_solution(maze, path)
```

> **Note:** By changing the horizon from $20$ to $12$ you should observe that the optimal policy changes.

### Value iteration

Run the following python code to obtain the optimal solution of the newly formulated MDP with value iteration.

```python
# Discount Factor
gamma   = 0.50;
# Accuracy treshold
epsilon = 0.001;
V, policy = mz.value_iteration(env, gamma, epsilon)
method = 'ValIter';
start  = (0,0);
path = env.simulate(start, policy, method)
# Show the shortest path
mz.animate_solution(maze, path)
```


# Problem Statement:

\begin{equation} H_{0} : p_{gate30} - p_{gate40} >= 0 \end{equation}

\begin{equation} H_{1} : p_{gate30} - p_{gate40} < 0 \end{equation}

```python
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline
```

## Read & Understand the data

```python
### Your Code Here ###
df = pd.read_csv('cookie_cats.csv')
df.head()
```

      userid
      version
      sum_gamerounds
      retention_1
      retention_7

      0
      116
      gate_30
      3
      False
      False

      1
      337
      gate_30
      38
      True
      False

      2
      377
      gate_40
      165
      True
      False

      3
      483
      gate_40
      1
      False
      False

      4
      488
      gate_40
      179
      True
      True

```python
df.shape
```

    (90189, 5)

```python
df.isnull().any().sum()
```

    0

```python
df.userid.duplicated().sum()
```

    0

### How many player in each group?

##### Hint: Use groupby with count

```python
### Your Code Here ###
df.groupby('version').count()
```

      userid
      sum_gamerounds
      retention_1
      retention_7

      version

      gate_30
      44700
      44700
      44700
      44700

      gate_40
      45489
      45489
      45489
      45489

### What is the percentage of users that came back the day after they installed?

```python
### Your Code Here ###
df.retention_1.mean()
```

    0.4452095044850259

### What is the percentage of users of each group [gate_30, gate_40] that came back the day after they installed?

```python
### Your Code Here ###
gate_30 = df.query("version == 'gate_30'").retention_1.mean()
gate_40 = df.query("version == 'gate_40'").retention_1.mean()
gate_30, gate_40

obs_sample = gate_30 - gate_40
obs_sample
```

    0.005905169787341458

### Bootstrap the data by resampling the dataset with replacement for retention_1

##### Hint:  use .sample method with frac = 1 and replace = True
##### Hint:  groupby the result of sampling by version column then select retention_1 column and apply mean as an agg function
##### Hint: take difference in mean between the 2 groups in each iteration and append it to a list

```python
df_30 = df.query("version == 'gate_30'")
df_40 = df.query("version == 'gate_40'")
```

```python
### Your Code Here ###
diffs = []
for _ in range(10000):
    sample_30 = np.random.choice([0,1],df_30.shape[0], p = [1-gate_30, gate_30])
    sample_40 = np.random.choice([0,1],df_40.shape[0], p = [1-gate_40, gate_40])
    diff = sample_30.mean() - sample_40.mean()
    diffs.append(diff)

```

### Plot the difference distribution

```python
### Your Code Here ###
plt.hist(diffs);
```

### At alpha level 0.05, should we reject the null ?
##### Hint: Calculate the STDerr, Simulate under the null, Calculate the p-value

```python
diffs = np.array(diffs)
```

```python
### Your Code Here ###
under_null = np.random.normal(0, diffs.std(), 10000)
```

```python
### Your Code Here ###
plt.hist(under_null)
plt.axvline(x= obs_sample, color = 'r');
```

```python
P_value = (under_null < obs_sample).mean()
P_value
```

    0.9653

**Since P_Value is (0.9648) > alpha (0.05), So we Fail to Reject the Null Hypothesis.**

```python

```


# Solar-powered pump

A solar-powered pump transports water from a source underground up to a tank. The PV solar panel converts solar energy into electrical power with an efficiency of $\eta_{\text{PV}} = 0.08$ and stores the electrical energy in a battery, which powers the pump continuously. The 24-hour annual average solar flux is SF = 225 W/m$^2$.

The isentropic efficiency of the pump is 0.58, and the pump performance is given as pressure head versus flow rate:

| Head (ft)     | Flow rate (gpm) |
| ------------- | --------------- |
| 60            | 0               |
| 60            | 2               |
| 57            | 4               |
| 52            | 6               |
| 44            | 8               |
| 33            | 10              |
| 18            | 12              |
| 0             | 14              |

The water source is 20 ft below ground level (where the tank is located), so the pump must provide a pressure rise equal to the 20 ft of water. There is also head loss due to friction in the pipe; the total head loss (pressure drop due to friction) between the inlet and outlet is:
\begin{equation}
\Delta P_{\text{pipe}} = 20 \left[ \text{ft H}_2 \text{O} \right] + 0.085 \left[ \frac{ \text{ft H}_2 \text{O} }{\text{gpm}^2} \right] \dot{V}^2
\end{equation}
where $\dot{V}$ is the volumetric flow rate.

We can assume water is an incompressible fluid here.

**Problem**:

- Determine the flow rate delivered by the pump
- Determine the area of the solar panel required by the system

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from scipy import optimize
import cantera as ct

from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity
```

## Determine the flow rate of water delivered

First, specify the input information:

```python
solar_flux = Q_(225, 'W/m^2')
efficiency_pump = 0.58
efficiency_solar = 0.08
```

Next, to model the pump, we can fit the performance data to a third-order polynomial:
\begin{equation}
\Delta P_{\text{pump}} = a_0 + a_1 \dot{V} + a_2 \dot{V}^2 + a_3 \dot{V}^3
\end{equation}

We can find this fit using the NumPy function `polyfit()`, and place the result in a `poly1d` object so we can easily evaluate the polynomial:

```python
# Construct NumPy arrays with the pressure head and
# flow rate from the table
head = Q_(np.asarray([60, 60, 57, 52, 44, 33, 18, 0]), 'ft')
flow_rate = Q_(np.asarray([0, 2, 4, 6, 8, 10, 12, 14]), 'gallon per minute')

# Perform the fit with degree 3, and place in a poly1d object
pump_fit = np.poly1d(np.polyfit(flow_rate.magnitude, head.magnitude, 3))

fig, ax = plt.subplots(figsize=(5, 3))

# Plot the measurements
ax.plot(flow_rate, head, 'o')

# Generate a dense sample of flow rate values, then plot the polynomial fit
flow_rate_dense = np.linspace(flow_rate[0].magnitude, flow_rate[-1].magnitude, 100)
ax.plot(flow_rate_dense, pump_fit(flow_rate_dense))

plt.xlabel(f'Flow rate ({flow_rate.units})')
plt.ylabel(f'Pressure head ({head.units})')
plt.grid(True)
plt.legend(['Pump data', 'Best fit curve'])
fig.tight_layout()
plt.show()
```

    <IPython.core.display.Javascript object>

To find the volumetric flow rate of the pump, we need to find the condition where the pressure head of the pump matches the pressure drop of the pipe between the inlet and outlet. We can do this by setting two $\Delta P$ expressions equal to each other, and the finding the root of this equation:
\begin{equation}
\Delta P_{\text{pump}} \left(\dot{V}\right) = \Delta P_{\text{pipe}} \left(\dot{V}\right)
\end{equation}

To find the root, we can use the [SciPy function `root_scalar`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.root_scalar.html), part of its `optimize` module. This needs needs us to provide a function that rearranges the above expression to equal zero, and returns that value. In other words, the function should return:
\begin{equation}
\Delta P_{\text{pump}} \left(\dot{V}\right) - \Delta P_{\text{pipe}} \left(\dot{V}\right)
\end{equation}
which will equal zero when the root is found.

```python
def pump_performance(volume_flow_rate, pump_fit):
    '''Find root of pump performance equations'''
    volume_flow_rate *= ureg('gal/min')
    deltaP_pipe = (
        Q_(20, 'ft') +
        Q_(0.085, 'ft/((gal/min)**2)') * volume_flow_rate**2
        )
    deltaP_pump = pump_fit(volume_flow_rate) * ureg.ft
    return (deltaP_pump - deltaP_pipe).to('ft').magnitude
```

Then, we call the `root_scalar` function, giving it the function we just created along with a range of possible values and the extra argument our function needs:

```python
sol = optimize.root_scalar(
    pump_performance, bracket=[0, 14],
    args=(pump_fit,),
    )

volume_flow_rate = Q_(sol.root, 'gal/min')
print(f'Pump flow rate: {volume_flow_rate: .2f}')
```

    Pump flow rate: 10.51 gallon / minute

## Determine area of solar panels required

To determine the area of solar panels required, we first need to calculate the work needed for the pump:
\begin{equation}
\dot{W}_p = \frac{\dot{m} v \Delta P_{\text{pump}} }{\eta_p} = \frac{\dot{V} \Delta P_{\text{pump}} }{\eta_p}
\end{equation}

First, however, we'll need to convert the pressure head (given in feet of water) to an actual pressure difference, using $h = \frac{P}{\rho g}$.

```python
pressure_head = Q_(pump_fit(volume_flow_rate), 'ft')
water = ct.Water()
pressure_drop = pressure_head * Q_(water.density, 'kg/m^3') * ureg.gravity

work_pump = (
    volume_flow_rate * pressure_drop.to('Pa') /
    efficiency_pump
    )
print(f'Pump work: {work_pump.to(ureg.watt): .2f}')
```

    Pump work: 100.13 watt

Finally, we can calculate the area required by relating the power generated by the solar panels to the rate of work required by the pump:
\begin{equation}
\text{SF} \, A \eta_{\text{PV}} = \dot{W}_p
\end{equation}

```python
area_solar = work_pump / (solar_flux * efficiency_solar)
print(f'Area of solar panels required: {area_solar.to("m^2"): .2f}')
```

    Area of solar panels required: 5.56 meter ** 2