# Naive Bayes and Hyperparameter Optimization

*Hayley Boyce, May 5th, 2021*

```python
# Importing our libraries
import pandas as pd
import altair as alt
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.model_selection import cross_validate, train_test_split
from sklearn.svm import SVR, SVC

import sys
sys.path.append('code/')
from display_tree import display_tree
from plot_classifier import plot_classifier
import matplotlib.pyplot as plt

# Preprocessing and pipeline
from sklearn.impute import SimpleImputer
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, MinMaxScaler
```

## House Keeping
- Quiz Today!
- Result of Polls
- Assignment due Monday
- Project groups this week
- Project instructions next week
- Heavy class today, more learning, less practice (sorry!)

## Lecture Learning Objectives

- Explain the naive assumption of naive Bayes.
- Predict targets by hands-on toy examples using naive Bayes.
- Use `scikit-learn`'s `MultiNomialNB`.
- Use `predict_proba` and explain its usefulness.
- Explain the need for smoothing in naive Bayes.
- Explain how `alpha` controls the fundamental tradeoff.
- Explain the need for hyperparameter optimization
- Carry out hyperparameter optimization using `sklearn`'s `GridSearchCV` and `RandomizedSearchCV`.

## Five Minute Recap/ Lightning Questions

- What kind of preprocessing must I do if I have a feature with categories that have an order to them?
- How many columns do I need for a binary feature?
- What tool do we use to preprocess all our pipelines and build a model without breaking the golden rule?
- Between `Pipeline()` and `make_pipeline()`, which one assigns names to the steps on our behalf?
- In text data, what are our features made up of?

### Some lingering questions

- How do I tune multiple hyperparameters at once?
- What algorithm works well with our `spam`, `non spam` problem?

## Naive Bayes introduction -  spam/non spam
Last lecture we saw this spam classification problem where we used `CountVectorizer()` to vectorize the text into features and used an `SVC` to classify each text message into either a class of `spam` or `non spam`.

$X = \begin{bmatrix}\text{"URGENT!! You have been selected to receive a £900 prize reward!",}\\ \text{"Lol your always so convincing."}\\ \text{"Congrats! 1 year special cinema pass for 2 is yours. call 09061209465 now!"}\\ \end{bmatrix}$ and $y = \begin{bmatrix}\text{spam} \\ \text{non spam} \\ \text{spam} \end{bmatrix}$

For years, the best spam filtering methods used naive Bayes.

Naive Bayes is based on Bayes' Theorem:

- This is our first probabilistic classifier where we think of learning as a problem of statistical inference.

- Other applications of Naive Bayes:
    - Folder ordering, document clustering, etc.
    - Sentiment analysis (e.g., movies, restaurants, etc.)
    - Classifying products into groups based on descriptions
## Naive Bayes from scratch

Let's do some naive Bayes calculations **by hand**🖐 🤚 .

Yes, there is going to be some math here but it's going to be really helpful in understanding how this algorithm works!

Below we have a few texts and they are classed as either being  **spam** or **non spam**.

```python
df = pd.DataFrame({'X': [
                        "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!",
                        "Lol you are always so convincing.",
                        "Sauder has interesting courses.",
                        "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!",
                        "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!",
                        "Sauder has been interesting so far." ],
                   'y': ["spam", "non spam", "non spam", "spam", "spam", "non spam"]})
df
```

      X
      y

      0
      URGENT!! As a valued network customer you have...
      spam

      1
      Lol you are always so convincing.
      non spam

      2
      Sauder has interesting courses.
      non spam

      3
      URGENT! You have won a 1 week FREE membership ...
      spam

      4
      Had your mobile 11 months or more? U R entitle...
      spam

      5
      Sauder has been interesting so far.
      non spam

We know that we need to encode categorical data and transform it to numeric data to use it with machine learning since categoric columns throw an error when we try to fit our model.

This sounds like a job for `CountVectorizer()` since we have words that need to be converted into features!

Here we are going to set `max_features=4` to make our calculations a little easier and `stop_words='english'` so we are getting meaningful words as features and not stop words.

```python
from sklearn.feature_extraction.text import CountVectorizer

count_vect = CountVectorizer(max_features = 4, stop_words='english')
data = count_vect.fit_transform(df['X'])
train_bow_df = pd.DataFrame(data.toarray(), columns=sorted(count_vect.vocabulary_), index=df['X'])

train_bow_df['target'] = df['y'].tolist()
train_bow_df
```

      free
      prize
      sauder
      urgent
      target

      X

      URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!
      0
      1
      0
      1
      spam

      Lol you are always so convincing.
      0
      0
      0
      0
      non spam

      Sauder has interesting courses.
      0
      0
      1
      0
      non spam

      URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!
      1
      1
      0
      1
      spam

      Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!
      1
      0
      0
      0
      spam

      Sauder has been interesting so far.
      0
      0
      1
      0
      non spam

Suppose we are given 2 text messages in and we want to find the targets for these examples, how do we do it using naive Bayes?

First, let's get a numeric representation of our text messages.

```python
test_texts = ["URGENT! Free!!", "I like Sauder"]
data = count_vect.transform(test_texts).toarray()
test_bow_df = pd.DataFrame(data, columns=count_vect.vocabulary_, index=test_texts)
test_bow_df
```

      urgent
      prize
      sauder
      free

      URGENT! Free!!
      1
      0
      0
      1

      I like Sauder
      0
      0
      1
      0

Let's look at the text: "**URGENT! Free!!**"

> Is this **spam** or **non spam**?

So what we want to know is:

$$P(\textrm{spam}|\textrm{"URGENT! Free!!"})$$

$$ \text{and} $$

$$P(\textrm{non spam}|\textrm{"URGENT! Free!!"})$$

We really only care which one of these is bigger and whichever probability is larger is how we can classify our sentence as **spam** or **non spam**.

$$P(\textrm{spam}|\textrm{"URGENT! Free!!"}) > P(\textrm{non spam}|\textrm{"URGENT! Free!!"})$$

Remember our Bayes' Theorem is the following:

$$\text{P}(Y|X) = \frac{\text{P}(X | Y) \text{P}(Y)}{\text{P}(X)}$$
In this case:

$X$ is the representation of the words in our text ie; $\text{free} = 1, \text{prize} = 0, \text{sauder} = 0,  \text{urgent} = 1$

$y$ is our target either spam or non spam

Substituting into Bayes rule we get:

$$\frac{P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0,  \text{urgent} = 1 |\textrm{spam})*P(\textrm{spam})}{P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1 )}>\frac{P(\text{free} = 1, \text{prize} = 0, \text{sauder} = 0, \text{urgent} = 1 |\textrm{non spam})*P(\textrm{non spam})}{P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1 )}$$

Now, there are two reasons naive Bayes is so easy:
1. We can cancel out the denominator which leads us to this:

$$P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1|\textrm{spam})*P(\textrm{spam}) > P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1|\textrm{non spam})*P(\textrm{non spam})$$

2. We can simplify the numerator

### Naive Bayes' approximation

We assume each feature (word) is conditionally independent. (Assume that all features in $X$ are mutually independent, conditional on the target class.)

- In general,
$$P(\text{message} \mid \text{spam}) = P(w_1, w_2, . . . , w_d \mid \text{spam}) \approx \prod_{i=1}^{d}P(w_i \mid \text{spam})$$

$$P(\text{message} \mid \text{non spam}) = P(w_1, w_2, . . . , w_d \mid \text{non spam}) \approx \prod_{i=1}^{d}P(w_i \mid \text{non spam})$$

That means simply:

$$\begin{equation}
\begin{split}
& P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1 \mid \text{spam}) \\
&\approx  P(\text{free} = 1 \mid \text{spam}) \times P(\text{prize} = 0 \mid \text{spam}) \times P(\text{sauder} = 0 \mid \text{spam}) \times  P(\text{urgent} = 1 \mid \text{spam})
\end{split}
\end{equation}$$

And for the other class **non spam**:

$$\begin{equation}
\begin{split}
& P(\text{free} = 1, \text{prize} = 0,\text{sauder} = 0, \text{urgent} = 1 \mid \text{non spam}) \\
&\approx P(\text{free} = 1 \mid \text{non spam}) \times P(\text{prize} = 0 \mid \text{non spam}) \times P(\text{sauder} = 0 \mid \text{non spam}) \times P(\text{urgent} = 1 \mid \text{non spam})
\end{split}
\end{equation}$$

So our equation has boiled down to is:

$$ P(\text{free} = 1 \mid \text{spam}) \times P(\text{prize} = 0 \mid \text{spam}) \times P(\text{sauder} = 0 \mid \text{spam}) \times P(\text{urgent} = 1 \mid \text{spam})*P(\textrm{spam}) >$$
$$ P(\text{free} = 1 \mid \text{non spam}) \times P(\text{prize} = 0 \mid \text{non spam}) \times P(\text{sauder} = 0 \mid \text{non spam}) \times P(\text{urgent} = 1 \mid \text{non spam}) *P(\textrm{non spam})$$

- Now we just need to calculate each of those probabilities which is easy!

### Estimating $P(\text{spam} \mid \text{message})$ (The left side of our equation)

$$P(\text{free} = 1 \mid \text{spam}) \times P(\text{prize} = 0 \mid \text{spam})  \times P(\text{sauder} = 0 \mid \text{spam}) \times P(\text{urgent} = 1 \mid \text{spam})*P(\textrm{spam}) $$

We need the following:
1. Prior probability:
    $P(\text{spam})$

2. Conditional probabilities:
    1. $P(\text{free} = 1 \mid \text{spam})$
    2. $P(\text{prize} = 0 \mid \text{spam})$
    3. $P(\text{sauder} = 0 \mid \text{spam})$
    4. $P(\text{urgent} = 1 \mid \text{spam})$

```python
train_bow_df
```

      free
      prize
      sauder
      urgent
      target

      X

      URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!
      0
      1
      0
      1
      spam

      Lol you are always so convincing.
      0
      0
      0
      0
      non spam

      Sauder has interesting courses.
      0
      0
      1
      0
      non spam

      URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!
      1
      1
      0
      1
      spam

      Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!
      1
      0
      0
      0
      spam

      Sauder has been interesting so far.
      0
      0
      1
      0
      non spam

- Prior probability
    - $P(\text{spam}) = 3/6$

- Conditional probabilities
    -  What is  $P(\text{free} = 1 \mid \text{spam})$ ??
       - Given target is spam, how often "free"= 1? $= 2/3$
    - $P(\text{prize} = 0 \mid \text{spam}) = 1/3$
    - $P(\text{sauder} = 0 \mid \text{spam}) = 3/3$
    - $P(\text{urgent} = 1 \mid \text{spam}) = 2/3$

Now we have everything we need to do our calculations!

$$P(\textrm{spam}|\text{free} = 1, \text{prize} = 0, \text{sauder} = 0,  \text{urgent} = 1) = P(\text{free} = 1|\textrm{spam})*P(\text{prize} = 0|\textrm{spam})*P(\textrm{sauder = 0}|\textrm{spam})*P(\text{urgent} = 1|\textrm{spam})*P(\textrm{spam})$$
$$=  \frac{2}{3} * \frac{1}{3}* \frac{3}{3} * \frac{2}{3} *\frac{3}{6} $$

```python
spam_prior = 3/6
sauder0_spam = 3/3
free1_spam = 2/3
prize0_spam = 1/3
urgent1_spam = 2/3
spam_prob = spam_prior * sauder0_spam * free1_spam * prize0_spam * urgent1_spam
spam_prob
```

    0.07407407407407407

Ok, So we've done our left side! Now we have to do the right!

### Estimating $P(\text{non spam} \mid \text{message})$  (The right side of our equation)

$$P(\text{free} = 1 \mid \text{ non spam}) \times P(\text{prize} = 0 \mid \text{non spam})  \times P(\text{sauder} = 0 \mid \text{non spam}) \times P(\text{urgent} = 1 \mid \text{non spam})*P(\textrm{non spam}) $$

Now we need the following:

1. Prior probability: $P(\text{non spam})$
2. Conditional probabilities:
    1. $P(\text{free} = 1 \mid \text{non spam})$
    2. $P(\text{prize} = 0 \mid \text{non spam})$
    3. $P(\text{sauder} = 0 \mid \text{non spam})$
    4. $P(\text{urgent} = 1 \mid \text{non spam})$

Again we use the data to calculate these probabilities.

```python
train_bow_df
```

      free
      prize
      sauder
      urgent
      target

      X

      URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!
      0
      1
      0
      1
      spam

      Lol you are always so convincing.
      0
      0
      0
      0
      non spam

      Sauder has interesting courses.
      0
      0
      1
      0
      non spam

      URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!
      1
      1
      0
      1
      spam

      Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!
      1
      0
      0
      0
      spam

      Sauder has been interesting so far.
      0
      0
      1
      0
      non spam

- Prior probability
    - $P(\text{non spam}) = 3/6$

- Conditional probabilities
    - What is $P(\text{free} = 1 \mid \text{non spam})$ ?
        - Given the target is non spam, how ofter "free"=1? $0/3$
    - $P(\text{prize} = 0 \mid \text{non spam}) = 3/3$
    - $P(\text{sauder} = 0 \mid \text{non spam}) =1/3$
    - $P(\text{urgent} = 1 \mid \text{non spam}) = 0/3$

Time for our calculation:

$$P(\textrm{non spam}|\text{free} = 1, \text{prize} = 0,\text{sauder} = 0,  \text{urgent} = 1) = P(\text{free} = 1|\textrm{non spam})*P( \text{prize} = 0|\textrm{non spam})*P(\textrm{sauder = 0}|\textrm{non spam})*P(\text{urgent} = 1|\textrm{non spam})*P(\textrm{non spam})$$
$$= \frac{1}{3} * \frac{0}{3} * \frac{3}{3}* \frac{0}{3} *\frac{3}{6} $$

```python
non_spam_prior = 3/6
sauder0_non_spam = 0/3
free1_non_spam = 1/3
prize0_non_spam = 1/3
urgent1_non_spam = 2/3
non_spam_prob = non_spam_prior * sauder0_non_spam * free1_non_spam * prize0_non_spam * urgent1_non_spam
non_spam_prob
```

    0.0

so our equation:

$$ P(\text{free} = 1 \mid \text{spam}) \times P(\text{prize} = 0 \mid \text{spam}) \times P(\text{sauder} = 0 \mid \text{spam}) \times P(\text{urgent} = 1 \mid \text{spam})*P(\textrm{spam}) >$$
$$ P(\text{free} = 1 \mid \text{non spam}) \times P(\text{prize} = 0 \mid \text{non spam}) \times P(\text{sauder} = 0 \mid \text{non spam}) \times P(\text{urgent} = 1 \mid \text{non spam}) *P(\textrm{non spam})$$

has been calculated to

0.07407407407407407 > 0.0

Since our left side is greater than the right side, our text is classified as **spam**!

We could normalize this result and say 100% spam and 0% non spam so that the probabilities add up to 100%.

Now let's verify our result using sklearn.

## Naive Bayes classifier

The main Naive Bayes classifier in sklearn is called `MultinomialNB` and exists in the `naive_bayes` module.

```python
from sklearn.naive_bayes import MultinomialNB
```

```python
train_bow_df
```

      free
      prize
      sauder
      urgent
      target

      X

      URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!
      0
      1
      0
      1
      spam

      Lol you are always so convincing.
      0
      0
      0
      0
      non spam

      Sauder has interesting courses.
      0
      0
      1
      0
      non spam

      URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!
      1
      1
      0
      1
      spam

      Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!
      1
      0
      0
      0
      spam

      Sauder has been interesting so far.
      0
      0
      1
      0
      non spam

Let's split up our data into our features and targets:

```python
X_train = train_bow_df.drop(columns='target')
y_train = train_bow_df['target']
```

Here I am selecting the first row of our test set which was the **URGENT! Free!!** text.

```python
test_bow_df.iloc[[0]]
```

      urgent
      prize
      sauder
      free

      URGENT! Free!!
      1
      0
      0
      1

Here we get a prediction of spam:

```python
nb = MultinomialNB(alpha=0)
nb.fit(X_train, y_train)
nb.predict(test_bow_df.iloc[[0]])
```

    /usr/local/lib/python3.8/site-packages/sklearn/naive_bayes.py:511: UserWarning: alpha too small will result in numeric errors, setting alpha = 1.0e-10
      warnings.warn('alpha too small will result in numeric errors, '

    array(['spam'], dtype='<U8')

Instead of using `predict`,  we can use something called `predict_proba()`  with Naive Bayes classifier which gives us the ***proba***bilities of each class happening.

- `predict` returns the class with the highest probability.
- `predict_proba` gives us the actual probability scores.
- Looking at the probabilities can help us understand the model.

We will look more into this in Lecture 7.

```python
prediction =  nb.predict_proba(test_bow_df.iloc[[0]])
pd.DataFrame(data =prediction,columns = nb.classes_)
```

      non spam
      spam

      0
      2.250000e-20
      1.0

We get the same probabilities as we did it by hand.

(Ok 2.250000e-20 is essentially 0 but due to computing and storage, python specifies this 0 as an extremely small number.)

What about this warning we see?

> 'alpha too small will result in numeric errors'

Well, let's look at our conditional probabilities again from the right side of our equation.

- Conditional probabilities
    - $P(\text{free} = 1 \mid \text{non spam}) = 0/3$
    - $P(\text{prize} = 0 \mid \text{non spam}) = 3/3$
    - $P(\text{sauder} = 0 \mid \text{non spam}) =  1/3$
    - $P(\text{urgent} = 1 \mid \text{non spam}) = 0/3$

Is it wise to say that given a text that is non spam the probability of free occurring is 0?

Not really. We only are using 6 examples here and setting this to 0 (and $P(\text{urgent} = 1 \mid \text{non spam}) = 0$)   is making the whole right side of the equation equal to 0.

Naive Bayes naively multiplies all the feature likelihoods together, and if any of the terms is zero, it's going to void all other evidence and the probability of the class is going to be zero.

This is somewhat problematic.

We have limited data and if we do not see a feature occurring with a class, it doesn't mean it would never occur with that class.

How can we fix this?

### A simple solution: Laplace smoothing

- The simplest way to avoid zero probabilities is to add a value($\alpha$) to all the counts. This is called **Laplace smoothing**

Generally, we set alpha ($\alpha$) equal to 1 and in `scikit-learn` we control it using hyperparameter `alpha`.

This means that we give an instance of every word appearing once with a target of spam, as well as a target of non spam.

By default `alpha=1.0` in `scikit-learn`.

Let's see what our probabilities are now using alpha=1.

```python
nb = MultinomialNB(alpha=1)
nb.fit(X_train, y_train)
pd.DataFrame(data = nb.predict_proba(test_bow_df.iloc[[0]]),
             columns = nb.classes_)
```

      non spam
      spam

      0
      0.235849
      0.764151

A bit smoother now, wouldn't you say?

### `alpha` hyperparameter and the fundamental tradeoff

- High alpha $\rightarrow$ underfitting
    - means we are adding large counts to everything and so we are diluting the data
- Low alpha $\rightarrow$ overfitting

## Naive Bayes on Real Data

let's try `scikit-learn`'s implementation of Naive Bayes on a modified version of Kaggle's [Disaster Tweets](https://www.kaggle.com/vstepanenko/disaster-tweets).

```python
tweets_df = pd.read_csv("data/tweets_mod.csv")
tweets_df
```

      text
      target

      0
      YOU THERE, PACHIRISU PUNK, PREPARE TO BE DESTR...
      0

      1
      Face absolutely flattened against the glass of...
      0

      2
      Bruhhhh I screamed when she said that 😭 MY HEA...
      0

      3
      Granting warrants to "authorise police to ente...
      0

      4
      Ang lala hahaha I woke up to a deluge of death...
      0

      ...
      ...
      ...

      3995
      As it seems to be fairly contagious, I'm think...
      1

      3996
      #BoundBrookFire Firefighters from several diff...
      1

      3997
      It is turning out to be a very violent storm a...
      1

      3998
      A raging fire in Bound Brook, New Jersey, on S...
      1

      3999
      Hazardous eruption a possibility after Philipp...
      1

4000 rows × 2 columns

Let's split it into our training and test sets as well as our features and target objects.

```python
train_df, test_df = train_test_split(tweets_df, test_size=0.2, random_state=123)
X_train, y_train = train_df["text"], train_df["target"]
X_test, y_test = test_df["text"], test_df["target"]
train_df.head()
```

      text
      target

      1420
      How low have you sunk Alice, just clickbait fo...
      0

      1638
      Watching this tonight as I was working yesterd...
      0

      616
      January 14, 2020 at about 08:30 am, personnel ...
      0

      184
      Next oil spill you drone strike the CEO's neig...
      0

      2075
      Another 6.0 aftershock has hit Puerto Rico aft...
      1

Next, we make a pipeline and cross-validate!

```python
pipe_nb = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1))
scores = cross_validate(pipe_nb, X_train, y_train, return_train_score=True)
pd.DataFrame(scores)
```

      fit_time
      score_time
      test_score
      train_score

      0
      0.091117
      0.017300
      0.796875
      0.948438

      1
      0.076795
      0.015993
      0.801562
      0.948438

      2
      0.071020
      0.015681
      0.801562
      0.946875

      3
      0.070094
      0.013736
      0.837500
      0.945703

      4
      0.076934
      0.014068
      0.814063
      0.944531

```python
pd.DataFrame(scores).mean()
```

    fit_time       0.077192
    score_time     0.015356
    test_score     0.810312
    train_score    0.946797
    dtype: float64

## Let's Practice

Using naive Bayes by hand, what class would naive Bayes predict for the second example "I like Sauder".

```python
train_bow_df
```

      free
      prize
      sauder
      urgent
      target

      X

      URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!
      0
      1
      0
      1
      spam

      Lol you are always so convincing.
      0
      0
      0
      0
      non spam

      Sauder has interesting courses.
      0
      0
      1
      0
      non spam

      URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!
      1
      1
      0
      1
      spam

      Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!
      1
      0
      0
      0
      spam

      Sauder has been interesting so far.
      0
      0
      1
      0
      non spam

```python
test_bow_df.iloc[[1]]
```

      urgent
      prize
      sauder
      free

      I like Sauder
      0
      0
      1
      0

Let's do some of the steps here:

**spam side**

1\. Prior probability:
$P(\text{spam}) = $

2\. Conditional probabilities:

2.1 $P(\text{free} = 0 \mid \text{spam}) = $
2.2 $P(\text{prize} = 0 \mid \text{spam}) = $
2.3 $P(\text{sauder} = 1 \mid \text{spam}) = $
2.4 $P(\text{urgent} = 0 \mid \text{spam}) = $

3\. $P(\textrm{spam}|\text{free} = 0, \text{prize} = 0, \text{sauder} = 1,  \text{urgent} = 0) = $

**non spam side**

4\. Prior probability:
$P(\text{non spam}) = $

5\. Conditional probabilities:
5.1 $P(\text{free} = 0 \mid \text{non spam}) = $
5.2 $P(\text{prize} = 0 \mid \text{non spam}) = $
5.3 $P(\text{sauder} = 1 \mid \text{non spam}) = $
5.4 $P(\text{urgent} = 0 \mid \text{non spam}) = $

6\. $P(\textrm{non spam}|\text{free} = 0, \text{prize} = 0, \text{sauder} = 1,  \text{urgent} = 0) =$

**Final Class**

7\. CLASS AS:

```{admonition} Solutions!
:class: dropdown

1/. $3/6$
2.1 $1/3$
2.2 $1/3$
2.3 $0/3$
2.4 $1/3$
3\. $\frac{1}{3} * \frac{1}{3}* \frac{0}{3} * \frac{1}{3} *\frac{3}{6} = 0$

4\. $3/6$
5.1 $3/3$
5.2 $3/3$
5.3 $2/3$
5.4 $3/3$

6\. $\frac{3}{3} * \frac{3}{3}* \frac{2}{3} * \frac{3}{3} *\frac{3}{6} = 1/3$
7\. Non spam

```

## Automated Hyperparameter Optimization

We’ve seen quite a few different hyperparameters for different models.

We’ve seen `max_depth` and `min_samples_split` for decision trees.

We’ve seen `n_neighbors` and `weights` for K-Nearest Neighbours and we’ve seen `gamma` and `C` for SVMs with RBF.

We’ve even seen hyperparameters for our transformations like `strategy` for our `SimpleImputer()`.

They are important and we’ve seen they can really help optimize your model, but we’ve also seen how difficult it can be to figure out how to set them.

### The problem with hyperparameters

- We may have a lot of them. (deep learning!)
- Picking reasonable hyperparameters is important -> it helps avoid underfit or overfit models.
- Nobody knows exactly how to choose them.
- May interact with each other in unexpected ways.
- The best settings depend on the specific data/problem.
- Can take a long time to execute.

### How to pick hyperparameters

- Manual hyperparameter optimization (What we've done so far)
    - We may have some intuition about what might work.
    - It takes a lot of work.

**OR...**

- **Automated hyperparameter optimization** (hyperparameter tuning)
    - Reduce human effort.
    - Less prone to error.
    - Data-driven approaches may be effective.
    - It may be hard to incorporate intuition.
    - Overfitting on the validation set.

### Automated hyperparameter optimization

- Exhaustive grid search: [`sklearn.model_selection.GridSearchCV`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)

- Randomized hyperparameter optimization: `sklearn.model_selection.RandomizedSearchCV`

### Let's Apply it

Let's bring back the cities dataset we worked with in previous lectures.

```python
cities_df = pd.read_csv("data/canada_usa_cities.csv")
train_df, test_df = train_test_split(cities_df, test_size=0.2, random_state=123)
X_train, y_train = train_df.drop(columns=['country']), train_df['country']
X_test, y_test = test_df.drop(columns=['country']), test_df['country']
X_train.head()
```

      longitude
      latitude

      160
      -76.4813
      44.2307

      127
      -81.2496
      42.9837

      169
      -66.0580
      45.2788

      188
      -73.2533
      45.3057

      187
      -67.9245
      47.1652

## Exhaustive grid search - Trying ALL the options

We import `GridSearchCV` from `sklearn.model_selection`

```python
from sklearn.model_selection import GridSearchCV
```

We need to first decide on our model and which hyperparameters we want to tune.

We are going to use an SVC classifier.

After that, we built a dictionary called `param_grid` and we specify the values we wish to look over for the hyperparameter.

```python
param_grid = {"gamma": [0.1, 1.0, 10, 100]}
```

Then we initiate our model:

```python
svc = SVC()
```

```python
grid_search = GridSearchCV(svc, param_grid, verbose=2)
```

Assigning `verbose` tells `GridSearchCV` to print some output while it's running.

```python
grid_search.fit(X_train, y_train)
```

    Fitting 5 folds for each of 4 candidates, totalling 20 fits
    [CV] gamma=0.1 .......................................................
    [CV] ........................................ gamma=0.1, total=   0.0s
    [CV] gamma=0.1 .......................................................
    [CV] ........................................ gamma=0.1, total=   0.0s
    [CV] gamma=0.1 .......................................................
    [CV] ........................................ gamma=0.1, total=   0.0s
    [CV] gamma=0.1 .......................................................
    [CV] ........................................ gamma=0.1, total=   0.0s
    [CV] gamma=0.1 .......................................................
    [CV] ........................................ gamma=0.1, total=   0.0s
    [CV] gamma=1.0 .......................................................
    [CV] ........................................ gamma=1.0, total=   0.0s
    [CV] gamma=1.0 .......................................................
    [CV] ........................................ gamma=1.0, total=   0.0s
    [CV] gamma=1.0 .......................................................
    [CV] ........................................ gamma=1.0, total=   0.0s
    [CV] gamma=1.0 .......................................................
    [CV] ........................................ gamma=1.0, total=   0.0s
    [CV] gamma=1.0 .......................................................
    [CV] ........................................ gamma=1.0, total=   0.0s
    [CV] gamma=10 ........................................................
    [CV] ......................................... gamma=10, total=   0.0s
    [CV] gamma=10 ........................................................
    [CV] ......................................... gamma=10, total=   0.0s
    [CV] gamma=10 ........................................................
    [CV] ......................................... gamma=10, total=   0.0s
    [CV] gamma=10 ........................................................
    [CV] ......................................... gamma=10, total=   0.0s
    [CV] gamma=10 ........................................................
    [CV] ......................................... gamma=10, total=   0.0s
    [CV] gamma=100 .......................................................
    [CV] ........................................ gamma=100, total=   0.0s
    [CV] gamma=100 .......................................................
    [CV] ........................................ gamma=100, total=   0.0s
    [CV] gamma=100 .......................................................
    [CV] ........................................ gamma=100, total=   0.0s
    [CV] gamma=100 .......................................................
    [CV] ........................................ gamma=100, total=   0.0s
    [CV] gamma=100 .......................................................
    [CV] ........................................ gamma=100, total=   0.0s

    [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
    [Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=1)]: Done  20 out of  20 | elapsed:    0.1s finished

    GridSearchCV(estimator=SVC(), param_grid={'gamma': [0.1, 1.0, 10, 100]},
                 verbose=2)

The nice thing about this is we can do this for multiple hyperparameters simultaneously as well.

```python
param_grid = {
    "gamma": [0.1, 1.0, 10, 100],
    "C": [0.1, 1.0, 10, 100]
}

svc = SVC()
grid_search = GridSearchCV(svc, param_grid, cv= 5, verbose=2, n_jobs=-1)

```

```python
grid_search.fit(X_train, y_train)
```

    Fitting 5 folds for each of 16 candidates, totalling 80 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:    2.4s
    [Parallel(n_jobs=-1)]: Done  80 out of  80 | elapsed:    2.6s finished

    GridSearchCV(cv=5, estimator=SVC(), n_jobs=-1,
                 param_grid={'C': [0.1, 1.0, 10, 100],
                             'gamma': [0.1, 1.0, 10, 100]},
                 verbose=2)

The grid in `GridSearchCV` stands for the way that it’s checking the hyperparameters.

Since there 4 options for each, grid search is checking every value in each hyperparameter to one another.

That means it’s checking 4 x 4 = 16 different combinations of hyperparameter values for the model.

In `GridSearchCV` we can specify the number of folds of cross-validation with the argument `cv`.

Since we are specifying `cv=5` that means that fit is called a total of 80 times (16 different combinations x 5 cross-validation folds).

Something new we've added here is `n_jobs=-1`.

This is a little more complex.

Setting this to -1 helps make this process faster by running hyperparameter optimization in parallel instead of in a sequence.

### Implement with Pipelines

```python
pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
        ("clf", SVC())])
```

After specifying the steps in a pipeline, a user must specify a set of values for each hyperparameter in `param_grid` as we did before but this time we specify the name of the step followed by two underscores `__` and the name of the hyperparameter.

This is because the pipeline would not know which hyperparameter goes with each step. Does `gamma` correspond to the hyperparameter in `SimpleImputer()` or `StandardScaler()`?

This now gives the pipeline clear instructions on which hyperparameters correspond with which step.

```python
param_grid = {
    "clf__gamma": [0.1, 1.0, 10, 100],
    "clf__C": [0.1, 1.0, 10, 100]
}
```

Notice that we named our steps in the pipeline, so `clf` corresponds to the model initialization of the SVM classifier.

If we used `make_pipeline()` remember that the function names the steps by default the lower case name of each transformation or model.

```python
pipe = make_pipeline(SimpleImputer(strategy="median"),
                    StandardScaler(),
                    SVC())
pipe
```

    Pipeline(steps=[('simpleimputer', SimpleImputer(strategy='median')),
                    ('standardscaler', StandardScaler()), ('svc', SVC())])

```python
param_grid = {
    "svc__gamma": [0.1, 1.0, 10, 100],
    "svc__C": [0.1, 1.0, 10, 100]
}
```

Now when we initiate `GridSearchCV`, we set the first argument to the pipeline name instead of the model name this time.

```python
grid_search = GridSearchCV(pipe, param_grid, cv=5, return_train_score=True, verbose=2, n_jobs=-1)
grid_search.fit(X_train, y_train);
```

    Fitting 5 folds for each of 16 candidates, totalling 80 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:    2.2s
    [Parallel(n_jobs=-1)]: Done  80 out of  80 | elapsed:    2.6s finished

Looking a bit closer these are the steps being performed with `GridSearchCV`.

```
for gamma in [0.1, 1.0, 10, 100]:
        for C in [0.1, 1.0, 10, 100]:
            for fold in folds:
                fit in training portion with the given C and gamma
                score on validation portion
            compute average score
    pick hyperparameters with the best score
```

In this case, we can see from the output that 80 executions are done, just like we calculated (4 x 4 x 5 = 80).

### Why a grid?

If we fix `C` with a value of 1 and loop over the values of 1, 10 and 100 for `gamma`.

This results in `100` having the best score with 0.82.

Next, we fix `gamma` at `100` since that was what we found was the most optimal when `C` was equal to 1.

When we loop over the values of 1, 10 and 100 for `C` we get the most optimal value to be 10.

So naturally, we would pick the values `100` for `gamma` and `10` for `C`.

HOWEVER - if we had performed every possible combination, we would have seen that the optimal values would have actually been `10` for both `gamma` and `C`.

The same thing is shown if we did it the other way around, first fixing `gamma` at a value of 1 and then looping over all possible values of `C`.

This time the most optimal combination is `gamma` equal to 1 and `C` equal to 100 which is again not the optimal value of 10 for each.

This is why it is so important not to fix either of the hyperparameters since it won’t necessarily help you find the most optimal values.

### Now what?

How do we know what the best hyperparameter values are after fitting?

We can extract the best hyperparameter values with `.best_params_` and their corresponding score with `.best_score_`.

```python
grid_search.best_params_

```

    {'svc__C': 10, 'svc__gamma': 1.0}

```python
grid_search.best_score_
```

    0.8208556149732621

We can extract the optimal classifier inside with `.best_estimator_`.

```python
best_model = grid_search.best_estimator_
```

This has already been fully fitted on with all the data and not just a portion from cross-validation so all we need to do is score!

```python
best_model.score(X_train, y_train)
```

    0.8502994011976048

```python
best_model.score(X_test, y_test)
```

    0.8333333333333334

We can either save it as a new model and fit and score on this new one *or* we can use the `grid_search` object directly and it will by default score using the optimal model.
These both give the same results.

```python
grid_search.score(X_train, y_train)
```

    0.8502994011976048

```python
grid_search.score(X_test, y_test)
```

    0.8333333333333334

The same can be done for `.predict()` as well, either using the saved model or using the `grid_search` object directly.

```python
best_model.predict(X_test)
```

    array(['Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'USA', 'USA', 'Canada', 'Canada',
           'Canada', 'Canada', 'USA', 'Canada', 'USA', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'USA', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'USA', 'USA', 'Canada',
           'Canada', 'Canada'], dtype=object)

```python
grid_search.predict(X_test)
```

    array(['Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'USA', 'USA', 'Canada', 'Canada',
           'Canada', 'Canada', 'USA', 'Canada', 'USA', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'Canada', 'USA', 'Canada',
           'Canada', 'Canada', 'Canada', 'Canada', 'USA', 'USA', 'Canada',
           'Canada', 'Canada'], dtype=object)

### Notice any problems?

This seems pretty nice and obeys the golden rule however the new problem is the execution time.

Think about how much time it would take if we had 5 hyperparameters each with 10 different values.

That would mean we would be needing to call `cross_validate()` 100,000 times!

Exhaustive search may become infeasible fairly quickly.

**Enter randomized hyperparameter search!**

### Randomized hyperparameter optimization

```python
from sklearn.model_selection import RandomizedSearchCV
```

```python
param_grid = {
    "svc__gamma": [0.1, 1.0, 10, 100],
    "svc__C": [0.1, 1.0, 10, 100]
}
```

```python
random_search = RandomizedSearchCV(pipe, param_grid, cv=5, verbose=2, n_jobs=-1, n_iter=10)
random_search.fit(X_train, y_train);
```

    Fitting 5 folds for each of 10 candidates, totalling 50 fits

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:    0.3s finished

Notice that we use the same arguments in `RandomizedSearchCV()` as in `GridSearchCV()` however with 1 new addition - `n_iter`.

This argument gives us more control and lets us restrict how many candidates are searched over.

`GridSearchCV()` conducts `cross_validate()` on every single possible combination of the hyperparameters specified in `param_grid`.

Now we can change that and control that using `n_iter` which will pick a random subset containing the specified number of combinations.

The last time when we used exhaustive grid search, we had 80 fits (4 x 4 x 5).

This time we see only 50 fits (10 combinations instead of 16 and 5 folds)!

### Continuous values for hyperparameter tuning - optional

For randomized grid search we can search over a range of continuous values instead of discrete values like in `GridSearchCV()`.

We can specify a range of values instead of a list of values for each hyperparameter.

```python
import scipy
```

```python
param_grid = {
    "svc__C": scipy.stats.uniform(0, 100),
    "svc__gamma": scipy.stats.uniform(0, 100)}
```

```python
random_gs = RandomizedSearchCV(pipe, param_grid, n_jobs=-1, cv=10, return_train_score=True, n_iter=10)
random_gs.fit(X_train, y_train);
```

```python
random_gs.best_params_
```

    {'svc__C': 30.962086187979253, 'svc__gamma': 17.526459755968204}

```python
random_gs.best_score_
```

    0.7452205882352942

```python
random_gs.score(X_test, y_test)
```

    0.7142857142857143

**How differently does exhaustive and random search score?**

```python
grid_search.score(X_test, y_test)
```

    0.8333333333333334

```python
random_search.score(X_test, y_test)
```

    0.8333333333333334

Here, (and often) they produce similar scores.

## The problem with hyperparameter tuning - overfitting the validation set

Since we are repeating cross-validation over and over again, it’s not necessarily unseen data anymore.

This may produce overly optimistic results.

If our dataset is small and if our validation set is hit too many times, we suffer from **optimization bias** or **overfitting the validation set**.

### Example: overfitting the validation set
Attribution: [Mark Scmidt](https://www.cs.ubc.ca/~schmidtm/)

This exercise helps explain the concept of overfitting on the validation set.

Consider a multiple-choice (a,b,c,d) "test" with 10 questions:

- If you choose answers randomly, the expected grade is 25% (no bias).
- If you fill out two tests randomly and pick the best, the expected grade is 33%.
    - overfitting ~8%.
- If you take the best among 10 random tests, the expected grade is ~47%.
- If you take the best among 100, the expected grade is ~62%.
- If you take the best among 1000, the expected grade is ~73%.
    - You have so many "chances" that you expect to do well.

**But on new questions, the "random choice" accuracy is still 25%.**

```python
# Code attributed to Rodolfo Lourenzutti

number_tests = [1, 2, 10, 100, 1000]
for ntests in number_tests:
    y = np.zeros(10000)
    for i in range(10000):
        y[i] = np.max(np.random.binomial(10.0, 0.25, ntests))
    print(
        "The expected grade among the best of %d tests is : %0.2f"
        % (ntests, np.mean(y) / 10.0)
    )
```

    The expected grade among the best of 1 tests is : 0.25
    The expected grade among the best of 2 tests is : 0.33
    The expected grade among the best of 10 tests is : 0.47
    The expected grade among the best of 100 tests is : 0.62
    The expected grade among the best of 1000 tests is : 0.74

If we instead used a 100-question test then:

- Expected grade from best over 1 randomly-filled tests is 25%.
- Expected grade from best over 2 randomly-filled tests is ~27%.
- Expected grade from best over 10 randomly-filled tests is ~32%.
- Expected grade from best over 100 randomly-filled tests is ~36%.
- Expected grade from best over 1000 randomly-filled tests is ~40%.

```python
# Code attributed to Rodolfo Lourenzutti

number_tests = [1, 2, 10, 100, 1000]
for ntests in number_tests:
    y = np.zeros(10000)
    for i in range(10000):
        y[i] = np.max(np.random.binomial(100.0, 0.25, ntests))
    print(
        "The expected grade among the best of %d tests is : %0.2f"
        % (ntests, np.mean(y) / 100.0)
    )
```

    The expected grade among the best of 1 tests is : 0.25
    The expected grade among the best of 2 tests is : 0.27
    The expected grade among the best of 10 tests is : 0.32
    The expected grade among the best of 100 tests is : 0.36
    The expected grade among the best of 1000 tests is : 0.40

The optimization bias **grows with the number of things we try**.
But, optimization bias **shrinks quickly with the number of examples**.
But it’s still non-zero and growing if you over-use your validation set!

Essentially our odds of doing well on a multiple-choice exam (if we are guessing) increases the more times we can repeat and randomly take the exam again.

Because we have so many chances you’ll eventually do well and perhaps this is not representative of your knowledge (remember you are randomly guessing)

The same occurs with selecting hyperparameters.

The more hyperparameters values and combinations we try, the more likely we will randomly get a better scoring model by chance and not because the model represents the data well.

This overfitting can be decreased somewhat by increasing the number of questions or in our case, the number of examples we have.

TLDR: If your test score is lower than your validation score, it may be because did so much hyperparameter optimization that you got lucky and the bigger data set that you have, the better.

## Let's Practice

1\. Which method will attempt to find the optimal hyperparameter for the data by searching every combination possible of hyperparameter values given?
2\. Which method gives you fine-grained control over the amount of time spent searching?
3\. If I want to search for the most optimal hyperparameter values among 3 different hyperparameters each with 3 different values how many trials of cross-validation would be needed?

$x= [1,2,3]$
$y= [4,5,6]$
$z= [7,8,9]$

**True or False**

4\. A Larger `n_iter` will take longer but will search over more hyperparameter values.
5\. Automated hyperparameter optimization can only be used for multiple hyperparameters.

```{admonition} Solutions!
:class: dropdown

1. Exhaustive Grid Search (`GridSearchCV`)
2. Randomized Grid Search (`RandomizedSearchCV`)
3. $3 * 3 * 3 = 27$
4. True
5. False
```

## Let's Practice - Coding

We are going to practice grid search using our basketball dataset that we have seen before.

```python
# Loading in the data
bball_df = pd.read_csv('data/bball.csv')
bball_df = bball_df[(bball_df['position'] =='G') | (bball_df['position'] =='F')]

# Define X and y
X = bball_df.loc[:, ['height', 'weight', 'salary']]
y = bball_df['position']

# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=7)

bb_pipe = Pipeline(
            steps=[("imputer", SimpleImputer(strategy="median")),
                   ("scaler", StandardScaler()),
                   ("knn", KNeighborsClassifier())])
```

1. Using the pipeline `bb_pipe` provided, create a grid of parameters to search over `param_grid`. Search over the values 1, 5, 10, 20, 30, 40, and 50 for the hyperparameter `n_neighbors` and 'uniform' and 'distance' for the hyperparameter `weights` (make sure to call them appropriately).
2. Use `GridSearchCV` to hyperparameter tune using cross-validate equal to 10 folds. Make sure to specify the arguments `verbose=2` and `n_jobs=-1`.
3. Train your  pipeline with grid search.
4. Find the best hyperparameter values. Make sure to print these results.
5. Lastly, score your model on the test set.

```python

```

## What We've Learned Today

- How to predict by using naive Bayes.
- How to use `scikit-learn`'s `MultiNomialNB`.
- What `predict_proba` is.
- Why we need smoothing in  naive Bayes.
- How to carry out hyperparameter optimization using `sklearn`'s `GridSearchCV` and `RandomizedSearchCV`.


```python
import jax.numpy as np
from jax import jit
from discopy import Tensor
Tensor.np = np

import discopy
```

```python
#******************************************
# Generate your corpus
#******************************************
# sentences can be of the `word1 word2 word3 ...`
# if you are generating the diagrams using templates
# then you may want to consider adding POS tags (N, V, Adj) for each word
# labels should be of the form [0,1] or [1,0] for binary classification

# TODO make sentences
train_sentences = []
dev_sentences = []
test_sentences = []

# TODO make labels
train_labels = []
dev_labels = []
test_labels = []

train_labels = np.array(train_labels)
dev_labels = np.array(dev_labels)
test_labels = np.array(test_labels)
```

```python
#******************************************
# Generate your diagrams
#******************************************
# TODO generate the DisCoCat diagrams for your corpus. There are 2 options:
# 1. Follow a few fixed templates (N V N, N V Adj N, ...)
# 2. Use the online tool to generate diagrams.
# If you are using the online tool, consider writing a script.

# Suggestion: print out some of your DisCoCat diagrams using
# `grammar.draw` to check that they make sense.

# TODO implement sentence2diagram
sentence2diagram = None

train_diagrams = [sentence2diagram(s) for s in train_sentences]
dev_diagrams = [sentence2diagram(s) for s in dev_sentences]
test_diagrams = [sentence2diagram(s) for s in test_sentences]

from discopy import Word, Ty, Cup, Id, grammar
from sympy.abc import phi

n, s = Ty('n'), Ty('s')

# example of how to make your template:
# then you use your pos tags to determine which template to use
def NVN_example(w0, w1, w2):
    words = Word(w0, n) @ Word(w1, n.r @ s @ n.l) @ Word(w2, n)
    cups = Cup(n, n.r) @ Id(s) @ Cup(n.l, n)

    return words >> cups

grammar.draw(NVN_example('Alice', 'likes', 'Bob'))
```

```python
#******************************************
# Generate your circuits
#******************************************
# TODO: Implement your own functorial ansatz conversion!
# The `qnlp-tutorial` has some code on this.
# https://discopy.readthedocs.io/en/main/notebooks/qnlp-tutorial.html
# You may also find discopy's `IQPansatz` function useful.

# Suggestion: Draw some DisCoCat diagrams alongside their circuit diagrams
# and compare them

# TODO implement diagram2circuit
# diagram2circuit = None

train_circuits = [diagram2circuit(d) for d in train_diagrams]
dev_circuits = [diagram2circuit(d) for d in dev_diagrams]
test_circuits = [diagram2circuit(d) for d in test_diagrams]

# You can define parameterised circuits in DisCoPy
# using sympy variables, then substituting them using `lambdify`
from discopy import drawing
from discopy.quantum import Id, Rz, CX, CRz
from sympy.abc import symbols, phi

sym = symbols("abc")
circ = Rz(sym) @ Id(1) >> CX >> CRz(phi)
drawing.equation(circ, circ.lambdify(sym, phi)(0.1, 0.2), symbol='-->')
```

```python
# These imports will come in handy when you want to run
# backend simulations.

# from qiskit import IBMQ
# from pytket.extensions.qiskit import AerBackend, IBMQBackend, IBMQEmulatorBackend
```

```python
#******************************************
# Counting parameters in your model
#******************************************
from sympy import default_sort_key
circuits = train_circuits + dev_circuits + test_circuits

# Make sure you sort your symbols as they are returned as a set.
parameters = sorted({
    s for circ in circuits for s in circ.free_symbols
}, key=default_sort_key)
```

```python
def normalise(predicts):
    # apply smoothing to predicts
    predicts = np.abs(predicts - 1e-9)
    predicts = predicts / predicts.sum()
    return predicts

def BCE_loss(predicts, labels):
    cross_entropies = np.sum(labels * np.log2(predicts))
    cost = -1/len(labels) * np.sum(cross_entropies)

    return cost
```

```python
def make_pred_fn(circuits):
    def pred_fn(params):
        predictions = np.array([
            normalise(np.abs(c.lambdify(*parameters)(*params).eval().array))
            for c in circuits
        ])
        return predictions

    return pred_fn
```

```python
train_pred_fn = jit(make_pred_fn(train_circuits))
dev_pred_fn = jit(make_pred_fn(dev_circuits))
test_pred_fn = jit(make_pred_fn(test_circuits))

import numpy as onp
# onp.random.seed(0)

x0 = np.array(onp.random.random(len(parameters)))
print("Performing Just-In-Time compilation, this may take a while...")
train_pred_fn(x0)
dev_pred_fn(x0)
```

    Performing Just-In-Time compilation, this may take a while...

    DeviceArray([[0.59450185, 0.40549812],
                 [0.04066788, 0.9593321 ],
                 [0.1270783 , 0.8729217 ],
                 ...,
                 [0.30980638, 0.6901936 ],
                 [0.47308612, 0.5269139 ],
                 [0.396312  , 0.603688  ]], dtype=float32)

```python
def make_cost_fn(pred_fn, labels):
    costs = []
    accs = []
    params_list = []
    def cost_fn(params, **kwargs):
        predictions = pred_fn(params)

        cost = BCE_loss(predictions, labels)
        acc = np.sum(np.round(predictions) == labels) / len(labels) / 2
        costs.append(cost)
        accs.append(acc)
        params_list.append(params)

        return cost

    return cost_fn, costs, accs, params_list
```

```python
# Rerun this cell onwards, so you can reuse the JITed prediction functions

train_cost_fn, train_costs, train_accs, spsa_params = make_cost_fn(train_pred_fn, train_labels)
dev_cost_fn, dev_costs, dev_accs, callback_params = make_cost_fn(dev_pred_fn, dev_labels)
test_cost_fn, test_costs, test_accs, _ = make_cost_fn(test_pred_fn, test_labels)
```

```python
from noisyopt import minimizeSPSA
minimizeSPSA(train_cost_fn, x0=x0, a=0.13, c=0.1, niter=2000, callback=dev_cost_fn)
```

         fun: DeviceArray(0.07338803, dtype=float32)
     message: 'terminated after reaching max number of iterations'
        nfev: 4000
         nit: 2000
     success: True
           x: DeviceArray([-0.10197008,  0.21064918,  0.28642726, ...,  0.9639627 ,
                  0.3812377 ,  0.22056638], dtype=float32)

```python
import matplotlib.pylab as plt
fig, ax1 = plt.subplots()
ax1.plot(train_costs[::2], color='#54436B')
ax1.set_ylabel('Cost')
ax1.set_xlabel('Iterations')
ax1.set_title('Training Set')

ax2 = ax1.twinx()
ax2.plot(train_accs[::2], color='#50CB93')
ax2.set_ylabel('Accuracy')
plt.show()
```

```python
fig, ax1 = plt.subplots()
ax1.plot(dev_costs, color='#A03C78')
ax1.set_ylabel('Cost')
ax1.set_xlabel('Iterations')
ax1.set_title('Development Set')

ax2 = ax1.twinx()
ax2.plot(dev_accs, color='#ED8E7C')
ax2.set_ylabel('Accuracy')
plt.show()
```

```python
# when you are happy with the development set results
# run your model on the testing set

# for p in callback_params:
#     test_cost_fn(p)
```

```python
fig, ax1 = plt.subplots()
ax1.plot(test_costs, color='#7C83FD')
ax1.set_ylabel('Cost')
ax1.set_xlabel('Iterations')
ax1.set_title('Testing Set')

ax2 = ax1.twinx()
ax2.plot(test_accs, color='#96BAFF')
ax2.set_ylabel('Accuracy')
plt.show()
```

```python

```


```python
import numpy as np
```

```python
import matplotlib.pyplot as plt
from pydgrid.plot_bokeh import plot_results
import sympy as sym
import pydae.build as db
from pydae.grid_urisi import unb_ri_si
import json
```

```python
data = {
        "buses":[
                 {"bus": "B1",  "pos_x":   0, "pos_y":   0, "units": "m", "U_kV":20.0},
                 {"bus": "B2",  "pos_x":  10, "pos_y":   0, "units": "m", "U_kV":0.4},
                 {"bus": "B3",  "pos_x": 100, "pos_y":  0, "units": "m", "U_kV":0.4},
                 {"bus": "B4",  "pos_x": 120, "pos_y":  0, "units": "m", "U_kV":0.4},
                 {"bus": "B2dc",  "pos_x":  10, "pos_y":  -50, "units": "m", "U_kV":0.4},
                 {"bus": "B3dc",  "pos_x": 100, "pos_y":  -50, "units": "m", "U_kV":0.4}
                ],
        "grid_formers":[
                        {"bus": "B1",
                        "bus_nodes": [1, 2, 3], "deg": [0, -120, -240],
                        "kV": [11.547, 11.547, 11.547]},
                        {"bus": "B2dc",
                        "bus_nodes":  [1, 2, 3], "deg": [0, 0, 0],
                        "kV": [0.8, 0.0,0.0]}
                       ],
        "transformers":[
                        {"bus_j": "B1",  "bus_k": "B2",  "S_n_kVA": 1000.0, "U_j_kV":20, "U_k_kV":0.4,
                         "R_cc_pu": 0.01, "X_cc_pu":0.04, "connection": "Dyn11",   "conductors_j": 3, "conductors_k": 4,
                         "monitor":True},
                       ],
        "lines":[
                 {"bus_j": "B2",  "bus_k": "B3",  "code": "UG1", "m": 200.0},
                 {"bus_j": "B3",  "bus_k": "B4",  "code": "UG1", "m": 10.0},
                 {"bus_j": "B2dc",  "bus_k": "B3dc",  "code": "UG1dc", "m": 200.0, 'monitor':True}
                ],
        "loads":[
                 {"bus": "B4" , "kVA": 0.0, "pf": 0.85,"type":"3P+N"}
                 #{"bus": "B3dc" ,  "bus_nodes": [1, 4], "kVA": 300.0, "pf": 1,"type":"1P+N"}
                ],
        "shunts":[
                 {"bus": "B2" , "R": 0.0001, "X": 0.0, "bus_nodes": [4,0]},
                 {"bus": "B2dc" , "R": 0.00001, "X": 0.0, "bus_nodes": [4,0]}
                 ],
        "line_codes":
            {"lv_cu_150":     {"Rph":0.167,"Xph":0.08, "Rn":0.167, "Xn": 0.08},
   "UG1dc":
    {"R":[[ 0.211,  0.0,  0.0,  0.0],
          [ 0.0,  0.211,  0.0,  0.0],
          [ 0.0,  0.0,  0.211,  0.0],
          [ 0.0,  0.0,  0.0,  0.211]],
     "X":[[ 0.0,  0.0,  0.0,  0.0],
          [ 0.0,  0.0,  0.0,  0.0],
          [ 0.0,  0.0,  0.0,  0.0],
          [ 0.0,  0.0,  0.0,  0.0]]
            }
       }}

```

```python
grid = unb_ri_si(data)

v_B2_list = []
i_B2_list = []
v_B3_list = []
i_B3_list = []
for ph in ['a','b','c']:
    v_B2_ph_r,v_B2_ph_i = sym.symbols(f'v_B2_{ph}_r,v_B2_{ph}_i',real=True)
    v_B2_list += [v_B2_ph_r + 1j*v_B2_ph_i]
    i_B2_ph_r,i_B2_ph_i = sym.symbols(f'i_B2_{ph}_r,i_B2_{ph}_i',real=True)
    i_B2_list += [i_B2_ph_r + 1j*i_B2_ph_i]
    v_B3_ph_r,v_B3_ph_i = sym.symbols(f'v_B3_{ph}_r,v_B3_{ph}_i',real=True)
    v_B3_list += [v_B3_ph_r + 1j*v_B3_ph_i]
    i_B3_ph_r,i_B3_ph_i = sym.symbols(f'i_B3_{ph}_r,i_B3_{ph}_i',real=True)
    i_B3_list += [i_B3_ph_r + 1j*i_B3_ph_i]
v_B2_abc = sym.Matrix([v_B2_list]).T
i_B2_abc = sym.Matrix([i_B2_list]).T
v_B3_abc = sym.Matrix([v_B3_list]).T
i_B3_abc = sym.Matrix([i_B3_list]).T

s_B2 = v_B2_abc.T * sym.conjugate(i_B2_abc)
s_B2 = sym.expand(s_B2)
s_B3 = v_B3_abc.T * sym.conjugate(i_B3_abc)
s_B3 = sym.expand(s_B3)

i_l_B2dc_B3dc_a_r,i_l_B2dc_B3dc_n_r = sym.symbols('i_l_B2dc_B3dc_a_r,i_l_B2dc_B3dc_n_r')
v_B3dc_a_r,v_B3dc_n_r = sym.symbols('v_B3dc_a_r,v_B3dc_n_r')
v_B2dc_a_r,v_B2dc_n_r = sym.symbols('v_B2dc_a_r,v_B2dc_n_r')

p_B2dc,q_B2 = sym.symbols('p_B2dc,q_B2')
i_B2_a_r,i_B2_a_i = sym.symbols('i_B2_a_r,i_B2_a_i',real=True)

s_B2_a = v_B2_abc[0] * sym.conjugate(i_B2_abc[0])
s_B2_b = v_B2_abc[1] * sym.conjugate(i_B2_abc[1])
s_B2_c = v_B2_abc[2] * sym.conjugate(i_B2_abc[2])

eq_i_B2_a_r =  sym.re(s_B2_a) - p_B2dc/3
eq_i_B2_b_r =  sym.re(s_B2_b) - p_B2dc/3
eq_i_B2_c_r =  sym.re(s_B2_c) - p_B2dc/3
eq_i_B2_a_i =  sym.im(s_B2_a) - q_B2/3
eq_i_B2_b_i =  sym.im(s_B2_b) - q_B2/3
eq_i_B2_c_i =  sym.im(s_B2_c) - q_B2/3

i_rms = sym.sqrt(i_B2_a_r**2+i_B2_a_i**2+1)
a = 2.92075867
b = 0.44961794
c = 0.02717091
p_simple = a + b*i_rms + c*i_rms*i_rms

p_vsc_loss = p_simple*6

eq_p_B2dc = p_B2dc + i_l_B2dc_B3dc_a_r * v_B2dc_a_r + i_l_B2dc_B3dc_n_r * v_B2dc_n_r + sym.Piecewise((p_vsc_loss, p_B2dc < 1), (p_vsc_loss, p_B2dc > 0),(0.0, True))

i_B2_a_r,i_B2_b_r,i_B2_c_r,i_B2_a_i,i_B2_b_i,i_B2_c_i,p_B2dc = sym.symbols('i_B2_a_r,i_B2_b_r,i_B2_c_r,i_B2_a_i,i_B2_b_i,i_B2_c_i,p_B2dc',real=True)
g_vsc = [eq_i_B2_a_r,eq_i_B2_b_r,eq_i_B2_c_r,eq_i_B2_a_i,eq_i_B2_b_i,eq_i_B2_c_i,eq_p_B2dc]
y_vsc = [i_B2_a_r,i_B2_b_r,i_B2_c_r,i_B2_a_i,i_B2_b_i,i_B2_c_i,p_B2dc]

p_B3,p_B3dc,q_B3 = sym.symbols('p_B3,p_B3dc,q_B3',real=True)
i_B3dc_a_r,i_B3dc_n_r = sym.symbols('i_B3dc_a_r,i_B3dc_n_r',real=True)

s_B3_a = v_B3_abc[0] * sym.conjugate(i_B3_abc[0])
s_B3_b = v_B3_abc[1] * sym.conjugate(i_B3_abc[1])
s_B3_c = v_B3_abc[2] * sym.conjugate(i_B3_abc[2])

eq_i_B3_a_r =  sym.re(s_B3_a) - p_B3/3
eq_i_B3_b_r =  sym.re(s_B3_b) - p_B3/3
eq_i_B3_c_r =  sym.re(s_B3_c) - p_B3/3
eq_i_B3_a_i =  sym.im(s_B3_a) - q_B3/3
eq_i_B3_b_i =  sym.im(s_B3_b) - q_B3/3
eq_i_B3_c_i =  sym.im(s_B3_c) - q_B3/3

i_B3_a_r,i_B3_b_r,i_B3_c_r,i_B3_a_i,i_B3_b_i,i_B3_c_i,p_B3dc = sym.symbols('i_B3_a_r,i_B3_b_r,i_B3_c_r,i_B3_a_i,i_B3_b_i,i_B3_c_i,p_B3dc',real=True)

i_rms = sym.sqrt(i_B3_a_r**2+i_B3_a_i**2+1)
a = 2.92075867
b = 0.44961794
c = 0.02717091
p_simple = a + b*i_rms + c*i_rms*i_rms

p_vsc_loss = p_simple*6

eq_p_B3dc = p_B3dc - p_B3 - sym.Piecewise((p_vsc_loss, p_B3dc < 1), (p_vsc_loss, p_B3dc > 0),(0.0, True))
eq_i_B3dc_a_r = i_B3dc_a_r + p_B3dc/(v_B3dc_a_r-v_B3dc_n_r+1e-8)
eq_i_B3dc_n_r = i_B3dc_n_r + p_B3dc/(v_B3dc_n_r-v_B3dc_a_r+1e-8)

g_vsc += [eq_i_B3_a_r,eq_i_B3_b_r,eq_i_B3_c_r,eq_i_B3_a_i,eq_i_B3_b_i,eq_i_B3_c_i,eq_p_B3dc,eq_i_B3dc_a_r,eq_i_B3dc_n_r]
y_vsc += [i_B3_a_r,i_B3_b_r,i_B3_c_r,i_B3_a_i,i_B3_b_i,i_B3_c_i,p_B3dc,i_B3dc_a_r,i_B3dc_n_r]
u_vsc = {'p_B3':0.0,'q_B3':0.0,'q_B2':0.0}
xy_0_dict_vsc = {'v_B3dc_a_r':800.0,'v_B3dc_n_r':10.0}
```

    [[0.211+0.j 0.   +0.j 0.   +0.j 0.   +0.j]
     [0.   +0.j 0.211+0.j 0.   +0.j 0.   +0.j]
     [0.   +0.j 0.   +0.j 0.211+0.j 0.   +0.j]
     [0.   +0.j 0.   +0.j 0.   +0.j 0.211+0.j]]

```python
params_dict  = grid.dae['params']
f_list = grid.dae['f']
x_list = grid.dae['x']
g_list = grid.dae['g'] + g_vsc
y_list = grid.dae['y'] + y_vsc
u_dict = grid.dae['u']
for ph in ['a','b','c']:
    u_dict.pop(f'i_B2_{ph}_r')
    u_dict.pop(f'i_B2_{ph}_i')
for ph in ['a','n']:
    u_dict.pop(f'i_B3dc_{ph}_r')

u_dict.update(u_vsc)
a = sym.Symbol('a')
grid.dae['xy_0_dict'].update(xy_0_dict_vsc)

h_dict = grid.dae['h_v_m_dict']

sys_dict = {'name':'grid_trafo_2bus',
           'params_dict':params_dict,
           'f_list':f_list,
           'g_list':g_list,
           'x_list':x_list,
           'y_ini_list':y_list,
           'y_run_list':y_list,
           'u_run_dict':u_dict,
           'u_ini_dict':u_dict,
           'h_dict':h_dict
           }

db.system(sys_dict)
db.sys2num(sys_dict)

data = json.dumps(grid.dae['xy_0_dict'], indent=4)
fobj = open("xy_0_dict.json","w")
fobj.write(data)
fobj.close()
```

    system without dynamic equations, adding dummy dynamic equation

```python
plot_results(grid)
```

Figure(

id = '1300', …)

above = [],

align = 'start',

aspect_ratio = None,

aspect_scale = 1,

background = None,

background_fill_alpha = 1.0,

background_fill_color = '#ffffff',

below = [LinearAxis(id='1311', ...)],

border_fill_alpha = 1.0,

border_fill_color = '#ffffff',

center = [Grid(id='1314', ...), Grid(id='1318', ...)],

css_classes = [],

disabled = False,

extra_x_ranges = {},

extra_y_ranges = {},

frame_height = None,

frame_width = None,

height = None,

height_policy = 'auto',

hidpi = True,

js_event_callbacks = {},

js_property_callbacks = {},

left = [LinearAxis(id='1315', ...)],

lod_factor = 10,

lod_interval = 300,

lod_threshold = 2000,

lod_timeout = 500,

margin = (0, 0, 0, 0),

match_aspect = False,

max_height = None,

max_width = None,

min_border = 5,

min_border_bottom = None,

min_border_left = None,

min_border_right = None,

min_border_top = None,

min_height = None,

min_width = None,

name = None,

outline_line_alpha = 1.0,

outline_line_cap = 'butt',

outline_line_color = '#e5e5e5',

outline_line_dash = [],

outline_line_dash_offset = 0,

outline_line_join = 'bevel',

outline_line_width = 1,

output_backend = 'canvas',

plot_height = 400,

plot_width = 600,

renderers = [GlyphRenderer(id='1336', ...), GlyphRenderer(id='1342', ...), GlyphRenderer(id='1348', ...)],

reset_policy = 'standard',

right = [],

sizing_mode = None,

subscribed_events = [],

tags = [],

title = Title(id='1301', ...),

title_location = 'above',

toolbar = Toolbar(id='1325', ...),

toolbar_location = 'right',

toolbar_sticky = True,

visible = True,

width = None,

width_policy = 'auto',

x_range = DataRange1d(id='1303', ...),

x_scale = LinearScale(id='1307', ...),

y_range = DataRange1d(id='1305', ...),

y_scale = LinearScale(id='1309', ...))

```python
u_dict
```

    {'v_B1_a_r': 11547.0,
     'v_B1_a_i': 0.0,
     'v_B1_b_r': -5773.499999999997,
     'v_B1_b_i': -9999.995337498915,
     'v_B1_c_r': -5773.5000000000055,
     'v_B1_c_i': 9999.99533749891,
     'v_B2dc_a_r': 800.0,
     'v_B2dc_a_i': 0.0,
     'v_B2dc_b_r': 0.0,
     'v_B2dc_b_i': 0.0,
     'v_B2dc_c_r': 0.0,
     'v_B2dc_c_i': 0.0,
     'i_B4_n_r': 0.0,
     'i_B4_n_i': 0.0,
     'i_B2_n_r': 0.0,
     'i_B2_n_i': 0.0,
     'i_B3_a_r': 0.0,
     'i_B3_a_i': 0.0,
     'i_B3_b_r': 0.0,
     'i_B3_b_i': 0.0,
     'i_B3_c_r': 0.0,
     'i_B3_c_i': 0.0,
     'i_B3_n_r': 0.0,
     'i_B3_n_i': 0.0,
     'i_B2dc_n_r': 0.0,
     'i_B2dc_n_i': 0.0,
     'i_B3dc_a_i': 0.0,
     'i_B3dc_b_r': 0.0,
     'i_B3dc_b_i': 0.0,
     'i_B3dc_c_r': 0.0,
     'i_B3dc_c_i': 0.0,
     'i_B3dc_n_i': 0.0,
     'p_B4_a': 0.0,
     'q_B4_a': -0.0,
     'p_B4_b': -0.0,
     'q_B4_b': 0.0,
     'p_B4_c': 0.0,
     'q_B4_c': 0.0,
     'p_B3': 0.0,
     'q_B3': 0.0,
     'q_B2': 0.0,
     'u_dummy': 1.0}

```python
y_list
```

    [v_B4_a_r,
     v_B4_a_i,
     v_B4_b_r,
     v_B4_b_i,
     v_B4_c_r,
     v_B4_c_i,
     v_B4_n_r,
     v_B4_n_i,
     v_B2_a_r,
     v_B2_a_i,
     v_B2_b_r,
     v_B2_b_i,
     v_B2_c_r,
     v_B2_c_i,
     v_B2_n_r,
     v_B2_n_i,
     v_B3_a_r,
     v_B3_a_i,
     v_B3_b_r,
     v_B3_b_i,
     v_B3_c_r,
     v_B3_c_i,
     v_B3_n_r,
     v_B3_n_i,
     v_B2dc_n_r,
     v_B2dc_n_i,
     v_B3dc_a_r,
     v_B3dc_a_i,
     v_B3dc_b_r,
     v_B3dc_b_i,
     v_B3dc_c_r,
     v_B3dc_c_i,
     v_B3dc_n_r,
     v_B3dc_n_i,
     i_t_B1_B2_a_r,
     i_t_B1_B2_a_i,
     i_t_B1_B2_b_r,
     i_t_B1_B2_b_i,
     i_t_B1_B2_c_r,
     i_t_B1_B2_c_i,
     i_l_B2dc_B3dc_a_r,
     i_l_B2dc_B3dc_a_i,
     i_l_B2dc_B3dc_b_r,
     i_l_B2dc_B3dc_b_i,
     i_l_B2dc_B3dc_c_r,
     i_l_B2dc_B3dc_c_i,
     i_l_B2dc_B3dc_n_r,
     i_l_B2dc_B3dc_n_i,
     i_B4_a_r,
     i_B4_a_i,
     i_B4_b_r,
     i_B4_b_i,
     i_B4_c_r,
     i_B4_c_i,
     i_B4_n_r,
     i_B4_n_i,
     i_B2_a_r,
     i_B2_b_r,
     i_B2_c_r,
     i_B2_a_i,
     i_B2_b_i,
     i_B2_c_i,
     p_B2dc,
     i_B3_a_r,
     i_B3_b_r,
     i_B3_c_r,
     i_B3_a_i,
     i_B3_b_i,
     i_B3_c_i,
     p_B3dc,
     i_B3dc_a_r,
     i_B3dc_n_r]

```python

```

## Losses

```python
params_30A = {
 'a_i': 1.95773625,
 'b_i': 0.32430738,
 'c_i':-0.22694456,
 'd_i': 0.0155737 ,
 'e_i':-0.01093976,
 'a_d': 0.96302242,
 'b_d': 0.02653386,
 'c_d': 0.05055759,
 'd_d': 0.01328264,
 'e_d': 0.01394945,
 'R_th_igbt_sink': 0.5167999063238943,
 'R_th_diode_sink': 0.8394969099630814,
 'R_th_sink_a': 0.6619445743152772}

params = params_30A

a_i = params['a_i']
b_i = params['b_i']
c_i = params['c_i']
d_i = params['d_i']
e_i = params['e_i']
a_d = params['a_d']
b_d = params['b_d']
c_d = params['c_d']
d_d = params['d_d']
e_d = params['e_d']

m = 0.95
fp = 0.8
i_rms = np.linspace(0,30,600)
p_igbt  = a_i + (b_i - c_i*m*fp)*i_rms + (d_i - e_i*m*fp)*i_rms*i_rms
p_diode = a_d + (b_d - c_d*m*fp)*i_rms + (d_d - e_d*m*fp)*i_rms*i_rms

p_switch = p_igbt + p_diode
#print(f'p_switch = {p_switch} W')
```

```python
fig_1,  axes   = plt.subplots(nrows=2,ncols=1, figsize=(7, 7))

m = 0.95
fp = 0.8
i_rms = np.linspace(0,30,600)
p_igbt  = a_i + (b_i - c_i*m*fp)*i_rms + (d_i - e_i*m*fp)*i_rms*i_rms
p_diode = a_d + (b_d - c_d*m*fp)*i_rms + (d_d - e_d*m*fp)*i_rms*i_rms

p_switch = p_igbt + p_diode

axes[0].plot(i_rms,p_igbt)
axes[0].plot(i_rms,p_diode)
axes[0].plot(i_rms,p_switch)
axes[1].plot(i_rms,p_switch)

m = 0.4
fp = 0.8
i_rms = np.linspace(0,30,600)
p_igbt  = a_i + (b_i - c_i*m*fp)*i_rms + (d_i - e_i*m*fp)*i_rms*i_rms
p_diode = a_d + (b_d - c_d*m*fp)*i_rms + (d_d - e_d*m*fp)*i_rms*i_rms

p_switch = p_igbt + p_diode

a =  2.92075867
b = 0.4496179432
c = 0.0271709136
p_simple = a + b*i_rms + c*i_rms*i_rms
axes[1].plot(i_rms,p_simple)
```

```python
i_rms*400*np.sqrt(3)
```

    array([    0.        ,    34.69884756,    69.39769513,   104.09654269,
             138.79539026,   173.49423782,   208.19308538,   242.89193295,
             277.59078051,   312.28962808,   346.98847564,   381.6873232 ,
             416.38617077,   451.08501833,   485.7838659 ,   520.48271346,
             555.18156102,   589.88040859,   624.57925615,   659.27810372,
             693.97695128,   728.67579884,   763.37464641,   798.07349397,
             832.77234154,   867.4711891 ,   902.17003666,   936.86888423,
             971.56773179,  1006.26657936,  1040.96542692,  1075.66427448,
            1110.36312205,  1145.06196961,  1179.76081718,  1214.45966474,
            1249.1585123 ,  1283.85735987,  1318.55620743,  1353.255055  ,
            1387.95390256,  1422.65275012,  1457.35159769,  1492.05044525,
            1526.74929282,  1561.44814038,  1596.14698794,  1630.84583551,
            1665.54468307,  1700.24353064,  1734.9423782 ,  1769.64122576,
            1804.34007333,  1839.03892089,  1873.73776846,  1908.43661602,
            1943.13546358,  1977.83431115,  2012.53315871,  2047.23200628,
            2081.93085384,  2116.6297014 ,  2151.32854897,  2186.02739653,
            2220.72624409,  2255.42509166,  2290.12393922,  2324.82278679,
            2359.52163435,  2394.22048191,  2428.91932948,  2463.61817704,
            2498.31702461,  2533.01587217,  2567.71471973,  2602.4135673 ,
            2637.11241486,  2671.81126243,  2706.51010999,  2741.20895755,
            2775.90780512,  2810.60665268,  2845.30550025,  2880.00434781,
            2914.70319537,  2949.40204294,  2984.1008905 ,  3018.79973807,
            3053.49858563,  3088.19743319,  3122.89628076,  3157.59512832,
            3192.29397589,  3226.99282345,  3261.69167101,  3296.39051858,
            3331.08936614,  3365.78821371,  3400.48706127,  3435.18590883,
            3469.8847564 ,  3504.58360396,  3539.28245153,  3573.98129909,
            3608.68014665,  3643.37899422,  3678.07784178,  3712.77668935,
            3747.47553691,  3782.17438447,  3816.87323204,  3851.5720796 ,
            3886.27092717,  3920.96977473,  3955.66862229,  3990.36746986,
            4025.06631742,  4059.76516499,  4094.46401255,  4129.16286011,
            4163.86170768,  4198.56055524,  4233.25940281,  4267.95825037,
            4302.65709793,  4337.3559455 ,  4372.05479306,  4406.75364063,
            4441.45248819,  4476.15133575,  4510.85018332,  4545.54903088,
            4580.24787845,  4614.94672601,  4649.64557357,  4684.34442114,
            4719.0432687 ,  4753.74211627,  4788.44096383,  4823.13981139,
            4857.83865896,  4892.53750652,  4927.23635409,  4961.93520165,
            4996.63404921,  5031.33289678,  5066.03174434,  5100.73059191,
            5135.42943947,  5170.12828703,  5204.8271346 ,  5239.52598216,
            5274.22482973,  5308.92367729,  5343.62252485,  5378.32137242,
            5413.02021998,  5447.71906755,  5482.41791511,  5517.11676267,
            5551.81561024,  5586.5144578 ,  5621.21330537,  5655.91215293,
            5690.61100049,  5725.30984806,  5760.00869562,  5794.70754319,
            5829.40639075,  5864.10523831,  5898.80408588,  5933.50293344,
            5968.20178101,  6002.90062857,  6037.59947613,  6072.2983237 ,
            6106.99717126,  6141.69601883,  6176.39486639,  6211.09371395,
            6245.79256152,  6280.49140908,  6315.19025665,  6349.88910421,
            6384.58795177,  6419.28679934,  6453.9856469 ,  6488.68449447,
            6523.38334203,  6558.08218959,  6592.78103716,  6627.47988472,
            6662.17873228,  6696.87757985,  6731.57642741,  6766.27527498,
            6800.97412254,  6835.6729701 ,  6870.37181767,  6905.07066523,
            6939.7695128 ,  6974.46836036,  7009.16720792,  7043.86605549,
            7078.56490305,  7113.26375062,  7147.96259818,  7182.66144574,
            7217.36029331,  7252.05914087,  7286.75798844,  7321.456836  ,
            7356.15568356,  7390.85453113,  7425.55337869,  7460.25222626,
            7494.95107382,  7529.64992138,  7564.34876895,  7599.04761651,
            7633.74646408,  7668.44531164,  7703.1441592 ,  7737.84300677,
            7772.54185433,  7807.2407019 ,  7841.93954946,  7876.63839702,
            7911.33724459,  7946.03609215,  7980.73493972,  8015.43378728,
            8050.13263484,  8084.83148241,  8119.53032997,  8154.22917754,
            8188.9280251 ,  8223.62687266,  8258.32572023,  8293.02456779,
            8327.72341536,  8362.42226292,  8397.12111048,  8431.81995805,
            8466.51880561,  8501.21765318,  8535.91650074,  8570.6153483 ,
            8605.31419587,  8640.01304343,  8674.711891  ,  8709.41073856,
            8744.10958612,  8778.80843369,  8813.50728125,  8848.20612882,
            8882.90497638,  8917.60382394,  8952.30267151,  8987.00151907,
            9021.70036664,  9056.3992142 ,  9091.09806176,  9125.79690933,
            9160.49575689,  9195.19460446,  9229.89345202,  9264.59229958,
            9299.29114715,  9333.98999471,  9368.68884228,  9403.38768984,
            9438.0865374 ,  9472.78538497,  9507.48423253,  9542.1830801 ,
            9576.88192766,  9611.58077522,  9646.27962279,  9680.97847035,
            9715.67731792,  9750.37616548,  9785.07501304,  9819.77386061,
            9854.47270817,  9889.17155574,  9923.8704033 ,  9958.56925086,
            9993.26809843, 10027.96694599, 10062.66579356, 10097.36464112,
           10132.06348868, 10166.76233625, 10201.46118381, 10236.16003138,
           10270.85887894, 10305.5577265 , 10340.25657407, 10374.95542163,
           10409.6542692 , 10444.35311676, 10479.05196432, 10513.75081189,
           10548.44965945, 10583.14850702, 10617.84735458, 10652.54620214,
           10687.24504971, 10721.94389727, 10756.64274484, 10791.3415924 ,
           10826.04043996, 10860.73928753, 10895.43813509, 10930.13698266,
           10964.83583022, 10999.53467778, 11034.23352535, 11068.93237291,
           11103.63122047, 11138.33006804, 11173.0289156 , 11207.72776317,
           11242.42661073, 11277.12545829, 11311.82430586, 11346.52315342,
           11381.22200099, 11415.92084855, 11450.61969611, 11485.31854368,
           11520.01739124, 11554.71623881, 11589.41508637, 11624.11393393,
           11658.8127815 , 11693.51162906, 11728.21047663, 11762.90932419,
           11797.60817175, 11832.30701932, 11867.00586688, 11901.70471445,
           11936.40356201, 11971.10240957, 12005.80125714, 12040.5001047 ,
           12075.19895227, 12109.89779983, 12144.59664739, 12179.29549496,
           12213.99434252, 12248.69319009, 12283.39203765, 12318.09088521,
           12352.78973278, 12387.48858034, 12422.18742791, 12456.88627547,
           12491.58512303, 12526.2839706 , 12560.98281816, 12595.68166573,
           12630.38051329, 12665.07936085, 12699.77820842, 12734.47705598,
           12769.17590355, 12803.87475111, 12838.57359867, 12873.27244624,
           12907.9712938 , 12942.67014137, 12977.36898893, 13012.06783649,
           13046.76668406, 13081.46553162, 13116.16437919, 13150.86322675,
           13185.56207431, 13220.26092188, 13254.95976944, 13289.65861701,
           13324.35746457, 13359.05631213, 13393.7551597 , 13428.45400726,
           13463.15285483, 13497.85170239, 13532.55054995, 13567.24939752,
           13601.94824508, 13636.64709265, 13671.34594021, 13706.04478777,
           13740.74363534, 13775.4424829 , 13810.14133047, 13844.84017803,
           13879.53902559, 13914.23787316, 13948.93672072, 13983.63556829,
           14018.33441585, 14053.03326341, 14087.73211098, 14122.43095854,
           14157.12980611, 14191.82865367, 14226.52750123, 14261.2263488 ,
           14295.92519636, 14330.62404393, 14365.32289149, 14400.02173905,
           14434.72058662, 14469.41943418, 14504.11828175, 14538.81712931,
           14573.51597687, 14608.21482444, 14642.913672  , 14677.61251957,
           14712.31136713, 14747.01021469, 14781.70906226, 14816.40790982,
           14851.10675739, 14885.80560495, 14920.50445251, 14955.20330008,
           14989.90214764, 15024.60099521, 15059.29984277, 15093.99869033,
           15128.6975379 , 15163.39638546, 15198.09523303, 15232.79408059,
           15267.49292815, 15302.19177572, 15336.89062328, 15371.58947084,
           15406.28831841, 15440.98716597, 15475.68601354, 15510.3848611 ,
           15545.08370866, 15579.78255623, 15614.48140379, 15649.18025136,
           15683.87909892, 15718.57794648, 15753.27679405, 15787.97564161,
           15822.67448918, 15857.37333674, 15892.0721843 , 15926.77103187,
           15961.46987943, 15996.168727  , 16030.86757456, 16065.56642212,
           16100.26526969, 16134.96411725, 16169.66296482, 16204.36181238,
           16239.06065994, 16273.75950751, 16308.45835507, 16343.15720264,
           16377.8560502 , 16412.55489776, 16447.25374533, 16481.95259289,
           16516.65144046, 16551.35028802, 16586.04913558, 16620.74798315,
           16655.44683071, 16690.14567828, 16724.84452584, 16759.5433734 ,
           16794.24222097, 16828.94106853, 16863.6399161 , 16898.33876366,
           16933.03761122, 16967.73645879, 17002.43530635, 17037.13415392,
           17071.83300148, 17106.53184904, 17141.23069661, 17175.92954417,
           17210.62839174, 17245.3272393 , 17280.02608686, 17314.72493443,
           17349.42378199, 17384.12262956, 17418.82147712, 17453.52032468,
           17488.21917225, 17522.91801981, 17557.61686738, 17592.31571494,
           17627.0145625 , 17661.71341007, 17696.41225763, 17731.1111052 ,
           17765.80995276, 17800.50880032, 17835.20764789, 17869.90649545,
           17904.60534302, 17939.30419058, 17974.00303814, 18008.70188571,
           18043.40073327, 18078.09958084, 18112.7984284 , 18147.49727596,
           18182.19612353, 18216.89497109, 18251.59381866, 18286.29266622,
           18320.99151378, 18355.69036135, 18390.38920891, 18425.08805648,
           18459.78690404, 18494.4857516 , 18529.18459917, 18563.88344673,
           18598.5822943 , 18633.28114186, 18667.97998942, 18702.67883699,
           18737.37768455, 18772.07653212, 18806.77537968, 18841.47422724,
           18876.17307481, 18910.87192237, 18945.57076994, 18980.2696175 ,
           19014.96846506, 19049.66731263, 19084.36616019, 19119.06500776,
           19153.76385532, 19188.46270288, 19223.16155045, 19257.86039801,
           19292.55924558, 19327.25809314, 19361.9569407 , 19396.65578827,
           19431.35463583, 19466.0534834 , 19500.75233096, 19535.45117852,
           19570.15002609, 19604.84887365, 19639.54772122, 19674.24656878,
           19708.94541634, 19743.64426391, 19778.34311147, 19813.04195903,
           19847.7408066 , 19882.43965416, 19917.13850173, 19951.83734929,
           19986.53619685, 20021.23504442, 20055.93389198, 20090.63273955,
           20125.33158711, 20160.03043467, 20194.72928224, 20229.4281298 ,
           20264.12697737, 20298.82582493, 20333.52467249, 20368.22352006,
           20402.92236762, 20437.62121519, 20472.32006275, 20507.01891031,
           20541.71775788, 20576.41660544, 20611.11545301, 20645.81430057,
           20680.51314813, 20715.2119957 , 20749.91084326, 20784.60969083])

```python
m = 0.7
fp= 0.8
b = (b_i - c_i*m*fp + b_d - c_d*m*fp)
c = (d_i - e_i*m*fp + d_d - e_d*m*fp)
print(f'b = {b}')
print(f'c = {c}')
```

    b = 0.4496179432
    c = 0.027170913600000002

```python
a_i+a_d
```

    2.92075867

```python
6*40/20000*100
```

    1.2

```python

```


**This is Chapter 8 in the first edition, Chapter 9-10 in the second edition. There are additional problems in the first edition that I thought were worth solving.**

I starred solutions to problems I was unsure about.

### First Edition

##### 8.1

Show that table-lookup TD($\lambda $) is a special case of general TD($\lambda $) as given by equations ((8.5)-(8.7)).

The function approximation (FA) case is a general expression of the tabular TD($\lambda$), where the TD error $\delta_t = r_{t+1} + \gamma V_t(s_{t+1}) - V_t(s_t)$ are identical in both cases, but we can express the tabular value function as $V_t(s) = \theta_{t}(s)$ where $\theta_{t}(s) \in \mathbb{R}$, to re-derive the tabular updates from 8.5-8.7. The update to the eligibility trace reduces as follows, $\vec{e}_{t} = \gamma \lambda \vec{e}_{t-1} + \nabla_{\vec{\theta}_t} V_t(s_t)  = \gamma \lambda \vec{e}_{t-1} + \nabla_{\vec{\theta}_t} \vec{\theta_t}(s_t) =  \gamma \lambda \vec{e}_{t-1} + \delta(s - s_t)  $, which is identical to the tabular case.

##### 8.2

State aggregation is a simple form of generalizing function approximation in which states are grouped together, with one table entry (value estimate) used for each group. Whenever a state in a group is encountered, the group's entry is used to determine the state's value, and when the state is updated, the group's entry is updated. Show that this kind of state aggregation is a special case of a gradient method such as (8.4).

The state aggregation simply replaces $V_t(s_t)$ in 8.4 with a function that maps a set of states grouped into $N$ groups, to a value for each group, $S_i \rightarrow v_i \in \mathbb{R}, \forall i \in \{0, 1, \dots, N\} $.

##### 8.3*

The equations given in this section are for the on-line version of gradient-descent TD($\lambda $). What are the equations for the off-line version? Give a complete description specifying the new approximate value function at the end of an episode, $V'$, in terms of the approximate value function used during the episode, $V$. Start by modifying a forward-view equation for TD($\lambda $), such as (8.4).

I'm assuming the off-line version is just using the forward view of eligibility traces instead of the backward view. The algorithm is thus given in section 9.5 of the second edition. Given an $n$ step return, we loop through an episode for each time step $t$, let $\tau = t- n-1$. If $\tau \geq 0$,we obtain the n-step return and update Equation 8.4 accordingly.

If the question is referring to off-policy, the off-policy implementation would be making updates to a greedy policy (as in chapter 6 & 7 of edition 1), while following an epsilon-greedy policy. 8.4 would then be $\vec{\theta_{t+1}} = \vec{\theta_t} + \alpha [R_t^\lambda - \max Q_t(s_{t+1}, a')] \nabla_{\vec{\theta_t}} Q_t(s_t, a_t) $. Given $\vec{\theta_{t+1}}$, we then have $V' = V(\vec{\theta_{t+1}})$ where $V$ is the value function at the beginning of the episode and $V'$ is the update value function. We see later in the chapter that off-policy function approximation can diverge.

##### 8.4*

For off-policy, we can show that the forward and backward views are identical along a similar line as in (7.4 1st edition). We know that for off-policy, we have the exact result that $R_t^\lambda - V_t(s_t) = \sum_{k=t}^{T-1} (\gamma \lambda)^{k-t} \delta_{t + k} $ where $\delta_{t + k} = r_{t+k} + \gamma V(s_{t_k+1}) - V_t(s_k)$ (7.4).

In addition, we know using Equation 8.7, that $e_t(s) = \sum_{k=0}^t (\gamma \lambda)^{t-k} \nabla_{\vec{\theta_t}} V_t(s_t)$ (similar to equation 7.8).

It is easy now to show that the forward and backward off-policy equations 8.4-8.7 are equivalent:

$
\begin{equation}
\begin{split}
\vec{\theta_{t+1}} =& \vec{\theta_t} + \alpha[R_t^\lambda - V_t(s_t)] \nabla_{\vec{\theta_t}} V_t(s_t) \\
=& \vec{\theta_t} + \alpha \sum_{k=t}^{T-1} (\gamma \lambda)^{k-t} \delta_k \nabla_{\vec{\theta_t}} V_t(s_t)\\
=& \vec{\theta_t} + \alpha \delta_t \sum_{k=0}^{t} (\gamma \lambda)^{t-k} \nabla_{\vec{\theta_t}} V_t(s_t)\\
=& \vec{\theta_t} + \alpha \delta_t \vec{e_t(s)}
\end{split}
\end{equation}
$

Step 2 to 3 above use the results found in Equations 7.9-7.12.

##### 8.5

How could we reproduce the tabular case within the linear framework?

The tabular case can be reproduced if $\vec{\phi_s}$ is an orthogonal basis of unit vectors over the entire state space for each $s \in S$. Each $\theta$ would then correspond exactly to a $V(s)$ when performing an inner product with $\vec{\phi_s}$ and $\vec{\theta}$.

##### 8.6
How could we reproduce the state aggregation case (see Exercise 8.4 **should be Exercise 8.2**) within the linear framework?

One can reproduce the state aggregation case by making $\vec{\phi_s}$ an orthogonal basis on the group space instead of the entire state space.

##### 8.7

Suppose we believe that one of two state dimensions is more likely to have an effect on the value function than is the other, that generalization should be primarily across this dimension rather than along it. What kind of tilings could be used to take advantage of this prior knowledge?

A tiling that is dense in the dimension that is important, but sparse in the other dimension, would be ideal in this case.

##### 8.8**

Describe how the actor-critic control method can be combined with gradient-descent function approximation.

One example would be to make the critic $\delta_t = r_{t+1} + \gamma Q_t(s_{t+1}, a_{t+1}) - Q(s_t, a_t) $, and we update $\theta$ as usual where the traces are updated as in semi-gradient SARSA (I'm not sure about a more complicated trace update as shown in 7.7 in the first edition). The actor could also use an update as in Equation 7.14, but since we are approximation $Q(s,a)$, it might make sense to also approximate $p(s,a)$. I think we can use $\theta$ as a function approximation to $p(s,a)$, or equivalently using $Q(s,a)$ as our actor. I'm not too sure about this.

##### 8.9

Look up the paper by Baird (1995) on the Internet and obtain his counterexample for Q-learning. Implement it and demonstrate the divergence.

I only implemented the DP counter-example from the book.

```python
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
```

```python

class DP_Baird:
    def __init__(self):
        self.alpha = 0.01
        self.epsilon = 0.01
        self.gamma = 0.99

        # x is the feature set for each state
        self.x = np.array([
                [0, 0, 0, 0, 0, 0, 0],
                [1, 2, 0, 0, 0, 0, 0],
                [1, 0, 2, 0, 0, 0, 0],
                [1, 0, 0, 2, 0, 0, 0],
                [1, 0, 0, 0, 2, 0, 0],
                [1, 0, 0, 0, 0, 2, 0],
                [2, 0, 0, 0, 0, 0, 1]
            ])

        # v is the value function, v = w * x
        self.v = np.zeros((7))
        # w is the weight for each feature in x
        self.w = np.array([1., 1., 1., 1., 1., 10., 1.])
        # delta_w is w after the gradient updated
        self.delta_w = self.w.copy()

    def dp_update(self):
        """
        Update V using the dynamic programming update for Baird's counterexample
        https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node90.html
        """
        self.delta_w *= 0
        # the approximation of the value function is linear
        self.v = self.x.dot(self.w)

        # obtain the DP updates for w for states 1-5
        for i in range(1, 6):
            # for states 1-5, you automatically wind up in state 6
            alpha_error = self.alpha * (self.gamma * self.v[6] - self.v[i])
            self.delta_w += alpha_error * self.x[i]

        # add the transitions from state 6 to the DP updates
        for i in range(7):
            # with epsilon probability you go to the terminal state
            alpha_error1 = self.alpha * self.epsilon * (0 - self.v[6])
            # with (1 - epsilon) probability you wind up in v[6]
            alpha_error2 = self.alpha * (1 - self.epsilon) * (self.gamma * self.v[6] - self.v[6])
            self.delta_w[i] += alpha_error1 * self.x[6, i]
            self.delta_w[i] += alpha_error2 * self.x[6, i]

        self.w += self.delta_w
        return self.w.copy()

```

```python
dp_baird = DP_Baird()

w_array = np.array([dp_baird.dp_update() for i in range(10000)])
```

```python
for idx, w in enumerate(w_array):
    w_array[idx] = [np.log10(ww) if ww > 0 else -np.log10(-ww + 1) for ww in w]

```

```python
plt.plot(w_array[:, 6])
plt.plot(w_array[:, 5])
```

### Second Edition

##### 9.1
$(N+1)^d$ since there are $N+1$ polynomial functions for each dimension $d$. Each basis function is a product of all states, thus there are $(N+1)^d$ functions.

##### 9.2

$N=2$, $c_i$ are just the exponents on state 1 and 2 for each feature.

##### 9.3

Because $N+1$ integer vectors can be assigned to each dimension $d$.

##### 9.4

Same as 8.7 above.

#### 10.1

Why have we not considered Monte Carlo methods in this chapter?

Monte Carlo methods only update state-action value functions at the end of each episode. Semi-gradient methods are necessary for tasks such as the Mountain Car example, since the episodes take a long time to end.

##### 10.2
Give pseudocode for semi-gradient one-step Expected Sarsa for control.

Similar to n-step SARSA, but:
- For each episode:
    - Initialize and store $S_0$
    - Select and store an action $A_0$, $\epsilon$-greedy wrt to $\hat{q}(S_0, \theta)$
    - For each step $t$ in episode:
        - Take action $A_t$
        - Observe and store next reward as $R_{t+1}$ and next state as $S_{t+1}$
        - $G_t \leftarrow R_{t+1} + \gamma V_{t+1} = R_{t+1} + \gamma \sum_a \pi(a|s_t) Q_{t-1}(s_t, a)$
        - $\theta \leftarrow \theta + \alpha [G_t - \hat{q}(S_t, A_t, \theta)] \nabla \hat{q}(S_t, A_t, \theta)$
        - Select and store $A_{t+1}$, $\epsilon$-greedy wrt to $\hat{q}(S_{t+1}, \theta)$

##### *11.1

Convert the equation of n-step off-policy TD (7.7) to semi-gradient
form. Give accompanying definitions of the return for both the episodic and continuing
cases.

$\theta_{t+n} = \theta_{t+n-1} + \alpha \rho^{t+n}_t[G_t^{(n)} - \hat{q}(S_t, A_t, \theta_{t+n-1})] \nabla\hat{q}(S_t, A_t, \theta_{t+n-1})$,

where $\rho_t^{t+n}$ is the importance sampling ratio ( is the relative probability under two policies of taking the n actions), and

$G_t^{(n)} = R_{t+1} + ... + \gamma^{n-1} R_{t+n} + \gamma^n \hat{q}(S_t, A_t, \theta_{t+n-1})$,  (episodic)

$G_t^{(n)} = R_{t+1} - \bar{R}_{t} + ... + R_{t+n} - \bar{R}_{t+n+1} + \hat{q}(S_t, A_t, \theta_{t+n-1})$, (continuing)

##### **11.2

Convert the equations of n-step Q(σ) (7.9, 7.13, 7.14, and 7.15)
to semi-gradient form. Give definitions that cover both the episodic and continuing
cases.

The semigradient update step would be:

$\theta_{t+n} = \theta_{t+n-1} + \alpha (1 - \sigma_{t+1} + \sigma_{t+1} \rho_{t+1}) (1 - \sigma_{t+2} + \sigma_{t+2} \rho_{t+2}) \cdot \cdot \cdot (1 - \sigma_{t+n-1} + \sigma_{t+n-1} \rho_{t+n-1}) [G_t^{(n)} - \hat{q}(S_t, A_t, \theta_{t+n-1})] \nabla\hat{q}(S_t, A_t, \theta_{t+n-1})$,

where $\rho$ is defined in 7.15, and 7.14 is the n-step return for the episodic case. For the continuing task, $\delta$ in 7.15 would need to be:

$\delta_t = R_{t+1} - \bar{R_t} + \gamma [\sigma_{t+1} Q_t(S_{t+1}, A_{t+1}) + (1 - \sigma_{t+1}V_{t+1})] - Q_{t-1}(S_t, A_t)$

```python

```