Predicting Bike Sharing Demand

with Supervised Machine Learning, by Linear Regression (Regression)

__1) The Goal:__ Build and train a regression model on the Capital Bike Share (Washington, D.C.) Kaggle data set to predict the total count of bikes rented during each hour covered by the test set, using only the information (time- and weather-related features) available prior to the rental period.

A practical application of such a project would help you answer the following question, as an example: “Given the forecasted weather conditions, how many bicycles can we expect to be rented out (city-wide) this Saturday at 2pm?”

__2) Get the Data:__ You are provided hourly rental data spanning two years. Download the [Capital Bike Share Dataset](https://www.kaggle.com/c/bike-sharing-demand/data) from Kaggle and explore the data in pandas.

__3) Split the Data:__ Already split: "train.csv" and "test.csv". The training set is comprised of the first 19 days of each month, while the test set is the 20th to the end of the month.

__4) Exploratory Data Analysis (EDA)__

__5)-9) Feature Engineering (FE), Train Model, Optimize Hyperparameters/Cross-Validation:__ Optimize the model iteratively, select features, try different regressors (e.g. Linear Regression, Random Forest Regressor, SVR)

__10) Calculate Test Score:__ Because this is a regression problem, we are not evaluating our models using accuracy! Submissions are evaluated on the Root Mean Squared Logarithmic Error RMSLE. The RMSLE is calculated as

\begin{equation} \sqrt{\frac{1}{n} \sum_{i=1}^n (\log(p_i + 1) - \log(a_i+1))^2 } \end{equation}
Benchmarks:
- RMSLE < 0.5, is excellent! ⭐⭐⭐
- RMSLE < 0.75 is also quite solid ⭐⭐
- RMSLE > 1.0 means that there is still some FE work to do.

__11) Deploy and Monitor:__ Submit your prediction to Kaggle.

---

Exploratory Data Analysis

```python
import pandas as pd

data = pd.read_csv('./raw data/train.csv', index_col='datetime', parse_dates=True)
data.head(1)
```

      season
      holiday
      workingday
      weather
      temp
      atemp
      humidity
      windspeed
      casual
      registered
      count

      datetime

      2011-01-01
      1
      0
      0
      1
      9.84
      14.395
      81
      0.0
      3
      13
      16

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 10886 entries, 2011-01-01 00:00:00 to 2012-12-19 23:00:00
    Data columns (total 11 columns):
     #   Column      Non-Null Count  Dtype
    ---  ------      --------------  -----
     0   season      10886 non-null  int64
     1   holiday     10886 non-null  int64
     2   workingday  10886 non-null  int64
     3   weather     10886 non-null  int64
     4   temp        10886 non-null  float64
     5   atemp       10886 non-null  float64
     6   humidity    10886 non-null  int64
     7   windspeed   10886 non-null  float64
     8   casual      10886 non-null  int64
     9   registered  10886 non-null  int64
     10  count       10886 non-null  int64
    dtypes: float64(3), int64(8)
    memory usage: 1020.6 KB

The data fields are:
__datetime:__ hourly date + timestamp
__season__: 1. spring,
    2. summer,
    3. fall,
    4. winter
__holiday:__ whether the day is considered a holiday
__workingday:__ whether the day is neither a weekend nor holiday
__weather__: 1. Clear, Few clouds, Partly cloudy, Partly cloudy
     2. Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist
     3. Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds
     4. Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog
__temp:__ temperature in Celsius
__atemp:__ "feels like" temperature in Celsius
__humidity:__ relative humidity
__windspeed:__ wind speed
__casual:__ number of non-registered user rentals initiated
__registered:__ number of registered user rentals initiated
__count:__ number of total rentals

__Questions for EDA:__
- Which factors contribute most to the number of bicycles being checked out over the course of a given day?
- How much of an impact does weather have on demand?
- How does the behavior of casual users differ from subscribers?

The value in the __count__ column is the one you need to predict. Beware that combining the columns __casual__ and __registered__ results in the column __count__. Therefore you should exclude those columns from your model!

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style='white')
```

```python
data.plot(); # too many data points -> resample, see below
```

```python
data.columns = ['Season', 'Holiday', 'Workingday', 'weather', 'Temp', 'RealFeel', 'Humidity', 'Wind Speed', 'Casual User', 'Registered User', 'Number of Bikes']
data.resample('M').sum().plot();
```

```python
#data_count = pd.DataFrame(data['count'])
ax = data.resample('D').sum().rolling(365).sum().plot();
ax.set_ylim(0, None);
```

```python
data.groupby(data.index.time).mean().plot();
```

```python
pivoted = data.pivot_table('count', index=data.index.time, columns=data.index.date)
#pivoted.iloc[:5, :5]
pivoted.head()
```

      2011-01-01
      2011-01-02
      2011-01-03
      2011-01-04
      2011-01-05
      2011-01-06
      2011-01-07
      2011-01-08
      2011-01-09
      2011-01-10
      ...
      2012-12-10
      2012-12-11
      2012-12-12
      2012-12-13
      2012-12-14
      2012-12-15
      2012-12-16
      2012-12-17
      2012-12-18
      2012-12-19

      00:00:00
      16.0
      17.0
      5.0
      5.0
      6.0
      11.0
      17.0
      25.0
      25.0
      5.0
      ...
      20.0
      31.0
      34.0
      32.0
      47.0
      94.0
      110.0
      28.0
      18.0
      41.0

      01:00:00
      40.0
      17.0
      2.0
      2.0
      6.0
      4.0
      7.0
      16.0
      12.0
      1.0
      ...
      4.0
      8.0
      21.0
      23.0
      26.0
      95.0
      96.0
      15.0
      15.0
      15.0

      02:00:00
      32.0
      9.0
      NaN
      1.0
      2.0
      2.0
      1.0
      16.0
      11.0
      3.0
      ...
      5.0
      1.0
      9.0
      8.0
      9.0
      69.0
      87.0
      5.0
      7.0
      3.0

      03:00:00
      13.0
      6.0
      NaN
      NaN
      NaN
      NaN
      NaN
      7.0
      4.0
      1.0
      ...
      4.0
      3.0
      10.0
      2.0
      12.0
      23.0
      38.0
      3.0
      5.0
      5.0

      04:00:00
      1.0
      3.0
      1.0
      2.0
      2.0
      1.0
      1.0
      1.0
      1.0
      3.0
      ...
      12.0
      8.0
      4.0
      8.0
      10.0
      6.0
      11.0
      5.0
      8.0
      7.0

5 rows × 456 columns

```python

```

```python

```

```python

```

For FE pipeline:

logarithm on the target column
interactions between the weekday and the hours because the hours in the weekend mean something else

or use a model that does it auto like a random forest regressor

```python

```

```python
now = pd.to_datetime('today')
now.tz_localize('UTC').tz_convert('Europe/Berlin')
#timezone conversion can be useful as well, though we don't need this this week.
```

    Timestamp('2021-01-27 11:48:50.257305+0100', tz='Europe/Berlin')

```python

```

```python

```

```python

```

```python

```

```python
cmap = sns.diverging_palette(239, 239, s=100, as_cmap=True)
sns.heatmap(xtrain.corr(), annot=True, center=0, cmap = cmap)
```

```python

```

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
```

#### Step 1: Read the file train.csv into Python and print a few rows.

```python
train = pd.read_csv('./raw data/train.csv')
train.head()
```

      PassengerId
      Survived
      Pclass
      Name
      Sex
      Age
      SibSp
      Parch
      Ticket
      Fare
      Cabin
      Embarked

      0
      1
      0
      3
      Braund, Mr. Owen Harris
      male
      22.0
      1
      0
      A/5 21171
      7.2500
      NaN
      S

      1
      2
      1
      1
      Cumings, Mrs. John Bradley (Florence Briggs Th...
      female
      38.0
      1
      0
      PC 17599
      71.2833
      C85
      C

      2
      3
      1
      3
      Heikkinen, Miss. Laina
      female
      26.0
      0
      0
      STON/O2. 3101282
      7.9250
      NaN
      S

      3
      4
      1
      1
      Futrelle, Mrs. Jacques Heath (Lily May Peel)
      female
      35.0
      1
      0
      113803
      53.1000
      C123
      S

      4
      5
      0
      3
      Allen, Mr. William Henry
      male
      35.0
      0
      0
      373450
      8.0500
      NaN
      S

#### Step 2: Calculate the number of surviving/non-surviving passengers and display it as a bar plot.

```python
i=sum(train['Survived'])

labels_step2 = ['surviving', 'non-surviving']
x = np.arange(len(labels_step2))
y = [sum(train['Survived']), (train['Survived'].count()-sum(train['Survived']))]

plt.bar(x, y)
plt.xticks(x, labels_step2, size=16)

plt.title('%i passangers survived' %i, fontsize=18)
plt.ylabel('Number of passangers', fontsize=16)
plt.show()
```

#### Step 3: Calculate the proportion of surviving 1st class passengers with regards to the total number of 1st class passengers.

```python
print('{}% of the 1st class passengers survived.'.format(100*round(train[(train['Pclass']==1) & (train['Survived']==1)]['Survived'].count()/sum(train['Pclass']==1),2)))
```

    63.0% of the 1st class passengers survived.

#### Step 4: Create a bar plot with separate bars for male/female passengers and 1st/2nd/3rd class passengers.

```python
ax = sns.countplot(x="Pclass", hue="Sex", data=train)
```

#### Step 5: Create a histogram showing the age distribution of passengers. Compare surviving/non-surviving passengers.

```python
ax = sns.histplot(x="Age", hue="Survived", data=train)
```

#### Step 6: Calculate the average age for survived and drowned passengers separately.

```python
print('The average age of the survived passengers is {}, and the average age of the drowned passengers is {}.'.format(round(train.groupby('Survived')['Age'].mean().tolist()[0]),round(train.groupby('Survived')['Age'].mean().tolist()[1])))
```

    The average age of the survived passengers is 31, and the average age of the drowned passengers is 28.

#### Step 7: Replace missing age values by the mean age.

```python
train['Age_mean_imputed'] = train['Age'].fillna(round(np.float64(train['Age'].mean()),1))
```

#### Step 8: Create a table counting the number of surviving/dead passengers separately for 1st/2nd/3rd class and male/female.

```python
pd.crosstab(train['Pclass'], [train['Sex'], train['Survived']], rownames=['Pclass'], colnames=['Sex', 'Survived'])
```

      Sex
      female
      male

      Survived
      0
      1
      0
      1

      Pclass

      1
      3
      91
      77
      45

      2
      6
      70
      91
      17

      3
      72
      72
      300
      47

 Feature Engineering

```python
train.fillna(0.0, inplace=True) # little crutch to make the data work
```

```python

```

 Train the Model

```python

```

```python

```

 Optimize Hyperparameters/Cross-Validation

```python

```

```python

```

Train Model, Optimize Hyperparameters/Cross-Validation

- Train a Logistic Regression classification model (alternative models: Decision Trees, Random Forest)
- Create features using one-hot encoding
- Calculate the train and test accuracy
- Calculate a cross-validation score
- Train a Random Forest classification model
- Submit the predictions to Kaggle
- Give a 5-minute lightning talk by the end of the week

#### 10) Calculate Test Score
- Use the test data set aside to calculate the test score.

#### 11) Deploy and Monitor

```python

```

```python

```

 Calculate Test Score

```python

```

```python

```

```python

```

 Evaluate the Model using the Test Data

- accuracy
- confussion matrix: precission, recall, F1-score

```python

```

```python

```

 Make Predictions

```python

```


```python
from scipy.stats import gaussian_kde
from scipy.interpolate import interp1d
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rc
rc('font', **{'family': 'serif', 'serif': ['Computer Modern']})
rc('text', usetex=True)
```

# Building the joint prior

In this repository there exists code to compute the conditional priors $p(\chi_\mathrm{eff}|q)$ and $p(\chi_q|q)$ (functions `chi_effective_prior_from_isotropic_spins` and `chi_p_prior_from_isotropic_spins`, respectively) on $\chi_\mathrm{eff}$ and $\chi_p$ corresponding to uniform and isotropic component spin priors. Each of these priors have been marginalized over all other spin degrees of freedom.

In some circumstances, though, we might want the *joint* prior $p(\chi_\mathrm{eff},\chi_p|q)$ acting on the two effective spin parameters. Although we were able to derive closed-form expressions for $p(\chi_\mathrm{eff}|q)$ and $p(\chi_q|q)$, I personally lack the will-power and/or attention span to derive an analytic expression for $p(\chi_\mathrm{eff},\chi_p|q)$. Instead, let's build a function to do this numerically.

First, note that the joint prior on $\chi_\mathrm{eff}$ and $\chi_p$ is *weird*. Demonstrate this by drawing random component spins, computing the corresponding effective spins, and plotting the resulting density.

```python
def chi_p(a1,a2,cost1,cost2,q):

    sint1 = np.sqrt(1.-cost1**2)
    sint2 = np.sqrt(1.-cost2**2)

    return np.maximum(a1*sint1,((3.+4.*q)/(4.+3.*q))*q*a2*sint2)

def chi_eff(a1,a2,cost1,cost2,q):

    return (a1*cost1 + q*a2*cost2)/(1.+q)

# Choose some fixed mass ratio
q = 0.5

# Draw random component spins and compute effective parameters
ndraws = 30000
random_a1s = np.random.random(ndraws)
random_a2s = np.random.random(ndraws)
random_cost1s = 2.*np.random.random(ndraws)-1.
random_cost2s = 2.*np.random.random(ndraws)-1.

# Plot!
random_chi_effs = chi_eff(random_a1s,random_a2s,random_cost1s,random_cost2s,q)
random_chi_ps = chi_p(random_a1s,random_a2s,random_cost1s,random_cost2s,q)

fig,ax =  plt.subplots()
ax.hexbin(random_chi_effs,random_chi_ps,cmap='Blues',gridsize=30)
ax.set_xlabel('$\chi_\mathrm{eff}$',fontsize=14)
ax.set_ylabel('$\chi_p$',fontsize=14)
plt.show()
```

There are a few visible features we need to worry about.
1. First, the prior distribution comes to a sharp point at $\chi_\mathrm{eff} = \chi_p = 0$; this is related to the fact that the marginal $p(\chi_\mathrm{eff}|q)$ is quite sharply peaked about the origin (see `Demo.ipynb`)
2. The concentration about $\chi_\mathrm{eff} = 0$ also implies that vanishingly few of our prior draws occur in the distant wings of the joint prior, at very negative or very positive  $\chi_\mathrm{eff}$.
3. In the vertical direction, we can see the same sharp drop and extended plateau as seen in the marginal $\chi_p$ prior in `Demo.ipynb`

Naively, we could just draw a bunch of prior samples and form a KDE over this space. The first two features listed above, though, make this extremely difficult. The extreme narrowness of $p(\chi_\mathrm{eff},\chi_p|q)$ near the origin means we must use an extremely small KDE bandwidth to accurately capture this behavior, but such a small bandwidth will accentuate sampling fluctuations elsewhere. Meanwhile, the fact that very few samples occur at very positive or very negative $\chi_\mathrm{eff}$ means that we will need to perform a vast number of draws (like, many millions) if we wish to accurately estimate the prior on posterior samples falling in these areas.

Recall that this prior remains *conditional* on $q$, and so we can't just build a single KDE (in which case we might tolerate having to perform a vast number of draws and slow KDE evaluation), but will need to build a new estimator every time we consider a different mass ratio.

Instead, let's leverage our knowledge of the marginal prior $p(\chi_\mathrm{eff}|q)$ and factor the joint prior as

\begin{equation}
p(\chi_\mathrm{eff},\chi_p|q) = p(\chi_p|\chi_\mathrm{eff},q) p(\chi_\mathrm{eff},q),
\end{equation}

so that we only have to worry about numerically constructing the one-dimensional distribution $p(\chi_p|\chi_\mathrm{eff},q)$.

Given $\chi_\mathrm{eff}$ and $q$, we will repeatedly draw $\{a_1,a_2,\cos t_1,\cos t_2\}$ consistent with $\chi_\mathrm{eff}$, and then construct the resulting distribution over $\chi_p$. In particular, we will regard

\begin{equation}
\cos t_1 = \frac{(1+q)\chi_\mathrm{eff} - q a_2 \cos t_2}{a_1}
\end{equation}

as a function of the $\chi_\mathrm{eff}$ and the three other component spin parameters. In making this choice, though, we are *really* drawing from a slice through

\begin{equation}
\frac{dP}{d a_1 da_2 d\chi_\mathrm{eff} d\cos t_2 } = \frac{dP}{d a_1 da_2 d \cos t_1 d\cos t_2} \frac{\partial \cos t_1}{\partial \chi_\mathrm{eff}}.
\end{equation}

Thus, in order to have properly sampled from the underlying uniform and isotropic distribution $dP/d a_1 da_2 d \cos t_1 d\cos t_2$, we will need to remember to divide out the Jacobian weights $\partial \cos t_1/\partial \chi_\mathrm{eff} = a_1/(1+q)$.

Let's try this in the following cell:

```python
# Fix some value for chi_eff and q
# Feel free to change these!
aMax = 1.
Xeff = 0.2
q = 0.5

# Draw random spin magnitudes.
# Note that, given a fixed chi_eff, a1 can be no larger than (1+q)*chi_eff,
# and a2 can be no larger than (1+q)*chi_eff/q
ndraws = 100000
a1 = np.random.random(ndraws)*aMax
a2 = np.random.random(ndraws)*aMax

# Draw random tilts for spin 2
cost2 = 2.*np.random.random(ndraws)-1.

# Finally, given our conditional value for chi_eff, we can solve for cost1
# Note, though, that we still must require that the implied value of cost1 be *physical*
cost1 = (Xeff*(1.+q) - q*a2*cost2)/a1

# While any cost1 values remain unphysical, redraw a1, a2, and cost2, and recompute
# Repeat as necessary
while np.any(cost1<-1) or np.any(cost1>1):
    to_replace = np.where((cost1<-1) | (cost1>1))[0]
    a1[to_replace] = np.random.random(to_replace.size)*aMax
    a2[to_replace] = np.random.random(to_replace.size)*aMax
    cost2[to_replace] = 2.*np.random.random(to_replace.size)-1.
    cost1 = (Xeff*(1.+q) - q*a2*cost2)/a1

Xp_draws = chi_p(a1,a2,cost1,cost2,q)
jacobian_weights = (1.+q)/a1
```

For comparison, let's also take a brute-force approach, drawing truly random component spins and saving those whose $\chi_\mathrm{eff}$ are "close to" the conditioned $\chi_\mathrm{eff}$ value specified above. This can take a while, depending on the values of $q$ and $\chi_\mathrm{eff}$ we've chosen...

```python
test_a1s = np.array([])
test_a2s = np.array([])
test_cost1s = np.array([])
test_cost2s = np.array([])

while test_a1s.size<30000:

    test_a1 = np.random.random()*aMax
    test_a2 = np.random.random()*aMax
    test_cost1 = 2.*np.random.random()-1.
    test_cost2 = 2.*np.random.random()-1.
    test_xeff = chi_eff(test_a1,test_a2,test_cost1,test_cost2,q)
    if np.abs(test_xeff-Xeff)<0.02:
        test_a1s = np.append(test_a1s,test_a1)
        test_a2s = np.append(test_a2s,test_a2)
        test_cost1s = np.append(test_cost1s,test_cost1)
        test_cost2s = np.append(test_cost2s,test_cost2)
```

Let's plot both approaches below. For completeness, also plot what happens if we *forget* the Jacobian factors, which gives a clear mismatch relative to the brute force draws.

```python
fig,ax = plt.subplots()
ax.hist(Xp_draws,density=True,bins=30,weights=jacobian_weights,label='Our approach')
ax.hist(Xp_draws,density=True,bins=30,histtype='step',ls='--',color='black',label='Our approach (w/out Jacobians)')
ax.hist(chi_p(test_a1s,test_a2s,test_cost1s,test_cost2s,q),density=True,histtype='step',bins=30,color='black',
       label='Brute force')
plt.legend()
ax.set_xlabel(r'$\chi_p$',fontsize=14)
ax.set_ylabel(r'$p(\chi_p|\chi_\mathrm{eff},q)$',fontsize=14)
plt.show()
```

We could stop here, KDE our (appropriately weighted) draws, and evaluate the KDE at a $\chi_p$ of interest. We want to be a bit more careful with the end points, though. If we KDE directly, some of our probability will leak out past our boundaries at $\chi_p = 0$ and $\chi_p = 1$.

```python
demo_kde = gaussian_kde(Xp_draws,weights=jacobian_weights)

fig,ax = plt.subplots()
ax.hist(Xp_draws,density=True,bins=30,weights=jacobian_weights)
ax.plot(np.linspace(-0.1,1.1,50),demo_kde(np.linspace(-0.1,1.1,50)),color='black',label='KDE')
plt.legend()
ax.set_xlabel(r'$\chi_p$',fontsize=14)
ax.set_ylabel(r'$p(\chi_p|\chi_\mathrm{eff},q)$',fontsize=14)
plt.show()
```

Even if we truncate to the interval $0 \leq \chi_p \leq 1$, we will still generically  end up in a situation where our prior does not go to zero at $\chi_p = 0$ and $\chi_p = 1$:

```python
# Integrate across (0,1) to obtain appropriate normalization
truncated_grid = np.linspace(0,1,100)
norm_constant = np.trapz(demo_kde(truncated_grid),truncated_grid)

fig,ax = plt.subplots()
ax.hist(Xp_draws,density=True,bins=30,weights=jacobian_weights)
ax.plot(truncated_grid,demo_kde(truncated_grid)/norm_constant,color='black',label='KDE')
plt.legend()
ax.set_xlabel(r'$\chi_p$',fontsize=14)
ax.set_ylabel(r'$p(\chi_p|\chi_\mathrm{eff},q)$',fontsize=14)
plt.show()
```

Instead, we will take a two step approach. First, use a KDE to evaluate $p(\chi_p|\chi_\mathrm{eff},q)$ across a grid of points well inside the boundaries at $0$ and $\mathrm{Max}(\chi_p)$. Then manually specificy the endpoints, with $p(\chi_p|\chi_\mathrm{eff},q) = 0$.

Note that the maximum value of $\chi_p$ given some $\chi_\mathrm{eff}$ is

\begin{equation}
\begin{aligned}
\mathrm{Max}(\chi_p) &= \mathrm{Max}\left[\mathrm{max}\left( s_{1p}, \frac{3+4q}{4+3q} q s_{2p}\right)\right] \\
&= \mathrm{Max}(s_{1p}),
\end{aligned}
\end{equation}

defining $s_p = a \sin t$ as the in-plane spin component. If we define $s_z = a \cos t$, then

\begin{equation}
\begin{aligned}
\mathrm{Max}(\chi_p)
&= \mathrm{Max}\sqrt{a^2_\mathrm{max}-s_{1z}^2} \\
&= \sqrt{a^2_\mathrm{max}-\mathrm{Min}(s_{1z}^2)} \\
&= \sqrt{a^2_\mathrm{max}-\mathrm{Min}\left[\left((1+q)\chi_\mathrm{eff} - q s_{2z}\right)^2\right]}
\end{aligned}
\end{equation}

where the minimum is taken over possible $s_{2z}$. If $(1+q)\chi_\mathrm{eff} \leq a_\mathrm{max} q$, then there is always some $s_{2z}$ available such that the bracketed term is zero, giving $\mathrm{Max}(\chi_p) = a_\mathrm{max}$. If, on the other hand, $(1+q)\chi_\mathrm{eff} > a_\mathrm{max} q$ then the bracketed term will necessarily always be non-zero, with its smallest value occurring at $s_{2z} = a_\mathrm{max}$. In this case, $\mathrm{Max}(\chi_p) = \sqrt{a^2_\mathrm{max}-\left((1+q)\chi_\mathrm{eff} - a_\mathrm{max} q\right)^2}$.

```python
# Compute maximum chi_p
if (1.+q)*np.abs(Xeff)/q<aMax:
    max_Xp = aMax
else:
    max_Xp = np.sqrt(aMax**2 - ((1.+q)*np.abs(Xeff)-q)**2.)

# Set up a grid slightly inside (0,max chi_p) and evaluate KDE
reference_grid = np.linspace(0.05*max_Xp,0.95*max_Xp,30)
reference_vals = demo_kde(reference_grid)

# Manually prepend/append zeros at the boundaries
reference_grid = np.concatenate([[0],reference_grid,[max_Xp]])
reference_vals = np.concatenate([[0],reference_vals,[0]])
norm_constant = np.trapz(reference_vals,reference_grid)

# Interpolate!
prior_vals = [np.interp(Xp,reference_grid,reference_vals) for Xp in truncated_grid]

fig,ax = plt.subplots()
ax.hist(Xp_draws,density=True,bins=30,weights=jacobian_weights)
ax.plot(truncated_grid,prior_vals/norm_constant,color='black',label='Our interpolant')
plt.legend()
ax.set_xlabel(r'$\chi_p$',fontsize=14)
ax.set_ylabel(r'$p(\chi_p|\chi_\mathrm{eff},q)$',fontsize=14)
plt.show()
```

This procedure is implemented in the function `chi_p_prior_given_chi_eff_q` appearing in `priors.py`.  For completeness, let's compare the output of this function against the result we got in this notebook.

```python
from priors import *
ndraws=100000
priors_from_function = [chi_p_prior_given_chi_eff_q(q,aMax,Xeff,xp,ndraws=ndraws,bw_method=1.*ndraws**(-1./5.)) for xp in reference_grid]
```

```python
fig,ax = plt.subplots()
ax.plot(reference_grid,priors_from_function,label='From priors.py')
ax.plot(reference_grid,reference_vals)
ax.set_xlabel(r'$\chi_p$',fontsize=14)
ax.set_ylabel(r'$p(\chi_p|\chi_\mathrm{eff},q)$',fontsize=14)
plt.legend()
plt.show()
```

```python

```


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

from chmp.bayes import Model, sample_latent, sample_prior
from chmp.ds import mpl_set, get_color_cycle, colormap, define
from chmp.experiment import Loop
from chmp.ml import get_variables, Sample
```

    /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
      return f(*args, **kwds)

# Implicit Variational Inference using GAN-style training

Based on:

- D Tran, R Ranganath, DM Blei, ["Hierarchical Implicit Models and Likelihood-Free Variational Inference"](https://arxiv.org/abs/1702.08896), 2017.
- F Huszár, ["Variational Inference using Implicit Models"](http://www.inference.vc/variational-inference-using-implicit-models/).

$$
\begin{align}
    \mathcal{L}_q
        &= \mathbb{E}_{q(z)} \log \frac{p(x, z)}{q(z)} \\
        &= \mathbb{E}_{q(z)} \log p(x|z) + \mathbb{E}_{q(z)} \log \frac{p(z)}{q(z)} \\
        &\approx \mathbb{E}_{q(z)} \log p(x|z) + \mathbb{E}_{q(z)} r(z)
\end{align}
$$

Where $r$ *maximizes*:

$$
    \mathcal{L}_r =
        \mathbb{E}_{p(z)} \big[ \log \sigma(r(z)) \big] +
        \mathbb{E}_{q(z)} \big[ \log\big( 1 - \sigma(r(z)) \big) \big]
$$

At optimiality $r^\star(z) = \log p(z) - \log q(z)$.

## Example: Gaussian

Target density:

$$
    p(x) = \mathcal{N}(x | \mu_p = 5, \sigma_p = 2)
$$

Generators:

$$
    x = \mu_q + \sigma_p \epsilon \sim q(x) \\
    \epsilon \sim \mathcal{N}(0, 1)
$$

Log-ratio approximation:
$$
    r(x) = b_0 + b_1 x + b_2 x^2
$$

Note, that $r(x)$ is able to fit the log ratio exactly.

```python
tf.reset_default_graph()

n_samples = 10

p_ = tf.random_normal((n_samples,), mean=5.0, stddev=2.0)

def sample_q():
    with tf.variable_scope('q', reuse=tf.AUTO_REUSE):
        mu_ = tf.get_variable('mu', initializer=0.0, dtype=tf.float32)
        sigma_ = tf.nn.softplus(tf.get_variable('sigma', initializer=1.0, dtype=tf.float32))

        return mu_ + sigma_ * tf.random_normal((n_samples,))

def apply_r(x):
    with tf.variable_scope('r', reuse=tf.AUTO_REUSE):
        b0_ = tf.get_variable('b', initializer=0.0, dtype=tf.float32)
        b1_ = tf.get_variable('b1', initializer=1.0, dtype=tf.float32)
        b2_ = tf.get_variable('b2', initializer=0.5, dtype=tf.float32)

        return b0_ + b1_ * x + b2_ * (x ** 2.0)

q_ = sample_q()
ratio_loss_ = -tf.reduce_mean(
    tf.log(tf.sigmoid(apply_r(p_))) +
    tf.log(1 - tf.sigmoid(apply_r(sample_q())))
)

kl_loss_ = -tf.reduce_mean(apply_r(sample_q()))

train_q_ = tf.train.AdamOptimizer(0.01).minimize(kl_loss_, var_list=get_variables('q/'))
train_r_ = tf.train.AdamOptimizer(0.05).minimize(ratio_loss_, var_list=get_variables('r/'))
```

```python
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for _ in range(10_000):
        sess.run(train_r_)
        sess.run(train_q_)

    q = [sess.run(q_) for _ in range(100)]
    p = [sess.run(p_) for _ in range(100)]
```

```python
c0, c1 = get_color_cycle(2)

plt.hist(np.reshape(q, -1), bins=21, normed=True, alpha=0.3, label='x ~ q', color=c1)
plt.hist(np.reshape(p, -1), bins=21, normed=True, alpha=0.3, label='x ~ p', color=c0)

t = np.linspace(-3, 13, 100)
plt.plot(
    t, np.exp(-0.5 * (t - 5.) ** 2.0 / (4.)) / np.sqrt(2. * np.pi * 4.),
    label='p(x)', color=c0,
)

mpl_set(xlabel='x', ylabel='Frequency', legend=True)
pass
```

## Mixture of Gaussians

```python
tf.reset_default_graph()

n_samples = 10

u_ = tf.cast(tf.random_uniform([n_samples]) > 0.4, tf.float32)
p_ = (
    u_ * tf.random_normal([n_samples], mean=-2, stddev=1.0) +
    (1 - u_) * tf.random_normal([n_samples], mean=4.0, stddev=0.5)
)

def sample_q():
    with tf.variable_scope('q', reuse=tf.AUTO_REUSE):
        mu_ = tf.get_variable('mu', initializer=np.random.uniform(-5., 5.0, size=(4,)).astype(np.float32), dtype=tf.float32)
        sigma_ = tf.nn.softplus(tf.get_variable('sigma', initializer=5.0 * np.ones(4, dtype=np.float32), dtype=tf.float32))

        theta_ = tf.get_variable('theta', initializer=0.1 * np.ones(4, dtype=np.float32), dtype=tf.float32)

        u_ = tf.contrib.distributions.RelaxedOneHotCategorical(0.1, logits=theta_).sample([n_samples])

        mu_ = tf.expand_dims(mu_, axis=0)
        sigma_ = tf.expand_dims(sigma_, axis=0)

        return tf.reduce_sum(
            u_ * (mu_ + sigma_ * tf.random_normal((n_samples, 4))),
            axis=-1,
        )

b_init_ = np.random.uniform(-1e-3, +1e-3, size=(3, 10)).astype(np.float32)
mu_init_ = np.random.uniform(-10., 10., size=(10,)).astype(np.float32)
sigma_init_ = (0.01 + np.random.normal(size=(10,)) ** 2.0).astype(np.float32)

def apply_r(x):
    with tf.variable_scope('r', reuse=tf.AUTO_REUSE):
        b0_ = tf.get_variable('b0', initializer=[1., 0.01, 0.01], dtype=tf.float32)
        b_ = tf.get_variable('b', initializer=b_init_, dtype=tf.float32)
        mu_ = tf.get_variable('mu', initializer=mu_init_, dtype=tf.float32)
        sigma_ = tf.get_variable('sigma', initializer=sigma_init_, dtype=tf.float32)

        x1 = tf.expand_dims(x, axis=-1)
        x2 = tf.expand_dims(x1, axis=-1)

        b_ = tf.expand_dims(b_, axis=0)
        n = tf.expand_dims(tf.expand_dims(tf.range(3, dtype=tf.float32), axis=0), axis=-1)

        mu_ = tf.expand_dims(mu_, axis=0)
        sigma_ = tf.expand_dims(sigma_, axis=0)

        base = b0_[0] + b0_[1] * x + b0_[2] * x ** 2.
        delta = tf.reduce_mean(
            tf.reduce_sum(b_ * (x2 - mu_) ** n, axis=1) * (tf.exp(-0.5 * (x1 - mu_) ** 2.0 / sigma_ ** 2.0) / sigma_),
            axis=1,
        )

        return base + delta

q_ = sample_q()
ratio_loss_ = -tf.reduce_mean(
    tf.log(tf.sigmoid(apply_r(p_))) +
    tf.log(1 - tf.sigmoid(apply_r(sample_q())))
)

kl_loss_ = -tf.reduce_mean(apply_r(sample_q()))

train_q_ = tf.train.AdamOptimizer(0.01).minimize(kl_loss_, var_list=get_variables('q/'))
train_r_ = tf.train.AdamOptimizer(0.1).minimize(ratio_loss_, var_list=get_variables('r/'))

t_ = tf.linspace(-10., 10., 100)
r_ = apply_r(t_)
```

```python
ratios = []

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for loop, iteration in Loop.over(range(10_000)): #10_000):
        if iteration % 200 == 0:
            ratios.append(sess.run([t_, r_]))
            sess.run(tf.verify_tensor_all_finite(q_, 'nans detected'))

        sess.run(train_r_)
        sess.run(train_q_)
        print(f'{loop}'.ljust(120), end='\r')

    q = [sess.run(q_) for _ in range(100)]
    p = [sess.run(p_) for _ in range(100)]

ratios = np.asarray(ratios)
```

    100%|██████████| 10000/10000 [00:28<00:00, 355.06it/s]

```python
c0, c1 = get_color_cycle(2)

plt.figure(figsize=(16, 4))
plt.subplot(1, 3, 1)
plt.hist(np.reshape(q, -1), range=(-10, +10), bins=31, normed=True, alpha=0.3, label='x ~ q', color=c1)
plt.hist(np.reshape(p, -1), range=(-10, +10), bins=31, normed=True, alpha=0.3, label='x ~ p', color=c0)

t = np.linspace(-10, 10, 100)
plt.plot(
    t,
    0.6 * np.exp(-0.5 * (t + 2.) ** 2.0) / np.sqrt(2. * np.pi) +
    0.4 * np.exp(-0.5 * ((t - 4) / 0.5) ** 2.0) / np.sqrt(2.0 * np.pi * 0.5 ** 2.0),
    label='p(x)', color=c0,
)

mpl_set(xlabel='x', ylabel='Frequency', legend=True)

plt.subplot(1, 3, 2)
plt.plot(np.sort(np.reshape(p, -1)), np.linspace(0, 1, np.size(p)), label='p', color=c0)
plt.plot(np.sort(np.reshape(q, -1)), np.linspace(0, 1, np.size(q)), label='q', color=c1)
plt.axvline(x=-2, color='0.5')
plt.axvline(x=4, color='0.5')
mpl_set(xlabel='x', ylabel='Emperical CDF', legend=True)

plt.subplot(1, 3, 3)
for color, ratio in zip(colormap(range(len(ratios))), ratios):
    plt.plot(ratio[0], ratio[1], color=color, alpha=0.2)

plt.plot(ratios[-1, 0], ratios[-1, 1], color='0.2', label='final')
plt.axhline(y=0, c='0.5')
mpl_set(xlabel='x', ylabel='Estimated log ratio r(x)', legend=True)
pass
```

## Linear regression

```python
n_features = 10
n_samples = 100

np.random.seed(1234)
w = np.random.normal(scale=4, size=n_features)
x = np.random.uniform(-10, +10, size=(n_samples, n_features))

y = x @ np.random.normal(w, scale=0.2)
```

```python
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.bar(np.arange(len(w)), w)

plt.subplot(1, 2, 2)
plt.hist(y - x @ w)
pass
```

```python
def build_implicit_losses(model, discriminator):
    model_scope = model.build(ensure_loss=False, latent_strategy=sample_prior)
    inference_scope = model.build(ensure_loss=False, latent_strategy=sample_latent)

    if model_scope['loss'] is not None or inference_scope['loss'] is not None:
        raise RuntimeError('cannot handle custom losses ...')

    latent = {*model_scope['latent'], *inference_scope['latent']}
    observed = {*model_scope['observed'], *inference_scope['observed']} & {*model_scope['p'], *inference_scope['p']}

    ratio_loss = 0
    for k in latent:
        ratio_loss += tf.reduce_mean(
            tf.log(tf.sigmoid(discriminator[k](model_scope['latent'][k]))) +
            tf.log(1 - tf.sigmoid(discriminator[k](inference_scope['latent'][k])))
        )

    kl_loss = 0
    for k in observed:
        px = inference_scope['p'][k]
        x = inference_scope['observed'][k]
        kl_loss += tf.reduce_mean(px.log_prob(x))

    for k in latent:
        kl_loss += tf.reduce_mean(discriminator[k](inference_scope['latent'][k]))

    return -kl_loss, -ratio_loss
```

```python
tf.reset_default_graph()

def discriminator(weights):
    with tf.variable_scope('discriminator', reuse=tf.AUTO_REUSE):
        x = tf.layers.dense(weights, 10, activation=tf.nn.relu, name='layer1')
        x = tf.layers.dense(x, 10, activation=tf.nn.relu, name='layer2')
        x = tf.layers.dense(x, 1, name='layer3')
    return x

with Model() as model:
    @model.observe
    def _(s):
        s.x = tf.placeholder(name='x', dtype=tf.float32, shape=(n_samples, n_features))
        s.y = tf.placeholder(name='y', dtype=tf.float32, shape=n_samples)

    @model.define
    def _(s):
        s.p.z = tf.distributions.Normal(loc=tf.zeros([n_samples, n_features]), scale=2.0)
        s.p.y = tf.distributions.Normal(tf.reduce_sum(s.x * s.z, axis=1), 1.0)

    @model.inference
    def _(s):
        pz = tf.distributions.Normal(
            tf.get_variable('mean', n_features, dtype=tf.float32),
            tf.nn.softplus(tf.get_variable('scale', n_features, dtype=tf.float32)),
        )
        s.q.z = Sample(pz.sample(n_samples))

kl_loss_, ratio_loss_ = build_implicit_losses(model, dict(z=discriminator))
x_, y_, z_ = model.get('x', 'y', 'z')

# NOTE: supplying var_list is important
train_r_ = tf.train.AdamOptimizer(0.01).minimize(ratio_loss_, var_list=get_variables('discriminator'))
train_q_ = tf.train.AdamOptimizer(0.001).minimize(kl_loss_, var_list=get_variables('inference'))
```

```python
losses = []
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for loop, iteration in Loop.over(range(20_000)):
        sess.run([train_r_])
        sess.run(train_q_, {y_: y, x_: x})

        if iteration % 50 == 0:
            _, kl_loss, ratio_loss = sess.run([
                tf.verify_tensor_all_finite(z_, 'nans detected'), kl_loss_, ratio_loss_
            ], {y_: y, x_: x})

            losses.append((kl_loss, ratio_loss))

        print(f'{loop}'.ljust(120), end='\r')
    z = sess.run(z_)

losses = np.asarray(losses)
```

    100%|██████████| 20000/20000 [00:56<00:00, 350.88it/s]

```python
@define
def _():
    c0, c1 = get_color_cycle(2)

    plt.figure(figsize=(16, 4))
    plt.subplot(1, 3, 1)
    plt.bar(np.arange(n_features) - 0.2, np.mean(z, axis=0), width=0.4, label='mean inferred')
    plt.bar(np.arange(n_features) + 0.2, w, width=0.4, label='true')
    plt.axhline(y=0, color='0.5', alpha=0.5, lw=0.75)
    mpl_set(legend=True, xlabel='Feature', ylabel='Weight')

    plt.subplot(1, 3, 2)
    plt.violinplot(z - w[None, :])
    plt.axhline(y=0, color='0.5', alpha=0.5, lw=0.75)
    mpl_set(xlabel='Feature', ylabel='Residual weight (inferred - true)')

    plt.subplot(1, 3, 3)
    plt.plot(losses[:, 0], color=c0)
    plt.yscale('log')
    plt.twinx()
    plt.plot(losses[:, 1], color=c1)
    plt.yscale('log')
```

# Implicit Variational Inference using Denoisers

Based on F Huszár, ["Variational Inference using Implicit Models IV"](http://www.inference.vc/variational-inference-using-implicit-models-part-iv-denoisers-instead-of-discriminators/) (2017).

Train a denoiser $F(z) = z - r(z)$ that takes a corrupted sample from the posteriro $z + \epsilon$ to the original value $z$. It is trained via the loss

$$
\begin{align}
    \mathcal{L}_r
        &= \int\mathrm{d}z\; \mathrm{d}\epsilon; q(z) f_\sigma(\epsilon)\; |F(z + \epsilon) - z|^2 \\
        &= \int\mathrm{d}z\; \mathrm{d}\epsilon; q(z) f_\sigma(\epsilon)\; |r(z + \epsilon) - \epsilon|^2
\end{align}
$$

With the zero-centered Gaussian $f_\sigma(\epsilon)$. The optimal $r$ will behave asymptotically as:

$$
\begin{align}
    F(z) &\approx z + \sigma^2 \frac{\partial}{\partial z} \log q(z) \\
    r(z) &\approx -\sigma^2 \frac{\partial}{\partial z} \log q(z)
\end{align}
$$

Reparametrize $q(z)$ as $z = t(\eta) \sim q(z), \eta \sim p(\eta)$.

Using these results, we can rewrite the gradient VI objective as:

$$
\begin{align}
    \frac{\partial}{\partial \phi} \mathcal{L}_q
        &= \frac{\partial}{\partial \phi} \mathbb{E}_{q(z)} \log \frac{p(x, z)}{q(z)} \\
        &= \mathbb{E}_{p(\eta)} \left[
            \frac{\partial}{\partial \phi} \log p(x, t(\eta)) +
           \frac{1}{\sigma^2} r(t(\eta)) \cdot \frac{\partial}{\partial \phi} t(\eta)
        \right]
\end{align}
$$

```python
n_features = 10
n_samples = 100

np.random.seed(1234)
w = np.random.normal(scale=4, size=n_features)
x = np.random.uniform(-10, +10, size=(n_samples, n_features))

y = x @ np.random.normal(w, scale=0.2)
```

```python
def build_denoiser_losses(model, denoiser, sigma=0.1):
    scope = model.build(ensure_loss=False, latent_strategy=sample_latent)

    if scope['loss'] is not None:
        raise RuntimeError('cannot handle custom losses ...')

    denoiser_loss = 0
    for k in scope['latent']:
        z = scope['latent'][k]
        eps = tf.random_normal(tf.shape(z), stddev=sigma)
        denoised = denoiser[k](z + eps) - eps

        denoiser_loss += tf.reduce_mean(denoised * denoised)

    kl_loss = 0
    for k in {*scope['observed']} & {*scope['p']}:
        px = scope['p'][k]
        x = scope['observed'][k]
        kl_loss += tf.reduce_mean(px.log_prob(x))

    for k in scope['latent']:
        pz = scope['p'][k]
        z = scope['latent'][k]

        kl_loss += tf.reduce_mean(pz.log_prob(z))
        kl_loss += tf.reduce_mean(tf.stop_gradient(denoiser[k](z)) * z) / (sigma ** 2.0)

    return -kl_loss, denoiser_loss
```

```python
tf.reset_default_graph()

def denoiser(weights):
    with tf.variable_scope('denoiser', reuse=tf.AUTO_REUSE):
        x = tf.layers.dense(weights, 20, activation=tf.nn.relu, name='layer1')
        x = tf.layers.dense(weights, 20, activation=tf.nn.relu, name='layer2')
        x = tf.layers.dense(x, n_features, name='layer3')
    return x

with Model() as model:
    @model.observe
    def _(s):
        s.x = tf.placeholder(name='x', dtype=tf.float32, shape=(n_samples, n_features))
        s.y = tf.placeholder(name='y', dtype=tf.float32, shape=n_samples)

    @model.define
    def _(s):
        s.p.z = tf.distributions.Normal(loc=tf.zeros([n_samples, n_features]), scale=2.0)
        s.p.y = tf.distributions.Normal(tf.reduce_sum(s.x * s.z, axis=1), 1.0)

    @model.inference
    def _(s):
        pz = tf.distributions.Normal(
            tf.get_variable('mean', n_features, dtype=tf.float32),
            tf.nn.softplus(tf.get_variable('scale', n_features, dtype=tf.float32)),
        )
        s.q.z = Sample(pz.sample(n_samples))

kl_loss_, denoiser_loss_ = build_denoiser_losses(model, dict(z=denoiser))
x_, y_, z_ = model.get('x', 'y', 'z')

# NOTE: supplying var_list is important
train_r_ = tf.train.AdamOptimizer(0.01).minimize(denoiser_loss_, var_list=get_variables('denoiser'))
train_q_ = tf.train.AdamOptimizer(0.001).minimize(kl_loss_, var_list=get_variables('inference'))
```

```python
losses = []
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for loop, iteration in Loop.over(range(20_000)):
        sess.run([train_r_])
        sess.run(train_q_, {y_: y, x_: x})

        if iteration % 50 == 0:
            _, kl_loss, denoiser_loss = sess.run([
                tf.verify_tensor_all_finite(z_, 'nans detected'), kl_loss_, denoiser_loss_
            ], {y_: y, x_: x})

            losses.append((kl_loss, denoiser_loss))

        print(f'{loop}'.ljust(120), end='\r')

    z = sess.run(z_)

losses = np.asarray(losses)
```

    100%|██████████| 20000/20000 [00:58<00:00, 343.15it/s]

```python
@define
def _():
    c0, c1 = get_color_cycle(2)

    plt.figure(figsize=(16, 4))
    plt.subplot(1, 3, 1)
    plt.bar(np.arange(n_features) - 0.2, np.mean(z, axis=0), width=0.4, label='mean inferred')
    plt.bar(np.arange(n_features) + 0.2, w, width=0.4, label='true')
    plt.axhline(y=0, color='0.5', alpha=0.5, lw=0.75)
    mpl_set(legend=True, xlabel='Feature', ylabel='Weight')

    plt.subplot(1, 3, 2)
    plt.violinplot(z - w[None, :])
    plt.axhline(y=0, color='0.5', alpha=0.5, lw=0.75)
    mpl_set(xlabel='Feature', ylabel='Residual weight (inferred - true)')

    plt.subplot(1, 3, 3)
    plt.plot(losses[:, 0], color=c0)
    plt.yscale('log')
    plt.twinx()
    plt.plot(losses[:, 1], color=c1)
    plt.yscale('log')
```

# GAN-like inference for entropy only

Approximate the entropy term as a ratio estimation problem:

$$
\begin{eqnarray}
    \mathcal{L}_q
        &=& \mathbb{E}_{q(z)} \left[ \log \frac{p(x|z) p(z)}{q(z)} \right] \\
        &=& \mathbb{E}_{q(z)} \left[ \log p(x|z) + \log \frac{p(z)}{t(z)} + \log \frac{t(z)}{q(z)} \right] \\
        &\approx& \mathbb{E}_{q(z)} \left[ \log p(x|z) + \log \frac{p(z)}{t(z)} + \log r(z) \right]
\end{eqnarray}
$$

Determine $r$ from the contrastive criterion:

$$
\begin{eqnarray}
    \mathcal{L}_r
        &=& \mathbb{E}_{t(z)} \big[ \log \sigma(r(z)) \big] +
            \mathbb{E}_{q(z)} \big[ \log\big( 1 - \sigma(r(z)) \big) \big]
\end{eqnarray}
$$

Maximum likelihood solution for $t$, e.g., a Gaussian approximation to the posterior,

$$
\begin{eqnarray}
    \mathcal{L}_t
        &=& \mathbb{E}_{q(z)} \log t(z)
\end{eqnarray}
$$

```python
# https://github.com/LMescheder/AdversarialVariationalBayes/blob/master/notebooks/Stan-example.ipynb
```

```python
# copied from https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started
# NOTE the distribution is bimodal, since tua can be pos. or neg..
# Accordingly eta will change sign too
model_code = """
data {
    int<lower=0> J;
    real y[J];
    real<lower=0> sigma[J];
}
parameters {
    real mu;
    real tau;
    real eta[J];
}
transformed parameters {
    real theta[J];
    for (j in 1:J) {
        theta[j] = mu + tau * eta[j];
    }
}
model {
    y ~ normal(theta, sigma);
    eta ~ normal(0, 1);
    mu ~ normal(0, 1);
    tau ~ normal(0, 1);
}
"""
data = dict(
    J=8,
    y=[2.8,  0.8, -0.3,  0.7, -0.1,  0.1, 1.8, 1.2],
    sigma=[0.8, 0.5, 0.8, 0.6,  0.5, .6, 0.5, 0.4],
)
```

```python
import pystan
```

```python
m = pystan.StanModel(model_code=model_code)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_30357cbae0edf6498c1820f11b608917 NOW.

```python
# use settings from Adversarial Variational Bayes:
# Unifying Variational Autoencoders and Generative Adversarial Networks
trace = m.sampling(data=data, iter=500_000, thin=50)
```

```python
plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.hist2d(trace['mu'], trace['tau'], bins=(31, 31), range=((-1, +3), (-3, +3)))
mpl_set(xlabel='mu', ylabel='tau')

plt.subplot(1, 2, 2)
plt.hist2d(trace['tau'], trace['eta'][:, 0], bins=(31, 31), range=((-3, +3), (-3, +3)))
mpl_set(xlabel='tau', ylabel='eta_0')
pass
```

```python
def gaussian_reference(items):
    with tf.variable_scope('reference', reuse=tf.AUTO_REUSE):
        return {
            k: tf.distributions.Normal(
                loc=tf.get_variable(name=f'{k}_loc', initializer=tf.zeros_like(v)),
                scale=tf.nn.softplus(tf.get_variable(name=f'{k}_scale', initializer=tf.ones_like(v))),
            )
            for k, v in items.items()
        }

def build_approx_entropy_losses(model, discriminator, reference=gaussian_reference):
    scope = model.build(ensure_loss=False, latent_strategy=sample_latent)
    # TODO: make this configurable
    reference_dists = reference(scope['latent'])
    reference_samples = {k: p.sample() for k, p in reference_dists.items()}

    if scope['loss'] is not None:
        raise RuntimeError('cannot handle custom losses ...')

    latent = {*scope['latent']}
    observed = {*scope['observed']} & {*scope['p']}

    ratio_loss = 0
    for k in latent:
        ratio_loss += tf.reduce_mean(
            tf.log(tf.sigmoid(discriminator(reference_samples))) +
            tf.log(1 - tf.sigmoid(discriminator(scope['latent'])))
        )

    kl_loss = 0
    for k in observed:
        px = scope['p'][k]
        x = scope['observed'][k]
        kl_loss += tf.reduce_mean(px.log_prob(x))

    for k in latent:
        x = scope['latent'][k]
        px = scope['p'][k]
        tx = reference_dists[k]

        kl_loss += tf.reduce_mean(px.log_prob(x)) - tf.reduce_mean(tx.log_prob(x))

    # add the entropy term approximation relative to the reference
    kl_loss += tf.reduce_mean(discriminator(scope['latent']))

    reference_loss = 0
    for k in reference_dists:
        reference_loss += tf.reduce_mean(reference_dists[k].log_prob(scope['latent'][k]))

    return -kl_loss, -ratio_loss, -reference_loss
```

```python
tf.reset_default_graph()

def discriminator(items):
    with tf.variable_scope('discriminator', reuse=tf.AUTO_REUSE):
        x = tf.concat([[items['mu'], items['tau']], items['eta']], axis=0)
        x = tf.reshape(x, [1, 10])
        x = tf.layers.dense(x, 10, activation=tf.nn.relu, name='layer1')
        x = tf.layers.dense(x, 10, activation=tf.nn.relu, name='layer2')
        x = tf.layers.dense(x, 1, name='layer3')
    return x

with Model() as model:
    @model.observe
    def _(s):
        s.y = tf.placeholder(name='y', dtype=tf.float32, shape=[1, 8])
        s.sigma = tf.placeholder(name='sigma', dtype=tf.float32, shape=[1, 8])

    @model.define
    def _(s):
        s.p.eta = tf.distributions.Normal(loc=tf.zeros([8]), scale=1.0)
        s.p.mu = tf.distributions.Normal(loc=0.0, scale=1.0)
        s.p.tau = tf.distributions.Normal(loc=0.0, scale=1.0)

        s.p.y = tf.distributions.Normal(
            loc=s.mu + s.tau * s.eta,
            scale=s.sigma,
        )

    @model.inference
    def _(s):
        x = tf.random_normal(shape=[10, 1])
        x = tf.layers.dense(x, 10, activation=tf.nn.relu)

        u = tf.random_normal(shape=[10, 1])
        u = tf.layers.dense(u, 10, activation=tf.nn.sigmoid)

        s.q.eta = Sample(u[:8, 0] * x[:8, 0])
        s.q.mu = Sample(u[8, 0] * x[8, 0])
        s.q.tau = Sample(u[9, 0] * x[9, 0])

kl_loss_, ratio_loss_, reference_loss_ = build_approx_entropy_losses(model, discriminator)
y_, sigma_, eta_, mu_, tau_ = model.get('y', 'sigma', 'eta', 'mu', 'tau')

# NOTE: supplying var_list is important
train_r_ = tf.train.AdamOptimizer(0.01).minimize(ratio_loss_, var_list=get_variables('discriminator'))
train_q_ = tf.train.AdamOptimizer(0.001).minimize(kl_loss_, var_list=get_variables('inference'))
train_reference_ = tf.train.AdamOptimizer(0.001).minimize(reference_loss_, var_list=get_variables('reference'))
```

# Misc

## Cauchy Reparametrization

```python
gamma = 0.5

plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)

u = np.random.uniform(size=10_000)
x = gamma * np.tan(np.pi * (u - 0.5))
_x = np.linspace(-gamma * 10, gamma * 10, 200)
_p = 1.0 / (np.pi * gamma * (1 + (_x / gamma) ** 2.0))

plt.hist(x, range=(-gamma * 10, gamma * 10), bins=31, normed=True, alpha=0.3)
plt.plot(_x, _p)
mpl_set(xlabel='x', ylabel='p(x)', title='Cauchy')

plt.subplot(1, 2, 2)

u = np.random.uniform(size=10_000)
x = gamma * np.tan(np.pi * 0.5 * u)
_x = np.linspace(0, gamma * 10, 200)
_p = 2.0 / (np.pi * gamma * (1 + (_x / gamma) ** 2.0))

plt.hist(x, range=(0, gamma * 10), bins=30, normed=True, alpha=0.3)
plt.plot(_x, _p)
mpl_set(xlabel='x', ylabel='p(x)', title='Half Cauchy')
```

## Ratio Estimation Derivation

Note: $r = r(z)$

$$
\begin{eqnarray}
    \sigma(r) + \sigma(-r)
        &=& \frac{1}{1 + e^{-r}} + \frac{1}{1 + e^{+r}}
\\
        &=& \frac{1 + e^{-r} + 1 + e^{+r} }{(1 + e^{-r}) (1 + e^{+r})}
\\
        &=& 1
\\
    \frac{\partial}{\partial r} \log \sigma(r)
        &=& \frac{e^{-r}}{1 + e^{-r}}
\\
        &=& \sigma(-r)
\\
    \frac{\partial}{\partial r} \log 1 - \sigma(r)
        &=& \frac{\partial}{\partial r} \log \sigma(-r)
\\
        &=& -\sigma(r)
\end{eqnarray}
$$

$$
\begin{eqnarray}
    \frac{\partial}{\partial r} \int \mathrm{d}z\; \left[
        p(z) \log \sigma(r) + q(z) \log \big( 1 - \sigma(r) \big)
    \right] &=& 0
\\
    \int \mathrm{d}z\; \left[ p(z) \sigma(-r) - q(z) \sigma(r) \right] &=& 0
\\
    \int \mathrm{d}z\; \left[ p(z) \sigma(-r) - q(z) (1 - \sigma(-r))\right] &=& 0
\\
    p(z) \sigma(-r) - q(z) (1 - \sigma(-r)) &=& 0
\\
    (p(z) + q(z)) \sigma(-r) &=& q(z)
\\
    \frac{1}{1 + e^r} &=& \frac{q(z)}{p(z) + q(z)}
\\
    e^r &=& \frac{p(z)}{q(z)}
\\
    r &=& \log \frac{p(z)}{q(z)}
\end{eqnarray}
$$

```python

```


```python
RAY_SWITCH = True # allow task-based parallel optimization by ray if set to True
```

```python
from sympy import init_printing; init_printing()

if RAY_SWITCH:
    import ray; ray.init()
    def raylize_func(func):
        @ray.remote
        def raylized_func(*args, **kwargs):
            return func(*args, **kwargs)
        return raylized_func

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all" # display all expression in one cell instead of the last one
```

## Initialization of Variables

```python
from sympy import integrate, Eq, symbols, Array, Function
from sinupy.algebra.tensor import cross
x, y, z = symbols('x, y, z', real=True); t = symbols("t", real=True)
E_x, E_y, E_z = symbols("E_x, E_y, E_z", real=True); E = Array([E_x, E_y, E_z]); E # Array([E_x, E_y, E_z]) or Array([E_x, 0, 0])
B_x, B_y, B_z = symbols("B_x, B_y, B_z", positive=True); B = Array([0, 0, B_z]); B # Array([B_x, B_y, B_z]) or Array([0, 0, B_z])
r_x, r_y, r_z = [f(t) for f in symbols("r_x, r_y, r_z", real=True, cls=Function)]; r = Array([r_x, r_y, r_z])
v_x, v_y, v_z = [f(t) for f in symbols("v_x, v_y, v_z", real=True, cls=Function)]; v = Array([v_x, v_y, v_z])
m = symbols('m', positive=True); q = symbols('q', nonzero=True, real=True)

```

## Set up the Newton Second Law Equation (ode system)

```python
Newton2 = Eq(q*E + q* cross(v, B), m * v.diff(t) );
diff_r_is_v = Eq(r.diff(t), v); diff_r_is_v
```

```python
from sympy import simplify, sqrt, trigsimp, refine, Q, sin, cos, pi, Abs
simplify = raylize_func(simplify)
trigsimp = raylize_func(trigsimp)
# [func = raylize_func(func) for func in [simplify,sqrt]]
import sympy.solvers.ode as ode
from wagglepy.symutil import divide_Array_Eq

# v_sol = ode.systems.dsolve_system(
#     divide_Array_Eq(Newton2))
# for i in range(3): v_sol[0][i]
# for i in range(3): v_sol[0][i].simplify()

v_r_sol = ode.systems.dsolve_system(
    [*divide_Array_Eq(Newton2),
     *divide_Array_Eq(diff_r_is_v)])
for comp_sol in v_r_sol[0]: comp_sol

v_r_sol[0] = ray.get([simplify.remote(comp_sol) for comp_sol in v_r_sol[0]]) if RAY_SWITCH \
    else [simplify(comp_sol) for comp_sol in v_r_sol[0]]

for comp_sol in v_r_sol[0]: comp_sol
```

```python

```

### Time Averaging

```python
from wagglepy.symutil.characteristics import min_period
from wagglepy.symutil.op import time_average_of_TrigFunc
tprime = symbols('t^{\prime}', real=True)
T = min_period(v_r_sol[0][0].rhs, t)

def subs_multiple_pi_trig_func(expr):
    return expr\
            .subs(sin((2*pi*q)/Abs(q)), 0)\
            .subs(sin((pi*q)/Abs(q)), 0)\
            .subs(cos((2*pi*q)/Abs(q)), 1)
def t_avg_Eq(eq, Tmin, Tmax):
        return Eq(
            time_average_of_TrigFunc(eq.lhs, t, T=[Tmin, Tmax]),
            time_average_of_TrigFunc(eq.rhs, t, T=[Tmin, Tmax]))
def simplify_rhs(eq):
    return Eq(eq.lhs, eq.rhs.simplify())

if RAY_SWITCH:
    t_avg_Eq = raylize_func(t_avg_Eq)
    subs_multiple_pi_trig_func = raylize_func(subs_multiple_pi_trig_func)
    simplify_rhs = raylize_func(simplify_rhs)

    # t_avg_EqSol.remote(v_r_sol[0][0])
    # t_avg_EqSol.remote(v_r_sol[0][1])
    # t_avg_EqSol.remote(v_r_sol[0][2])
    # print("The above formulas might look too cumbersome, let's simplify them.")
    futures = []
    for i, comp_sol in enumerate(v_r_sol[0]):
        t_avg_comp_future = t_avg_Eq.remote(comp_sol, 0, T) if i < 3 \
            else t_avg_Eq.remote(comp_sol, tprime, tprime+T)
        futures.append(
                simplify_rhs.remote(
                    subs_multiple_pi_trig_func.remote(
                        trigsimp.remote(
                                subs_multiple_pi_trig_func.remote(t_avg_comp_future),
                        method="fu")
                    )
                ))

    t_avg_Eq_list = ray.get(futures)
    for comp_sol in t_avg_Eq_list: comp_sol
else:
    pass

```

```python

```

```python

```

## Visualization of The Electron Trace Interacting with a Static E.M. Field

```python
from scipy import constants
import numpy as np
from sympy import lambdify, Symbol
v_r_sol_N = [comp_sol.rhs\
             .subs([(E_x, 6e5), (E_y, 6e5), (E_z, 1e4)])\
             .subs([(B_x, 0.0), (B_y, 0.0), (B_z, 5e-2)])\
             .subs([('C1', 1e3), ('C2', 0.0), ('C3', 0.0), ('C4', 0.0), ('C5', 0.0), ('C6', 0.0)])\
             .subs(q, constants.e).subs(m, constants.m_p) for comp_sol in v_r_sol[0]]
for i in range(6): v_r_sol_N[i]
```

```python
t1 = 2e-4
dt = 6e-8
num_of_t = int(t1 / dt); num_of_t
t_arr = np.arange(num_of_t) * dt

x_arr, y_arr, z_arr = (lambdify(t, v_r_sol_N[i], 'numpy')(t_arr) for i in [3, 4, 5])
# x_rest_arr = x_arr - a_0**2 / 4 * (t_arr-x_arr)
# z_arr = np.zeros_like(x_arr)

vx_arr, vy_arr, vz_arr = (lambdify(t, v_r_sol_N[i], 'numpy')(t_arr) for i in [0, 1, 2])
v_arr = np.sqrt(vx_arr**2 + vy_arr**2 + vz_arr**2)

```

```python
import plotly.graph_objects as go

fig = go.Figure(data=go.Scatter3d(
    x=x_arr, y=y_arr, z=z_arr,
    marker=dict(
        size=2,
        color=v_arr,
        colorscale='Viridis',
    ),
    line=dict(
        width=2,
        color=v_arr,
        colorscale='Viridis',
    )
))

fig.update_layout(
    scene=dict(
        aspectratio = dict( x=1, y=1, z=1.0 ),
        aspectmode = 'data'))
# fig.show()

```

## Bugs to be reported to Sympy

```python
from sympy import sin, sinh, pi, symbols
a = symbols("a", positive=True)
b, t = symbols("b, t", real=True)
sin(a*t).period(t)
```

```python
sin(-a*t).period(t) # AttributeError: 'Mul' object has no attribute 'period'
sin((-1)*a*t).period(t) # AttributeError: 'Mul' object has no attribute 'period'
```

```python
sin(b*t).period(t)
```

```python

```


# Worksheet 6

```
%matplotlib inline
```

## Question 1

Explain the shooting method for BVPs.

### Answer Question 1

The boundary value problem for $y(x)$ with boundary data at $x = a, b$ is converted to an initial value problem for $y(x)$ by, at first, guessing the additional (initial) boundary data $z$ at $x = a$ that is required for a properly posed (i.e., completely specified) IVP. The IVP can then be solved using any appropriate solver to get some solution $y(x; z)$ that depends on the guessed initial data $z$. By comparing against the required boundary data at $y(b)$ we can check if we have the correct solution of the original BVP. To be precise, we can write

\begin{equation}
  \phi (z) = \left. y(x; z) \right|_{x=b} − y(b),
\end{equation}

a nonlinear equation for $z$. At the root where $\phi(z) = 0$ we have the appropriate initial data $z$ such that the solution of the IVP is also a solution of the original BVP. The root of this nonlinear equation can be found using any standard method such as bisection or the secant method.

## Question 2

Give a complete algorithm for solving the BVP

\begin{equation}
  y'' − 3 y' + 2y = 0, \quad y(0) = 0, \quad y(1) = 1
\end{equation}

using the finite difference method. Include the description of the grid, the grid spacing, the treatment of the boundary conditions, the finite difference operators and a description of the linear system to be solved. You do not need to say which method would be used to solve the linear system, but should mention any special properties of the system that might make it easier to solve.

### Answer Question 2

We first choose the grid. We will use $N + 2$ point to cover the domain $x \in [0, 1]$; this implies that we have a grid spacing $h = 1 / (N + 1)$ and we can explicitly write the coordinates of the grid points as

\begin{equation}
  x_i = h i, \quad i = 0, 1, \dots , N + 1.
\end{equation}

We denote the value of the (approximate finite difference) solution at the grid points as $y_i (\approx y(x_i))$. We will impose the boundary conditions using

\begin{align}
  y_0 & = y(0) & y_{N +1} & = y(1) \\
      & = 0    &          & = 1.
\end{align}

We will use central differencing which gives

\begin{align}
  \left. y'(x)  \right|_{x = x_i} & \approx \frac{y_{i+1} − y_{i−1}}{2 h}, \\
  \left. y''(x) \right|_{x = x_i} & \approx \frac{y_{i+1} + y_{i−1} - 2 y_i}{h^2}.
\end{align}

We can then substitute all of these definitions into the original definition to find the finite difference
equation that holds for the interior points $i = 1, \dots , N$:

\begin{equation}
  y_{i+1} \left( 1 − \frac{3}{2} h \right) + y_i \left( −2 + 2 h^2 \right) + y_{i−1} \left( 1 + \frac{3}{2} h \right) = 0.
\end{equation}

This defines a linear system for the unknowns $y_i , i = 1, \dots , N$ of the form

\begin{equation}
  T {\bf y} = {\bf f}.
\end{equation}

We can see that the matrix $T$ is tridiagonal and has the form

\begin{equation}
  T =
    \begin{pmatrix}
      -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & 0 & 0 & \dots & 0 \\
      1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 & 0 & \dots & 0 \\
      0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 & \dots & 0 \\
      0 & 0 & \ddots & \ddots & \ddots & \dots & 0 \\
      0 & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 \\
      0 & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h \\
      0 & \dots & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2
    \end{pmatrix}
\end{equation}

The right hand side vector results from the boundary data and is

\begin{equation}
  {\bf f} = \begin{pmatrix} - \left( 1 + \tfrac{3}{2} h \right) y_0 \\ 0 \\ \vdots \\ 0 \\ - \left( 1 - \tfrac{3}{2} h \right) y_{N+1} \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ \vdots \\ 0 \\ 0 \end{pmatrix}.
\end{equation}

As the system is given by a tridiagonal matrix it is simple and cheap to solve using, e.g., the Thomas algorithm.

## Question 3

Explain how your algorithm would have to be modified to solve the BVP where the boundary condition at $x = 1$ becomes the Neumann condition

\begin{equation}
  y'(1) = 1 + \frac{e}{e − 1}.
\end{equation}

### Answer Question 3

First a finite difference representation of the boundary condition is required. A first order representation would be to use backward differencing

\begin{equation}
  \frac{y_{N + 1} − y_N}{h} = 1 + \frac{e}{e - 1}.
\end{equation}

This can be rearranged to give

\begin{equation}
  y_{N + 1} = y_N + h \left( 1 + \frac{e}{e − 1} \right).
\end{equation}

So now whenever we replaced $y(1)$ as represented by $y_{N + 1}$ by the boundary value in the previous algorithm we must instead replace it with the above equation which uses the known boundary data and unknown interior values.

Explicitly, this modifies the matrix $T$ to

\begin{equation}
  T =
    \begin{pmatrix}
      -2 + 2 h^2 & 1 - \tfrac{3}{2} h & 0 & 0 & 0 & \dots & 0 \\
      1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 & 0 & \dots & 0 \\
      0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 & \dots & 0 \\
      0 & 0 & \ddots & \ddots & \ddots & \dots & 0 \\
      0 & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h & 0 \\
      0 & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 & 1 - \frac{3}{2} h \\
      0 & \dots & \dots & \dots & 0 & 1 + \tfrac{3}{2} h & -2 + 2 h^2 + \color{red}{\left(1 - \frac{3}{2} h \right)}
    \end{pmatrix}
\end{equation}

and the right hand side vector ${\bf f}$ to

\begin{equation}
  {\bf f} = \begin{pmatrix} 0 \\ 0 \\ \vdots \\ 0 \\ \color{red}{- \left( 1 - \frac{3}{2} h \right) h \left( 1 + \frac{e}{e - 1} \right)} \end{pmatrix}.
\end{equation}

## Coding Question 1

Write a simple shooting method to solve the BVP
\begin{equation}
  y'' − 3 y' + 2 y = 0, \quad y(0) = 0, \quad y(1) = 1.
\end{equation}

Use standard black-box methods to solve the ODE, rewritten in first order form, and either a simple bisection method or the standard black-box methods to find the root. Compare your estimate against the answer

\begin{equation}
  y(x) = \frac{e^{2 x − 1} − e^{x − 1}}{e − 1}.
\end{equation}

### Answer Coding Question 1

```
def shooting_Dirichlet(f, ivp_interval, guess_interval, y_bc, method = 'brentq', tolerance = 1.e-8, MaxSteps = 100):
    """Solve the BVP z' = f(x, z) on x \in ivp_interval = [a, b] where z = [y, y'], subject to boundary conditions y(a) = y_bc[0], y(b) = y_bc[1]."""
    import numpy as np
    import scipy.integrate as si
    import scipy.optimize as so

    # Define the function computing the error in the boundary condition at b
    def shooting_phi(guess):
        """Internal function for the root-finding"""
        import numpy as np
        import scipy.integrate as si

        # The initial conditions from the guess and the boundary conditions
        y0 = [y_bc[0], guess]
        # Solve the IVP
        y = si.odeint(f, y0, np.linspace(ivp_interval[0], ivp_interval[1]))

        # Compute the error at the final point
        return y[-1, 0] - y_bc[1]

    # Choose between the root-finding methods
    if (method == 'bisection'):
        guess_min = guess_interval[0]
        guess_max = guess_interval[1]

        phi_min = shooting_phi(guess_min)
        phi_max = shooting_phi(guess_max)
        assert(phi_min * phi_max < 0.0)
        for i in range(MaxSteps):
            guess = (guess_min + guess_max) / 2.0
            phi = shooting_phi(guess)
            if (phi_min * phi < 0.0):
                guess_max = guess
                phi_max = phi
            else:
                guess_min = guess
                phi_min = phi
            if (abs(phi) < tolerance) or (guess_max - guess_min < tolerance):
                break
    elif (method == 'brentq'):
        guess = so.brentq(shooting_phi, guess_interval[0], guess_interval[1])
    else:
        raise Exception("method parameter must be in ['brentq', 'bisection']")

    # The initial conditions from the boundary, and the now "correct" value from the root-find
    y0 = [y_bc[0], guess]
    # Solve the IVP
    x = np.linspace(ivp_interval[0], ivp_interval[1])
    y = si.odeint(f, y0, x)

    return [x, y]

# Define the specific ODE to be solved
def f_bvp(y, x):
    """First order form of the above ODE"""
    import numpy as np

    dydx = np.zeros_like(y)

    dydx[0] = y[1]
    dydx[1] = 3.0 * y[1] - 2.0 * y[0]

    return dydx

# Define the exact solution for comparison
def y_exact(x):
    """Exact solution as given above."""
    import numpy as np

    return (np.exp(2.0*x - 1.0) - np.exp(x - 1.0)) / (np.exp(1.0) - 1.0)

# Now test it on the BVP to be solved
import numpy as np

x, y_brentq = shooting_Dirichlet(f_bvp, [0.0, 1.0], [-10.0, 10.0], [0.0, 1.0])
x, y_bisection = shooting_Dirichlet(f_bvp, [0.0, 1.0], [-10.0, 10.0], [0.0, 1.0], method = 'bisection')

import matplotlib.pyplot as plt

plt.figure(figsize = (12, 8))
plt.plot(x, y_brentq[:, 0], 'kx', x, y_bisection[:, 0], 'ro', x, y_exact(x), 'b-')
plt.xlabel('$x$', size = 16)
plt.ylabel('$y$', size = 16)
plt.legend(('Shooting, brentq method', 'Shooting, bisection', 'Exact'), loc = "upper left")

plt.figure(figsize = (12, 8))
plt.semilogy(x, np.absolute(y_brentq[:, 0] - y_exact(x)), 'kx', x, np.absolute(y_bisection[:, 0] - y_exact(x)), 'ro')
plt.xlabel('$x$', size = 16)
plt.ylabel('$|$Error$|$', size = 16)
plt.legend(('Shooting, brentq method', 'Shooting, bisection'), loc = "lower right")
plt.show()

```

## Coding Question 2

Implement your finite difference algorithm algorithm above to solve this BVP, using a standard black-box linear system solver. Show that your result converges to the correct answer.

### Answer Coding Question 2

```
def bvp_FD_Dirichlet(p, q, f, interval, y_bc, N = 100):
    """Solve linear BVP y'' + p(x) y' + q(x) y = f(x) on the given interval = [a, b] using y(a) = y_bc[0], y(b) = y_bc[1]."""
    import numpy as np
    import scipy.linalg as la

    h = (interval[1] - interval[0]) / (N + 1.0)
    # The grid, including boundaries, and set up final solution (fix at boundaries)
    x = np.linspace(interval[0], interval[1], N+2)
    y = np.zeros_like(x)
    y[0]  = y_bc[0]
    y[-1] = y_bc[1]
    # Set up diagonal entries of the matrix. Call sub-diagonal, diagonal, and super-diagonal vectors VE, VF, VG.
    VE = 1.0 - h / 2.0 * p(x[2:-1])
    VF = -2.0 + h**2 * q(x[1:-1])
    VG = 1.0 + h / 2.0 * p(x[1:-2])
    # Set up RHS vector F
    F = h**2 * f(x[1:-1])
    # Include boundary contributions
    F[0]  -= y_bc[0] * (1.0 - h / 2.0 * p(x[1]))
    F[-1] -= y_bc[1] * (1.0 + h / 2.0 * p(x[-2]))
    # Be lazy: set up full matrix
    T = np.diag(VE, -1) + np.diag(VF) + np.diag(VG, +1)
    y[1:-1] = la.solve(T, F)

    return [x, y]

# Define the problem to be solved

def bvp_p(x):
    """Term proportional to y' in definition of BVP"""
    import numpy as np

    return -3.0 * np.ones_like(x)

def bvp_q(x):
    """Term proportional to y in definition of BVP"""
    import numpy as np

    return 2.0 * np.ones_like(x)

def bvp_f(x):
    """Term on RHS in definition of BVP"""
    import numpy as np

    return np.zeros_like(x)

# Define the exact solution for comparison
def y_exact(x):
    """Exact solution as given above."""
    import numpy as np

    return (np.exp(2.0*x - 1.0) - np.exp(x - 1.0)) / (np.exp(1.0) - 1.0)

# Now solve the problem
import numpy as np

x, y = bvp_FD_Dirichlet(bvp_p, bvp_q, bvp_f, [0.0, 1.0], [0.0, 1.0])

import matplotlib.pyplot as plt

plt.figure(figsize = (12, 8))
plt.plot(x, y, 'kx', x, y_exact(x), 'b-')
plt.xlabel('$x$', size = 16)
plt.ylabel('$y$', size = 16)
plt.legend(('Finite difference solution', 'Exact'), loc = "upper left")

# Now do a convergence test
import scipy.linalg as la

levels = np.array(range(4, 10))
Npoints = 2**levels
err_2norm = np.zeros(len(Npoints))
for i in range(len(Npoints)):
    x, y = bvp_FD_Dirichlet(bvp_p, bvp_q, bvp_f, [0.0, 1.0], [0.0, 1.0], Npoints[i])
    err_2norm[i] = la.norm(y - y_exact(x), 2) / np.sqrt(Npoints[i])

# Best fit to the errors
h = 1.0 / Npoints
p = np.polyfit(np.log(h), np.log(err_2norm), 1)

fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.loglog(h, err_2norm, 'kx')
plt.loglog(h, np.exp(p[1]) * h**(p[0]), 'b-')
plt.xlabel('$h$', size = 16)
plt.ylabel('$\|$Error$\|_1$', size = 16)
plt.legend(('Finite difference errors', "Best fit line slope {0:.3}".format(p[0])), loc = "upper left")

plt.show()
```

## Coding Question 3

Modify your algorithm for the Neumann boundary condition above. Check that it converges to the same answer as for the Dirichlet case.

### Answer Coding Question 3

```
def bvp_FD_DirichletNeumann(p, q, f, interval, y_bc, N = 100):
    """Solve linear BVP y'' + p(x) y' + q(x) y = f(x) on the given interval = [a, b] using y(a) = y_bc[0], y'(b) = y_bc[1]."""
    import numpy as np
    import scipy.linalg as la

    h = (interval[1] - interval[0]) / (N + 1.0)
    # The grid, including boundaries, and set up final solution (fix at boundaries)
    x = np.linspace(interval[0], interval[1], N+2)
    y = np.zeros_like(x)
    y[0]  = y_bc[0]
    # Neumann boundary condition at the right end, so value of solution unknown

    # Set up diagonal entries of the matrix. Call sub-diagonal, diagonal, and super-diagonal vectors VE, VF, VG.
    VE = 1.0 - h / 2.0 * p(x[2:-1])
    VF = -2.0 + h**2 * q(x[1:-1])
    VG = 1.0 + h / 2.0 * p(x[1:-2])
    # Set up RHS vector F
    F = h**2 * f(x[1:-1])
    # Include boundary contributions
    F[0]  -= y_bc[0] * (1.0 - h / 2.0 * p(x[1]))
    # Neumann boundary condition at the right end - modify matrix and RHS vector
    VF[-1] += (1.0 + h / 2.0 * p(x[-2]))
    F[-1] -= (1.0 + h / 2.0 * p(x[-2])) * h * y_bc[1]
    # Be lazy: set up full matrix
    T = np.diag(VE, -1) + np.diag(VF) + np.diag(VG, +1)
    y[1:-1] = la.solve(T, F)
    # Finally set the solution at the right boundary
    y[-1] = y[-2] + h * y_bc[1]

    return [x, y]

# Define the problem to be solved

def bvp_p(x):
    """Term proportional to y' in definition of BVP"""
    import numpy as np

    return -3.0 * np.ones_like(x)

def bvp_q(x):
    """Term proportional to y in definition of BVP"""
    import numpy as np

    return 2.0 * np.ones_like(x)

def bvp_f(x):
    """Term on RHS in definition of BVP"""
    import numpy as np

    return np.zeros_like(x)

# Define the exact solution for comparison
def y_exact(x):
    """Exact solution as given above."""
    import numpy as np

    return (np.exp(2.0*x - 1.0) - np.exp(x - 1.0)) / (np.exp(1.0) - 1.0)

# Now solve the problem
import numpy as np

x, y = bvp_FD_DirichletNeumann(bvp_p, bvp_q, bvp_f, [0.0, 1.0], [0.0, 1.0 + np.exp(1.0) / (np.exp(1.0) - 1.0)])

import matplotlib.pyplot as plt

plt.figure(figsize = (12, 8))
plt.plot(x, y, 'kx', x, y_exact(x), 'b-')
plt.xlabel('$x$', size = 16)
plt.ylabel('$y$', size = 16)
plt.legend(('Finite difference solution', 'Exact'), loc = "upper left")

# Now do a convergence test
import scipy.linalg as la

levels = np.array(range(4, 10))
Npoints = 2**levels
err_DN_2norm = np.zeros(len(Npoints))
for i in range(len(Npoints)):
    x, y = bvp_FD_DirichletNeumann(bvp_p, bvp_q, bvp_f, [0.0, 1.0], [0.0, 1.0 + np.exp(1.0) / (np.exp(1.0) - 1.0)], Npoints[i])
    err_DN_2norm[i] = la.norm(y - y_exact(x), 2) / np.sqrt(Npoints[i])

# Best fit to the errors
h = 1.0 / Npoints
p = np.polyfit(np.log(h), np.log(err_DN_2norm), 1)

fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.loglog(h, err_DN_2norm, 'kx')
plt.loglog(h, np.exp(p[1]) * h**(p[0]), 'b-')
plt.xlabel('$h$', size = 16)
plt.ylabel('$\|$Error$\|_2$', size = 16)
plt.legend(('Finite difference errors (Neumann BC)', "Best fit line slope {0:.3}".format(p[0])), loc = "upper left")

plt.show()
```

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../../IPythonNotebookStyles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h2 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h3 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    div.text_cell_render{
        font-family: Gill, Verdana, Arial, Helvetica, sans-serif;
        line-height: 110%;
        font-size: 120%;
        width:700px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 12pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

> (The cell above executes the style for this notebook. It closely follows the style used in the [12 Steps to Navier Stokes](http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/) course.)