# Silver as constraint to large scale PV development

**Samuele Lo Piano** [slopiano@gmail.com](mailto:s.lopiano@gmail.com)
In this example we will analyse whether silver could constrain a large-scale
development of solar photovoltaics. Silver is used in the contact metallization
of crystalline-silicon-based PV cells. Remarkable efforts are being allocated
towards silver-paste-use reduction. On the other hand, solar photovoltaics
modules are being installed at an exponential increasing pace. Other parameters
are also included in our model, i.e.:

* other (non-industrial and industrial non-PV) uses of silver

* increase in efficiency of the modules

We will perform
a sensitivity analysis of a model aimed at defining silver extraction capacity
and the compatibility with its biophysical budget up to the year 2050.

Will
silver use in photovoltaics be the most pressing constraint or would other
sector impose more severe burdens? How about the extraction capacity - on the
base of the silver mining trends, could silver mining respond to a significant
uprise in silver demand? And finally, how about silver natural budget, can we be
sure the expansion of photovoltaics will not be constrained by silver natural
availability?

Let's start off by setting the plot environment and importing the
relevant packages.

```
# ipython magic
%matplotlib notebook
%load_ext autoreload
%autoreload 2
```

```
%matplotlib inline

# plot configuration
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

plt.style.use("ggplot")
# import seaborn as sns # sets another style
matplotlib.rcParams['lines.linewidth'] = 3
fig_width, fig_height = (7.0,5.0)
matplotlib.rcParams['figure.figsize'] = (fig_width, fig_height)

# font = {'family' : 'sans-serif',
#         'weight' : 'normal',
#         'size'   : 18.0}
# matplotlib.rc('font', **font)  # pass in the font dict as kwar
```

```
# import modules
import numpy as np
from scipy import special
import sobol_seq
import pandas as pd
from statsmodels.tsa.arima_model import ARIMA
import seaborn as sns
from mpl_toolkits.mplot3d import Axes3D
```

The yearly silver demand can be described as such:

$$
\begin{equation}
m_{Ag,yearly}(t)
=  \frac{m_{Ag,PV} e^{-Ag_{contact paste}\,t} PC_{PV} e^{-PC_{PVinstall}\,t}}{PV_{efficiency} e^{cell_{efficiency}\,t}}\  + Ag_{nonPV}(t)\
\label{_auto1} \tag{1}
\end{equation}
$$

Where $t$ is time relative to the base
year (2017); $m_{Ag,PV}$ is the amount of silver paste used per unit of
photovoltaic cell (100 mg/cell) in the starting year (2017); $PV_{PC}$ is the
yearly amount of first-generation crystalline-silicon-based power capacity
installed in 2017 (74 GW); $Ag_{nonPV}$ is the amount of silver used outside of
the solar photovoltaic sector; $PV_{efficiency}$ represents the typical cell
efficiency (4.27 W/cell, corresponding to 17.5%) in the base year. The typical
loss in efficiency from cells to modules has been considered as insignificant
for the purposes of the present accounting.

## Definining the variables and their distributions

The distributions of our parameters have been defined from
uniform Sobol distributions, which have been conveniently re-scaled or
transformed to other size and distribution shapes.

```
# Sample basis distribution
run = 50

sampleTot = sobol_seq.i4_sobol_generate(12,run*2**10)
```

The pace at which the silver-paste use for the contact metallization of solar
photovoltaic cells has been defined through a stakeholder-
engagment exercise.
Field experts have been contacted and have been asked to produce a guesstimate
on the silver use for contact metallization to the year 2050. These figures have
been complemented with a report, whose findings have been extrapolated to the
year 2050 - [ITRPV2018](http://www.itrpv.net/Reports/Downloads/).

```
# Current use of silver paste (mg/cell) - year 2017

ITRPV2018silverCell2017 = 100
ITRPVsilverCell2028 = 50
ITRPV2018decreasePace= - 1/(2028-2017) * np.log(ITRPVsilverCell2028/ITRPV2018silverCell2017)

def decreasePaceSilver2050(silverCell):
    return - 1/(2050-2017) * np.log(silverCell/ITRPV2018silverCell2017)

def mu(x):
    return np.mean(x)

def sigma(x):
    return np.std(x)
```

The upper boundary is conversely defined from the higher figure of the
distribution. In order to generate this latter, the figures provided have been
weighted according to the frequency of the responses received. A sample of
quasi-random numbers is eventually obtained from the truncated normal
distribution. A convenient size is chosen, i.e. $2^{10}$ - more than 1,000
values. The reader is encouraged to evaluate the effect of the sample size on
the basis of what has been illustrated for the convergence pace of the test
functions in [another notebook of this series](testfunctions3.ipynb). Here the
effect of the variability of the sample has been averaged out by repeating the
simulation 50 times.

```
silverCell2050DecreasePace = [ITRPV2018decreasePace, ITRPV2018decreasePace, \
decreasePaceSilver2050(ITRPV2018silverCell2017*0.01), decreasePaceSilver2050(ITRPV2018silverCell2017*0.01),\
decreasePaceSilver2050(10), decreasePaceSilver2050(15),decreasePaceSilver2050((20+10)/2), 0.05]
```

The growth in photovoltaic-cell efficiency has been taken as deterministic in
this study.

```
# PV Cell Efficiency growth up to the year 2050 (W/cell)

PVCellEfficiency2017 = 4.27

PVCellEfficiency2050 = 6.1

PVCellEfficiencyIncreasePace = 1/(2050-2017) * np.log(PVCellEfficiency2050/PVCellEfficiency2017)
```

Analogously, the distribution for the trend in the installed PV power capacity
is based on reports produced by energy agencies and research institutes (IEA,
Greenpeace and the Fraunhofer institute). It has been assumed the market share
of silicon-based PV solar panels would remain constant at 90%.

```
# Current expansion of PV power capacity (GW) - year 2017

PVPC2017 = 82
CrystallineSiliconShare = 0.9
IEA2050 = 4670
Greenpeace2050 = 9295
Fraunhofer2050 = 30700
CPVPC2017 = 402

IEA2050_CrystallineSiliconShare = IEA2050 * CrystallineSiliconShare
Greenpeace2050_CrystallineSiliconShare = Greenpeace2050 * CrystallineSiliconShare
Fraunhofer2050_CrystallineSiliconShare = Fraunhofer2050 * CrystallineSiliconShare
PVPC2017_CrystallineSiliconShare = PVPC2017 * CrystallineSiliconShare

PVPC2050IncreasePace = [0.025, 0.112, 0.061]
```

From which a truncated normal distribution has been generated. The low-
discrepancy Sobol sequence can be converted into other shape transformation by
evaluating the inverse probability distribution. As the value of probability
distribution shall be the same in order to maintain the low-discrepancy
properties of the Sobol sequence, one can apply an inverse transformation to
assure consistency.

The figures have been defined on the basis of the
[Thompson-Reuter/Silver Institute report](http://www.silverinstitute.org/wp-
content/uploads/2017/08/WSS2017.pdf) as regards the trends of silver use in
other industrial application and non-PV industrial applications. Their future
projections are based on the autoregressive integrated moving average
[ARIMA](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) approach.

```
# Non-industrial + Industrial Non-PV
otherAg = pd.DataFrame([28.8,29.4,32.8,26.0,30.7,32.5,29.3,33.4,33.2,34.0,29.9,28.7], index=[y for y in range(2006,2018)]).T
```

The trend for the yearly extraction capacity has been retrieve from the USGS
figures and is based on the projection from post World-War-II onwards.

```
# Yearly extraction capacity

Extraction = pd.DataFrame([3.97,5.22,5.44,5.57,6.32,6.21,6.7,6.9,6.67,7,7.02,7.19,7.43,6.91,7.32,7.37,7.65,7.78,7.73,8.01,8.3,8.03,8.56,9.2,9.36,9.17,9.38,9.7,9.26,9.43,9.84,10.3,
10.7,10.8,10.7,11.2,11.5,12.1,13.1,13.1,13,14,15.5,16.4,16.6,15.6,14.9,14.1,14,14.9,15.1,16.5,17.2,17.6,18.1,18.7,18.8,18.8,20,20.8,20.1,20.8,21.3,22.3,23.3,23.3,24.3,25.7,
26.8,27.6,25.7,25], index=[y for y in range(1946,2018)],columns=['Global Production'])
```

As regards the biophysical available budget, the figures for the ultimate
recoverable resources have been obtained by the literature. A uniform
distribution has been selected also in this case, whose extremes are the most
conservative and the most optmistic figures available in the literature.

### Output functions

We are now ready to generate the output functions: i) the
demand from the photovoltaics sector is evaluated along ii) the overall silver
demand and iii) its cumulative figure up to the year 2050. iv) The derivative of
the yearly demand has also been computed. In order to analyse any viability
constraint (technical incapacity to mine adequate supply of minerals to face the
demand on a year-to-year base) and any potential feasibility constraint
(availability of silver natural budget), the trends of these variables have also
been assessed below. As regards the former, it is assumed silver from old-scrap
recycling provides only 20% of the yearly supply. This figure is in line with
the recent global trends reported. The latter is evaluated by comparing the
figures for the cumulative extraction against silver natural budget.

```
run = 50

S_silverCellR = []
S_PVPCR = []
S_otherAgR = []
T_silverCellR = []
T_PVPCR = []
T_otherAgR = []
yearlySilverR = []
CumulativeSilverR = []
PVPC2050R = []
silverCellR = []
otherAgR = []
silverDeltaR = []
DifferenceR = []

for r in range(run):
    sample = sampleTot[int(r*len(sampleTot)/run):int((r+1)*len(sampleTot)/run)].T

    silverCellDistribution2 = sigma(silverCell2050DecreasePace)*2**0.5*special.erfinv(sample[0]*(1-special.erf(-mu(silverCell2050DecreasePace)/
    (sigma(silverCell2050DecreasePace)*2**0.5)))+special.erf(-mu(silverCell2050DecreasePace)/
    (sigma(silverCell2050DecreasePace)*2**0.5)))+mu(silverCell2050DecreasePace)

    silverCellDistribution2B = sigma(silverCell2050DecreasePace)*2**0.5*special.erfinv(sample[6]*(1-special.erf(-mu(silverCell2050DecreasePace)/
    (sigma(silverCell2050DecreasePace)*2**0.5)))+special.erf(-mu(silverCell2050DecreasePace)/
    (sigma(silverCell2050DecreasePace)*2**0.5)))+mu(silverCell2050DecreasePace)

    PVPC2050IncreasePaceDist = sigma(PVPC2050IncreasePace)*2**0.5*special.erfinv(sample[1]*(1-special.erf(-mu(PVPC2050IncreasePace)/
    (sigma(PVPC2050IncreasePace)*2**0.5)))+special.erf(-mu(PVPC2050IncreasePace)/
    (sigma(PVPC2050IncreasePace)*2**0.5)))+mu(PVPC2050IncreasePace)

    PVPC2050IncreasePaceDistB = sigma(PVPC2050IncreasePace)*2**0.5*special.erfinv(sample[7]*(1-special.erf(-mu(PVPC2050IncreasePace)/
    (sigma(PVPC2050IncreasePace)*2**0.5)))+special.erf(-mu(PVPC2050IncreasePace)/
    (sigma(PVPC2050IncreasePace)*2**0.5)))+mu(PVPC2050IncreasePace)

    Noise = sigma(sample[2])*2**0.5*special.erfinv(2*sample[2]-1)
    NoiseB = sigma(sample[8])*2**0.5*special.erfinv(2*sample[8]-1)

    # fit model Silver
    model = ARIMA(otherAg.iloc[-1]+Noise[:len(otherAg.T)], order=(0, 1, 1))
    model_fit = model.fit(disp=False)
    # make prediction
    yhat = model_fit.predict(len(otherAg.iloc[-1]), 44, typ='levels')
    yhat.index = yhat.index + 2006
    forecast, stderr, conf = model_fit.forecast(33)

    modelB = ARIMA(otherAg.iloc[-1]+NoiseB[:len(otherAg.T)], order=(0, 1, 1))
    model_fitB = modelB.fit(disp=False)
    # make prediction
    yhatB = model_fitB.predict(len(otherAg.iloc[-1]), 44, typ='levels')
    yhatB.index = yhat.index
    forecastB, stderrB, confB = model_fitB.forecast(33)

    otherAgDist = pd.DataFrame([((forecast[i]-conf[i][0])/1.96)*2**0.5*special.erfinv(sample[3]*(1-special.erf(-forecast[i]/
    (((forecast[i]-conf[i][0])/1.96)*2**0.5)))+special.erf(-forecast[i]/
    (((forecast[i]-conf[i][0])/1.96)*2**0.5)))+forecast[i] for i in range(len(forecast))],index=[i for i in range(2018,2051)])

    otherAgDistB = pd.DataFrame([((forecastB[i]-confB[i][0])/1.96)*2**0.5*special.erfinv(sample[9]*(1-special.erf(-forecastB[i]/
    (((forecastB[i]-confB[i][0])/1.96)*2**0.5)))+special.erf(-forecastB[i]/
    (((forecastB[i]-confB[i][0])/1.96)*2**0.5)))+forecastB[i] for i in range(len(forecastB))],index=[i for i in range(2018,2051)])

    # Resources and reserves
    distributionSilverReserveResource = sample[4]*(4000-530)+530
    distributionSilverReserveResource = sample[10]*(4000-530)+530

    # fit model Extraction
    model2 = ARIMA(Extraction['Global Production']+Noise[:len(Extraction)], order=(2, 1, 1))
    model_fit2 = model2.fit(disp=False)
    # make prediction
    yhat2 = model_fit2.predict(len(Extraction), 105, typ='levels')
    yhat2.index = yhat2.index+1946
    forecast2, stderr2, conf2 = model_fit2.forecast(33)

    model2B = ARIMA(Extraction['Global Production']+NoiseB[:len(Extraction)], order=(2, 1, 1))
    model_fit2B = model2B.fit(disp=False)
    # make prediction
    yhat2B = model_fit2B.predict(len(Extraction), 105, typ='levels')
    yhat2B.index = yhat2B.index+1946
    forecast2B, stderr2B, conf2B = model_fit2B.forecast(33)

    # Extraction distribution
    ExtractionDist = pd.DataFrame([((forecast2[i]-conf2[i][0])/1.96)*2**0.5*special.erfinv(sample[5]*(1-special.erf(-forecast2[i]/
    (((forecast2[i]-conf2[i][0])/1.96)*2**0.5)))+special.erf(-forecast2[i]/
    (((forecast2[i]-conf2[i][0])/1.96)*2**0.5)))+forecast2[i] for i in range(len(forecast2))],index = otherAgDist.index)

    ExtractionDistB = pd.DataFrame([((forecast2B[i]-conf2B[i][0])/1.96)*2**0.5*special.erfinv(sample[11]*(1-special.erf(-forecast2B[i]/
    (((forecast2B[i]-conf2B[i][0])/1.96)*2**0.5)))+special.erf(-forecast2B[i]/
    (((forecast2B[i]-conf2B[i][0])/1.96)*2**0.5)))+forecast2B[i] for i in range(len(forecast2B))],index = otherAgDistB.index)

    # Generation of the yearly silver output (1,000 metric tonnes)
    PVDemand = pd.DataFrame([(ITRPV2018silverCell2017 * np.exp(-silverCellDistribution2*i1)/(PVCellEfficiency2017 * \
    np.exp(PVCellEfficiencyIncreasePace*i1)))* PVPC2017 * CrystallineSiliconShare * np.exp(PVPC2050IncreasePaceDist*i1)/1000 for i1
    in range (2051-2018)],index=otherAgDist.index)

    PVDemandAB1 = pd.DataFrame([(ITRPV2018silverCell2017 * np.exp(-silverCellDistribution2B*i1)/(PVCellEfficiency2017 * \
    np.exp(PVCellEfficiencyIncreasePace*i1)))* PVPC2017 * CrystallineSiliconShare * np.exp(PVPC2050IncreasePaceDist*i1)/1000 for i1
    in range (2051-2018)],index=otherAgDist.index)

    PVDemandAB2 = pd.DataFrame([(ITRPV2018silverCell2017 * np.exp(-silverCellDistribution2*i1)/(PVCellEfficiency2017 * \
    np.exp(PVCellEfficiencyIncreasePace*i1)))* PVPC2017 * CrystallineSiliconShare * np.exp(PVPC2050IncreasePaceDistB*i1)/1000 for i1
    in range (2051-2018)],index=otherAgDist.index)

    PVDemandB = pd.DataFrame([(ITRPV2018silverCell2017 * np.exp(-silverCellDistribution2B*i1)/(PVCellEfficiency2017 * \
    np.exp(PVCellEfficiencyIncreasePace*i1)))* PVPC2017 * CrystallineSiliconShare * np.exp(PVPC2050IncreasePaceDistB*i1)/1000 for i1
    in range (2051-2018)],index=otherAgDist.index)

    # Generation of the cumulative silver output (1,000 metric tonnes)
    yearlySilver = PVDemand+otherAgDist
    yearlySilverAB1 = PVDemandAB1+otherAgDist
    yearlySilverAB2 = PVDemandAB2+otherAgDist
    yearlySilverAB3 = PVDemand+otherAgDistB
    yearlySilverB = PVDemandB+otherAgDistB

    CumulativeSilver = yearlySilver.cumsum()

    # Derivative Yearly Silver Demand
    derivativeYearlySilver = yearlySilver.diff()

    # Yearly Delta (viability, technological constraint)

    silverDelta = ExtractionDist - 0.8*yearlySilver

    # Cumulative Delta (feasibility, biophysical constraint)

    Difference = distributionSilverReserveResource-CumulativeSilver

    S_silverCell = 1-(0.5*((yearlySilverB-yearlySilverAB1)**2).mean(axis=1))/yearlySilverB.var(axis=1,ddof=0)
    S_PVPC = 1-(0.5*((yearlySilverB-yearlySilverAB2)**2).mean(axis=1))/yearlySilverB.var(axis=1,ddof=0)
    S_otherAg = 1-(0.5*((yearlySilverB-yearlySilverAB3)**2).mean(axis=1))/yearlySilverB.var(axis=1,ddof=0)
    T_silverCell = (0.5*((yearlySilver-yearlySilverAB1)**2).mean(axis=1))/yearlySilver.var(axis=1,ddof=0)
    T_PVPC = (0.5*((yearlySilver-yearlySilverAB2)**2).mean(axis=1))/yearlySilver.var(axis=1,ddof=0)
    T_otherAg = (0.5*((yearlySilver-yearlySilverAB3)**2).mean(axis=1))/yearlySilver.var(axis=1,ddof=0)

    S_silverCellR.append(S_silverCell)
    S_PVPCR.append(S_PVPC)
    S_otherAgR.append(S_otherAg)
    T_silverCellR.append(T_silverCell)
    T_PVPCR.append(T_PVPC)
    T_otherAgR.append(T_otherAg)
    yearlySilverR.append(yearlySilver)
    PVPC2050R.append(PVPC2050IncreasePaceDist)
    silverCellR.append(silverCellDistribution2)
    CumulativeSilverR.append(CumulativeSilver)
    otherAgR.append(otherAgDist)
    silverDeltaR.append(silverDelta)
    DifferenceR.append(Difference)

S_silverCell_df = pd.DataFrame(S_silverCellR)
S_PVPC_df = pd.DataFrame(S_PVPCR)
S_otherAg_df = pd.DataFrame(S_otherAgR)
T_silverCell_df = pd.DataFrame(T_silverCellR)
T_PVPC_df = pd.DataFrame(T_PVPCR)
T_otherAg_df = pd.DataFrame(T_otherAgR)
```

```
df20503d = pd.concat([pd.Series(np.array(silverCellR).flatten()),pd.Series(np.array(PVPC2050R).flatten()),
pd.concat([oa.loc[2050] for oa in otherAgR],axis=0).reset_index(drop=True),
pd.concat([yr.loc[2050] for yr in yearlySilverR],axis=0).reset_index(drop=True),
pd.concat([cr.loc[2050] for cr in CumulativeSilverR],axis=0).reset_index(drop=True),
pd.concat([sd.loc[2050] for sd in silverDeltaR],axis=0).reset_index(drop=True),
pd.concat([df.loc[2050] for df in DifferenceR],axis=0).reset_index(drop=True)],axis=1)
df20503d.columns = ['silverCell','PVPC','nonPVsilver','yearlySilver','CumulativeSilver','yearly Delta','Difference']
```

Each of the distributions generated can be visually inspected to acknowledge its
trend in the inquired timeframe with an **uncertainty analysis**.

```
for i2, row in PVDemand.iterrows():
    plt.plot([i2 for col in range(len(PVDemand.columns))], row, c = 'y', label = 'Silver PV Demand' if i2 == 2018 else "")
plt.legend()
plt.show()
```

```
for i2, row in yearlySilver.iterrows():
    plt.plot([i2 for col in range(len(yearlySilver.columns))], row, c = 'r', label = 'Silver Total Demand' if i2 == 2018 else "")
plt.legend()
plt.show()
```

```
for i2, row in CumulativeSilver.iterrows():
    plt.plot([i2 for col in range(len(CumulativeSilver.columns))], row, c = 'g', label = 'Cumulative Silver Demand' if i2 == 2018 else "")
    plt.scatter(i2, row.mean(), c = 'm', label = 'Cumulative Silver demand mean' if i2 == 2018 else "")
plt.legend()
plt.show()
```

```
for i2, row in derivativeYearlySilver.iterrows():
    plt.plot([i2 for col in range(len(derivativeYearlySilver.columns))], row, c = 'k', label = 'derivative Silver' if i2 == 2018 else "")
plt.legend()
plt.show()
```

The trends of viability and feasibility can also be visually inspected.

```
for i2, row in silverDelta.iterrows():
    plt.plot([i2 for col in range(len(silverDelta.columns))], row, c = 'c')
plt.show()
```

```
for i2, row in Difference.iterrows():
    plt.plot([i2 for col in range(len(Difference.columns))], row, c = 'w', label = 'Difference' if i2 == 2018 else "")
    plt.scatter(i2, row.mean(), c = 'm', label = 'Difference mean' if i2 == 2018 else "")
plt.legend()
plt.show()
```

```
for i, row in silverDelta.iterrows():
    plt.bar(i,row[row>0].count(),color='b')
    plt.bar(i,row[row<0].count(),bottom=row[row>0].count(),color='r')
plt.show()
```

```
for i, row in Difference.iterrows():
    plt.bar(i,row[row>0].count(),color='b')
    plt.bar(i,row[row<0].count(),bottom=row[row>0].count(),color='r')
plt.show()
```

One can see that for some combination of values, the expansion of photovoltaics
power capacity may not be feasible or viable. Precisely, constraints may appear
from 2035 onwards. Viability issues seem more likely.

## Sensitivity analysis
What are the parameters whose uncertainty affects the output the most? This
question is of primary importance if one wants to address where the potential
constraint to large-scale development of solar photovoltaics would most likely
come from. To this end, sensitivity analysis can play a role by helping us to
assess which parameters uncertainty will affect the yearly silver demand the
most and with it all the other derived variables.

We will take advantage of
three toolkits to apportion the uncertainty in the output to the input
parameters uncertainty:

i) scatter plots for a rough visual correlation;

ii) Smirnov plots for a more quantitative visual inference;

iii) Variance-based sensitivity indices.

### Scatter plots

These visualizations allow to evaluate
how the uncertainty in the output varies along with the value of the input
variables. These plots can be interpreted in this way: How much may the output
variance be reduced by fixing the input? Is an overall trend visible or only a
random pattern? The input parameter's uncertainty plays a tangible role on the
output's uncertainty if one can discern a trend. Linear regressions have also
been drawn on the scatter plots to enhance the visibility of these effects. The
trends suggests an increasing importance over the course of the years for the PV
variables and the reverse for non-PV silver.

```
for i2, row in yearlySilver.iterrows():
    plt.scatter(silverCellDistribution2, row, s = 2, label = 'silverCellDistribution'+str(i2))
    plt.plot(np.unique(silverCellDistribution2), np.poly1d(np.polyfit(silverCellDistribution2, row, 1))(np.unique(silverCellDistribution2)), color='r')
    plt.legend()
    plt.show()
```

```
for i2, row in yearlySilver.iterrows():
    plt.scatter(silverCellDistribution2, row, s = 2, label = 'silverCellDistribution'+str(i2))
    plt.plot(np.unique(silverCellDistribution2), np.poly1d(np.polyfit(silverCellDistribution2, row, 1))(np.unique(silverCellDistribution2)), color='r')
    plt.legend()
    plt.show()
```

```
for i2, row in yearlySilver.iterrows():
    plt.scatter(otherAgDist.loc[i2], row, s = 2, label = 'Ag_nonPV'+str(i2))
    plt.plot(np.unique(otherAgDist.loc[i2]), np.poly1d(np.polyfit(otherAgDist.loc[i2], row, 1))(np.unique(otherAgDist.loc[i2])), color='b')
    plt.legend()
    plt.show()
```

### Kolmogorov-Smirnov test

This tool allows to define the importance on the
variable on the basis of the distance between the 'behavioural' distribution and
the 'non-behavioural' one. Behavioural corresponds to the feasibility/viability
area. The opposite applies to the non-behavioural rangwe
[saltelli_global_2008](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470725184).
This approach is part of the *Monte Carlo Filtering* methodology, more precisely
to the *Regional Sensitivity Analysis* framework. The charts below are shown for
the year 2050. The interested reader is invited to replicate the test for the
other years as exercise.

```
#Smirnov-Kolmorov Test

BVsilverCell=df20503d['silverCell'][df20503d['yearly Delta']>0].sort_values()
BVsilverCellCum = BVsilverCell.cumsum()/BVsilverCell.sum()
NBVsilverCell=df20503d['silverCell'][df20503d['yearly Delta']<0].sort_values()
NBVsilverCellCum = NBVsilverCell.cumsum()/NBVsilverCell.sum()

ax1 = plt.plot(BVsilverCell,BVsilverCellCum,label='silverCell_B',c='b')
plt.plot(NBVsilverCell,NBVsilverCellCum,label='silverCell_NB',c='r')
plt.legend()
```

```
BFsilverCell=df20503d['silverCell'][df20503d['Difference']>0].sort_values()
BFsilverCellCum = BFsilverCell.cumsum()/BFsilverCell.sum()
NBFsilverCell=df20503d['silverCell'][df20503d['Difference']<0].sort_values()
NBFsilverCellCum = NBFsilverCell.cumsum()/NBFsilverCell.sum()

ax1 = plt.plot(BFsilverCell,BFsilverCellCum,label='silverCell_B',c='b')
plt.plot(NBFsilverCell,NBFsilverCellCum,label='silverCell_NB',c='r')
plt.legend()
```

```
BVPVPC=df20503d['PVPC'][df20503d['yearly Delta']>0].sort_values()
BVPVPCCum = BVPVPC.cumsum()/BVPVPC.sum()
NBVPVPC=df20503d['PVPC'][df20503d['yearly Delta']<0].sort_values()
NBVPVPCCum = NBVPVPC.cumsum()/NBVPVPC.sum()

ax1 = plt.plot(BVPVPC,BVPVPCCum,label='PVPC_B',c='b')
plt.plot(NBVPVPC,NBVPVPCCum,label='PVPC_NB',c='r')
plt.legend()
```

```
BFPVPC=df20503d['PVPC'][df20503d['Difference']>0].sort_values()
BFPVPCCum = BFPVPC.cumsum()/BFPVPC.sum()
NBFPVPC=df20503d['PVPC'][df20503d['Difference']<0].sort_values()
NBFPVPCCum = NBFPVPC.cumsum()/NBFPVPC.sum()

ax1 = plt.plot(BFPVPC,BFPVPCCum,label='PVPC_B',c='b')
plt.plot(NBFPVPC,NBFPVPCCum,label='PVPC_NB',c='r')
plt.legend()
```

```
#Smirnov-Kolmorov Test

BVnonPVsilver=df20503d['nonPVsilver'][df20503d['yearly Delta']>0].sort_values()
BVnonPVsilverCum = BVnonPVsilver.cumsum()/BVnonPVsilver.sum()
NBVnonPVsilver=df20503d['nonPVsilver'][df20503d['yearly Delta']<0].sort_values()
NBVnonPVsilverCum = NBVnonPVsilver.cumsum()/NBVnonPVsilver.sum()

ax1 = plt.plot(BVnonPVsilver,BVnonPVsilverCum,label='nonPVsilver_B',c='b')
plt.plot(NBVnonPVsilver,NBVnonPVsilverCum,label='nonPVsilver_NB',c='r')
plt.legend()
```

```
#Smirnov-Kolmorov Test

BFnonPVsilver=df20503d['nonPVsilver'][df20503d['Difference']>0].sort_values()
BFnonPVsilverCum = BFnonPVsilver.cumsum()/BFnonPVsilver.sum()
NBFnonPVsilver=df20503d['nonPVsilver'][df20503d['Difference']<0].sort_values()
NBFnonPVsilverCum = NBFnonPVsilver.cumsum()/NBFnonPVsilver.sum()

ax1 = plt.plot(BFnonPVsilver,BFnonPVsilverCum,label='nonPVsilver_B',c='b')
plt.plot(NBFnonPVsilver,NBFnonPVsilverCum,label='nonPVsilver_NB',c='r')
plt.legend()
```

The two curves represent the cumulative distribution probabilities. The larger
is the maximum distance between the curves, the more important is the input
parameter's uncertainty on the output uncertainty. For an in-depth view of the
methodology, one can have a look at [the above-mentioned
contribution](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470725184).
The trends for the three variables inquired suggest silver non-PV use may be the
most important variable in 2050 when assessing techno-economic (viability) and
biophysical (feasibility) constraints. Yet the approach is far from being
flawless as it addresses a factor significance, which may be different from
importance (a Smirnov-insignificance result does not necessary imply non-
importance). Furthermore, compensatory effects may occur along with more complex
interactions unable to detect in the test's graphical framework.

### Variance-based Sensitivity Analysis

When one strives for a precise quantitative
information, Sobol sensitivity indices are a very convenient mean to achieve it.
We report here the first-order indices along with total indices in order to
acknowledge for their potential higher order interaction. The model is additive
with the exception of the potential interaction between silver use in PV cells
and power capacity installed. The indices have been computed for the year 2050
and are shown in the table below. One can easily compute the indices also for
other years.

The formula for the estimators are those of [Saltelli et al.
2010](https://www.sciencedirect.com/science/article/pii/S0010465509003087)
Jansen 1999, respectively (references therein, Table 2) for the first-order
sensitivity index and the total sensitivity index. Their expressions are

$$
\begin{equation}
S_i = V(Y)-\frac{1}{2N}
\sum^N_{j=1}({f(B)_j}-{f(A_B^{(i)})_j})^2
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation}
T_i = \frac{1}{2N} \sum^N_{j=1}({f(A)_j}-{f(A_B^{(i)})_j})^2
\label{_auto3} \tag{3}
\end{equation}
$$

The two sample matrices A and B have
been obtained by slicing column-wise our Sobol sample distributions.

```
# Sensitivity indices plots

df_l = [S_silverCell_df,S_PVPC_df,T_silverCell_df-S_silverCell_df,S_otherAg_df]
df_names = ['S_silverCell_df','S_PVPC_df','S_2order_silverCellPVPC','S_otherAg_df']
colors = ['b','r','g','k']
props = [dict(boxes=co, whiskers=co, medians=co, caps=co) for co in colors]
fig, ax = plt.subplots(figsize=(20,10))
patch = []
for id, dfl in enumerate(df_l):
    dfl.plot(kind='box',ax=ax,label=df_names[id],positions=[c+id/len(df_l) for c in dfl],color=props[id],
            patch_artist=True,showfliers=False)
    plt.ylim(-0.1,1.1)
plt.legend()
plt.show()
```

Black: Silver use in non-photovoltaics; Blue: Silver use in contact
metallization; Red: Expansion of solar power capacity; Green: Second order index
for the interaction between silver use in contact metallization and the
expansion of solar power capacity.

```
fig = plt.figure(figsize=(20,10))
ax = fig.gca(projection='3d')
surf=ax.plot_trisurf(df20503d['silverCell'], df20503d['PVPC'], df20503d['CumulativeSilver'], cmap=plt.cm.jet, linewidth=0.01)
ax.set_xlabel('Silver Contact metallization decrease pace')
ax.set_ylabel('PV Power Capacity increase pace')
ax.set_zlabel('Cumulative Silver 2050 (kton)')
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

```
fig = plt.figure(figsize=(20,10))
ax = fig.gca(projection='3d')
surf=ax.plot_trisurf(df20503d['silverCell'], df20503d['PVPC'], df20503d['yearlySilver'], cmap=plt.cm.jet, linewidth=0.01)
ax.set_xlabel('Silver Contact metallization decrease pace')
ax.set_ylabel('PV Power Capacity increase pace')
ax.set_zlabel('Yearly Silver 2050 (kton)')
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

## Conclusion

The importance of the pace of reducing use of silver for contact
metallization along with the increase in the installed PV power capacity
augments over the course of the years as witnessed by the sensitivity indices.
Higher order interactions between the two parameters can be easily appreciated.


# 第3章　ニューラルネットワーク

## 3.2 活性化関数

anacondaのバージョンはanaconda3-2020.02(python3.7.6)です。

python -Vで確認してください。

注記

20210128時点でpython3.8ではcondaでinstallできるtensorflowが対応していないため、python3.7のanacondaを使います。

このレジュメはmacで作っています。

私はpythonをpyenvで管理し、anacondaを使用します。

Pythonライブラリーは極力condaで管理します。

condaとpipはできるだけ混ぜないようにしたほうが安全です。

pyenvでanaconda3-2020.011をインストールしglobal設定します。

pyenv install anaconda3-2020.11

pyenv global anaconda3-2020.11

その後python3.7の仮想環境をつくりactivateします。

conda create -n py37 jupyter jupyterlab tensorflow matplotlib scikit-learn
conda activate py37

conda upgrade -all

プロンプトが

(py37)[Hostname]%

となっていることを確認してjupyter labを立ち上げます。

もしパッケージがimport出来ない場合はコンソールでpy37をactivateし、conda install コマンドで追加インストールします。

最低限必要なパッケージをimportしておきます。

```python
import os
import numpy as np
import matplotlib.pylab as plt
```

### 活性化関数

以下の３つの活性化関数について式の定義と実装を説明します。
１）ステップ関数

２）シグモイド関数

３）ReLU関数

### 3.2.1 ステップ関数の実装

スッテプ関数の式
$$
\begin{equation}
  y(x) =\begin{cases}
 1 & \text{$x>0$} \\
 0 & \text{$x \leq 0$}\\
 \end{cases}
 \end{equation}
$$

```python
### ステップ関数の定義

def step_function(x):
    return np.array(x>0, dtype=np.int)
```

```python
#ステップ関数の表示

x=np.arange(-5.0,5.0,0.1)
y1=step_function(x)
plt.plot(x,y1,color='blue')
plt.ylim(-0.1, 1.1)
```

### 3.2.2 シグモイド関数の実装

シグモイド関数の式
$$
\begin{equation}
    y=\frac{1}{1+\exp(x)}\\
\end{equation}
$$

```python
### シグモイド関数の定義

def sigmoid_function(x):
    return 1/(1+np.exp(-x))
```

```python
#シグモイド関数の表示

y2=sigmoid_function(x)
plt.plot(x,y1,color='red',linestyle="dashed")
plt.plot(x,y2,color='blue',linestyle="solid")
plt.ylim(-0.1, 1.1)
plt.legend({'step function','sigmoid function'})
```

### 3.2.3 ReLU関数の実装

ReLU (Rectified Linear Unit) 関数の式
$$
\begin{equation}
  y(x) =\begin{cases}
 x & \text{$x>0$} \\
 0 & \text{$x \leq 0$}\\
 \end{cases}
 \end{equation}
$$

```python
#ReLU関数の定義

def ReLU_function(x):
    return np.maximum(0,x)
```

```python
y3=ReLU_function(x)
x=np.arange(-5,5,0.1)
plt.plot(x,y3,color='blue',linestyle='solid')
#plt.ylim(-0.1, 1.1)
```

## 3.3 多次元配列の計算
ここではpythonの行列表現と行列計算について確認します。

```python
A = np.array([1, 2, 3, 4])
print(A)
```

    [1 2 3 4]

```python
np.ndim(A)
```

    1

```python
A.shape
```

    (4,)

```python
B = np.array([[1, 2], [3, 4], [5, 6]])
print(B)
```

    [[1 2]
     [3 4]
     [5 6]]

```python
np.ndim(B)
```

    2

```python
B.shape
```

    (3, 2)

```python
A = np.array([[1, 2], [3, 4]])
print(A)
```

    [[1 2]
     [3 4]]

```python
B = np.array([[5, 6], [7, 8]])
print(B)
```

    [[5 6]
     [7 8]]

```python
np.dot(A, B)
```

    array([[19, 22],
           [43, 50]])

$$
    \left(
        \begin{array}{cc}
        1 & 2\\
        3 & 4\\
        \end{array}
    \right)
    \left(
        \begin{array}{cc}
        5 & 6\\
        7 & 8\\
        \end{array}
    \right) =
    \left(
        \begin{array}{cc}
        19 & 22\\
        43 & 50\\
        \end{array}
    \right)
$$

## 3.4 3層ニューラルネットワークの実装

```python
def init_network():
    network = {}
    network['W1'] = np.array([[0.1, 0.3, 0.5], [0.2, 0.4, 0.6]])
    network['b1'] = np.array([0.1, 0.2, 0.3])
    network['W2'] = np.array([[0.1, 0.4], [0.2, 0.5],[0.3, 0.6]])
    network['b2'] = np.array([0.1, 0.2])
    network['W3'] = np.array([[0.1, 0.3], [0.2, 0.4]])
    network['b3'] = np.array([0.1, 0.2])

    return network
```

```python
def identity_function(x):
    y = x

    return x
```

```python
def forward(network, x):
    W1, W2, W3 = network['W1'], network['W2'], network['W3']
    b1, b2, b3 = network['b1'], network['b2'], network['b3']

    a1 = np.dot(x, W1) + b1
    z1 = sigmoid_function(a1)

    a2 = np.dot(z1, W2) + b2
    z2 = sigmoid_function(a2)

    a3 = np.dot(z2, W3) + b3
    y = identity_function(a3)

    return y

```

```python
network = init_network()
x = np.array([1.0, 0.5])
y = forward(network, x)

print(y)
```

    [0.31682708 0.69627909]

## 3.5 出力層の設計

### 3.5.1（素朴な）softmax関数

合計が１になる。分類器の出力で使用する。
$$
    y_k=\frac{exp(a_k)}{\sum_{k=1}^{n} exp(a_i)}
$$

```python
def simple_softmax(a):
    exp_a=np.exp(a)
    sum_exp_a=np.sum(exp_a)
    y = exp_a / sum_exp_a

    return y
```

```python
a=[10,3,2,1,3]
b=simple_softmax(a)
print(b)
```

    [9.97722562e-01 9.09805211e-04 3.34698633e-04 1.23128746e-04
     9.09805211e-04]

```python
np.sum(b)
```

    1.0

#### オーバーフローで正しく計算されない場合
$exp(1000)$は無限大として扱われてうまく計算できない

```python
a = np.array([1010, 1000, 990])
b = simple_softmax(a)
print(b)
```

    [nan nan nan]

    /Users/tomo/.pyenv/versions/anaconda3-2020.11/envs/py37/lib/python3.7/site-packages/ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp

    /Users/tomo/.pyenv/versions/anaconda3-2020.11/envs/py37/lib/python3.7/site-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in true_divide
      after removing the cwd from sys.path.

### 3.5.2 softmax関数のオーバーフロー対策

$$
    y_k=\frac{exp(a_k)}{\sum_{k=1}^{n} exp(a_i)} =
    \frac{C exp(a_k)}{C \sum_{i=1}^n exp(a_i)}\\
    =\frac{exp(a_k + logC)}{\sum_{i=1}^n exp(a_i + logC)}\\
    =\frac{exp(a_k + C')}{\sum_{i=1}^n exp(a_i + C')}
$$

分子と分母に同じ数を掛けても同じ計算をしていることになる。

つまり$exp$内で何かの定数を分子、分母に足し算（もしくは引き算）してsoftmax関数に入力しても値は変わらない。

$C'$としては入力信号の最大値を使うのが一般的。

```python
c = np.max(a)
d = simple_softmax(a-c)
print(d)
```

    [9.99954600e-01 4.53978686e-05 2.06106005e-09]

```python
print(np.sum(d))
```

    1.0

```python
#オーバーフロー対策済softmax関数の定義

def softmax(a):
    c = np.max(a)
    exp_a = np.exp(a-c) #オーバーフロー対策
    sum_exp_a = np.sum(exp_a)
    y = exp_a / sum_exp_a

    return y
```

これで先程と異なりオーバーフローによるエラーがなくなり、正しく計算できるようになった。

```python
d = softmax(a)
print(d)
```

    [9.99954600e-01 4.53978686e-05 2.06106005e-09]

```python
sum(d)
```

    1.0

## 3.6 手書き文字の認識
### 3.6.1 MINISTデータセット

ゼロから作るDeep Learningと異なり、ここではkerasでやってみます。

この部分は教科書でも学習について話しているのではなく、学習済の重みを使って順伝播の流れを見てい来ます。

本質的には、入力の次元と重み行列サイズ、出力次元の関係が理解できればOKです。

隠れ層として全結合層を使用し、活性化関数は恒等関数を、手書き数字データを1次元ベクトルに変換するところは教科書と同じです。

```python
# Tensorflow2ではkerasが組み込まれているのでそちらを使う
import tensorflow as tf
from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint
from tensorflow.keras.layers import Activation, Dense, Input
from tensorflow.keras.models import Model
from tensorflow.keras.utils import to_categorical
from sklearn.model_selection import train_test_split
```

### kerasに含まれるMINISTデータの読み込み

```python
from tensorflow.keras.datasets import mnist
(x_train0, y_train), (x_test, y_test) = mnist.load_data()
```

データの次元の確認です。

テストデータ数は60000で、画像は28x28 pixelsサイズです。

```python
print(x_train0.shape)
```

    (60000, 28, 28)

テストデータの最初＝0番目(pythonは０から始まる)の画像を可視化してみましょう。

5であることがわかります。

ピクセル値は0-255の範囲です。

```python
img = x_train0[0]
plt.imshow(img, cmap='gray')
plt.colorbar()
```

対応するラベルデータy_train[0]が５で有ることを確認します。

```python
y_train[0]
```

    5

以下では、テストデータをテストデータ48000と検証データ12000に手作業で分離します。

学習時にtestデータと検証データを分けることも可能ですが、それはまたいずれ。

```python
(x_train, x_val, y_train, y_val) = train_test_split(x_train0, y_train, test_size=0.2)
```

```python
x_train.shape
```

    (48000, 28, 28)

```python
x_val.shape
```

    (12000, 28, 28)

```python
y_train.shape
```

    (48000,)

```python
y_val.shape
```

    (12000,)

```python
plt.imshow(x_train[1], cmap='gray')
plt.colorbar()
```

```python
y_train[1]
```

    8

```python
plt.imshow(x_val[100], cmap='gray')
```

```python
y_val[100]
```

    6

学習データ(28x28)を1次元ベクトルに変換に変換します。

```python
x_train=x_train.reshape(48000,28*28)
x_train.shape
```

    (48000, 784)

同様に検証データ、テストデータもベクトル化しましょう。

```python
x_val=x_val.reshape(12000,28*28)

x_test=x_test.reshape(10000,28*28)
```

### モデルの定義
入力は748x1次元データです。

隠れそうとして使うのは全結合層２層。<br?
活性化関数は恒等関数を使用します。$y=x$

隠れ層1に50units、隠れ層2に100units、出力層は10unitsとなります。

行列としては

入力層　 隠れ層１　 隠れ層２　出力層  出力

1x784  784x50 　 50x100  100x10 →10

２番目以降、隣り合う数字が一致していれば行列演算が可能です。

```python
def model_functional_api():

    inputs = Input (shape=(28*28, ))
    x = Dense(50, activation ='linear')(inputs)
    x = Dense(100, activation = 'linear')(x)
    predictions = Dense(10, activation ='softmax')(x)

    model = Model(inputs=inputs, outputs = predictions)
    return model
```

### データの前処理
ピクセル値0-255を0-1に規格化し、

ラベルデータはone-hot表現(例えば３は[0, 0, 0, 1, 0,・・・]という表現になる）にする関数preprocessを定義します。

```python
def preprocess(data, label=False):
    if label:
        data = to_categorical(data)
    else:
        data = data.astype('float32') / np.max(data)

    return data
```

画像データの規格化処理です。

```python
X_train = preprocess(x_train)
X_val = preprocess(x_val)
X_test = preprocess(x_test)
```

ラベルのone-hot表現化処理です。

```python
Y_train = preprocess(y_train, label = True)
Y_val = preprocess(y_val, label = True)
Y_test = preprocess(y_test, label = True)
```

one-hot表現になっていることを確認します。

```python
Y_train[1]
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 1., 0.], dtype=float32)

先程定義したモデル関数でモデル=modelを作成し、その構造を確認しましょう。

```python
model = model_functional_api()
model.summary()
```

    Model: "model"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_1 (InputLayer)         [(None, 784)]             0
    _________________________________________________________________
    dense (Dense)                (None, 50)                39250
    _________________________________________________________________
    dense_1 (Dense)              (None, 100)               5100
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                1010
    =================================================================
    Total params: 45,360
    Trainable params: 45,360
    Non-trainable params: 0
    _________________________________________________________________

この部分は最適化関数、ロス関数などの設定です。

詳細はは4章以降ででてくるので後ほど。

```python
model.compile(optimizer ='rmsprop',
             loss='categorical_crossentropy',
             metrics = ['accuracy'])
```

## 3.6.3 バッチ処理の部分
バッチサイズ＝100にしているので先程の重み行列の計算は

100x784  784x50 50x100 100x10 →100x10

となります。

```python
history=model.fit(X_train, Y_train,
                  epochs=5,
                  batch_size=100,
                  validation_data = (X_val, Y_val))
```

    Train on 48000 samples, validate on 12000 samples
    Epoch 1/5
    48000/48000 [==============================] - 3s 67us/sample - loss: 0.4068 - accuracy: 0.8842 - val_loss: 0.3277 - val_accuracy: 0.9097
    Epoch 2/5
    48000/48000 [==============================] - 2s 39us/sample - loss: 0.3045 - accuracy: 0.9133 - val_loss: 0.3001 - val_accuracy: 0.9194
    Epoch 3/5
    48000/48000 [==============================] - 2s 38us/sample - loss: 0.2915 - accuracy: 0.9173 - val_loss: 0.2946 - val_accuracy: 0.9206
    Epoch 4/5
    48000/48000 [==============================] - 2s 40us/sample - loss: 0.2836 - accuracy: 0.9195 - val_loss: 0.3098 - val_accuracy: 0.9168
    Epoch 5/5
    48000/48000 [==============================] - 2s 41us/sample - loss: 0.2791 - accuracy: 0.9227 - val_loss: 0.2953 - val_accuracy: 0.9206

```python
classes = model.predict(X_test, batch_size = 128)
```

```python
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

epochs = range(1, len(acc)+1)
plt.plot(epochs, acc, 'b', label='Training accuracy')
plt.plot(epochs, val_acc, 'r', label='Val accuracy')
plt.legend()
```

```python
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc)+1)
plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'r', label='Val accuracy')
plt.legend()
```

```python
classes.astype
```

    <function ndarray.astype>

```python
classes.shape
```

    (10000, 10)

```python
plt.imshow(classes[:30,:10])
plt.colorbar()
```

お疲れさまでした。４章に続きます。


Clone the tutorial repo

```python
import world
import utils
from world import cprint
import register
import torch
import numpy as np
from tensorboardX import SummaryWriter
import time
import Procedure
from os.path import join
# ==============================
utils.set_seed(world.seed)
print(">>SEED:", world.seed)
# ==============================

from register import dataset
from torch import nn

from dataloader import BasicDataset
import dataloader
# import odeblock as ode
```

    ██╗  ████████╗    ██████╗  ██████╗███████╗
    ██║  ╚══██╔══╝   ██╔═══██╗██╔════╝██╔════╝
    ██║     ██║█████╗██║   ██║██║     █████╗
    ██║     ██║╚════╝██║   ██║██║     ██╔══╝
    ███████╗██║      ╚██████╔╝╚██████╗██║
    ╚══════╝╚═╝       ╚═════╝  ╚═════╝╚═╝

    Current cuda device  0
    loading [../data/gowalla]
    810128 interactions for training
    217242 interactions for testing
    gowalla Sparsity : 0.0008396216228570436
    gowalla is ready to go
    ===========config================
    {'A_n_fold': 100,
     'A_split': False,
     'K': 4,
     'bigdata': False,
     'bpr_batch_size': 2048,
     'decay': 0.0001,
     'dropout': 0,
     'dual_res': False,
     'keep_prob': 0.6,
     'latent_dim_rec': 64,
     'learnable_time': False,
     'lightGCN_n_layers': 3,
     'lr': 0.001,
     'lr_time': 0.0001,
     'multicore': 0,
     'pretrain': 0,
     'pretrained_file_name': 'ltocf',
     'solver': 'euler',
     'test_u_batch_size': 100,
     'time_split': 4}
    cores for test: 32
    comment: lt-ncf
    tensorboard: 1
    LOAD: 0
    Weight path: ./checkpoints
    Test Topks: [20]
    using bpr loss
    adjoint method: False
    ===========end===================
    >>SEED: 2020

```python
if world.adjoint:
    from torchdiffeq import odeint_adjoint as odeint
else:
    from torchdiffeq import odeint

class ODEFunction(nn.Module):
    """
       ## linear GCN (non-time-dependent) in ODE function
    """
    def __init__(self, Graph):
        super(ODEFunction, self).__init__()
        self.g = Graph

    def forward(self, t, x):
        """
        ## linear GCN (non-time-dependent) in ODE function

        \begin{align}
        $ \boldsymbol{E}_{k} = \boldsymbol{A}\boldsymbol{E}_{k-1} $
        \end{align}
        """
        out = torch.sparse.mm(self.g, x)
        return out

class ODEBlock(nn.Module):
    def __init__(self, odeFunction, solver, init_time, final_time):
        super(ODEBlock, self).__init__()
        self.odefunc = odeFunction
        self.integration_time = torch.tensor([init_time,final_time]).float()
        self.solver = solver

    def forward(self, x):
        self.integration_time = self.integration_time.type_as(x)
        out = odeint(func=self.odefunc, y0=x, t=self.integration_time, method=self.solver)
        return out[1]

def ODETimeSplitter(num_split, K):
        eta = K / num_split
        return [i*eta for i in range(1, num_split)]

def ODETimeSetter(num_split, K):
        eta = K/ num_split
        return [torch.tensor([i*eta], dtype=torch.float32, requires_grad=True, device='cuda') for i in range(1, num_split)]

```

```python
#FIXED
class LTOCF(nn.Module):
    def __init__(self,
                 config:dict,
                 dataset:BasicDataset):
        super(LTOCF, self).__init__()
        self.config = config
        self.dataset : dataloader.BasicDataset = dataset
        self.__init_weight()
        self.__init_ode()

    def __init_weight(self):
        self.num_users  = self.dataset.n_users
        self.num_items  = self.dataset.m_items
        self.latent_dim = self.config['latent_dim_rec']
        self.n_layers = self.config['lightGCN_n_layers']
        self.keep_prob = self.config['keep_prob']
        self.A_split = self.config['A_split']
        self.embedding_user = torch.nn.Embedding(
            num_embeddings=self.num_users, embedding_dim=self.latent_dim)
        self.embedding_item = torch.nn.Embedding(
            num_embeddings=self.num_items, embedding_dim=self.latent_dim)
        if self.config['pretrain'] == 0:
            nn.init.normal_(self.embedding_user.weight, std=0.1)
            nn.init.normal_(self.embedding_item.weight, std=0.1)
            world.cprint('use NORMAL distribution initilizer')
        else:
            print('use pretarined data')
        self.f = nn.Sigmoid()
        self.Graph = self.dataset.getSparseGraph()
        print(f"lgn is already to go(dropout:{self.config['dropout']})")

    def __init_ode(self):
        self.time_split = self.config['time_split'] # init the number of time split

        self.odetime_splitted = ODETimeSplitter(self.time_split, self.config['K'])
        self.ode_block_1 = ODEBlock(ODEFunction(self.Graph), self.config['solver'], 0, self.odetime_splitted[0])
        self.ode_block_2 = ODEBlock(ODEFunction(self.Graph), self.config['solver'], self.odetime_splitted[0], self.odetime_splitted[1])
        self.ode_block_3 = ODEBlock(ODEFunction(self.Graph), self.config['solver'], self.odetime_splitted[1], self.odetime_splitted[2])
        self.ode_block_4 = ODEBlock(ODEFunction(self.Graph), self.config['solver'], self.odetime_splitted[2], self.config['K'])

    def get_time(self):
        ode_times=list(self.odetime_1)+ list(self.odetime_2)+ list(self.odetime_3)
        return ode_times

    def __dropout_x(self, x, keep_prob):
        size = x.size()
        index = x.indices().t()
        values = x.values()
        random_index = torch.rand(len(values)) + keep_prob
        random_index = random_index.int().bool()
        index = index[random_index]
        values = values[random_index]/keep_prob
        g = torch.sparse.FloatTensor(index.t(), values, size)
        return g

    def __dropout(self, keep_prob):
        if self.A_split:
            graph = []
            for g in self.Graph:
                graph.append(self.__dropout_x(g, keep_prob))
        else:
            graph = self.__dropout_x(self.Graph, keep_prob)
        return graph

    def computer(self):
        """
        propagate methods for LT-NCF
        """
        users_emb = self.embedding_user.weight
        items_emb = self.embedding_item.weight
        all_emb = torch.cat([users_emb, items_emb])
        embs = [all_emb]

        """
        layers
        """
        all_emb_1 = self.ode_block_1(all_emb)
        all_emb_1 = all_emb_1 - all_emb
        embs.append(all_emb_1)

        all_emb_2 = self.ode_block_2(all_emb_1)
        all_emb_2 = all_emb_2 - all_emb_1
        embs.append(all_emb_2)

        all_emb_3 = self.ode_block_3(all_emb_2)
        all_emb_3 = all_emb_3 - all_emb_2
        embs.append(all_emb_3)

        all_emb_4 = self.ode_block_4(all_emb_3)
        all_emb_4 = all_emb_4 - all_emb_3
        embs.append(all_emb_4)

        embs = torch.stack(embs, dim=1)
        light_out = torch.mean(embs, dim=1)

        users, items = torch.split(light_out, [self.num_users, self.num_items])
        return users, items

    def getUsersRating(self, users):
        all_users, all_items = self.computer()
        users_emb = all_users[users.long()]
        items_emb = all_items
        rating = self.f(torch.matmul(users_emb, items_emb.t()))
        return rating

    def getEmbedding(self, users, pos_items, neg_items):
        all_users, all_items = self.computer()
        users_emb = all_users[users]
        pos_emb = all_items[pos_items]
        neg_emb = all_items[neg_items]
        users_emb_ego = self.embedding_user(users)
        pos_emb_ego = self.embedding_item(pos_items)
        neg_emb_ego = self.embedding_item(neg_items)
        return users_emb, pos_emb, neg_emb, users_emb_ego, pos_emb_ego, neg_emb_ego

    def bpr_loss(self, users, pos, neg):
        (users_emb, pos_emb, neg_emb,
        userEmb0,  posEmb0, negEmb0) = self.getEmbedding(users.long(), pos.long(), neg.long())
        reg_loss = (1/2)*(userEmb0.norm(2).pow(2) +
                         posEmb0.norm(2).pow(2)  +
                         negEmb0.norm(2).pow(2))/float(len(users))
        pos_scores = torch.mul(users_emb, pos_emb)
        pos_scores = torch.sum(pos_scores, dim=1)
        neg_scores = torch.mul(users_emb, neg_emb)
        neg_scores = torch.sum(neg_scores, dim=1)

        loss = torch.mean(torch.nn.functional.softplus(neg_scores - pos_scores))

        return loss, reg_loss

    def forward(self, users, items):
        # compute embedding
        all_users, all_items = self.computer()
        users_emb = all_users[users]
        items_emb = all_items[items]
        inner_pro = torch.mul(users_emb, items_emb)
        gamma     = torch.sum(inner_pro, dim=1)
        return gamma
```

```python
if world.dataset in ['gowalla', 'yelp2018', 'amazon-book']:
    dataset = dataloader.Loader(path="../data/"+world.dataset)
```

    loading [../data/gowalla]
    810128 interactions for training
    217242 interactions for testing
    gowalla Sparsity : 0.0008396216228570436
    gowalla is ready to go

```python
# Recmodel = register.MODELS[world.model_name](world.config, dataset)
Recmodel = LTOCF(world.config, dataset)

Recmodel = Recmodel.to(world.device)
if world.config['learnable_time'] == False:
    bpr = utils.BPRLoss(Recmodel, world.config)
elif world.config['learnable_time'] == True:
    bpr_t = utils.BPRLossT(Recmodel, world.config)

weight_file = utils.getFileName()
pretrained_weight_file = utils.getPretrainedFileName(world.config['pretrained_file_name'])
print(f"load and save to {pretrained_weight_file}")
if world.LOAD:
    try:
        Recmodel.load_state_dict(torch.load(pretrained_weight_file,map_location=torch.device('cpu')))
        world.cprint(f"loaded model weights from {pretrained_weight_file}")
    except FileNotFoundError:
        print(f"{pretrained_weight_file} not exists, start from beginning")
Neg_k = 1

save_name = time.strftime("%m-%d-%Hh%Mm-") + "-" + world.dataset + "-" + world.model_name + "-" + world.config['solver'] + "-adjoint_" + str(world.adjoint) + "-learnable_t_" + str(world.config['learnable_time']) + "-dual_res_" + str(world.config['dual_res']) + "-lr" + str(world.config['lr']) + "-lr_time" + str(world.config['lr_time']) + "-decay" + str(world.config['decay'])+ "-" + world.comment

# init tensorboard
if world.tensorboard:
    w : SummaryWriter = SummaryWriter(join(world.BOARD_PATH, save_name))
else:
    w = None
    world.cprint("not enable tensorflowboard")
```

    use NORMAL distribution initilizer
    loading adjacency matrix
    successfully loaded...
    don't split the matrix
    lgn is already to go(dropout:0)
    load and save to /home/bigdyl/jeongwhanchoi/tutorial21/LT-OCF/code/pretrain/ltocf

```python
for epoch in range(world.TRAIN_epochs+1):
    start = time.time()
    if epoch %10 == 0:
        cprint("[TEST]")
        Procedure.Test(dataset, Recmodel, epoch, w, world.config['multicore'])

    if world.model_name == 'ltocf':
        if world.config['learnable_time'] == False:
            output_information = Procedure.BPR_train_original(dataset, Recmodel, bpr, epoch, neg_k=Neg_k,w=w)
            print(f'EPOCH[{epoch+1}/{world.TRAIN_epochs}] {output_information}')
        else:
            output_information, times_list= Procedure.BPR_train_ode(dataset, Recmodel, bpr_t, epoch, neg_k=Neg_k,w=w)
            print(f'EPOCH[{epoch+1}/{world.TRAIN_epochs}] {output_information}')
            print(times_list)
```

```python
!python main.py --dataset="gowalla" --model="ltocf" --solver="rk4" --adjoint=False --learnable_time=False --dual_res=False --K=4 --lr=1e-3 --decay=1e-4 --topks="[20]" --comment="fixed_time" --tensorboard=1 --gpuid=0
```

    ██╗  ████████╗    ██████╗  ██████╗███████╗
    ██║  ╚══██╔══╝   ██╔═══██╗██╔════╝██╔════╝
    ██║     ██║█████╗██║   ██║██║     █████╗
    ██║     ██║╚════╝██║   ██║██║     ██╔══╝
    ███████╗██║      ╚██████╔╝╚██████╗██║
    ╚══════╝╚═╝       ╚═════╝  ╚═════╝╚═╝

    Current cuda device  0
    >>SEED: 2020
    loading [../data/gowalla]
    810128 interactions for training
    217242 interactions for testing
    gowalla Sparsity : 0.0008396216228570436
    gowalla is ready to go
    ===========config================
    {'A_n_fold': 100,
     'A_split': False,
     'K': 4.0,
     'bigdata': False,
     'bpr_batch_size': 2048,
     'decay': 0.0001,
     'dropout': 0,
     'dual_res': False,
     'keep_prob': 0.6,
     'latent_dim_rec': 64,
     'learnable_time': False,
     'lightGCN_n_layers': 3,
     'lr': 0.001,
     'lr_time': 0.0001,
     'multicore': 0,
     'pretrain': 0,
     'pretrained_file_name': 'ltocf',
     'solver': 'rk4',
     'test_u_batch_size': 100,
     'time_split': 4}
    cores for test: 32
    comment: fixed_time
    tensorboard: 1
    LOAD: 0
    Weight path: ./checkpoints
    Test Topks: [20]
    using bpr loss
    adjoint method: False
    ===========end===================
    use NORMAL distribution initilizer
    loading adjacency matrix
    successfully loaded...
    don't split the matrix
    lgn is already to go(dropout:0)
    load and save to /home/bigdyl/jeongwhanchoi/tutorial21/LT-OCF/code/pretrain/ltocf
    [TEST]
    Inference time: 11.9485s
    {'precision': array([0.00398218]), 'recall': array([0.01336332]), 'ndcg': array([0.01033115])}
    ^C
    Traceback (most recent call last):
      File "/home/bigdyl/jeongwhanchoi/tutorial21/LT-OCF/code/main.py", line 56, in <module>
        output_information = Procedure.BPR_train_original(dataset, Recmodel, bpr, epoch, neg_k=Neg_k,w=w)
      File "/home/bigdyl/jeongwhanchoi/tutorial21/LT-OCF/code/Procedure.py", line 41, in BPR_train_original
        cri = bpr.stageOne(batch_users, batch_pos, batch_neg)
      File "/home/bigdyl/jeongwhanchoi/tutorial21/LT-OCF/code/utils.py", line 40, in stageOne
        loss.backward()
      File "/home/bigdyl/anaconda3/envs/tutorial/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward
        torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
      File "/home/bigdyl/anaconda3/envs/tutorial/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
        Variable._execution_engine.run_backward(
    KeyboardInterrupt

```python

```


```python
#This notebook also uses the `(some) LaTeX environments for Jupyter`
#https://github.com/ProfFan/latex_envs wich is part of the
#jupyter_contrib_nbextensions package

from myhdl import *
from myhdlpeek import Peeker
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from sympy import *
init_printing()

import itertools

#EE drawing tools in python from https://cdelker.bitbucket.io/SchemDraw/
import SchemDraw as schem
import SchemDraw.elements as e
import SchemDraw.logic as l

#https://github.com/jrjohansson/version_information
%load_ext version_information
%version_information myhdl, myhdlpeek, numpy, pandas, matplotlib, sympy, itertools, SchemDraw
```

Software	Version

Python	3.6.5 64bit [GCC 7.2.0]

IPython	6.4.0

OS	Linux 4.15.0 33 generic x86_64 with debian buster sid

myhdl	0.10

myhdlpeek	0.0.7

numpy	1.14.3

pandas	0.23.0

matplotlib	2.2.2

sympy	1.4.dev

itertools	The 'itertools' distribution was not found and is required by the application

SchemDraw	0.3.0

Mon Sep 17 12:11:24 2018 MDT

```python
z, r, DisAng=symbols('z, r, Omega')
zFunc=Eq(z, r*exp(1j*DisAng)); zFunc
```

```python
zFuncN=lambdify((r, DisAng), zFunc.rhs, dummify=False)

zr=np.arange(-1.5, 1.5+.03, .03); zi=np.copy(zr)
zR, zI=np.meshgrid(zr, zi)
zN=zR+1j*zI

rN=1.0
AngThetaN=np.arange(0, 1+.005, .005)*2*np.pi
zAtR1=zFuncN(rN, AngThetaN)
```

```python
%matplotlib notebook

def Zplot(zR, zI, HzNMag, HzAtR1NMag, HzNPhase, HzAtR1NPhase, title):

    fig = plt.figure()

    #plot the z space mag
    axZmag = fig.add_subplot(221, projection='3d')
    Mags=axZmag.plot_surface(zR, zI, HzNMag, cmap=plt.get_cmap('tab20'))
    axZmag.plot(np.real(zAtR1), np.imag(zAtR1), HzAtR1NMag, 'r-', label='r=1')
    axZmag.set_xlabel('Re'); axZmag.set_ylabel('Im'); axZmag.set_zlabel('Mag')
    axZmag.legend(loc='best')
    fig.colorbar(Mags)

    #plot the z space phase
    axZph = fig.add_subplot(222, projection='3d')
    Phase=axZph.plot_surface(zR, zI, HzNPhase, cmap=plt.get_cmap('tab20'))
    axZph.plot(np.real(zAtR1), np.imag(zAtR1), HzAtR1NPhase, 'r-', label='r=1')
    axZph.set_xlabel('Re'); axZph.set_ylabel('Im'); axZph.set_zlabel('Phase')
    axZph.legend(loc='best')
    fig.colorbar(Phase)

    axBodeM=fig.add_subplot(212)
    Mline=axBodeM.plot(AngThetaN, HzAtR1NMag, label='FTMag')
    axBodeM.set_ylabel('Mag')

    axBodeP=axBodeM.twinx()
    Pline=axBodeP.plot(AngThetaN, np.rad2deg(HzAtR1NPhase), 'g--', label='FTPhase')
    axBodeP.set_ylabel('Phase [deg]')
    axBodeP.set_xlabel('Ang')

    lns = Mline+Pline
    labs = [l.get_label() for l in lns]
    axBodeP.legend(lns, labs, loc='best')

    fig.suptitle(title)
    fig.show()

```

```python
N=symbols('N')
DelayH=z**(-N); DelayH
```

```python
DelaySupH=simplify(DelayH.subs(zFunc.lhs, zFunc.rhs)); DelaySupH
```

```python
DelaySupH=simplify(DelaySupH.subs(r, 1)); DelaySupH
```

```python
DelayHN=lambdify((z, N), DelayH, dummify=False)
Nvalue=2
HzN=DelayHN(zN, N=Nvalue); HzN.shape
HzNMag=np.abs(HzN); HzNPhase=np.angle(HzN)

HAtR1=zFuncN(rN, AngThetaN)
HzAtR1N=DelayHN(zAtR1, N=Nvalue)
HzAtR1NMag=np.abs(HzAtR1N); HzAtR1NPhase=np.angle(HzAtR1N)
```

```python
def DelayExplorer(N=1):
    Nvalue=N

    HzN=DelayHN(zN, N=Nvalue); HzN.shape
    HzNMag=np.abs(HzN); HzNPhase=np.angle(HzN)

    HAtR1=zFuncN(rN, AngThetaN)
    HzAtR1N=DelayHN(zAtR1, N=Nvalue)
    HzAtR1NMag=np.abs(HzAtR1N); HzAtR1NPhase=np.angle(HzAtR1N)
    Zplot(zR, zI, HzNMag, HzAtR1NMag, HzNPhase, HzAtR1NPhase,
          f'z Delay order N={N}')
```

```python
# will add widgets for interative later
DelayExplorer(N=1)
```

```python
@block
def Delay(x, y, read, clk):
    '''
        Z delay bulding block for a CIC filter

        Inputs:
            x (bitVec): the x(n) data in feed
            ------------------------
            read (bool): the exstiror calc hold input. calc is done only if
            `read` is True

            clk(bool): clock feed
            rst(bool): reset feed

        Outputs:
            y (data):  the y(n+1) output of y(n+1)=x(n)

    '''
    @always(clk. posedge)
    def logic():
        if read:
            y.next=x
    return logic
```

```python
Peeker.clear()
x=Signal(modbv(0)[BitWidth:]); Peeker(x, 'x')
y=Signal(modbv(0)[BitWidth:]); Peeker(y, 'y')

read, clk=[Signal(bool(0)) for  _ in range(2)]
Peeker(ena_in, 'ena_in'); Peeker(clk, 'clk')

DUT=Delay(x, y, read, clk)

DateCol=pd.DataFrame(columns=['x', 'y', 'ena_in'])

def Delay_TB():

    @always(delay(1))  ## delay in nano seconds
    def clkGen():
        clk.next = not clk

    @instance
    def stimulus():
        Tested_read=False
        count=0
        while 1:

            if Tested_read==False and count<=2:
                print(f'Tested_read: {Tested_ena}, count:{count}')
            elif Tested_read==False and count>2:
                print(f'Tested_read: {Tested_ena}, count:{count}')

                ena_in.next=True
                Tested_ena=True
                x.next=0

            if Tested_ena  and count>2:
                x.next=x+1

            if count> 2*BitWidth:
                raise StopSimulation

            DateCol.loc[count]=[int(x),int(y), int(ena_in)]

            count+=1
            yield clk.posedge

    return instances()
sim = Simulation(DUT, Delay_TB(), *Peeker.instances()).run()
```


#**Chapter 4 – Training Models**

     Open the original notebook

Let's begin like in the last notebook: importing a few common modules, ensuring MatplotLib plots figures inline and preparing a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so once again we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20.

You don't need to worry about understanding everything that is written in this section.

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Is this notebook running on Colab or Kaggle?
IS_COLAB = "google.colab" in sys.modules

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

# Common imports
import numpy as np
import os

# To make this notebook's output stable across runs
rnd_seed = 42
rnd_gen = np.random.default_rng(rnd_seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "classification"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)
```

In this notebook we will be working with the [*Iris Flower Dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set), in which the length and width of both the sepals and petals of three types of Iris flowes were recorded. For reference, these are pictures of the three flowers:

 In order: Iris Setosa,  Iris Versicolor, and Iris Virginica

Photo Credits:[Kosaciec szczecinkowaty Iris setosa](https://en.wikipedia.org/wiki/File:Kosaciec_szczecinkowaty_Iris_setosa.jpg) by [Radomil Binek](https://commons.wikimedia.org/wiki/User:Radomil) licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [Blue flag flower close-up (Iris versicolor)](https://en.wikipedia.org/wiki/File:Iris_versicolor_3.jpg)by Danielle Langlois licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [image of Iris virginica shrevei](https://en.wikipedia.org/wiki/File:Iris_virginica.jpg) by [Frank Mayfield](https://www.flickr.com/photos/33397993@N05) licensed under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/deed.en).

As you can imagine, this dataset is normally used to train *multiclass*/*multinomial* classification algorithms and not *binary* classification algorithms, since there *are* more than 2 classes.

"*Three classes, even!*" - an observant TA

For this exercise, however, we will be implemented the binary classification algorithm referred to as the *logistic regression* algorithm (also called logit regression).

```python
# Let's load the Iris Dataset
from sklearn import datasets
iris = datasets.load_iris()

# Print out some information about the data
print(f'Keys in Iris dictionary: \n{list(iris.keys())}\n\n')
print(iris.DESCR)

# And load the petal lengths and widths as our input data
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

print(iris['data_module'])

# The target data labels Setosa as 0, Versicolor as 1, and Virginica as 2. For
# this exercise we will be using only the Versicolor and Virgina sets.
bin_indices = np.logical_or(y==1,y==2)
bin_X = X[bin_indices]
bin_y = (y[bin_indices]==2).astype(np.uint8) # convert to binary
```

    Keys in Iris dictionary:
    ['data', 'target', 'frame', 'target_names', 'DESCR', 'feature_names', 'filename', 'data_module']

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...
    sklearn.datasets.data

We now have a set of binary classification data we can use to train an algorithm.

As we saw during our reading, we need to define three things in order to train our algorithm: the type of algorithm we will train, the cost function (which will tell us how close our prediction is to the truth), and a method for updating the parameters in our model according to the value of the cost function (e.g., the gradient descent method).

Let's begin by defining the type of algorithm we will use. We will train a logistic regression model to differentiate between two classes. A reminder of how the logistic regression algorithm works is given below.

The logistic regression algorithm will thus take an input $t$ that is a linear combination of the features:

 $t_{\small{n}} = \beta_{\small{0}} + \beta_{\small{1}} \cdot X_{1,n} + \beta_{\small{2}} \cdot X_{2,n}$

where
* $n$ is the ID of the sample
* $X_{\small{1}}$ represents the petal length
* $X_{\small{1}}$ represents the petal width

This input is then fed into the logistic function, $\sigma$:
\begin{align}
\sigma: t\mapsto \dfrac{1}{1+e^ {-t}}
\end{align}

Let's plot it below to remember the shape of the function

```python
t = np.arange(-4,4,.1)
def logistic(in_val):
    # Return the value of the logistic function
    return 1/(1 + np.exp(- in_val))

fig, ax = plt.subplots()

ax.axvline(0, c='black', alpha=1)
ax.axhline(0, c='black', alpha=1)

[ax.axhline(y_val, c='black', alpha=0.5, linestyle='dotted') for y_val in (0.5,1)]

plt.autoscale(axis='x', tight=True)

ax.plot(t, logistic(t));
ax.set_xlabel('$t$')
ax.set_ylabel('$\\sigma\\  \\left(t\\right)$')
fig.tight_layout()
```

With the logistic function, we define inputs resulting in $\sigma\geq.5$ as belonging to the ***one*** class, and any value below that is considered to belong to the ***zero*** class.

We now have a function which lets us map the value of the petal length and width to the class to which the observation belongs (i.e., whether the length and width correspond to Iris Versicolor or Iris Virginica). However, there is a parameter vector **$\theta$** with a number of parameters that we do not have a value for:
 $\theta = [ \beta_{\small{0}}, \beta_{\small{1}}$, $\beta_{\small{2}} ]$

**Q1) Set up an array of random numbers between 0 and 1 representing the $\theta$ vector.**

Hint:  Use `rnd_gen`! If you're not sure how to use it, consult the `default_rng` documentation [at this link](https://numpy.org/doc/stable/reference/random/generator.html). For instance, you may use the `random` method of `rnd_gen`.

```python
import random
vec_0 = np.random.uniform(0, 1, 3)
theta = vec_0
theta
```

    array([0.96169723, 0.20190635, 0.95737101])

In order to determine whether a set of $\beta$ values is better than the other, we need to quantify well the values are able to predict the class. This is where the cost function comes in.

The cost function, $c$, will return a value close to zero when the prediction, $\hat{p}$, is correct and a large value when it is wrong. In a binary classification problem, we can use the log loss function. For a single prediction and truth value, it is given by:
\begin{align}
        \text{c}(\hat{p},y) = \left\{
        \begin{array}{cl}
        -\log(\hat{p})& \text{if}\; y=1\\
        -\log(1-\hat{p}) & \text{if}\; y=0
        \end{array}
        \right.
    \end{align}

However, we want to apply the cost function to an n-dimensional set of predictions and truth values. Thankfully, we can find the average value of the log loss function $J$ for an an-dimensional set of $\hat{y}$ & $y$ as follows:

\begin{align}
        \text{J}(\mathbf{\hat{p}},y) = - \dfrac{1}{n} \sum_{i=1}^{n}
        \left[ y_i\cdot \log\left( \hat{p}_i \right) \right] +
        \left[ \left( 1 - y_i \right) \cdot \log\left( 1-\hat{p}_i \right) \right]
    \end{align}

We now have a formula that can be used to calculate the average cost over the training set of data.

**Q2) Define a log_loss function that takes in an arbitrarily large set of prediction and truths**

Hint 1: You need to encode the function $J$ above, for which Numpy's functions may be quite convenient (e.g., [`log`](https://numpy.org/doc/stable/reference/generated/numpy.log.html), [`mean`](https://numpy.org/doc/stable/reference/generated/numpy.mean.html), etc.)

Hint 2: Asserting the dimensions of the vector is a good way to check that your function is working correctly. [Here's a tutorial on how to use `assert`](https://swcarpentry.github.io/python-novice-inflammation/10-defensive/index.html#assertions). For instance, to assert that two vectors `X` and `Y` have the same dimension, you may use:
```
assert X.shape==Y.shape
```

Now let's code 💻

```python
def log_loss(p_hat, y, epsilon=1e-7):
  # Write your code here.
  # We can also run into problems if p_hat = 0, so add an _epsilon_ term
  # when evaluating log(p_hat).
  log_p_hat = np.log(epsilon + p_hat)

  J = ((y*log_p_hat) + ((1-y)*np.log(1-p_hat)) )
 #J = -1/len(y)*( sum( (y*log_p_hat) + ((1-y)*np.log(1-p_hat))) )

  # -1/n * np.sum((y * log_p_hat) + ((1 - y) * np.log(1 - log_p_hat)))

  # After calculating J, assert that J has the same shape as p_hat and y
  assert J.shape == y.shape
  assert J.shape == p_hat.shape

  return J
```

We now have a way of quantifying how good our predictions are. The final thing needed for us to train our algorithm is figuring out a way to update the parameters in a way that improves the average quality of our predictions.

**Warning**: we'll go into a bit of math below

Let's look at the change in a single parameter within $\theta$: $\beta_1$ (given $X_{1,i} = X_1$, $\;\hat{p}_{i} = \hat{p}$, $\;y_{i} = y$). If we want to know what the effect of changing the value of $\beta_1$ will have on the log loss function we can find this with the partial derivative:
$
        \dfrac{\partial J}{\partial \beta_1}
$

This may not seem very helpful by itself - after all, $\beta_1$ isn't even in the expression of $J$. But if we use the chain rule, we can rewrite the expression as:

        $\dfrac{\partial J}{\partial \hat{p}} \cdot
        \dfrac{\partial \hat{p}}{\partial \theta} \cdot
        \dfrac{\partial \theta}{\partial \beta_1}$

We'll spare you the math (feel free to verify it youself, however!):

$\dfrac{\partial J}{\partial \hat{p}} =  \dfrac{\hat{p} - y}{\hat{p}(1-\hat{p})}, \quad
        \dfrac{\partial \hat{p}}{\partial \theta} = \hat{p} (1-\hat{p}), \quad
        \dfrac{\partial \theta}{\partial \beta_1} = X_1 $

and thus
$
        \dfrac{\partial J}{\partial \beta_1} = (\hat{p} - y) \cdot X_1
$

We can calculate the partial derivative for each parameter in $\theta$ which, as you may have realized, is simply the $\theta$ gradient of $J$: $\nabla_{\theta}(J)$

With all of this information, we can now write $\nabla_{\theta} J$ in terms of the error, the feature vector, and the number of samples we're training on!

$\nabla_{\mathbf{\theta}^{(k)}} \, J(\mathbf{\theta^{(k)}}) = \dfrac{1}{n} \sum\limits_{i=1}^{n}{ \left ( \hat{p}^{(k)}_{i} - y_{i} \right ) \mathbf{X}_{i}}$

Note that here $k$ represents the iteration of the parameters we are currently on.

We now have a gradient we can calculate and use in the batch gradient descent method! The updated parameters will thus be:

\begin{align}
{\mathbf{\theta}^{(k+1)}} = {\mathbf{\theta}^{(k)}} - \eta\,\nabla_{\theta^{(k)}}J(\theta^{(k)})
\end{align}

Where $\eta$ is the learning rate parameter. It's also worth pointing out that $\;\hat{p}^{(k)}_i = \sigma\left(\theta^{(k)}, X_i\right) $

In order to easily calculate the input to the logistic regression, we'll multiply the $\theta$ vector with the X data, and as we have a non-zero bias  $\beta_0$ we'd like to have an X matrix whose first column is filled with ones.

\begin{align}
    X_{\small{with\ bias}} = \begin{pmatrix}
        1 & X_{1,0} & X_{2,0}\\
        1 & X_{1,1} & X_{2,1}\\
        &...&\\
        1 & X_{1,n} & X_{2,n}
        \end{pmatrix}
\end{align}

**Q3) Prepare the `X_with_bias` matrix (remember to use the `bin_X` data and not just `X`). Write a function called `predict` that takes in the parameter vector $\theta$ and the `X_with_bias` matrix and evaluates the logistic function for each of the samples.**

Hint 1: You recently learned how to initialize arrays in the `Numpy` notebook [at this link](https://nbviewer.org/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/S1_2_Numpy.ipynb). There are many ways to add a columns of 1 to `bin_X`, for instance using [`np.concatenate`](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html) or [`np.append`](https://numpy.org/doc/stable/reference/generated/numpy.append.html).

Hint 2:  To clarify, the function `predict` calculates $\hat{p}$ from $\beta$ and $\boldsymbol{X}$.

Hint 3: In practice, to calculate the logistic function for each sample, you may follow the equations [higher up in the notebook](#logit) and (1) calculate $t$ from $\beta$ and $\boldsymbol{X_{\mathrm{with\ bias}}}$ before (2) applying the logistic function $\sigma$ to $t$.

```python
# Prepare the X_with_bias matrix

#X_with_bias = np.c_[np.ones([len(X), 1]), X]
```

```python
ones = np.ones(bin_X.shape[0])
ones.shape = ones[:, None].shape
X_with_bias = np.concatenate((ones, bin_X), axis=1)
```

```python
# Write your function predict here
def predict(tetha,X_with_bias):
    t0 = tetha[0]*X_with_bias[:,0] + tetha[1]*X_with_bias[:,1] + tetha[2]*X_with_bias[:,2]
    s = 1/ (1+np.exp(-t0))
    return s
```

**Q4) Now that you have a `predict` function, write a `gradient_calc` function that calculates the gradient for the logistic function.**

Hint 1: You'll have to feed `theta`, `X`, and `y` to the `gradient_calc` function.

Hint 2: You can use [this equation](#grad_eq) to calculate the gradient of the cost function.

```python
# Write your code here

def grad_calc(theta, p_hat,X, y): # len(x) = N
    return np.mean((p_hat-y)*X.T)
#((1/len(X))*np.sum((p_hat - y)*X.T))

```

We can now write a function that will train a logistic regression algorithm!

Your `logistic_regression` function needs to:
* Take in a set of training input/output data, validation input/output data, a number of iterations to train for, a set of initial parameters $\theta$, and a learning rate $\eta$
* At each iteration:
 * Generate a set of predictions on the training data. Hint: You may use your function `predict` on inputs `X_train` from the training set.
 * Calculate and store the loss function for the training data at each iteration. Hint: You may use your function `log_loss` on inputs `X_train` and outputs `y_train` from the training set.
 * Calculate the gradient. Hint: You may use your function `grad_calc`.
 * Update the $\theta$ parameters. Hint: You need to implement [this equation](#grad_descent).
 * Generate a set of predictions on the validation data using the updated parameters. Hint: You may use your function `predict` on inputs `X_valid` from the validation set.
 * Calculate and store the loss function for the validation data. Hint: You may use your function `log_loss` on inputs `X_valid` and outputs `y_valid` from the validation set.
 * Bonus: Calculate and store the accuracy of the model on the training and validation data as a metric!
* Return the final set of parameters $\theta$ & the stored training/validation loss function values (and the accuracy, if you did the bonus)

**Q5) Write the `logistic_regression` function**

```python
# Write your code here
# Write your code here
from logging import logProcesses
def logistic_regression(X_train, y_train,X_valid, y_valid, iters, theta, eta):

    losses_train = []
    losses_valid = []

    for i in range(iters):
        #generate predictions
        p_hat_pred = predict(theta,X_train)
        # Gradient and theta
        gd = grad_calc(theta,p_hat_pred,X_train,y_train)
        theta = theta - eta*gd
        # Predictions from validation set
        y_pred = predict(theta, X_valid)
        # Validation set loss function
        Loss_v = -np.mean(log_loss(y_pred, y_valid, epsilon=1e-7))
        losses_valid.append(Loss_v)
        # Test set loss function
        loss = -np.mean(log_loss(p_hat = p_hat_pred, y = y_train, epsilon=1e-7))
        losses_train.append(loss)
    losses = [losses_train,losses_valid]
    return losses, theta

```

**¡¡¡Important Note!!!**

The notebook assumes that you will return
1. a Losses list, where Losses[0] is the training loss and Losses[1] is the validation loss
2. a tuple with the 3 final coefficients ($\beta_0$, $\beta_1$, $\beta_2$)

The code for visualizing the bonus accuracy is not included - but it should be simple enough to do in a way similar to that which is done with the losses.

---------------------

Now that we have our logistic regression function, we're all set to train our algorithm! Or are we?

There's an important data step that we've neglected up to this point - we need to split the data into the train, validation, and test datasets.

```python
test_ratio = 0.2
validation_ratio = 0.2
total_size = len(X_with_bias)

test_size = int(total_size * test_ratio)
validation_size = int(total_size * validation_ratio)
train_size = total_size - test_size - validation_size

rnd_indices = rnd_gen.permutation(total_size)

X_train = X_with_bias[rnd_indices[:train_size]]
y_train = bin_y[rnd_indices[:train_size]]
X_valid = X_with_bias[rnd_indices[train_size:-test_size]]
y_valid = bin_y[rnd_indices[train_size:-test_size]]
X_test = X_with_bias[rnd_indices[-test_size:]]
y_test = bin_y[rnd_indices[-test_size:]]
```

Now we're ready!

**Q6) Train your logistic regression algorithm. Use 5000 iterations, $\eta$=0.1**

Hint: It's time to use the `logistic_regression` function you defined in Q5.

```python
# Complete the code
losses, coeffs = logistic_regression(X_train, y_train, X_valid, y_valid, 50000, theta, 0.1)
```

Let's see how our model did while learning!

```python
# Produce the Loss Function Visualization Graphs
fig, ax = plt.subplots(figsize=(18,8))
ax.plot(losses[0], color='blue', label='Training', linewidth=3);
ax.plot(losses[1], color='orange', label='Validation', linewidth=3);
ax.legend();
ax.set_ylabel('Log Loss')
ax.set_xlabel('Iterations')
ax.set_title('Loss Function Graph')
ax.autoscale(axis='x', tight=True)
fig.tight_layout();

# Let's get predictions from our model for the training, validation, and testing
# datasets
y_hat_train = (predict(X_train, coeffs)>=.5).astype(int)
y_hat_valid = (predict(X_valid, coeffs)>=.5).astype(int)
y_hat_test = (predict(X_test, coeffs)>=.5).astype(int)

y_sets = [ [y_hat_train, y_train],
           [y_hat_valid, y_valid],
           [y_hat_test, y_test] ]

def accuracy_score(y_hat, y):
    assert(y_hat.size==y.size)
    return (y_hat == y).sum()/y.size

[accuracies.append(accuracy_score(y_set[0],y_set[1])) for y_set in y_sets]

printout= (f'Training Accuracy:{accuracies[0]:.1%} \n'
           f'Validation Accuracy:{accuracies[1]:.1%} \n')

# Add the testing accuracy only once you're sure that your model works!

print(printout)
```

Congratulations on training a logistic regression algorithm from scratch! Once you're done with the upcoming environmental science applications notebook, feel free to come back to take a look at the challenges 😀

## Challenges

* **C1)** Add L2 Regularization to training function

* **C2)** Add early stopping to the training algorithm! Stop training when the accuracy is >=90%

* **C3)** Implement a softmax regression model (It's multiclass logistic regression 🙂)


# Getting Started with SYMPAIS
[](https://colab.research.google.com/github/ethanluoyc/sympais/blob/master/notebooks/getting_started.ipynb)

## Setup

```python
try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False
```

### Install SYMPAIS

```python
# (TODO(yl): Simplify when we make this public)
GIT_TOKEN = ""
if IN_COLAB:
    !pip install -U pip setuptools wheel
    if GIT_TOKEN:
        !pip install git+https://{GIT_TOKEN}@github.com/ethanluoyc/sympais.git#egg=sympais
    else:
        !pip install git+https://github.com/ethanluoyc/sympais.git#egg=sympais
```

### Download and install pre-built RealPaver v0.4

```python
if IN_COLAB:
    !curl -L "https://drive.google.com/uc?export=download&id=1_Im0Ot5TjkzaWfid657AV_gyMpnPuVRa" -o realpaver
    !chmod u+x realpaver
    !cp realpaver /usr/local/bin
```

```python
import jax
import jax.numpy as jnp

from sympais import tasks
from sympais import methods
from sympais.methods import run_sympais, run_dmc
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib
import numpy as onp
import math

%load_ext autoreload
%autoreload 2
%matplotlib inline
```

## Load a task

```python
task = tasks.Sphere(nd=3)
```

```python
task.profile
```

```python
task.constraints
```

```python
task.domains
```

## Run DMC baseline

```python
dmc_output = run_dmc(task, seed=0, num_samples=int(1e8))
print(dmc_output)
```

## Run SYMPAIS

```python
sympais_output = run_sympais(
    task,
    key=jax.random.PRNGKey(0),
    num_samples=int(1e6),
    num_proposals=100,
    tune=False,
    init='realpaver',
    num_warmup_steps=500,
    window_size=100
)
print(sympais_output)
```

## Create your own problem

In this section, we will show how to implement a new probabilistic analysis
task similar to the sphere task above.

A probabilistic ananlysis `Task` consists of an input `Profile` $p(\mathbf{x})$ and a list of constraints `cs`. A user create a new `Task` either by calling the super class constructor or subclassing the base class.

Consider a two-dimensional problems where we would like to know the probablity that the
the inputs $x \in [-10, 10]$ and $y \in [-10, 10]$ are jointly in the interior of a two-dimensional _cube_. The set of constraints is
$$
\begin{align}
    x + y &\leq 1.0, \\
    x + y &\geq -1.0, \\
    y - x &\geq -1.0, \\
    y - x &\leq 1.0.
\end{align}
$$

First, let's import the related modules used for defining the tasks

```python
import sympy
from sympais import tasks
from sympais import profiles
from sympais import distributions as dist
```

### Independent profile

We will first show how to define a task when the input variables are _independent_.
We use `Profile` for defining the input distribution and SymPy expressions for defining the constraints.

The `Profile` uses the following iterface. To create a customized profile, the user needs to implement
`profile.log_prob` and `profile.sample` functions. Note that unlike numpyro distributions, the samples
are represented as a dictionary from variable names to their values. This is so that it is easier to integrate
with a symbolic execution engine.

```python
help(profiles.Profile)
```

When the input random variables are independent, we provide a convenience `IndependentProfile` class which allows you to specify the per-component distribution. `IndependentProfile` implements `sample` and `log_prob` by dispatching to the individual components and then aggretating the results.

We are now ready to define a task for the `cube` problem. The code is shown below.

```python
class IndependentCubeTask(tasks.Task):
    def __init__(self):
        profile = profiles.IndependentProfile({
            "x": dist.Normal(loc=-2, scale=1),
            "y": dist.Normal(loc=-2, scale=1)
        })
        domains = {"x": (-10., 10.), "y": (-10., 10.)}
        b = 1.0
        x = sympy.Symbol("x")
        y = sympy.Symbol("y")
        c1 = x + y <= b   # type: sympy.Expr
        c2 = x + y >= -b  # type: sympy.Expr
        c3 = y - x >= -b  # type: sympy.Expr
        c4 = y - x <= b  # type: sympy.Expr
        super().__init__(profile, [c1, c2, c3, c4], domains)
```

Let us create some helper functions for visualizing the profile and the constraints.

```python
b = 1.

def f1(x):
    return b - x
def f2(x):
    return -b - x
def f3(x):
    return -b + x
def f4(x):
    return b + x

x = sympy.Symbol('x')
x1, = sympy.solve(f1(x)-f3(x))
x2, = sympy.solve(f1(x)-f4(x))
x3, = sympy.solve(f2(x)-f3(x))
x4, = sympy.solve(f2(x)-f4(x))

y1 = f1(x1)
y2 = f1(x2)
y3 = f2(x3)
y4 = f2(x4)

N = 200
X, Y = jnp.meshgrid(jnp.linspace(-4,4,N), jnp.linspace(-4, 4, N))
xr = jnp.linspace(-3, 3, 100)

def plot_constraints(ax):
    ax.plot(x1, y1, 'k', markersize=5)
    ax.plot(x2, y2, 'k', markersize=5)
    ax.plot(x3, y3, 'k', markersize=5)
    ax.plot(x4, y4, 'k', markersize=5)
    ax.fill([x1,x2,x4,x3],[y1,y2,y4,y3],'gray', alpha=0.5);

    y1r = f1(xr)
    y2r = f2(xr)
    y3r = f3(xr)
    y4r = f4(xr)
    ax.plot(xr, y1r, 'w--')
    ax.plot(xr, y2r, 'w--')
    ax.plot(xr, y3r, 'w--')
    ax.plot(xr, y4r, 'w--')
```

```python
cube_task = IndependentCubeTask()
```

```python
logp = cube_task.profile.log_prob(
        {'x': X.reshape(-1), "y": Y.reshape(-1)}).reshape((N, N))

fig, ax = plt.subplots(1, 1, figsize=(3,3))
ax.contourf(X, Y, logp, levels=20, cmap='Blues_r')

plot_constraints(ax)
ax.set(xlim=(-3,2), ylim=(-3,2), xlabel='$x$', ylabel='$y$');
```

### Correlated profile

In the general case, the inputs may be correlated. In this case, the user needs to provide a custom implementation
of `Profile`. We will show how to do this for the case where $x$ and $y$ are jointly Gaussian.

```python
from numpyro import distributions as numpyro_dist

class CorrelatedProfile(profiles.Profile):
    def __init__(self):
        self._dist = numpyro_dist.MultivariateNormal(
            loc=jnp.array([-2, -2]), covariance_matrix=jnp.array([[1.0, 0.8], [0.8, 1.5]])
        )
    def sample(self, rng, sample_shape=()):
        samples = self._dist.sample(rng, sample_shape=sample_shape)
        # We needs the [..., ] to maintain batch dimensions.
        return {'x': samples[..., 0], 'y': samples[..., 1]}

    def log_prob(self, samples):
        samples = jnp.stack([samples['x'], samples['y']], -1)
        return self._dist.log_prob(samples)

class CorrelatedCubeTask(tasks.Task):
    def __init__(self):
        b = 1.0
        x = sympy.Symbol("x")
        y = sympy.Symbol("y")
        c1 = x + y <= b   # type: sympy.Expr
        c2 = x + y >= -b  # type: sympy.Expr
        c3 = y - x >= -b  # type: sympy.Expr
        c4 = y - x <= b  # type: sympy.Expr
        profile = CorrelatedProfile()
        domains = {"x": (-10., 10.), "y": (-10., 10.)}
        super().__init__(profile, [c1, c2, c3, c4], domains)
```

All of the benchmarks are define similarly to the examples shown above.
If you are interested, check our the source code in src/sympais/tasks for more examples.

```python
correlated_cube_task = CorrelatedCubeTask()
logp = correlated_cube_task.profile.log_prob(
        {'x': X.reshape(-1), "y": Y.reshape(-1)}).reshape((N, N))

fig, ax = plt.subplots(1, 1, figsize=(3,3))
ax.contourf(X, Y, logp, levels=20, cmap='Blues_r')
plot_constraints(ax)
ax.set(xlim=(-3,2), ylim=(-3,2), xlabel='$x$', ylabel='$y$');
```

### Run samplers

Now we have our new task definitions, let's run DMC and SYMPAIS on these tasks.

```python
dmc_output = run_dmc(correlated_cube_task, seed=0, num_samples=int(1e8), batch_size=int(1e6))
print(dmc_output)
```

```python
sympais_output = run_sympais(
    correlated_cube_task,
    key=jax.random.PRNGKey(0),
    num_samples=int(1e6),
    num_proposals=100,
    tune=False,
    init='realpaver',
    num_warmup_steps=500,
    window_size=100
)
print(sympais_output)
```


# MHD Equation with CentPy in 2d

### Import packages

```python
# Install the centpy package
!pip install centpy
```

    Requirement already satisfied: centpy in /usr/local/lib/python3.6/dist-packages (0.1)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from centpy) (1.18.5)

```python
# Import numpy and centpy for the solution
import numpy as np
import centpy
```

```python
# Imports functions from matplotlib and setup for the animation
import matplotlib.pyplot as plt
from matplotlib import animation
from IPython.display import HTML
```

### Equation

We solve the equations of ideal magnetohydrodynamics in 1D

\begin{equation}
\partial_t
\begin{bmatrix}
\rho \\ \rho v_x \\ \rho v_y \\ \rho v_z \\ B_x \\ B_y \\ B_z \\ E
\end{bmatrix}
+
\partial_x
\begin{bmatrix}
\rho v_x \\ \rho v_x^2 + p^* - B_x^2 \\ \rho v_x v_y - B_x B_y \\
\rho v_x v_z - B_x B_z \\ 0 \\ B_y v_x - B_x v_y \\ B_z v_x - B_x v_z \\
(E+p^*) v_x - B_x (B_x v_x + B_y v_y + B_z v_Z)
\end{bmatrix}
+
\partial_y
\begin{bmatrix}
\rho v_y \\ \rho v_y v_x - B_y B_x \\ \rho v_y^2 + p^* - B_y^2  \\
\rho v_y v_z - B_y B_z \\ B_x v_y - B_y v_x \\ 0 \\ B_z v_y - B_y v_z \\
(E+p^*) v_y - B_y (B_x v_x + B_y v_y + B_z v_Z)
\end{bmatrix}
= 0
\end{equation}

where the total pressure is given by

\begin{equation}
p^* = p + \frac{1}{2} (B_x^2 + B_y^2 + B_z^2)
\end{equation}

with the equation of state

\begin{equation}
p = (\gamma-1) \left(E-\frac{1}{2} \rho (v_x^2+v_y^2+v_z^2) -
\frac{1}{2}(B_x^2 + B_y^2 + B_z^2)\right), \qquad \gamma=5/3
\end{equation}

The solution is computed on the domain $(x,y,t)\in([0,2\pi]\times[0,2\pi]\times[0,3])$ with initial data for an *Orszag-Tang vortex system*:

\begin{equation}
(\rho, v_x, v_y, v_z, B_x, B_y, B_z, p)_{t=0} =
(\gamma^2, -\sin y, \sin x, 0, -\sin y, \sin (2x), 0, \gamma)
\end{equation}

and Dirichlet boundary data set by initial data on each boundary. The solution is computed using a 288$\times$288 mesh and CFL number 0.75.

```python
pars = centpy.Pars2d(
    x_init=0., x_final=2.*np.pi,
    y_init=0., y_final=2.*np.pi,
    J=144, K=144,
    t_final=3.0,
    dt_out=0.05,
    cfl=0.75,
    scheme="sd2",)
pars.gamma = 5./3
```

```python
# MHD equation
class MHD2d(centpy.Equation2d):

    # Helper functions for the equation

    def pressure(self, u):
        return (self.gamma - 1.0) * (
            u[:, :, 7]
            - 0.5 * (u[:, :, 1] ** 2 + u[:, :, 2] ** 2 + u[:, :, 3] ** 2) / u[:, :, 0]
            - 0.5 * (u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2)
        )

    def pressure_star(self, u):
        return self.pressure(u) + 0.5 * (
            u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2
        )

    def initial_data(self):
        u = np.zeros((self.J + 4, self.K + 4, 8))
        x = self.xx.T
        y = self.yy.T
        gamma = self.gamma
        dx = self.dx
        dy = self.dy

        u[:, :, 0] = gamma ** 2
        u[:, :, 1] = u[:, :, 0] / dy * (np.cos(y + 0.5 * dy) - np.cos(y - 0.5 * dy))
        u[:, :, 2] = 0.0
        u[:, :, 3] = -u[:, :, 0] / dx * (np.cos(x + 0.5 * dx) - np.cos(x - 0.5 * dx))
        u[:, :, 4] = 1.0 / dy * (np.cos(y + 0.5 * dy) - np.cos(y - 0.5 * dy))
        u[:, :, 5] = 0.0
        u[:, :, 6] = ( -0.5 / dx * (np.cos(2.0 * (x + 0.5 * dx)) - np.cos(2.0 * (x - 0.5 * dx))) )

        I1 = ( -0.125/dy*(u[:, :, 0] + 1.0)*(np.sin(2.0 * (y + 0.5 * dy)) - np.sin(2.0 * (y - 0.5 * dy))) )
        I2 = ( -0.125/dx*u[:, :, 0]*(np.sin(2.0 * (x + 0.5 * dx)) - np.sin(2.0 * (x - 0.5 * dx))) )
        I3 = ( -0.0625 / dx * (np.sin(4.0 * (x + 0.5 * dx)) - np.sin(4.0 * (x - 0.5 * dx))) )
        u[:, :, 7] = 3.0 + 0.5 * u[:, :, 0] + I1 + I2 + I3

        return u

    def boundary_conditions(self, u):  # periodic
        # x-boundary
        u[0] = u[-4]
        u[1] = u[-3]
        u[-2] = u[2]
        u[-1] = u[3]
        # y-boundary
        u[:, 0] = u[:, -4]
        u[:, 1] = u[:, -3]
        u[:, -2] = u[:, 2]
        u[:, -1] = u[:, 3]

    def flux_x(self, u):
        f = np.empty_like(u)

        p_star = self.pressure_star(u)

        f[:, :, 0] = u[:, :, 1]
        f[:, :, 1] = u[:, :, 1] ** 2 / u[:, :, 0] + p_star - u[:, :, 4] ** 2
        f[:, :, 2] = u[:, :, 1] * u[:, :, 2] / u[:, :, 0] - u[:, :, 4] * u[:, :, 5]
        f[:, :, 3] = u[:, :, 1] * u[:, :, 3] / u[:, :, 0] - u[:, :, 4] * u[:, :, 6]
        f[:, :, 4] = 0.0
        f[:, :, 5] = (
            u[:, :, 1] * u[:, :, 5] / u[:, :, 0] - u[:, :, 4] * u[:, :, 2] / u[:, :, 0]
        )
        f[:, :, 6] = (
            u[:, :, 1] * u[:, :, 6] / u[:, :, 0] - u[:, :, 4] * u[:, :, 3] / u[:, :, 0]
        )
        f[:, :, 7] = (u[:, :, 7] + p_star) * u[:, :, 1] / u[:, :, 0] - u[:, :, 4] * (
            u[:, :, 4] * u[:, :, 1] / u[:, :, 0]
            + u[:, :, 5] * u[:, :, 2] / u[:, :, 0]
            + u[:, :, 6] * u[:, :, 3] / u[:, :, 0]
        )

        return f

    def flux_y(self, u):
        g = np.empty_like(u)

        p_star = self.pressure_star(u)

        g[:, :, 0] = u[:, :, 3]
        g[:, :, 1] = u[:, :, 3] * u[:, :, 1] / u[:, :, 0] - u[:, :, 4] * u[:, :, 6]
        g[:, :, 2] = u[:, :, 3] * u[:, :, 2] / u[:, :, 0] - u[:, :, 5] * u[:, :, 6]
        g[:, :, 3] = u[:, :, 3] ** 2 / u[:, :, 0] + p_star - u[:, :, 6] ** 2

        g[:, :, 4] = (
            u[:, :, 3] * u[:, :, 4] / u[:, :, 0] - u[:, :, 6] * u[:, :, 1] / u[:, :, 0]
        )
        g[:, :, 5] = (
            u[:, :, 3] * u[:, :, 5] / u[:, :, 0] - u[:, :, 6] * u[:, :, 2] / u[:, :, 0]
        )
        g[:, :, 6] = 0.0
        g[:, :, 7] = (u[:, :, 7] + p_star) * u[:, :, 3] / u[:, :, 0] - u[:, :, 6] * (
            u[:, :, 4] * u[:, :, 1] / u[:, :, 0]
            + u[:, :, 5] * u[:, :, 2] / u[:, :, 0]
            + u[:, :, 6] * u[:, :, 3] / u[:, :, 0]
        )

        return g

    def spectral_radius_x(self, u):
        rho = u[:, :, 0]
        vx = u[:, :, 1] / rho
        vy = u[:, :, 2] / rho
        vz = u[:, :, 3] / rho
        p = (self.gamma - 1.0) * (
            u[:, :, 7]
            - 0.5 * rho * (vx ** 2 + vy ** 2 + vz ** 2)
            - 0.5 * (u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2)
        )
        A = self.gamma * p / rho
        B = (u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2) / rho
        cfx = np.sqrt(
            0.5 * (A + B + np.sqrt((A + B) ** 2 - 4 * A * u[:, :, 4] ** 2 / rho))
        )
        # cfy = np.sqrt(
        #     0.5 * (A + B + np.sqrt((A + B) ** 2 - 4 * A * u[:, :, 6] ** 2 / rho))
        # )

        return np.abs(vx) + cfx

    def spectral_radius_y(self, u):
        rho = u[:, :, 0]
        vx = u[:, :, 1] / rho
        vy = u[:, :, 2] / rho
        vz = u[:, :, 3] / rho
        p = (self.gamma - 1.0) * (
            u[:, :, 7]
            - 0.5 * rho * (vx ** 2 + vy ** 2 + vz ** 2)
            - 0.5 * (u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2)
        )
        A = self.gamma * p / rho
        B = (u[:, :, 4] ** 2 + u[:, :, 5] ** 2 + u[:, :, 6] ** 2) / rho
        # cfx = np.sqrt(
        #     0.5 * (A + B + np.sqrt((A + B) ** 2 - 4 * A * u[:, :, 4] ** 2 / rho))
        # )
        cfy = np.sqrt(
            0.5 * (A + B + np.sqrt((A + B) ** 2 - 4 * A * u[:, :, 6] ** 2 / rho))
        )

        return np.abs(vy) + cfy
```

### Solution

```python
eqn = MHD2d(pars)
soln = centpy.Solver2d(eqn)
soln.solve()
```

### Animation

```python
# Animation
j0 = slice(2,-2)

fig = plt.figure(figsize=(8,6))
ax = plt.axes(xlim=(soln.x_init,soln.x_final), ylim=(soln.y_init, soln.y_final))

ax.set_title("MHD Density")
ax.set_xlabel("x")
ax.set_ylabel("y")

img=ax.imshow(soln.u_n[1,j0,j0,0], extent=[0, 6.3, 0, 6.3], origin='lower',
           cmap='inferno', alpha=0.5)
fig.colorbar(img)

def animate(i):
    ax.collections = []
    ax.contour(soln.x[j0], soln.y[j0], soln.u_n[1+i,j0,j0,0], 20, colors='black',linewidths=0.1)
    img.set_array(soln.u_n[i,j0,j0,0])
    img.autoscale()

plt.close()
anim = animation.FuncAnimation(fig, animate, frames=soln.Nt, interval=100, blit=False);
HTML(anim.to_html5_video())
```