# Home assigment 1

Please give your name below:

```python
name=''
```

## Exercise 1

In nature Uranium is commonly available in the form of two isotopes: $^{235}U$ and $^{238}U$. Let us define Uranium enrichment as:
\begin{equation}
\epsilon= \frac{N_{^{235}U}}{N_{^{235}U}+N_{^{238}U}}
\end{equation}

where $N$ indicates the number of atoms in a given volume. The half-lives are $4.51 \times 10^9$ and $7.13 \times 10^8$ yr for $^{238}U$ and $^{235}U$, respectively. Today, the natural enrichment is $0.72 \%$, whereas a typical LWR fuel has an enrichment of $2-5 w\%$. It is notable that in the past natural uranium contained enough $^{235}U$ that LWRs could have been built without requiring further enrichment. In fact, evidance shows that cca $1.7 \times 10^9$ yr ago at certain locations such as in Oklo, Gabon the conditions were ideal for the formation of [natural nuclear fission reactors](https://en.wikipedia.org/wiki/Natural_nuclear_fission_reactor).

- Derive the equation that describes the evolution of the enrichment in time. (hint: use markdown and Latex to document your derivation)
- Define a program function which evaluates the enrichment at a certain time in the past. The function should be able to handle array-like inputs (hint: you can use numpy).
- Evaluate the function and print the enrichment
    - at the birth of Earth ($4.5 \times 10^9$ years ago)
    - at the time when the Oklo reactor "operated" ($1.7 \times 10^9$ years ago)
- Plot the enrichment of uranium versus time in the past. Include a vertical line (`plt.axvline`) at the two times highlighted above

### Solution

Include your derivation here.

```python
# include your code here

def epsilon(t):
    """Function to evaluate Uranium enrichment at time t. The parameter t can be both float or array."""
    return eps

#do the printing and plotting here
```

```python

```


%load_ext autoreload
%autoreload 2

```python
import model_funcs
import numpy as np
import sympy as sm
from scipy import interpolate
from bokeh.io import output_notebook,show
from bokeh.layouts import row, column
output_notebook()
```

    Loading BokehJS ...

In the following representation, we have divided the project in six sections.
1. Introduction to our simple macro model
2. Solving the simple macro model
3. Introduction and solution to the micromodel
4. Solving the modified solowmodel
5. Different methods to solve the model
6. Solving the model for different $\theta$

# 1. Our Model

Our model is the classic solowmodel, but instead of having a fixed savings rate, in each period the savings rate is decided by a micro-household optimization problem. We will get back to this later but first we will solve the simple Solow model with a fixed savingsrate.

**Macro foundations**

The basic solow model consists of the following equations:
$$
Y_t = A K^{\alpha} L^{1-\alpha}
$$$$
K_{t+1}=s_{K}\cdot Y_{t}+\left(1-\delta\right)K_{t}
$$$$
L_{t+1}=\left(1+n\right) L_{t}
$$

By denoting $k_{t},y_{t}=\frac{K_{t}}{L_{t}},\frac{Y_{t}}{L_{t}}$ the following transition equation can be derived:
$$
k_{t+1}= \frac{1}{1+n} \left(s_{Kt}\cdot A k_{t}^{\alpha} + \left(1+\delta\right)k_{t} \right)
$$

# 2. Solving the basic Solow model

For a fixed savings rate we know that basic Solow model has convergenge to steady state (Inada-conditions). Convergence will imply $k_{t+1}=k_{t}=k^{*}$ thus we have from the transtition equation that:
$$
k^{*}= \frac{1}{1+n} \left(s_{Kt}\cdot A {k^{*}}^{\alpha} + \left(1+\delta\right)k^{*} \right)
$$
And can use sympy to derive our steady state capital for a given fixed savings will be:

```python
sm.init_printing(use_unicode=True)
# Function that solve the transtion equation
ss_k = model_funcs.steadystate()
ss_k
```

```python
sm.init_printing(use_unicode=False)
```

We can also simulate the model with a function defined in model_funcs.py and show it visually.
First we set up reasonable parameter values

$\alpha$, with a Cobb-Douglas production function, this is the incomeshare of capital, it is set to $\frac{1}{3}$, a ordinary macro-assumption.

$\delta$, capital depreciation, is set to $5\%$.

$n$, population growth is set to $0.8\%$ representing average european population growth.

$s_{K}$, the fixed savings rate is set to $20\%$.

$A$ and $L_{0}$ is both normalized to 1.

$K_{0}$ is set to 5, below the steady state.

```python
# Reasonable parameters:
#Macro:
alpha = 1/3
delta = 0.05
n = 0.008
sk = 0.2

# production:
a = 1
k0 = 5
l0 = 1

# Our timeline:
timeline = 200

# Steady state capital pr. worker:
k_star = model_funcs.find_ssk_k(sk,a,delta,n,alpha)
k_stars  =[k_star for i in range(timeline)]

# The evolution of the normal solow model:
trad_k_pr, trad_y_pr = model_funcs.solowwalk(
    k0, a, l0, n, sk, alpha, delta, timeline)

p = model_funcs.plotting("T",['Solow path', 'Steady state'], range(timeline), [trad_k_pr,k_stars],
                    y_name='Capital pr. capita', title = 'Traditional Solowpath',
                    legendlocation ='bottom_center', width = 600)
show(p,notebook_handle=True)
```

<Bokeh Notebook handle for In[4]>

# 3. The micro model

Now that we have seen the solow model with a fixed savingsrate. We will show how the the savingsrate is formed in our micromodel:

**Micro-foundation of the savingsrate**

Representative household utilityfunction in single periode:
$$
u\left(c_{t}\right)=\frac{c^{1-\theta}-1}{1-\theta}
$$
When the household sums its total utility for mulitiple future periods, it discounts future consumption for immediate consumption by $1-\beta$ so utility for all periods from period 0 to period T, given a vector C of consumption in each period is:
$$
U\left(C\right)=\sum_{t=0}^{T}\beta^{t}\frac{c_{t}^{1-\theta}-1}{1-\theta}
$$
We assume that the household understands how the solow model works, e.i. how production for consumption is made and how capital accumulation works. From the calulations above we have:
$$
y_{t}=A k_{t}^{\alpha} \Rightarrow
c_{t} = \left(1-s_{Kt}\right) A k_{t}^{\alpha}
$$ $$
k_{t+1}= \frac{1}{1+n} \left(s_{Kt}\cdot A k_{t}^{\alpha} + \left(1+\delta\right)k_{t} \right)
$$
This is our 'budget constraint' more precisely defined as production constraints for all periods

So the final optimizationproblem is:
$$
\max_{s_{K_t}}U\left(C\right)=\sum_{t=0}^{T}\beta^{t}\frac{c_{t}^{1-\theta}-1}{1-\theta}
$$
$$ s.t.$$$$
c_{t} = \left(1-s_{Kt}\right) A k_{t}^{\alpha},\;t\in [0,T]
$$$$
k_{t}=
\begin{cases}
k_{0}& \text{if }t=0\\
\frac{1}{1+n} \left(s_{Kt-1}\cdot A k_{t-1}^{\alpha} + \left(1+\delta\right)k_{t-1} \right)& \text{if }t\in\left[1,T\right]
\end{cases}
$$$$
k_{0} \text{ given}
$$
It's important to note that although the optimization, $s_{Kt}$, is solved for all periods. Only the solution for t=0 is used, since the problem is solved again in the next period. E.g. in each period the household formulates a plan for current and future consumption. In the current period the plan is pursued but in the following period a new plan is formulated, which is then also only followed for one period.

First we set up microparameter values:

$\theta$, determining the representative household's aversion to fluctations in consumption is as base, set to 0.5, but values below and above are also looked at, as we don't know an approximate of the true value

$\beta$, determining how much the representative households values immediate above future consumption, is set to 0.99. Representing a low preference.

$t$, the amount of future time periods considered by the household, is set to 40 for two reasons:

1. Computational limitations, honestely speed becomes a factor when we simulate a more foresighted representative household. Later we will use other solutionmethod that allows us to relax this assumption.
2. It approximately represents the average remaining lifetime of the working population. As we asumme no 'warm glow' from bequesting, this will also be the timeline that the population cares about on average.

```python
# Parameters
#Micro:
theta = 0.5
beta = 0.99

t = 40

# time preference utility weights and population growth
# are precomputed to save time:
weight = np.array([beta**i for i in range(t)])
l = np.array([l0*(1+n)**i for i in range(t)])
```

**Solving the micro-foundation numerically**

First we look at how, for different thetas, optimal savings rate in period t=0, depends on the level of initial capital:

```python
'''We find the prefered savings rate in period t=0
for different thetas an initial capitals'''
k_n = 100
thetas = [0.3,0.5,0.7,1.5]
sks = [np.zeros(k_n) for i in range(len(thetas))]
k0s = np.linspace(0.001,30,k_n)

for j in range(len(thetas)):
    for i in range(k_n):
            sks[j][i] = model_funcs.optimal_sks(
                t, a, l, weight, delta, alpha, thetas[j], k0s[i])

# We save the result for theta=0.5 as a interpolater for use in Macro model
sk_interpers = [interpolate.RegularGridInterpolator([k0s], sks[i],
        bounds_error=False,fill_value=None) for i in range(len(thetas))]
```

```python
''' We use Bokeh to plot and have defined a function in model_funcs.py
that calls the plot to bokeh '''

y_names=[]
# \u03B8 is code for theta
for theta in thetas:
    y_names.append(f'\u03B8 = {theta:.1f}')

p = model_funcs.plotting("Capital pr. capita",y_names, k0s, sks, width=600)
show(p,notebook_handle=True)
```

<Bokeh Notebook handle for In[7]>

So our models generally implies a negative relationship between amount of capital pr. capita. This makes well intuitive sense, since marginal product of capital is relatively smaller, when there is more of it, and thus the household will choose to consume more now, rather than making a bad investment for future consumption. This also bodes well for our model having convergence, to explain why, first we plot the same graph, but with a line indicating steady state for a given fixed savings rate. To do this we use the below formula derived ealier and plot it for varying $s_{K}$.
$$
k^{*}=\left(\frac{B\cdot s_{K}}{\delta+n}\right)^{\frac{1}{1-\alpha}}
$$

```python
sksopt = model_funcs.find_ssk_sk(k0s,a,delta,n,alpha)
```

```python
y_names.append('Steady state')
sks.append(sksopt)
```

```python
figure1 = model_funcs.plotting("Capital pr. capita",y_names, k0s, sks, title='Optimal savings rate for varying \u03B8 and capital pr. capita',width= 600)
show(figure1,notebook_handle=True)
```

<Bokeh Notebook handle for In[10]>

Now we can see that we have convergence. Furthermore we acutaully have faster convergence, than in the traditional solow model. For at fixed savings rate $k_{t}<k^{*}$, the transition equation would imply $\Delta k_{t}>0$. But now, if $k_{t}<k^{*}$ the chosen savings rate by the household will be bigger than at the steady state-level, savings will thus be bigger, implying a bigger $\Delta k_{t}$ when $k_{t} \rightarrow k^{*}$ we will see $s_{Kt} \rightarrow s_{K}^{*}$ and the chosen savings rate will be fixed.

Since we have simple solow model, we know that the real optimal fixed savings rate for maximum overall consumption in the long run is acutally $s_{K}^{**}=\alpha$ which in our case is $\frac{1}{3}$, but we see that this steady state can't be achieved in our model. This is because our representative household is shortsighted and values immediate consumption above future consumption.

**Plotting the consumption plan**

Another way of visually showing convergence is by plotting the consumption plan of the representative houshold. We will do this below. We slightly alter our model when doing this because we now assume that the representative household plans for more periods ahead, e.g. considers and cares about consumption in the next many many periods. Because we now do this we can say that the consummer will stick to the plan made in period 1, in the following period untill it hits steady state where it will stay, since there are no chocks in our model.

```python
model_funcs.plotting_plan()
```

    Output()

    HBox(children=(IntSlider(value=200, description='T', max=500, min=10, orientation='vertical', step=10), FloatS…

We can see that we hit steady state for almost all parameter values (for large $\theta$s convergence is so slow, that T has to be increased). Thus this way of solving the model is viable.

# 4. The modified solow walk

Now that we have showed in theory that the model will have convergence, we will show it with a simulation of our model, we will look at a timerange of 150 periods and compare how fast this model convergences relative to the traditional solow-model, with a fixed savings rate set at the steady state level savings of our model.

```python
timeline = 150
k0 = 5
'''
This function finds the implied steady state values of s_k and k_t in our model,
basicly by finding the crossing in the above figure.
'''

k_star, sk_star = model_funcs.find_ss_interp(sk_interpers[1], a, n, delta, alpha)

# The evolution of the normal solow model:
trad_k_pr, trad_y_pr = model_funcs.solowwalk(
    k0, a, l0, n, sk_star, alpha, delta, timeline)

# The evolution of the modified solowmodel:
k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
    k0, l0, a,n, alpha, delta, sk_interpers[1], timeline)

```

```python
# Again plotting with the help of bokeh:
# These are the ones that are shown:
y_names = ['Model path', 'Fixed Savings rate','Steady state level']

ys = [k_pr ,trad_k_pr, [k_star for i in range(timeline)]]

p1 = model_funcs.plotting("T",y_names, range(timeline), ys,
                     y_name= 'Capital pr. capita',legendlocation ='bottom_center',
                    title = 'Evolution of capital')

ys2 = [sk_path*100, [sk_star*100 for i in range(timeline)]]

p2 = model_funcs.plotting("T",['Model','Steady state/Fixed'], range(timeline), ys2,
                     y_name= 'Savings rate',legendlocation ='top_center',
                    title = 'Evolution of the savings rate',y_unit=' %')

show(row(p1,p2),notebook_handle=True)
```

<Bokeh Notebook handle for In[13]>

Thus we see the faster convergence to the same steady state. we get the same result if our intial $k_{0}>k^{*}$. Lets try with $k_{0}=25$ implying that in the first period, the representative household will actually not want to save any money for the future in the first period.

```python
k0 = 25
# Exactly the same as before
k_star, sk_star = model_funcs.find_ss_interp(sk_interpers[1], a, n, delta, alpha)

trad_k_pr, trad_y_pr = model_funcs.solowwalk(
    k0, a, l0, n, sk_star, alpha, delta,timeline)

k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
    k0, l0, a,n, alpha, delta, sk_interpers[1], timeline)
```

```python
y_names = ['Model_path', 'Fixed Savings rate','Steady state level']
ys = [k_pr ,trad_k_pr, [k_star for i in range(timeline)]]
p1 = model_funcs.plotting("T",y_names, list(range(timeline)), ys,
                     y_name= 'Capital pr. capita',legendlocation ='top_center',
                    title='Evolution of capital')
ys2 = [sk_path*100, [sk_star*100 for i in range(timeline)]]
p2 = model_funcs.plotting("T",['Model','Steady state/Fixed'], list(range(timeline)), ys2,
                     y_name= 'Savings rate', legendlocation ='bottom_center',
                     title ='Evolution of savings rate',y_unit=' %')
show(row(p1,p2),notebook_handle=True)
```

<Bokeh Notebook handle for In[15]>

# 5. Different ways of solving the model

After initially making this project, Jeppe has explained to us, a faster way of solving this model (And also that the name of the model is actually Ramsey). Here we will solve this model, timing the speed of the the two solutions we have utilized, and also comparing their results.
This new of solving the model actually also requires a slight altering of the assumptions of the model. Instead of assuming that the representative household consideres and values comsumption in a limited time, we assume that this timeframe is, in theory unlimited, but in our model a large enough number of periods to imply convergence (We test 200 periods, if that's not enough we use 400). Going from a considered time frame of 40 to 200 results in a slighter higher savningsrate in steady state, since the household values the future relatively more now.

```python
k0 = 8
theta = 0.5
%load_ext line_profiler
```

First we look at the time profile for the interpolation solution

```python
def full_solution_interp():
    k_n = 100
    sks = np.zeros(k_n)
    k0s = np.linspace(0.001,30,k_n)

    for i in range(k_n):
        sks[i] = model_funcs.optimal_sks(
            t, a, l, weight, delta, alpha, theta, k0s[i])

    # We save these results as a interpolater for use in Macro model
    sk_interper = interpolate.RegularGridInterpolator([k0s], sks,
            bounds_error=False,fill_value=None)

    # We then are able to simulate the model:
    k_star, sk_star = model_funcs.find_ss_interp(sk_interper, a, n, delta, alpha)

    k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
        k0, l0, a,n, alpha, delta, sk_interper, timeline)

def loop(n):
    for i in range(n):
        full_solution_interp()
```

```python
# This takes about a minut to run
%lprun -f full_solution_interp -f loop loop(10)
```

    Timer unit: 1e-06 s

    Total time: 69.9715 s
    File: <ipython-input-17-f1b28ac403ea>
    Function: full_solution_interp at line 1

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
         1                                           def full_solution_interp():
         2        10          9.0      0.9      0.0      k_n = 100
         3        10         47.0      4.7      0.0      sks = np.zeros(k_n)
         4        10        910.0     91.0      0.0      k0s = np.linspace(0.001,30,k_n)
         5
         6
         7      1010       2853.0      2.8      0.0      for i in range(k_n):
         8      1000       2000.0      2.0      0.0          sks[i] = model_funcs.optimal_sks(
         9      1000   69708811.0  69708.8     99.6              t, a, l, weight, delta, alpha, theta, k0s[i])
        10
        11                                               # We save these resoluts as a interpolater for use in Macro model
        12        10         26.0      2.6      0.0      sk_interper = interpolate.RegularGridInterpolator([k0s], sks,
        13        10       1131.0    113.1      0.0              bounds_error=False,fill_value=None)
        14
        15                                               # We then are able to simulate the model:
        16        10      13416.0   1341.6      0.0      k_star, sk_star = model_funcs.find_ss_interp(sk_interper, a, n, delta, alpha)
        17
        18        10         29.0      2.9      0.0      k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
        19        10     242278.0  24227.8      0.3          k0, l0, a,n, alpha, delta, sk_interper, timeline)

    Total time: 69.9759 s
    File: <ipython-input-17-f1b28ac403ea>
    Function: loop at line 21

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
        21                                           def loop(n):
        22        11        258.0     23.5      0.0      for i in range(n):
        23        10   69975646.0 6997564.6    100.0          full_solution_interp()

We see that almost all of the computing time for this method is in the setup of this model, e.i. computing the optimal savingsrates for the  interpeter of the capital levels.

```python
def loop_ram(n):
    for i in range(n):
        model_funcs.find_ss_ramsey(
            a, n, beta, delta, alpha, theta, k0=k0,l0=l0,path=True)
```

```python
%lprun -f model_funcs.find_ss_ramsey -f loop_ram loop_ram(10)
```

    Timer unit: 1e-06 s

    Total time: 34.0084 s
    File: /Users/Asker/Desktop/OneDrive - Koebenhavns Universitet/Documents/Uni/4. semester/Programering og numerisk analyse /Projects/projects-2019-unity/modelproject/model_funcs.py
    Function: find_ss_ramsey at line 363

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
       363                                           def find_ss_ramsey(a, n, beta, delta, alpha, theta, k0=10,l0=1,path=False):
       364
       365                                               '''
       366                                               Finding the steady state of the model by solving the comsumer optimization
       367                                               once, and finding the point in comsumption plan where savings stabilizes.
       368
       369                                               Args:
       370                                                       a (float)       : The A parameter in the production function
       371                                                       n (float)       : Population growth
       372                                                       beta (float)    : Time discounter of the consumer
       373                                                       delta (float)   : Depreciation rate
       374                                                       alpha (float)   : Alpha in the productionfunction
       375                                                       theta(float)    : Parameter of consumers considered utilityfunction
       376                                                       k0 (float)      : Inital level of capital
       377                                                       l0 (float)      : Inital level of population
       378                                                                       (The two parameters above are only necessary
       379                                                                       if the funcitons fails to converge.)
       380                                                       path (bool)     : If true will also return the Solow path to steady state
       381                                               Returns:
       382                                                       k_star          : Capital level in steady state
       383                                                       sk_star         : The now fixed savings rate, when steady state is reached.
       384                                                       sks[:ss_i]      : The path of the planned savings rate, up untill convergence is reached
       385                                                       k_pr_path       : The path of capital untill convergence
       386                                               '''
       387
       388        10         18.0      1.8      0.0      t = 200
       389        10        838.0     83.8      0.0      weight = np.array([beta**i for i in range(t)])
       390        10       3906.0    390.6      0.0      l = l0*np.array([(1+n)**i for i in range(t)])
       391
       392        10   33995370.0 3399537.0    100.0      sks = optimal_sks(t, a, l, weight, delta, alpha, theta, k0, first=False)
       393        10         33.0      3.3      0.0      ss_sk = False
       394
       395                                               # We only search for steady state in middle of the comsumptionp plan,
       396                                               # since sometimes the savingsrate is constant at 0 at either the begining or end
       397        10         44.0      4.4      0.0      i = int(0.25*t)
       398        10         22.0      2.2      0.0      while i < int(0.75*t):
       399        10       1659.0    165.9      0.0          if np.isclose(sks[i],sks[i+1]):
       400        10         29.0      2.9      0.0              ss_sk = sks[i]
       401        10         17.0      1.7      0.0              break
       402
       403                                                   i+=1
       404
       405        10       2002.0    200.2      0.0      if ss_sk==False:
       406                                                   # We try again with a longer timerframe
       407                                                   t = 400
       408                                                   weight = np.array([beta**i for i in range(t)])
       409                                                   l = l0*np.array([(1+n)**i for i in range(t)])
       410
       411                                                   sks = optimal_sks(t, a, l, weight, delta, alpha, theta, k0, first=False)
       412                                                   i = int(0.25*t)
       413                                                   while i < int(0.75*t):
       414                                                       if np.isclose(sks[i],sks[i+1]):
       415                                                           ss_sk=sks[i]
       416                                                           break
       417
       418                                                       i += 1
       419
       420                                               # Convergence period:
       421        10         19.0      1.9      0.0      ss_i = i
       422
       423                                               # We use the analytical solution to find the steady state of capital for this savings rate:
       424        10        115.0     11.5      0.0      ss_k = find_ssk_k(ss_sk,a,delta,n,alpha)
       425
       426        10         14.0      1.4      0.0      if path:
       427        10         21.0      2.1      0.0          k_pr_path = [k0/l0]
       428       500        843.0      1.7      0.0          for i in range(1,ss_i):
       429       490        775.0      1.6      0.0              k_pr_path.append(transition_eq(
       430       490       2650.0      5.4      0.0                  a,k_pr_path[i-1],n,sks[i-1],alpha,delta))
       431        10         35.0      3.5      0.0          return ss_sk, ss_k, sks[:ss_i], k_pr_path
       432
       433                                               else:
       434                                                   return ss_sk, ss_k

    Total time: 34.0118 s
    File: <ipython-input-19-87176433393a>
    Function: loop_ram at line 1

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================

We see that most of the time is still spend calculating the optimal savingrate, but overall a lot less time is needed since the optimizer is now only called once instead of a 100 (albeit it is now optmizing over 200 variables instead of 30). So in conclusion the new solving method is overall faster, but if the interpolater is already calculated (as we did during the visualization of the Micro-problem, interpolation will be faster, since this micro problem is the only part of the problem that takes a notable time to run.

Now we will compare the conclusions of the two different solving methods.

```python
k_star, sk_star = model_funcs.find_ss_interp(sk_interpers[1], a, n, delta, alpha)

k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
    k0, l0, a,n, alpha, delta, sk_interpers[1], timeline)

ss_sk, ss_k, sk_path_2, k_pr_2 = model_funcs.find_ss_ramsey(
    a, n, beta, delta, alpha, theta, k0=k0,l0=l0,path =True)

p1 = model_funcs.plotting("T",['Capital pr. capita','Steady sate'], range(len(k_pr)),
                          [k_pr,[k_star for i in range(len(k_pr))]], y_name= 'Capital pr. capita',
                          legendlocation ='bottom_center',
                         title = 'Evolution of capital for method 1')

p2 = model_funcs.plotting("T",['Capital pr. capita','Steady sate'], range(len(k_pr_2)),
                          [k_pr_2,[ss_k for i in range(len(k_pr_2))]], y_name= 'Capital pr. capita',
                          legendlocation ='bottom_center',
                         title = 'Evolution of capital for method 2')

p3 = model_funcs.plotting("T",['Savings rate','Steady state'], range(len(sk_path)),
                          [sk_path*100,[sk_star*100 for i in range(len(sk_path))]],
                    y_name= 'Savings rate',legendlocation ='top_center',
                    title = 'Evolution of the savings rate for method 1',y_unit=' %')

p4 = model_funcs.plotting("T",['Savings rate','Steady state'], range(len(sk_path_2)),
                          [sk_path_2*100,[ss_sk*100 for i in range(len(sk_path_2))]],
                    y_name= 'Savings rate',legendlocation ='top_center',
                    title = 'Evolution of the savings rate for method 2',y_unit=' %')

show(column(row(p1,p2),row(p3,p4)),notebook_handle=True)
```

<Bokeh Notebook handle for In[21]>

As expected the second method, with a longer considered timeframe, gives us a slightly higher savingsrate and thus higher capital in the steady state.

# 6. The model for different values of $\theta$

Now we look at the evolution of the model for different values of theta in the household utility function. Both for initially low and initially high $k_{0}$.

Here we use the interpolation method, since the microproblem was solved for different values of $\theta$ earlier in the project.

```python
k_prs_low = []
sk_paths_low = []

k0 = 5
for i in range(4):
    k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
        k0, l0, a,n, alpha, delta, sk_interpers[i], timeline)
    k_prs_low.append(k_pr)
    sk_paths_low.append(sk_path)

k_prs_high = []
sk_paths_high = []

k0 = 20
for i in range(4):
    k_pr, y_pr, sk_path = model_funcs.new_mod_solowwalk(
        k0, l0, a,n, alpha, delta, sk_interpers[i], timeline)
    k_prs_high.append(k_pr)
    sk_paths_high.append(sk_path)

```

```python
theta_names = ['\u03B8 = 0.3', '\u03B8 = 0.5','\u03B8 = 0.7','\u03B8 = 1.5']

p1 = model_funcs.plotting("T",theta_names, list(range(timeline)), k_prs_low,
                     y_name= 'Capital pr. capita',legendlocation ='bottom_center',
                    title = 'Evolution of the capita pr. capita for different values of \u03B8')

p2 = model_funcs.plotting("T",theta_names, list(range(timeline)), sk_paths_low,
                     y_name= 'Savings rate',legendlocation ='top_center',
                    title = 'Evolution of the savingsrate for different values of \u03B8')

p3 = model_funcs.plotting("T",theta_names, list(range(timeline)), k_prs_high,
                     y_name= 'Capital pr. capita',legendlocation ='top_center',
                    title = 'Evolution of the capita pr. capita for different values of \u03B8')

p4 = model_funcs.plotting("T",theta_names, list(range(timeline)), sk_paths_high,
                     y_name= 'Savings rate',legendlocation ='bottom_center',
                    title = 'Evolution of the savingsrate for different values of \u03B8')

```

Before we show the figures of the simulation of the macro model, we remind ourselfs of the figure showing the optimal savingsrate for different $\theta$s
The crossing of the steady state-line with the lines of optimal savings rate, shows our steady state for the given $\theta$. Also it would make sence if the steeper the optimalsavings-line is, the faster the convergence towards the steady state will be. Since the savingsrate will be adjusted more, and the capitalaccumulation will be faster.

```python
show(figure1,notebook_handle=True)
```

<Bokeh Notebook handle for In[24]>

```python
show(row(p1,p2),notebook_handle=True)
show(row(p3,p4),notebook_handle=True)
```

<Bokeh Notebook handle for In[25]>

Overall we see that a higher $\theta$ gives a slower rate of convergence and a lower steady state level of optimal savingsrate.
And with that we end the project for now, tough the model leaves space for plenty of extensions.


```julia
using Zygote
```

```julia
f(x) = 2x
```

    f (generic function with 1 method)

```julia
df(x) = gradient(f,x)
```

    df (generic function with 1 method)

```julia
df(1.0)
```

    (2.0,)

```julia
using Symbolics
```

```julia
@variables x
```

\begin{equation}
\left[
\begin{array}{c}
x \\
\end{array}
\right]
\end{equation}

```julia
df(x)
```

    (2,)

```julia
x
```

\begin{equation}
x
\end{equation}

```julia
h(x) = x^2 + 2x + 1
g(x) = sin(x)
f(x) = g(h(x))
df(x) = gradient(f,x)
df(1.0)
```

    (-2.6145744834544478,)

```julia
@variables x
df(x)
```

    (2cos(1 + x^2 + 2x) + 2x*cos(1 + x^2 + 2x),)

```julia

```


# Constraining Cosmological Parameters by Using Lensing Data

## 1. Data Preparation

In this notebook, we will use data from strong gravitational lensing observation to constrain various cosmological parameters. This work utilizes Leaf & Melia (2018; https://arxiv.org/abs/1805.08640) paper as the main reference. The data was taken from their Table 1.

```python
# import necessary modules
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

%matplotlib inline

# set basic figure parameters
import matplotlib as mpl
mpl_param = {'figure.figsize'   : [8.0, 6.0],
             'savefig.dpi'      : 100,
             'axes.titlesize'   : 'xx-large',
             'axes.labelsize'   : 'xx-large',
             'text.usetex'      : False,
             'font.family'      : 'serif'}
mpl.rcParams.update(mpl_param)

# read lensing data which is taken from Leaf et al. (2018)
data = pd.read_csv('lensing_data.csv')

data = data[data.name != 'J0850-0347'] # outlier removal

# top rows in the data
data.head()
```

      name
      zl
      zs
      sigma_ap
      err_sigma_ap
      theta_E
      survey
      theta_ap
      theta_eff
      sigma_0
      err_sigma_0

      0
      J0151+0049
      0.517
      1.364
      219
      39
      0.68
      BELLS
      1.0
      0.89
      226
      40

      1
      J0747+4448
      0.437
      0.897
      281
      52
      0.61
      BELLS
      1.0
      1.24
      286
      53

      2
      J0747+5055
      0.438
      0.898
      328
      60
      0.75
      BELLS
      1.0
      2.87
      323
      59

      3
      J0801+4727
      0.483
      1.518
      98
      24
      0.49
      BELLS
      1.0
      0.57
      103
      25

      4
      J0830+5116
      0.530
      1.332
      268
      36
      1.14
      BELLS
      1.0
      1.10
      274
      37

In lensing system, we have so called $D$ parameter, which can be derived from observation and defined as:
\begin{equation}
    D^{\rm obs} = \frac{c^2 \theta_{\rm E}}{4 \pi \sigma_0^2}
\end{equation}

where the measurements error can be propagated with

\begin{equation}
    \sigma_{D^{\rm obs}} = D^{\rm obs} \sqrt{\bigg(\frac{\sigma_{\theta_{\rm E}}}{\theta_{\rm E}}\bigg)^2 + \bigg( \frac{2 \sigma_{\sigma_0}}{\sigma_0}\bigg)^2 + \sigma_X^2}
\end{equation}

Therefore, we can write the equation above into the codes below.

```python
# define function to calculate angular diameter distance as in Equation 3
# this is used for observational data
def D_obs(theta_E, sigma_0):

    # need to convert theta_E from arcsec to radian
    theta_E = np.deg2rad(theta_E/3600)

    # speed of light in km/s
    c = 3*1e5

    return c**2 * theta_E / (4*np.pi * sigma_0**2)

# error of observed angular diameter distance as in Equation 10
# this doesn't include the sigma_x
def D_obs_err(theta_E, err_theta_E, sigma_0, err_sigma_0):

    # systematic error
    sigma_X = 0.12

    return D_obs(theta_E, sigma_0) \
            * np.sqrt((err_theta_E/theta_E)**2 \
            + (2*err_sigma_0/sigma_0)**2 \
            + sigma_X**2)
```

```python
# calculate the angular diameter distance
Dobs = D_obs(data['theta_E'], data['sigma_0'])
Dobs_err = D_obs_err(data['theta_E'], data['theta_E']*0.05, data['sigma_0'], data['err_sigma_0'])
```

```python
# plot as in Figure 2 in their paper

plt.errorbar(x=data['sigma_0'].values,
             y=Dobs.values,
             xerr=data['err_sigma_0'].values,
             yerr=Dobs_err.values,
             fmt='bo', alpha=0.6)

plt.xlabel('$\sigma_0$ (km s$^{-1}$)')
plt.ylabel('$D^{\\rm obs}$')
plt.ylim(0, 2)
```

As we can see that there are some objects with $D^{\rm obs} > 1$. For a best fitting result, we have to exclude these outliers.

```python
# cut outliers based on page 3 in the paper
data = data[Dobs < 1]

# recalculate the angular diameter distance
Dobs = D_obs(data['theta_E'], data['sigma_0'])
Dobs_err = D_obs_err(data['theta_E'], data['theta_E']*0.05,
                     data['sigma_0'], data['err_sigma_0'])
```

## 2. Angular Diameter Distance

Theoretically, the angular diameter distance for certain cosmological model can be described as this equation:

\begin{equation}
    D(z_1, z_2) = \frac{c}{H_0 (1+z_2)}
        \times \int_{z_1}^{z_2}{
            \frac{dz'}{\sqrt{
                \Omega_{\rm m}(1+z')^3 + \Omega_{\rm r}(1+z')^4 + \Omega_{\rm de}(1+z')^{3(1+w_{\rm de})
               }}}}
\end{equation}

Then, the $D^{\rm th}$ is defined as

\begin{equation}
    D^{\rm th} = \frac{D(z_1, z_2)}{D(0, z_2)}
\end{equation}

Clearly, $D$ does not depend on $H_0$ for any considered models. These equations can be written as below.

```python
from scipy import integrate

# make angular diameter distance calculator
# equivalent to astropy model for cosmolo.angular_diameter_distance_z1z2()
# based on Equation 4 in Leaf et al. (2018)
def ang_dist (z_1, z_2, Omega_m, Omega_lambda, wde):

    Omega_r = 0

    # integration part
    # integration is calculated from redshift=z_lensing to redshift=z_source
    fn = lambda z: (Omega_r*(1+z)**4. \
                    + Omega_m*(1+z)**3 \
                    + Omega_lambda*(1+z)**(3*(1+wde)) \
                    )**-0.5

    # return array values
    return c/(H0*(1+z_2)) \
        * np.asarray([integrate.quad(fn, _z[0], _z[1])[0] for _z in list(zip(z_1, z_2))])

# D theoretical based on Equation 7 in Leaf et al. (2018)
def D_theory(X, Omega_m, Omega_lambda, wde):
    z_1, z_2 = X
    return ang_dist(z_1, z_2, Omega_m, Omega_lambda, wde) \
            / ang_dist(0*z_1, z_2, Omega_m, Omega_lambda, wde)
```

## 3. Maximum Likelihood Fitting

We will use maximization of the likelihood function to constrain and compare the models. First, we calculate $D^{\rm obs}$ and $D^{\rm th}$. Then, the join likelihood function for all parameters, based on a flat Bayesian prior, is

\begin{equation}
    \mathcal{L} = \prod_{i} \frac{1}{\sqrt{2 \pi} \sigma_{D_i}} \exp{\bigg[-\frac{\chi^2}{2} \bigg]}
\end{equation}

where for each measurement

\begin{equation}
    \chi^2_i = \frac{(D_i^{\rm obs} - D_i^{\rm th})^2}{\sigma^2_{D_i}}
\end{equation}

In python, those equations can be written as below.

```python
# define likelihood function as in Equation 11 in Leaf et al. (2018)
def lnlike(theta, X, y, yerr):

    Omega_m, Omega_lambda, wde = theta
    model = D_theory(X, Omega_m, Omega_lambda, wde)

    # chi-square
    chi2 = ((y-model)**2)/yerr**2

    return np.sum(1/(np.sqrt(2*np.pi)*yerr) * (np.exp(-chi2/2)))
```

```python
# assign input parameters to new variables
X = (data['zl'].values, data['zs'].values)
y = Dobs
yerr = Dobs_err

# some constants
c = 3 * 1e5 # km/s
H0 = 67.7 #km / (Mpc s)

# Omega_m, Omega_lambda, wde
initial_params = [0.3, 0.7, -1]

from scipy import optimize

# optimize module minimizes functions whereas we would like to maximize the likelihood
# that's why I put the minus(-) sign
nll = lambda *args: -lnlike(*args)
result = optimize.minimize(nll, initial_params, args=(X, y, yerr))
m_ml, b_ml, wde_ml = result["x"]

print ('======================================')
print ('Maximum Likelihood Result')
print ('Omega_m = %.2f (%f)' %(m_ml, 0))
print ('Omega_lambda = %.2f (%f)' %(b_ml, 0))
print ('w_de = %.2f (%f)' %(wde_ml, 0))
print ('======================================')
```

    ======================================
    Maximum Likelihood Result
    Omega_m = 2.65 (0.000000)
    Omega_lambda = 3.72 (0.000000)
    w_de = -3.70 (0.000000)
    ======================================

The result is far from current known value. Therefore, we need to improve the fitting method.

## 4. Markov Chain Monte Carlo Fitting

In order to improve our fitting accuracy, we will use emcee, which is pure-Python implementation of Goodman & Weare's Affine Invariant Markov chain Monte Carlo (MCMC) Ensemble sampler. These pages (http://emcee.readthedocs.io/en/stable/) will show you how to use it. We would like to marginalize over some "nuisance parameters" and find an estimate of the posterior probability function (the distribution of parameters that is consistent with dataset) for others.

We will define uniform (so-called "uninformative") priors on initial parameters ($\Omega_{\rm m}$, $\Omega_{\Lambda}$, and $w_{\rm de}$). Then, combining this with the definition of lnlike from above, the full log-probability function can be calculated.

```python
# define prior for LambdaCDM cosmology model
def lnprior(theta):
    Omega_m, Omega_lambda, wde = theta

    if -1.05 <= wde <= -0.95 \
    and 0.95 <= Omega_m + Omega_lambda <= 1.05:
        return 0
    return -np.inf

# define the full probability
def lnprob(theta, X, y, yerr):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, X, y, yerr)
```

Next, we start by initializing the walkers in a tiny Gaussian ball around the maximum likelihood result and set up the sampler.

```python
# 3 parameters to fit and set up 100 walkers
ndim, nwalkers = 3, 500

# initiate walkers position around initial parameters
pos = [initial_params + 1e-3*np.random.randn(ndim) for i in range(nwalkers)]

import emcee
import sys

# set up the sampler
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(X, y, yerr), threads=3)

# number of iteration, along with progress bar width
nsteps = 1000
width = 30

print ('running MCMC.....')

# run the MCMC
for i, result in enumerate(sampler.sample(pos, iterations=nsteps)):
    n = int((width+1) * float(i) / nsteps)
    sys.stdout.write("\r[{0}{1}]".format('#' * n, ' ' * (width - n)))
sys.stdout.write("\n")

# take the sampler after 50 steps
samples = sampler.chain[:, 50:, :].reshape((-1, ndim))
```

    running MCMC.....
    [                              ]

    /home/irhamta/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: invalid value encountered in double_scalars
      if sys.path[0] == '':
    /home/irhamta/anaconda3/lib/python3.6/site-packages/scipy/integrate/quadpack.py:385: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      warnings.warn(msg, IntegrationWarning)

    [                              ]

    /home/irhamta/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: invalid value encountered in double_scalars
      if sys.path[0] == '':
    /home/irhamta/anaconda3/lib/python3.6/site-packages/scipy/integrate/quadpack.py:385: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      warnings.warn(msg, IntegrationWarning)

    [#                             ]

    /home/irhamta/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:12: RuntimeWarning: invalid value encountered in double_scalars
      if sys.path[0] == '':
    /home/irhamta/anaconda3/lib/python3.6/site-packages/scipy/integrate/quadpack.py:385: IntegrationWarning: The occurrence of roundoff error is detected, which prevents
      the requested tolerance from being achieved.  The error may be
      underestimated.
      warnings.warn(msg, IntegrationWarning)

    [##############################]

The walkers start in small distributions around the maximum likelihood values and then they quickly wander and start exploring the full posterior distribution. In fact, after fewer than 50 steps, the samples seem pretty well "burnt-in". That is a hard statement to make quantitatively but for now, we will just accept it and discard the initial 50 steps and flatten the chain so that we have a flat list of samples

Now that we have this list of samples, let's make one of the most useful plots you can make with your MCMC results: a corner plot.

```python
import corner

Omega_m = 0.307
Omega_r = 0 * 1e-5 # too small, so I just set it to zero
Omega_lambda = 1 - Omega_m
wde = -1

# plot parameter distributions
fig = corner.corner(samples, labels=["$\Omega_m$", "$\Omega_\Lambda$", "$w_{\\rm de}$"],
                      truths=[Omega_m, Omega_lambda, wde])

plt.savefig('result/lensing.png')
plt.show()

# obtain the value
m_mcmc, b_mcmc, wde_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]),
                             zip(*np.percentile(samples, [16, 50, 84],
                                                axis=0)))

print ('============================================================================')
print ('MCMC Result')
print ('Omega_m = ', m_mcmc)
print ('Omega_lambda = ', b_mcmc)
print ('w_de = ', wde_mcmc)
print ('============================================================================')

# save obtained parameters
output_data = pd.DataFrame({'omega_m': samples[:, 0],
                            'omega_l': samples[:, 1],
                            'wde'    : samples[:, 2]})
output_data.to_csv('result/output_lensing.csv', index=False)
```

As the final result, we obtained best parameters for $\Lambda$CDM cosmology as:

$\Omega_{\rm m} = 0.33_{-0.10}^{+0.09}$

$\Omega_\Lambda = 0.67_{-0.09}^{+0.10}$

$w_{\rm de} = -1.00_{-0.03}^{+0.04}$

which is consistent compared to Leaf & Melia (2018) result. Then, we can overplot the lensing contour result with other observations, such as Planck's CMB and supernova type Ia observation.

```python
# =============================================================================
# Plot Lensing Result
# =============================================================================

from scipy import ndimage

lensing_data = pd.read_csv('result/output_lensing.csv')

H_lensing, xbins_lensing, ybins_lensing = np.histogram2d(lensing_data['omega_m'],
                                                          lensing_data['omega_l'],
                                                          bins=100)

H_lensing = ndimage.gaussian_filter(H_lensing, 2)
sortH_lensing = np.sort(H_lensing.flatten())
cumH_lensing = sortH_lensing.cumsum()
lvl68_lensing = sortH_lensing[cumH_lensing > cumH_lensing.max()*0.32].min()
lvl95_lensing = sortH_lensing[cumH_lensing > cumH_lensing.max()*0.05].min()
lvl99_lensing = sortH_lensing[cumH_lensing > cumH_lensing.max()*0.003].min()

c1 = plt.contour(H_lensing.T, [lvl99_lensing, lvl95_lensing, lvl68_lensing], colors='b',
                 extent=(xbins_lensing[0], xbins_lensing[-1], ybins_lensing[0], ybins_lensing[-1]),
                 label='Lensing')

c1.collections[0].set_label('Lensing')

# =============================================================================
# Plot CMB data from Planck observation
# =============================================================================

# path to data planck data folder
path = './'

# MCMC chain samples
cmb_data = np.loadtxt(path + 'base_plikHM_TTTEEE_lowl_lowE_lensing_1.txt')

# load the column names for the samples
column_names = np.loadtxt(path + 'base_plikHM_TTTEEE_lowl_lowE_lensing.paramnames',
                          dtype=np.str, usecols=[0])

# make a data frame with column names and samples
# first two columns are not important
planck = pd.DataFrame(cmb_data[:,2:], columns=column_names)

# insert parameters to variable
planck_omegaM = planck['omegam*']
planck_omegaL = planck['omegal*']

H, xbins, ybins = np.histogram2d(planck_omegaM ,planck_omegaL, bins=100)

H = ndimage.gaussian_filter(H, 2)
sortH = np.sort(H.flatten())
cumH = sortH.cumsum()
lvl68 = sortH[cumH > cumH.max()*0.32].min()
lvl95 = sortH[cumH > cumH.max()*0.05].min()
lvl99 = sortH[cumH > cumH.max()*0.003].min()

c0 = plt.contour(H.T, [lvl99, lvl95, lvl68],
              colors='k', extent=(xbins[0], xbins[-1], ybins[0], ybins[-1]),
              label='Planck')
c0.collections[0].set_label('Planck-CMB')

# =============================================================================
# Plot Supernova Result
# =============================================================================

supernova_data = pd.read_csv('result/output_supernova.csv')

H_supernova, xbins_supernova, ybins_supernova = np.histogram2d(supernova_data['omega_m'],
                                                               supernova_data['omega_l'],
                                                               bins=100)

H_supernova = ndimage.gaussian_filter(H_supernova, 2)
sortH_supernova = np.sort(H_supernova.flatten())
cumH_supernova = sortH_supernova.cumsum()
lvl68_supernova = sortH_supernova[cumH_supernova > cumH_supernova.max()*0.32].min()
lvl95_supernova = sortH_supernova[cumH_supernova > cumH_supernova.max()*0.05].min()
lvl99_supernova = sortH_supernova[cumH_supernova > cumH_supernova.max()*0.003].min()

#c2 = plt.contour(H_supernova.T, [lvl99_supernova, lvl95_supernova, lvl68_supernova], colors='r',
#                 extent=(xbins_supernova[0], xbins_supernova[-1], ybins_supernova[0], ybins_supernova[-1]),
#                 label='Lensing')
#c2.collections[0].set_label('Supernova')

plt.xlabel('$\Omega_m$')
plt.ylabel('$\Omega_\Lambda$')
plt.legend()
plt.xlim(0.2, 0.45)
plt.ylim(0.6, 0.8)
```

```python

```


# Support Vector Machines

SVM hypothesis and cost function can be derived from logistic regression hypothesis. For logistic regression hypothesis is defined as:
$$
h_{\theta} = \frac{1}{1+e^{-\theta^{T}x}}
$$

where:
* if $y=1$, we want $h_{\theta}(x) \approx 1$, $\theta^T x \gg 0 $
* if $y=1$, we want $h_{\theta}(x) \approx 0$, $\theta^T x \ll 0 $

Cost for a single example can be written as:

$$
-\bigg(y log\big(h_{\theta}(x)\big) + (1 - y) log\big(1 - h_{\theta}(x)\big)\bigg) \\=\\ -y log\frac{1}{1+e^{-\theta^{T}x}} -(1-y)log(1 - \frac{1}{1+e^{-\theta^{T}x}})
$$

And finally, from here, the whole optimization process for regularized logistic regression is defined as:
$$
\underset{\theta}{\min} \frac{1}{m}\bigg[\sum_{i=1}^{m}y^{(i)}\big( -log h_{\theta}(x^{(i)})\big) + (1 - y^{(i)})\big( -log(1 - h_{\theta}(x^{(i)}))\big) \bigg] + \frac{\lambda}{2m}\sum_{j=1}^{n}\theta_{j}^{2}
$$
where $\lambda$ is the regularization parameter.

For SVMs the regularization parameter is $C$ and is defined as the inverse of $\lambda$. This parameter is now applied to the left portion of optimization term.

Instead of logarithmic cost, SVM's cost is linear (is shown in figure below the code in the next cell) and optimization for SVM is as follows:
$$
\underset{\theta}{\min} C\bigg[\sum_{i=1}^{m}y^{(i)}\big(cost_{1}(\theta^T x^{(i)})\big) + (1 - y^{(i)})\big(cost_{0}(\theta^T x^{(i)})\big) \bigg] + \frac{1}{2}\sum_{j=1}^{n}\theta_{j}^{2}
$$

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')
%matplotlib inline
from sklearn.svm import SVC
from scipy.io import loadmat
from sympy import Symbol, diff
```

```python
# domain
x1 = -3
x2 = 3
x = np.linspace(x1, x2, 100)

# logreg cost
c_lr = lambda x: - np.log(1/(1 + np.exp(-x)))
c_lr_vals = c_lr(x)

# 1st derivative of logreg cost
h = 0.01
diff1_approx = lambda x :(c_lr(x+h)-c_lr(x-h))/(2*h)
diff1_act = lambda x: -1/(1+np.exp(x))

# svm cost
k = -0.645 # manually configured slope
c_svm = [k * x - k if x<=1 else 0 for x in x]

# visualize
plt.figure(figsize=(12,8))
plt.plot(x, c_lr(x), label='logistic regression cost')
plt.plot(x, diff1_act(x), label='theoretical derivative of logreg cost')
plt.plot(x, diff1_approx(x), marker='x', markevery=5, linestyle='None', label='numerical derivative of logreg cost')
plt.plot(x, c_svm, label='svm cost')

plt.xlabel('x')
plt.ylabel('y')
plt.legend(loc='best')
plt.show()
```

## 1 Support vector machines - examples

### 1.1 Example dataset 1

In this example, 2D data should be separated by a linear boundary. Because of a single positive outlier at about (0.1, 4.1), regularization parameter C will drasticly affect the position of the linear boundary.

Informally, the C parameter is a positive value that controls the penalty for misclassified training examples; a large C parameter tells the SVM to try to classify **all** the examples correctly. It plays a role similar to $\frac{1}{\lambda}$, where $\lambda$ is the regularization parameter for regularized logistic regression.

```python
dataset1 = loadmat('data/ex6data1.mat')
print(dataset1['__header__'])
X = dataset1['X']
y = dataset1['y']
```

    b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Nov 13 14:28:43 2011'

```python
def plotData(X, y):
    pos = np.where(y==1)[0]
    neg = np.where(y==0)[0]
    plt.plot(X[pos, 0], X[pos, 1],
             marker='+',
             color='black',
             markersize=7,
             linestyle='None',
             label='pos examps')

    plt.plot(X[neg, 0], X[neg, 1],
             marker='o',
             color='red',
             markersize=7,
             linestyle='None',
             label='neg examps')
    plt.xlabel('x1')
    plt.ylabel('x2')

def trainSVM(X, y, C, kernel, tol, max_iter):
    return SVC(C, kernel=kernel, tol=tol, max_iter=max_iter)

def visualizeBoundary(X, y, model):
    # make classification predictions over a grid of values
    x1plot = np.linspace(np.min(X[:, 0]), np.max(X[:, 0]), num=100)
    x2plot = np.linspace(np.min(X[:, 1]), np.max(X[:, 1]), num=100)
    preds = np.zeros((len(x1plot), len(x2plot)))
    for i, x1 in enumerate(x1plot):
        for j, x2 in enumerate(x2plot):
            preds[i,j] = model.predict(np.array([x1, x2]).reshape(1, -1))

    #X1, X2 = np.meshgrid(x1plot, x2plot)
    contr = plt.contour(x1plot, x2plot, preds.T)
```

```python
plotData(X, y)
plt.legend(loc='lower left')
plt.show()
```

#### C = 1

```python
# train a model
C = 1
kernel = 'linear'
tol = 1e-3
max_iter=1000
model = trainSVM(X, y, C, kernel, tol, max_iter)
model.fit(X, y.ravel())
```

    SVC(C=1, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,
        decision_function_shape='ovr', degree=3, gamma='scale', kernel='linear',
        max_iter=1000, probability=False, random_state=None, shrinking=True,
        tol=0.001, verbose=False)

```python
# plot the SVM boundary
plotData(X, y)
visualizeBoundary(X, y, model)
plt.axis([np.min(X[:, 0])-0.2, np.max(X[:, 0])+0.2,
         np.min(X[:, 1])-0.2, np.max(X[:, 1])+0.2])
plt.legend(loc='lower left')
plt.show()
```

#### Changing the C parameter

```python
Cs = [0.1, 1, 10, 100]
for i, C in enumerate(Cs):
    kernel = 'linear'
    tol = 1e-2
    max_iter=1000
    model = trainSVM(X, y, C, kernel, tol, max_iter)
    model.fit(X, y.ravel())

    plt.figure(figsize=(12,9))
    plt.subplot(2,2,i+1)
    plotData(X, y)
    plt.axis([np.min(X[:, 0])-0.2, np.max(X[:, 0])+0.2,
             np.min(X[:, 1])-0.2, np.max(X[:, 1])+0.2])
    visualizeBoundary(X, y, model)
    plt.title(f'C = {C}')
    plt.legend(loc='lower left')
    plt.show()
```

### 1.2 SVM for non-linearly separable data

#### 1.2.1 Gaussian kernel

To find non-linear decision boundaries with the SVM, input features have to be mapped to higher dimension (in order to separate the data linearly with a hyperplane).

The most used mapping function is Gaussian kernel or radial basis function. Gaussian kernel is similarity function that measures the 'distance' between a pair of examples, $(x^{(i)}, x^{(j)})$. The Gaussian kernel is parametrized by a bandwidth parameter, $\sigma$, which determines how fast the similarity metrics decreases to 0 as the examples are further apart.
$$
K_{gasussian}(x^{(i)}, x^{(j)}) = exp(-\frac{||{x^{(i)} - x^{(j)}||}^2}{2\sigma^{2}} = exp\bigg(-\frac{\sum_{k=1}^{n}(x_{k}^{(i)} - x_{k}^{(j)})^2)}{2\sigma^2}\bigg)
$$

```python
def gaussianKernel(x1, x2, sigma):
    x1 = x1.flatten()
    x2 = x2.flatten()

    return np.exp(-np.sum((x1-x2)**2)/(2*sigma**2))
```

```python
x1 = np.array([1, 2, 1])
x2 = np.array([0, 4, -1])
sigma = 2

sim = gaussianKernel(x1, x2, sigma)
```

```python
print(f'Gaussian kernel between x1={x1} and x2={x2} with sigma={sigma} is {sim}')
print(f'Expected value ~ 0.324652')
```

    Gaussian kernel between x1=[1 2 1] and x2=[ 0  4 -1] with sigma=2 is 0.32465246735834974
    Expected value ~ 0.324652

#### 1.2.2 Example dataset 2

This dataset contains data without linear decision boundary that separates the positive and negative examples.

```python
dataset2 = loadmat('data/ex6data2.mat')
print(dataset2['__header__'])
X = dataset2['X']
y = dataset2['y']
```

    b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Nov 13 14:28:54 2011'

```python
plt.figure(figsize=(12, 8))
plotData(X, y)
plt.legend(loc='best')
plt.show()
```

```python
# rbf is ootb sklearn gaussian kernel
# instead of sigma it takes gamma=1/sigma^2 as parameter
C = 1
sigma = 0.1
model = SVC(C=C,
            kernel='rbf',
            gamma=sigma**(-2),
           )
model.fit(X, y.ravel())

plt.figure(figsize=(12, 8))
plotData(X, y)
plt.axis([np.min(X[:, 0])-0.05, np.max(X[:, 0])+0.05,
         np.min(X[:, 1])-0.05, np.max(X[:, 1])+0.05])
visualizeBoundary(X, y, model)
plt.legend(loc='lower left')
plt.show()
```

#### 1.2.2 Example dataset 2

In the provided dataset, *ex6data3.mat*, the given variables are $X$,
$y$, $Xval$, $yval$.
The task is to use the cross validation set to determine the best C and $\sigma$ parameter to use.

For both C and $\sigma$, suggested values are 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30. All possible pairs of values for C and $\sigma$ should be examined, where the total number of models for suggested list is 64 (8^2).

```python
dataset3 = loadmat('data/ex6data3.mat')
print(dataset3['__header__'])
X = dataset3['X']
y = dataset3['y']
Xval = dataset3['Xval']
yval = dataset3['yval']
```

    b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Nov 13 14:29:06 2011'

```python
plt.figure(figsize=(12, 8))
plotData(X, y)
plt.legend(loc='best')
plt.show()
```

```python
def cvParams(X, y, Xval, yval):
    params = [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30]

    opt_accuracy = 0
    opt_C, opt_sigma = 0, 0

    for i, C in enumerate(params):
        for j, sigma in enumerate(params):
            model = SVC(C, kernel='rbf', gamma=1/(sigma**2))
            model.fit(X, y.ravel())
            predictions = model.predict(Xval)
            accuracy = np.mean(predictions.ravel() == yval.ravel())
            if accuracy > opt_accuracy:
                opt_accuracy = accuracy
                opt_C = C
                opt_sigma = sigma
    return opt_C, opt_sigma, opt_accuracy
```

```python
C, sigma, acc = cvParams(X, y, Xval, yval)
model = SVC(C, kernel='rbf', gamma=1/(sigma**2))
model.fit(X, y.ravel())

plt.figure(figsize=(12, 8))
plotData(X, y)
plt.axis([np.min(X[:, 0])-0.05, np.max(X[:, 0])+0.05,
         np.min(X[:, 1])-0.05, np.max(X[:, 1])+0.05])
visualizeBoundary(X, y, model)
plt.legend(loc='lower left')
plt.title(f'accuracy = {acc*100}%')
plt.show()
```

## 2 Spam classification

TBA

```python

```


# Using the Symbolic Fresnel Routines

**Scott Prahl**

April 2019

```python
import sympy
import pypolar.sym_fresnel as sym_fresnel

sympy.init_printing(use_unicode=True)
```

```python
theta = sympy.Symbol('theta')
m = sympy.Symbol('m')
sym_fresnel.t_par_amplitude(m,theta)
```

```python

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Reflection and Transmission
***

Referring to Section 2.6, practical radar problems involve waves propagating in bounded regions and interacting with media of differing constitutive parameters.  These media may include the targets of interest to the radar system as well as other regions such as rain, buildings, trees, and birds.  Therefore, it is beneficial to study the reflection and refraction of electromagnetic waves occurring when a wave traveling in a given medium impinges on another medium with a different set of constitutive parameters.

The reflection and transmission coefficients for perpendicular polarization are given by (Equations 2.102 and 2.103)

\begin{align}
\Gamma_{{\scriptstyle TE}} &= \frac{\eta_2 \cos \theta_i - \eta_1 \cos \theta_t}{\eta_2 \cos \theta_i + \eta_1 \cos \theta_t},\\ \nonumber \\
\mathrm{T}_{{\scriptstyle TE}} &= \frac{2\, \eta_2 \cos \theta_i}{\eta_2 \cos \theta_i + \eta_1 \cos \theta_t}\nonumber
\end{align}

The reflection and transmission coefficients for parallel polarization are given by (Equations 2.114 and 2.115)

\begin{align}
\Gamma_{{\scriptstyle TM}} &= \frac{\eta_2 \cos \theta_t - \eta_1 \cos \theta_i}{\eta_2 \cos \theta_t + \eta_1 \cos \theta_i} \label{eq:parallel_reflection_coefficient},\\ \nonumber \\
\mathrm{T}_{{\scriptstyle TM}} &= \frac{2 \, \eta_2 \cos \theta_i}{\eta_2 \cos \theta_t + \eta_1 \cos \theta_i}.
\end{align}
***

Begin by getting library path

```python
import lib_path
```

Set the operating frequency (Hz), th relative permittivity, the relative permeability, and the conductivity (S/m) for the different regions using the `array` routine from `scipy`

```python
from numpy import array

frequency = 300e6

relative_permittivity = array([1.3, 2.8])

relative_permeability = array([1.0, 1.0])

conductivity = array([0.01, 0.01])
```

Set up the keyword args

```python
kwargs = {'frequency':              frequency,

          'relative_permittivity':  relative_permittivity,

          'relative_permeability':  relative_permeability,

          'conductivity':           conductivity}
```

Calculate the critical angle and Brewster angle using the `plane_waves` routine from `wave_propagation`

```python
from Libs.wave_propagation import plane_waves

critical_angle = plane_waves.critical_angle(**kwargs)

brewster_angle = plane_waves.brewster_angle(**kwargs)

# Display the results

print('Critical Angle {:.1f}'.format(critical_angle))

print('Brewster Angle {:.1f}'.format(brewster_angle))
```

    Critical Angle 81.1+51.0j
    Brewster Angle 54.8+3.0j

Set the incident angles using the `linspace` routine from `scipy`

```python
from scipy import linspace

from scipy.constants import pi

incident_angle = linspace(0., 0.5 * pi, 1000)
```

Set up the keyword args

```python
kwargs = {'frequency': frequency,

          'incident_angle': incident_angle,

          'relative_permittivity': relative_permittivity,

          'relative_permeability': relative_permeability,

          'conductivity': conductivity}
```

Calculate the reflection and transmission coefficients using the `plane_waves` routines from `wave_propagation`

```python
reflection_coefficient_te, transmission_coefficient_te, reflection_coefficient_tm, transmission_coefficient_tm = plane_waves.reflection_transmission(**kwargs)
```

Display the reflection and transmission coefficients using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

from scipy import degrees

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Set up the axes

fig, axes1 = plt.subplots()

axes2 = axes1.twinx()

# Display the reflection coefficients

axes1.plot(degrees(incident_angle), abs(reflection_coefficient_te), 'b', label='|$\Gamma_{TE}$|')

axes1.plot(degrees(incident_angle), abs(reflection_coefficient_tm), 'b--', label='|$\Gamma_{TM}$|')

# Display the transmission coefficients

axes2.plot(degrees(incident_angle), abs(transmission_coefficient_te), 'r', label='|$T_{TE}$|')

axes2.plot(degrees(incident_angle), abs(transmission_coefficient_tm), 'r--', label='|$T_{TM}$|')

# Set the plot title and labels

axes1.set_title('Plane Wave Reflection and Transmission', size=14)

axes1.set_xlabel('Incident Angle (degrees)', size=12)

axes1.set_ylabel('|Reflection Coefficient|', size=12)

axes2.set_ylabel('|Transmission Coefficient|', size=12)

# Set the tick label size

axes1.tick_params(labelsize=12)

axes2.tick_params(labelsize=12)

# Set the legend

axes1.legend(loc='upper right', prop={'size': 10})

axes2.legend(loc='upper left', prop={'size': 10})

# Turn on the grid

axes1.grid(linestyle=':', linewidth=0.5)
```