# PS7

In this problem set, we will use the symplest type of recurrent network, and Elman network, to try to learn a symple language..
# Instructions

Remember to do your problem set in Python 3. Fill in `#YOUR CODE HERE`.

Make sure:
- that all plots are scaled in such a way that you can see what is going on (while still respecting specific plotting instructions)
- that the general patterns are fairly represented.
- to label all x- and y-axes, and to include a title.

**N.B.** The ideas in this notebook draw heavily from the readings

 Elman, J. L. (1990). Finding structure in time. _Cognitive Science, 14_, 179-211.

McClelland, J., & Rumelhart, M. (1986). Past tenses of English verbs. In McClelland, J. and Rumelhart, D. (Eds.) _Parallel distributed processing: Explorations in the microstructure of cognition. Vol. 2: Applications_ (pp. 216-271). Cambridge, MA: MIT Press.

If you are confused about some of the ideas in this notebook or would like further clarification, we recommend having a look there.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

import warnings
warnings.filterwarnings('ignore')

```

## Introduction

One of the most successful (and controversial) applications of neural
        networks has been as models of human language. Specifically, contrary to the rules/symbols approach proposed by Chomsky, neural networks have been used to demonstrate that distributed representations can give rise to language learning. You will test whether a
simple neural network is capable of learning the rule underlying a
context-free language.

The language $a^nb^n$, being the set of all strings containing a
sequence of $a$'s followed by a sequence of $b$'s of the same length
is a simple example of a language that can be generated by a
context-free grammar but not a finite-state grammar (because a finite state grammar cannot keep track of the number of times a string has occurred in the sentence). Human languages
exhibit similar long-range constraints -- for example, a plural noun
at the start of a sentence affects conjugation of a verb at the end,
regardless of what intervenes. Some criticisms of applications of
neural networks to human languages are based upon their apparent
reliance on local sequential structure, which makes them seem much
more similar to finite-state grammars than to context-free
grammars. In other words, simple neural networks will mostly use the most recent word to predict the next one. An interesting question to explore is thus whether a
recurrent neural network can learn to generalize a simple rule
characterizing a long-range dependency, such as the rule underlying
$a^nb^n$.

Recall that an "Elman" network, as discussed by Elman (1990), is a
recurrent network where the activation of the hidden units at the
previous timestep are used as input to the hidden units on the current
timestep. This type of network architecture allows the
network to learn about sequential dependencies in the input data. In this notebook we will evaluate whether such a network can learn an $a^nb^n$
grammar. Here we formalize learning a grammar as being able to correctly
predict what the next item in a sequence should be given the
rules of the grammar. Therefore, the output node represents the
networks's prediction for what the next item in the sequence (the next
input) will be -- it outputs a $1$ if it thinks the current input will
be followed by an $a$, and outputs a $0$ if it thinks the current
input will be followed by a $b$.

## Q1. Data	[5pts, SOLO]
We will use the `abdata.npz` dataset for this problem. Make sure that the data file is in the same directory as your notebook while working on the problem set.

This dataset has two keys.
- The array `train_data` contains the sequence we will use to train our network.
- The array `test_data` contains the sequence we will use to evaluate the
network.

In both `train_data` and `test_data` a $1$ represents an $a$ and a $0$ represents a $b$.

`train_data` was constructed by concatenating a randomly ordered
set of strings of the form $a^nb^n$, with $n$ ranging from 1 to 11.
The frequency of sequences for a given value of $n$ in the training set
are given by `np.ceil(50/n)`, thus making them inversely proportional to $n$.
The `np.ceil` function returns the smallest integer greater or equal to
its input. For example, `np.ceil(3)` is 3, but `np.ceil(3.1)` is
4 . `test_data` contains an ordered sequence of strings of the form
$a^nb^n$, with $n$ increasing from 1 to 18 over the length of the
string.

Take a look at the data! You can first print the variables. Next plot the first 100 values of train_data and the first 100 values of test_data in two subplots. Make sure to label the axes and the titles. Upload your plot to gradescope as PS7_Q1.png

```python
ab_data = np.load("abdata.npz")
ab_data.keys()

# look at train_data
print('train_data')
train_data = ab_data['train_data']
print(train_data[:30])
print(len(train_data))

## look at test_data
print('test_data')
test_data = ab_data['test_data']
print(test_data)

## plot the data
#YOUR CODE HERE

figure.savefig('PS7_Q1.png')
```

## Q2. Input/output [3 pts HELP]
In order to train your network, you will need both training *input* and
training *output*.

That is, you need a sequence of inputs of the form
$a^nb^n$, and a corresponding sequence with the correct output for
each item in the input sequence.

For this problem we're going to use `train_data[:-1]` as the
input training sequence, and `train_data[1:]` as the output
training sequence.

Explain in gradescope why `train_data[:-1]` and `train_data[1:]` are appropriate input and output
sequences. If you're confused by what the sequences
`train_data[:-1]` and `train_data[1:]` look like,
try creating them in a cell and compare them to `train_data`.

## Elman network (provided)
We have provided you with a function, `train_Elman`, which takes four arguments:
- `input` -- the training input sequence
- `output` -- the training output sequence
- `num_hidden` -- the number of hidden units
- `num_iters` -- the number of training iterations: network needs to be trained on many iterations

`train_Elman` will:

1) create a network with one input node, the specified number of hidden units, and one output node

2) train the network on the training data for the specified number of iterations.

The network sees the
training data one input at a time (in our case, it sees a single $1$
or $0$ per time step).

```python
def train_Elman(inputs, outputs, num_hidden, num_iters):
    """
    Initializes and trains an Elman network. For details see Elman (1990).

    Parameters
    ----------
    inputs : numpy array
        A one dimensional sequence of input values to the network

    outputs : numpy array
        A one-dimensional sequence of desired output values for each of the
        items in inputs

    num_hidden : int
        The number of hidden units to use in the network

    num_iters : int
        The number of training iterations to run the network for

    Returns
    -------
    net : dict
        Dictionary object containing the trained network weights for each layer.
        Key 1 corresponds to the weights from the visibles to the hidden units,
        key 2 corresponds to the weights from the hiddens to the output units.

    NOTE: Poorly-Python-ported from trainElman.m, which in turn was adapted from
    code from http://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/matlab/
    recurrent/ which in turn was adapted from Elman (1990) :-)
    """
    np.random.seed(seed=1)

    # Parameters
    # increment to the derivative of the transfer function (Fahlman's trick)
    DerivIncr = 0.2
    Momentum  = 0.05
    LearnRate = 0.001

    num_input  = 1
    num_output = 1
    num_train  = inputs.shape[0]

    if inputs.ndim == 2:
        num_input  = inputs.shape[0]
        num_output = outputs.shape[0]
        num_train  = inputs.shape[1]

    if not all([outputs.ndim == inputs.ndim,
               inputs.shape[0] == outputs.shape[0]]):
        raise ValueError('unequal number of input and output examples')

    # create a dictionary to hold the network weights
    net = {}
    net[1] = np.random.rand(num_hidden, num_input + num_hidden + 1) - 0.5
    net[2] = np.random.rand(num_output, num_hidden + 1) - 0.5

    # the context layer
    # zeros because it is not active when the network starts
    Result1 = np.zeros((num_hidden, num_train))

    # the row of ones is the bias
    Inputs = np.vstack((inputs, np.ones(num_train)))
    Desired = outputs

    delta_w1 = 0.
    delta_w2 = 0.

    # Training
    for ii in range(num_iters):
        # Recurrent state
        # includes current inputs, as well as the output of the hidden layer
        # from the previous time step
        Input1 = np.vstack((Inputs, np.hstack([np.zeros((num_hidden,1)),  Result1[:,:-1]])))

        # Forward propagate activations
        # input --> hidden
        NetIn1 = np.dot(net[1], Input1)
        Result1 = np.tanh(NetIn1)

        # Hidden --> output
        # we again add a row of ones for bias
        Input2 = np.vstack((Result1, np.ones(num_train)))
        NetIn2 = np.dot(net[2], Input2)
        Result2 = np.tanh(NetIn2)

        # Backprop errors
        # output --> hidden
        Result2Error = Result2 - Desired
        In2Error = Result2Error * (DerivIncr + np.cosh(NetIn2)**(-2))

        # hidden --> input
        Result1Error = np.dot(net[2].T, In2Error)
        In1Error = Result1Error[:-1, :] * (DerivIncr + np.cosh(NetIn1)**(-2))

        # Calculate weight updates
        dw2 = np.dot(In2Error, Input2.T)
        dw1 = np.dot(In1Error, Input1.T)

        delta_w2 = -LearnRate * dw2 + Momentum * delta_w2
        delta_w1 = -LearnRate * dw1 + Momentum * delta_w1

        net[2] = net[2] + delta_w2
        net[1] = net[1] + delta_w1
    return net

```

## Q3 - learn anbn language [HELP 5 pts]
Complete the function `anbn_learner` below to train an "Elman"
network with two hidden units using the provided function `train_Elman` (remember
to use the input **train_data[:-1]** and output **train_data[1:]** sequences from Q2).

Train the network for *100 iterations*, and return the final output of the network.
We provide test cases. If you got the function right, the following cell should print "Success". Otherwise, it will give you an error message that will help with debugging.

Copy your code into gradescope.

```python
def anbn_learner(train_data):
    """
    Creates an "Elman" neural network with two hidden units and trains it
    on the provided data.

    Parameters
    ----------
    train_data: numpy array of shape (n,)
        the data on which to train the Elman network

    Returns
    -------
    net: dictionary with 2 keys
        a dictionary containing the weights of the network. Valid keys are 1 and 2.
        key 1 is for the weights between the input and the hidden units, and
        key 2 is for the weights between the hidden units and the output units.
    """
    # YOUR CODE HERE

```

```python
"""Check that anbn_learner returns the correct output"""
from numpy.testing import assert_array_equal
from nose.tools import assert_equal, assert_almost_equal

# check that abdata hasn't been modified
ab = np.load("data/abdata.npz")
assert_array_equal(test_data, ab['test_data'], "test_data array has changed")
assert_array_equal(train_data, ab['train_data'], "train_data array has changed")

# generate test data
traindata = np.zeros(20)
traindata[10:] = 1.

net = anbn_learner(traindata)

# check that net has the correct shape and type
assert_equal(type(net), dict, "net should be a dict of network weights")
assert_equal(len(net), 2, "incorrect number of layers in net")
assert_equal(list(net.keys()), [1,2], "keys for net should be 1 and 2")

# check the dimensions of the weight matrices
assert_equal(net[1].shape, (2,4), "invalid network weights for the input -> hidden layer")
assert_equal(net[2].shape, (1,3), "invalid network weights for the hidden -> output layer")

# check the weight matrix sums to the correct value on testdata
assert_almost_equal(np.sum(net[1]), -1.9326, places=4, msg="weights for input --> hidden layer are incorrect")
assert_almost_equal(np.sum(net[2]), 0.01825, places=4, msg="weights for hidden --> output layer are incorrect")
print("Success!")
```

## Q4 - checking the trained network [5 pts, solo]
Once the network is trained (*your anbn_learner function should pass the test case in the previous cell)*, you can test it on a new set of sequences
and evaluate its predictions to see how well it has learned the target
 grammar. To generate predictions from the trained network, we use the provided function `predict_Elman`.

Use your `anbn_learner` function on `train_data` to train a network, then use your trained network (by passing it as the first input to `predict_Elman` function) to predict the sequences in `test_data`.

A) The `predict_Elman` function returns an array of predicted values with the same dimensions as the input.Plot your prediction (sequence index number on x axis, predicted values on y axis) and upload the figure to gradescope as PS7_Q4.png.

B) Look back at your plot of the test data in Q1 - what do you think of the network's predictions? Do your predictions approximate the testing data? Explain why you think they do, or why you think they do not, by referring to the mechanisms of recurrent networks and the nature of the data.

```python

def predict_Elman(net, inputs):
    """
    Uses the Elman network parameterized by the weights in net to generate
    predictions for the elements in inputs.

    Parameters
    ----------
    net : dict
        Dictionary object containing the trained network weights and
        recurrent connections as produced by train_Elman. Key 1 corresponds
        to the weights from the visibles to the hidden units, key 2
        corresponds to the weights from the hiddens to the output units.

    inputs : numpy array
        A one dimensional sequence of input values to the network

    Returns
    -------
    outputs : numpy array
        An array containing the predictions generated by the Elman network for the
        items in inputs.

    NOTE: Poorly-Python-ported from predictElman.m, which in turn was adapted from
    code from http://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/matlab/
    recurrent/ which in turn was adapted from Elman (1990) :-)
    """
    num_output = 1
    num_hidden = net[1].shape[0]
    num_train  = inputs.shape[0]

    if inputs.ndim == 2:
        num_train = inputs.shape[1]

    Inputs = np.vstack((inputs, np.ones([1, num_train])))
    Result1 = np.zeros([num_hidden, 1])

    outputs = np.zeros(num_train)

    for i in range(num_train):
        Input1 = np.append(Inputs[:, i], Result1)
        NetIn1 = np.dot(net[1], Input1)
        Result1 = np.tanh(NetIn1)

        Input2 = np.append(Result1, np.ones((1, 1)))
        NetIn2 = np.dot(net[2], Input2)
        outputs[i] = np.tanh(NetIn2)
    return outputs

```

```python
## YOUR CODE HERE

figure.savefig('PS7_Q4.png')

```

## Q5 - Quantifying the model performance [2pts, HELP]

How well does the network do at predicting the next letter? Has it learned the language? Let's look more carefully.

To quantify how well the network performs we are going to look at how much the predicted sequence deviates from expectations. The squared error (SE) for a prediction $p_i$ in the prediction vector ${\bf p}$
compared to a target value ${y_i}$ in the target vector ${\bf y}$ is

\begin{equation}
SE_i = (p_i-y_i)^2
\end{equation}

That is, the squared error is just the squared difference between the
predicted and target value.

Complete the function `squared_error`, which takes in an array of test data and an array of
predictions. The function should return an error array **with the same number of elements as the test data**, containing the SE for each
of the predictions of the network compared against the corresponding value in `test_data`.

Remember that the predictions refer to the _next_ item in the sequence
(e.g.  `predictions[0]` should be compared to
`test_data[1]`, etc.). You should append an $a$ (coded as a $1$) to the end of your test data to equate the array sizes (describing the start of a new sequence of $a^nb^n$).

We have provided test cases again in the cell below. If your function is right, it should print success. Otherwise, it will give you an error with feedback.

Copy your function into gradescope.

```python
def squared_error(predictions, test_data):
    """
    Uses equation 1 to compute the SE for each of the predictions made
    by the network.

    Parameters
    ----------
    predictions: numpy array of shape (n,)
        an array of predictions from the Elman network

    test_data: numpy array of shape (n,)
        the array of test data from which predictions were generated

    Returns
    -------
    se_vector: numpy array of shape (n,)
        an array containing the SE for each of items in predictions
    """
    # YOUR CODE HERE

```

```python
#YOUR OWN TESTS HERE
```

```python
"""Check that squared_error returns the correct output"""
from nose.tools import assert_equal

# generate test data
pred = np.array([1, 0, 1])
test = np.array([0, 1, 0])
se = squared_error(pred, test)

# check that squared_error returns the correct output for testdata
#assert_equal(se.dtype, np.float64, "squared_error should return an array of floats")
assert_equal(se.shape, (3,), "squared_error returned an array of the incorrect size on the validate testdata")
assert_array_equal(se, np.zeros(3), "squared_error should return all zeros on the validate testdata")

# check that squared_error compares the correct elements
pred = np.zeros(1)
test = np.zeros(1)
se = squared_error(pred, test)
assert_equal(se, np.ones(1), "squared_error([0],[0]) should have returned a 1 (did you remember to append an a to testdata?")

print("Success!")
```

---

## Q6 [5 pts, SOLO]
Train the network on the train_data, then apply it to the test_data: try to predict test data using the weights of the trained network. Measure the resulting squared error.

Use matplotlib to plot a bar graph of the squared error for each training example. You should have the sequence iteration number on the x axis, and the error values on y axis. Don't forget to provide a title and labels your $x$ and $y$ axes!

If you have difficulty interpreting this graph, you may want to
examine a few of the values in `test_data`, `predictions`, and your `mse_vector` to see how they are related.
Upload your figure PS7_Q6.png to gradescope.

```python

# YOUR CODE HERE

# create the figure
fig, axis = plt.subplots()
axis.set_xlim([0.0, 350.0]), axis.set_ylim([0.0,.7])

# YOUR CODE HERE
axis.bar(np.arange(len(se_vector)),se_vector)
```

## Q7 [SOLO, 5pts]

To get a better idea of what is going on, let's have a look at the specific values in `test_data` where the prediction error spikes. Use the provided code below and look at its output.

At which points in test_data do the large errors occur? Explain which parts of the data the network is predicting ok, and which parts it's not predicting well. Answer in gradescope.

```python
# prints the 3 values preceding and 2 values following the spot where
# the prediction error >= 0.5
error_spike_idxs = np.argwhere(se_vector >= 0.5) + 1
error_spike_idxs = error_spike_idxs[:-1]

for i in error_spike_idxs:
    print('3 values preceding MSE spike: {}\tValue at MSE spike: {}'
          '\t\t2 values following MSE spike: {}'\
          .format(test_data[i[0]-3:i[0]], test_data[i[0]], test_data[i[0]+1:i[0]+3]))
```

## Q8.1. Conclusions [SOLO 5 pts]

Earlier we said that we can evaluate whether the network has learned
the grammar by looking at the predictions it makes. If the network has
learned the $a^nb^n$ grammar, in what cases should it make correct
predictions? When should it make incorrect predictions?

Do your predictions about when the network should make correct/incorrect predictions if it has learned the $a^nb^n$ grammar match the the times when the network makes large errors, as identified in Q7? Did the network learn the $a^nb^n$ language?

---

## Q8.2. Conclusion [SOLO 5pts]
At what level of the Chomsky hierarchy is the $a^nb^n$ grammar?

How does this compare to the level of most natural languages? Specifically, what level of Chomsky's hierarchy describes most of the natural languages, and are these levels higher relative to that of $a^nb^n$?

Use this to explore the implications of your results from Q7 and Q8.1 for using the Elman network to model the relationships present in human language. Is the Elman network likely to be sufficient to capture human language satisfactorily?

# Submission

### Attention! Code submission requirement!

When you're done with your problem set, do the following:
- Upload your answers in Gradescope's PS7.
- Convert your Jupyter Notebook into a `.py` file by doing so:

- Submit the `.py` file you just created in Gradescope's PS7-code.

---


```python
import pandas as pd
import numpy as np
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import xlrd
import seaborn as sns
from scipy.optimize import fsolve
from sympy import Eq, Symbol, solve
from scipy import stats
import xlrd
from scipy import signal
from matplotlib.ticker import (MultipleLocator, AutoMinorLocator)
from matplotlib import rc, rcParams
%matplotlib inline
```

# Event 1

```python
df1=pd.read_excel("2003Nov20.xlsx")
df1.replace([9999999,99999.9,9999.99,999.99,99.99,99999],np.nan,inplace=True)

nan_rows = df1[df1.isnull().any(1)]
nan_rows
```

      Year
      Day
      Hour
      Min
      Bz
      Vsw
      Nsw
      Psw
      AE
      SYM-H

      9
      2003
      324
      0
      9
      1.33
      NaN
      NaN
      NaN
      31
      0

      17
      2003
      324
      0
      17
      -2.52
      NaN
      NaN
      NaN
      33
      -1

      25
      2003
      324
      0
      25
      0.06
      NaN
      NaN
      NaN
      35
      -2

      41
      2003
      324
      0
      41
      -0.52
      NaN
      NaN
      NaN
      34
      -2

      52
      2003
      324
      0
      52
      NaN
      443.2
      4.56
      1.79
      40
      -3

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1305
      2003
      324
      21
      45
      -16.79
      NaN
      NaN
      NaN
      1022
      -390

      1314
      2003
      324
      21
      54
      -14.62
      NaN
      NaN
      NaN
      1057
      -388

      1324
      2003
      324
      22
      4
      NaN
      NaN
      NaN
      NaN
      495
      -398

      1325
      2003
      324
      22
      5
      NaN
      NaN
      NaN
      NaN
      490
      -398

      1389
      2003
      324
      23
      9
      -11.87
      NaN
      NaN
      NaN
      746
      -334

126 rows × 10 columns

```python
df1.to_csv("2003-11-20.csv",index=False)
```

```python

df2=df1[["Bz","Vsw","Nsw","AE","SYM-H"]]
df2.columns=["bz","vsw","nsw","ae","sym"]

rc('font', weight='bold')
rcParams['text.latex.preamble'] = [r'\usepackage{sfmath} \boldmath']

import warnings
warnings.filterwarnings("ignore")
sns.set_style(style="ticks")
sns.set_context("paper",font_scale=1.5, rc={"lines.linewidth": 2})

fig, axs = plt.subplots(5, sharex=True,
                        gridspec_kw={'hspace': 0.2},constrained_layout=True,
                        figsize=(16,9))
fig.suptitle('OMNI Parameters on 20 November 2003',color="black",
            x=0.5,y=1.03,fontsize=22,fontweight="bold")
tick=np.arange(0,1440+60,60)
color=["yellow","white",'black']
colors=["magenta","black","red","green","orange"]
#hourly_ticks =  30 * np.arange(48)
cc=['$B_{\\rmz}^{\\rmGSM}$\n(nT)',
    '$V_{\\rmsw}$\n(km/s)','Nsw\n(n/cc)','AE\n(nT)','SYM-H\n(nT)']
for i,c in enumerate(df2.columns):
    axs[i].plot(df2.index, df2[c],color=colors[i],alpha=1.0)
    #axs[i].plot(df1.index, df1[c],color="black",alpha=1.0,label="1-Min data")
    #if i==0:
        #axs[i].plot(hourly_ticks,x["bz"],color="red",alpha=1.0)
        #axs[i].plot(hourly_ticks,x["bz"],color="red",alpha=1.0,label="30-Min Average")
    #if i==4:
        #axs[i].plot(dfnew.rolling(30, center=True).mean()["sym"])
        #axs[i].plot(hourly_ticks,x["sym"],color="red",alpha=1.0)
    #axs[i].legend()
    if i==1:
        ax2=axs[i].twinx()
        p1,=ax2.plot(df1.index,df1["Psw"],linestyle="-",color="blue")
        tkw = dict(size=4, width=1.5)
        ax2.tick_params(axis='y', colors=p1.get_color(),labelsize=14, **tkw)
        ax2.set_ylim([0,30])
        ax2.set_yticks([0,10,20,30])
        ax2.yaxis.label.set_color(p1.get_color())
        ax2.set_ylabel("Psw\n(nPa)",fontsize=22,labelpad=5,fontweight="bold")
        ax2.tick_params(which='major', length=8,top=True, right=True,direction="in",labelsize=20)
        ax2.grid(axis="y",linestyle='--',color="blue",alpha=0.5)

    #ylabel management
    if i==0:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2],fontweight="bold")
    elif i==2:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2],fontweight="bold")
    elif i==1:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2],fontweight="bold")
    elif i==3:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=15,color=color[2],fontweight="bold")
    else:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=15,color=color[2],fontweight="bold")

#grid
    axs[i].grid(axis="both",linestyle='--',linewidth=1.5)

#xlim
    axs[i].set_xlim([0,1440])

#xticks and xticks label
    axs[i].set_xticks(tick)
    #axs[i].set_xticklabels(["\n00:00\n03-Aug", "02:00", "04:00", "06:00","08:00", "10:00","12:00",
     #                      "14:00", "16:00", "18:00", "20:00", "22:00", "\n00:00"],
      #                    rotation=0)
    axs[i].set_xticklabels(["0\n20-Nov", "1", "2", "3","4", "5","6",
                           "7", "8", "9", "10", "11","12","13","14","15","16","17","18","19","20",
                            "21","22","23", "0\n21-Nov"],
                          rotation=0)
    #axs[i].set_xticklabels(np.arange(0,25))
    axs[i].set_xlabel('Time (UT)',fontsize=20,color=color[2],labelpad=-5,fontweight="bold")
    tkw = dict(size=6, width=2,labelcolor=color[2])
    axs[i].tick_params(axis='y', labelsize=20,**tkw,direction="in")
    axs[i].tick_params(axis='x',labelsize=20, **tkw,direction="in")

#
#ylim of each subplot
    if i==0:
        axs[i].set_ylim([-60,40])
        axs[i].set_yticks(np.arange(-60,60,20))

    if i==1:
        axs[i].set_ylim([400,800])
        axs[i].set_yticks(np.arange(400,900,100))
    if i==2:
        axs[i].set_ylim([0,30])
        axs[i].set_yticks(np.arange(0,40,10))
    if i==3:
        axs[i].set_ylim([0,3000])
        axs[i].set_yticks(np.arange(0,4000,1000))
    if i==4:
        axs[i].set_ylim([-500,100])
        axs[i].set_yticks(np.arange(-500,300,200))

        req=dict(size=20,color=color[2])
        axs[i].text(660,-300,"Main Phase",rotation=0,**req)
        axs[i].text(490,-430,"Initial",rotation=90,**req)
        axs[i].text(1110,-300,"Recovery Phase",rotation=0,**req)
        #axs[i].axhline(0)
    #axs.xaxis.set_minor_locator(AutoMinorLocator(6))

    axs[i].xaxis.set_minor_locator(MultipleLocator(60))
    axs[i].tick_params(which='major', length=4,labelcolor=color[2],direction="in")
    axs[i].tick_params(which='minor', length=4,labelcolor=color[2],direction="in")
    #axs[i].minorticks_on()
    for xc in [486,536,1097]:
            axs[i].axvline(x=xc,ymin=0,ymax=1,ls='--',color=color[2],linewidth=2,zorder=0, clip_on=False)

#label outer
for ax in axs:
    ax.label_outer()

#figure description
fig.tight_layout(pad=2.0)
fig.savefig("2003paper.png",dpi=300,bbox_inches='tight')

```

```python

```

```python

```

# Event 2

```python
df=pd.read_excel("2015June22.xlsx")
df.replace([9999999,99999.9,9999.99,999.99,99.99,99999],np.nan,inplace=True)

nan_rows = df[df.isnull().any(1)]
nan_rows
```

      Year
      Day
      Hour
      Min
      Bz
      Vsw
      Nsw
      Psw
      AE
      SYM-H

      11
      2015
      173
      0
      11
      -0.22
      NaN
      NaN
      NaN
      57
      9

      12
      2015
      173
      0
      12
      -4.89
      NaN
      NaN
      NaN
      57
      9

      13
      2015
      173
      0
      13
      -4.83
      NaN
      NaN
      NaN
      58
      9

      17
      2015
      173
      0
      17
      -5.30
      NaN
      NaN
      NaN
      73
      9

      18
      2015
      173
      0
      18
      -4.76
      NaN
      NaN
      NaN
      74
      9

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1292
      2015
      173
      21
      32
      19.80
      NaN
      NaN
      NaN
      1404
      -85

      1300
      2015
      173
      21
      40
      3.27
      NaN
      NaN
      NaN
      1012
      -91

      1301
      2015
      173
      21
      41
      4.44
      NaN
      NaN
      NaN
      925
      -93

      1404
      2015
      173
      23
      24
      9.27
      NaN
      NaN
      NaN
      292
      -98

      1405
      2015
      173
      23
      25
      10.76
      NaN
      NaN
      NaN
      291
      -98

199 rows × 10 columns

```python
df.to_csv("2015-06-22.csv",index=False)
```

```python

df3=df[["Bz","Vsw","Nsw","AE","SYM-H"]]
df3.columns=["bz","vsw","nsw","ae","sym"]

import warnings
warnings.filterwarnings("ignore")
sns.set_style(style="ticks")
sns.set_context("paper",font_scale=1.5, rc={"lines.linewidth": 2})
rc('font', weight='bold')
rcParams['text.latex.preamble'] = [r'\usepackage{sfmath} \boldmath']
fig, axs = plt.subplots(5, sharex=True,
                        gridspec_kw={'hspace': 0.2},constrained_layout=True,
                        figsize=(16,9))
fig.suptitle('OMNI Parameters on 22 June 2015',color="black",
            x=0.5,y=1.05,fontsize=22,fontweight="bold")
tick=np.arange(0,1440+60,60)
color=["yellow","white","black"]
colors=["magenta","black","red","green","orange"]
#hourly_ticks =  30 * np.arange(48)
cc=['$B_{\\rmz}^{\\rmGSM}$\n(nT)',
    '$V_{\\rmsw}$\n(km/s)','Nsw\n(n/cc)','AE\n(nT)','SYM-H\n(nT)']
for i,c in enumerate(df3.columns):
    axs[i].plot(df3.index, df3[c],color=colors[i],alpha=1.0)
    #axs[i].plot(df1.index, df1[c],color="black",alpha=1.0,label="1-Min data")
    #if i==0:
        #axs[i].plot(hourly_ticks,x["bz"],color="red",alpha=1.0)
        #axs[i].plot(hourly_ticks,x["bz"],color="red",alpha=1.0,label="30-Min Average")
    #if i==4:
        #axs[i].plot(dfnew.rolling(30, center=True).mean()["sym"])
        #axs[i].plot(hourly_ticks,x["sym"],color="red",alpha=1.0)
    #axs[i].legend()
    if i==1:
        ax2=axs[i].twinx()
        p1,=ax2.plot(df3.index,df["Psw"],linestyle="-",color="blue")
        tkw = dict(size=4, width=1.5)
        ax2.tick_params(axis='y', colors=p1.get_color(),labelsize=14, **tkw)
        ax2.set_ylim([0,60])
        ax2.set_yticks(np.arange(0,80,20))
        ax2.yaxis.label.set_color(p1.get_color())
        ax2.set_ylabel("Psw\n(nPa)",fontsize=22,labelpad=5)
        ax2.tick_params(which='major', length=8,top=True, right=True,direction="in",labelsize=20)
        ax2.grid(axis="y",linestyle='--',color="blue",alpha=0.5)

    #ylabel management
    if i==0:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2])
        req=dict(size=20,color='green')
        axs[i].text(330,45,"1st CME",rotation=0,**req)
        axs[i].text(1050,45,"2nd CME",rotation=0,**req)
    elif i==2:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2],fontweight="bold")
    elif i==1:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=20,color=color[2],fontweight="bold")
    elif i==3:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=15,color=color[2],fontweight="bold")
    else:
        axs[i].set_ylabel(cc[i],fontsize=22,labelpad=15,color=color[2],fontweight="bold")

#grid
    axs[i].grid(axis="both",linestyle='--',linewidth=1.5)

#xlim
    axs[i].set_xlim([0,1440])

#xticks and xticks label
    axs[i].set_xticks(tick)
    #axs[i].set_xticklabels(["\n00:00\n03-Aug", "02:00", "04:00", "06:00","08:00", "10:00","12:00",
     #                      "14:00", "16:00", "18:00", "20:00", "22:00", "\n00:00"],
      #                    rotation=0)
    axs[i].set_xticklabels(["0\n22-June", "1", "2", "3","4", "5","6",
                           "7", "8", "9", "10", "11","12","13","14","15","16","17","18","19","20",
                            "21","22","23", "0\n23-June"],
                          rotation=0)
    #axs[i].set_xticklabels(np.arange(0,25))
    axs[i].set_xlabel('Time (UT)',fontsize=20,color=color[2],labelpad=-5,fontweight="bold")
    tkw = dict(size=6, width=2,labelcolor=color[2])
    axs[i].tick_params(axis='y', labelsize=20,**tkw,direction="in")
    axs[i].tick_params(axis='x',labelsize=20, **tkw,direction="in")

#
#ylim of each subplot
    if i==0:
        axs[i].set_ylim([-40,40])
        axs[i].set_yticks(np.arange(-40,60,20))

    if i==1:
        axs[i].set_ylim([200,800])
        axs[i].set_yticks(np.arange(200,1000,200))
    if i==2:
        axs[i].set_ylim([0,80])
        axs[i].set_yticks(np.arange(0,100,20))
    if i==3:
        axs[i].set_ylim([0,3000])
        axs[i].set_yticks(np.arange(0,4000,1000))
    if i==4:
        axs[i].set_ylim([-150,100])
        axs[i].set_yticks(np.arange(-150,250,100))

        req=dict(size=20,color=color[2])
        axs[i].text(1165,0,"Main",rotation=90,**req)
        axs[i].text(1120,-360, "Initial",rotation=90,**req)
        axs[i].text(1260,-40,"Recovery\nPhase",rotation=0,**req)
        #axs[i].axhline(0)
    #axs.xaxis.set_minor_locator(AutoMinorLocator(6))

    axs[i].xaxis.set_minor_locator(MultipleLocator(60))
    axs[i].tick_params(which='major', length=4,labelcolor=color[2],direction="in")
    axs[i].tick_params(which='minor', length=4,labelcolor=color[2],direction="in")
    #axs[i].minorticks_on()
    for xc in [1117,1155,1217]:
            axs[i].axvline(x=xc,ymin=0,ymax=1,ls='--',color=color[2],linewidth=1.5,zorder=0, clip_on=False)

    axs[i].axvline(x=346,ymin=0,ymax=1,ls='-',color='green',linewidth=1.5,zorder=0, clip_on=False)
    axs[i].axvline(x=1117,ymin=0,ymax=1,ls='-',color='green',linewidth=1.5,zorder=0, clip_on=False)

#label outer
for ax in axs:
    ax.label_outer()

#figure description
fig.tight_layout(pad=2.0)
fig.savefig("2015paper.png",dpi=300,bbox_inches='tight')

```

```python

```

```python

```


\title{Upsampling and Downsampling in myHDL}
\author{Steven K Armour}
\maketitle

# Python Libraries Utilized

```python
import numpy as np
import scipy.signal as sig
import pandas as pd

from sympy import *
init_printing()
from IPython.display import display, Math, Latex

from myhdl import *
from myhdlpeek import Peeker

import matplotlib.pyplot as plt
%matplotlib inline
```

# Acknowledgments
The orgianl Interpolation Decimation componetswritten in myHDL was done by [Christopher Felton](https://github.com/cfelton)
(myHDL.old version [here](https://github.com/jandecaluwe/site-myhdl-retired/blob/master/_ori/pages/projects/gciccomplete.txt))

Author of myHDL [Jan Decaluwe](http://www.myhdl.org/users/jandecaluwe.html) and the author of the myHDL Peeker [XESS Corp.](https://github.com/xesscorp/myhdlpeek)

# Refrances

# Down Sampling via Decimation

In DownSmapling by decmiation the input signal $x(n)$ is feed into the downsampler and the output is then shifted to the downshifted signal $x_d(Dn)$ where the down sampleing factor $D\geq$. In a analog time system the Downsampling action would be intupirted to be exsapnsive time scaling. In the discrate space downsampleing a alreaddy sampled signal is the action of skiping every $D$ inputs to the dowmsample. Effectivly removing, aka decmating, those inputs by the action of skiping them

```python
TestValues=pd.read_pickle('SinGen16.pkl')
TestValues.head(16)
plt.stem(TestValues['GenValue'])
None
```

```python
1/TestValues['Time[s]'][1]
```

```python

```

The test data that is being used has 200 samples in 20$mu$ sec with a sampling rate of 10 MHz. We can see what the result of downsampling will be via the following

```python
D=5
plt.stem(TestValues['GenValue'][::D])
```

```python
plt.stem(TestValues['Time[s]'][::D], TestValues['GenValue'][::D])
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
```

by the nyqest condidtion that the lowest sampling rate must be $f_s>2\max{f_a}$ and by using a downsampler the myquest condtion becomes $f_s>2D\max{f_a}$. Therefore the upper bound of D in order to not incurre the penility of ailissing is $D<\dfrac{f_s}{2\max{f_a}}$ where ofcource $D$ is an integer satifying affor mentioned condidtion

# myHDL implimintaion of a Decimator

in python we simpling use slicing to performthe action of Decimation blindly, however in order to implimint this at the hardware level the layers of aprstarion in slicing must be peeled back

When the slice ```x[::D]``` is taken we are telling python to return the nth value after egnoring D intermited values. How python (or any other programing language does this is) is by estaplishing a counter that counts eather up to and down from the the skip factor, returing the now nth value and then reseting that counter and performing that action to a end condition is meet. In order to do this via a single HDL object it will have be broken down into two internal componets. One componet is the afored mentioned counter and the other is the counter watcher that allows the nth input to pass through it if counter reaches the skip number D.

```python
def Decimator(x_in, y_out,  clk, rst, D_parm=2):
    """
    Parameters:
    -----------
    D_parm: The Decimation (downsampling factor) value in base 10

    Ports:
    ------
    x_in:    2's compliment input
    y_out:    2's compliment output
    clk:  System clock, must be equal to or greater than the max
          sample rate after interpolation or before decimation
    rst:  System reset
    """

    D_base2= int(np.ceil(np.log2(D_parm)))
    Count=Signal(intbv(0, max=2**D_base2))

    @always(clk.posedge)
    def counter():
        if rst:
            Count.next=0
        else:
            #recyle the count
            if Count==D_parm-1:
                Count.next=0
            #up incrimite counter upwards to D_parm
            else:
                Count.next=Count+1

    @always(clk.posedge)
    def action():
        if rst:
            y_out.next=0
        else:
            #counter has terminated
            if Count==0:
                y_out.next=x_in
            else:
                y_out.next=0

    return instances()
```

```python
Peeker.clear()

x_in=Signal(intbv(0, max=TestValues['GenValue'].max()+1,
                  min=TestValues['GenValue'].min()-1))
Peeker(x_in, 'x_in')

y_out=Signal(intbv(0, max=TestValues['GenValue'].max()+1,
                   min=TestValues['GenValue'].min()-1))
Peeker(y_out, 'y_out')
DecimatorTracker=[]

clk=Signal(bool(0)); Peeker(clk, 'clk')
rst=Signal(bool(0)); Peeker(rst, 'rst')

DUT=Decimator(x_in=x_in, y_out=y_out, clk=clk, rst=rst, D_parm=5)

def Decimator_TB():
    TestValueGen=TestValues['GenValue'].iteritems()

    @always(delay(1))
    def clkgen():
        clk.next = not clk

    @instance
    def stimules():
        for step, val in TestValueGen:
            x_in.next=int(val)
            if step==50:
                rst.next=True
            else:
                rst.next=False

            DecimatorTracker.append(int(y_out))
            yield clk.negedge

        raise StopSimulation

    return instances()

```

```python
sim = Simulation(DUT, Decimator_TB(), *Peeker.instances()).run()
DecimatorTracker=np.array(DecimatorTracker)
Peeker.to_wavedrom(start_time=40, stop_time=60, tock=True)
```

```python
plt.stem(DecimatorTracker)
```

```python
plt.stem(DecimatorTracker[np.nonzero(DecimatorTracker)])
```

# Upsampling via Interpolation

Unlike downsampling that simply ignores input values via a built in counter and passing zeros inplace of the ignored values. Upsampleing must add intermidite values between the given input. The travial way is padding a zeros between the inputs by the upsampling factor $U$; and wile this does acomplish the act of incrassing the samples all that has been done is zero padding. This can be seen as follows for a upsampling of $U=5$

```python
original=list(np.array(TestValues['GenValue']))
```

```python
U=5
zeroPadding= [0]*((U+1)*len(original)-1)
zeroPadding[::U+1]=original; InterpoloteData=pd.DataFrame(zeroPadding, columns=['GenValue'])
plt.stem(InterpoloteData['GenValue'])
```

```python
InterpoloteData.head((U+2)*2)
```

      GenValue

      0
      29196

      1
      0

      2
      0

      3
      0

      4
      0

      5
      0

      6
      19259

      7
      0

      8
      0

      9
      0

      10
      0

      11
      0

      12
      5123

      13
      0

```python
zeroPadding=np.array(zeroPadding)
zeroPadding[zeroPadding==0]=np.nan
zeroPadding[0]=0
InterpoloteData=pd.DataFrame(zeroPadding, columns=['GenValue'])
```

as can be seen this zero intermidite padding has not done much but to incress the number of samples. In order to make this non trival the intermited padded values must take on a value other than zero. We can accomplish this by means of interpolation by the sample and hold method. Wich happens to be rectanlge impulse function in the discrate time domain and in frequancy domn becomes the sinc function (more detail explination [here](https://en.wikipedia.org/wiki/Zero-order_hold) & [here](https://en.wikipedia.org/wiki/Rectangular_function))

```python
InterpoloteData['GenValue'].interpolate(method='zero', inplace=True)
```

```python
plt.stem(InterpoloteData['GenValue'])
```

as shown by using a zero order hold interpolation we can artificsly increass the sampling; and by artfical it we can see clearly that the outcome of the interpolation is not a perfect sinusiode

```python
plt.stem(InterpoloteData['GenValue'][:200])
```

```python

```


```
%load_ext autoreload
```

```
autoreload 2
```

```
%matplotlib inline
```

```
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym

import inputs
import models
import shooting

from __future__ import division
```

```
# define some workers skill
x, a, b = sym.var('x, a, b')
skill_cdf = (x - a) / (b - a)
skill_params = {'a': 1.0, 'b': 2.0}
skill_bounds = [skill_params['a'], skill_params['b']]

workers = inputs.Input(var=x,
                       cdf=skill_cdf,
                       params=skill_params,
                       bounds=skill_bounds,
                       )

# define some firms
y = sym.var('y')
productivity_cdf = (y - a) / (b - a)
productivity_params = skill_params
productivity_bounds = skill_bounds

firms = inputs.Input(var=y,
                     cdf=productivity_cdf,
                     params=productivity_params,
                     bounds=productivity_bounds,
                     )
```

```
# Developped country

R, l, r, rho, gamma, eta, L, A, kapa = sym.var('R, l, r, rho, gamma, eta, L, A, kapa')

# First set k to the optimum (look at Capital notebook)
k_star = (2*6**(-0.5)*(A*eta**2*kapa*x**0.5/R)**0.5*sym.cos(0.333333333333333*sym.acos(3*6**0.5*x**(-0.5)*(l*x*y/r)**0.25*(R*x**(-0.5)/(A*eta**2*kapa))**0.5*(-eta + 1)/(2*eta))))**(4)

F = A*kapa*(eta*(k_star*x)**rho + (1- eta)*(y*(l/r))**rho)**(gamma/rho)

F_params = {'R':0.13099, 'rho':0.25, 'gamma':0.5, 'eta':0.89, 'L':4.2, 'A': 1.0, 'kapa':1.0}
```

Checking capital...

```
# Try out positive
model = models.Model('positive',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = shooting.ShootingSolver(model=model)
```

```
solver.solve(10, tol=1e-6, number_knots=100, atol=1e-15, rtol=1e-12)
```

    Success! All workers and firms are matched

```
solver.solution
```

      $\mu(x)$
      $\theta(x)$
      $w(x)$
      $\pi(x)$

      x

      1.000000
       1.000000
       3.000178
       0.061186
       2.914672

      1.010101
       1.003401
       2.939583
       0.062447
       2.939121

      1.020202
       1.006872
       2.880465
       0.063728
       2.963565

      1.030303
       1.010415
       2.822776
       0.065031
       2.988006

      1.040404
       1.014030
       2.766469
       0.066354
       3.012442

      1.050505
       1.017718
       2.711500
       0.067700
       3.036875

      1.060606
       1.021481
       2.657826
       0.069067
       3.061303

      1.070707
       1.025319
       2.605406
       0.070456
       3.085728

      1.080808
       1.029235
       2.554202
       0.071869
       3.110149

      1.090909
       1.033229
       2.504174
       0.073305
       3.134567

      1.101010
       1.037303
       2.455287
       0.074764
       3.158981

      1.111111
       1.041457
       2.407506
       0.076248
       3.183391

      1.121212
       1.045694
       2.360799
       0.077757
       3.207798

      1.131313
       1.050015
       2.315132
       0.079290
       3.232202

      1.141414
       1.054421
       2.270475
       0.080850
       3.256603

      1.151515
       1.058913
       2.226799
       0.082436
       3.281000

      1.161616
       1.063494
       2.184074
       0.084048
       3.305395

      1.171717
       1.068163
       2.142274
       0.085688
       3.329786

      1.181818
       1.072924
       2.101372
       0.087356
       3.354175

      1.191919
       1.077778
       2.061342
       0.089052
       3.378560

      1.202020
       1.082725
       2.022161
       0.090778
       3.402943

      1.212121
       1.087769
       1.983804
       0.092533
       3.427323

      1.222222
       1.092909
       1.946248
       0.094319
       3.451700

      1.232323
       1.098149
       1.909473
       0.096135
       3.476075

      1.242424
       1.103490
       1.873456
       0.097983
       3.500447

      1.252525
       1.108933
       1.838178
       0.099864
       3.524817

      1.262626
       1.114481
       1.803619
       0.101777
       3.549184

      1.272727
       1.120134
       1.769759
       0.103725
       3.573549

      1.282828
       1.125896
       1.736581
       0.105706
       3.597911

      1.292929
       1.131768
       1.704067
       0.107723
       3.622271

      ...
      ...
      ...
      ...
      ...

      1.707071
       1.498031
       0.786404
       0.233426
       4.619494

      1.717172
       1.511000
       0.771421
       0.237960
       4.643787

      1.727273
       1.524221
       0.756697
       0.242591
       4.668080

      1.737374
       1.537700
       0.742227
       0.247320
       4.692371

      1.747475
       1.551441
       0.728006
       0.252151
       4.716661

      1.757576
       1.565451
       0.714030
       0.257087
       4.740950

      1.767677
       1.579736
       0.700294
       0.262129
       4.765239

      1.777778
       1.594301
       0.686795
       0.267281
       4.789526

      1.787879
       1.609153
       0.673528
       0.272546
       4.813812

      1.797980
       1.624298
       0.660489
       0.277926
       4.838098

      1.808081
       1.639742
       0.647675
       0.283425
       4.862382

      1.818182
       1.655492
       0.635080
       0.289046
       4.886666

      1.828283
       1.671554
       0.622702
       0.294792
       4.910948

      1.838384
       1.687937
       0.610537
       0.300666
       4.935230

      1.848485
       1.704646
       0.598581
       0.306671
       4.959511

      1.858586
       1.721689
       0.586830
       0.312812
       4.983790

      1.868687
       1.739074
       0.575281
       0.319092
       5.008069

      1.878788
       1.756808
       0.563932
       0.325514
       5.032348

      1.888889
       1.774900
       0.552777
       0.332082
       5.056625

      1.898990
       1.793357
       0.541815
       0.338801
       5.080901

      1.909091
       1.812189
       0.531042
       0.345674
       5.105177

      1.919192
       1.831403
       0.520454
       0.352706
       5.129452

      1.929293
       1.851008
       0.510050
       0.359901
       5.153726

      1.939394
       1.871014
       0.499825
       0.367264
       5.177999

      1.949495
       1.891429
       0.489777
       0.374798
       5.202272

      1.959596
       1.912264
       0.479904
       0.382509
       5.226544

      1.969697
       1.933529
       0.470201
       0.390402
       5.250815

      1.979798
       1.955232
       0.460667
       0.398482
       5.275085

      1.989899
       1.977386
       0.451300
       0.406753
       5.299355

      2.000000
       2.000000
       0.442095
       0.415222
       5.323623

100 rows × 4 columns

```
sol1 = solver.solution
```

```
sol1.plot()
plt.show()
```

```
xs = []
thetas = []
ys = []
for i in range(100):
    xs.append(sol1.index[i])
    ys.append(sol1.iloc[i,0])
    thetas.append(sol1.iloc[i,1])
eval_k = sym.lambdify((x,y,l,r,A,kapa,eta,R), k_star)
ks = []
for a in range(len(xs)):
    ks.append(eval_k(xs[a],ys[a],thetas[a],1,1,1,0.89,0.13099))
```

```
plt.plot(xs, ks, label="Capital")
plt.legend(loc=2)
plt.xlim(1,2)
plt.title("Rich country")
plt.show()
```

```
solver.residuals.plot(logy=True, alpha=0.5)
plt.show()
```

```
# Not-so-Developped country

F_params2 = {'R':0.83684, 'rho':0.25, 'gamma':0.5, 'eta':0.89, 'L':5.46, 'A': 0.5105, 'kapa':1.0}

model2 = models.Model('positive',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params2)

solver = shooting.ShootingSolver(model=model2)
```

```
solver.solve(1.0, tol=1e-6, number_knots=100, atol=1e-15, rtol=1e-12)
```

    Success! All workers and firms are matched

```
solver.solution
```

      $\mu(x)$
      $\theta(x)$
      $w(x)$
      $\pi(x)$

      x

      1.000000
       1.000000
       2.517564
       0.012033
       0.174572

      1.010101
       1.004046
       2.475778
       0.012236
       0.175642

      1.020202
       1.008160
       2.434783
       0.012442
       0.176712

      1.030303
       1.012343
       2.394558
       0.012651
       0.177781

      1.040404
       1.016597
       2.355084
       0.012863
       0.178848

      1.050505
       1.020922
       2.316341
       0.013078
       0.179915

      1.060606
       1.025319
       2.278312
       0.013297
       0.180980

      1.070707
       1.029789
       2.240978
       0.013518
       0.182045

      1.080808
       1.034334
       2.204322
       0.013743
       0.183109

      1.090909
       1.038954
       2.168328
       0.013971
       0.184171

      1.101010
       1.043651
       2.132980
       0.014203
       0.185233

      1.111111
       1.048426
       2.098261
       0.014438
       0.186294

      1.121212
       1.053280
       2.064158
       0.014676
       0.187354

      1.131313
       1.058213
       2.030655
       0.014918
       0.188413

      1.141414
       1.063228
       1.997739
       0.015164
       0.189472

      1.151515
       1.068326
       1.965396
       0.015414
       0.190529

      1.161616
       1.073508
       1.933613
       0.015667
       0.191586

      1.171717
       1.078774
       1.902376
       0.015924
       0.192642

      1.181818
       1.084128
       1.871675
       0.016186
       0.193697

      1.191919
       1.089568
       1.841497
       0.016451
       0.194751

      1.202020
       1.095098
       1.811829
       0.016720
       0.195805

      1.212121
       1.100719
       1.782662
       0.016994
       0.196858

      1.222222
       1.106431
       1.753984
       0.017272
       0.197910

      1.232323
       1.112237
       1.725785
       0.017554
       0.198962

      1.242424
       1.118138
       1.698054
       0.017840
       0.200012

      1.252525
       1.124135
       1.670781
       0.018132
       0.201062

      1.262626
       1.130230
       1.643957
       0.018428
       0.202112

      1.272727
       1.136424
       1.617573
       0.018728
       0.203160

      1.282828
       1.142719
       1.591619
       0.019034
       0.204208

      1.292929
       1.149117
       1.566086
       0.019344
       0.205256

      ...
      ...
      ...
      ...
      ...

      1.707071
       1.525247
       0.799952
       0.037870
       0.247748

      1.717172
       1.537981
       0.786590
       0.038513
       0.248775

      1.727273
       1.550931
       0.773426
       0.039169
       0.249803

      1.737374
       1.564102
       0.760457
       0.039837
       0.250829

      1.747475
       1.577498
       0.747680
       0.040517
       0.251856

      1.757576
       1.591123
       0.735093
       0.041211
       0.252882

      1.767677
       1.604982
       0.722693
       0.041918
       0.253908

      1.777778
       1.619078
       0.710476
       0.042639
       0.254933

      1.787879
       1.633418
       0.698441
       0.043374
       0.255959

      1.797980
       1.648005
       0.686584
       0.044123
       0.256984

      1.808081
       1.662843
       0.674904
       0.044887
       0.258008

      1.818182
       1.677939
       0.663397
       0.045665
       0.259032

      1.828283
       1.693297
       0.652061
       0.046459
       0.260056

      1.838384
       1.708923
       0.640893
       0.047269
       0.261080

      1.848485
       1.724821
       0.629892
       0.048094
       0.262103

      1.858586
       1.740997
       0.619054
       0.048936
       0.263126

      1.868687
       1.757456
       0.608378
       0.049795
       0.264149

      1.878788
       1.774205
       0.597862
       0.050671
       0.265172

      1.888889
       1.791249
       0.587502
       0.051564
       0.266194

      1.898990
       1.808593
       0.577297
       0.052476
       0.267216

      1.909091
       1.826245
       0.567245
       0.053406
       0.268238

      1.919192
       1.844210
       0.557344
       0.054354
       0.269259

      1.929293
       1.862494
       0.547590
       0.055323
       0.270280

      1.939394
       1.881105
       0.537984
       0.056310
       0.271301

      1.949495
       1.900048
       0.528522
       0.057319
       0.272322

      1.959596
       1.919331
       0.519202
       0.058347
       0.273342

      1.969697
       1.938960
       0.510023
       0.059398
       0.274362

      1.979798
       1.958943
       0.500982
       0.060469
       0.275382

      1.989899
       1.979287
       0.492079
       0.061564
       0.276402

      2.000000
       2.000000
       0.483310
       0.062680
       0.277421

100 rows × 4 columns

```
solver.residuals.plot(logy=True, alpha=0.5)
plt.show()
```

```
sol2 = solver.solution
```

```
sol2.plot()
plt.show()
```

```
xs2 = []
thetas2 = []
ys2 = []
for i in range(100):
    xs2.append(sol2.index[i])
    ys2.append(sol2.iloc[i,0])
    thetas2.append(sol2.iloc[i,1])
```

```
ks2 = []
for a in range(len(xs)):
    ks2.append(eval_k(xs2[a],ys2[a],thetas2[a],1,0.5105,1,0.89,0.83684))
```

```
plt.plot(xs2, ks2, label="Capital")
plt.legend(loc=1)
plt.xlim(1,2)
plt.title("Poor Country")
plt.show()
```

```
#plt.plot(xs, ks, label="Capital - Rich")
sol1.plot()
sol2.plot()
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.xlim(1,2)
plt.show()
```

```
fig = plt.figure()
ax1 = fig.add_subplot(111)
ax1.plot(xs, ys, label="Land quality- Rich")
ax1.plot(xs, thetas, label="Labour- Rich")
ax1.plot(xs2, ks2, label="Capital - Poor")
ax1.plot(xs2, ys2, label="Land quality- Poor")
ax1.plot(xs2, thetas2, label="Labour- Poor")
ax1.plot(xs, ks, label="Capital - Rich")

ax1.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

plt.xlim(1,2)
plt.title("Both Countries")
plt.show()
```

```
fig = plt.figure()
ax1 = fig.add_subplot(111)
plt.plot(xs, ks, label="Rich")

ax2 = ax1.twinx()
plt.plot(xs2, ks2, label="Poor", c='g')
ax2.set_ylabel('Poor')

ax1.legend(bbox_to_anchor=(1.1, 1), loc=2, borderaxespad=0.)
ax2.legend(bbox_to_anchor=(1.065, 0.95), loc=2, borderaxespad=1)
plt.xlim(1,2)
plt.title("Capital")
plt.show()
```

```

```