Name: Minh T. Nguyen

Future Scientist Program - ACC - DoWhy Package

Estimating the Effect of a Member Rewards Program

### Credits:

1. This Algorithms: https://github.com/microsoft/dowhy/blob/master/docs/source/example_notebooks/dowhy_example_effect_of_memberrewards_program.ipynb

2. CI tutorial: https://github.com/amit-sharma/causal-inference-tutorial/tree/master/python

3. Other Application: https://microsoft.github.io/dowhy/example_notebooks/tutorial-causalinference-machinelearning-using-dowhy-econml.html#Why-causal-inference?

4. Package: https://microsoft.github.io/dowhy/

5. Paper: https://paperswithcode.com/paper/dowhy-an-end-to-end-library-for-causal

###1. Abstract:
*The DoWhy package can be used to estimate the effect of a subscription or  rewards program for customers

*Suppose that Target has a membershp rewards program where customers receive additional benefits if they sign up. We want to know if the program is effective.

###-> What is the impact of offering the membership rewards program on the total sales?
###-> If the current members had not signed up for the program, how much less would they have spent on the websites?

*This is the problem related to Average Treatment Effect on the Treated (ATT)

###2. Formulating the Causal Model

The rewarding program of Target was introduced in Jan 2019. The features are data of all monthly transactions of every user and on the time of singup for those who chose to signup for the rewards program.

```python
# Creating some simulated data for the example
import pandas as pd
import numpy as np

num_users = 10000
num_months = 12

# Create tuples of users and signup months
signup_months = np.random.choice(np.arange(1, num_months), num_users) * np.random.randint(0,2,size=num_users)

# Generating table of dataset
df = pd.DataFrame({
    'user_id': np.repeat(np.arange(num_users), num_months),
    'signup_month': np.repeat(signup_months, num_months), # signup months == 0 means that the customer did not sign up
    'month': np.tile(np.arange(1, num_months+1), num_users), # months are from 1 to 12
    'spend': np.random.poisson(500, num_users*num_months) #np.random.beta(a=2, b=5, size=num_users * num_months)*1000 # centered at 500
    })

# Assigning a treatment value based on the signup month. If not signup then the treatment is False, else True
df["treatment"] = (1-(df["signup_month"]==0)).astype(bool)

# Simulating effect of month (monotonically increasing - customers buy the most in December)
df["spend"] = df["spend"] - df["month"]*10

# The Treatment Effect (simulating a simple treatment effect of 100)
after_signup = (df["signup_month"] < df["month"]) & (df["signup_month"] !=0)
df.loc[after_signup,"spend"] = df[after_signup]["spend"] + 100

df
```

      user_id
      signup_month
      month
      spend
      treatment

      0
      0
      10
      1
      483
      True

      1
      0
      10
      2
      512
      True

      2
      0
      10
      3
      464
      True

      3
      0
      10
      4
      466
      True

      4
      0
      10
      5
      456
      True

      ...
      ...
      ...
      ...
      ...
      ...

      119995
      9999
      5
      8
      496
      True

      119996
      9999
      5
      9
      516
      True

      119997
      9999
      5
      10
      511
      True

      119998
      9999
      5
      11
      480
      True

      119999
      9999
      5
      12
      496
      True

120000 rows × 5 columns

###3. The Consideration on Time

*Time plays a crucial role in modeling this problem. Why? Rewards signup can affect the future transactions, but not those that happened before it. In fact, the transaction prior to the rewards signup can be assumed to CAUSE the rewards signup decision. Thus, we can split up the variables for each user in term of

(1) Activity prior to the treatment (causes the treatment) - the signup program

(2) Activity after the treatment (is the outcome of applying treatmennt) - the signup program

HERE, many important variables that affect signup and total spend are missing (e.g., the type of products bought, length of a user's account, geography, etc.). So we'll need a node denoting Unobserved Confounders.

```python
!pip install dowhy
```

    Requirement already satisfied: dowhy in /usr/local/lib/python3.7/dist-packages (0.6)
    Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from dowhy) (0.10.2)
    Requirement already satisfied: pydot>=1.4 in /usr/local/lib/python3.7/dist-packages (from dowhy) (1.4.2)
    Requirement already satisfied: pandas>=0.24 in /usr/local/lib/python3.7/dist-packages (from dowhy) (1.1.5)
    Requirement already satisfied: sympy>=1.4 in /usr/local/lib/python3.7/dist-packages (from dowhy) (1.7.1)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from dowhy) (0.22.2.post1)
    Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.7/dist-packages (from dowhy) (1.19.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from dowhy) (1.4.1)
    Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from dowhy) (2.5)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels->dowhy) (0.5.1)
    Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/lib/python3.7/dist-packages (from pydot>=1.4->dowhy) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->dowhy) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24->dowhy) (2018.9)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.4->dowhy) (1.2.1)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->dowhy) (1.0.1)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->dowhy) (4.4.2)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from patsy>=0.4.0->statsmodels->dowhy) (1.15.0)

```python
import os, sys
sys.path.append(os.path.abspath("../../../"))
import dowhy

# Setting the signup month (for ease of analysis)
i = 6
```

```python
causal_graph = """digraph {
treatment[label="Program Signup in month i"];
pre_spends;
post_spends;
Z->treatment;
U[label="Unobserved Confounders"];
pre_spends -> treatment;
treatment->post_spends;
signup_month->post_spends; signup_month->pre_spends;
signup_month->treatment;
U->treatment; U->pre_spends; U->post_spends;
}"""

# Post-process the data based on the graph and the month of the treatment (sign up program)
df_i_signupmonth = df[df.signup_month.isin([0,i])].groupby(["user_id", "signup_month", "treatment"]).apply(
    lambda x: pd.Series({'pre_spends': np.sum(np.where(x.month < i, x.spend,0))/np.sum(np.where(x.month<i, 1,0)),
                        'post_spends': np.sum(np.where(x.month > i, x.spend,0))/np.sum(np.where(x.month>i, 1,0)) })
    ).reset_index()

print(df_i_signupmonth) # Reprint the updated table

model = dowhy.CausalModel(data=df_i_signupmonth,
                     graph=causal_graph.replace("\n", " "),
                     treatment="treatment",
                     outcome="post_spends")

# model.view_model()
from IPython.display import Image, display
display(Image(filename="causal_model.png"))
```

###4. Identifying the Causal Effect

For simplicy, says we assume that unobserved confounding does not play a big part

```python
identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)

print(identified_estimand)
```

    Estimand type: nonparametric-ate

    ### Estimand : 1
    Estimand name: backdoor
    Estimand expression:
         d
    ────────────(Expectation(post_spends|signup_month,pre_spends))
    d[treatment]
    Estimand assumption 1, Unconfoundedness: If U→{treatment} and U→post_spends then P(post_spends|treatment,signup_month,pre_spends,U) = P(post_spends|treatment,signup_month,pre_spends)

    ### Estimand : 2
    Estimand name: iv
    Estimand expression:
    Expectation(Derivative(post_spends, [Z])*Derivative([treatment], [Z])**(-1))
    Estimand assumption 1, As-if-random: If U→→post_spends then ¬(U →→{Z})
    Estimand assumption 2, Exclusion: If we remove {Z}→{treatment}, then ¬({Z}→post_spends)

    ### Estimand : 3
    Estimand name: frontdoor
    No such variable found!

Based on the graph, DoWhy determines that the signup month and amount spent in the pre-treatment months (signup_months, pre_spend) need to be conditioned on

### 5. Estimating the effect

We now estimate the effect based on the backdoor estimate, setting the targe units to "att"

```python
estimate = model.estimate_effect(identified_estimand,
                                 method_name="backdoor1.propensity_score_matching",
                                target_units="att")
print(estimate)
```

    /usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

    *** Causal Estimate ***

    ## Identified estimand
    Estimand type: nonparametric-ate

    ## Realized estimand
    b: post_spends~treatment+signup_month+pre_spends
    Target units: att

    ## Estimate
    Mean value: 103.72095671981776

###What does these feedback means?

*The analysis tell us the Average Treatmetn Effect on the Treated (ATT). That is, the average effect on the total spend for the customers that signed up for the Rewarding Program in month i=6 (compared to the case where they had not signed up).

*Note: We can similarly calculalte the effect for customers who signed up in any other month by changing the value i = 6

*Note that the estimation suffers from left and right-censoring

1. Left-censoring: If the customer signs up in the first month, we do not have enough transaction history to match them to similar customers who did not sign up (and thus apply the backdoor identified estimation)

2. Right-censoring: if the customer signs up in the lastmonth, we do not have enought future (post-treatment) transactions to estimate the outcocme after sing up.

*Thus, even if the effect of signup was the same across all months, the estimated effects may be different by month of signup, due to lack of data (and thus high variance in estimated pre-treatment or post-treatment transactions activity).

### 6. Refuting the estimate:

*We refute the estimate using the placebo treatment refuter. This refuter substitutes the treatment by an independent random variable and checks whether our estimate now goes to zero

```python
refutation = model.refute_estimate(identified_estimand, estimate, method_name="placebo_treatment_refuter",
                     placebo_type="permute", num_simulations=2)
print(refutation)
```

    /usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)
    /usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

    Refute: Use a Placebo Treatment
    Estimated effect:103.72095671981776
    New effect:0.6421791951404714
    p value:0.11916793187897057

```python

```


# POS part-of speech 词性标注
## 任务描述
给定一个字符序列，标注出每个字符的词性
$$S = w_1,w_2.w_3,w_4,w_5$$
$$Z=z_1,z_2,z_3,z_4,z_5$$
## 解决方法
现根据标注好的词性表进行训练，之后根据输入的字符序列，标注出各个字符对应的词性
$$S'=w_1',w_2',w_3',w_4',w_5'$$
$$Z'=?$$
## 理论基础
**Noisy Channel Model**
$$\begin{align}P(Z|S) &\propto P(S|Z)P(Z)\\
&=P(w_1w_2w_3...w_n|z_1z_2z_3...z_n)P(z_1z_2z_3...z_n)\\
&=\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)P(z_2|z_1)...P(z_n|z_{n-1})
\end{align}$$
其中$\prod_{i=1}^nP(w_i|z_i)$被称为是条件概率，$P(z_1)P(z_2|z_1)...P(z_n|z_{n-1}$是语言模型。
$$\begin{align}
\hat Z &= argmaxP(Z|S)\\
&=argmax\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)\prod_{j=2}^nP(z_j|z_{j-1})\\
&\propto log(\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)\prod_{j=2}^nP(z_j|z_{j-1}))\\
&=argmax\sum_{i=1}^nlog(w_i|z_i)+logP(z_1)+\sum_{j=2}^nlogP(z_j|z_{j-1})
\end{align}$$
为了求出$\hat Z$则需要计算上述公式，我们需要计算$\sum_{i=1}^nlog(w_i|z_i)$，$logP(z_1)$，$\sum_{j=2}^nlogP(z_j|z_{j-1})$，分别记为$A, \pi, B$

```python
tag2id, id2tag = {}, {}# 词性字典，词性转id，id转词性
word2id, id2word = {}, {}# 单词id字典，id单词字典
for line in open('traindata.txt','r',encoding='utf-8'):# 读取文件，生成相关字典
    items = line.split('/')
    word, tag = items[0],items[1].strip()

    if word not in word2id:
        word2id[word] = len(word2id)
        id2word[len(id2word)] = word
    if tag not in tag2id:
        tag2id[tag] = len(tag2id)
        id2tag[len(id2tag)] = tag
M = len(word2id) # M:词典的大小，# of words in dictionary
N = len(tag2id) # N：词性的种类个数，# of tags in tag dictionary
```

```python
print(M,N)
# print(word2id, id2word)
# print(tag2id, id2tag)
```

    18978 54

构建$A, B, \pi$

```python
import numpy as np
pi = np.zeros(N) # 初始化pi向量，表示出现在第一个位置的各个词性的概率
A = np.zeros((N,M)) # A[i][j]表示在第i个词性下单词是第j个的概率 ，发射矩阵
B = np.zeros((N,N)) # B[i][j]表示从第j个词性转移到第i个词性的概率，转移矩阵
```

```python
# 根据训练数据，统计相关的数据，然后标准化，得到相应的A, B, pi
prev_tag = ""
for line in open("traindata.txt","r",encoding="utf-8"):
    items = line.split("/")
    wordID, tagID = word2id[items[0]],tag2id[items[1].strip()]
    if prev_tag == "":
        pi[tagID] += 1
        A[tagID][wordID] += 1
        prev_tag = tagID
    else:
        A[tagID][wordID] += 1
        B[tag2id[prev_tag]][tagID] += 1
    if items[0] == ".":
        prev_tag = ""
    else:
        prev_tag = items[1].strip()
# normalization
pi = pi/sum(pi)
for i in range(N):
    A[i] /= sum(A[i])
    B[i] /= sum(B[i])
```

```python
print(A[0])
print(pi)
```

    [5.67741935e-04 0.00000000e+00 0.00000000e+00 ... 5.16129032e-05
     0.00000000e+00 0.00000000e+00]
    [1.81425219e-01 0.00000000e+00 1.00037051e-02 3.33456836e-03
     3.95208102e-03 3.68037545e-02 1.11646289e-01 3.66802519e-02
     6.17512659e-04 3.81622823e-02 8.76867976e-03 5.18710634e-02
     6.02692355e-02 2.47005064e-04 2.17240953e-01 0.00000000e+00
     1.48203038e-03 6.05162406e-03 8.64517723e-04 2.47005064e-04
     0.00000000e+00 4.73014697e-02 0.00000000e+00 7.16314684e-03
     1.72903545e-03 2.09954304e-03 7.53365444e-02 6.36038039e-02
     2.59355317e-03 1.85253798e-03 5.92812153e-03 1.97604051e-03
     2.84055823e-03 0.00000000e+00 0.00000000e+00 2.71705570e-03
     5.92812153e-03 5.92812153e-03 9.88020254e-04 3.70507595e-04
     1.23502532e-04 0.00000000e+00 0.00000000e+00 1.85253798e-03
     0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
     0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
     0.00000000e+00 0.00000000e+00]

```python
def log(v):# 计算log函数值
    if v == 0:
        return np.log(v+0.0001)
    else:
        return np.log(v)
```

使用viterbi算法计算

```python
def viterbi(x, pi, A, B):
    x = [word2id[word] for word in x.split(" ")]#对输入的句子进行切分
    T = len(x)

    dp = np.zeros((T,N))
    for j in range(N):
        dp[0][j] = log(pi[j])+log(A[j][x[0]])# 矩阵第一列初始化
    ptr = np.array([[0 for x in range(N)] for y in range(T)])
    for i in range(1, T): # 每个单词
        for j in range(N): # 每个词性
            dp[i][j] = -99999
            for k in range(N): # 从每个k到达j
                score = dp[i-1][k] + log(B[k][j]) + log(A[j][x[i]])
                if score > dp[i][j]:
                    dp[i][j] = score
                    ptr[i][j] = k
    # decoding:把做好的路径打印出来
    best_seq = [0]*T
    # step1: 找出对应于最后一个单词的词性
    best_seq[T-1] = np.argmax(dp[T-1])
    for i in range(T-2, -1, -1):
        best_seq[i] = ptr[i+1][best_seq[i+1]]
    # 将词性序列打印出来
    for i in range(len(best_seq)):
        print(id2tag[best_seq[i]])
```

```python
x = "Social security number , passport number and details about the services provided for the payment"
viterbi(x,pi,A,B)
```

    DT
    NN
    NN
    ,
    DT
    NN
    CC
    NNS
    IN
    DT
    NNS
    VBN
    IN
    DT
    NN

```python

```


```python
import numpy as np
import scipy as sp
import scipy.stats
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
```

Table of Contents

1   Bayesian Theory

Bayesian is not the opposite of frequentist, but it holds a constrasting view comparing with frequentists.

In the first chapter, we review Bayes' Theorem and give an overview of application in statistics and econometrics.

If all you want know is what Bayesian methods are, this is the only chapter you need to read.

#  Bayesian Theory

The bayesian theory is built upon the most famous **Bayes' Theorem**, which is derived from condition probability and joint probability.

The **law of joint probability** says that
$$
P(x, y)=P(x \mid y) P(y)
$$
or
$$
P(x, y)=P(y \mid x) P(x)
$$
Combine them, we will obtain the formula of Bayes' Theorem.
$$
P(x \mid y)=\frac{P(y \mid x) P(x)}{P(y)}
$$

To make it meaningful in the context of statistics/econometrics, replace the symbols will provide a refreshing view
$$
P(\theta \mid Y)=\frac{P(Y \mid \theta) P(\theta)}{P(Y)}
$$
where $Y$ is the data, $\theta$ is the vector of parameters you would like to make inferrence. The formula summarizes what we know about $\theta$ about having seen the sample of observations on $Y$.

In Bayesian terms
\begin{align}
\text{prior:}&\ P(\theta)\\
\text{likelihood:}&\ P(Y|\theta)\\
\text{marginal likelihood:}&\ P(Y)\\
\text{posterior:}&\ P(\theta|Y)\\
\end{align}

To update prior beliefs into posterior beliefs conditional on observaed data, the process is called **Bayesian Inference**.

More often, we ignore $P(Y)$ due to its irrelevance to $\theta$, the formula simplies into
$$
P(\theta \mid Y) \propto P(Y \mid \theta) P(\theta)
$$

#  Model Comparison

If you have multiple models in mind, Bayesian framework can provide model comparison too. Suppose we have $m$ different models, $M_i$ for $i=1, ..., m$, the posterior for the parameters claculated using $M_i$ is
$$
P\left(\theta^{i} \mid Y, M_{i}\right)=\frac{P\left(Y \mid \theta^{i}, M_{i}\right) P\left(\theta^{i} \mid M_{i}\right)}{P\left(Y \mid M_{i}\right)}
$$

We use $i$ to signal that for each model has its own posterior, likelihood and prior.

However, to compare models, we need to use evaluate the **posterior model probability**
$$
\operatorname{P}\left(M_{i} \mid Y\right) \propto P\left(Y \mid M_{i}\right) \operatorname{P}\left(M_{i}\right),
$$

If we have two models, we can construct the **posterior odds ratio** as
$$
\frac{P\left(M_{1} \mid Y\right)}{P\left(M_{2} \mid Y\right)}=\frac{P\left(Y \mid M_{1}\right)}{P\left(Y \mid M_{2}\right)} \frac{P\left(M_{1}\right)}{P\left(M_{2}\right)}
$$
Specifically, this part is called **Bayes factor**
$$
BF_{12}=\frac{P\left(Y \mid M_{1}\right)}{P\left(Y \mid M_{2}\right)}
$$
which is a the ratio of _marginal likelihoods_ for model $1$ and model $2$. Furthermore, marginal likelihood can be decomposed into
$$
P\left(Y \mid M_{i}\right)=\int_{\theta_{i}} P\left(Y \mid \theta_{i}, M_{i}\right) P\left(\theta_{i} \mid M_{i}\right) d \theta_{i}
$$

#  Bernoulli Example

An example will help clarify how we use the Bayes' Theorem for inference and prediction.

The Bernoulli pmf is commonly written as
$$
f(k; p) = p^{k}(1-p)^{1-k}\ \text{ for }\ \ k\in \{0, 1\}
$$
Watch out the notation that $k$ equals either $0$ or $1$, not $k \in (0, 1)$.

The meaning of $0$ and $1$ usually represents "happened" and "not happened". Not suppose we have collects a sample of $K= (k_1, k_2, k_3,...,k_T)$.

The joint Bernoulli pmf is
$$
f(K|p)=p^{k_1}(1-p)^{1-k_1}p^{k_2}(1-p)^{1-k_2}p^{k_3}(1-p)^{1-k_3}...p^{k_T}(1-p)^{1-k_T}=p^{\sum_{i=0}^T k_i}(1-p)^{T-\sum_{i=0}^T k_i}
$$

which is the **likelihood function** given all possible $p$.

Naturally, we need prior too.

The onf of most frequently used prior in Bayesian inference is **Beta distribution**, because it models the probability of a probability.

For instance, you can ask a question, what's the probability that a random person has diabetes? You could give an educated guess around $30\%$, then you go to collect data on the street that asking random person provide a blood sample test.

Recall that we have mentioned pdf of Beta distribution
\begin{equation}
f(p, \alpha, \beta)=\frac{\Gamma(\alpha+b) p^{\alpha-1}(1-p)^{\beta-1}}{\Gamma(\alpha) \Gamma(\beta)}
\end{equation}
However in Bayesian inference, we usually combine those gamma functions into a beta function term.
\begin{equation}
f(p, \alpha, \beta)=\frac{1}{B(\alpha, \beta)} p^{\alpha-1}(1-p)^{\beta-1}
\end{equation}
where subscript $0$ represent **prior**.

Posterior is straightforward by evoking Bayes' Theorem.
$$
f(p|K) \propto \left(p^{\sum_{i=0}^T k_i}(1-p)^{T-\sum_{i=0}^T k_i}\right)\frac{1}{B(\alpha_0, \beta_0)} p^{\alpha_0-1}(1-p)^{\beta_0-1}
$$
Since we have already using $\propto$, we just need to keep the kernal of beta distribution, which simplies to
$$
f(p|K) \propto \left(p^{\sum_{i=0}^T k_i}(1-p)^{T-\sum_{i=0}^T k_i}\right) p^{\alpha_0-1}(1-p)^{\beta_0-1}
$$
Collect terms we have the kernal of posterior distribution, which is also a beta distribution.
$$
f(p|K) \propto p^{\sum_{i=0}^T k_i+\alpha_0-1}(1-p)^{T-\sum_{i=0}^T k_i+\beta_0-1}
$$

We will see more detailed examples in next chapter, we just derived posterior analytically by using **conjugate prior**. It makes posterior takes the same function form as prior, but with updated parameters. In this case the posterior parameters are
$$
\alpha=\sum_{i=0}^T k_i+\alpha_0\\
\beta=T-\sum_{i=0}^T k_i+\beta_0
$$

Let's give some meaning to the example.

The beta distribution here models our prior belief, let's say the parameters are set as
$$
\alpha_0 = 4\\
\beta_0 = 7
$$
The sample is
$$
K= (1, 0, 0, 0, 1, 0, 1, 1, 0, 0)
$$
which indicates $T=10$. Therefore the posterior parameters are
$$
\alpha = 4+4=8\\
\beta=10-4+7=13
$$
Plot prior and posterior together.

```python
x_beta = np.linspace(0, 1, 200)
y_beta_prior = sp.stats.beta(a=4, b=7).pdf(x_beta)
y_beta_posterior = sp.stats.beta(a=8, b=13).pdf(x_beta)

fig, ax = plt.subplots(figsize=(6, 6))
ax.plot(x_beta, y_beta_prior, label='prior')
ax.plot(x_beta, y_beta_posterior, label='posterior')
ax.set_title('Beta-Bernoulli Conjugate')
ax.legend()
plt.show()
```

##  Reporting Results

We can provide numerical features of posterior distribution such as mean and standard deviation as **point estimates**. Because the posterior are not simulated, we have formula for mean and variance which can be found from any textbook or Wikipedia.
$$
\mathrm{E}[X]=\frac{\alpha}{\alpha+\beta}\\
\operatorname{var}[X]=\frac{\alpha \beta}{(\alpha+\beta)^{2}(\alpha+\beta+1)}
$$

```python
a, b = 8, 13
mean_post = a/(a+b)
std_post =  (a*b)/((a+b)**2*(a+b+1))
print('mean of posterior: {}'.format(mean_post))
print('std of posterior: {}'.format(std_post))
```

    mean of posterior: 0.38095238095238093
    std of posterior: 0.010719439290867862

Reporting **Bayesian Credible Interval** is also a common statistic that automatically provided in some statistic softwares. The Bayesian Credible Interval differs the Frequentist Confidence Interval mostly in a philosophical sense.

Frequentists assume the parameters are fixed but confidence intervals are random, so $95\%$ confidence interval means if the experiement repeats $100$ times, there are round $95$ times the confidence interval will include the true parameter.

However Bayesian holds a contrasting view that parameters are random, but the credible intervals are fixed. As an example, the shaded are is the $95\%$ credible interval. It indicates that there is $95\%$ chance the interval include the true parameter.

```python
lb = sp.stats.beta(a=8, b=13).ppf(.025)
ub = sp.stats.beta(a=8, b=13).ppf(.975)
x_ci = np.linspace(lb, ub, 100)
y_ci = sp.stats.beta(a=8, b=13).pdf(x_ci)

x_beta = np.linspace(0, 1, 100)
y_beta_posterior = sp.stats.beta(a=8, b=13).pdf(x_beta)

fig, ax = plt.subplots(figsize=(6, 6))
ax.plot(x_beta, y_beta_posterior)
ax.fill_between(x_ci, y_ci, color='tomato')

ax.set_title('Bayesian Credible Interval')
plt.show()
```

```python

```

```python

```

Chapter 1 - Overview of Bayesian Approach.ipynb


# Linear Regression (Botswana)

In this code, we use linear regression on the analytical forms of the Green Solow model to find its parameters. We show that the parameters found using linear regression are not consistent, and each analytical equation gives different/ conflicting parameters.

```julia
# #generating the environment, this has to be done the first time this code is run
# using Pkg
# Pkg.generate("regression_Y")

# Pkg.add("ModelingToolkit")
# Pkg.add("OrdinaryDiffEq")
# Pkg.add("LinearAlgebra")
# Pkg.add("Plots")
# Pkg.add("StatsModels")
# Pkg.add("Plots")
# Pkg.add("DataFrames")
# Pkg.add("CSV")
# Pkg.add("GLM")
# Pkg.add("Statistics")
# Pkg.add("Flux")
# Pkg.add("StatsBase")
```

```julia
println(@__DIR__)
using Pkg
# Pkg.status()
# Pkg.instantiate()
Pkg.activate(string(@__DIR__, "/regression_Y/"))
Pkg.status()
```

    /home/ziyu/solow/NODE/Neural Ordinary Differential Equations for the Regression and Prediction of Macroeconomics Data under the Green Solow Model/Model -1 Analytical Solution Regression

     Activating environment at `~/solow/NODE/Neural Ordinary Differential Equations for the Regression and Prediction of Macroeconomics Data under the Green Solow Model/Model -1 Analytical Solution Regression/regression_Y/Project.toml`

    Project regression_Y v0.1.0
    Status `~/solow/NODE/Neural Ordinary Differential Equations for the Regression and Prediction of Macroeconomics Data under the Green Solow Model/Model -1 Analytical Solution Regression/regression_Y/Project.toml`
      [336ed68f] CSV v0.8.4
      [a93c6f00] DataFrames v0.22.5
      [587475ba] Flux v0.11.1
      [38e38edf] GLM v1.4.0
      [961ee093] ModelingToolkit v5.10.1
      [1dea7af3] OrdinaryDiffEq v5.51.1
      [91a5bcdd] Plots v1.6.12
      [2913bbd2] StatsBase v0.33.3
      [3eaba693] StatsModels v0.6.21
      [37e2e46d] LinearAlgebra
      [10745b16] Statistics

```julia
using ModelingToolkit
using OrdinaryDiffEq
using LinearAlgebra
using Statistics
using Flux
using StatsBase
using DataFrames
# using Plots
```

# Importing data

```julia
using CSV
raw_data = CSV.read(string(@__DIR__,"/compileddf.csv"),normalizenames = true, DataFrame)
```

	Year	Albania_et	Albania_yt	Albania_Popt	Albania_nt	Albania_Ktotal	Albania_kt

	Int64	Float64?	Float64?	Float64	Float64?	Float64?	Float64?

50 rows × 834 columns (omitted printing of 827 columns)

1	1959	missing	missing	1.57133e6	missing	missing	missing

2	1960	1.25819	0.001451	1.62311e6	0.0329562	missing	missing

3	1961	1.37419	0.001463	1.67664e6	0.0329743	missing	missing

4	1962	1.43996	0.001511	1.72814e6	0.0307175	missing	missing

5	1963	1.18168	0.001563	1.77972e6	0.029846	missing	missing

6	1964	1.11174	0.001616	1.83228e6	0.0295379	missing	missing

7	1965	1.1661	0.001675	1.88365e6	0.028035	missing	missing

8	1966	1.33306	0.001738	1.93296e6	0.0261784	missing	missing

9	1967	1.36375	0.001804	1.98406e6	0.0264345	missing	missing

10	1968	1.51955	0.001869	2.0387e6	0.0275385	missing	missing

11	1969	1.55897	0.001932	2.10026e6	0.0301982	missing	missing

12	1970	1.75324	0.002004	2.15661e6	0.0268295	8341.5	0.00386787

13	1971	1.9895	0.002084	2.20872e6	0.0241615	9139.71	0.00413801

14	1972	2.51591	0.002165	2.26355e6	0.0248266	9979.96	0.00440898

15	1973	2.3039	0.002273	2.29569e6	0.0141976	10938.6	0.00476486

16	1974	1.84901	0.002282	2.34792e6	0.0227526	12095.4	0.00515151

17	1975	1.91063	0.002289	2.40111e6	0.0226515	13168.9	0.0054845

18	1976	2.01358	0.002299	2.45482e6	0.0223713	14067.9	0.0057307

19	1977	2.27588	0.002309	2.50905e6	0.0220888	15281.5	0.00609056

20	1978	2.53063	0.002319	2.56338e6	0.0216528	16651.3	0.00649584

21	1979	2.89821	0.002331	2.61768e6	0.0211842	18108.5	0.00691776

22	1980	1.93506	0.002347	2.67141e6	0.020527	19475.2	0.00729022

23	1981	2.69302	0.00236	2.72427e6	0.0197873	20820.0	0.00764241

24	1982	2.62486	0.002373	2.7801e6	0.0204917	22260.3	0.00800703

25	1983	2.68324	0.002387	2.83732e6	0.0205813	22660.8	0.00798669

26	1984	2.69429	0.0024	2.896e6	0.0206826	23848.4	0.00823495

27	1985	2.65802	0.002413	2.9567e6	0.0209596	24653.7	0.00833826

28	1986	2.66536	0.002428	3.01544e6	0.0198671	26068.0	0.00864484

29	1987	2.41406	0.002443	3.07532e6	0.0198588	27256.4	0.00886296

30	1988	2.3316	0.002459	3.13684e6	0.0200051	28346.3	0.00903656

⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

```julia
# for USA

USA_raw = raw_data[:, ["United_States_kt", "United_States_et", "United_States_yt", "United_States_nt"]]
delete!(USA_raw, 1:10)
USA_raw[!,:t] = 1:40

USA_y_reg = USA_raw[:,[1,3,2,5]]
DataFrames.rename!(USA_y_reg, [:k, :y, :e, :t])
USA_y_reg = USA_y_reg[2:end,:]

USA_y_reg[!, :yty0] = USA_y_reg[!, :y]/USA_y_reg[!, :y][1]
USA_y_reg[!, :lgyty0] = log.(USA_y_reg[!, :yty0])
USA_y_reg[!, :ktk0] = USA_y_reg[!, :k]/USA_y_reg[!, :k][1]
USA_y_reg[!, :lgktk0] = log.(USA_y_reg[!, :ktk0])
USA_y_reg
```

	k	y	e	t	yty0	lgyty0	ktk0	lgktk0

	Float64	Float64?	Float64?	Int64	Float64	Float64	Float64	Float64

39 rows × 8 columns

1	0.0632449	0.01503	21.1113	2	1.0	0.0	1.0	0.0

2	0.0651057	0.015304	20.9802	3	1.01823	0.018066	1.02942	0.0289973

3	0.067476	0.015944	21.7486	4	1.06081	0.0590344	1.0669	0.0647568

4	0.0706719	0.016689	22.5106	5	1.11038	0.104702	1.11743	0.111033

5	0.0743356	0.016491	21.5029	6	1.09721	0.0927666	1.17536	0.161575

6	0.0764952	0.016284	20.4022	7	1.08343	0.0801348	1.20951	0.190214

7	0.0777832	0.016975	21.1576	8	1.12941	0.121693	1.22987	0.20691

8	0.0809296	0.017567	21.5325	9	1.1688	0.155974	1.27962	0.246565

9	0.0851758	0.018373	21.973	10	1.22242	0.200834	1.34676	0.297702

10	0.0892787	0.018789	21.7804	11	1.2501	0.223223	1.41163	0.344749

11	0.0922673	0.018577	20.7865	12	1.23599	0.211876	1.45889	0.377674

12	0.0947202	0.018856	19.7668	13	1.25456	0.226783	1.49767	0.403913

13	0.0960069	0.018325	18.5905	14	1.21923	0.198218	1.51802	0.417405

14	0.0948384	0.01892	18.5715	15	1.25882	0.230171	1.49954	0.405159

15	0.0955838	0.020123	18.9768	16	1.33886	0.291815	1.51133	0.412988

16	0.0969398	0.020717	18.8823	17	1.37838	0.320906	1.53277	0.427075

17	0.100121	0.021236	18.7207	18	1.41291	0.34565	1.58307	0.459367

18	0.102487	0.021788	19.3503	19	1.44963	0.371311	1.62047	0.482718

19	0.104387	0.022499	20.0104	20	1.49694	0.403423	1.65053	0.501095

20	0.105472	0.023059	20.0758	21	1.5342	0.428008	1.66768	0.511432

21	0.105757	0.023201	19.3228	22	1.54365	0.434147	1.67218	0.514129

22	0.10488	0.022849	19.0562	23	1.52023	0.418859	1.65831	0.505801

23	0.104205	0.023298	19.1394	24	1.5501	0.438319	1.64764	0.499342

24	0.105723	0.023616	19.3471	25	1.57126	0.451876	1.67164	0.513808

25	0.107934	0.024279	19.3609	26	1.61537	0.479564	1.70661	0.534507

26	0.110946	0.024603	19.2765	27	1.63693	0.49282	1.75423	0.56203

27	0.112828	0.02523	19.496	28	1.67864	0.517986	1.78399	0.578852

28	0.115402	0.026052	19.6904	29	1.73333	0.550046	1.82468	0.601403

29	0.118977	0.026849	19.5792	30	1.78636	0.58018	1.8812	0.631912

30	0.123078	0.027735	19.7272	31	1.84531	0.612647	1.94605	0.665802

⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

## Parameters for the equation for y(t)

Start by regressing the equation for $y(t)$. To regress the following equation, the analytical solution of the Green Solow Model equation:

$\frac{y(t)}{y(0)} = \frac{k(t)^\alpha}{k(0)^\alpha} \times e^{g_B \times t} = \left(\frac{k(t)}{k(0)}\right)^\alpha \times e^{g_B \times t}$

Can take the log of both sides to get:

$\log\left(\frac{y(t)}{y(0)}\right) = \alpha \log\left(\frac{k(t)}{k(0)}\right) + g_B\times t$

```julia
using GLM

#regression of USA y data
ols_y_USA = lm(@formula(lgyty0 ~ lgktk0 + t), USA_y_reg)

```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,CholeskyPivoted{Float64,Array{Float64,2}}}},Array{Float64,2}}

    lgyty0 ~ 1 + lgktk0 + t

    Coefficients:
    ──────────────────────────────────────────────────────────────────────────────
                       Coef.  Std. Error      t  Pr(>|t|)   Lower 95%    Upper 95%
    ──────────────────────────────────────────────────────────────────────────────
    (Intercept)  -0.0245329   0.00779418  -3.15    0.0033  -0.0403402  -0.00872552
    lgktk0        0.00126064  0.0705531    0.02    0.9858  -0.141828    0.144349
    t             0.0197244   0.00156593  12.60    <1e-14   0.0165486   0.0229003
    ──────────────────────────────────────────────────────────────────────────────

Note that the value of $\alpha$ can be found through this regression, via the coefficient for $log\frac{k(t)}{k(0)}$

```julia
#returns the coefficients, which are the values of $\alpha$ and $g_B$

USA_alpha = coef(ols_y_USA)[2]
USA_gB = coef(ols_y_USA)[3]

println("alpha is ", USA_alpha)
println("gB is ", USA_gB)
```

    alpha is 0.0012606403056080499
    gB is 0.019724449042539376

## Parameters for the equation for e(t)

Regressing the equation for $e(t)$. To regress the following equation, the analytical solution of the Green Solow Model equation:

\begin{gather}
    \frac{e(t)}{e(0)}
    = \left( \frac{k(t)}{k(0)}\right)^\alpha \times e^{(g_B -g_A) \times t}
\end{gather}

Can take the log of both sides to get:

$\log\left(\frac{e(t)}{e(0)}\right) = \alpha \log\left(\frac{k(t)}{k(0)}\right) + (g_B-g_A)\times t$

```julia
USA_raw[!,:t] = 1:40
USA_e_reg = USA_raw[:,[1,3,2,5]]
DataFrames.rename!(USA_e_reg, [:k, :y, :e, :t])
USA_e_reg = USA_e_reg[2:end,:]

USA_e_reg[!, :ete0] = USA_e_reg[!, :e]/USA_y_reg[!, :e][1]
USA_e_reg[!, :lgete0] = log.(USA_e_reg[!, :ete0])
USA_e_reg[!, :ktk0] = USA_e_reg[!, :k]/USA_e_reg[!, :k][1]
USA_e_reg[!, :lgktk0] = log.(USA_e_reg[!, :ktk0])
# print(USA_y_reg)

ols_e_USA = lm(@formula(lgete0 ~ lgktk0 + t), USA_e_reg)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,CholeskyPivoted{Float64,Array{Float64,2}}}},Array{Float64,2}}

    lgete0 ~ 1 + lgktk0 + t

    Coefficients:
    ──────────────────────────────────────────────────────────────────────────────
                       Coef.  Std. Error      t  Pr(>|t|)    Lower 95%   Upper 95%
    ──────────────────────────────────────────────────────────────────────────────
    (Intercept)   0.00689264  0.0144505    0.48    0.6363  -0.0224143   0.0361996
    lgktk0       -0.136058    0.130806    -1.04    0.3052  -0.401345    0.129229
    t             5.92741e-5  0.00290324   0.02    0.9838  -0.00582877  0.00594732
    ──────────────────────────────────────────────────────────────────────────────

Upon observation, we can see that the values for $\alpha$ already differ among these two regessions.

```julia
# #calculating the variance and standard errors
# X = Array{Float64}(undef, 39, 3).+1
# Y = Array{Float64}(undef, 39, 1)
# Y[:,1] = USA_e_reg[!, :lgete0]
# X[:,2] = USA_e_reg[!, :lgktk0]
# X[:,3] = USA_e_reg[!, :t]
# beta = [0.00689264; -0.136058; 0.0000592741]
# MSE = norm(Y-X*beta, 2)^2
# variance = MSE.* inv(transpose(X)*X)
# standard_errors = Array{Float64}(undef, 1, 3)
# standard_errors = sqrt.([variance[1,1], variance[2,2], variance[3,3]]./(39-3))
```

## Parameters for the equation for k(t)

\begin{equation}
    k(t) = (\frac{\frac{(1-\alpha)\times s\times y(0)}{k(0)^\alpha} \times \frac{e^{(1-\alpha)\times (\delta+n+g_B)\times t + g_B \times t}}{(1-\alpha)\times (\delta+n+g_B) + g_B } +C}{e^{(1-\alpha)\times (\delta +n+g_B)\times t }})^\frac{1}{1-\alpha}
\end{equation}
Where:
\begin{equation}
    C = k(0)^{1-\alpha} - \frac{(1-\alpha)\times s\times y(0)}{{k(0)^\alpha}\times((1-\alpha)\times (\delta+n+g_B) + g_B )}
\end{equation}

We can simplify the equation to:
\begin{equation}
    k(t) ^ {(1-\alpha)}= \frac{(1-\alpha)\times s\times y(0)}{k(0)^\alpha \times ((1-\alpha)\times (\delta+n+g_B) + g_B )}  \times e^{g_B \times t} +C\times {e^{(\alpha-1)\times (\delta +n+g_B)\times t }}
\end{equation}

Which is equivalent to:
\begin{equation}
    k(t) ^ {(1-\alpha)}= \frac{(1-\alpha)\times s\times y(0)}{k(0)^\alpha \times ((1-\alpha)\times (\delta+n+g_B) + g_B )}  \times e^{g_B \times t} +(k(0)^{1-\alpha} - \frac{(1-\alpha)\times s\times y(0)}{{k(0)^\alpha}\times((1-\alpha)\times (\delta+n+g_B) + g_B )})\times {e^{(\alpha-1)\times (\delta +n+g_B)\times t }}
\end{equation}

Finding of coefficients based on:
https://math.stackexchange.com/questions/2249200/exponential-regression-with-two-terms-and-constraints
https://scikit-guess.readthedocs.io/en/latest/_downloads/4cd313a50f7e08ab81758ce0bd661bc3/Regressions-et-equations-integrales.pdf

```julia
function regression_system(dataframe)
regression_system_matrix = zeros(5,5)

regression_system_matrix[1,1] = sum(dataframe[:, :SSk].^2)
regression_system_matrix[1,2] = sum(dataframe[:, :SSk].*dataframe[:, :Sk])
regression_system_matrix[1,3] = sum(dataframe[:, :SSk].*dataframe[:, :t].^2)
regression_system_matrix[1,4] = sum(dataframe[:, :SSk].*dataframe[:, :t])
regression_system_matrix[1,5] = sum(dataframe[:, :SSk])

regression_system_matrix[2,1] = regression_system_matrix[1,2]
regression_system_matrix[2,2] = sum(dataframe[:, :Sk].^2)
regression_system_matrix[2,3] = sum(dataframe[:, :Sk].*dataframe[:, :t].^2)
regression_system_matrix[2,4] = sum(dataframe[:, :Sk].*dataframe[:, :t])
regression_system_matrix[2,5] = sum(dataframe[:, :Sk])

regression_system_matrix[3,1] = sum(dataframe[:, :SSk].*dataframe[:, :t].^2)
regression_system_matrix[3,2] = sum(dataframe[:, :Sk].*dataframe[:, :t].^2)
regression_system_matrix[3,3] = sum(dataframe[:, :t].^4)
regression_system_matrix[3,4] = sum(dataframe[:, :t].^3)
regression_system_matrix[3,5] = sum(dataframe[:, :t].^2)

regression_system_matrix[4,1] = sum(dataframe[:, :SSk].*dataframe[:, :t])
regression_system_matrix[4,2] = sum(dataframe[:, :Sk].*dataframe[:, :t])
regression_system_matrix[4,3] = sum(dataframe[:, :t].^3)
regression_system_matrix[4,4] = sum(dataframe[:, :t].^2)
regression_system_matrix[4,5] = sum(dataframe[:, :t].^1)

regression_system_matrix[5,1] = sum(dataframe[:, :SSk])
regression_system_matrix[5,2] = sum(dataframe[:, :Sk])
regression_system_matrix[5,3] = sum(dataframe[:, :t].^2)
regression_system_matrix[5,4] = sum(dataframe[:, :t].^1)
regression_system_matrix[5,5] = length(dataframe[:, :t])
# print(regression_system_matrix)

regression_system_vector = zeros(5,1)
regression_system_vector[1] = sum(dataframe[:, :SSk].*dataframe[:, :kalpha])
regression_system_vector[2] = sum(dataframe[:, :Sk].*dataframe[:, :kalpha])
regression_system_vector[3] = sum(dataframe[:, :kalpha].*dataframe[:, :t].^2)
regression_system_vector[4] = sum(dataframe[:, :kalpha].*dataframe[:, :t].^1)
regression_system_vector[5] = sum(dataframe[:, :kalpha])
# print(regression_system_vector)

ABCDE = inv(regression_system_matrix) * regression_system_vector
# print(ABCDE)
A = ABCDE[1]
B = ABCDE[2]

return A, B, regression_system_matrix, regression_system_vector
end
```

    regression_system (generic function with 1 method)

```julia
function unrelated_coefficients(dataframe,p ,q)
unrelated_coefficient_matrix = zeros(3,3)

unrelated_coefficient_matrix[1,1] = length(dataframe[:, :t])
unrelated_coefficient_matrix[1,2] = sum(exp.(p.*dataframe[:, :t]))
unrelated_coefficient_matrix[1,3] = sum(exp.(q.*dataframe[:, :t]))

unrelated_coefficient_matrix[2,1] = sum(exp.(p.*dataframe[:, :t]))
unrelated_coefficient_matrix[2,2] = sum(exp.(2*p.*dataframe[:, :t]))
unrelated_coefficient_matrix[2,3] = sum(exp.((p+q).*dataframe[:, :t]))

unrelated_coefficient_matrix[3,1] = sum(exp.(q*dataframe[:, :t]))
unrelated_coefficient_matrix[3,2] = sum(exp.((p+q).*dataframe[:, :t]))
unrelated_coefficient_matrix[3,3] = sum(exp.(2*q.*dataframe[:, :t]))
# print(unrelated_coefficient_matrix)

unrelated_coefficient_vector = zeros(3,1)
unrelated_coefficient_vector[1] = sum(dataframe[:, :kalpha])
unrelated_coefficient_vector[2] = sum(dataframe[:, :kalpha].*exp.(p.*dataframe[:, :t]))
unrelated_coefficient_vector[3] = sum(dataframe[:, :kalpha].*exp.(q.*dataframe[:, :t]))
# print(unrelated_coefficient_vector)

abc = inv(unrelated_coefficient_matrix) * unrelated_coefficient_vector
a,b,c = abc
return a,b,c, unrelated_coefficient_matrix, unrelated_coefficient_vector
end
```

    unrelated_coefficients (generic function with 1 method)

Proceed with regression for k

We solve for a,b,c,p,q to get the regression equation:

\begin{equation}
    y = k(t) ^{1-\alpha} = a +b\times e ^{p \times t} +c \times e ^{q \times t}
\end{equation}

Therefore we solve for k(t):
$k(t) = y^\frac{1}{1-\alpha}$

```julia
USA_k_reg = USA_raw[:,[1,5]]
DataFrames.rename!(USA_k_reg, [:k, :t])
USA_k_reg = USA_k_reg[2:end,:]

USA_k_reg[!, :kalpha] = USA_k_reg[!,:k].^(1-USA_alpha) #already sorted in increasing order of k and t

USA_k_reg[!, :Sk] = convert(UnitRange{Float64},0:(size(USA_k_reg[!,:k])[1]-1))
USA_k_reg[!, :SSk] = convert(UnitRange{Float64},0:(size(USA_k_reg[!,:k])[1]-1))

for i in 1:(size(USA_k_reg[!,:k])[1]-1)
    USA_k_reg[i+1, :Sk] = USA_k_reg[i, :Sk] + 0.5*(USA_k_reg[i+1, :kalpha]+USA_k_reg[i, :kalpha])*(USA_k_reg[i+1, :t]-USA_k_reg[i, :t])
    USA_k_reg[i+1, :SSk] = USA_k_reg[i, :SSk] + 0.5*(USA_k_reg[i+1, :Sk]+USA_k_reg[i, :Sk])*(USA_k_reg[i+1, :t]-USA_k_reg[i, :t])
end
# last(USA_k_reg,5)

USA_A,USA_B,USA_regression_system_matrix,USA_regression_system_vector = regression_system(USA_k_reg)

USA_p = 0.5(USA_B+sqrt(USA_B^2+4*USA_A))
USA_q = 0.5(USA_B-sqrt(USA_B^2+4*USA_A))
print(" p = ", USA_p,"\n q = ",USA_q)

USA_a,USA_b,USA_c,USA_unrelated_coefficient_matrix,USA_unrelated_coefficient_vector = unrelated_coefficients(USA_k_reg, USA_p, USA_q)
print("\n a = ", USA_a,"\n b = ",USA_b, "\n c = ",USA_c)

#solving for predicted values of k
USA_k_reg[!,:k_pred] = (USA_a .+ USA_b .* exp.(USA_p.*USA_k_reg[!,:t]) .+ USA_c .* exp.(USA_q.*USA_k_reg[!,:t])).^(1/(1-USA_alpha))
USA_k_reg
```

     p = 0.12426465648334557
     q = -0.1121990327546415
     a = 0.10164676326837592
     b = 0.0004884705570055945
     c = -0.05278388438077906

	k	t	kalpha	Sk	SSk	k_pred

	Float64	Int64	Float64	Float64	Float64	Float64

39 rows × 6 columns

1	0.0632449	2	0.0634654	0.0	0.0	0.059886

2	0.0651057	3	0.0653303	0.0643979	0.0321989	0.0644347

3	0.067476	4	0.0677057	0.130916	0.129856	0.0685208

4	0.0706719	5	0.0709083	0.200223	0.295425	0.0721957

5	0.0743356	6	0.0745795	0.272967	0.53202	0.0755062

6	0.0764952	7	0.0767435	0.348628	0.842818	0.078494

7	0.0777832	8	0.078034	0.426017	1.23014	0.0811973

8	0.0809296	9	0.0811865	0.505627	1.69596	0.0836505

9	0.0851758	10	0.0854407	0.588941	2.24325	0.0858849

10	0.0892787	11	0.0895511	0.676437	2.87594	0.0879294

11	0.0922673	12	0.0925449	0.767485	3.5979	0.0898102

12	0.0947202	13	0.0950021	0.861258	4.41227	0.0915518

13	0.0960069	14	0.0962909	0.956905	5.32135	0.0931769

14	0.0948384	15	0.0951204	1.05261	6.32611	0.094707

15	0.0955838	16	0.0958671	1.1481	7.42646	0.0961624

16	0.0969398	17	0.0972254	1.24465	8.62284	0.0975625

17	0.100121	18	0.100412	1.34347	9.9169	0.0989264

18	0.102487	19	0.102781	1.44507	11.3112	0.100273

19	0.104387	20	0.104685	1.5488	12.8081	0.10162

20	0.105472	21	0.105772	1.65403	14.4095	0.102988

21	0.105757	22	0.106057	1.75994	16.1165	0.104395

22	0.10488	23	0.105178	1.86556	17.9293	0.105862

23	0.104205	24	0.104502	1.9704	19.8472	0.107411

24	0.105723	25	0.106023	2.07566	21.8703	0.109063

25	0.107934	26	0.108238	2.18279	23.9995	0.110843

26	0.110946	27	0.111254	2.29254	26.2372	0.112779

27	0.112828	28	0.113139	2.40474	28.5858	0.114898

28	0.115402	29	0.115716	2.51916	31.0477	0.117234

29	0.118977	30	0.119296	2.63667	33.6257	0.11982

30	0.123078	31	0.123403	2.75802	36.323	0.122698

⋮	⋮	⋮	⋮	⋮	⋮	⋮

```julia
print(ols_y_USA)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,CholeskyPivoted{Float64,Array{Float64,2}}}},Array{Float64,2}}

    lgyty0 ~ 1 + lgktk0 + t

    Coefficients:
    ──────────────────────────────────────────────────────────────────────────────
                       Coef.  Std. Error      t  Pr(>|t|)   Lower 95%    Upper 95%
    ──────────────────────────────────────────────────────────────────────────────
    (Intercept)  -0.0245329   0.00779418  -3.15    0.0033  -0.0403402  -0.00872552
    lgktk0        0.00126064  0.0705531    0.02    0.9858  -0.141828    0.144349
    t             0.0197244   0.00156593  12.60    <1e-14   0.0165486   0.0229003
    ──────────────────────────────────────────────────────────────────────────────

The simplest value to compare is to $g_B$ from the regression of $y(t)$, which should correspond to $p$ in the regression of $k(t)$. These values do not match.

# Fitting of values using linear regression

Using each linear regression to regenerate the values of k(t), y(t) and e(t) from initial conditions, we find that the regressions actually fit the data well despite the regression coefficients/ Green Solow model parameters being inconsistent between equations. This suggests that it may be difficult to fit all three equations with constant and consistent parameters.

```julia
print(mean(USA_raw[!, "United_States_nt"]))
```

    0.010453575379948777

```julia
USA_pred = USA_k_reg[:,[1,2]]
DataFrames.rename!(USA_pred, [:k,:t])
USA_pred[:,:k_pred] = USA_k_reg[!,:k_pred]

USA_pred[!,:y_pred] = USA_y_reg[1,:y].*exp.(coef(ols_y_USA)[2]*log.(USA_k_reg[!,:k_pred]./USA_k_reg[1,:k_pred])
            .+ coef(ols_y_USA)[1]
            .+ coef(ols_y_USA)[3].*USA_k_reg[!,:t])

USA_pred[!,:e_pred] = USA_e_reg[1,:e].*exp.(coef(ols_e_USA)[2]*log.(USA_k_reg[!,:k_pred]./USA_k_reg[1,:k_pred])
            .+ coef(ols_e_USA)[1]
            .+ coef(ols_e_USA)[3].*USA_k_reg[!,:t])
USA_pred[!,:y_actual] = USA_y_reg[!,:y]
USA_pred[!,:e_actual] = USA_e_reg[!,:e]
USA_pred

```

	k	t	k_pred	y_pred	e_pred	y_actual	e_actual

	Float64	Int64	Float64	Float64	Float64	Float64?	Float64?

39 rows × 7 columns

1	0.0632449	2	0.059886	0.0152559	21.2598	0.01503	21.1113

2	0.0651057	3	0.0644347	0.0155612	21.0503	0.015304	20.9802

3	0.067476	4	0.0685208	0.0158724	20.8762	0.015944	21.7486

4	0.0706719	5	0.0721957	0.0161897	20.7296	0.016689	22.5106

5	0.0743356	6	0.0755062	0.0165131	20.6047	0.016491	21.5029

6	0.0764952	7	0.078494	0.0168429	20.4974	0.016284	20.4022

7	0.0777832	8	0.0811973	0.0171791	20.4044	0.016975	21.1576

8	0.0809296	9	0.0836505	0.017522	20.3232	0.017567	21.5325

9	0.0851758	10	0.0858849	0.0178716	20.2516	0.018373	21.973

10	0.0892787	11	0.0879294	0.0182282	20.1881	0.018789	21.7804

11	0.0922673	12	0.0898102	0.0185918	20.1312	0.018577	20.7865

12	0.0947202	13	0.0915518	0.0189626	20.0799	0.018856	19.7668

13	0.0960069	14	0.0931769	0.0193408	20.033	0.018325	18.5905

14	0.0948384	15	0.094707	0.0197264	19.9899	0.01892	18.5715

15	0.0955838	16	0.0961624	0.0201198	19.9496	0.020123	18.9768

16	0.0969398	17	0.0975625	0.0205209	19.9116	0.020717	18.8823

17	0.100121	18	0.0989264	0.0209301	19.8752	0.021236	18.7207

18	0.102487	19	0.100273	0.0213474	19.8399	0.021788	19.3503

19	0.104387	20	0.10162	0.021773	19.805	0.022499	20.0104

20	0.105472	21	0.102988	0.0222071	19.7702	0.023059	20.0758

21	0.105757	22	0.104395	0.0226499	19.7349	0.023201	19.3228

22	0.10488	23	0.105862	0.0231015	19.6987	0.022849	19.0562

23	0.104205	24	0.107411	0.0235621	19.6609	0.023298	19.1394

24	0.105723	25	0.109063	0.0240319	19.6213	0.023616	19.3471

25	0.107934	26	0.110843	0.0245111	19.5793	0.024279	19.3609

26	0.110946	27	0.112779	0.0249999	19.5344	0.024603	19.2765

27	0.112828	28	0.114898	0.0254985	19.4861	0.02523	19.496

28	0.115402	29	0.117234	0.0260071	19.434	0.026052	19.6904

29	0.118977	30	0.11982	0.0265259	19.3775	0.026849	19.5792

30	0.123078	31	0.122698	0.0270552	19.3162	0.027735	19.7272

⋮	⋮	⋮	⋮	⋮	⋮	⋮	⋮

```julia
pred = Array(USA_pred[!, [:k_pred, :y_pred, :e_pred]])
kdata = USA_pred[!, :k]
ydata = USA_pred[!, :y_actual]
edata = USA_pred[!, :e_actual]
USA_k_pred = USA_pred[!, :k_pred]
USA_y_pred = USA_pred[!, :y_pred]
USA_e_pred = USA_pred[!, :e_pred]
```

    39-element Array{Float64,1}:
     21.25978728758043
     21.05032063675229
     20.876198256237366
     20.729560779300837
     20.60471897187664
     20.497424820771403
     20.404422711731332
     20.323160940988302
     20.251599451146507
     20.188077650463605
     20.131221034220932
     20.079873601915654
     20.033047862223604
      ⋮
     19.434004634514558
     19.37752829260339
     19.316203777641533
     19.249540127079378
     19.177056452375098
     19.098288970731346
     19.01279919036143
     18.92018301624989
     18.82008044765211
     18.71218544658236
     18.59625548011186
     18.47212019109886

```julia
#checking the implementation of MSE
# println("(USA Mean Squared Errors) kt: ", Flux.mse(USA_k_reg[!, :k_pred], USA_k_reg[!, :k]),
#     " yt: ", Flux.mse(USA_y_pred, USA_y_reg[!, :y]),
#     " et: ", Flux.mse(USA_e_pred, USA_e_reg[!, :e]))
println("(USA Mean Squared Errors) kt: ", (sqL2dist(USA_k_reg[!, :k_pred], identity.(USA_k_reg[!, :k]))/length(USA_k_reg[!, :k])),
    " yt: ", (sqL2dist(USA_y_pred, identity.(USA_y_reg[!, :y]))/length(USA_y_reg[!, :y])),
    " et: ", (sqL2dist(USA_e_pred, identity.(USA_e_reg[!, :e]))/length(USA_e_reg[!, :e])))
println("(USA Root Mean Squared Errors) kt: ", (L2dist(USA_k_reg[!, :k_pred], identity.(USA_k_reg[!, :k]))/length(USA_k_reg[!, :k])),
    " yt: ", (L2dist(USA_y_pred, identity.(USA_y_reg[!, :y]))/length(USA_y_reg[!, :y])),
    " et: ", (L2dist(USA_e_pred, identity.(USA_e_reg[!, :e]))/length(USA_e_reg[!, :e])))
#checking the implementation of MAE
# println("(USA Mean Absolute Errors) kt: ", Flux.mae(USA_k_reg[!, :k_pred], USA_k_reg[!, :k]),
#     " yt: ", Flux.mae(USA_y_pred, USA_y_reg[!, :y]),
#     " et: ", Flux.mae(USA_e_pred, USA_e_reg[!, :e]))
println("(USA Mean Absolute Errors) kt: ",
    L1dist(convert(Array{Float64},USA_k_reg[!, :k_pred]),USA_k_reg[!, :k])/length(USA_k_reg[!, :k]),
    " yt: ", L1dist(convert(Array{Float64},USA_y_pred),identity.(USA_y_reg[!, :y]))/length(USA_y_reg[!, :y]),
    " et: ", L1dist(convert(Array{Float64},USA_e_pred),identity.(USA_e_reg[!, :e]))/length(USA_e_reg[!, :e]))

function r2_impl(data,pred)
tss = sum((data .-mean(data)).^2)
rss = sum((data .-pred).^2)
r2 = 1- rss/tss
return r2
end
#checking the implementation of R2
# println("(USA R2 Correlation Coefficient) yt: ", r2(ols_y_USA),
#     " et: ", r2(ols_e_USA))
# println("(USA R2 Correlation Coefficient) yt: ", r2_impl(USA_y_reg[!, :lgyty0], predict(ols_y_USA)),
#     " et: ", r2_impl(USA_e_reg[!, :lgete0], predict(ols_e_USA)))
println("(USA R2 Correlation Coefficient) kt: ", r2_impl(USA_k_reg[!, :k], USA_k_reg[!, :k_pred]),
    " yt: ", r2_impl(USA_y_reg[!, :y], USA_y_pred),
    " et: ", r2_impl(USA_e_reg[!, :e], USA_e_pred))

function Ar2_impl(data,pred, p)
tss = sum((data .-mean(data)).^2)
rss = sum((data .-pred).^2)
r2 = 1- rss/tss
n = size(data)[1]
ar2 = 1 - abs((1 - r2)*(n-1)/(n-p-1))
return ar2
end

#checking the implementation of adjusted R2
# println("(USA Adjusted R2) yt: ", adjr2(ols_y_USA),
#     " et: ", adjr2(ols_e_USA))
# println("(USA Adjusted R2) yt: ", Ar2_impl(USA_y_reg[!, :lgyty0], predict(ols_y_USA),2),
#     " et: ", Ar2_impl(USA_e_reg[!, :lgete0], predict(ols_e_USA),2))
# 2 parameters are coeff of lge, coeff of t, y intercept is excluded
# for k, there are 4 parameters excluding intercept
println("(USA Adjusted R2) kt: ", Ar2_impl(USA_k_reg[!, :k], USA_k_reg[!, :k_pred], 4),
    " yt: ", Ar2_impl(USA_y_reg[!, :y], USA_y_pred, 2),
    " et: ", Ar2_impl(USA_e_reg[!, :e], USA_e_pred, 2))

println("(USA Mean Percentage Error) kt: ", mean((USA_k_reg[!, :k].- USA_k_reg[!, :k_pred])./USA_k_reg[!, :k])*100,
    " yt: ", mean((USA_y_reg[!, :y].- USA_y_pred)./USA_y_reg[!, :y])*100,
    " et: ", mean((USA_e_reg[!, :e].- USA_e_pred)./USA_e_reg[!, :e])*100)

println("(USA Mean Absolute Percentage Error) kt: ", mean(abs.((USA_k_reg[!, :k].- USA_k_reg[!, :k_pred])./USA_k_reg[!, :k]))*100,
    " yt: ", mean(abs.((USA_y_reg[!, :y].- USA_y_pred)./USA_y_reg[!, :y]))*100,
    " et: ", mean(abs.((USA_e_reg[!, :e].- USA_e_pred)./USA_e_reg[!, :e]))*100)

println("(USA Mean Arctangent Absolute Percentage Error) kt: ", mean(atan.(abs.((USA_k_reg[!, :k].- USA_k_reg[!, :k_pred])./USA_k_reg[!, :k])))*100,
    " yt: ", mean(atan.(abs.((USA_y_reg[!, :y].- USA_y_pred)./USA_y_reg[!, :y])))*100,
    " et: ", mean(atan.(abs.((USA_e_reg[!, :e].- USA_e_pred)./USA_e_reg[!, :e])))*100)

println("(USA Mean Bias Error) kt: ", mean(USA_k_reg[!, :k].- USA_k_reg[!, :k_pred]),
    " yt: ", mean(USA_y_reg[!, :y].- USA_y_pred),
    " et: ", mean(USA_e_reg[!, :e].- USA_e_pred))

function loglikelihood_calc(data,pred)
rss = sum((data .-pred).^2)
n = length(pred)
return -n/2*(log(2pi*rss/n)+1)
end
# implementation from julia: https://github.com/JuliaStats/GLM.jl/blob/950579166c52ff1cbbeab75cbc23235b30640802/src/lm.jl

#checking implementation for loglikelihood
# println("(USA Loglikelihood) yt: ", loglikelihood(ols_y_USA),
#     " et: ", loglikelihood(ols_e_USA))
# println("(USA Loglikelihood) yt: ", loglikelihood_calc(USA_y_reg[!, :lgyty0], predict(ols_y_USA)),
#     " et: ", loglikelihood_calc(USA_e_reg[!, :lgete0], predict(ols_e_USA)))
println("(USA Loglikelihood) kt: ", loglikelihood_calc(USA_k_reg[!, :k], USA_k_reg[!, :k_pred]),
    " yt: ", loglikelihood_calc(USA_y_reg[!, :y], USA_y_pred),
    " et: ", loglikelihood_calc(USA_e_reg[!, :e], USA_e_pred))

function aic_calc(data, pred, params)
    L = loglikelihood_calc(data,pred)
    return 2*params-2*L
end

#checking implementation for aic
# println("(USA Aikaike Information Criterion) yt: ", aic(ols_y_USA),
#     " et: ", aic(ols_e_USA))
# println("(USA Aikaike Information Criterion) yt: ", aic_calc(USA_y_reg[!, :lgyty0], predict(ols_y_USA),4),
#     " et: ", aic_calc(USA_e_reg[!, :lgete0], predict(ols_e_USA),4))
# 4 degrees of freedom in the model: intercept, two coefficients and variance

println("(USA Aikaike Information Criterion) kt: ", aic_calc(USA_k_reg[!, :k], USA_k_reg[!, :k_pred], 6),
    " yt: ", aic_calc(USA_y_reg[!, :y], USA_y_pred, 4),
    " et: ", aic_calc(USA_e_reg[!, :e], USA_e_pred, 4))

function aicc_calc(data, pred, params)
    n = length(pred)
    L = loglikelihood_calc(data,pred)
    return -2*L + 2*params + 2*params*(params+1)/(n-params-1)
end

# checking implementation for aicc
# println("(USA Corrected AIC) yt: ", aicc(ols_y_USA),
#     " et: ", aicc(ols_e_USA))
# println("(USA Corrected AIC) yt: ", aicc_calc(USA_y_reg[!, :lgyty0], predict(ols_y_USA),4),
#     " et: ", aicc_calc(USA_e_reg[!, :lgete0], predict(ols_e_USA),4))
# 4 degrees of freedom in the model: intercept, two coefficients and variance

println("(USA Corrected AIC) kt: ", aicc_calc(USA_k_reg[!, :k], USA_k_reg[!, :k_pred], 6),
    " yt: ", aicc_calc(USA_y_reg[!, :y], USA_y_pred, 4),
    " et: ", aicc_calc(USA_e_reg[!, :e], USA_e_pred, 4))

function bic_calc(data, pred, params)
    n = length(pred)
    L = loglikelihood_calc(data,pred)
    return -2*L + params*log(n)
end

# checking implementation for bic_calc
# println("(USA Corrected BIC) yt: ", bic(ols_y_USA),
#     " et: ", bic(ols_e_USA))
# println("(USA Corrected BIC) yt: ", bic_calc(USA_y_reg[!, :lgyty0], predict(ols_y_USA),4),
#     " et: ", bic_calc(USA_e_reg[!, :lgete0], predict(ols_e_USA),4))
# 4 degrees of freedom in the model: intercept, two coefficients and variance

println("(USA BIC) kt: ", bic_calc(USA_k_reg[!, :k], USA_k_reg[!, :k_pred], 6),
    " yt: ", bic_calc(USA_y_reg[!, :y], USA_y_pred, 4),
    " et: ", bic_calc(USA_e_reg[!, :e], USA_e_pred, 4))
```

    (USA Mean Squared Errors) kt: 5.678948483119817e-6 yt: 2.1385827570016916e-7 et: 0.6480757609654627
    (USA Root Mean Squared Errors) kt: 0.0003815941085364643 yt: 7.405096576529957e-5 et: 0.1289082123227323
    (USA Mean Absolute Errors) kt: 0.0019583915517779305 yt: 0.00035563186255282205 et: 0.6490451855870963
    (USA R2 Correlation Coefficient) kt: 0.9920351502065367 yt: 0.9915186572251804 et: 0.401868779518018
    (USA Adjusted R2) kt: 0.9910981090543645 yt: 0.9910474715154682 et: 0.36863926726901897
    (USA Mean Percentage Error) kt: -0.04528992632422997 yt: -0.029006754324289948 et: 0.6661073688872203
    (USA Mean Absolute Percentage Error) kt: 1.9010133878548163 yt: 1.6186890227671928 et: 3.226550905701008
    (USA Mean Arctangent Absolute Percentage Error) kt: 1.9004564742376495 yt: 1.6181823433974636 et: 3.223464340464821
    (USA Mean Bias Error) kt: 3.1530937239381956e-8 yt: 8.470225898391878e-7 et: 0.16334750601626732
    (USA Loglikelihood) kt: 180.19691434233724 yt: 244.1414671408362 et: -46.88052314430435
    (USA Aikaike Information Criterion) kt: -348.3938286846745 yt: -480.2829342816724 et: 101.7610462886087
    (USA Corrected AIC) kt: -345.7688286846745 yt: -479.1064636934371 et: 102.93751687684399
    (USA BIC) kt: -338.4124588078966 yt: -473.6286876971538 et: 108.41529287312729

```julia
using LaTeXStrings
using Plots

```

```julia
pred[:,1:2]
```

    39×2 Array{Float64,2}:
     0.059886   0.0152559
     0.0644347  0.0155612
     0.0685208  0.0158724
     0.0721957  0.0161897
     0.0755062  0.0165131
     0.078494   0.0168429
     0.0811973  0.0171791
     0.0836505  0.017522
     0.0858849  0.0178716
     0.0879294  0.0182282
     0.0898102  0.0185918
     0.0915518  0.0189626
     0.0931769  0.0193408
     ⋮
     0.117234   0.0260071
     0.11982    0.0265259
     0.122698   0.0270552
     0.125911   0.027595
     0.129507   0.0281457
     0.133543   0.0287075
     0.13808    0.0292806
     0.143188   0.0298652
     0.148946   0.0304617
     0.155442   0.0310701
     0.162778   0.0316909
     0.171066   0.0323242

```julia
function plot_graphmodel1(nnpred,country,nntstepsfull,kdata,ydata,edata)
        upscale = 0.9 #8x upscaling in resolution
    fntsm = Plots.font("sans-serif", pointsize=round(12.0*upscale))
    fntlg = Plots.font("sans-serif", pointsize=round(16.0*upscale))
    default(titlefont=fntlg, guidefont=fntlg, tickfont=fntsm, legendfont=fntsm)
    default(size=(550*upscale,400*upscale)) #Plot canvas size
    default(dpi=150)

#     fctime = collect(Int(floor(size(nntstepsfull,1)*0.8))+1:1:size(nntstepsfull,1))
#             println(fctime)
#             println(nntstepsfull)
    plotkyenn12ic= scatter(xaxis=nntstepsfull,nnpred[:,1:2],
        label=[L"k_{sol}" L"y_{sol}"],color=[:red :blue],legend=(0.102,0.955),
        markershape=[:circle :diamond],markersize=4,
        ylabel=L"k^c_t,y^t_c",right_margin = 15Plots.mm)

    plot!(plotkyenn12ic,[kdata,ydata],label=[L"k_{data}" L"y_{data}"],linestyle=[:solid :dash],
        color=[:red :blue],ylim=(0,1.5.*round(maximum(kdata);sigdigits=1)),foreground_color_legend = nothing)

#     scatter!(plotkyenn12ic,fctime,nnpred[1:2,fctime]',
#         color=[:red :blue],markershape=[:circle :diamond], markersize=4,label=nothing) #markersize = 8 for predict

    pr = twinx()

    scatter!(pr,nntstepsfull,nnpred[:,3],label=L"e_{sol}",
        color=:green,legend=(0.89,0.954), markershape = [:utriangle], markersize=4, right_margin = 15Plots.mm,
        foreground_color_legend = nothing,background_color_legend = nothing)

# #     scatter!(pr,fctime,nnpred[3,fctime],
# #         label=L"e_{sol}",color=:green,legend=(0.89,0.954), markershape = [:utriangle], markersize=4,
# #         ylabel=L"e^t_c",right_margin = 15Plots.mm,
# #         foreground_color_legend = nothing,background_color_legend = nothing)

    plot!(pr,edata,label=L"e_{data}",color=:green,ylim=(0,1.7.*round(maximum(edata);sigdigits=1)),
                linestyle=:dot)

    xpts = collect(0:5:5*ceil(length(nntstepsfull)/5))
#     if xpts[end] != length(nntsteps)
#         push!(xpts,length(nntsteps))
        xpts[1]=1
#     end

   plot(plotkyenn12ic,xlabel = "years",xticks=xpts,box=:on,linewidth=2*upscale)
#     display(StatsPlots.plot(pl,xlabel = "Days post 21 March 2020",xticks=0:10:110,tickfont=12,
#         guidefont = 16,legendfontsize=12,legend=(0.1,0.92),
#         dpi=300,box=:on,linewidth=2*upscale))
# file_dir1 = @__DIR__
#     savefig(string(file_dir1,"/",country,"model1_maxdata"))
end

plot_graphmodel1(pred,"USA",range(1, 39, length = 39),kdata,ydata,edata)
```

```julia

```