# ConsIndShockModel: Consumption With Shocks

```python
# Initial imports and notebook setup, click arrow to show
import sys
import os

from HARK.ConsumptionSaving.ConsIndShockModel import *
import HARK.ConsumptionSaving.ConsumerParameters as Params
from HARK.utilities import plotFuncsDer, plotFuncs
from time import clock
mystr = lambda number : "{:.4f}".format(number)
```

Defines classes to solve canonical consumption-saving models with idiosyncratic shocks to income.  All models here assume CRRA utility with geometric discounting, no bequest motive, and income shocks are fully transitory or fully permanent.

ConsIndShockModel currently solves three types of models:
1. A basic "perfect foresight" consumption-saving model with no uncertainty.
2. A consumption-saving model with risk over transitory and permanent income shocks.
3. The model described in (2), with an interest rate for debt that differs from the interest rate for savings.

See [NARK](https://github.com/econ-ark/NARK) for information on variable naming conventions.
See [HARK documentation](https://github.com/econ-ark/HARK/tree/master/Documentation) for brief mathematical descriptions of the models being solved.  Detailed mathematical references are referenced _in situ_ below.

## Perfect Foresight CRRA Utility Consumer

The $\mathtt{PerfForesightConsumerType}$ class defines the solution for the problem of a consumer with Constant Relative Risk Aversion utility
$$\newcommand{\CRRA}{\rho}$$
\begin{equation}
U(C) = \frac{C^{1-\CRRA}}{1-\rho}
\end{equation}
has perfect foresight about everything except the (stochastic) date of death, which occurs with constant probability implying a "survival probability" $\newcommand{\LivPrb}{\aleph}\LivPrb < 1$.  Permanent labor income $P_t$ grows from period to period by a factor $\Gamma_t$.  At the beginning of each period $t$, the consumer has some amount of market resources $M_t$ (which includes both market wealth and currrent income) and must choose how much of those resources to consume $C_t$ and how much to retain in a riskless asset $A_t$ which will earn return factor $R$. The agent's flow of utility $U(C_t)$ from consumption is geometrically discounted by factor $\beta$. Between periods, the agent survives with probability $\newcommand{\LivFac}{\aleph}{\LivFac_{t}}$ which results in a further downweighting of future utility because the consumer does not receive utility after death.  For notational simplicity, we omit $\LivFac$ from the statement of the problem (effectively assuming $\LivFac=1$).

The agent's problem can be written in Bellman form as:
$$\newcommand{\DiscFac}{\beta}\renewcommand{\LivFac}{\aleph}{}$$
\begin{eqnarray*}
V_t(M_t,P_t) &=& \max_{C_t}~U(C_t) ~+ \phantom{\LivFac} \DiscFac V_{t+1}(M_{t+1},P_{t+1}), \\
& s.t. & \\
%A_t &=& M_t - C_t, \\
M_{t+1} &=& R (M_{t}-C_{t}) + Y_{t+1}, \\
P_{t+1} &=& \Gamma_{t+1} P_t, \\
\end{eqnarray*}

A particular perfect foresight agent's problem can be characterized by values of risk aversion $\rho$, discount factor $\beta$, and return factor $R$, along with sequences of income growth factors $\{ \Gamma_t \}$ and survival probabilities $\{\LivPrb_t = \LivPrb = 1\}$ (which are allowed to vary by age but which for present purposes we will assume are time invariant at $\LivPrb$.  To keep things simple, let's forget about "sequences" of income growth and mortality, and just think about an $\textit{infinite horizon}$ consumer with constant income growth and survival probability of $\LivFac=1$.

Solve the model described above and in [PerfForesightCRRA](http://econ.jhu.edu/people/ccarroll/public/lecturenotes/consumption/PerfForesightCRRA)

```python
PFexample = PerfForesightConsumerType(**Params.init_perfect_foresight)
PFexample.cycles = 0 # Make this type have an infinite horizon

PFexample.solve()
PFexample.unpackcFunc()

# Plot the perfect foresight consumption function
print('Linear consumption function:')
mMin = PFexample.solution[0].mNrmMin
plotFuncs(PFexample.cFunc[0],mMin,mMin+10)

PFexample.timeFwd()
PFexample.T_sim = 120 # Set number of simulation periods
PFexample.track_vars = ['mNrmNow']
PFexample.initializeSim()
PFexample.simulate()
```

## Consumer with idiosyncratic income shocks

Solve a model like the one analyzed in [BufferStockTheory](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/)

Specifically, our new type of consumer receives two income shocks at the beginning of each period: a completely transitory shock $\newcommand{\tShkEmp}{\theta}{\tShkEmp_t}$ and a completely permanent shock $\newcommand{\pShk}{\psi}{\pShk_t}$.  Moreover, lenders will not let the agent borrow money such that his ratio of end-of-period assets $A_t$ to permanent income $P_t$ is less than $\underline{a}$.  As with the perfect foresight problem, this model can be framed in terms of $\textit{normalized}$ variables, e.g. $m_t \equiv M_t/P_t$.  (See [here](http://econ.jhu.edu/people/ccarroll/papers/BufferStockTheory/) for all the theory).

\begin{eqnarray*}
v_t(m_t) &=& \max_{c_t} {~} U(c_t) + \phantom{\LivPrb} \beta  \mathbb{E}_{t} [(\Gamma_{t+1}\psi_{t+1})^{1-\rho} v_{t+1}(m_{t+1}) ], \\
a_t &=& m_t - c_t, \\
a_t &\geq& \underline{a}, \\
m_{t+1} &=& R/(\Gamma_{t+1} \psi_{t+1}) a_t + \theta_{t+1}, \\
\mathbb{E}[\psi]=\mathbb{E}[\theta] &=& 1, \\
u(c) &=& \frac{c^{1-\rho}}{1-\rho}.
\end{eqnarray*}

HARK represents agents with this kind of problem as instances of the class $\texttt{IndShockConsumerType}$.  To create an $\texttt{IndShockConsumerType}$, we must specify the same set of parameters as for a $\texttt{PerfForesightConsumerType}$, as well as an artificial borrowing constraint $\underline{a}$ and a stochastic process for the income shocks.

The user can specify any desired discrete approximation to a continuous distribution for the IID shocks.  We have built-in tools for constructing thes as discrete equiprobably approximations to lognormal, which is the default assumption.

```python
IndShockExample = IndShockConsumerType(**Params.init_idiosyncratic_shocks)
IndShockExample.cycles = 0 # Make this type have an infinite horizon

start_time = clock()
IndShockExample.solve()
end_time = clock()
print('Solving a consumer with idiosyncratic shocks took ' + mystr(end_time-start_time) + ' seconds.')
IndShockExample.unpackcFunc()
IndShockExample.timeFwd()

# Plot the consumption function and MPC for the infinite horizon consumer
print('Concave consumption function:')
plotFuncs(IndShockExample.cFunc[0],IndShockExample.solution[0].mNrmMin,5)
print('Marginal propensity to consume function:')
plotFuncsDer(IndShockExample.cFunc[0],IndShockExample.solution[0].mNrmMin,5)

# Compare the consumption functions for the perfect foresight and idiosyncratic
# shock types.  Risky income cFunc asymptotically approaches perfect foresight cFunc.
print('Consumption functions for perfect foresight vs idiosyncratic shocks:')
plotFuncs([PFexample.cFunc[0],IndShockExample.cFunc[0]],IndShockExample.solution[0].mNrmMin,100)

# Compare the value functions for the two types
if IndShockExample.vFuncBool:
    print('Value functions for perfect foresight vs idiosyncratic shocks:')
    plotFuncs([PFexample.solution[0].vFunc,IndShockExample.solution[0].vFunc],
                  IndShockExample.solution[0].mNrmMin+0.5,10)

# Simulate some data; results stored in mNrmNow_hist, cNrmNow_hist, and pLvlNow_hist
IndShockExample.T_sim = 120
IndShockExample.track_vars = ['mNrmNow','cNrmNow','pLvlNow']
IndShockExample.makeShockHistory() # This is optional, simulation will draw shocks on the fly if it isn't run.
IndShockExample.initializeSim()
IndShockExample.simulate()
```

## Idiosyncratic shocks consumer with a finite lifecycle

Models of this kinds are described in [SolvingMicroDSOPs](http://econ.jhu.edu/people/ccarroll/SolvingMicroDSOPs) and an example is solved in the [SolvingMicroDSOPs REMARK](https://github.com/econ-ark/REMARK/blob/master/REMARKs/SolvingMicroDSOPs.md).

```python
LifecycleExample = IndShockConsumerType(**Params.init_lifecycle)
LifecycleExample.cycles = 1 # Make this consumer live a sequence of periods -- a lifetime -- exactly once

start_time = clock()
LifecycleExample.solve()
end_time = clock()
print('Solving a lifecycle consumer took ' + mystr(end_time-start_time) + ' seconds.')
LifecycleExample.unpackcFunc()
LifecycleExample.timeFwd()

# Plot the consumption functions during working life
print('Consumption functions while working:')
mMin = min([LifecycleExample.solution[t].mNrmMin for t in range(LifecycleExample.T_cycle)])
plotFuncs(LifecycleExample.cFunc[:LifecycleExample.T_retire],mMin,5)

# Plot the consumption functions during retirement
print('Consumption functions while retired:')
plotFuncs(LifecycleExample.cFunc[LifecycleExample.T_retire:],0,5)
LifecycleExample.timeRev()

# Simulate some data; results stored in mNrmNow_hist, cNrmNow_hist, pLvlNow_hist, and t_age_hist
LifecycleExample.T_sim = 120
LifecycleExample.track_vars = ['mNrmNow','cNrmNow','pLvlNow','t_age']
LifecycleExample.initializeSim()
LifecycleExample.simulate()
```

## "Cyclical" consumer type
Make and solve a "cyclical" consumer type who lives the same four quarters repeatedly.
The consumer has income that greatly fluctuates throughout the year.

```python
CyclicalExample = IndShockConsumerType(**Params.init_cyclical)
CyclicalExample.cycles = 0

start_time = clock()
CyclicalExample.solve()
end_time = clock()
print('Solving a cyclical consumer took ' + mystr(end_time-start_time) + ' seconds.')
CyclicalExample.unpackcFunc()
CyclicalExample.timeFwd()

# Plot the consumption functions for the cyclical consumer type
print('Quarterly consumption functions:')
mMin = min([X.mNrmMin for X in CyclicalExample.solution])
plotFuncs(CyclicalExample.cFunc,mMin,5)

# Simulate some data; results stored in cHist, mHist, bHist, aHist, MPChist, and pHist
CyclicalExample.T_sim = 480
CyclicalExample.track_vars = ['mNrmNow','cNrmNow','pLvlNow','t_cycle']
CyclicalExample.initializeSim()
CyclicalExample.simulate()
```

## Agent with a kinky interest rate (Rboro > RSave)

Models of this kind are analyzed in [A Theory of the Consumption Function, With
and Without Liquidity Constraints](http://www.econ2.jhu.edu/people/ccarroll/ATheoryv3JEP.pdf)
and the [expanded edition](http://www.econ2.jhu.edu/people/ccarroll/ATheoryv3NBER.pdf).

```python
KinkyExample = KinkedRconsumerType(**Params.init_kinked_R)
KinkyExample.cycles = 0 # Make the Example infinite horizon

start_time = clock()
KinkyExample.solve()
end_time = clock()
print('Solving a kinky consumer took ' + mystr(end_time-start_time) + ' seconds.')
KinkyExample.unpackcFunc()
print('Kinky consumption function:')
KinkyExample.timeFwd()
plotFuncs(KinkyExample.cFunc[0],KinkyExample.solution[0].mNrmMin,5)

KinkyExample.T_sim = 120
KinkyExample.track_vars = ['mNrmNow','cNrmNow','pLvlNow']
KinkyExample.initializeSim()
KinkyExample.simulate()
```


# **Project 4 - Q Learning**

For this project, you will be tasked with both implementing and explaining key components of the Q-learning algorithm.

All the code deliverables has to be provided within this notebook.

# 1 - Packages
Let's first import all the packages that you will need during this assignment.

*
[numpy](https://numpy.org/) - is the main package for scientific computing with Python
*
[matplotlib](https://matplotlib.org/) - is a plotting library
*
[gym](https://gym.openai.com/docs/) - Gym is a toolkit for developing and comparing reinforcement learning algorithms.
*
[gym.spaces](http://gym.openai.com/docs/) - Every environment comes with an action_space and an observation_space. These attributes are of type Space, and they describe the format of valid actions and observations.
*
[time](https://docs.python.org/3/library/time.html?highlight=time#module-time) - will be used to track how much time each computation takes
*
[copy](https://docs.python.org/3/library/copy.html) - A copy is sometimes needed so one can change one copy without changing the other.
*
[Threading](https://docs.python.org/3/library/threading.html) - This module constructs higher-level threading interfaces on top of the lower level thread module.
*
[Collections](https://docs.python.org/2/library/collections.html) - This module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict, list, set, and tuple.

```python
#######################################################################
# Authors:
# Nathan Margaglio (nathanmargaglio@gmail.com)
# Mihir Hemant Chauhan (mihirhem@buffalo.edu)
# Qian Cheng (qcheng2@buffalo.edu)
#######################################################################
import numpy as np
import matplotlib.pyplot as plt
import gym
import gym.spaces
import time
import copy
import threading
import time
import collections
import math
import random
import sys
from collections import deque, defaultdict
```

## Basic Environment
Here we define our grid-world environment. No need to make any changes.

```python
class GridEnvironment(gym.Env):
    metadata = { 'render.modes': ['human'] }

    def __init__(self, normalize=False, size=4):
        self.observation_space = gym.spaces.Box(0, size, (size,))
        self.action_space = gym.spaces.Discrete(4)
        self.max_timesteps = size*2 + 1
        self.normalize = normalize
        self.size = size

        # Generate State Transition Table
        self.transition_matrix = []
        for x in range(size + 1):
            state_x = []
            for y in range(size + 1):
                state_y = []
                for a in range(4):
                    one_hot = np.zeros(4)
                    one_hot[a] = 1
                    state_y.append(one_hot)
                state_x.append(state_y)
            self.transition_matrix.append(state_x)

    def transition_func(self, x, y, action, return_probs=False):
        probs = self.transition_matrix[x][y][action]
        if return_probs:
            return probs
        else:
            return np.random.choice(len(probs), p=probs)

    def _get_distance(self, x, y):
        return abs(x[0] - y[0]) + abs(x[1] - y[1])

    def reset(self):
        self.timestep = 0
        self.agent_pos = [0, 0]
        self.goal_pos = [self.size, self.size]
        self.state = np.zeros((self.size + 1, self.size + 1))
        self.state[tuple(self.agent_pos)] = 1
        self.state[tuple(self.goal_pos)] = 0.5
        self.prev_distance = self._get_distance(self.agent_pos, self.goal_pos)
        return np.array(self.agent_pos)/1.

    def step(self, action):
        action_taken = self.transition_func(self.agent_pos[0], self.agent_pos[1], action)
        self.state = np.random.choice(self.observation_space.shape[0])
        if action_taken == 0:
            self.agent_pos[0] += 1
        if action_taken == 1:
            self.agent_pos[0] -= 1
        if action_taken == 2:
            self.agent_pos[1] += 1
        if action_taken == 3:
            self.agent_pos[1] -= 1

        self.agent_pos = np.clip(self.agent_pos, 0, self.size)
        self.state = np.zeros((self.size + 1, self.size + 1))
        self.state[tuple(self.agent_pos)] = 1
        self.state[tuple(self.goal_pos)] = 0.5

        current_distance = self._get_distance(self.agent_pos, self.goal_pos)
        if current_distance < self.prev_distance:
            reward = 1
        elif current_distance > self.prev_distance:
            reward = -1
        else:
            reward = -1
        self.prev_distance = current_distance

        self.timestep += 1
        if self.timestep >= self.max_timesteps or current_distance == 0:
            done = True
        else:
            done = False
        info = {}

        obs = self.agent_pos
        if self.normalize:
            obs = obs/self.size
        return obs, reward, done, info

    def render(self, mode='human'):
        plt.imshow(self.state)
```

```python
env = GridEnvironment()
obs = env.reset()
env.render()
```

## Random Agent
This runs the environment with a random agent that just takes random actions. Neither does he learn, nor remember anything. Try to run it!

```python
class RandomAgent:
    def __init__(self, env):
        self.env = env
        self.observation_space = env.observation_space
        self.action_space = env.action_space

    def policy(self, observation):
        return np.random.choice(self.action_space.n)

    def step(self, observation, verbose=False):
        return self.policy(observation)
```

```python
env = GridEnvironment(normalize=True)
agent = RandomAgent(env)

obs = env.reset()
done = False
agent.epsilon = 0
env.render()
plt.show()

while not done:
    print(obs)
    action = agent.step(obs, verbose=True)
    obs, reward, done, info = env.step(action)
    env.render()
    plt.show()
```

## Heuristic Agent
This runs the environment with a heuristic agent. No need to make any changes. Try to run it!

```python
class HeuristicAgent:
    def __init__(self, env):
        self.env = env
        self.observation_space = env.observation_space
        self.action_space = env.action_space

    def policy(self, observation):
        # 0 - down
        # 1 - up
        # 2 - right
        # 3 - left
        if (observation[0] < 1.):
            return 0
        if (observation[1] < 1.):
            return 2
        return 0

    def step(self, observation, verbose=False):
        if verbose:
            print(observation)
        return self.policy(observation)
```

```python
env = GridEnvironment(normalize=True)
agent = HeuristicAgent(env)

obs = env.reset()
done = False
agent.epsilon = 0
env.render()
plt.show()

while not done:
    action = agent.step(obs, verbose=True)
    obs, reward, done, info = env.step(action)
    env.render()
    plt.show()
```

## Tabular Q-Learning
This is where you need to define policy and update Q tables.
For policy.

*
[np.argmax](https://docs.scipy.org/doc/numpy/reference/generated/numpy.argmax.html) - Returns the indices of the maximum values along an axis.

### Epsilon

Our agent will randomly select its action at first by a certain percentage, called ‘exploration rate’ or ‘epsilon’. This is because at first, it is better for the agent to try all kinds of things before it starts to see the patterns. When it is not deciding the action randomly, the agent will predict the reward value based on the current state and pick the action that will give the highest reward. We want our agent to decrease the number of random action, as it goes, so we indroduce an exponential-decay epsilon, that eventually will allow our agent to explore the evironment. \\

###  Task 1: Implement policy function.

**Instructions:**
- Our agent will randomly select its action at first by a certain percentage, called ‘exploration rate’ or ‘epsilon’. This is because at first, it is better for the agent to try all kinds of things before it starts to see the patterns. Select a random uniform number. If it's less than epsilon, return the random choice action space.
- When it is not deciding the action randomly, the agent will predict the reward value based on the current state and pick the action that will give the highest reward.
\begin{align} \notag
\pi\left(s_{t}\right)=\underset{a \in A}{\operatorname{argmax}} Q_{\theta}\left(s_{t}, a\right)
\end{align}
- Return the policy
- Please note, that the name for all the variables should start with self, thus

epsilon $\rightarrow$ self.epsilon

action_space $\rightarrow$ self.action_space

###  Task 2: Update Q-table

**Instructions:**
            \begin{align} \notag
            Q^{n e w}\left(s_{t}, a_{t}\right) \leftarrow(1-\alpha) \cdot \underbrace{Q\left(s_{t}, a_{t}\right)}_{\text {old value }}+\underbrace{\alpha}_{\text {learning rate }} \cdot \overbrace{(\underbrace{r_{t}}_{\text {reward }} + \underbrace{\gamma}_{\text {discount factor }} \underbrace{\max _{a} Q\left(s_{t+1}, a\right)}_{a})}^{\text {learned value }}
            \end{align}

### Training
### Environment
First, we initialize our environment.  The environment, loosely structured like [OpenAI's Gym Environments](https://gym.openai.com/), has three main methods: `reset`, `step` and `render`. You'll only need `reset` and `step` here.

- When we call **reset**, we initialize the environment with a fresh episode. This allows us to effectively run through episodes (only needing to call reset at the beginning of an episode), but, more importantly, `reset()` returns the environment's initial state.

- The **step** method accepts an action as a parameter (which, for this example, is an integer in [0, 3]), processes the action, and returns the new state, the reward for performing the action, and a boolean indicating if the run is over.

### Agent
When we initialize the agent, we must pass both a `environment` into QLearningAgent function.
###  Task 3: Implement the training algorithm

**Instructions:**
- After initialization, pass the initial state to obs. Then check if it's already done. If done = False, you'll keep going. While it's not done, you'll need to update `state`, `action`,`reward` and `next_state`. You can get action by `step` the current state on agent. Use `copy` to record the current state. `step` the current action on environment to return the new state, the reward for performing the action, a boolean indicating if the run is over and some other information. Add the new reward on the total rewards. Use `copy` to save the new state returned by `step`. Update the `state`, `action`, `reward`, `next_state` of agent.

```python
class QLearningAgent:
    def __init__(self, env, epsilon=1.0, lr=0.1, gamma=0.9):
        self.env = env
        self.observation_space = env.observation_space
        self.action_space = env.action_space
        q_table_dim = env.observation_space.shape[0] + 1
        self.q_table = np.zeros((q_table_dim, q_table_dim, env.action_space.n))
        #self.q_table = defaultdict(lambda: np.zeros(self.action_space.n))
        self.epsilon = epsilon
        self.lr = lr
        self.gamma = gamma

    def policy(self, observation):
      # Code for policy (Task 1) (30 points)
      observation = observation.astype(int)
      if random.random()>self.epsilon:
        return np.argmax(self.q_table[observation[0]][observation[1]])
      else:
        return np.random.choice(env.action_space.n)

    def step(self, observation):
      return self.policy(observation)

    def update(self, state, action, reward, next_state):
        state = state.astype(int)
        next_state = next_state.astype(int)
        # Code for updating Q Table (Task 2) (20 points)
        current = self.q_table[state[0]][state[1]][action]
        Qsa_next = np.max(self.q_table[next_state[0]][next_state[1]]) if next_state is not None else 0
        target = (reward + (self.gamma*Qsa_next))
        new_value = current + self.lr * (target - current)
        return new_value

    def set_epsilon(self, epsilon):
        self.epsilon = epsilon
```

```python
env = GridEnvironment() # note: we do not normalize
agent = QLearningAgent(env)
episodes = 1000 # number of games we want the agent to play
delta_epsilon = agent.epsilon/episodes

# total_rewards = []
epsilons = [agent.epsilon]

nA = env.action_space.n
plot_every = 100
tmp_score = deque(maxlen=plot_every)
avg_score = deque(maxlen=episodes)
best_avg_reward = -math.inf
best_rewards = []

# Training Process (Task 3) (20 points)
for i_episode in range(1,episodes):
        state = env.reset()
        score = 0
        while True:
            state = state.astype(int)
            action = agent.policy(state)
            next_state, reward, done, info = env.step(action)
            score += reward
            next_state = next_state.astype(int)
#            print(state, action, next_state, state.shape)
            agent.q_table[state[0]][state[1]][action] = agent.update(state,action,reward,next_state)
            state = next_state
            if done:
                tmp_score.append(score)
                break
#             if(i_episode%100)==0:
#                 avg_score.append(np.mean(tmp_score))
        if (i_episode >= 100):
            avg_reward = np.mean(tmp_score)
            avg_score.append(avg_reward)
            if avg_reward > best_avg_reward:
                best_avg_reward = avg_reward
                best_rewards.append(best_avg_reward)
            agent.set_epsilon(agent.epsilon*delta_epsilon)
            epsilons.append(agent.epsilon)
        print("\rEpisode {}/{} || Best average reward {}".format(i_episode, episodes, best_avg_reward), end="")
        sys.stdout.flush()

        if i_episode == episodes: print('\n')
```

    Episode 999/1000 || Best average reward 8.0

#### Visualize $\epsilon$
Plot our value of $\epsilon$ over each episode

```python
plt.xlabel('Episode')
plt.ylabel('$\epsilon$')
plt.plot(epsilons)
```

#### Visualize Rewards
Plot total_rewards per episode.  We apply a rolling mean of window $10$ to visualize easier.

```python
window = 10
plt.xlabel('Episode')
plt.ylabel('Total Reward (SMA 10)')
plt.plot([np.mean(list(avg_score)[tr:tr+window]) for tr in range(window, len(avg_score))])
```

```python
env = GridEnvironment()

obs = env.reset()
done = False
agent.epsilon = 0
env.render()
plt.show()

while not done:
    action = agent.step(obs)
    obs, reward, done, info = env.step(action)
    env.render()
    plt.show()
```

Temporal Difference/Temporal_Difference_SARSA_MAX_Q_Learning for Grid World.ipynb


```
import tensorflow as tf

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

#building all kinds of evaluating parameters
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score
from sklearn.metrics import precision_score, recall_score
from sklearn.metrics import f1_score, matthews_corrcoef
from sklearn.metrics import confusion_matrix

from sklearn.preprocessing import RobustScaler
```

```
from google.colab import drive
drive.mount('/content/drive/')
```

    Mounted at /content/drive/

```
p = np.linspace(0.01, 1.0, 50)
# print (p)
logp = -np.log(p)
ones = np.ones(len(p))
def fterm(gamma):
  return (ones - p)**(gamma)

fl2 = fterm(2) * logp

fl0 = fterm(0) * logp
fl5 = fterm(5) * logp
fl0d5 = fterm(0.5) * logp

fig=plt.figure(figsize=(9, 7))
plt.plot(p, logp, 'b-', label='CE Loss')

plt.plot(p, fl0d5, linestyle='-.', color='magenta', label=r'Focal Loss: $\gamma=0.5$')
plt.plot(p, fl2, 'r.', label=r'Focal Loss: $\gamma=2$')
plt.plot(p, fl5, linestyle='--', color='orange', label=r'Focal Loss: $\gamma=5$')
# plt.text(0.1, 4, )
plt.legend(fontsize=12)
plt.xlabel('Probability of Ground Truth Class', fontsize=12)
plt.ylabel('Loss', fontsize=12)

# print (fterm(2))
# plt.savefig('/content/drive/My Drive/Colab Notebooks/Focal_Loss.png', dpi=200)
```

```
credit_df = pd.read_csv('/content/drive/My Drive/Colab Notebooks/creditcard.csv', sep=',')
credit_df.head(3)
```

      Time
      V1
      V2
      V3
      V4
      V5
      V6
      V7
      V8
      V9
      V10
      V11
      V12
      V13
      V14
      V15
      V16
      V17
      V18
      V19
      V20
      V21
      V22
      V23
      V24
      V25
      V26
      V27
      V28
      Amount
      Class

      0
      0.0
      -1.359807
      -0.072781
      2.536347
      1.378155
      -0.338321
      0.462388
      0.239599
      0.098698
      0.363787
      0.090794
      -0.551600
      -0.617801
      -0.991390
      -0.311169
      1.468177
      -0.470401
      0.207971
      0.025791
      0.403993
      0.251412
      -0.018307
      0.277838
      -0.110474
      0.066928
      0.128539
      -0.189115
      0.133558
      -0.021053
      149.62
      0

      1
      0.0
      1.191857
      0.266151
      0.166480
      0.448154
      0.060018
      -0.082361
      -0.078803
      0.085102
      -0.255425
      -0.166974
      1.612727
      1.065235
      0.489095
      -0.143772
      0.635558
      0.463917
      -0.114805
      -0.183361
      -0.145783
      -0.069083
      -0.225775
      -0.638672
      0.101288
      -0.339846
      0.167170
      0.125895
      -0.008983
      0.014724
      2.69
      0

      2
      1.0
      -1.358354
      -1.340163
      1.773209
      0.379780
      -0.503198
      1.800499
      0.791461
      0.247676
      -1.514654
      0.207643
      0.624501
      0.066084
      0.717293
      -0.165946
      2.345865
      -2.890083
      1.109969
      -0.121359
      -2.261857
      0.524980
      0.247998
      0.771679
      0.909412
      -0.689281
      -0.327642
      -0.139097
      -0.055353
      -0.059752
      378.66
      0

```
print ('dataframe shape: ', credit_df.shape)
```

    dataframe shape:  (284807, 31)

```
# check the label distribution

sns.countplot(x='Class', data=credit_df, palette='hls')
plt.xlabel('Label', fontsize=12)
plt.ylabel('Counts', fontsize=12)
plt.show()
```

we see here extremely imbalanced data in action here. Fradulent cases are negligible and that's why it gets difficult to identify the fraud cases ...

```
print ('Cases without frauds: ', len(credit_df[credit_df['Class']==0]))
print ('Cases with frauds: ', len(credit_df[credit_df['Class']==1]))
```

    Cases without frauds:  284315
    Cases with frauds:  492

```
print ('check for null values in each column: ', credit_df.isna().sum())
```

    check for null values in each column:  Time      0
    V1        0
    V2        0
    V3        0
    V4        0
    V5        0
    V6        0
    V7        0
    V8        0
    V9        0
    V10       0
    V11       0
    V12       0
    V13       0
    V14       0
    V15       0
    V16       0
    V17       0
    V18       0
    V19       0
    V20       0
    V21       0
    V22       0
    V23       0
    V24       0
    V25       0
    V26       0
    V27       0
    V28       0
    Amount    0
    Class     0
    dtype: int64

No null values in the data-frame!!

If you see values in the respective columns, _Amount_ and _Time_ are not scaled but other columns are scaled. So to include Amount and Time as features we need to scale them.
Here we can use StandardScaler/RobustScaler to scale these features.

```
## Scale Amount and Time

robust_scaler = RobustScaler()

credit_df[['Amount', 'Time']] = robust_scaler.fit_transform(credit_df[['Amount', 'Time']])

credit_df.head(3)
```

      Time
      V1
      V2
      V3
      V4
      V5
      V6
      V7
      V8
      V9
      V10
      V11
      V12
      V13
      V14
      V15
      V16
      V17
      V18
      V19
      V20
      V21
      V22
      V23
      V24
      V25
      V26
      V27
      V28
      Amount
      Class

      0
      -0.994983
      -1.359807
      -0.072781
      2.536347
      1.378155
      -0.338321
      0.462388
      0.239599
      0.098698
      0.363787
      0.090794
      -0.551600
      -0.617801
      -0.991390
      -0.311169
      1.468177
      -0.470401
      0.207971
      0.025791
      0.403993
      0.251412
      -0.018307
      0.277838
      -0.110474
      0.066928
      0.128539
      -0.189115
      0.133558
      -0.021053
      1.783274
      0

      1
      -0.994983
      1.191857
      0.266151
      0.166480
      0.448154
      0.060018
      -0.082361
      -0.078803
      0.085102
      -0.255425
      -0.166974
      1.612727
      1.065235
      0.489095
      -0.143772
      0.635558
      0.463917
      -0.114805
      -0.183361
      -0.145783
      -0.069083
      -0.225775
      -0.638672
      0.101288
      -0.339846
      0.167170
      0.125895
      -0.008983
      0.014724
      -0.269825
      0

      2
      -0.994972
      -1.358354
      -1.340163
      1.773209
      0.379780
      -0.503198
      1.800499
      0.791461
      0.247676
      -1.514654
      0.207643
      0.624501
      0.066084
      0.717293
      -0.165946
      2.345865
      -2.890083
      1.109969
      -0.121359
      -2.261857
      0.524980
      0.247998
      0.771679
      0.909412
      -0.689281
      -0.327642
      -0.139097
      -0.055353
      -0.059752
      4.983721
      0

```
X_labels = credit_df.drop(['Class'], axis=1)
y_labels = credit_df['Class']

print ('x shape and type: ', X_labels.shape, type(X_labels))
print ('y shape: ', y_labels.shape)

X_labels = X_labels.to_numpy(dtype=np.float64)
y_labels = y_labels.to_numpy(dtype=np.float64)

print ('x shape and type: ', X_labels.shape, type(X_labels))

print ('check few yvals: ', y_labels[0:3])

y_lab_cat = tf.keras.utils.to_categorical(y_labels, num_classes=2, dtype='float32')

print ('categorical y_lab: ', y_lab_cat[0:3])
```

    x shape and type:  (284807, 30) <class 'pandas.core.frame.DataFrame'>
    y shape:  (284807,)
    x shape and type:  (284807, 30) <class 'numpy.ndarray'>
    check few yvals:  [0. 0. 0.]
    categorical y_lab:  [[1. 0.]
     [1. 0.]
     [1. 0.]]

```
x_train, x_test, y_train, y_test = train_test_split(X_labels, y_lab_cat, test_size=0.3,
                                                    stratify=y_lab_cat, shuffle=True)

# stratify ensure same proportion of class labels

print('train data shape and type: ', x_train.shape, type(x_train))
print ('test data shape: ', x_test.shape)

print ('check few y vals: ', y_train[0:3])
```

    train data shape and type:  (199364, 30) <class 'numpy.ndarray'>
    test data shape:  (85443, 30)
    check few y vals:  [[1. 0.]
     [1. 0.]
     [1. 0.]]

```
from tensorflow.keras.layers import  Dense, Input, Activation, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import activations
from tensorflow import keras

from keras import backend as K
```

```
def simple_model():
  input_data  = Input(shape=(x_train.shape[1], ))
  x = Dense(64)(input_data)
  x = Activation(activations.relu)(x)
  x = Dense(32)(x)
  x = Activation(activations.relu)(x)
  x = Dense(2)(x)
  x = Activation(activations.softmax)(x)

  model = Model(inputs=input_data, outputs=x, name='Simple_Model')

  return model
```

```
def simple_model1():
  input_data  = Input(shape=(x_train.shape[1], ))
  x = Dense(64)(input_data)
  x = Activation(activations.relu)(x)
  x = Dense(32)(x)
  x = Activation(activations.relu)(x)
  x = Dense(2)(x)
  x = Activation(activations.softmax)(x)

  model = Model(inputs=input_data, outputs=x, name='Simple_Model')

  return model
```

```
simple_model = simple_model()
simple_model.summary()
```

```
# simple_model.compile(optimizer=Adam(learning_rate=1e-2), loss='sparse_categorical_crossentropy',
#                       metrics=['acc'])

simple_model.compile(optimizer=Adam(learning_rate=5e-3), loss='categorical_crossentropy',
                      metrics=['acc'])
```

```
simple_model.fit(x_train, y_train, validation_split=0.2, epochs=5, shuffle=True, batch_size=256)
```

    Epoch 1/5
    624/624 [==============================] - 2s 2ms/step - loss: 0.0393 - acc: 0.9834 - val_loss: 0.0034 - val_acc: 0.9994
    Epoch 2/5
    624/624 [==============================] - 1s 2ms/step - loss: 0.0028 - acc: 0.9995 - val_loss: 0.0032 - val_acc: 0.9995
    Epoch 3/5
    624/624 [==============================] - 1s 2ms/step - loss: 0.0027 - acc: 0.9994 - val_loss: 0.0028 - val_acc: 0.9994
    Epoch 4/5
    624/624 [==============================] - 1s 2ms/step - loss: 0.0023 - acc: 0.9994 - val_loss: 0.0029 - val_acc: 0.9995
    Epoch 5/5
    624/624 [==============================] - 1s 2ms/step - loss: 0.0021 - acc: 0.9995 - val_loss: 0.0034 - val_acc: 0.9992

    <tensorflow.python.keras.callbacks.History at 0x7f7d92e1b4a8>

```
def conf_matrix(predictions, class_types, y_test_argmax=False):
    ''' Plots conf. matrix and classification report '''
    if y_test_argmax==True:
      cm=confusion_matrix(np.argmax(y_test, axis=1), np.argmax(np.round(predictions), axis=1))
      cr=classification_report(np.argmax(y_test, axis=1),
                                np.argmax(np.round(predictions), axis=1),
                                target_names=[class_types[i] for i in range(len(class_types))])
      print(cr)
    else:
      cm=confusion_matrix(y_test, np.argmax(np.round(predictions), axis=1))
      print("Classification Report:\n")
      cr=classification_report(y_test,
                                  np.argmax(np.round(predictions), axis=1),
                                  target_names=[class_types[i] for i in range(len(class_types))])
      print(cr)
    plt.figure(figsize=(6,6))

    sns_hmp = sns.heatmap(cm, annot=True, xticklabels = [class_types[i] for i in range(len(class_types))],
                yticklabels = [class_types[i] for i in range(len(class_types))], fmt="d", cmap="YlGnBu")

    plt.xlabel('Pred Class')
    plt.ylabel('True Class')
    fig = sns_hmp.get_figure()
    # fig.savefig('/content/gdrive/My Drive/Colab Notebooks/resnet/heatmap.png', dpi=250)
```

```
y_pred = simple_model.predict(x_test, batch_size=64)

# y_pred_argmax = np.argmax(y_pred, axis=1)
```

```
conf_matrix(y_pred, ['Real', 'Fraud'], y_test_argmax=True)
```

```
print ('check first 3 y pred: ', y_pred[0:3])
print ('check first 3 y_test: ', y_test[0:3])
print ('check argmax of pred:',  np.argmax(np.round(y_pred[0:3]), axis=1))
```

    check first 3 y pred:  [[9.9999940e-01 5.8592815e-07]
     [9.9996805e-01 3.1988002e-05]
     [9.9940896e-01 5.9102214e-04]]
    check first 3 y_test:  [[1. 0.]
     [1. 0.]
     [1. 0.]]
    check argmax of pred: [0 0 0]

### Focal Loss:

We will see that cross-entropy loss is a special case of Focal loss.

Definition :

$\text{FL}(p_t) = - \alpha _t \, \left(1-p_t\right)^{\gamma}\, \text{log}(p_t)$

$$\begin{equation}p_t = \begin{cases}p\, ; & \text{if}\, \, y = 1 \\ 1-p\, ; & \text{otherwise} \end{cases} \end{equation}$$

When $\gamma = 0, \alpha _t = 1$, then FL is Cross Entropy loss.

```
## how does it affect ?

y_example_true = [1.0]
y_example_pred = [0.90]
y_example_pred0 = [0.95]
y_example_pred1 = [0.20]

binary_ce = tf.keras.losses.BinaryCrossentropy()
loss_close = binary_ce(y_example_true, y_example_pred).numpy()
loss_veryclose = binary_ce(y_example_true, y_example_pred0).numpy()
loss_far = binary_ce(y_example_true, y_example_pred1).numpy()

print ('CE loss when pred is close to true: ', loss_close)
print ('CE loss when pred is very close to true: ', loss_veryclose)
print ('CE loss when pred is far from true: ', loss_far)

focal_factor_close = (1.0-0.90)**2 ## (take gamma = 2, as in paper)
focal_factor_veryclose = (1.0-0.95)**2
focal_factor_far = (1.0-0.20)**2 ## ()

print ('\n')

print ('focal loss when pred is close to true: ', loss_close*focal_factor_close)
print ('focal loss when pred is very close to true: ', loss_veryclose*focal_factor_veryclose)

print ('focal loss when pred is far from true: ', loss_far*focal_factor_far)
```

    CE loss when pred is close to true:  0.10536041
    CE loss when pred is very close to true:  0.051293183
    CE loss when pred is far from true:  1.6094373

    focal loss when pred is close to true:  0.0010536041110754007
    focal loss when pred is very close to true:  0.00012823295779526255
    focal loss when pred is far from true:  1.0300399017333985

Penalize way more when prediction is far from true, i.e. focus on the wrongly classified samples

```
import tensorflow_addons as tfa
```

```
fl_tensorflow = tfa.losses.SigmoidFocalCrossEntropy(alpha=1.0, gamma=2.0)
fl_close = fl_tensorflow(y_example_true, y_example_pred).numpy()
fl_far = fl_tensorflow(y_example_true, y_example_pred1).numpy()

print ('check values: ', fl_close, fl_far)
```

    check values:  0.008925734 1.03004

Custom loss function

```
def focal_loss_custom(alpha, gamma):
  def binary_focal_loss(y_true, y_pred):

    fl = tfa.losses.SigmoidFocalCrossEntropy(alpha=alpha, gamma=gamma)
    # pred_argmax = np.argmax(np.round(y_pred), axis=1)
    # pred_argmax_tensor = tf.convert_to_tensor(pred_argmax, dtype=tf.float32)
    # y_true_tensor = tf.convert_to_tensor(y_true, dtype=tf.float32)
    y_true_K = K.ones_like(y_true)
    focal_loss = fl(y_true, y_pred)
    # focal_loss = tf.reduce_mean(focal_loss)
    return focal_loss
  return binary_focal_loss
```

```
simple_model1 = simple_model1()
simple_model1.compile(optimizer=Adam(learning_rate=5e-3), loss=focal_loss_custom(alpha=0.2, gamma=2.0), metrics=['acc'])
```

```
simple_model1.fit(x_train, y_train, validation_split=0.2, epochs=5, shuffle=True, batch_size=256)
```

    Epoch 1/5
    624/624 [==============================] - 2s 2ms/step - loss: 0.0135 - acc: 0.9844 - val_loss: 0.0012 - val_acc: 0.9993
    Epoch 2/5
    624/624 [==============================] - 1s 2ms/step - loss: 8.6970e-04 - acc: 0.9994 - val_loss: 0.0013 - val_acc: 0.9991
    Epoch 3/5
    624/624 [==============================] - 1s 2ms/step - loss: 8.5765e-04 - acc: 0.9994 - val_loss: 0.0011 - val_acc: 0.9993
    Epoch 4/5
    624/624 [==============================] - 1s 2ms/step - loss: 7.4777e-04 - acc: 0.9994 - val_loss: 0.0012 - val_acc: 0.9993
    Epoch 5/5
    624/624 [==============================] - 1s 2ms/step - loss: 8.3562e-04 - acc: 0.9994 - val_loss: 0.0012 - val_acc: 0.9993

    <tensorflow.python.keras.callbacks.History at 0x7f7a3594e390>

```
y_pred = simple_model1.predict(x_test, batch_size=32)

```

```
confusion_matrix(np.argmax(y_test, axis=1), np.argmax(np.round(y_pred), axis=1))
```

    array([[85289,     6],
           [   47,   101]])

```
conf_matrix(y_pred, ['Real', 'Fraud'], y_test_argmax=True)
```

```
print ('check first 3 ypred: ', y_pred[0:4])
print ('check first 3 ytest: ', y_test[0:4])

print (np.argmax(np.round(y_pred[0:4]), axis=1))

print (tf.cast(y_test[0:4], dtype=tf.float32))
print (K.ones_like(y_test[0:4]))
```

    check first 3 ypred:  [[0.9667234  0.03327663]
     [0.9822015  0.01779843]
     [0.99318063 0.00681934]
     [0.9779823  0.02201766]]
    check first 3 ytest:  [[1. 0.]
     [1. 0.]
     [1. 0.]
     [1. 0.]]
    [0 0 0 0]
    tf.Tensor(
    [[1. 0.]
     [1. 0.]
     [1. 0.]
     [1. 0.]], shape=(4, 2), dtype=float32)
    tf.Tensor(
    [[1. 1.]
     [1. 1.]
     [1. 1.]
     [1. 1.]], shape=(4, 2), dtype=float32)

```

```


# Lucas_Task20

### 3 - Regarding the distribution problem for the elbow muscles presented in this text:
a. Test different initial values for the optimization.

b. Test other values for the elbow angle where the results are likely to change.

```python
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

```python
r_ef = np.loadtxt('C:/Users/ebm/Downloads/Lucas/r_elbowflexors.mot', skiprows=7)
f_ef = np.loadtxt('C:/Users/ebm/Downloads/Lucas/f_elbowflexors.mot', skiprows=7)
```

```python
m_ef = r_ef*1
m_ef[:, 2:] = r_ef[:, 2:]*f_ef[:, 2:]
```

```python
labels = ['Biceps long head', 'Biceps short head', 'Brachialis']
fig, ax = plt.subplots(nrows=1, ncols=3, sharex=True, figsize=(10, 4))
ax[0].plot(r_ef[:, 1], r_ef[:, 2:])
#ax[0].set_xlabel('Elbow angle $(\,^o)$')
ax[0].set_title('Moment arm (m)')
ax[1].plot(f_ef[:, 1], f_ef[:, 2:])
ax[1].set_xlabel('Elbow angle $(\,^o)$', fontsize=16)
ax[1].set_title('Maximum force (N)')
ax[2].plot(m_ef[:, 1], m_ef[:, 2:])
#ax[2].set_xlabel('Elbow angle $(\,^o)$')
ax[2].set_title('Maximum torque (Nm)')
ax[2].legend(labels, loc='best', framealpha=.5)
ax[2].set_xlim(np.min(r_ef[:, 1]), np.max(r_ef[:, 1]))
plt.tight_layout()
plt.show()
```

```python
a_ef = np.array([624.3, 435.56, 987.26])/50  # 50 N/cm2
print(a_ef)
```

    [ 12.486    8.7112  19.7452]

#### Test 1

```python
M = 40  # desired torque at the elbow
iang = 80  # which will give the closest value to 90 degrees
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 40
    x0 = [ 51.78948736  31.97872006  83.65566559]
    r * x0 = 6.11695790214

#### Test 2

```python
M = 80  # desired torque at the elbow
iang = 40  # which will give the closest value to 90 degrees
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 80
    x0 = [ 62.58312845  42.94361276  97.71786525]
    r * x0 = 5.38553888055

#### Test 3

```python
M = 50  # desired torque at the elbow
iang = 60  # which will give the closest value to 90 degrees
r  = r_ef[iang, 2:]
f0 = f_ef[iang, 2:]
a  = a_ef
m  = m_ef[iang, 2:]
x0 = f_ef[iang, 2:]/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 50
    x0 = [ 60.67171435  39.21827211  93.34593473]
    r * x0 = 6.61186324316

### 4 - In an experiment to estimate forces of the elbow flexors, through inverse dynamics it was found an elbow flexor moment of 10 Nm. Consider the following data for maximum force (F0), moment arm (r), and pcsa (A) of the brachialis, brachioradialis, and biceps brachii muscles: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A (cm2): 33, 8, 23, respectively (data from Robertson et al. (2013)).

a. Use static optimization to estimate the muscle forces.

b. Test the robustness of the results using different initial values for the muscle forces.

c. Compare the results for different cost functions.

#### Test 1

```python
#Valores fornecidos: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A(cm²): 33, 8, 23

M = 40  # desired torque at the elbow
iang = 80 # which will give the closest value to 90 degrees
r  = np.array([2, 5, 4])
f0 = np.array([1000, 250, 700])
a  = np.array([33, 8, 23])
m  = r*f0
x0 = f0/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 40
    x0 = [ 100.   25.   70.]
    r * x0 = 605.0

#### Test 2

```python
#Valores fornecidos: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A(cm²): 33, 8, 23

M = 80  # desired torque at the elbow
iang = 40 # which will give the closest value to 90 degrees
r  = np.array([2, 5, 4])
f0 = np.array([1000, 250, 700])
a  = np.array([33, 8, 23])
m  = r*f0
x0 = f0/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 80
    x0 = [ 100.   25.   70.]
    r * x0 = 605.0

#### Test 3

```python
#Valores fornecidos: F0 (N): 1000, 250, 700; r (cm): 2, 5, 4; A(cm²): 33, 8, 23

M = 50  # desired torque at the elbow
iang = 60 # which will give the closest value to 90 degrees
r  = np.array([2, 5, 4])
f0 = np.array([1000, 250, 700])
a  = np.array([33, 8, 23])
m  = r*f0
x0 = f0/10  # far from the correct answer for the sum of torques
print('M =', M)
print('x0 =', x0)
print('r * x0 =', np.sum(r*x0))
```

    M = 50
    x0 = [ 100.   25.   70.]
    r * x0 = 605.0