# Assignment 2 - Semi-gradient TD with a Neural Network

Welcome to Course 3 Programming Assignment 2. In the previous assignment, you implemented semi-gradient TD with State Aggregation for solving a **policy evaluation task**. In this assignment, you will implement **semi-gradient TD with a simple Neural Network** and use it for the same policy evaluation problem.

You will implement an agent to evaluate a fixed policy on the 500-State Randomwalk. As you may remember from the previous assignment, the 500-state Randomwalk includes 500 states. Each episode begins with the agent at the center and terminates when the agent goes far left beyond state 1 or far right beyond state 500. At each time step, the agent selects to move either left or right with equal probability. The environment determines how much the agent moves in the selected direction.

**In this assignment, you will:**
- Implement stochastic gradient descent method for state-value prediction.
- Implement semi-gradient TD with a neural network as the function approximator and Adam algorithm.
- Compare performance of semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

## Packages
We import the following libraries that are required for this assignment:

- [numpy](www.numpy.org) : Fundamental package for scientific computing with Python.
- [matplotlib](http://matplotlib.org) : Library for plotting graphs in Python.
- [RL-Glue](http://www.jmlr.org/papers/v10/tanner09a.html) : Library for reinforcement learning experiments.
- [tqdm](https://tqdm.github.io/) : A package to display progress bar when running experiments.
- BaseOptimizer : An abstract class that specifies the optimizer API for Agent.
- plot_script : Custom script to plot results.
- RandomWalkEnvironment : The Randomwalk environment script from Course 3 Assignment 1.

```python
# Do not modify this cell!

# Import necessary libraries
# DO NOT IMPORT OTHER LIBRARIES - This will break the autograder.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import os, shutil
from tqdm import tqdm

from rl_glue import RLGlue
from environment import BaseEnvironment
from agent import BaseAgent
from optimizer import BaseOptimizer
import plot_script
from randomwalk_environment import RandomWalkEnvironment
```

## Section 1: Create semi-gradient TD with a Neural Network
In this section, you will implement an Agent that learns with semi-gradient TD with a neural network. You will use a neural network with one hidden layer. The input of the neural network is the one-hot encoding of the state number. We use the one-hot encoding of the state number instead of the state number itself because we do not want to build the prior knowledge that integer number inputs close to each other have similar values. The hidden layer contains 100 rectifier linear units (ReLUs) which pass their input if it is bigger than one and return 0 otherwise. ReLU gates are commonly used in neural networks due to their nice properties such as the sparsity of the activation and having non-vanishing gradients. The output of the neural network is the estimated state value. It is a linear function of the hidden units as is commonly the case when estimating the value of a continuous target using neural networks.

The neural network looks like this:

For a given input, $s$, value of $s$ is computed by:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

where $W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$  are the parameters of the network and will be learned when training the agent.

## 1-1: Implement helper methods

Before implementing the agent, you first implement some helper functions which you will later use in agent's main methods.

### Implement `get_value()`
First, you will implement get_value() method which feeds an input $s$ into the neural network and returns the output of the network $v$ according to the equations above. To implement get_value(), take into account the following notes:

- `get_value()` gets the one-hot encoded state number denoted by s as an input.
- `get_value()` receives the weights of the neural network as input, denoted by weights and structured as an array of dictionaries. Each dictionary corresponds to weights from one layer of the neural network to the next. Each dictionary includes $W$ and $b$. The shape of the elements in weights are as follows:
    - weights[0]["W"]: num_states $\times$ num_hidden_units
    - weights[0]["b"]: 1 $\times$ num_hidden_units
    - weights[1]["W"]: num_hidden_units $\times$ 1
    - weights[1]["b"]: 1 $\times$ 1

- The input of the neural network is a sparse vector. To make computation faster, we take advantage of input sparsity. To do so, we provided a helper method `my_matmul()`. **Make sure that you use `my_matmul()` for all matrix multiplications except for element-wise multiplications in this notebook.**
- The max operator used for computing $x$ is element-wise.

```python
def my_matmul(x1, x2):
    """
    Given matrices x1 and x2, return the multiplication of them
    """

    result = np.zeros((x1.shape[0], x2.shape[1]))
    x1_non_zero_indices = x1.nonzero()
    if x1.shape[0] == 1 and len(x1_non_zero_indices[1]) == 1:
        result = x2[x1_non_zero_indices[1], :]
    elif x1.shape[1] == 1 and len(x1_non_zero_indices[0]) == 1:
        result[x1_non_zero_indices[0], :] = x2 * x1[x1_non_zero_indices[0], 0]
    else:
        result = np.matmul(x1, x2)
    return result
```

```python
#GRADED FUNCTION: [get_value]

def get_value(s, weights):
    """
    Compute value of input s given the weights of a neural network
    """
    ### Compute the ouput of the neural network, v, for input s (3 lines)

    ### START CODE HERE ###

    phi = my_matmul(s, weights[0]['W']) + weights[0]['b']
    x = np.maximum(0, phi)
    v = my_matmul(x, weights[1]['W']) + weights[1]['b']

    ### END CODE HERE ###
    return v
```

Run the following code to test your implementation of the `get_value()` function:

```python
## Test Code for get_value() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 10
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_value_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

estimated_value = get_value(s, weights)
print ("Estimated value: {}".format(estimated_value))
assert(np.allclose(estimated_value, np.array([[-0.21915705]])))

print ("Passed the assert!")

```

    Estimated value: [[-0.21915705]]
    Passed the assert!

**Expected output**:

    Estimated value: [[-0.21915705]]

### Implement `get_gradient()`
You will also implement `get_gradient()` method which computes the gradient of the value function for a given input, using backpropagation. You will later use this function to update the value function.

As you know, we compute the value of a state $s$ according to:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

To update the weights of the neural network ($W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$), we compute the gradient of $v$ with respect to the weights according to:

$$
\begin{align}
\frac{\partial v}{\partial W^{[0]}} &= s^T(W^{[1]T} \odot I_{x>0}) \\
\frac{\partial v}{\partial b^{[0]}} &= W^{[1]T} \odot I_{x>0} \\
\frac{\partial v}{\partial W^{[1]}} &= x^T \\
\frac{\partial v}{\partial b^{[1]}} &= 1
\end{align}
$$
where $\odot$ denotes element-wise matrix multiplication and $I_{x>0}$ is the gradient of the ReLU activation function which is an indicator whose $i$th element is 1 if $x[i]>0$ and 0 otherwise.

```python
#GRADED FUNCTION: [get_gradient]

def get_gradient(s, weights):
    """
    Given inputs s and weights, return the gradient of v with respect to the weights
    """

    ### Compute the gradient of the value function with respect to W0, b0, W1, b1 for input s (6~8 lines)
    # grads[0]["W"] = ?
    # grads[0]["b"] = ?
    # grads[1]["W"] = ?
    # grads[1]["b"] = ?
    # Note that grads[0]["W"], grads[0]["b"], grads[1]["W"], and grads[1]["b"] should have the same shape as
    # weights[0]["W"], weights[0]["b"], weights[1]["W"], and weights[1]["b"] respectively
    # Note that to compute the gradients, you need to compute the activation of the hidden layer (x)

    grads = [dict() for i in range(len(weights))]

    ### START CODE HERE ###

    phi = my_matmul(s, weights[0]['W']) + weights[0]['b']
    x = np.maximum(0, phi)

    grads[0]['W'] = my_matmul(s.T, (weights[1]['W'].T * (x > 0) ) )
    grads[0]['b'] = weights[1]['W'].T * (x > 0)
    grads[1]['W'] = x.T
    grads[1]['b'] = 1

    ### END CODE HERE ###

    return grads

```

Run the following code to test your implementation of the `get_gradient()` function:

```python
## Test Code for get_gradient() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_gradient_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

grads = get_gradient(s, weights)

grads_answer = np.load("asserts/get_gradient_grads.npz")

print("grads[0][\"W\"]\n", grads[0]["W"], "\n")
print("grads[0][\"b\"]\n", grads[0]["b"], "\n")
print("grads[1][\"W\"]\n", grads[1]["W"], "\n")
print("grads[1][\"b\"]\n", grads[1]["b"], "\n")

assert(np.allclose(grads[0]["W"], grads_answer["W0"]))
assert(np.allclose(grads[0]["b"], grads_answer["b0"]))
assert(np.allclose(grads[1]["W"], grads_answer["W1"]))
assert(np.allclose(grads[1]["b"], grads_answer["b1"]))

print("Passed the asserts!")

```

    grads[0]["W"]
     [[0.         0.        ]
     [0.         0.        ]
     [0.         0.        ]
     [0.76103773 0.12167502]
     [0.         0.        ]]

    grads[0]["b"]
     [[0.76103773 0.12167502]]

    grads[1]["W"]
     [[0.69198983]
     [0.82403662]]

    grads[1]["b"]
     1

    Passed the asserts!

**Expected output**:

    grads[0]["W"]
     [[0.         0.        ]
     [0.         0.        ]
     [0.         0.        ]
     [0.76103773 0.12167502]
     [0.         0.        ]]

    grads[0]["b"]
     [[0.76103773 0.12167502]]

    grads[1]["W"]
     [[0.69198983]
     [0.82403662]]

    grads[1]["b"]
     [[1.]]

### Implement stochastic gradient descent method for state-value prediction
In this section, you will implement stochastic gradient descent (SGD) method for state_value prediction. Here is the basic SGD update for state-value prediction with TD:

$$\mathbf{w_{t+1}} = \mathbf{w_{t}} + \alpha \delta_t \nabla \hat{v}(S_t,\mathbf{w_{t}})$$

At each time step, we update the weights in the direction  $g_t = \delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$ using a fixed step-size $\alpha$. $\delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1},\mathbf{w_{t}}) - \hat{v}(S_t,\mathbf{w_t})$ is the TD-error. $\nabla \hat{v}(S_t,\mathbf{w_{t}})$ is the gradient of the value function with respect to the weights.

The following cell includes the SGD class. You will complete the `update_weight()` method of SGD assuming that the weights and update g are provided.

**As you know, in this assignment, we structured the weights as an array of dictionaries. Note that the updates $g_t$, in the case of TD, is $\delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$. As a result, $g_t$ has the same structure as $\nabla \hat{v}(S_t,\mathbf{w_t})$ which is also an array of dictionaries.**

```python
#GRADED FUNCTION: [SGD]

class SGD(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the stochastic gradient descent method.

        Assume optimizer_info dict contains:
        {
            step_size: float
        }
        """
        self.step_size = optimizer_info.get("step_size")

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """
        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update weights (1 line)
                # weights[i][param] = None

                ### START CODE HERE ###

                weights[i][param] = weights[i][param] + self.step_size * g[i][param]

                ### END CODE HERE ###

        return weights
```

Run the following code to test your implementation of the `update_weights()` function:

```python
# Do not modify this cell!

## Test Code for update_weights() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1

weights_data = np.load("asserts/update_weights_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

g_data = np.load("asserts/update_weights_g.npz")
g = [dict() for i in range(num_hidden_layer+1)]
g[0]["W"] = g_data["W0"]
g[0]["b"] = g_data["b0"]
g[1]["W"] = g_data["W1"]
g[1]["b"] = g_data["b1"]

test_sgd = SGD()
optimizer_info = {"step_size": 0.3}
test_sgd.optimizer_init(optimizer_info)
updated_weights = test_sgd.update_weights(weights, g)

# updated weights asserts
updated_weights_answer = np.load("asserts/update_weights_updated_weights.npz")

print("updated_weights[0][\"W\"]\n", updated_weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", updated_weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", updated_weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", updated_weights[1]["b"], "\n")

assert(np.allclose(updated_weights[0]["W"], updated_weights_answer["W0"]))
assert(np.allclose(updated_weights[0]["b"], updated_weights_answer["b0"]))
assert(np.allclose(updated_weights[1]["W"], updated_weights_answer["W1"]))
assert(np.allclose(updated_weights[1]["b"], updated_weights_answer["b1"]))

print("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.17899492  0.53656321]
     [ 0.58008221  1.47666572]
     [ 1.01909411 -1.10248056]
     [ 0.72490408  0.06828853]
     [-0.20609725  0.69034095]]

    updated_weights[0]["b"]
     [[-0.18484533  0.92844539]]

    updated_weights[1]["W"]
     [[0.70488257]
     [0.58150878]]

    updated_weights[1]["b"]
     [[0.88467086]]

    Passed the asserts!

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.17899492  0.53656321]
     [ 0.58008221  1.47666572]
     [ 1.01909411 -1.10248056]
     [ 0.72490408  0.06828853]
     [-0.20609725  0.69034095]]

    updated_weights[0]["b"]
     [[-0.18484533  0.92844539]]

    updated_weights[1]["W"]
     [[0.70488257]
     [0.58150878]]

    updated_weights[1]["b"]
     [[0.88467086]]

### Adam Algorithm
In this assignment, instead of using SGD for updating the weights, we use a more advanced algorithm called Adam. The Adam algorithm improves the SGD update with two concepts: adaptive vector step-sizes and momentum. It keeps estimates of the mean and second moment of the updates, denoted by $\mathbf{m}$ and $\mathbf{v}$ respectively:
$$\mathbf{m_t} = \beta_m \mathbf{m_{t-1}} + (1 - \beta_m)g_t \\
\mathbf{v_t} = \beta_v \mathbf{v_{t-1}} + (1 - \beta_v)g^2_t
$$

Given that $\mathbf{m}$ and $\mathbf{v}$ are initialized to zero, they are biased toward zero. To get unbiased estimates of the mean and second moment, Adam defines $\mathbf{\hat{m}}$ and $\mathbf{\hat{v}}$ as:
$$ \mathbf{\hat{m_t}} = \frac{\mathbf{m_t}}{1 - \beta_m^t} \\
\mathbf{\hat{v_t}} = \frac{\mathbf{v_t}}{1 - \beta_v^t}
$$

The weights are then updated as follows:
$$ \mathbf{w_t} = \mathbf{w_{t-1}} + \frac{\alpha}{\sqrt{\mathbf{\hat{v_t}}}+\epsilon} \mathbf{\hat{m_t}}
$$

When implementing the agent you will use the Adam algorithm instead of SGD because it is more efficient. We have already provided you the implementation of the Adam algorithm in the cell below. You will use it when implementing your agent.

```python
class Adam(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the Adam algorithm.

        Assume optimizer_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
        }
        """

        self.num_states = optimizer_info.get("num_states")
        self.num_hidden_layer = optimizer_info.get("num_hidden_layer")
        self.num_hidden_units = optimizer_info.get("num_hidden_units")

        # Specify Adam algorithm's hyper parameters
        self.step_size = optimizer_info.get("step_size")
        self.beta_m = optimizer_info.get("beta_m")
        self.beta_v = optimizer_info.get("beta_v")
        self.epsilon = optimizer_info.get("epsilon")

        self.layer_size = np.array([self.num_states, self.num_hidden_units, 1])

        # Initialize Adam algorithm's m and v
        self.m = [dict() for i in range(self.num_hidden_layer+1)]
        self.v = [dict() for i in range(self.num_hidden_layer+1)]

        for i in range(self.num_hidden_layer+1):

            # Initialize self.m[i]["W"], self.m[i]["b"], self.v[i]["W"], self.v[i]["b"] to zero
            self.m[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.m[i]["b"] = np.zeros((1, self.layer_size[i+1]))
            self.v[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.v[i]["b"] = np.zeros((1, self.layer_size[i+1]))

        # Initialize beta_m_product and beta_v_product to be later used for computing m_hat and v_hat
        self.beta_m_product = self.beta_m
        self.beta_v_product = self.beta_v

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """

        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update self.m and self.v
                self.m[i][param] = self.beta_m * self.m[i][param] + (1 - self.beta_m) * g[i][param]
                self.v[i][param] = self.beta_v * self.v[i][param] + (1 - self.beta_v) * (g[i][param] * g[i][param])

                ### compute m_hat and v_hat
                m_hat = self.m[i][param] / (1 - self.beta_m_product)
                v_hat = self.v[i][param] / (1 - self.beta_v_product)

                ### update weights
                weights[i][param] += self.step_size * m_hat / (np.sqrt(v_hat) + self.epsilon)

        ### update self.beta_m_product and self.beta_v_product
        self.beta_m_product *= self.beta_m
        self.beta_v_product *= self.beta_v

        return weights

```

## 1-2: Implement Agent Methods
In this section, you will implement `agent_init()`, `agent_start()`, `agent_step()`, and `agent_end()`.

In `agent_init()`, you will:

   - specify the neural network structure by filling self.layer_size with the size of the input layer, hidden layer, and output layer.
   - initialize the network's parameters. We show the parameters as an array of dictionaries, self.weights, where each dictionary corresponds to weights from one layer to the next. Each dictionary includes $W$ and $b$. To initialize the parameters, you will use a normal distribution with mean 0 and standard deviation $\sqrt{\frac{2}{\text{# input of each node}}}$. This initialization heuristic is commonly used when using ReLU gates and helps keep the output of a neuron from getting too big or too small. To initialize the network's parameters, use **self.rand_generator.normal()** which draws random samples from a normal distribution. The parameters of self.rand_generator.normal are mean of the distribution, standard deviation of the distribution, and output shape in the form of tuple of integers.

In `agent_start()`, you will:
   - specify self.last_state and self.last_action.

In `agent_step()` and `agent_end()`, you will:
   - compute the TD error using $v(S_t)$ and $v(S_{t+1})$. To compute the value function for $S_t$ and $S_{t+1}$, you will get their one-hot encoding using `one_hot()` method that we provided below. You feed the one-hot encoded state number to the neural networks using `get_value()` method that you implemented above. Note that `one_hot()` method returns the one-hot encoding of a state as a numpy array of shape (1, num_states).
   - retrieve the gradients using `get_gradient()` function that you implemented.
   - use Adam_algorithm that we provided to update the neural network's parameters, self.weights.
   - use `agent_policy()` method to select actions with. (only in `agent_step()`)

```python
def one_hot(state, num_states):
    """
    Given num_state and a state, return the one-hot encoding of the state
    """
    # Create the one-hot encoding of state
    # one_hot_vector is a numpy array of shape (1, num_states)

    one_hot_vector = np.zeros((1, num_states))
    one_hot_vector[0, int((state - 1))] = 1

    return one_hot_vector
```

```python
#GRADED FUNCTION: [Agent]

class TDAgent(BaseAgent):
    def __init__(self):
        self.name = "td_agent"
        pass

    def agent_init(self, agent_info={}):
        """Setup for the agent called when the experiment first starts.

        Set parameters needed to setup the semi-gradient TD with a Neural Network.

        Assume agent_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            discount_factor: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
            seed: int
        }
        """

        # Set random seed for weights initialization for each run
        self.rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set random seed for policy for each run
        self.policy_rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set attributes according to agent_info
        self.num_states = agent_info.get("num_states")
        self.num_hidden_layer = agent_info.get("num_hidden_layer")
        self.num_hidden_units = agent_info.get("num_hidden_units")
        self.discount_factor = agent_info.get("discount_factor")

        ### Define the neural network's structure (1 line)
        # Specify self.layer_size which shows the number of nodes in each layer
        # self.layer_size = np.array([None, None, None])
        # Hint: Checkout the NN diagram at the beginning of the notebook

        ### START CODE HERE ###

        self.layer_size = np.array( [self.num_states, self.num_hidden_units, 1] )

        ### END CODE HERE ###

        # Initialize the neural network's parameter (2 lines)
        self.weights = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):

            ### Initialize self.weights[i]["W"] and self.weights[i]["b"] using self.rand_generator.normal()
            # Note that The parameters of self.rand_generator.normal are mean of the distribution,
            # standard deviation of the distribution, and output shape in the form of tuple of integers.
            # To specify output shape, use self.layer_size.

            ### START CODE HERE ###

            self.weights[i]['W'] = self.rand_generator.normal( 0, np.sqrt(2/(self.layer_size[i])), (self.layer_size[i], self.layer_size[i+1]) )
            self.weights[i]['b'] = self.rand_generator.normal( 0, np.sqrt(2/(self.layer_size[i])),(1,self.layer_size[i+1]) )

            ### END CODE HERE ###

        # Specify the optimizer
        self.optimizer = Adam()
        optimizer_info = {"num_states": agent_info["num_states"],
                          "num_hidden_layer": agent_info["num_hidden_layer"],
                          "num_hidden_units": agent_info["num_hidden_units"],
                          "step_size": agent_info["step_size"],
                          "beta_m": agent_info["beta_m"],
                          "beta_v": agent_info["beta_v"],
                          "epsilon": agent_info["epsilon"]}
        self.optimizer.optimizer_init(optimizer_info)

        self.last_state = None
        self.last_action = None

    def agent_policy(self, state):

        ### Set chosen_action as 0 or 1 with equal probability.
        chosen_action = self.policy_rand_generator.choice([0,1])
        return chosen_action

    def agent_start(self, state):
        """The first method called when the experiment starts, called after
        the environment starts.
        Args:
            state (Numpy array): the state from the
                environment's evn_start function.
        Returns:
            The first action the agent takes.
        """
        ### select action given state (using self.agent_policy()), and save current state and action (2 lines)
        # self.last_state = ?
        # self.last_action = ?

        ### START CODE HERE ###

        self.last_state = state
        self.last_action = self.agent_policy(self.last_state)

        ### END CODE HERE ###

        return self.last_action

    def agent_step(self, reward, state):
        """A step taken by the agent.
        Args:
            reward (float): the reward received for taking the last action taken
            state (Numpy array): the state from the
                environment's step based, where the agent ended up after the
                last step
        Returns:
            The action the agent is taking.
        """

        ### Compute TD error (5 lines)
        # delta = None

        ### START CODE HERE ###
        delta = reward + self.discount_factor * get_value(one_hot(state, self.num_states), self.weights) - get_value(one_hot(self.last_state,self.num_states), self.weights)
        ### END CODE HERE ###

        ### Retrieve gradients (1 line)
        # grads = None

        ### START CODE HERE ###

        grads = get_gradient( one_hot(self.last_state, self.num_states), self.weights )

        ### END CODE HERE ###

        ### Compute g (1 line)
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                ### START CODE HERE ###

                g[i][param] = delta * grads[i][param]

                ### END CODE HERE ###

        ### update the weights using self.optimizer (1 line)
        # self.weights = None

        ### START CODE HERE ###

        self.weights = self.optimizer.update_weights(self.weights, g)

        ### END CODE HERE ###

        ### update self.last_state and self.last_action (2 lines)

        ### START CODE HERE ###

        self.last_state = state
        self.last_action = self.agent_policy(self.last_state)

        ### END CODE HERE ###

        return self.last_action

    def agent_end(self, reward):
        """Run when the agent terminates.
        Args:
            reward (float): the reward the agent received for entering the
                terminal state.
        """

        ### compute TD error (3 lines)
        # delta = None

        ### START CODE HERE ###

        delta = reward - get_value( one_hot(self.last_state, self.num_states), self.weights )

        ### END CODE HERE ###

        ### Retrieve gradients (1 line)
        # grads = None

        ### START CODE HERE ###

        grads = get_gradient( one_hot(self.last_state, self.num_states), self.weights )

        ### END CODE HERE ###

        ### Compute g (1 line)
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                ### START CODE HERE ###

                g[i][param] = delta * grads[i][param]

                ### END CODE HERE ###

        ### update the weights using self.optimizer (1 line)
        # self.weights = None

        ### START CODE HERE ###

        self.weights = self.optimizer.update_weights(self.weights, g)

        ### END CODE HERE ###

    def agent_message(self, message):
        if message == 'get state value':
            state_value = np.zeros(self.num_states)
            for state in range(1, self.num_states + 1):
                s = one_hot(state, self.num_states)
                state_value[state - 1] = get_value(s, self.weights)
            return state_value
```

Run the following code to test your implementation of the `agent_init()` function:

```python
## Test Code for agent_init() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.25,
              "discount_factor": 0.9,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

print("layer_size: {}".format(test_agent.layer_size))
assert(np.allclose(test_agent.layer_size, np.array([agent_info["num_states"],
                                                    agent_info["num_hidden_units"],
                                                    1])))

print("weights[0][\"W\"] shape: {}".format(test_agent.weights[0]["W"].shape))
print("weights[0][\"b\"] shape: {}".format(test_agent.weights[0]["b"].shape))
print("weights[1][\"W\"] shape: {}".format(test_agent.weights[1]["W"].shape))
print("weights[1][\"b\"] shape: {}".format(test_agent.weights[1]["b"].shape), "\n")

assert(test_agent.weights[0]["W"].shape == (agent_info["num_states"], agent_info["num_hidden_units"]))
assert(test_agent.weights[0]["b"].shape == (1, agent_info["num_hidden_units"]))
assert(test_agent.weights[1]["W"].shape == (agent_info["num_hidden_units"], 1))
assert(test_agent.weights[1]["b"].shape == (1, 1))

print("weights[0][\"W\"]\n", (test_agent.weights[0]["W"]), "\n")
print("weights[0][\"b\"]\n", (test_agent.weights[0]["b"]), "\n")
print("weights[1][\"W\"]\n", (test_agent.weights[1]["W"]), "\n")
print("weights[1][\"b\"]\n", (test_agent.weights[1]["b"]), "\n")

agent_weight_answer = np.load("asserts/agent_init_weights_1.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_weight_answer["b1"]))

print("Passed the asserts!")

```

    layer_size: [5 2 1]
    weights[0]["W"] shape: (5, 2)
    weights[0]["b"] shape: (1, 2)
    weights[1]["W"] shape: (2, 1)
    weights[1]["b"] shape: (1, 1)

    weights[0]["W"]
     [[ 1.11568467  0.25308164]
     [ 0.61900825  1.4172653 ]
     [ 1.18114738 -0.6180848 ]
     [ 0.60088868 -0.0957267 ]
     [-0.06528133  0.25968529]]

    weights[0]["b"]
     [[0.09110115 0.91976332]]

    weights[1]["W"]
     [[0.76103773]
     [0.12167502]]

    weights[1]["b"]
     [[0.44386323]]

    Passed the asserts!

**Expected output**:

    layer_size: [5 2 1]
    weights[0]["W"] shape: (5, 2)
    weights[0]["b"] shape: (1, 2)
    weights[1]["W"] shape: (2, 1)
    weights[1]["b"] shape: (1, 1)

    weights[0]["W"]
     [[ 1.11568467  0.25308164]
     [ 0.61900825  1.4172653 ]
     [ 1.18114738 -0.6180848 ]
     [ 0.60088868 -0.0957267 ]
     [-0.06528133  0.25968529]]

    weights[0]["b"]
     [[0.09110115 0.91976332]]

    weights[1]["W"]
     [[0.76103773]
     [0.12167502]]

    weights[1]["b"]
     [[0.44386323]]

Run the following code to test your implementation of the `agent_start()` function:

```python
# Do not modify this cell!

## Test Code for agent_start() ##

agent_info = {"num_states": 500,
              "num_hidden_layer": 1,
              "num_hidden_units": 100,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 10
             }

# Suppose state = 250
state = 250

test_agent = TDAgent()
test_agent.agent_init(agent_info)
test_agent.agent_start(state)

print("Agent state: {}".format(test_agent.last_state))
print("Agent selected action: {}".format(test_agent.last_action))

assert(test_agent.last_state == 250)
assert(test_agent.last_action == 1)

print("Passed the asserts!")

```

    Agent state: 250
    Agent selected action: 1
    Passed the asserts!

**Expected output**:

    Agent state: 250
    Agent selected action: 1

Run the following code to test your implementation of the `agent_step()` function:

```python
# Do not modify this cell!

## Test Code for agent_step() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_step_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 3
start_state = 3
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and the next state observed was State 1
reward = 10.0
next_state = 1
test_agent.agent_step(reward, next_state)

# updated weights asserts
print("updated_weights[0][\"W\"]\n", test_agent.weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", test_agent.weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", test_agent.weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", test_agent.weights[1]["b"], "\n")

agent_updated_weight_answer = np.load("asserts/agent_step_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))

# last_state and last_action assert
print("Agent last state:", test_agent.last_state)
print("Agent last action:", test_agent.last_action, "\n")

assert(test_agent.last_state == 1)
assert(test_agent.last_action == 1)

print ("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.23397791 -0.48152743]
     [ 0.72792093 -0.15829832]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.29798822 0.96254535]]

    updated_weights[1]["W"]
     [[0.76628754]
     [0.11486511]]

    updated_weights[1]["b"]
     [[0.58530057]]

    Agent last state: 1
    Agent last action: 1

    Passed the asserts!

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.23397791 -0.48152743]
     [ 0.72792093 -0.15829832]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.29798822 0.96254535]]

    updated_weights[1]["W"]
     [[0.76628754]
     [0.11486511]]

    updated_weights[1]["b"]
     [[0.58530057]]

    Agent last state: 1
    Agent last action: 1

Run the following code to test your implementation of the `agent_end()` function:

```python
# Do not modify this cell!

## Test Code for agent_end() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_end_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 4
start_state = 4
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and reached the terminal state
reward = 10.0
test_agent.agent_end(reward)

# updated weights asserts
print("updated_weights[0][\"W\"]\n", test_agent.weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", test_agent.weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", test_agent.weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", test_agent.weights[1]["b"], "\n")

agent_updated_weight_answer = np.load("asserts/agent_end_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))

print ("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.17531054 -0.51043162]
     [ 0.75062903 -0.13736817]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.30846523 0.95937346]]

    updated_weights[1]["W"]
     [[0.68861703]
     [0.15986364]]

    updated_weights[1]["b"]
     [[0.586074]]

    Passed the asserts!

**Expected output:**

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.17531054 -0.51043162]
     [ 0.75062903 -0.13736817]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.30846523 0.95937346]]

    updated_weights[1]["W"]
     [[0.68861703]
     [0.15986364]]

    updated_weights[1]["b"]
     [[0.586074]]

## Section 2 - Run Experiment

Now that you implemented the agent, we can run the experiment. Similar to Course 3 Programming Assignment 1, we will plot the learned state value function and the learning curve of the TD agent. To plot the learning curve, we use Root Mean Squared Value Error (RMSVE).

## 2-1: Run Experiment for Semi-gradient TD with a Neural Network

We have already provided you the experiment/plot code, so you can go ahead and run the two cells below.

Note that running the cell below will take **approximately 12 minutes**.

```python
# Do not modify this cell!

true_state_val = np.load('data/true_V.npy')
state_distribution = np.load('data/state_distribution.npy')

def calc_RMSVE(learned_state_val):
    assert(len(true_state_val) == len(learned_state_val) == len(state_distribution))
    MSVE = np.sum(np.multiply(state_distribution, np.square(true_state_val - learned_state_val)))
    RMSVE = np.sqrt(MSVE)
    return RMSVE

# Define function to run experiment
def run_experiment(environment, agent, environment_parameters, agent_parameters, experiment_parameters):

    rl_glue = RLGlue(environment, agent)

    # save rmsve at the end of each episode
    agent_rmsve = np.zeros((experiment_parameters["num_runs"],
                            int(experiment_parameters["num_episodes"]/experiment_parameters["episode_eval_frequency"]) + 1))

    # save learned state value at the end of each run
    agent_state_val = np.zeros((experiment_parameters["num_runs"],
                                environment_parameters["num_states"]))

    env_info = {"num_states": environment_parameters["num_states"],
                "start_state": environment_parameters["start_state"],
                "left_terminal_state": environment_parameters["left_terminal_state"],
                "right_terminal_state": environment_parameters["right_terminal_state"]}

    agent_info = {"num_states": environment_parameters["num_states"],
                  "num_hidden_layer": agent_parameters["num_hidden_layer"],
                  "num_hidden_units": agent_parameters["num_hidden_units"],
                  "step_size": agent_parameters["step_size"],
                  "discount_factor": environment_parameters["discount_factor"],
                  "beta_m": agent_parameters["beta_m"],
                  "beta_v": agent_parameters["beta_v"],
                  "epsilon": agent_parameters["epsilon"]
                 }

    print('Setting - Neural Network with 100 hidden units')
    os.system('sleep 1')

    # one agent setting
    for run in tqdm(range(1, experiment_parameters["num_runs"]+1)):
        env_info["seed"] = run
        agent_info["seed"] = run
        rl_glue.rl_init(agent_info, env_info)

        # Compute initial RMSVE before training
        current_V = rl_glue.rl_agent_message("get state value")
        agent_rmsve[run-1, 0] = calc_RMSVE(current_V)

        for episode in range(1, experiment_parameters["num_episodes"]+1):
            # run episode
            rl_glue.rl_episode(0) # no step limit

            if episode % experiment_parameters["episode_eval_frequency"] == 0:
                current_V = rl_glue.rl_agent_message("get state value")
                agent_rmsve[run-1, int(episode/experiment_parameters["episode_eval_frequency"])] = calc_RMSVE(current_V)
            elif episode == experiment_parameters["num_episodes"]: # if last episode
                current_V = rl_glue.rl_agent_message("get state value")

        agent_state_val[run-1, :] = current_V

    save_name = "{}".format(rl_glue.agent.name).replace('.','')

    if not os.path.exists('results'):
                os.makedirs('results')

    # save avg. state value
    np.save("results/V_{}".format(save_name), agent_state_val)

    # save avg. rmsve
    np.savez("results/RMSVE_{}".format(save_name), rmsve = agent_rmsve,
                                                   eval_freq = experiment_parameters["episode_eval_frequency"],
                                                   num_episodes = experiment_parameters["num_episodes"])

# Run Experiment

# Experiment parameters
experiment_parameters = {
    "num_runs" : 20,
    "num_episodes" : 1000,
    "episode_eval_frequency" : 10 # evaluate every 10 episode
}

# Environment parameters
environment_parameters = {
    "num_states" : 500,
    "start_state" : 250,
    "left_terminal_state" : 0,
    "right_terminal_state" : 501,
    "discount_factor" : 1.0
}

# Agent parameters
agent_parameters = {
    "num_hidden_layer": 1,
    "num_hidden_units": 100,
    "step_size": 0.001,
    "beta_m": 0.9,
    "beta_v": 0.999,
    "epsilon": 0.0001,
}

current_env = RandomWalkEnvironment
current_agent = TDAgent

# run experiment
run_experiment(current_env, current_agent, environment_parameters, agent_parameters, experiment_parameters)

# plot result
plot_script.plot_result(["td_agent"])

shutil.make_archive('results', 'zip', 'results')
```

You plotted the learning curve for 1000 episodes. As you can see the RMSVE is still decreasing. Here we provide the pre-computed result for 5000 episodes and 20 runs so that you can see the performance of semi-gradient TD with a neural network after being trained for a long time.

Does semi-gradient TD with a neural network find a good approximation within 5000 episodes?

As you may remember from the previous assignment, semi-gradient TD with 10-state aggregation converged within 100 episodes. Why is TD with a neural network slower?

Would it be faster if we decrease the number of hidden units? Or what about if we increase the number of hidden units?

## 2-2: Compare Performance of Semi-gradient TD with a Neural Network and Semi-gradient TD with Tile-coding

In this section, we compare the performance of semi-gradient TD with a Neural Network and semi-gradient TD with tile-coding. Tile-coding is a kind of coarse coding that uses multiple overlapping partitions of the state space to produce features. For tile-coding, we used 50 tilings each with 6 tiles. We set the step-size for semi-gradient TD with tile-coding to $\frac{0.1}{\text{# tilings}}$. See the figure below for the comparison between semi-gradient TD with tile-coding and semi-gradient TD with a neural network and Adam algorithm. This result is for 5000 episodes and 20 runs:

How are the results?

Semi-gradient TD with tile-coding is much faster than semi-gradient TD with a neural network. Why?

Which method has a lower RMSVE at the end of 5000 episodes?

### Wrapping up!

You have successfully implemented Course 3 Programming Assignment 2.

You have implemented **semi-gradient TD with a Neural Network and Adam algorithm** in 500-state Random Walk.

You also compared semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

From the experiments and lectures, you should be more familiar with some of the strengths and weaknesses of using neural networks as the function approximator for an RL agent. On one hand, neural networks are powerful function approximators capable of representing a wide class of functions. They are also capable of producing features without exclusively relying on hand-crafted mechanisms. On the other hand, compared to a linear function approximator with tile-coding, neural networks can be less sample efficient. When implementing your own Reinforcement Learning agents, you may consider these strengths and weaknesses to choose the proper function approximator for your problems.

---

**Note**: Apart from using the 'Submit' button in the notebook, you have to submit an additional `zip` file containing the 'npy' files that were generated from running the experiment cells. To do so:

1. Generate the zip file by running the experiment cells in the notebook. On the top of the notebook, navigate to 'File->Open' to open the directory view of this assignment. Select "results.zip" and click on "Download". Alternatively, you can download just the results folder and run "zip -jr results.zip results/" (The flag 'j' is required by the grader!).
2. Go to the "My submission" tab and click on "+ Create submission".
3. Click on "C3M2 Data-file Grader" and upload your `results.zip`.

**These account for 40% of the marks, so don't forget to do so!**

```python

```

Reinforcement Learning Specialization/3. Prediction and Control with Function Approximation/Assignment2-v7.ipynb


```python
# magics: ensures that any changes to the modules loaded below will be re-loaded automatically
%load_ext autoreload
%autoreload 2
%matplotlib inline

# Load packages
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')

# Load user written modules
import tools
import model
import vfi
import egm
import fd
```

## Model description

The simple specification of our model in discrete time is given by the recursive problem
\begin{equation}
	\begin{aligned}
		v(a_t)  &=  \max_{c}  u(c_t) + \beta \mathrm{E}_t \big[ v(a_{t+1})\big] \\
		& \text{s.t.} \\
		a_{t+1} &= y_{t+1} + (1+r)(a_t - c_t). \\
        a_t & \geq 0
	\end{aligned}
\end{equation}
where
\begin{equation}
y_t = \begin{cases}
y_1 \quad \text{w. prob.} \quad \pi \\
y_2 \quad \text{w. prob.} \quad 1-\pi
\end{cases}
\end{equation}

and instantaneous utility is given by
\begin{equation}
u(c_t) = \begin{cases}
\frac{c^{1-\eta} - 1}{1-\eta} \quad & \text{for} \quad \eta \neq 1 \\
\log c_t \quad & \text{for} \quad \eta = 1
\end{cases}
\end{equation}

First we setup all parameters used in the solution algortihms

```python
par = model.setup()
```

## Solve by value function iteration (VFI)

We can then solve the model using value function iteration. Write a bit about the algorithm here.

```python
sol_vfi = vfi.solve_VFI(par)
```

Check how many iterations are used to find the convergent solution

```python
print("Using value function iteration required " + str(sol_vfi.it) +" iterations before convergence")
```

    Using value function iteration required 267 iterations before convergence

## Solve by the endogenous grid method (EGM)

It seems that to solve the model using the endogenous grid methods, we again rely on the contraction mapping theorem. Hence, the only difference from VFI is that we update each iteration more efficiently.

The idea of EGM is to use the Euler equation to solve the problem:
\begin{equation}
c_t^{-\rho} = (1+r)\beta \mathrm{E}_t \Big[ \big(c_{t+1}(a_{t+1})\big)^{-\rho} \Big] = (1+r)\beta  \mathrm{E}_t \Big[ \big(c_{t+1}\big((1+r)a_t-c_t\big) \big)^{-\rho} \Big]
\end{equation}
In the RHS i've inserted the transition rule for assets. Note that in our model assets are the same as the state ****** Ved ikke helt hvad der menes her.

Run the EGM algortihm

```python
par = model.setup()
sol_egm = egm.solve_EGM(par)
```

Again, we check the number of iterations used

```python
print("Using value function iteration required " + str(sol_egm.it) +" iterations before convergence")
```

    Using value function iteration required 56 iterations before convergence

Besides using considerably less iterations, we also note that each iteration should be much quicker for EGM as we can back out optimal consumption given last iteration WITHOUT using any solver

To assert that our algorithms have run correctly, we can compare the converged policy functions derived from each algorithm

```python
fig = plt.figure(figsize=(14,5))

ax = fig.add_subplot(1,2,1)

ax.plot(sol_egm.a, sol_egm.c, linestyle = ':', color = '0.4')
ax.plot(sol_vfi.a, sol_vfi.c, linestyle = '-', color = '0.7')
ax.set_xlabel(f"Assets, $a_t$")
ax.set_ylabel(f"Consumption, $c^\star_t$")
ax.set_title(f'Policy function')
ax.set_xlim([-1,20])

plt.show()
```

We can see that the two solutions are quite close.

We now extend the EGM algortihm to solve a general Markov process for wages. The model is therefore
\begin{equation}
	\begin{aligned}
		v_j(a_t)  &=  \max_{c}  u(c_t) + \beta \Big[ P_{jj} v_j(a_{t+1}) + (1-P_{jj})v_{-j}(a_{t+1}) \big] \\
		& \text{s.t.} \\
		a_{t+1} &= y_j + (1+r)(a_t - c_t). \\
        a_t & \geq 0
	\end{aligned}
\end{equation}
for $j \in \{1,2\}$.

We have the transition matrix between states
\begin{equation}
\mathbf{P} = \begin{pmatrix}
P_{11} & 1 - P_{11} \\
1 - P_{22} & P_{22}
\end{pmatrix}
\end{equation}
where $P_{11}$ is the probability of of staying in state 1 conditional on being in state 1 and $P_{22}$ is the probability of staying in state 2 conditional on being in state 2. (Kommenter på, hvilken state vi antager er hhv. employment og unemployment)
\end{equation}

**Run the new and improved EGM solver**

```python
par = model.setup()
sol_egm_2d = egm.solve_EGM_2d(par)
```

```python
fig = plt.figure(figsize=(8,6))

ax = fig.add_subplot(1,1,1)

ax.plot(sol_egm_2d.a[0,:], sol_egm_2d.c[0,:], linestyle = ':', color = 'red', label = '$y_1$')
ax.plot(sol_egm_2d.a[1,:], sol_egm_2d.c[1,:], linestyle = ':', color = 'blue', label = '$y_2$')
# ax.plot(sol_egm_2d.a[1,:10], sol_egm_2d.a[1,:10], linestyle = '--', color = '0.6') # Check with 45 degree line. Seems correct
ax.set_xlabel(f"Assets, $a_t$")
ax.set_ylabel(f"Consumption, $c^\star_t$")
ax.set_title(f'Policy function')
ax.set_xlim([-1,20])

ax.legend(frameon=True)

plt.plot()
```

```python
print("We again used " + str(sol_egm_2d.it) +" iterations before convergence")
```

    We again used 56 iterations before convergence

## Finite Difference (FD) ##

Some test code here

```python
par = model.setup()
```

```python
sol_fd = fd.solve_fd(par)

fig = plt.figure(figsize=(14,5))

ax = fig.add_subplot(1,2,1)
ax.spy(sol_fd.A,markersize=1)
plt.show()
```

```python
fig = plt.figure(figsize=(14,5))

ax = fig.add_subplot(1,2,1)

ax.plot(sol_fd.a[:-1], sol_fd.c[0,:], linestyle = ':', color = '0.4')
ax.plot(sol_fd.a[:-1], sol_fd.c[1,:], linestyle = ':', color = '0.4')

# ax.plot(sol_vfi.a, sol_vfi.c, linestyle = '-', color = '0.7')

# ax.set_xlabel(f"Assets, $a_t$")
# ax.set_ylabel(f"Consumption, $c^\star_t$")
# ax.set_title(f'Policy function')
# ax.set_xlim([-1,20])
# ax.set_ylim([-.2,3.5])

plt.show()
```


# The $z$-Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Theorems

The theorems of the $z$-transform relate basic operations applied to discrete signals to their equivalents in the spectral domain. They are of use for the computation of $z$-transforms of signals composed from modified [standard signals](../continuous_signals/standard_signals.ipynb) and for the computation of the response of systems to an input signal. The theorems allow further to predict the consequences of modifying a signal or system by certain operations.

### Convolution Theorem

The convolution theorem states that the $z$-transform of the linear convolution of two discrete signals $x[k]$ and $y[k]$ is equal to the scalar multiplication of their $z$-transforms $X(z)$ and $Y(z)$

\begin{equation}
\mathcal{Z} \{ x[k] * y[k] \} = X(z) \cdot Y(z)
\end{equation}

under the assumption that both transforms $X(z) = \mathcal{Z} \{ x[k] \}$ and $Y(z) = \mathcal{Z} \{ y[k] \}$ exist, respectively. The ROC of the linear convolution $x(t) * y(t)$ includes at least the intersection of the ROCs of $x[k]$ and $y[k]$

\begin{equation}
\text{ROC} \{ x[k] * y[k] \} \supseteq \text{ROC} \{ x[k] \} \cap \text{ROC} \{ y[k] \}
\end{equation}

The theorem can be proven by introducing the [definition of the linear convolution](../discrete_systems/linear_convolution.ipynb) into the [definition of the $z$-transform](definition.ipynb) and changing the order of summation

\begin{align}
\mathcal{Z} \{ x[k] * y[k] \} &= \sum_{k = -\infty}^{\infty} \left( \sum_{\kappa = -\infty}^{\infty} x[\kappa] \cdot y[k - \kappa] \right) z^{-k} \\
&= \sum_{\kappa = -\infty}^{\infty} \left( \sum_{k = -\infty}^{\infty} y[k - \kappa] \, z^{-k} \right) x[\kappa] \\
&= Y(z) \cdot \sum_{\kappa = -\infty}^{\infty} x[\kappa] \, z^{-\kappa} \\
&= Y(z) \cdot X(z)
\end{align}

The convolution theorem is very useful in the context of linear time-invariant (LTI) systems. The output signal $y[k]$ of an LTI system is given as the convolution of the input signal $x[k]$ with its impulse response $h[k]$. The signals and their relation can be represented alternatively in the spectral domain using the convolution theorem. This leads to the following two equivalent representations of an LTI system

Calculation of the system response by transforming the problem into the spectral domain can be beneficial since this replaces the evaluation of the linear convolution by a scalar multiplication. In many cases this procedure simplifies the calculation of the system response significantly. The convolution theorem can also be useful to derive a $z$-transform. The key is here to express the signal as convolution of two other signals for which the $z$-transforms are known. This is illustrated by the following example.

**Example**

The [linear convolution of two rectangular signals](../discrete_systems/linear_convolution.ipynb#Finite-Length-Signals) $\text{rect}_N[k] * \text{rect}_M[k]$ results in a finite-length signal of trapezoidal shape. The $z$-transform of this signal is derived by application of the convolution theorem together with the [transform of the rectangular signal](definition.ipynb#Transformation-of-the-Rectangular-Signal). This results in

\begin{equation}
\mathcal{Z} \{ \text{rect}_N[k] * \text{rect}_M[k] \}
= \sum_{k=0}^{N-1} z^{-k} \cdot \sum_{k=0}^{M-1} z^{-k}
= \begin{cases}
\frac{1 - z^{-N}}{1 - z^{-1}} \cdot \frac{1 - z^{-M}}{1 - z^{-1}} & \text{for } z \neq 1 \\
N \cdot M & \text{for } z = 1
\end{cases}
\end{equation}

The ROC of the rectangular signal is $z \in \mathbb{C} \setminus \{ 0 \}$. The ROC of the convolution of two rectangular signals is also $z \in \mathbb{C} \setminus \{ 0 \}$, since in general no poles and zeros cancel out. The poles and zeros of $\mathcal{Z} \{ \text{rect}_N[k] * \text{rect}_M[k] \}$ are computed and illustrated in the following.

```python
%matplotlib inline
import sympy as sym
from pole_zero_plot import pole_zero_plot
sym.init_printing()

N = 5
M = 7

z = sym.symbols('z', complex=True)
k = sym.symbols('k', integer=True)

X = sym.summation(z**(-k), (k,0,N-1)) * sym.summation(z**(-k), (k,0,M-1))

poles = sym.roots(sym.denom(X) * z, z)
zeros = sym.roots(sym.numer(X) * z**(N+M-2), z)

pole_zero_plot(poles, zeros)
```

**Exercise**

* Determine the locations of the zeros of the $z$-transform $\mathcal{Z} \{ \text{rect}_N[k] \}$ of a rectangular signal.
* How are the locations of the zeros from above example related to these?

### Shift Theorem

The [shift of a signal](../discrete_signals/operations.ipynb#Shift) $x(k - \kappa)$ for $\kappa \in \mathbb{Z}$ can be expressed by the convolution of the signal $x[k]$ with a shifted Dirac impulse

\begin{equation}
x[k - \kappa] = x[k] * \delta[k - \kappa]
\end{equation}

This follows from the sifting property of the Dirac impulse. Applying a two-sided $z$-transform to the left- and right-hand side and exploiting the convolution theorem yields

\begin{equation}
\mathcal{Z} \{ x[k - \kappa] \} = X(z) \cdot z^{- \kappa}
\end{equation}

where $X(z) = \mathcal{Z} \{ x[k] \}$ is assumed to exist. Note that $\mathcal{Z} \{ \delta[k - \kappa] \} = z^{- \kappa}$ can be derived from the definition of the two-sided $z$-transform together with the sifting property of the Dirac impulse. The $z$-transform of a shifted signal is given by multiplying the $z$-transform of the original signal with $z^{-\kappa}$. The ROC of $x[k-\kappa]$ is given as the ROC of $x[k]$, whereby the points $0$ and $\infty$ have to be regarded separately due to the term $z^{- \kappa}$ in its $z$-transform

\begin{equation}
\text{ROC} \{ x[k - \kappa] \} \supseteq \text{ROC} \{ x[k] \} \setminus \{0, \infty \}
\end{equation}

This result is known as shift theorem of the $z$-transform. For a causal signal $x[k]$ and $\kappa > 0$ the shift theorem of the one-sided $z$-transform is equal to the shift theorem of the two-sided transform.

**Example**

Let's consider a system which delays a signal by a given number of samples. Its output signal $y[k] = \mathcal{H} \{ x[k] \}$ reads

\begin{equation}
y[k] = x[k-\kappa]
\end{equation}

where $\kappa \in \mathbb{N}$. Applying the two-sided $z$-transform to the left- and right-hand side yields

\begin{equation}
Y(z) = z^{-\kappa} \cdot X(z)
\end{equation}

where $Y(z) = \mathcal{Z} \{ y[k] \}$ and $X(z) = \mathcal{Z} \{ y[k] \}$. The transfer function $H(z)$ of the system can be deduced from the spectral representation of the system output $Y(z) = H(z) \cdot X(z)$ as

\begin{equation}
H(z) = z^{- \kappa}
\end{equation}

Delaying a signal is a frequently applied operation in discrete signal processing. Block diagrams illustrating signal processing algorithms frequently contain blocks that constitute delays. These blocks are commonly annotated by the corresponding transfer function $z^{- \kappa}$.

**Exercise**

* Draw a block diagramm illustrating the following difference equation $y[k] = x[k] + 2 x[k-1] + x[k-3]$.

### Modulation Theorem

The complex modulation of a signal $x[k]$ is defined as $z_0^{k} \cdot x[k]$ with $z_0 \in \mathbb{C} \setminus \{ 0 \}$. The $z$-transform of a modulated signal is derived by introducing it into the definition of the two-sided $z$-transform

\begin{equation}
\mathcal{Z} \left\{ z_0^k \cdot x[k] \right\} =
\sum_{k = -\infty}^{\infty} x[k] z_0^k \, z^{-k} =
\sum_{k = -\infty}^{\infty} x[k] \, \left( \frac{z}{z_0} \right)^{-k}
= X \left( \frac{z}{z_0} \right)
\end{equation}

where $X(z) = \mathcal{Z} \{ x[k] \}$. Modulation of the signal $x[k]$ leads to a complex scaling of the $z$-plane by the factor $z_0$. The complex scaling of $z$ can be expressed as

\begin{equation}
\frac{z}{z_0} = \frac{|z|}{|z_0|} \; e^{j (\arg(z) - \arg(z_0))}
\end{equation}

where $\arg(z)$ and $\arg(z_0)$ denote the argument (phase) of $z$ and $z_0$, respectively. It follows from this representation that the complex scaling by $z_0$ can be interpreted as a scaling and rotation of the $z$-plane.

The ROC is also complex scaled

\begin{equation}
\text{ROC} \{ z_0^k \cdot x[k] \} = \{z: \frac{z}{z_0} \in \text{ROC} \{ x[k] \} \}
\end{equation}

This relation is known as modulation theorem.

**Example**

The $z$-transform of an harmonic exponential signal $e^{j \Omega_0 k}$ of finite duration $N$ is derived the following. The signal can be expressed as $e^{j \Omega_0 k} \cdot \text{rect}_N[k]$ which allows to apply the modulation theorem with $z_0 = e^{j \Omega_0}$. This yields

\begin{equation}
\mathcal{Z} \{ e^{j \Omega_0 k} \cdot \text{rect}_N[k] \} = \sum_{k=0}^{N-1} \left( \frac{z}{e^{j \Omega_0}} \right)^{-k} = e^{j \Omega_0 k} z^{-k}
\end{equation}

for $z \in \mathbb{C} \setminus \{ 0 \}$.

### Reversal

The two-sided $z$-transform of the reversed signal $x[-k]$ of $x[k]$ is given as

\begin{equation}
\mathcal{Z} \{ x[-k] \} = \sum_{k=-\infty}^{\infty} x[-k] \, z^{-k} = \sum_{k=-\infty}^{\infty} x[k] \, z^k =
\sum_{k=-\infty}^{\infty} x[k] \, \left( \frac{1}{z} \right)^{-k} = X \left( \frac{1}{z} \right)
\end{equation}

where $X(z) = \mathcal{Z} \{ x[k] \}$ denotes the $z$-transform of $x[k]$. The revesal of a discrete signal leads to an inversion of the complex frequency $z$ in its $z$-transform. The ROC is given as

\begin{equation}
\text{ROC} \{ x[-k] \} = \{z: \frac{1}{z} \in \text{ROC} \{ x[k] \} \}
\end{equation}

**Example**

The $z$-transform of the anticausal complex exponential signal $x[k] = - z_0^k \cdot \epsilon[k - 1]$ is derived by applying a series of transformations to the complex exponential signal $z_0[k] \cdot \epsilon[k]$. Reversal of the complex exponential signal yields

\begin{equation}
\mathcal{Z} \{ z_o^{-k} \cdot \epsilon[-k] \} = \frac{z^{-1}}{z^{-1} - z_0}
\end{equation}

Now the shift theorem is applied to the shifted signal

\begin{equation}
\mathcal{Z} \{ z_o^{-k-1} \cdot \epsilon[-k-1] \} = z \cdot \frac{z^{-1}}{z^{-1} - z_0}
\end{equation}

The factor $z_0^{-1}$ is a constant with respect to the $z$-transform. This yields

\begin{equation}
\mathcal{Z} \{ z_o^{-k} \cdot \epsilon[-k-1] \} = z_0 z \cdot \frac{z^{-1}}{z^{-1} - z_0}
\end{equation}

By noting that $z_o^{-k} = \left( \frac{1}{z_o} \right)^k$ we get

\begin{equation}
\mathcal{Z} \{ \left( \frac{1}{z_o} \right)^{k} \cdot \epsilon[-k-1] \} = z_0^{-1} z \cdot \frac{z^{-1}}{z^{-1} - z_0^{-1}} = \frac{z}{z_0 - z}
\end{equation}

From this, the $z$-transform of the anticausal complex exponential signal follows as

\begin{equation}
\mathcal{Z} \{ - z_0^{k} \cdot \epsilon[-k-1] \} =  \frac{z}{z - z_0}
\end{equation}

By considering the influence of above steps on the ROC, it is derived as

\begin{equation}
|z| < |z_0|
\end{equation}

Note that the $z$-transform of the anticausal exponential signal is equal to the transform of the causal exponential signal. However, both differ with respect to their ROCs. Above results can be confirmed by numerical evaluation of the definition of the $z$-transform

```python
z0 = sym.symbols('z0', complex=True)
sym.summation(z0**k * z**(-k), (k, -sym.oo, -1))
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Regresión con Redes Neuronales

Empleando diferentes *funciones de pérdida* y *funciones de activación* las **redes neuronales** pueden resolver
efectivamente problemas de **regresión.**
En esta libreta se estudia el ejemplo de [California Housing](http://www.spatial-statistics.com/pace_manuscripts/spletters_ms_dir/statistics_prob_lets/html/ms_sp_lets1.html)
donde el propósito es predecir el valor medio de una casa según 8 atributos.

## Descripción general del conjunto de datos

El conjunto de datos `California Housing` está hecho de 9 variables numéricas, donde 8 son las *características* y 1 es la variable objetivo.
Este conjunto de datos fue creado en 1990 basándose en el censo poblacional realizado por el gobierno de EUA. La estructura del conjunto de datos
es simple: cada línea en el archivo de datos cuenta por un **bloque** poblacional que consta de entre 600 y 3000 personas. Por cada *bloque*
se tienen 8 características de cada casa y su costo medio.

Empleando *redes neuronales* se pretende predecir el costo de las casas por bloque.

## Atributos del conjunto de datos

Este conjunto de datos cuenta con 8 *atributos*, descritos a continuación, con la etiqueta como viene en el conjunto de datos de `scikit-learn`:

- **MedInc**, *Ingresos promedio por bloque*
- **HouseAge**, *Antigüedad promedio por casa en el bloque*
- **AveRooms**, *Número promedio de cuartos por casa en el bloque*
- **AveBedrms**, *Número promedio de recámaras por casa en el bloque*
- **Population**, *Población total del bloque*
- **AveOccup**, *Ocupancia promedio por casa en el bloque*
- **Latitude**, *Latitud del bloque*
- **Longitude**, *Longitud del bloque*

Y la *variable respuesta* es:

- **MedValue**, *Costo promedio por casa en el distrito*

```python
import tensorflow as tf
from sklearn import datasets, metrics, model_selection, preprocessing
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
```

```python
# Importar el conjunto de datos California Housing
cali_data = datasets.fetch_california_housing()
```

## Visualización de datos

```python
# Realizar una visualización general de la relación entre atributos del conjunto de datos
sns.pairplot(pd.DataFrame(cali_data.data, columns=cali_data.feature_names))
plt.show()
```

Con estas figuras se pueden observar algunas características interesantes:

- *Primero*, todas las variables importan en el modelo. Esto significa que el modelo de regresión viene pesado por todas las
  características y se requiere que el modelo sea *robusto* ante esta situación.
- *Segundo*, hay algunas características que tienen relación *lineal* entre ellas, como lo es **AveRooms** y **AveBedrms**.
  Esto puede ayudar a discriminar ciertas características que no tienen mucho peso sobre el modelo y solamente utilizar
  aquellas que influyen mucho más. A esta parte del *procesamiento de datos* se le conoce como **selección de características**
  y es una rama específica de la *inteligencia computacional.*
- *Tercero*, la línea diagonal muestra la relación *distribución* de cada una de las características. Esto es algo importante
  de estudiar dado que algunas características muestran *distribuciones* conocidas y este hecho se puede utilizar para
  emplear técnicas estadísticas más avanzadas en el **análisis de regresión.**

Sin embargo, en toda esta libreta se dejarán las 8 características para que sean pesadas en el modelo final.

```python
# Separar todos los datos y estandarizarlos
X = cali_data.data
y = cali_data.target
# Crear el transformador para estandarización
std = preprocessing.StandardScaler()
X = std.fit_transform(X)
X = np.array(X).astype(np.float32)
y = std.fit_transform(y.reshape(-1, 1))
y = np.array(y).astype(np.float32)
```

Dado que los datos vienen en diferentes unidades y escalas, siempre se debe estandarizar los datos de alguna forma. En particular
en esta libreta se emplea la normalización de los datos, haciendo que tengan *media* $\mu = 0$ y *desviación estándar* $\sigma = 1$.

```python
# Separar en conjunto de entrenamiento y prueba
x_train, x_test, y_train, y_test = model_selection.train_test_split(
    X, y, test_size=0.2, random_state=49
)
```

```python
# Definir parámetros generales de la Red Neuronal
pasos_entrenamiento = 1000
tam_lote = 30
ratio_aprendizaje = 0.01
```

## Estructura o *topología* de la red neuronal

Para esta regresión se pretende utilizar una *red neuronal* de **dos capas ocultas**, con *funciones de activación* **ReLU**,
la **primera** capa oculta cuenta con 25 neuronas mientras que la **segunda** cuenta con 50.
La **capa de salida** *no* tiene función de activación, por lo que el modelo lineal queda de la siguiente forma

$$ \hat{y}(x) = \sum_{i=1}^{8} \alpha_i \cdot x_i + \beta_i$$

donde $\alpha_i$ son los *pesos* de la *capa de salida*, mientras que $\beta_i$ son los *sesgos*.

```python
# Parámetros para la estructura general de la red
# Número de neuronas por capa
n_capa_oculta_1 = 25
n_capa_oculta_2 = 50
n_entrada = X.shape[1]
n_salida = 1
```

```python
# Definir las entradas de la red neuronal
x_entrada = tf.placeholder(tf.float32, shape=[None, n_entrada])
y_entrada = tf.placeholder(tf.float32, shape=[None, n_salida])
```

```python
# Diccionario de pesos
pesos = {
    "o1": tf.Variable(tf.random_normal([n_entrada, n_capa_oculta_1])),
    "o2": tf.Variable(tf.random_normal([n_capa_oculta_1, n_capa_oculta_2])),
    "salida": tf.Variable(tf.random_normal([n_capa_oculta_2, n_salida])),
}
# Diccionario de sesgos
sesgos = {
    "b1": tf.Variable(tf.random_normal([n_capa_oculta_1])),
    "b2": tf.Variable(tf.random_normal([n_capa_oculta_2])),
    "salida": tf.Variable(tf.random_normal([n_salida])),
}
```

    WARNING:tensorflow:From /home/edwin/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.

```python
def propagacion_adelante(x):
    # Capa oculta 1
    # Esto es la mismo que Ax + b, un modelo lineal
    capa_1 = tf.add(tf.matmul(x, pesos["o1"]), sesgos["b1"])
    # ReLU como función de activación
    capa_1 = tf.nn.relu(capa_1)
    # Capa oculta 1
    # Esto es la mismo que Ax + b, un modelo lineal
    capa_2 = tf.add(tf.matmul(capa_1, pesos["o2"]), sesgos["b2"])
    # ReLU como función de activación
    capa_2 = tf.nn.relu(capa_2)

    # Capa de salida
    # Nuevamente, un modelo lineal
    capa_salida = tf.add(tf.matmul(capa_2, pesos["salida"]), sesgos["salida"])

    return capa_salida
```

```python
# Implementar el modelo y sus capas
y_prediccion = propagacion_adelante(x_entrada)
```

## Función de pérdida

Para la función de pérdida se emplea la [función de Huber](https://en.wikipedia.org/wiki/Huber_loss) definida como

\begin{equation}
    L_{\delta} \left( y, f(x) \right) =
    \begin{cases}
        \frac{1}{2} \left( y - f(x) \right)^2  & \text{para} \vert y - f(x) \vert \leq \delta, \\
        \delta \vert y - f(x) \vert - \frac{1}{2} \delta^2   & \text{en cualquier otro caso.}
    \end{cases}
\end{equation}

Esta función es [robusta](https://en.wikipedia.org/wiki/Robust_regression) lo cual está hecha para erradicar el peso de posibles
valores atípicos y puede encontrar la verdadera relación entre las características sin tener que recurrir a metodologías paramétricas
y no paramétricas.

## Nota

Es importante mencionar que el valor de $\delta$ en la función de Huber es un **hiperparámetro** que debe de ser ajustado mediante *validación cruzada*
pero no es realiza en esta libreta por limitaciones de equipo y rendimiento en la ejecución de esta libreta.

```python
# Definir la función de costo
f_costo = tf.reduce_mean(tf.losses.huber_loss(y_entrada, y_prediccion, delta=2.0))
# f_costo = tf.reduce_mean(tf.square(y_entrada - y_prediccion))
optimizador = tf.train.AdamOptimizer(learning_rate=ratio_aprendizaje).minimize(f_costo)
```

    WARNING:tensorflow:From /home/edwin/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/losses/losses_impl.py:448: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.cast instead.

```python
# Primero, inicializar las variables
init = tf.global_variables_initializer()
```

```python
# Función para evaluar la precisión de clasificación
def precision(prediccion, real):
    return tf.sqrt(tf.losses.mean_squared_error(real, prediccion))
```

## Precisión del modelo

Para evaluar la precisión del modelo se emplea la función [RMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation) (Root Mean Squared Error)
definida por la siguiente función:

$$ RMSE = \sqrt{\frac{\sum_{i=1}^{N} \left( \hat{y}_i - y_i \right)^2}{N}} $$

Para crear un mejor estimado, se empleará validación cruzada de 5 pliegues.

```python
# Crear el plegador para el conjunto de datos
kf = model_selection.KFold(n_splits=5)
kf_val_score_train = []
kf_val_score_test = []
```

```python
# Crear un grafo de computación
with tf.Session() as sess:
    # Inicializar las variables
    sess.run(init)

    for tr_idx, ts_idx in kf.split(x_train):
        # Comenzar los pasos de entrenamiento
        # solamente con el conjunto de datos de entrenamiento
        for p in range(pasos_entrenamiento):
            # Minimizar la función de costo
            minimizacion = sess.run(
                optimizador,
                feed_dict={x_entrada: x_train[tr_idx], y_entrada: y_train[tr_idx]},
            )
            # Cada tamaño de lote, calcular la precisión del modelo
            if p % tam_lote == 0:
                prec_entrenamiento = sess.run(
                    precision(y_prediccion, y_entrada),
                    feed_dict={x_entrada: x_train[tr_idx], y_entrada: y_train[tr_idx]},
                )
                kf_val_score_train.append(prec_entrenamiento)
                prec_prueba = sess.run(
                    precision(y_prediccion, y_entrada),
                    feed_dict={x_entrada: x_train[ts_idx], y_entrada: y_train[ts_idx]},
                )
                kf_val_score_test.append(prec_prueba)
    # Prediccion final, una vez entrenado el modelo
    pred_final = sess.run(
        precision(y_prediccion, y_entrada),
        feed_dict={x_entrada: x_test, y_entrada: y_test},
    )
    pred_report = sess.run(y_prediccion, feed_dict={x_entrada: x_test})
    print("Precisión final: {0}".format(pred_final))
```

    Precisión final: 0.5226657390594482

```python
print("Precisión RMSE para entrenamiento: {0}".format(np.mean(kf_val_score_train)))
print("Precisión RMSE para entrenamiento: {0}".format(np.mean(kf_val_score_test)))
```

    Precisión RMSE para entrenamiento: 0.9006941914558411
    Precisión RMSE para entrenamiento: 0.9006157517433167

Aquí se muestra el valor de *RMSE* final para cada parte, entrenamiento y prueba. Se puede observar que hay muy poco sobreajuste,
y si se quisiera corregir se puede realizar aumentando el número de neuronas, de capas, cambiando las funciones de activación,
entre muchas otras cosas.


# Equilibre énergétique de l'échangeur

On note :
- $m_{dro}$ le débit massique du fluide dans le dromotherme en $\frac{kg}{s}$

- $Cp_{dro}$ la capacité thermique du fluide circulant dans le dromotherme en  $\frac{J}{kg K}$

- $m_{sto}$ le débit massique du fluide dans le système géothermique du stockage en $\frac{kg}{s}$

- $Cp_{sto}$ la capacité thermique du fluide circulant dans le système géothermique du stockage en $\frac{J}{kg K}$

On pose $coeff = \frac{m_{dro} * Cp_{dro}}{m_{sto} * Cp_{sto}}$

## Cas 1 : $coeff <= 1$

On a

\begin{equation}
T_{inj\_dro} = T_{sor\_dro} - eff * (T_{sor\_dro} - T_{sor\_sto})
\end{equation}

Et :

\begin{equation}
T_{inj\_sto} = T_{sor\_sto} + coeff * (T_{sor\_dro} - T_{inj\_dro})
\end{equation}

On peut donc écrire :

\begin{equation}
T_{inj\_sto} = T_{sor\_sto} + coeff * (T_{sor\_dro} - T_{sor\_dro} + eff * (T_{sor\_dro} - T_{sor\_sto}))
\end{equation}

Ce qui laisse :

\begin{equation}
T_{inj\_sto} = T_{sor\_sto} + coeff * eff * (T_{sor\_dro} - T_{sor\_sto})
\end{equation}

# Echanges énérgétiques au niveau du système géothermique équipant le système de stockage

Si k est le coefficient du système énergétique exprimé en W/K, et en notant y la température dans le massif de stockage, on a :

\begin{equation}
m_{sto} * Cp_{sto} * (T_{inj\_sto} - T_{sor\_sto}) = k * (\frac{T_{inj\_sto} + T_{sor\_sto}}{2} - y )
\end{equation}

Soit :

\begin{equation}
T_{inj\_sto}  (m_{sto} * Cp_{sto} - \frac{k}{2})  =  (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto} - k y
\end{equation}

Ou encore :

\begin{equation}
T_{inj\_sto}  =  \frac{ (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto} - k y }{m_{sto} * Cp_{sto} - \frac{k}{2}}
\end{equation}

# Bilan

On obtient 2 expressions de $T_{inj\_sto}$, permettant ainsi de formuler $T_{sor\_sto}$ fonction de y et de $T_{sor\_dro}$ seulement :

\begin{equation}
T_{sor\_sto} + coeff * eff * (T_{sor\_dro} - T_{sor\_sto}) = \frac{(m_{sto} * Cp_{sto} + \frac{k}{2} )  T_{sor\_sto} - k y} {m_{sto} * Cp_{sto} - \frac{k}{2}}
\end{equation}

En posant $ B = coeff * eff  * (m_{sto} * Cp_{sto} - \frac{k}{2})$

On obtient :

\begin{equation}
(m_{sto} * Cp_{sto} - \frac{k}{2})*T_{sor\_sto} + B * (T_{sor\_dro} - T_{sor\_sto}) =  (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto} - k  y
\end{equation}

Soit :

\begin{equation}
(m_{sto} * Cp_{sto} - \frac{k}{2} - B - m_{sto} * Cp_{sto} - \frac{k}{2}) *  T_{sor\_sto} = - B * T_{sor\_dro} - k y
\end{equation}

Ou encore :

\begin{equation}
( k + B ) *  T_{sor\_sto} = ky + B * T_{sor\_dro}
\end{equation}

Il reste :

\begin{equation}
T_{sor\_sto} = \frac{ky + B * T_{sor\_dro}}{k+B}
\end{equation}

# Couplage avec la PAC

On note :

- $P_{geo}$ la puissance géothermique extraite en  W
- $besoin$ le besoin en chauffage du bâtiment en W
- $P_{elec}$ la puissance électrique de la PAC en  W
- COP  le coefficient de performance de la PAC

On a : $besoin = P_{elec} + P_{geo}$

Et
\begin{equation}
COP=\frac{besoin}{P_{elec}}
\end{equation}

De ces deux expressions, on obtient la relation entre la puissance géothermique et le besoin du bâtiment:

\begin{equation}
P_{geo}=\frac{COP-1}{COP} * besoin
\end{equation}

La puissance géothermique s'exprime aussi : $P_{geo} = m_{pac} Cp_{pac} ( T_{inj\_pac}-T_{sor\_pac} )$

On en déduit la température de sortie du fluide dans la PAC:

\begin{equation}
T_{sor\_pac} = T_{inj\_pac} - \frac{P_{geo}}{m_{pac} * Cp_{pac}}
\end{equation}

En écrivant l'équilibre énergétique entre la PAC et le stockage, comme on a écrit celui entre le dromotherm et stockage, on a :

\begin{equation}
T_{sor\_pac}  =  \frac{ (m_{pac} * Cp_{pac} + \frac{k}{2})  T_{inj\_pac} - k y }{m_{pac} * Cp_{pac} - \frac{k}{2}}
\end{equation}

Les deux expressions de $T_{sor\_pac}$ nous permettent d'obtenir $T_{inj\_pac}$ :

\begin{equation}
T_{inj\_pac}  = y-\frac{C Pgeo}{k}
\end{equation}

avec

\begin{equation}
C  = 1-\frac{k}{2 m_{pac} Cp_{pac}}
\end{equation}

## Bâtiment

Le bâtiment est assimilé à un modèle R3C.
On note :

- Tconsigne : température de consigne en °C

- Tint : température intérieure en °C

- Text : température extérieure en °C

- Rm : Résistance thermique des murs (K/W)

- Ri : Résistance superficielle intérieure (K/W)

- Rf : résistance de fuite (infiltrations+vitre+renouvellement d'air) K/W

Par analogie électrique, on assimile les températures à des tensions et les puissances à des intensités
en première approximation, on a donc :

\begin{equation}
    \frac {T_{int}-T_{ext}}{R_m+R_i}+ \frac {T_{int} -T_{ext}}{R_f} + C \frac{dT_{int}}{dt}=Qchauffage=besoin
\end{equation}

Soit :

\begin{equation}
    C \frac{dT_{int}}{dt}=Qchauffage- \frac {T_{int}-T_{ext}}{R_m+R_i}-\frac {T_{int} -T_{ext}}{R_f}
\end{equation}

Pour maintenir Tint constante et égale à Tconsigne, on doit donc développer :

\begin{equation}
   Qchauffage=(Tconsigne-Text)*(\frac {1}{Rm+Ri}+ \frac {1}{Rf} )
\end{equation}

# RECAPITULATIF DE TOUTES LES EQUATIONS DU SYSTEME

## Dromotherm

\begin{equation}
C_s h_s\frac{\partial T_s}{\partial t}=(1-\alpha)R_g(t)+ R_{atm}(t) -\epsilon\sigma T_s^{4}-H_v(T_s-T_{ext}(t))-r_{s/d}(T_s-T_f)
\end{equation}

\begin{equation}
((1-\phi) C_d h_d + \phi C_f h_d ) \frac {\partial T_f}{\partial t} + K_p C_f h_d \frac {\partial T_f}{\partial x}=r_{s/d}(T_s-T_f) - r_{d/b1}(T_f-T_{b1})
\end{equation}

\begin{equation}
C_b h_{b1}\frac{\partial T_{b1}}{\partial t}=r_{d/b1}(T_f-T_{b1})- r_{b1/b2}(T_{b1}-T_{b2})
\end{equation}

\begin{equation}
C_b h_{b2}\frac{\partial T_{b2}}{\partial t}= r_{b1/b2}(T_{b1}-T_{b2})
\end{equation}

## Echangeur thermique

\begin{equation}
T_{inj\_dro} = T_{sor\_dro} - eff * (T_{sor\_dro} - T_{sor\_sto})
\end{equation}

## Stockage

\begin{equation}
T_{inj\_sto}  =  \frac{ (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto} - k T_{stock} }{m_{sto} * Cp_{sto} - \frac{k}{2}}
\end{equation}

\begin{equation}
T_{sor\_sto} = \frac{k T_{stock} + B * T_{sor\_dro}}{k+B}
\end{equation}

## PAC

\begin{equation}
T_{sor\_pac} = T_{inj\_pac} - \frac{P_{geo}}{m_{pac} * Cp_{pac}}
\end{equation}

\begin{equation}
T_{inj\_pac}  = T_{stock}-\frac{C Pgeo}{k}
\end{equation}

## Température stockage
###  1er cas :Dromotherm seul en marche

\begin{equation}
    m_{sable} Cp_{sable} \frac {dT_{stock}}{dt}=m_{sto} Cp_{sto} ( T_{inj\_sto} - T_{sor\_sto} )
\end{equation}

### 2ème cas :PAC seule  en marche

\begin{equation}
    m_{sable} Cp_{sable} \frac {dT_{stock}}{dt}= m_{pac} Cp_{pac} (T_{sor\_pac}- T_{inj\_pac})
\end{equation}

### 3me cas : Dromotherm et PAC en marche

\begin{equation}
    m_{sable} Cp_{sable} \frac {dT_{stock}}{dt}=m_{sto} Cp_{sto} ( T_{inj\_sto} - T_{sor\_sto} ) + m_{pac} Cp_{pac} (T_{sor\_pac}- T_{inj\_pac})
\end{equation}

## Bâtiment
\begin{equation}
   besoin_{chauffage}=(Tconsigne-Text)*(\frac {1}{Rm+Ri}+ \frac {1}{Rf} )
\end{equation}

\begin{equation}
   besoin_{ECS}= Volume_{ballon}*N_{pers}*(T_{ballon}-T{eau})*Cp_{eau}/(24*3600)
\end{equation}

# SCHEMA NUMERIQUE

Soit $t_i $, l'instant courant.

## Dromotherm
Le schema numérique se résume à :

\begin{equation}
   T_{sor\_dro}(t_i)=D(t_i,T(t_{i-1}),T_{inj\_dro}(t_{i-1})
\end{equation}

avec :

\begin{equation}
   T(t_i)=[Ts(t_i),Tf(t_i),Tb1(t_i),Tb2(t_i)]
\end{equation}

## Echangeur thermique

\begin{equation}
T_{inj\_dro}(t_i) = T_{sor\_dro}(t_i) - eff * (T_{sor\_dro}(t_i) - T_{sor\_sto}(t_i))
\end{equation}

## Stockage

\begin{equation}
T_{inj\_sto}(t_i)  =  \frac{ (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto}(t_i) - k T_{stock}(t_i) }{m_{sto} * Cp_{sto} - \frac{k}{2}}
\end{equation}

\begin{equation}
T_{sor\_sto}(t_i) = \frac{k T_{stock}(t_i) + B * T_{sor\_dro}(t_i)}{k+B}
\end{equation}

## PAC

\begin{equation}
T_{inj\_pac}(t_i)  = T_{stock}(t_i)-\frac{C Pgeo(t_i)}{k}
\end{equation}

\begin{equation}
T_{sor\_pac}(t_i) = T_{inj\_pac}(t_i) - \frac{P_{geo}(t_i)}{m_{pac} * Cp_{pac}}
\end{equation}

## Température stockage

Les équations régissant la température du stockage dans chaque est une équation différentielle d'ordre 1 sous la forme:

\begin{equation}
     \frac {dT_{stock}}{dt}= F (t,T_{stock})
\end{equation}

Suivant la méthode Euler explicite on a :
\begin{equation}
 T_{stock}(t_{i})=T_{stock}(t_{i})+ dt *F (t_{i-1},T_{stock}(t_{i-1})
\end{equation}
## Bâtiment

\begin{equation}
   besoin_{chauffage}(t_i)=(Tconsigne-Text(t_i))*(\frac {1}{Rm+Ri}+ \frac {1}{Rf} )
\end{equation}

\begin{equation}
   besoin_{ECS}(t_i)= Volume_{ballon}*N_{pers}*(T_{ballon}-T{eau}(t_i))*Cp_{eau}/(24*3600)
\end{equation}

# ALGORITHME DE RESOLUTION

## Phase d'initialisation (t=0)
 A l'instant initial, tout le système est supposé être à l'arrêt et toutes les températures égales à une valeur $T_{ini}$

 ## Phase itérative ( t>0)

 La procédure itérative de calcul des inconnues se présente comme suit:

 ### I- Choix du  fonctionnement du démonstrateur

 Le démonstrateur peut fonctionner de 8  manières. L'utilisateur choisit:

En été:

 1. Dromotherm ON et PAC ON

 2. Dromotherm ON et PAC OFF

 3. Dromotherm OFF et PAC ON

 4. Dromotherm OFF et PAC OFF

En hivers:

1. Dromotherm ON et PAC ON

2. Dromotherm ON et PAC OFF

3. Dromotherm OFF et PAC ON

4. Dromotherm OFF et PAC OFF

NB: PAC ON = chauffage et/ ou ECS

### II- Calculer  La variation temporelle de la Température du stockage suivant chaque cas

1. Dromotherm ON et PAC ON

   $der=\frac{m_{sto} Cp{sto} (T_{inj\_sto}(t_{i})-T_{sor\_sto}(t_{i}))-P_{geo}(t_i)}{m_{sable} Cp{sable}}  $

2. Dromotherm ON et PAC OFF

   $der=\frac{m_{sto} Cp{sto} (T_{inj\_sto}(t_{i})-T_{sor\_sto}(t_{i}))}{m_{sable} Cp{sable}}  $

3. Dromotherm OFF et PAC ON

    $der=\frac{-P_{geo}(t_i)}{m_{sable} Cp{sable}}  $

4. Dromotherm OFF et PAC OFF

    $der=0$

 ### III- Calculer la Température du stockage par la méthode de Euler explicite

 $T_{stock}(t_{i})=T_{stock}(t_{i})+ dt *der$

 ### IV- Calculer les températures des fluides circulant dans le domotherm, l'échangeur et la PAC

 1. Si Dromotherm ON

   Calculer $T_{sor\_dro}(t_i)$ en fonction de $T_{inj\_dro}(t_{i-1})$ et du débit unitaire qdro_u

   Calcluer $T_{sor\_sto}(t_i)$ par $T_{sor\_sto}(t_i) = \frac{k T_{stock}(t_i) + B * T_{sor\_dro}(t_i)}{k+B}$

   Calculer $T_{inj\_sto}(t_i)$ par $ T_{inj\_sto}(t_i) =  \frac{ (m_{sto} * Cp_{sto} + \frac{k}{2})  T_{sor\_sto}(t_i) - k        T_{stock}(t_i) }{m_{sto} * Cp_{sto} - \frac{k}{2}}$

   Calculer $T_{inj\_dro}(t_i)$ par $T_{inj\_dro}(t_i) = T_{sor\_dro}(t_i) - eff * (T_{sor\_dro}(t_i) - T_{sor\_sto}(t_i))$

 2. Si Dromotherm OFF

  Calculer $T_{sor\_dro}(t_i)$ en fonction de $T_{inj\_dro}(t_{i-1})$ et du débit nul

  Calculer $T_{inj\_dro}(t_i)$ par $T_{inj\_dro}(t_i) = T_{sor\_dro}(t_i-1)$

  Calcluer $T_{sor\_sto}(t_i)$ par $T_{sor\_sto}(t_i) = T_{sor\_sto}(t_i-1)$

  Calculer $T_{inj\_sto}(t_i)$ par $ T_{inj\_sto}(t_i) = T_{inj\_sto}(t_i-1) $

 3. Si la PAC ON

 Calculer $T_{inj\_pac}(t_i)$ par  $T_{inj\_pac}(t_i) = T_{stock}(t_i)-\frac{C Pgeo(t_i)}{k}$

 Calculer $T_{sor\_pac}(t_i)$ par   $T_{sor\_pac}(t_i)= T_{inj\_pac}(t_i) - \frac{P_{geo}(t_i)}{m_{pac} * Cp_{pac}}$

 4. Si la PAC OFF

 Calculer $T_{inj\_pac}(t_i)$ par  $T_{inj\_pac}(t_i) = T_{inj\_pac}(t_i-1)$

 Calculer $T_{sor\_pac}(t_i)$ par   $T_{sor\_pac}(t_i)= T_{sor\_pac}(t_i-1)$

 La procédure recommence à l'étape I

```python

```