# Project 4: Reinforcement Learning
### Train a Smartcab How to Drive
Uirá Caiado. Jul 7, 2016

#### Abstract

*A smartcab is a self-driving car from the not-so-distant future that ferries people from one arbitrary location to another. In this project, I will design the AI driving agent for the smartcab using reinforcement learning. This [area of machine learning](https://en.wikipedia.org/wiki/Reinforcement_learning) is inspired by behaviorist psychology and consists in training the agent by reward and punishment without needing to specify how the task is to be achieved. The agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.*

## 1. Introduction

In this section, I will present a brief introduction to reinforcement learning and to the problem addressed by this project.

### 1.1. Reinforcement Learning

As explained by \cite{Mohri_2012}, reinforcement learning is the study of planning and learning in a scenario where a learner (or agent) actively interacts with the environment to achieve a particular goal. The achievement of the agent's goal is typically measured by the reward he receives from the environment and which he seeks to maximize.

\cite{Kaelbling_1996} state that the most significant difference between reinforcement learning and supervised learning is that there is no presentation of input/output pairs. Instead, they explained that after choosing an action, the agent is told the immediate reward and the following state, but is not told which action would have been in its best long-term interests. It is necessary for the agent to gather useful experience about the possible system states, actions, transitions and rewards actively to act optimally.

Defining a general formulation of the problem based on a Markov Decision Process (MDP), as proposed by \cite{Mitchell}, the agent can perceive a set $S$ os distinct states of its environment and has a set $A$ of actions that it can perform. So, at each discrete time step $t$, the agent senses the current state $s_t$ and choose to take an action $a_t$. The environment responds by providing the agent a reward $r_t=r(s_t, a_t)$ and by producing the succeeding state $s_{t+1}=\delta(s_t, a_t)$. The functions $r$ and $\delta$ only depend on the current state and action (it is [memoryless](https://en.wikipedia.org/wiki/Markov_process)), are part of the environment and are not necessarily known to the agent.

The task of the agent is to learn a policy $\pi$ that maps each state to an action ($\pi: S \rightarrow A$), selecting its next action $a_t$ based solely on the current observed state $s_t$, that is $\pi(s_t)=a_t$. The optimal policy, or control strategy, is the one that produces the greatest possible cumulative reward over time. So, stating that:

$$V^{\pi}(s_t)= r_t + \gamma r_{t+1} + \gamma^2 r_{t+1} + ... = \sum_{i=0}^{\infty} \gamma^{i} r_{t+i}$$

Where $V^{\pi}(s_t)$ is also called the discounted cumulative reward and it represents the cumulative value achieved by following an policy $\pi$ from an initial state $s_t$ and $\gamma \in [0, 1]$ is a constant that determines the relative value of delayed versus immediate rewards. If we set $\gamma=0$, only immediate rewards is considered. As $\gamma \rightarrow 1$, future rewards are given greater emphasis relative to immediate reward. The optimal policy $\pi^{*}$ that will maximizes $V^{\pi}(s_t)$ for all states $s$ can be written as:

$$\pi^{*} = \underset{\pi}{\arg \max} \, V^{\pi} (s)\,\,\,\,\,, \,\, \forall s$$

As learning $\pi^{*}: S \rightarrow A$ directly is difficult because the available training data does not provide training examples of the form $(s, a)$, in the section 4 I will implement the Q-learning algorithm for estimating the optimal policy.

### 1.2. What Will Be Done

The goal of this project is to design the AI driving agent for the smartcab that operates in an idealized grid-like city.

The smartcab is able to sense whether the traffic light is green for its direction of movement and whether there is a car at the intersection on each of the incoming roadways (and which direction they are trying to go). In addition to this, each trip has an associated timer that counts down every time step. If the timer is at $0$ and the destination has not been reached, the trip is over, and a new one may start.

It should receive the inputs mentioned above at each time step t, and generate an output move, that consists on to stay put at the current intersection, move one block forward, one block left, or one block right (no backward movement).

The smartcab also should receive a reward for each successfully completed trip. A trip is considered “successfully completed” if the passenger is dropped off at the desired destination within a pre-specified time bound. It also gets a smaller reward for each correct move executed at an intersection. It gets a minor penalty for a wrong move and a larger penalty for violating traffic rules and/or causing an accident.

Based on the rewards and penalties it gets, the agent should learn an optimal policy for driving on city roads, obeying traffic rules correctly, and trying to reach the destination within a goal time.

## 2. Implement a Basic Driving Agent

In this section, I will implement a basic driving agent that accepts specified inputs ans produces a valid output.

```
Udacity:

Implement the basic driving agent, which processes the following inputs at each time step:

Next waypoint location, relative to its current location and heading, Intersection state (traffic light and presence of cars), and, Current deadline value (time steps remaining), And produces some random move/action (None, 'forward', 'left', 'right'). Don’t try to implement the correct strategy! That’s exactly what your agent is supposed to learn.

Run this agent within the simulation environment with enforce_deadline set to False (see run function in agent.py), and observe how it performs. In this mode, the agent is given unlimited time to reach the destination. The current state, action taken by your agent and reward/penalty earned are shown in the simulator.

In your report, mention what you see in the agent’s behavior. Does it eventually make it to the target location?

It is asked to:

- Agent accepts inputs: Student is able to implement the desired interface to the agent that accepts specified inputs.
- Produces a valid output: The driving agent produces a valid output (one of None, ‘forward’, ‘left’, ‘right’) in response to the inputs.
- Runs in simulator: The driving agent runs in the simulator without errors. Rewards and penalties do not matter - it’s okay for the agent to make mistakes.
```

### 2.1. My Grid-Like World

Let's start by testing the output produced by my current *world*, that will be used to analyze the results of this project. This *world* is a grid-like city, with roads going North-South and East-West. Other vehicles may be present on the streets, but no pedestrians. There is a traffic light at each intersection that can be in one of two states: North-South open or East-West open. US right-of-way rules apply: On a green light, you can turn left only if there is no oncoming traffic at the intersection coming straight. On a red light, you can turn right if there is no oncoming traffic turning left or traffic from the left going straight.

We are told to assume that a higher-level planner assigns a route to the smartcab, splitting it into waypoints at each intersection. The time in this world is quantized and at any instant, the smartcab is at some intersection. Therefore, the next waypoint is always either one block straight ahead, one block left, one block right, one block back or exactly there (reached the destination). Below is a sample of the log file generated by the simulator.

```python
import random
from smartcab.environment import Agent, Environment
from smartcab.planner import RoutePlanner
from smartcab.simulator import Simulator
import smartcab.agent as agent; reload(agent);
```

```python
e = Environment()  # create environment (also adds some dummy traffic)
a = e.create_agent(agent.LearningAgent)  # create agent
e.set_primary_agent(a, enforce_deadline=True)  # specify agent to track
sim = Simulator(e, update_delay=0.5, display=False)
```

```python
# this test was performed before I implement the codes
sim.run(n_trials=1)
```

    Simulator.run(): Trial 0
    2016-07-16 16:29:32,947;Environment.reset(): Trial set up with start = (5, 2), destination = (2, 4), deadline = 25
    2016-07-16 16:29:32,960;RoutePlanner.route_to(): destination = (2, 4)
    2016-07-16 16:29:33,466;LearningAgent.update(): deadline = 25, inputs = {'light': 'red', 'oncoming': None, 'right': None, 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:33,966;LearningAgent.update(): deadline = 24, inputs = {'light': 'red', 'oncoming': None, 'right': None, 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:34,466;LearningAgent.update(): deadline = 23, inputs = {'light': 'red', 'oncoming': None, 'right': None, 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:34,966;LearningAgent.update(): deadline = 22, inputs = {'light': 'red', 'oncoming': None, 'right': 'forward', 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:35,466;LearningAgent.update(): deadline = 21, inputs = {'light': 'green', 'oncoming': None, 'right': 'forward', 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:35,966;LearningAgent.update(): deadline = 20, inputs = {'light': 'green', 'oncoming': None, 'right': 'forward', 'left': None}, action = None, reward = 0.0
    2016-07-16 16:29:36,466;LearningAgent.update(): deadline = 19, inputs = {'light': 'green', 'oncoming': None, 'right': 'forward', 'left': None}, action = None, reward = 0.0

### 2.3. The Basic Agent

As mentioned before, I am going to implement a smartcab that processes the following inputs at each time step:

- *Next waypoint location*: about its current position and heading
- *Intersection state*: traffic light and presence of cars
- *Current deadline value*: time steps remaining

For the purposes of this project, the first implementation of the agent should produce just some random move at each time $t$, such that $a_t \in (None,\, forward,\, left,\, right)$. I will not implement the correct strategy because it is precisely what my agent is supposed to learn. Below I am going to simulate $100$ different trials with the `enforce_deadline` set to False and will save the logs produced by the agent to observe how it performs. In this mode, the agent is given unlimited time to reach the destination.

```python
import smartcab.eda as eda; reload(eda);
s_fname = 'log/sim_Mon_Jul_18_003801_2016.txt'
l_aux, t_aux = eda.simple_counts(s_fname)
```

    Number of Trials: 100
    Times that the agent reached the target location: 69
    Times the agent reached the hard deadline: 31
    Times the agent SUCCESSFULLY reached the target location: 21
    Counting of moves made:
    {'forward': 2018, 'None': 2085, 'right': 2035, 'left': 2094}

As can be seen above, even using just random moves, the agent still was able to reach the target destination roughly $20\%$ of the trials at the deadline stipulated by the Planner. If we considering all the times it reached the location, it was able to complete the route almost $70\%$ of the trials. Now, I am going to produce same basic statistics about the times that it reached the target location.

```python
import pandas as pd
df_last = pd.DataFrame(l_last)
ax = (df_last.ix[df_last.success==1, 'steps']).hist(bins=15, normed=True)
ax.set_title("Steps The Agent Had Taken\nto Reach the Taget Location\n", fontsize=15);
ax.set_ylabel("% Frequency")
ax.set_xlabel("Number of Steps");
```

```python
df_count = pd.DataFrame(df_last.ix[df_last.success==1, 'steps'])
df_count.columns = ["Number Of Steps"]
df_count.describe()
```

      Number Of Steps

      count
      69.000000

      mean
      59.884058

      std
      36.539880

      min
      2.000000

      25%
      24.000000

      50%
      60.000000

      75%
      91.000000

      max
      130.000000

It took 59 steps on average, and the variation was huge. The agent has taken from 2 steps to 130 to finish the route. In the section 4, I will try to improve that.

### 2.4. Identify and Update State

```
Udacity:

It is asked to:

- Reasonable states identified: Student has identified states that model the driving agent and environment, along with a sound justification. Justify why you picked these set of states, and how they model the agent and its environment.
-  Agent updates state: The driving agent updates its state when running, based on current input. The exact state does not matter, and need not be correlated with inputs, but it should change during a run.
```

To complete the implementation of my Basic Agent, I still need to decide how it will represent its state internally. Considering the inputs that it receives before performs any action and, consequently, receives a reward, I initially will use a tuple using all of them to represent the current state of my agent, in the form $(inputs,\, next\_waypoint,\, deadline)$. I believe that it will represent a reasonable set of states that my agent could use to gather more information from the environment.

$Inputs$ is a dictionary with the state of the traffic light for the agents' direction of movement and the direction of the random agents on each of the incoming roadways (if any). $next\_waypoint$ is the direction of the target location and $deadline$, a timer.

I believe that $next\_waypoint$ and $inputs$ are natural choices, given that the first gives the direction of the target and the second provides information about the environment state. The inclusion of $deadline$ might help the agent choose to explore the environment when it has more time available, for example. Later on, we will see how significant this single decision is when dealing with Reinforcement Learning problems.

So, let's count the number of stated that I get when I enforce deadline in my simulation on $100$ trials.

```python
s_fname = 'log/sim_Wed_Jul_20_234917_2016.txt'
from collections import defaultdict
d_count = defaultdict(int)

with open(s_fname) as fr:
    set_states = set()
    for idx, row in enumerate(fr):
        s_aux = row.strip().split(';')[1]
        if 'LearningAgent.update' in s_aux:
            f_aux, l_aux = s_aux[24:].split('action = ')
            state = f_aux.strip() + ','.join(l_aux.split(",")[1:])
            d_count[state] += 1

print 'number of states in the state space: {}'.format(len(d_count))
```

    number of states in the state space: 891

```python
import pandas as pd
ax = pd.Series(d_count.values()).hist(bins=7)
ax.set_xlabel("State Count")
ax.set_ylabel("Frequency")
ax.set_title("Distribution of Number of Times \nEach State Has Appeared\n", fontsize=15);
```

The figure above presents the frequency of the counting of each state. For instance, around $600$ states appeared just two times, more than $100$ appeared four times and so on. Curious the agent's behavior distribution looks like a log-normal distribution.

## 3. Implement Q-Learning

In this section, I will briefelly explained the concept of Q-learning and implement an algorithm to learn the Q function.

### 3.1. The Q-Function

As mention before, to learn an optimal policy $\pi^{*}$, I can't learn a function $\pi^{*}: S \rightarrow A$ that maps a state to the optimal action directly. There is no such information upfront to be used as training data. Instead, as \cite{Mitchell} explained, the only available information is the sequence of immediate rewards $r(s_i, a_i)$ for $i=1,\, 2,\, 3,\,...$

So, as we are trying to maximize the cumulative rewards $V^{*}(s_t)$ for all states $s$, the agent should prefer $s_1$ over $s_2$ wherever $V^{*}(s_1) > V^{*}(s_2)$. Given that the agent must choose among actions and not states, and it isn't able to perfectly predict the immediate reward and immediate successor for every possible state-action transition, we also must learn $V^{*}$ indirectly.

To solve that, we define a function $Q(s, \, a)$ such that its value is the maximum discounted cumulative reward that can be achieved starting from state $s$ and applying action $a$ as the first action. So, we can write:

$$Q(s, \, a) = r(s, a) + \gamma V^{*}(\delta(s, a))$$

As $\delta(s, a)$ is the state resulting from applying action $a$ to state $s$ (the successor) chosen by following the optimal policy, $V^{*}$ is the cumulative value of the immediate successor state discounted by a factor $\gamma$. Thus,  what we are trying to achieve is

$$\pi^{*}(s) = \underset{a}{\arg \max} Q(s, \, a)$$

Thus, the optimal policy can be obtained even if the agent just uses the current action $a$ and state $s$ and chooses the action that maximizes $Q(s,\, a)$. Also, it is important to notice that the function above implies that the agent can select optimal actions even when it has no knowledge of the functions $r$ and $\delta$. In the next subsection, we will see how \cite{Mitchell} defined a reliable way to estimate training values for $Q$, given only a sequence of immediate rewards $r$.

### 3.2. Learning $Q$

As we have seen, learning the Q function corresponds to learning the optimal policy. According to \cite{Mohri_2012}, the optimal state-action value function $Q^{*}$ is defined for all $(s, \, a) \in S \times A$ as the expected return for taking the action $a \in A$ at the state $s \in S$, following the optimal policy. So, it can be written as \cite{Mitchell} suggested:

$$V^{*}(s) = \underset{a'}{\arg \max} Q(s, \, a')$$

Using this relationship, we can write a recursive definition of Q function, such that:

$$Q(s, \, a) = r(s, a) + \gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')$$

The recursive nature of the function above implies that our agent doesn't know the actual $Q$ function. It just can estimate $Q$, that we will refer as $\hat{Q}$. It will represents is hypothesis $\hat{Q}$ as a large table that attributes each pair $(s\, , \, a)$ to a value for $\hat{Q}(s,\, a)$ - the current hypothesis about the actual but unknown value $Q(s, \, a)$. I will initialize this table with zeros, but it could be filled with random numbers, according to \cite{Mitchell}. Still according to him, the agent repeatedly should observe its current state $s$ and do the following:
- Observe the current state $s$ and:
    - Choose some action $a$ and execute it
    - Receive the immediate reward $r = r(s, a)$
    - initialize the table entry $\hat{Q}(s, \, a)$ to zero if there is no entry $(s, \, a)$
    - Observe the new state $s' = \delta(s, \,a)$.
    - Updates the table entry for $\hat{Q}(s, \, a)$ following:
        - $\hat{Q}(s, \, a) \leftarrow r + \gamma \underset{a'}{\max} \hat{Q}(s', \, a')$
- $s \leftarrow s'$

Note this training rule is suited to a deterministic Markov decision process (where $r(s,\, a)$ and $\delta(s,\, a)$ are deterministics) and uses the agent's current $\hat{Q}$ values for the new state $s'$ to refine its estimate of $\hat{Q}(s, \,a)$ for the previous state $s$. It is tricky. So, let's take a look at its performance.

```
Udacity:

It is asked to:

- Agent updates Q-values: The driving agent updates a table/mapping of Q-values correctly, implementing the Q-Learning algorithm.
- Picks the best action: Given the current set of Q-values for a state, it picks the best available action.
- Changes in behavior explained: Student has reported the changes in behavior observed, and provided a reasonable explanation for them. What changes do you notice in the agent’s behavior?
```

```python
import smartcab.eda as eda; reload(eda)
s_fname = 'log/sim_Mon_Jul_25_015925_2016.txt'
l_aux, t_aux = eda.simple_counts(s_fname)
```

    Number of Trials: 100
    Times that the agent reached the target location: 55
    Times the agent reached the hard deadline: 45
    Times the agent SUCCESSFULLY reached the target location: 55
    Counting of moves made:
    {'forward': 486, 'None': 411, 'right': 857, 'left': 463}

The agent is reaching its destination much more frequently than before. When the agent selected its action randomly, it reached its destination just 21% of the times. Now, it is reaching 55%. Now the agent is always taking the "safest action":  takes the action with the biggest reward. It is not bad, but looking at the number of explored actions by state in the Q-table produced by this simulation, we can notice something curious

```python
df_q_table = pd.read_csv('log/qtable.txt', sep='\t')
df_count = pd.DataFrame(4 - ((df_q_table.isnull())).sum(axis=1))
df_count.columns= ['Number of Explored Actions']
df_count['count'] = 1
df_grouped = df_count.groupby('Number of Explored Actions').sum()
df_grouped = df_grouped * 1. / df_grouped.sum()
df_grouped
```

      count

      Number of Explored Actions

      1
      0.562814

      2
      0.211055

      3
      0.158291

      4
      0.067839

The agent explored just one possible action in $56\%$ of the states in the Q-Table. According to \cite{Mitchell}, there are some conditions to ensure that the algorithm converges toward a $\hat{Q}$ equal to the true $Q$ function. One of them is that the system should be a deterministic MDP. Another is that the agent must select actions in a way that it visits every possible state-action pair infinitely often. Looking at the number of times this implementation of the agent explored all $4$ possibles actions (less than $7\%$ of the states), I would say that it isn't true. In the next section, I will try to improve te agent's performance.

## 4. Enhance the Driving Agent

In this section, I will try to improve the driving agent to make it consistently reaches the destination within allotted time and the net reward positive.
```
Udacity:

It is asked to:
- Improvements reported: Specific improvements made by the student beyond the basic Q-Learning implementation have been reported, including at least one parameter that was tuned along with the values tested. The corresponding results for each value are also reported.
- Agent learns a feasible policy within 100 trials: The driving agent is able to consistently reach the destination within allotted time, with net reward remaining positive.
```

### 4.1. Experimentation Strategies

One of the issues of the current strategy is that the agent could overcommit to actions that presented positive $\hat{Q}$ values early in the simulation, failing to explore other actions that could present even higher values. \cite{Mitchell} proposed to use a probabilistic approach to select actions, assigning higher probabilities to action with high $\hat{Q}$ values, but given to every action at least a nonzero probability. So, I will implement the following relation:

$$P(a_i\, | \,s ) = \frac{k ^{\hat{Q}(s, a_i)}}{\sum_j k^{\hat{Q}(s, a_j)}}$$

Where $P(a_i\, | \,s )$ is the probability of selecting the action $a_i$ given the state $s$. The constant $k$ is positive and determines how strongly the selection favors action with high $\hat{Q}$ values. Let's look at the number of times the algorithm choose to explore when we vary the value of the $k$.

```python
import smartcab.eda as eda; reload(eda);
import pandas as pd
```

```python
s_fname = 'log/sim_Wed_Aug__3_025249_2016.txt'
d1, d2, d3 = eda.performance_measures(s_fname)
df_plot = pd.DataFrame(d1) * 1.
df_plot.index = [u'exploration', u'exploitation']
ax = (df_plot / df_plot.sum()).T.plot()
ax.set_xlabel('$k$')
ax.set_ylabel('Ratio')
ax.set_title('Exploration vs. Exploitation Ratio When\n Choosing Different Values To $k$',
             fontsize=15);
```

As can be seen above, large values of $k$ assigned a higher probability of the agent exploits what it already has learned. Small values have made the agent favor the exploration of new actions. Now, let's see how the different values of $k$ impacted on the times the agent completed the routes.

```python
ax = pd.Series(d3).plot()
ax.set_xlabel('$k$')
ax.set_ylabel('On-Time Ratio')
ax.set_title('Successfully Compleated Routes When\n Choosing Different Values To $k$',
             fontsize=15);
```

The agent presented a very steep improvement as $k$ was approaching $3$. Above this value, the completed trips ratio just showed a slight growth. Now, let's check how the number of trials changed the average reward received by the agent.

```python
import matplotlib.pyplot as plt
```

```python
df_plot = pd.DataFrame(d2).iloc[10:, :]
df_plot = df_plot.unstack().reset_index()
df_plot.columns = ['k', 'trials', 'average reward']

def dateplot(x, y, **kwargs):
    ax = plt.gca()
    data = kwargs.pop("data")
    data.plot(x=x, y=y, ax=ax, **kwargs)

g = sns.FacetGrid(df_plot, col="k", col_wrap=4, size=2.5)
g = g.map_dataframe(dateplot, "trials", "average reward")
g.fig.suptitle("Cumulated Average Reward by Value of $k$",
               fontsize=16, y=1.04);
```

Clearly, small values of $k$ are worst than large values. As I want that my agent has some probability of exploring new actions, and it seems that $k=3$ is the best choice (the "slope" is not so steep, and the start value is pretty high), I will use this parameter to perform the next simulations.

### 4.2. Nondeterministic Rewards and Actions

The last modification that I want to introduce is when the environment is non-deterministic. As I said before, the algorithm used is appropriate to the deterministic case. On a nondeterministic environment, the reward function $r(s,\,a)$ and action transition function $\delta(s,\,a)$ may have probabilistic outcomes. As there are other agents in the smartcab world, I believe that it is a proper assumption.

As explained by \cite{Mitchell}, to handle nondeterministic MDPs, it is needed to redefine the value $V^{*}$ to be expressed as a expected value. So, jumping to the $Q$ function, we can re-express $Q$ recursively as:

\begin{equation}
\begin{aligned}
Q(s, \, a) &= E \left [r(s, a) \right ] + E \left [\gamma \, \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  E \left [\underset{a'}{\max} \, Q(\delta(s,\, a), \, a')\right ]\\
&= E \left [r(s, a) \right ] + \gamma \,  \sum_{s'} P\left(s' \,| \,s, a   \right) \underset{a'}{\max} \, Q(\delta(s,\, a), \, a')
\end{aligned}
\end{equation}

Where $P\left(s' \,| \,s, a   \right)$ is the probability that taking action $a$ in state $s$ will produce the next state $s'$. To implement that, we need to assume that the $\hat{Q}$ values in our table may be wrong, due to the nondeterministic behavior of our environment. So, \cite{Mitchell} suggested to modifying the training rule by introducing a decaying weighted average factor to the current $\hat{Q}$ value and to the revised estimate, such that:

$$\hat{Q}(s, \, a) \leftarrow  (1 - \alpha_{n})\, \hat{Q}_{n-1}(s, \, a) +  \alpha_{n}\left[ r + \gamma \underset{a'}{\max} \hat{Q}_{n-1}(s', \, a') \right ]$$

Where $\alpha_{n} = \frac{1}{1 + visits_{n}(s, \,a)}$ is the decaying factor and $visits_{n}$ is the total number of times this state-action $(s, \, a)$ pair has been visited. So, let's put it in practice.

```python
import smartcab.eda as eda; reload(eda)
s_fname = 'log/sim_Fri_Jul_29_180801_2016.txt'
d1, d2, d3 = eda.performance_measures(s_fname)
l_aux, t_aux = eda.simple_counts(s_fname)
```

    Number of Trials: 100
    Times that the agent reached the target location: 58
    Times the agent reached the hard deadline: 42
    Times the agent SUCCESSFULLY reached the target location: 58
    Counting of moves made:
    {'forward': 550, 'None': 542, 'right': 887, 'left': 458}

Looking just at the times the agent has completed the route, it doesn't look like we made much progress. I will compare the performance of all the agents created so far in the last section.

### 4.3. The last parameter to Tune: Gamma

There is just one remain parameter of the model that was not explored yet:  the $\gamma$. As already explained, the $\gamma$ in the $\hat{Q}$ function is a factor used to discount the immediate successor state to be used to update $\hat{Q}$ table. As $\gamma$ approaches $0$, less "memory" the agent has. So, let's check what happens when we vary this parameter about the average cumulative reward of the learning agent:

```python
s_fname = 'log/sim_Wed_Aug__3_030042_2016.txt'
d2, d3 = eda.performance_measures_gamma(s_fname)
```

```python
df_plot = pd.DataFrame(d2).iloc[10:, :]
df_plot = df_plot.unstack().reset_index()
df_plot.columns = ['$\gamma$', 'trials', 'average reward']

def dateplot(x, y, **kwargs):
    ax = plt.gca()
    data = kwargs.pop("data")
    data.plot(x=x, y=y, ax=ax, **kwargs)

g = sns.FacetGrid(df_plot, col="$\gamma$", col_wrap=3, size=2.5)
g = g.map_dataframe(dateplot, "trials", "average reward")
g.fig.suptitle("Cumulated Average Reward by Value of Gamma",
               fontsize=16, y=1.04);
```

In the last section, I will use $\gamma = 0.3$ to perform the final tests.

## 5. Conclusion

In this section, I will compare the performance of the algorithms implemented so far and then I will show how a little change can make a huge difference in a reinforcement learning problem.

### 5.1. Final Remarks

```
Udacity:

It is asked to:
- Final agent performance discussed: A description is provided of what an ideal or optimal policy would be. The performance of the final driving agent is discussed and compared to how close it is to learning the stated optimal policy. Report what changes you made to your basic implementation of Q-Learning to achieve the final version of the agent. How well does it perform? Does your agent get close to finding an optimal policy, i.e. reach the destination in the minimum possible time, and not incur any penalties?
```

The goal of the whole project was to build an agent that obeys traffic rules correctly and reachs the destination within an allotted time. So, the "perfect" agent would be the one that always come to the destination in the goal time without any penalty.

However, as our environment is non-deterministic, to always reach the destination on time is not reasonable. Also, as the environment penalizes the agent even when it makes a correct move, but in a distinct direction from the target location, it isn't fair to expect that the perfect agent never incurs any penalty.

So, the best one would be the one that learns a reasonable policy fast and receives the maximum average cumulative reward possible. Average because we should weight the rewards received by the number of steps used to avoid discrepancies. In this sense, let's first compare how the modifications made on this project reflected on our agent when we ensure deadline. The improved agent will be set with $\gamma=0.5$ and $k=3$.

```python
s_improved_agent = 'log/sim_Wed_Aug__3_030628_2016.txt'
s_basic_qlearning = 'log/sim_Mon_Jul_25_015925_2016.txt'
s_random_agent = 'log/sim_072316_142208.txt'
l_fname = [s_improved_agent, s_basic_qlearning, s_random_agent]
l_agent = ["Improved Q-learning", "Deterministic Q-learning", "Random"]
```

```python
import smartcab.eda as eda
import pandas as pd
d_rtn = {}
for s_fname, s_key in zip(l_fname, l_agent):
    l_aux, t_aux = eda.simple_counts(s_fname, b_print=False)
    d_rtn[s_key] = t_aux[1] * 1./100.

df_rtn = pd.DataFrame(pd.Series(d_rtn))
df_rtn.columns = ["On-Time Ratio"]
df_rtn.ix[l_agent, :]
```

      On-Time Ratio

      Improved Q-learning
      0.64

      Deterministic Q-learning
      0.55

      Random
      0.42

```python
d_rtn = {}
for s_fname, s_key in zip(l_fname, l_agent):
    d2, d3 = eda.performance_measures_gamma(s_fname, s_gamma="1")
    d_rtn[s_key] = d2[1]

ax = pd.DataFrame(d_rtn).ix[10:, l_agent].plot()
ax.set_xlabel("trials")
ax.set_ylabel("average reward")
ax.set_title("Cumulated Average Reward by Learner\n", fontsize=15);
```

As can be seen above, the improved agent did better than the other implementations. It reached the destination $64\%$ of the trials on time and the final average reward by action was greater than the others. It was achieved by tuning the parameters, including an experimentation strategy and changing the $\hat{Q}$ function to deal with the non-deterministic environment, as explained before. The random agent reached the destination on time just $42\%$ of the trials and the agent that used the deterministic version of $\hat{Q}$ function, $55\%$. Although the final agent is still incurring penalties, it is closer to a perfect agent than it was when using just random moves. In the next subsection, I will show how to change the behavior of the agent radically.

### 5.2. Blowing Up Everything: Changing The Intern State

```
Udacity reviewer:

Can you briefly provide some further justification here as to why each of these input variables are necessary for learning the surrounding environment and traffic laws by observing numerous variation of the environment at each intersection? Also, if a specific input is omitted, please include some discussion as to why. Likewise, can you explain whether or not deadline is necessary for the state space? For instance, due to the number of states that the deadline variable can take, will it blow up our state space into a size that cannot be feasibly explored by the agent in 100 trials if included? Keep in mind that there would be too many states here for the agent to be able to visit and in order to learn the value of each action, your agent will need to visit every state many times over.
```

As pointed out by the Udacity Reviewer, something that I should have taken into account was the size of state space resulted by my choice of agents' intern state representation. I included an Experimentation Strategy and addressed a non-deterministic MDP environment exactly due to the number of states that the agent would need to learn. It turned out being the right choice, as demonstrated in the last subsection.

However, I believe that I underestimated this single decision. My current state is represented by the tuple $(inputs,\, next\_waypoint,\, deadline)$. The $deadline$ is assigned by the Environment in a non-deterministic way, what makes my state space unlikely to learn in $100$ trials. Let's see what happens If I discard this single variable in the performance of all my $LearningAgent$:

```python
s_improved_agent = 'log/sim_Wed_Aug__3_030921_2016.txt'
s_basic_qlearning = 'log/sim_Mon_Jul_25_015925_2016.txt'
s_random_agent = 'log/sim_072316_142208.txt'
l_fname = [s_improved_agent, s_basic_qlearning, s_random_agent]
l_agent = ["Improved Q-learning", "Deterministic Q-learning", "Random"]
```

```python
d_rtn = {}
for s_fname, s_key in zip(l_fname, l_agent):
    l_aux, t_aux = eda.simple_counts(s_fname, b_print=False)
    d_rtn[s_key] = t_aux[1] * 1./100.

df_rtn = pd.DataFrame(pd.Series(d_rtn))
df_rtn.columns = ["On-Time Ratio"]
df_rtn.ix[l_agent, :]
```

      On-Time Ratio

      Improved Q-learning
      0.90

      Deterministic Q-learning
      0.55

      Random
      0.42

```python
d_rtn = {}
for s_fname, s_key in zip(l_fname, l_agent):
    d2, d3 = eda.performance_measures_gamma(s_fname, s_gamma="1")
    d_rtn[s_key] = d2[1]

ax = pd.DataFrame(d_rtn).ix[:, l_agent].plot()
ax.set_xlabel("trials")
ax.set_ylabel("average reward")
ax.set_title("Cumulated Average Reward by Learner\n", fontsize=15);
```

Using this inter-state representation, the final version of the agent is much closer to a perfect agent. It reached the destination at $90\%$ of the trials and learned most of what need to learn in less than $60$ trials.

## 6. Reflection

Something really valuable that I learned from reviewers is the importance of the state space representation and how it affects the performance of the learning agent.

Still, it's hard to say how better would be the performance of a perfect agent to contrast to the current implementation, given the stochastic environment. An option would be to implement a learning agent that already knows the traffic rules and just need to learn how to find his way to the destination on time.

*Style notebook and change matplotlib defaults*

```python
#loading style sheet
from IPython.core.display import HTML
HTML( open('ipython_style.css').read())
```

```python
#changing matplotlib defaults
%matplotlib inline
import seaborn as sns
sns.set_palette("deep", desat=.6)
sns.set_context(rc={"figure.figsize": (8, 4)})
sns.set_style("whitegrid")
sns.set_palette(sns.color_palette("Set2", 10))
```

```python

```


---

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

# Propiedades de la Solución de  Schwarzschild
---

## Resumen

Este cuaderno utiliza [SageMath](https://www.sagemath.org) y [SageManifolds](https://sagemanifolds.obspm.fr/index.html) para calcular algunas propiedades geométricas del espacio-tiempo de Schwarzschild incluyendo  simbolos de Christoffel, el tensor de Riemann, el tensor de Ricci y el escalar de Kretschmann.

Nota: Este cuaderno fue creado utilizando SageMath >= 9.4

```python
version()
```

    'SageMath version 9.4, Release Date: 2021-08-22'

Activamos la visualización matemática via LaTeX,

```python
%display latex
```

## 1.  Definición del Espacio-Tiempo

El espaciotiempo se define como una variedad Lorentziana 4-dimensional,

```python
MU = Manifold(4, 'MU', structure='Lorentzian')
print(MU)
```

    4-dimensional Lorentzian manifold MU

### 1.1. Coordenadas Estandard

Se introducen las coordenadas esfericas (estandard) o **coordenadas de Schwarzschild-Droste**  utilizando el método `.chart()` del objeto variedad`MU`. El argumento del método `.chart()` es una cadena de texto que define cada una de las coordenadas y sus rangos, si son diferentes de $(-\infty, +\infty)$, al igual que el sìmbolo LaTeX que las representará.

Note que las variables de `Python` para cada coordenada son declaradas dentro del operador `<...>`en el lado izquierdo. La variable `X` denotará la carta coordenada.

```python
X.<t,r,th,ph> = MU.chart(r"t r:(0,+oo) th:(0,pi):\theta ph:(0,2*pi):\phi")
X
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(MU,(t, r, {\theta}, {\phi})\right)\]

## 2. Tensor Metrico

Para definir la métrica de Schwarzschild, es necesario declarar el parámetro de masa $M$ como una variable simbólica utilizando al función `var()`. Adicionalmente asumiremos que este parámetro toma valores positivos.

```python
M = var('M')
assume(M>=0)
```

Ahora definimos el tensor métrico

```python
g = MU.metric()
g[0,0] = -(1-2*M/r)
g[1,1] = 1/(1-2*M/r)
g[2,2] = r^2
g[3,3] = (r*sin(th))^2
g.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}g = \left( \frac{2 \, M}{r} - 1 \right) \mathrm{d} t\otimes \mathrm{d} t + \left( -\frac{1}{\frac{2 \, M}{r} - 1} \right) \mathrm{d} r\otimes \mathrm{d} r + r^{2} \mathrm{d} {\theta}\otimes \mathrm{d} {\theta} + r^{2} \sin\left({\theta}\right)^{2} \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\]

```python
g[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
\frac{2 \, M}{r} - 1 & 0 & 0 & 0 \\
0 & -\frac{1}{\frac{2 \, M}{r} - 1} & 0 & 0 \\
0 & 0 & r^{2} & 0 \\
0 & 0 & 0 & r^{2} \sin\left({\theta}\right)^{2}
\end{array}\right)\]

## 3. Simbolos de Christoffel (Conexiones)

Los simbolos de Christoffel (de segunda especie) para la métrica $g$ en las coordenadas de Schwarzschild-Droste se definen como

\begin{equation}
\Gamma^\alpha_{\mu \nu} = \frac{1}{2} g^{\alpha \sigma} \left[ \partial_\mu g_{\sigma \nu} + \partial_\nu g_{\mu \sigma } - \partial_\sigma g_{\mu \nu} \right]
\end{equation}

Estas componentes se pueden obtener mediante el método `.christoffel_symbols_display()` del objeto métrica, `g`. Por defecto, solo se mostrarán las componentes no-nulas y no-redundantes (i.e. se tiene en cuenta la propiedad de simetría $\Gamma^\alpha_{\mu \nu} = \Gamma^\alpha_{\nu \mu}$).

De igual forma, utilizando el símbolo `?` al final del comando, i.e. `g.christoffel_symbols_display?` se visualiza la información y opciones disponibles para el método.

```python
g.christoffel_symbols_display?
```

```python
g.christoffel_symbols_display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\begin{array}{lcl} \Gamma_{ \phantom{\, t} \, t \, r }^{ \, t \phantom{\, t} \phantom{\, r} } & = & -\frac{M}{2 \, M r - r^{2}} \\ \Gamma_{ \phantom{\, r} \, t \, t }^{ \, r \phantom{\, t} \phantom{\, t} } & = & -\frac{2 \, M^{2} - M r}{r^{3}} \\ \Gamma_{ \phantom{\, r} \, r \, r }^{ \, r \phantom{\, r} \phantom{\, r} } & = & \frac{M}{2 \, M r - r^{2}} \\ \Gamma_{ \phantom{\, r} \, {\theta} \, {\theta} }^{ \, r \phantom{\, {\theta}} \phantom{\, {\theta}} } & = & 2 \, M - r \\ \Gamma_{ \phantom{\, r} \, {\phi} \, {\phi} }^{ \, r \phantom{\, {\phi}} \phantom{\, {\phi}} } & = & {\left(2 \, M - r\right)} \sin\left({\theta}\right)^{2} \\ \Gamma_{ \phantom{\, {\theta}} \, r \, {\theta} }^{ \, {\theta} \phantom{\, r} \phantom{\, {\theta}} } & = & \frac{1}{r} \\ \Gamma_{ \phantom{\, {\theta}} \, {\phi} \, {\phi} }^{ \, {\theta} \phantom{\, {\phi}} \phantom{\, {\phi}} } & = & -\cos\left({\theta}\right) \sin\left({\theta}\right) \\ \Gamma_{ \phantom{\, {\phi}} \, r \, {\phi} }^{ \, {\phi} \phantom{\, r} \phantom{\, {\phi}} } & = & \frac{1}{r} \\ \Gamma_{ \phantom{\, {\phi}} \, {\theta} \, {\phi} }^{ \, {\phi} \phantom{\, {\theta}} \phantom{\, {\phi}} } & = & \frac{\cos\left({\theta}\right)}{\sin\left({\theta}\right)} \end{array}\]

También es posible acceder a cada componente particular,

```python
g.christoffel_symbols()[0,1,0]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}-\frac{M}{2 \, M r - r^{2}}\]

Podemos comprobar que la simetría en los indices inferiores se satisface,

```python
g.christoffel_symbols()[0,0,1] == g.christoffel_symbols()[0,1,0]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{True}\]

## 4. Tensor de Curvatura

El tensor de curvatura de Riemann se define como

\begin{equation}
R^\mu_{\,\, \nu \rho \sigma} = \frac{\partial \Gamma^\mu_{ \nu \sigma}}{\partial x^{\rho}} - \frac{\partial \Gamma^\mu_{ \nu \rho}}{\partial x^{\sigma}} + \Gamma^{\alpha}_{\nu \sigma}\Gamma^\mu_{\rho \alpha} - \Gamma^{\alpha}_{\ \nu \rho}\Gamma^\mu_{\sigma \alpha}
\end{equation}

Este puede calculares mediante el método `.riemann()` de la métrica,

```python
Riemann = g.riemann()
print(Riemann)
```

    Tensor field Riem(g) of type (1,3) on the 4-dimensional Lorentzian manifold MU

**Es importante notar que este es un tensor de orden (1,3)**.

```python
Riemann
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Riem}\left(g\right)\]

Para visualizar sus componentes utilizamos el método `.display()`,

```python
Riemann.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Riem}\left(g\right) = \left( -\frac{2 \, M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial t }\otimes \mathrm{d} r\otimes \mathrm{d} t\otimes \mathrm{d} r + \left( \frac{2 \, M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial t }\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} t -\frac{M}{r} \frac{\partial}{\partial t }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t\otimes \mathrm{d} {\theta} + \frac{M}{r} \frac{\partial}{\partial t }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t -\frac{M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial t }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t\otimes \mathrm{d} {\phi} + \frac{M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial t }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t -\frac{2 \, {\left(2 \, M^{2} - M r\right)}}{r^{4}} \frac{\partial}{\partial r }\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} r + \frac{2 \, {\left(2 \, M^{2} - M r\right)}}{r^{4}} \frac{\partial}{\partial r }\otimes \mathrm{d} t\otimes \mathrm{d} r\otimes \mathrm{d} t -\frac{M}{r} \frac{\partial}{\partial r }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r\otimes \mathrm{d} {\theta} + \frac{M}{r} \frac{\partial}{\partial r }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r -\frac{M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial r }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r\otimes \mathrm{d} {\phi} + \frac{M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial r }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r + \left( \frac{2 \, M^{2} - M r}{r^{4}} \right) \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} {\theta} + \left( -\frac{2 \, M^{2} - M r}{r^{4}} \right) \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} t\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t + \left( -\frac{M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} {\theta} + \left( \frac{M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} r\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r + \frac{2 \, M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi} -\frac{2 \, M \sin\left({\theta}\right)^{2}}{r} \frac{\partial}{\partial {\theta} }\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta} + \left( \frac{2 \, M^{2} - M r}{r^{4}} \right) \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} {\phi} + \left( -\frac{2 \, M^{2} - M r}{r^{4}} \right) \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} t\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t + \left( -\frac{M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} {\phi} + \left( \frac{M}{2 \, M r^{2} - r^{3}} \right) \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} r\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r -\frac{2 \, M}{r} \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi} + \frac{2 \, M}{r} \frac{\partial}{\partial {\phi} }\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\]

```python
Riemann[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left[\left[\left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, -\frac{2 \, M}{2 \, M r^{2} - r^{3}}, 0, 0\right], \left[\frac{2 \, M}{2 \, M r^{2} - r^{3}}, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, -\frac{M}{r}, 0\right], \left[0, 0, 0, 0\right], \left[\frac{M}{r}, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, -\frac{M \sin\left({\theta}\right)^{2}}{r}\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[\frac{M \sin\left({\theta}\right)^{2}}{r}, 0, 0, 0\right]\right]\right], \left[\left[\left[0, -\frac{2 \, {\left(2 \, M^{2} - M r\right)}}{r^{4}}, 0, 0\right], \left[\frac{2 \, {\left(2 \, M^{2} - M r\right)}}{r^{4}}, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, -\frac{M}{r}, 0\right], \left[0, \frac{M}{r}, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, -\frac{M \sin\left({\theta}\right)^{2}}{r}\right], \left[0, 0, 0, 0\right], \left[0, \frac{M \sin\left({\theta}\right)^{2}}{r}, 0, 0\right]\right]\right], \left[\left[\left[0, 0, \frac{2 \, M^{2} - M r}{r^{4}}, 0\right], \left[0, 0, 0, 0\right], \left[-\frac{2 \, M^{2} - M r}{r^{4}}, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, -\frac{M}{2 \, M r^{2} - r^{3}}, 0\right], \left[0, \frac{M}{2 \, M r^{2} - r^{3}}, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, \frac{2 \, M \sin\left({\theta}\right)^{2}}{r}\right], \left[0, 0, -\frac{2 \, M \sin\left({\theta}\right)^{2}}{r}, 0\right]\right]\right], \left[\left[\left[0, 0, 0, \frac{2 \, M^{2} - M r}{r^{4}}\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[-\frac{2 \, M^{2} - M r}{r^{4}}, 0, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, -\frac{M}{2 \, M r^{2} - r^{3}}\right], \left[0, 0, 0, 0\right], \left[0, \frac{M}{2 \, M r^{2} - r^{3}}, 0, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, -\frac{2 \, M}{r}\right], \left[0, 0, \frac{2 \, M}{r}, 0\right]\right], \left[\left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right], \left[0, 0, 0, 0\right]\right]\right]\right]\]

Cada componente particular se visualiza utilizando los índices correspondientes. Por ejemplo, la componente $R^0_{\ \,101} = R^t_{\ \,trt}$ será

```python
Riemann[0,1,1,0]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\frac{2 \, M}{2 \, M r^{2} - r^{3}}\]

```python
Riemann[0,1,0,1]
```

Para obtener el tensor de curvatura del tipo (0,4) se utiliza el método `.down()`:

```python
Riemann_dddd = Riemann.down(g)
Riemann_dddd
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mbox{Tensor field of type (0,4) on the 4-dimensional Lorentzian manifold MU}\]

```python
Riemann_dddd.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}-\frac{2 \, M}{r^{3}} \mathrm{d} t\otimes \mathrm{d} r\otimes \mathrm{d} t\otimes \mathrm{d} r + \frac{2 \, M}{r^{3}} \mathrm{d} t\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} t + \left( -\frac{2 \, M^{2} - M r}{r^{2}} \right) \mathrm{d} t\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t\otimes \mathrm{d} {\theta} + \left( \frac{2 \, M^{2} - M r}{r^{2}} \right) \mathrm{d} t\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t -\frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \mathrm{d} t\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t\otimes \mathrm{d} {\phi} + \frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \mathrm{d} t\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t + \frac{2 \, M}{r^{3}} \mathrm{d} r\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} r -\frac{2 \, M}{r^{3}} \mathrm{d} r\otimes \mathrm{d} t\otimes \mathrm{d} r\otimes \mathrm{d} t + \left( \frac{M}{2 \, M - r} \right) \mathrm{d} r\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r\otimes \mathrm{d} {\theta} + \left( -\frac{M}{2 \, M - r} \right) \mathrm{d} r\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r + \left( \frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \right) \mathrm{d} r\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r\otimes \mathrm{d} {\phi} + \left( -\frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \right) \mathrm{d} r\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r + \left( \frac{2 \, M^{2} - M r}{r^{2}} \right) \mathrm{d} {\theta}\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} {\theta} + \left( -\frac{2 \, M^{2} - M r}{r^{2}} \right) \mathrm{d} {\theta}\otimes \mathrm{d} t\otimes \mathrm{d} {\theta}\otimes \mathrm{d} t + \left( -\frac{M}{2 \, M - r} \right) \mathrm{d} {\theta}\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} {\theta} + \left( \frac{M}{2 \, M - r} \right) \mathrm{d} {\theta}\otimes \mathrm{d} r\otimes \mathrm{d} {\theta}\otimes \mathrm{d} r + 2 \, M r \sin\left({\theta}\right)^{2} \mathrm{d} {\theta}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi} -2 \, M r \sin\left({\theta}\right)^{2} \mathrm{d} {\theta}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta} + \frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \mathrm{d} {\phi}\otimes \mathrm{d} t\otimes \mathrm{d} t\otimes \mathrm{d} {\phi} -\frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \mathrm{d} {\phi}\otimes \mathrm{d} t\otimes \mathrm{d} {\phi}\otimes \mathrm{d} t + \left( -\frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \right) \mathrm{d} {\phi}\otimes \mathrm{d} r\otimes \mathrm{d} r\otimes \mathrm{d} {\phi} + \left( \frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \right) \mathrm{d} {\phi}\otimes \mathrm{d} r\otimes \mathrm{d} {\phi}\otimes \mathrm{d} r -2 \, M r \sin\left({\theta}\right)^{2} \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi} + 2 \, M r \sin\left({\theta}\right)^{2} \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\otimes \mathrm{d} {\phi}\otimes \mathrm{d} {\theta}\]

```python
Riemann_dddd.display_comp()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\begin{array}{lcl} X_{ \, t \, r \, t \, r }^{ \phantom{\, t}\phantom{\, r}\phantom{\, t}\phantom{\, r} } & = & -\frac{2 \, M}{r^{3}} \\ X_{ \, t \, r \, r \, t }^{ \phantom{\, t}\phantom{\, r}\phantom{\, r}\phantom{\, t} } & = & \frac{2 \, M}{r^{3}} \\ X_{ \, t \, {\theta} \, t \, {\theta} }^{ \phantom{\, t}\phantom{\, {\theta}}\phantom{\, t}\phantom{\, {\theta}} } & = & -\frac{2 \, M^{2} - M r}{r^{2}} \\ X_{ \, t \, {\theta} \, {\theta} \, t }^{ \phantom{\, t}\phantom{\, {\theta}}\phantom{\, {\theta}}\phantom{\, t} } & = & \frac{2 \, M^{2} - M r}{r^{2}} \\ X_{ \, t \, {\phi} \, t \, {\phi} }^{ \phantom{\, t}\phantom{\, {\phi}}\phantom{\, t}\phantom{\, {\phi}} } & = & -\frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \\ X_{ \, t \, {\phi} \, {\phi} \, t }^{ \phantom{\, t}\phantom{\, {\phi}}\phantom{\, {\phi}}\phantom{\, t} } & = & \frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \\ X_{ \, r \, t \, t \, r }^{ \phantom{\, r}\phantom{\, t}\phantom{\, t}\phantom{\, r} } & = & \frac{2 \, M}{r^{3}} \\ X_{ \, r \, t \, r \, t }^{ \phantom{\, r}\phantom{\, t}\phantom{\, r}\phantom{\, t} } & = & -\frac{2 \, M}{r^{3}} \\ X_{ \, r \, {\theta} \, r \, {\theta} }^{ \phantom{\, r}\phantom{\, {\theta}}\phantom{\, r}\phantom{\, {\theta}} } & = & \frac{M}{2 \, M - r} \\ X_{ \, r \, {\theta} \, {\theta} \, r }^{ \phantom{\, r}\phantom{\, {\theta}}\phantom{\, {\theta}}\phantom{\, r} } & = & -\frac{M}{2 \, M - r} \\ X_{ \, r \, {\phi} \, r \, {\phi} }^{ \phantom{\, r}\phantom{\, {\phi}}\phantom{\, r}\phantom{\, {\phi}} } & = & \frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \\ X_{ \, r \, {\phi} \, {\phi} \, r }^{ \phantom{\, r}\phantom{\, {\phi}}\phantom{\, {\phi}}\phantom{\, r} } & = & -\frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \\ X_{ \, {\theta} \, t \, t \, {\theta} }^{ \phantom{\, {\theta}}\phantom{\, t}\phantom{\, t}\phantom{\, {\theta}} } & = & \frac{2 \, M^{2} - M r}{r^{2}} \\ X_{ \, {\theta} \, t \, {\theta} \, t }^{ \phantom{\, {\theta}}\phantom{\, t}\phantom{\, {\theta}}\phantom{\, t} } & = & -\frac{2 \, M^{2} - M r}{r^{2}} \\ X_{ \, {\theta} \, r \, r \, {\theta} }^{ \phantom{\, {\theta}}\phantom{\, r}\phantom{\, r}\phantom{\, {\theta}} } & = & -\frac{M}{2 \, M - r} \\ X_{ \, {\theta} \, r \, {\theta} \, r }^{ \phantom{\, {\theta}}\phantom{\, r}\phantom{\, {\theta}}\phantom{\, r} } & = & \frac{M}{2 \, M - r} \\ X_{ \, {\theta} \, {\phi} \, {\theta} \, {\phi} }^{ \phantom{\, {\theta}}\phantom{\, {\phi}}\phantom{\, {\theta}}\phantom{\, {\phi}} } & = & 2 \, M r \sin\left({\theta}\right)^{2} \\ X_{ \, {\theta} \, {\phi} \, {\phi} \, {\theta} }^{ \phantom{\, {\theta}}\phantom{\, {\phi}}\phantom{\, {\phi}}\phantom{\, {\theta}} } & = & -2 \, M r \sin\left({\theta}\right)^{2} \\ X_{ \, {\phi} \, t \, t \, {\phi} }^{ \phantom{\, {\phi}}\phantom{\, t}\phantom{\, t}\phantom{\, {\phi}} } & = & \frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \\ X_{ \, {\phi} \, t \, {\phi} \, t }^{ \phantom{\, {\phi}}\phantom{\, t}\phantom{\, {\phi}}\phantom{\, t} } & = & -\frac{{\left(2 \, M^{2} - M r\right)} \sin\left({\theta}\right)^{2}}{r^{2}} \\ X_{ \, {\phi} \, r \, r \, {\phi} }^{ \phantom{\, {\phi}}\phantom{\, r}\phantom{\, r}\phantom{\, {\phi}} } & = & -\frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \\ X_{ \, {\phi} \, r \, {\phi} \, r }^{ \phantom{\, {\phi}}\phantom{\, r}\phantom{\, {\phi}}\phantom{\, r} } & = & \frac{M \sin\left({\theta}\right)^{2}}{2 \, M - r} \\ X_{ \, {\phi} \, {\theta} \, {\theta} \, {\phi} }^{ \phantom{\, {\phi}}\phantom{\, {\theta}}\phantom{\, {\theta}}\phantom{\, {\phi}} } & = & -2 \, M r \sin\left({\theta}\right)^{2} \\ X_{ \, {\phi} \, {\theta} \, {\phi} \, {\theta} }^{ \phantom{\, {\phi}}\phantom{\, {\theta}}\phantom{\, {\phi}}\phantom{\, {\theta}} } & = & 2 \, M r \sin\left({\theta}\right)^{2} \end{array}\]

```python
Riemann_dddd[0,1,0,1]
```

```python
Riemann_dddd[0,1,1,0]
```

```python
Riemann_dddd[1,0,0,1]
```

```python
Riemann_dddd[1,0,0,1]
```

## 5. Tensor de Ricci

El tensor de Ricci se calcula con el método `.ricci()` de la metrica,

```python
Ricci = g.ricci()
Ricci
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Ric}\left(g\right)\]

Como era de esperar, todas sus componentes son cero en este caso,

```python
Ricci.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Ric}\left(g\right) = 0\]

```python
Ricci[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0
\end{array}\right)\]

### Tensor de Ricci Tensor (Version Alternativa)

Otra forma de calcular este tensor es contrayendo el trensor de Riemann

\begin{equation}
R_{\mu \nu} = R^\alpha_{\,\, \mu \alpha \nu} = g ^{\alpha \beta} R_{\alpha \mu \beta \nu}
\end{equation}

```python
Ric = g.up(g)['^{ab}'] * Riemann.down(g)['_{acbd}']
Ricci[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0
\end{array}\right)\]

### 5.1. Escalar de Curvatura de Ricci

El escalar de curvatura puede obtenerse contrayendo la métrica con el tesnor de Ricci,

```python
R = g['_{ab}'] * Ricci.up(g)['^{ab}']
print(R)
```

    Scalar field dzero(∂/∂th) on the 4-dimensional Lorentzian manifold MU

```python
R.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\begin{array}{llcl} \mathrm{d}0\left( \frac{\partial}{\partial {\theta} } \right) : & MU & \longrightarrow & \mathbb{R} \\ & \left(t, r, {\theta}, {\phi}\right) & \longmapsto & 0 \end{array}\]

La expresión simbólica de un campo escalar se realiza utilizando el método `.expr()`:

```python
R.expr()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}0\]

También es posible utilizar el método `.ricci_scalar()`

```python
R = g.ricci_scalar()
R.expr()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}0\]

## 6. Escalar de Kretschmann

El escalar de Kretschmann es el "cuadrado" del tensor de Riemann,

\begin{equation}
K = R_{\mu \nu \rho \sigma} R^{\mu \nu \rho \sigma}
\end{equation}

Para calcularlo, se necesita el tensor de curvtura con todos los indices abajo y con todos los indices arriba,  i.e. $R_{\mu \nu \rho \sigma}$ y
$R^{\mu \nu \rho \sigma}$. Estos se evaluan utilizando los métodos`.down()` y `.up()`. Luego se realiza la contracción de los índices en el orden establecido,

```python
K = Riemann.down(g)['_{abcd}'] * Riemann.up(g)['^{abcd}']
K
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mbox{Scalar field on the 4-dimensional Lorentzian manifold MU}\]

```python
K.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\begin{array}{llcl} & MU & \longrightarrow & \mathbb{R} \\ & \left(t, r, {\theta}, {\phi}\right) & \longmapsto & \frac{48 \, M^{2}}{r^{6}} \end{array}\]

```python
K.expr()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\frac{48 \, M^{2}}{r^{6}}\]

### Summary of methods

```
Manifold.chart()
Manifold.metric()
```

```
metric.christoffel_symbols_display()
metric.christoffel_symbols()[]
metric.riemann()
metric.ricci()
```

```
tensor.display()
tensor[:]
tensor.expr()
tensor.up(metric)
tensor.down(metric)
```

```python
g.display?
```

```python

```


# **4.3: Cyclic pullout of textile fabrics and CFRP sheets**

[](https://moodle.rwth-aachen.de/mod/page/view.php?id=619022)

        Starting point

Once we have seen that there is a relation between the shape of the constitutive law and the  stress redistribution process within a bond zone which has an immediate consequence on the shape of the pullout curve, we extend our horizon to the case of **non-monotonic loading**.

        Where are we heading

To motivate a more general development of the model for the bond-slip behavior let us consider the case of non-monotonic loading. What happens in the material structure of the bond if the load is reduced and than it grows again?

# **TRC pullout test modeled using plastic material model**

## Test results

Consider again the test double-sided pullout test that was used to motivate the development of plastic material model in [notebook 4.1](4_1_PO_multilinear_unloading.ipynb#trc_study_monotonic) which demonstrated that no unloading branch could be reproduced. We reprint the test results for convenience here with the goal to examine the ability of the plastic model to reproduce the unloading stiffness of the specimen with the total length of 300 mm, i.e. $L_\mathrm{b} = 150$ mm.

## Pullout model setup

Let us first construct the model with the same geometry as specified in [notebook 3.2](../tour3_nonlinear_bond/3_2_anchorage_length.ipynb#trc_parameters). Note again that the maximum control displacement $w_\mathrm{max}$ is
the half of the crack opening displacement measured in the test. Therefore `w_max` has been set to 3 mm.

```python
%matplotlib widget
import matplotlib.pyplot as plt
import numpy as np
from bmcs_cross_section.pullout import PullOutModel1D
po_trc = PullOutModel1D(n_e_x=100, w_max=3) # mm
po_trc.geometry.L_x=150 # [mm]
po_trc.time_line.step = 0.02
po_trc.cross_section.trait_set(A_m=1543, A_f=16.7, P_b=10);
```

## Material model with kinematic hardening

To apply the elastic-plastic model in this pullout model, set the the `elasto-plasticity` option for the `material_model` attribute. Then, the plastic model with the parameters $\tau_Y = \bar{\tau}, K, \gamma$ introduced in the [notebook 3.2](../tour3_nonlinear_bond/3_2_anchorage_length.ipynb#trc_pullout_study) can be set to the attribute `material_model_` using the `trait_set` method as follows

| parameter | name | value | unit |
| - | - | -: | - |
| bond stiffness | `E_b` | 6.4 | MPa |
| elastic limit stress | `tau_bar` | 4 | MPa |
| isotropic hardening modulus | `K` | 0 | MPa |
| kinematic hardening modulus | `gamma` | 1.0 | MPa |

Using these parameters, we aim to reproduce the same shape of the [bond slip law that was used previously in notebook 3.2](../tour3_nonlinear_bond/3_2_anchorage_length.ipynb#trc_parameters), represented as a tri-linear function for the simulation of the monotonically increasing load.

```python
po_trc.material_model='elasto-plasticity'
po_trc.material_model_.trait_set(
    E_m=28000, E_f=170000, E_b=6.4, K = 0, gamma=1.0, tau_bar=4
);
po_trc.material_model_.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Load scenario

Finally, following the same mimic, we change the loading scenario from `monotonic` to `cyclic` and can set the parameters of the cyclic loading through the attribute with the trailing underscore `loading_scenario_`.
To verify the loading scenario we directly render the model component using the `interact` method.

```python
po_trc.loading_scenario = 'cyclic'
po_trc.loading_scenario_.trait_set(number_of_cycles=2,
                              unloading_ratio=0.8,
                            number_of_increments=200,
                              amplitude_type='constant',
                              loading_range='non-symmetric');
po_trc.loading_scenario_.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Run the simulation
Apparently, we apply directly the full control displacement, then reduce it to 80% and reload again to the maximum. Our intention is not to simulate the whole history displayed in the [test results](trc_test_results) but to compare the unloading stiffness reproduced by the model.
The model is now ready for execution so that let us run it and render the user interface to inspect the results.

```python
po_trc.reset()
po_trc.run()
po_trc.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Discussion of the study

 - The predicted response reproduces the experimental response reasonably well. For the **short bond length** used in the test, the slip and shear profiles are nearly uniform along the bond zone.  - **Task:** Change the bond length to 200 mm and compare the response with the pull-out curve depicted in the [test results](#trc_test_results)
 - The main focus of the study at hand was the question, if the **unloading stiffness** corresponds to the initial stiffness, similarly to the test response. Since this is the case, the conclusion can be made, that the bond behavior in the tested TRC specimens is primarily governed by plasticity.
 - Since the model uses **linear hardening**, the pull-out response has a linear inelastic branch up to the point of unloading. The non-linear shape of the experimentally obtained curve cannot be reproduced in a better way. **Non-linear hardening** function instead of a single hardening modulus $\gamma$ can be theoretically included into the framework of plasticity. However, such enhancement requires a non-trivial systematic calibration procedure which would be able to uniquely associate the inelastic mechanisms governing the response to the hardening process, which is a non-trivial task. Such effort would only be justified for a material combination with large amount of practical applications.

# **CFRP sheet test with softening plastic material**

It is interesting to examine the ability of the plastic model with negative hardening, i.e. softening, to simulate the cyclic response of the CFRP sheets.

## Pullout model setup
Let us construct the model with these parameters and check the shape of the bond-slip relation

```python
%matplotlib widget
import matplotlib.pyplot as plt
import numpy as np
from bmcs_cross_section.pullout import PullOutModel1D

A_f = 16.67 # [mm^2]
A_m = 1540.0 # [mm^2]
p_b = 100.0 #
E_f = 170000 # [MPa]
E_m = 28000 # [MPa]
pm = PullOutModel1D()
pm.sim.tline.step = 0.01 # 100 time increments
pm.cross_section.trait_set(A_f=A_f, P_b=p_b, A_m=A_m)
pm.geometry.L_x = 300 # length of the specimen [mm]
pm.w_max = 2.8 # maximum control displacement [mm]
pm.n_e_x = 100 # number of finite elements
```

## Material model with isotropic softening

The bond-slip law identified during the [**Tour 3:** (CFRP sheet test)](../tour3_nonlinear_bond/3_1_nonlinear_bond.ipynb#cfrp_bond_slip) can be reproduced by setting the parameters of
the plastic material model with isotropic hardening to:
| parameter | name | value | unit |
| - | - | -: | - |
| bond stiffness | `E_b` | 80 | MPa |
| elastic limit stress | `tau_bar` | 8 | MPa |
| isotropic hardening modulus | `K` | -20 | MPa |
| kinematic hardening modulus | `gamma` | 0 | MPa |

```python
pm.material_model='elasto-plasticity'
pm.material_model_.trait_set(
    E_m=28000, E_f=170000, E_b=80, K = -20, gamma=0, tau_bar=8
)
pm.material_model_.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

        What is the difference compared to a trilinear bond-slip law used before?

Apparently, the shape is identical to the [trilinear bond slip law](../tour3_nonlinear_bond/3_1_nonlinear_bond.ipynb#cfrp_trilinear_bond) law applied in the first analysis of this test setup. However, now unloading in inelastic regime would be governed by initial stiffness.
It is important to remark that the model has been augmented with an additional feature that prevents softening into negative stresses. To explain this minor but essential extension recal the shape of the yield condition for plastic model with isotropic hardening
\begin{align}
f = | \tau | - (\tau_Y - Z)
\end{align}
where the hardening stress is given as $Z = K z$. With the negative value of hardening modulus $K$ and implicitly positive value of the hardening state variable $z$ we can notice that the size of the elastic domain $(\tau_Y - Z)$ shrinks with increasing $z$. Without any treatment, the elastic domain could reach a negative size, which is nonphysical. Thus, the criterion of the positive size of the elastic domain is used to define a third branch in the plastic model.

**In fact,** with this necessary extension, we introduce **damage into a plastic model**, because we explicitly set the material stiffness at a predefined value of deformation, namely at the value of slip inducing the loss of elastic domain. As we will lear in Tour 5, relating the loss of stiffness to the measure of strain is the general concept behind damage models. :

In this context, where we introduced an argument of physical admissibility, it is worth reminding that inelastic behavior implicitly induces **dissipation of energy**. A sound formulation of material models is only possible in the framework of thermodynamic laws that can guarantee an admissible behavior for any loading history. The evaluation of energy dissipation and its importance for objectivity and robustness of material models will be illuminated in **Tour 6**.

With the material model at hand, let us explore the response of the CFRP sheets in a cyclic loading scenario with symmetric, increasing amplitudes

## Load scenario

To see the effect of propagating softening plasticity upon unloading, let us apply four loading cycles oscillating applying the pull-out and push-in load in each cycle. Note that with this,
a compressive stress is transferred by the FRP sheet upon push-in loading, which is unrealistic.
Such response might be expected from a test on CFRP bars. Still the purpose of the study is
to show the consequences of the applied assumptions on a broad scale of loading configuration and check the plausibility of the model response for a wider range of loading scenarios.

```python
pm.loading_scenario = 'cyclic'
pm.loading_scenario_.trait_set(number_of_cycles=4,
                              unloading_ratio=0.5,
                               number_of_increments=200,
                              amplitude_type='increasing',
                              loading_range='symmetric')
pm.run()
pm.interact()
```

    VBox(children=(HBox(children=(VBox(children=(Tree(layout=Layout(align_items='stretch', border='solid 1px black…

## Discussion of the study

 - By checking the `history` showing the field variables along the bond zone, we can observe the the alternating tension and compression profiles in the reinforcement and in the matrix.
 - The shear transfer is again taking place only on a short length due to the softening nature of the bond behavior.
 - The yielding of the bond within the process zone occurs both under pullout and under push-in load as apparent from the existence of the hysteretic loop.
 - In contrast to the [study of TRC test](#trc_results), the unloading stiffness does not correspond to the initial stiffness. This question is discussed in a more detail below.

        Stimulating question

# **Qualitative comparison of the two studies**

## **How to interpret unloading stiffness at structural level?**

In the classification of plasticity and damage models explained in  [4.1](4_1_PO_multilinear_unloading.ipynb#damage_and_plasticity) we have learned that in a single material point, plastic behavior unloads with initial stiffness and induces a persistent deformation $s_\mathrm{pl}$ at zero level of load. On the other hand, unloading of material point governed by damage behavior would recover all deformation and reach zero deformation at zero stress. **Can we assume such behavior of both class of models at the structural level?**

Comparing the slopes of the first unloading branch obtained for the [TRC test](trc_test) and of the [CFRP test](cfrp_test), we can notice that in the former case, the unloading stiffness is indeed equal to the initial stiffness. In this case, governed by the hardening bond-slip law, the shear stress increases and decreases upon loading and unloading in all material points simultaneously. On the other hand, in the case of [CFRP test](cfrp_test) governed by softening, the debonded zone reaches zero stress level and does not contribute to stress transfer any more. As a result, the effective length of the bond zone becomes shorter, so that the structural stiffness upon unloading must be smaller than the initial stiffness.

**The conclusion** is, that an application of a plastic model with softening,
is only valid up to the point at which the elastic domain vanished. From that point on,
a full damage must be assumed. Therefore, the kind of model used in the simulation of the CFRP test can be regarded as a first version of a combined damage-plasticity model.

## **When to apply isotropic and when kinematic hardening?**

In the two examples above, both kinematic and isotropic hardening has been used. In the first yielding step and upon the first unloading, there is no difference between the two parameters. They will only show a different behavior upon cyclic loading scenario. A correct distinction is only possible based on a test delivering several unloading and reloading cycles. Correct identification of the parameters is a demanding task which calls for a systematic and automated support in terms of calibration algorithms and validation procedures.

      4.2 Basic concept of plasticity

    5.1 Pullout simulation using damage model

```python

```


# About This Notebook

This notebook shows how to implement **Low-Rank Tensor Completion with Truncated Nuclear Norm minimization (LRTC-TNN)** on some real-world data sets. For an in-depth discussion of LRTC-TNN, please see our article [1].

[1] Xinyu Chen, Jinming Yang, Lijun Sun (2020). A Nonconvex Low-Rank Tensor Completion Model for Spatiotemporal Traffic Data Imputation. arXiv.2003.10271. [PDF]

## Quick Run

This notebook is publicly available for any usage at our data imputation project. Please check out [**transdim - GitHub**](https://github.com/xinychen/transdim).

## Low-Rank Tensor Completion

We start by importing the necessary dependencies.

```python
import numpy as np
from numpy.linalg import inv as inv
```

### Tensor Unfolding (`ten2mat`) and Matrix Folding (`mat2ten`)

Using numpy reshape to perform 3rd rank tensor unfold operation. [[**link**](https://stackoverflow.com/questions/49970141/using-numpy-reshape-to-perform-3rd-rank-tensor-unfold-operation)]

```python
def ten2mat(tensor, mode):
    return np.reshape(np.moveaxis(tensor, mode, 0), (tensor.shape[mode], -1), order = 'F')
```

```python
X = np.array([[[1, 2, 3, 4], [3, 4, 5, 6]],
              [[5, 6, 7, 8], [7, 8, 9, 10]],
              [[9, 10, 11, 12], [11, 12, 13, 14]]])
print('tensor size:')
print(X.shape)
print('original tensor:')
print(X)
print()
print('(1) mode-1 tensor unfolding:')
print(ten2mat(X, 0))
print()
print('(2) mode-2 tensor unfolding:')
print(ten2mat(X, 1))
print()
print('(3) mode-3 tensor unfolding:')
print(ten2mat(X, 2))
```

    tensor size:
    (3, 2, 4)
    original tensor:
    [[[ 1  2  3  4]
      [ 3  4  5  6]]

     [[ 5  6  7  8]
      [ 7  8  9 10]]

     [[ 9 10 11 12]
      [11 12 13 14]]]

    (1) mode-1 tensor unfolding:
    [[ 1  3  2  4  3  5  4  6]
     [ 5  7  6  8  7  9  8 10]
     [ 9 11 10 12 11 13 12 14]]

    (2) mode-2 tensor unfolding:
    [[ 1  5  9  2  6 10  3  7 11  4  8 12]
     [ 3  7 11  4  8 12  5  9 13  6 10 14]]

    (3) mode-3 tensor unfolding:
    [[ 1  5  9  3  7 11]
     [ 2  6 10  4  8 12]
     [ 3  7 11  5  9 13]
     [ 4  8 12  6 10 14]]

```python
def mat2ten(mat, tensor_size, mode):
    index = list()
    index.append(mode)
    for i in range(tensor_size.shape[0]):
        if i != mode:
            index.append(i)
    return np.moveaxis(np.reshape(mat, list(tensor_size[index]), order = 'F'), 0, mode)
```

### Singular Value Thresholding (SVT) for TNN

```python
def svt_tnn(mat, alpha, rho, theta):
    """This is a Numpy dependent singular value thresholding (SVT) process."""
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    vec = s.copy()
    vec[theta :] = s[theta :] - alpha / rho
    vec[vec < 0] = 0
    return np.matmul(np.matmul(u, np.diag(vec)), v)
```

**Understanding these codes**:

- **`line 1`**: Necessary inputs including any input matrix $\boldsymbol{X}$, weight of Truncated Nuclear Norm (TNN) regularization $\alpha$, learning rate $\rho$, and positive integer number $\theta$ for nuclear norm truncation.

- **`line 2`**: Compute the Singular Value Decomposition (SVD) for any matrix $\boldsymbol{X}$ with `numpy.linalg.svd` (i.e., SVD function in `Numpy`'s linear algebra package).

- **`line 3-5`**: Truncate singular values $\sigma_{\theta+1},...$ with the following rule:

\begin{equation}
\sigma_{i}=\left[\sigma_{i}(\boldsymbol{X})-\frac{\alpha}{\rho}\right]_{+}.
\end{equation}

- **`line 6`**: Return the resulted matrix.

**Potential alternative for this**:

This is a competitively efficient algorithm for implementing SVT-TNN.

```python
def svt_tnn(mat, alpha, rho, theta):
    tau = alpha / rho
    [m, n] = mat.shape
    if 2 * m < n:
        u, s, v = np.linalg.svd(mat @ mat.T, full_matrices = 0)
        s = np.sqrt(s)
        idx = np.sum(s > tau)
        mid = np.zeros(idx)
        mid[:theta] = 1
        mid[theta:idx] = (s[theta:idx] - tau) / s[theta:idx]
        return (u[:, :idx] @ np.diag(mid)) @ (u[:, :idx].T @ mat)
    elif m > 2 * n:
        return svt_tnn(mat.T, tau, theta).T
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    idx = np.sum(s > tau)
    vec = s[:idx].copy()
    vec[theta:idx] = s[theta:idx] - tau
    return u[:, :idx] @ np.diag(vec) @ v[:idx, :]
```

compute_mape: Compute the value of Mean Absolute Percentage Error (MAPE).

compute_rmse: Compute the value of Root Mean Square Error (RMSE).

> Note that $$\mathrm{MAPE}=\frac{1}{n} \sum_{i=1}^{n} \frac{\left|y_{i}-\hat{y}_{i}\right|}{y_{i}} \times 100, \quad\mathrm{RMSE}=\sqrt{\frac{1}{n} \sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}},$$ where $n$ is the total number of estimated values, and $y_i$ and $\hat{y}_i$ are the actual value and its estimation, respectively.

```python
def compute_rmse(var, var_hat):
    return np.sqrt(np.sum((var - var_hat) ** 2) / var.shape[0])
```

```python
def compute_mape(var, var_hat):
    return np.sum(np.abs(var - var_hat) / var) / var.shape[0]
```

### Define LRTC-TNN Function with `Numpy`

```python
def LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter):
    """Low-Rank Tenor Completion with Truncated Nuclear Norm, LRTC-TNN."""

    dim = np.array(sparse_tensor.shape)
    pos_missing = np.where(sparse_tensor == 0)
    pos_test = np.where((dense_tensor != 0) & (sparse_tensor == 0))

    X = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{X}}
    T = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{T}}
    Z = sparse_tensor.copy()
    last_tensor = sparse_tensor.copy()
    snorm = np.sqrt(np.sum(sparse_tensor ** 2))
    it = 0
    while True:
        rho = min(rho * 1.05, 1e5)
        for k in range(len(dim)):
            X[k] = mat2ten(svt_tnn(ten2mat(Z - T[k] / rho, k), alpha[k], rho, np.int(np.ceil(theta * dim[k]))), dim, k)
        Z[pos_missing] = np.mean(X + T / rho, axis = 0)[pos_missing]
        T = T + rho * (X - np.broadcast_to(Z, np.insert(dim, 0, len(dim))))
        tensor_hat = np.einsum('k, kmnt -> mnt', alpha, X)
        tol = np.sqrt(np.sum((tensor_hat - last_tensor) ** 2)) / snorm
        last_tensor = tensor_hat.copy()
        it += 1
        if (it + 1) % 50 == 0:
            print('Iter: {}'.format(it + 1))
            print('RMSE: {:.6}'.format(compute_rmse(dense_tensor[pos_test], tensor_hat[pos_test])))
            print()
        if (tol < epsilon) or (it >= maxiter):
            break

    print('Imputation MAPE: {:.6}'.format(compute_mape(dense_tensor[pos_test], tensor_hat[pos_test])))
    print('Imputation RMSE: {:.6}'.format(compute_rmse(dense_tensor[pos_test], tensor_hat[pos_test])))
    print()

    return tensor_hat
```

**Understanding these codes**:

- **`line 18-19`**: Update $\boldsymbol{\mathcal{Z}}_{k}^{l+1},k=1,2,3$.

- **`line 20-22`**: Update $\boldsymbol{\mathcal{X}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{X}}_{k}^{l+1}=\mathcal{P}_{\Omega}(\boldsymbol{\mathcal{Y}})+\mathcal{P}_{\Omega}^{\perp}\left(\boldsymbol{\mathcal{Z}}_{k}^{l+1}-\frac{1}{\rho}\boldsymbol{\mathcal{T}}_{k}^{l}\right),k=1,2,3.
\end{equation}

- **`line 23`**: Update $\boldsymbol{\mathcal{T}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{T}}_{k}^{l+1}=\boldsymbol{\mathcal{T}}_{k}^{l}+\rho_k\left(\boldsymbol{\mathcal{X}}_{k}^{l+1}-\boldsymbol{\mathcal{Z}}_{k}^{l+1}\right).
\end{equation}

### Guangzhou urban traffic speed data set

```python
import numpy as np
import time
import scipy.io

## 30% RM
r = 0.3
print('Missing rate = {}'.format(r))
missing_rate = r

## Random Missing (RM)
dense_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape
np.random.seed(1000)
sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim2, dim3) + 0.5 - missing_rate)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.25
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()

## 70% RM
r = 0.7
print('Missing rate = {}'.format(r))
missing_rate = r

## Random Missing (RM)
dense_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape
np.random.seed(1000)
sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim2, dim3) + 0.5 - missing_rate)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.2
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()

## 90% RM
r = 0.9
print('Missing rate = {}'.format(r))
missing_rate = r

## Random Missing (RM)
dense_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape
np.random.seed(1000)
sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim2, dim3) + 0.5 - missing_rate)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-4
theta = 0.1
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 5.51797

    Iter: 100
    RMSE: 3.00147

    Imputation MAPE: 0.070093
    Imputation RMSE: 2.99615

    Running time: 115 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 5.32424

    Iter: 100
    RMSE: 3.58887

    Imputation MAPE: 0.0836949
    Imputation RMSE: 3.57911

    Running time: 151 seconds

    Missing rate = 0.9
    Iter: 50
    RMSE: 4.05595

    Iter: 100
    RMSE: 4.05

    Imputation MAPE: 0.0950771
    Imputation RMSE: 4.05026

    Running time: 167 seconds

```python
import numpy as np
import time
import scipy.io

for r in [0.3, 0.7]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    ## Non-random Missing (NM)
    dense_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
    dim1, dim2, dim3 = dense_tensor.shape
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim3) + 0.5 - missing_rate)[:, None, :]

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.05
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 5.06925

    Iter: 100
    RMSE: 4.08494

    Imputation MAPE: 0.0965249
    Imputation RMSE: 4.08516

    Running time: 152 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 5.14929

    Iter: 100
    RMSE: 4.30086

    Imputation MAPE: 0.101477
    Imputation RMSE: 4.30102

    Running time: 123 seconds

```python
import numpy as np
import time
import scipy.io
np.random.seed(1000)

missing_rate = 0.3

## Block-out Missing (BM)
dense_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape

dim_time = dim2 * dim3
block_window = 6
vec = np.random.rand(int(dim_time / block_window))
temp = np.array([vec] * block_window)
vec = temp.reshape([dim2 * dim3], order = 'F')

sparse_tensor = mat2ten(ten2mat(dense_tensor, 0) * np.round(vec + 0.5 - missing_rate)[None, :], np.array([dim1, dim2, dim3]), 0)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()
```

    Iter: 50
    RMSE: 5.27782

    Iter: 100
    RMSE: 3.97509

    Imputation MAPE: 0.094045
    Imputation RMSE: 3.96451

    Running time: 147 seconds

### Hangzhou metro passenger flow data set

```python
import numpy as np
import time
import scipy.io

for r in [0.3, 0.7, 0.9]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    ## Random Missing (RM)
    dense_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
    dim1, dim2, dim3 = dense_tensor.shape
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim2, dim3) + 0.5 - missing_rate)

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.10
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 25.2411

    Iter: 100
    RMSE: 24.944

    Imputation MAPE: 0.186277
    Imputation RMSE: 24.9491

    Running time: 15 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 29.1678

    Iter: 100
    RMSE: 29.5341

    Imputation MAPE: 0.201632
    Imputation RMSE: 29.5459

    Running time: 19 seconds

    Missing rate = 0.9
    Iter: 50
    RMSE: 37.7603

    Iter: 100
    RMSE: 38.0407

    Imputation MAPE: 0.229517
    Imputation RMSE: 38.0515

    Running time: 23 seconds

```python
import numpy as np
import time
import scipy.io

for r in [0.3, 0.7]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    ## Non-random Missing (NM)
    dense_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
    dim1, dim2, dim3 = dense_tensor.shape
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim3) + 0.5 - missing_rate)[:, None, :]

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.10
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 49.406

    Iter: 100
    RMSE: 47.6061

    Imputation MAPE: 0.193862
    Imputation RMSE: 47.5992

    Running time: 21 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 42.8749

    Iter: 100
    RMSE: 41.8305

    Imputation MAPE: 0.226381
    Imputation RMSE: 41.8327

    Running time: 14 seconds

```python
import numpy as np
import time
import scipy.io
np.random.seed(1000)

missing_rate = 0.3

## Block-out Missing (BM)
dense_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')['tensor'].transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape

dim_time = dim2 * dim3
block_window = 6
vec = np.random.rand(int(dim_time / block_window))
temp = np.array([vec] * block_window)
vec = temp.reshape([dim2 * dim3], order = 'F')

sparse_tensor = mat2ten(ten2mat(dense_tensor, 0) * np.round(vec + 0.5 - missing_rate)[None, :], np.array([dim1, dim2, dim3]), 0)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()
```

    Iter: 50
    RMSE: 28.8777

    Iter: 100
    RMSE: 29.2744

    Imputation MAPE: 0.21374
    Imputation RMSE: 29.2814

    Running time: 20 seconds

### Seattle freeway traffic speed data set

```python
import numpy as np
import pandas as pd
import time
import scipy.io

for r in [0.3, 0.7, 0.9]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    ## Random missing (RM)
    dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0).values
    dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288]).transpose(0, 2, 1)
    dim1, dim2, dim3 = dense_tensor.shape
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim2, dim3) + 0.5 - missing_rate)

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.30
    if r > 0.8:
        rho = 5e-5
        theta = 0.10
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 5.86304

    Iter: 100
    RMSE: 3.1126

    Imputation MAPE: 0.0480528
    Imputation RMSE: 3.10965

    Running time: 263 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 5.89289

    Iter: 100
    RMSE: 3.79913

    Imputation MAPE: 0.0613629
    Imputation RMSE: 3.7982

    Running time: 228 seconds

    Missing rate = 0.9
    Iter: 50
    RMSE: 4.86097

    Iter: 100
    RMSE: 4.81283

    Imputation MAPE: 0.081929
    Imputation RMSE: 4.81347

    Running time: 260 seconds

```python
import numpy as np
import pandas as pd
import time
import scipy.io

for r in [0.3, 0.7]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    ## Non-random Missing (NM)
    dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0).values
    dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288]).transpose(0, 2, 1)
    dim1, dim2, dim3 = dense_tensor.shape
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim3) + 0.5 - missing_rate)[:, None, :]

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.05
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 5.10438

    Iter: 100
    RMSE: 4.43494

    Imputation MAPE: 0.074171
    Imputation RMSE: 4.43533

    Running time: 249 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 6.14372

    Iter: 100
    RMSE: 5.4049

    Imputation MAPE: 0.0928467
    Imputation RMSE: 5.40446

    Running time: 239 seconds

```python
import numpy as np
import scipy.io
np.random.seed(1000)

missing_rate = 0.3

## Block-out Missing (BM)
dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0).values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288]).transpose(0, 2, 1)
dim1, dim2, dim3 = dense_tensor.shape
block_window = 12
vec = np.random.rand(int(dim2 * dim3 / block_window))
temp = np.array([vec] * block_window)
vec = temp.reshape([dim2 * dim3], order = 'F')
sparse_tensor = mat2ten(dense_mat * np.round(vec + 0.5 - missing_rate)[None, :], np.array([dim1, dim2, dim3]), 0)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()
```

    Iter: 50
    RMSE: 6.60377

    Iter: 100
    RMSE: 5.69047

    Imputation MAPE: 0.0981021
    Imputation RMSE: 5.69791

    Running time: 220 seconds

### Portland highway traffic volume data set

```python
import numpy as np
import pandas as pd
import time
import scipy.io

for r in [0.3, 0.7, 0.9]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    # Random Missing (RM)
    dense_mat = np.load('../datasets/Portland-data-set/volume.npy')
    dim1, dim2 = dense_mat.shape
    dim = np.array([dim1, 96, 31])
    dense_tensor = mat2ten(dense_mat, dim, 0)
    np.random.seed(1000)
    sparse_tensor = mat2ten(dense_mat * np.round(np.random.rand(dim1, dim2) + 0.5 - missing_rate), dim, 0)

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.10
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 16.6038

    Iter: 100
    RMSE: 15.6579

    Imputation MAPE: 0.172064
    Imputation RMSE: 15.6594

    Running time: 693 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 19.5941

    Iter: 100
    RMSE: 19.2446

    Imputation MAPE: 0.204501
    Imputation RMSE: 19.2494

    Running time: 698 seconds

    Missing rate = 0.9
    Iter: 50
    RMSE: 23.7422

    Iter: 100
    RMSE: 24.1852

    Imputation MAPE: 0.244044
    Imputation RMSE: 24.1911

    Running time: 646 seconds

```python
import numpy as np
import pandas as pd
import time
import scipy.io

for r in [0.3, 0.7]:
    print('Missing rate = {}'.format(r))
    missing_rate = r

    # Non-random Missing (NM)
    dense_mat = np.load('../datasets/Portland-data-set/volume.npy')
    dim1, dim2 = dense_mat.shape
    dim = np.array([dim1, 96, 31])
    dense_tensor = mat2ten(dense_mat, dim, 0)
    np.random.seed(1000)
    sparse_tensor = dense_tensor * np.round(np.random.rand(dim1, dim[2]) + 0.5 - missing_rate)[:, None, :]

    start = time.time()
    alpha = np.ones(3) / 3
    rho = 1e-5
    theta = 0.10
    epsilon = 1e-4
    maxiter = 100
    LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
    end = time.time()
    print('Running time: %d seconds'%(end - start))
    print()
```

    Missing rate = 0.3
    Iter: 50
    RMSE: 19.2963

    Iter: 100
    RMSE: 18.5759

    Imputation MAPE: 0.193184
    Imputation RMSE: 18.5804

    Running time: 722 seconds

    Missing rate = 0.7
    Iter: 50
    RMSE: 38.5421

    Iter: 100
    RMSE: 38.2218

    Imputation MAPE: 0.269787
    Imputation RMSE: 38.2252

    Running time: 656 seconds

```python
import numpy as np
import scipy.io
np.random.seed(1000)

missing_rate = 0.3

## Block-out Missing (BM)
dense_mat = np.load('../datasets/Portland-data-set/volume.npy')
dim1, dim2 = dense_mat.shape
dim = np.array([dim1, 96, 31])
dense_tensor = mat2ten(dense_mat, dim, 0)
block_window = 4
vec = np.random.rand(int(dim2 / block_window))
temp = np.array([vec] * block_window)
vec = temp.reshape([dim2], order = 'F')
sparse_tensor = mat2ten(dense_mat * np.round(vec + 0.5 - missing_rate)[None, :], dim, 0)

start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 100
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
print()
```

    Iter: 50
    RMSE: 23.3329

    Iter: 100
    RMSE: 23.0543

    Imputation MAPE: 0.230816
    Imputation RMSE: 23.0534

    Running time: 697 seconds

### License

This work is released under the MIT license.