## THIS IS JUST BRAINSTORMING

### Copyright © 2020 Jacob Valdez

under the [MIT License](https://opensource.org/licenses/MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# Nodal Reinforcement Learning

Abstract and pictures/videos *en simulo*. Maybe also simplified diagrams of AGI node and network.

# Introduction

AGI-0 rests on the hypotheses that:
- encoding the principles of [free energy minimization](https://en.wikipedia.org/wiki/Free_energy_principle) in a multilayer hidden Markov chain prediction unit will enable unsupervised, open-ended learning
- competitive ensembles of prediction units will succeed in overcoming the [no free lunch theorom](https://en.wikipedia.org/wiki/No_free_lunch_theorem)
- ~~supervised multiagent simulations will enable learning mirroring features that transfer successfully to humans~~
- embedded natural language self examination objectives will automatically align agent behavior with encoded human values from environment reinforcement without any supervised reward administration
- **the realization of these hypotheses will achieve artificial general intelligence**

Discuss my theory of hierarchial conditioning here

Each child only provides additional conditioning context but is not required for a module to produce an output. (Kind of like how [conditioning gpt-2 with a longer input sentence gives you more control over its output](https://colab.research.google.com/drive/12_EzHGVE8ZnJ-MA9W_8tQuP6xHpWb1X3?usp=sharing))

# Methodology

$\newcommand{\ab}{\mathcal{A}}$
$\newcommand{\pr}{\mathcal{P}}$
$\newcommand{\pol}{\mathcal{\pi}}$
$\newcommand{\o}[1][t]{{o}_{#1}}$
$\newcommand{\otrue}[1][t]{{o}_{#1}^{true}}$
$\newcommand{\opast}[1][t-T_o:t]{{o}_{#1}}$
$\newcommand{\otruepast}[1][t-T_o:t]{{o}_{#1}^{true}}$
$\newcommand{\s}[1][t]{{s}_{#1}}$
$\newcommand{\spred}[1][t+1]{{s}_{#1}^{pred}}$
$\newcommand{\oimagpast}[1][\tau-T_o:\tau]{{o}_{#1}^{imag}}$
$\newcommand{\oimag}[1][\tau]{{o}_{#1}^{imag}}$
$\newcommand{\simag}[1][\tau]{{s}_{#1}^{imag}}$
$\newcommand{\spredimag}[1][\tau+p]{{s}_{#1}^{pred,imag}}$
$\newcommand{\stargetimag}[1][\tau+p]{{s}_{#1}^{*,imag}}$
$\newcommand{\starget}[1][t+1]{{s}_{#1}^*}$
$\newcommand{\otarget}[1][t+1]{{o}_{#1}^*}$
$\newcommand{\sblind}[1][\tau+p]{{s}_{#1}^{blind}}$
$\newcommand{\pe}[1][t]{pe_{#1}}$
$\newcommand{\n}[1][i]{n_{#1}}$
$\newcommand{\nparent}{\mathcal{N_{pa}}}$
$\newcommand{\nchildren}{\mathcal{N_{ch}}}$
$\newcommand{\U}{\mathcal{U}}$
$\newcommand{\Ex}[1][ ]{ \underset{#1}{ \mathbb{E} } }$
$ \newcommand{\vect}[1]{\boldsymbol{#1}} $
$\def\falert{f_{alert}}$
$\def\NN{N\kern-0.2em N}$
$\def\DS{\mathcal{D}}$
$\def\IE{\mathcal{E}_I}$
$\def\CER{\mathcal{C}}$
$\newcommand{norm}[1]{\bigl\lvert\bigl\rvert{#1}\bigr\lvert\bigr\rvert}$

## Architecture

Nodal-RL operates by minimizing predictive entropy. It does this by decomposing the state space with a [structured probabblistic model](https://en.wikipedia.org/wiki/Graphical_model) into sensory, information, and actuator nodes. Each *information node* on the graph only models a subset of all information. The directed and potentially cyclic model $\langle\mathcal{N}\kern-0.15em,\mathcal{E}\rangle$ may look like:

Above, among many other partial models, is modeled the information that $\n[tokens]$ and $\n[visual\ features]$ supply to $\n[nat\ lang\ I]$, the information $\n[nat\ lang\ I]$ supplies to $\n[nat\ lang\ II]$, and the partial information $\n[nat\ lang\ II]$ adds to a composite understanding of the world. This composite understanding is captured in the state variable $\n[comp].\s[ ]$. Instead of producing its state from directly observing sensory data, $\n[comp]$ only observes its parent nodes' states $\n[comp].\o[ ]=\{\n[nat\ lang\ II].\s[ ],\n[motion\ prim].\s[ ],\n[visual\ features].\s[ ]\}$.

An information node $\n$ attempts to build an internal model of its observed variables $P(\o[t+p]|\opast)$. It does this by first *abstracting* the information from its observable data trajectory into an internal *state* $\s=\ab(\opast)$. It then attempts to model the trajectory of that internal state with a *predictor* function $\spred=\pr(\s)$. Predictive error $D_{KL}(\spred,\s[t+1])$ and predictive uncertainty $H(\spred)$ then become the objectives to cumulatively minimize.

The *policy* $\pol$ specfies *target observations* $\otarget$ that approach these objectives. Target observations belonging to a child layer influence their parent's *target state* (e.g.: $\n[nat\ lang\ II].\otarget \stackrel{affects}{\rightarrow} \n[nat\ lang\ I].\starget$). Since a node may have multiple children, the target state is detirmined by a $\n[ch].w_{\mathcal{Pa}_i}$ weighted mean of the node children's target observations $\{\n[ch].\otarget\ |\ \forall\n[ch]\in\n\smash{.}\nchildren\}$ concatenated with the node's own predicted state $\n\smash{.}\spred$. Additionally, proabblistic certainty weighting, ($\CER(x)=0=$ very indecisive) weakens the influence of vague target distributions.

$\text{weighted ave}(\{\langle w_{pred},\spred\rangle\}\cup\{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\otarget), \n[ch].\otarget\rangle\ |\ \forall\n[ch]\in\nchildren\})$

Nodes do not compute updates simultaneously or even at the same frequency though. They are individually updated at periods of $\n\smash{.}p=\n\smash{.}f^{-1}$. As a result, $\n[ch].\otarget[t+\n\smash{.}p]$ may not exist. To avoid blocking, child node target observations are instead queried for the most recent existing record $\n[ch].\otarget[\tau]\ |\ \exists\n[ch].\otarget[\tau];\ \tau=\max [t-N_o:t+p];$ An exponential recency decay $e^{-\n[ch].\lambda(t+p-\tau)}$ additionally weights the target observation. The target state is thus:

$\starget[t+p]=\text{weighted ave}(\{\langle w_{pred},\spred[t+p]\rangle\} \cup \{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\otarget[\tau])\times e^{-\n[ch].\lambda(t+p-\tau)}, \n[ch].\otarget[\tau]\rangle\ |\ \exists\n[ch].\otarget[\tau];\ \tau=\max [t-N_o:t+p];\ \forall\n[ch]\in\nchildren\})$

Target state in turn biases the target observation via the policy $\otarget=\pol(\s,\starget)$. Since the target state may be a weighted mean of several stochastic variables, child target observations with a high entropy will have less influence in the weighted mean distribution-even if its weight is large. The more certain (lower entropy) a target observation is, the greator influence it will have on its parent target state.

This makes sibling nodes form an implicit team. One node $\n[abs\ per\ I]$ may have a very clear understanding of the dynamics of one subspace of the observation space $\mathcal{O}_i\subset\mathcal{O}$, hence producing low entropy predictions and strong opinions on what $\otarget[t+p]$ should be. At the same time, its sibling $\n[abs\ per\ II]$ may not have knowledge of that domain and therefore produce highly irrational, entropic observation targets. The result of averaging the two nodes' target observations is a distribution with relatively little influence by $\n[abs\ per\ II]$. Togethor, nodes form a network of experts each specializing in unique domains to give the agent open-ended intelligence.

## Stochastic Models

To allow information nodes to produce multimodal and multivariate observation and state distributions, the following reparametrization trick is employed to layer input concatenated with a multidimensional uniform distribution:

$\vect{y}=\sigma(W[\vect{x};\vect{\zeta}]+b)\qquad\vect{\zeta}\thicksim\{\mathcal{U(0,1)}\}^{N_\zeta}$

Since this design employs continuous distributions, certainty $\CER$ is measured by the relative entropy of the uniform distribution with respect to a target distribution $P$:

$\CER(P)=-D_{KL}(\mathcal{U}(a,b)||P)$

This above definition demands a definition of activation bounds $a$ and $b$ for each distribution. These are regularized distribution-wise by:

$\loss_{bounds}(x)=\begin{cases}f(a-x)\qquad&x\lt a\\0\qquad& a\le x\le b\\f(x-b)\qquad& x\gt b\end{cases}$

with $f(x)=\{x,x^2,x^n,e^x,\dots\}$ being any differentiable monotonically increasing function on $(0,\infty)$.

The certainty of various distributions zero as they are more unpredictable and increases to above one as they become more certain of the likelihood of a random variable.

## Dreaming

Not all observations collected online are equally relevant to future tasks. Additionally, when observations add little information to an already highly unpredictable state trajectory, it becomes pointless to informatively utilize the next state prediction. In these situations, informations nodes maintain productivity by *dreaming*.

Dreaming makes a shift from closed to open loop control. It involves relying less on observed data and more on anticipated observations. It mixes the true observation $\otrue$ with last timestep's target observation $\otarget[t]$ by a nonlinear function $\falert$ to produce the observed processed by all other functions:

$\o=\falert\otrue+(1-\falert)\otarget[t]$

where $\falert$ increases alertness when surprising, unexpected, or informative observations occur $\falert\propto I(\otrue)$ and decreases alertness when predictability is low $\falert\propto (H(\s[t-p]))^{-1}$. To ensure the transition to dreaming and back to alertness is not jittery, these factors are smoothed with a rolling average and transformed with $\tanh$:

$\overline{\underline{\textbf{Algorithm:}\ \text{Percieved Observation }\o}}$

$1.\ d_t^{ave}=d_t^{ave}+\alpha_{d,1}\frac{I(\otrue)}{H(\s[t-p])}$

$2.\ \falert=\tanh[\alpha_{d,2}(d_t^{ave}-\alpha_{d,3})]$

$3.\ \mathbf{return}\ \o=\falert\otrue+(1-\falert)\otarget[t]$

where $\alpha_{d,1}\lt1$ is small number that influences the rolling average latency, $\alpha_{d,2}\gt1$ is large and makes the transition from dream to alert state abrupt, and $\alpha_{d,3}$ is the minimum average ratio of information to entropy that must be provided by observations for $\n$ to pay attention to them.

Since dreamed state trajectories will generally have a larger entropy than states based on true observations, dreaming node target observations have a correspondingly high entropy and generally do not compete with target observations from sibling nodes that are not dreaming.

## Running

Information nodes join an asynchronously updating netowrk with sensory nodes and actuator nodes. Unless specified, variables are implied to belong to $\n$ ($\n.p=p$).

### Sensory Nodes

Sensory and simply publish states:

$\overline{\underline{\textbf{Algorithm:}\ \text{Sensory Node }\n{\ Update}}}$

$1.\ \s=\text{get sensory data}$

$6.\ \textbf{wait}\ p\ \text{timesteps}$

$3.\ \textbf{repeat}$

### Actuator Nodes

Actuator nodes attempts to fulfill their childrens' target observations. At the actuator level, distributions must finally be sampled into real numbers.

$\overline{\underline{\textbf{Algorithm:}\ \text{Actuator Node }\n{\ Update}}}$

$1.\ \starget[t+p]=\text{weighted ave}(\{\langle w_{pred},\spred[t+p]\rangle\} \cup \{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\otarget[\tau])\times e^{-\n[ch].\lambda(t+p-\tau)}, \n[ch].\otarget[\tau]\rangle\ |\ \exists\n[ch].\otarget[\tau];\ \tau=\max [t-N_o:t+p];\ \forall\n[ch]\in\nchildren\})$

$2.\ \text{execute } \overset{\small{N_s}}{\Ex}[\starget[t+p]] \quad\rhd\text{execute weighted target state sampling }N_s\text{ times}$

$3.\ \textbf{wait}\ p\ \text{timesteps}$

$4.\ \s[t+p]=\text{get actual result}$

$5.\ \textbf{repeat}$

### Information Nodes

Information nodes simulatenously engage in bottom-up processing and propagate top-down conditioning:

$\overline{\underline{\textbf{Algorithm:}\ \text{Information Node }\n{\ Update}}}$

$1.\ \mathbf{store}\ \otrue=\rm{concat}(\n[parent].\s|\forall\n[parent]\in\nparent)$

$2.\ d_t^{ave}=d_t^{ave}+\alpha_{d,1}\frac{I(\otrue)}{H(\s[t-p])}$

$3.\ \falert=\tanh[\alpha_{d,2}(d_t^{ave}-\alpha_{d,3})]$

$4.\ \o=\falert\otrue+(1-\falert)\otarget[t]$

$5.\ \s=\ab(\opast[t-N_o:t])$

$6.\ \spred[t+p]=\pr(\s)$

$7.\ \starget[t+p]=\text{weighted ave}(\{\langle w_{pred},\spred[t+p]\rangle\} \cup \{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\otarget[\tau])\times e^{-\n[ch].\lambda(t+p-\tau)}, \n[ch].\otarget[\tau]\rangle\ |\ \exists\n[ch].\otarget[\tau];\ \tau=\max [t-N_o:t+p];\ \forall\n[ch]\in\nchildren\})$

$8.\ \otarget[t+p]=\pol(\s,\starget[t+p])$

$9.\ \textbf{wait}\ p\ \text{timesteps}$

$10.\ \textbf{repeat}$

## Training

Training is performed online, on-policy, and on-predictor with new, prioritized, and imagined data. Minibatches are only one rollout sequence of observations ($N=1$) from every point in time in an episode to allow training large architectures and keep policies fresh. Dreaming provides novel observation trajectories for the abstractor and child information nodes to learn. However information nodes do not train their policy or predictor on its own dreamed trajectories ($\o=\otrue$ when optimizing).

### Learning the Policy

$\def\loss{\mathcal{L}}$
$\def\obj{\mathcal{J}}$
The policy aims to produce target observations $\otarget[t+p]=\pol(\s,\starget[t+p])$ which cumulatively:
- maximize target state achievement $D_{KL}\bigl(\starget[t+p]||\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)$
- maximize resulting state certainty $\CER\bigl(\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)$
- maximize informative observations targeted $\CER\bigl(\pol(\s,\starget[t+p])\bigr)$

These objectives are combined as:

$\loss_\pol=\underbrace{\beta_{\pol,D}\norm{\ln{D_{KL}\bigl(\starget[t+p]||\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)}}}_\text{target state achievement}-\underbrace{\beta_{\pol,\CER}\norm{\ln{\CER\bigl(\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)}}}_\text{resulting state certainty}-\underbrace{\beta_{\pol,I}\norm{\ln{\CER\bigl(\pol(\s,\starget[t+p])\bigr)}}}_\text{informative observations targeted}$

This attempts to make the policy descend the gradient of predictive error and entropy while attempting to please its children's desired target observations. That is, it should produce a target observation which will make the abstractor produce an abstraction $\ab([\opast;\pol(\s,\starget[t+p])])$ as close as possible to $\starget[t+p]$. The last objective is added because informative events often reduce cumulative predictive entropy. By searching out unexpected, *informative* events, an agent is able to learn a more robust internal model $\Pr(\o[t+p]|\opast)$.

### Learning the Predictor

Ideally, the predictor aims to model its observations perfectly $\spred[t+p]=\s[t+p]$. This means it should be able to maintain an accurate predicted state trajectory s.t. $\pr(\pr(\pr(\s)))\approxeq \s[t+3p]$. That requires maximizing predictive certainty $\CER(\pr(\s))$. Jointly, these objectives compose:

$\obj_\pr=\underbrace{\beta_{\pr,D}\norm{\ln{\bigl[D_{KL}(\pr(\s)||\s[t+p])\bigr]}}}_\text{accuracy} - \underbrace{\beta_{\pr,\CER}\norm{\ln{ \CER(\pr(\s))}}}_\text{certainty}$

Objectives are computed componentwise and only then normalized to encourage modeling only a few variables accurately rather than attempting to poorly model a joint distribution.

### Learning the Abstractor

In some information nodes, the abstractor may be frozen with pretrained weights. When trainable however, its optimization attempts to capture variables relevant to both the policy's and predictor's losses. Although dreaming may provide novel observation trajectories for the abstractor to represent only true observations are used to optimize $\ab$. However, child information nodes do optimize on the novel state trajectories generated by dreaming.

### Putting it all togethor

Optimization begins at each record in time of an information episoe $\IE=\langle\otruepast,\nchildren \text{(target observation sequences only)}\rangle$ by predicting the rollout for $T_{rollout}$ timesteps. Then cumulative policy and predictor losses are minimized by optimizing policy, predictor, and abstractor weights. Importantly, this optimization process takes place for nodes individually, allowing very large node architectures to integrate in an agent that no one processing unit could support.

$\begin{equation}\overline{\underline{\mathbf{Algorithm}:\ \text{Train Information Node }\n}} \\
1.\ \ \mathbf{for}\ t\in[\IE.T_{start}+p,\IE.T_{start}+2p,\dots,\IE.T_{end}]:\qquad\rhd\text{precompute }\s[ ],\spred[ ],\starget[ ],\otarget[ ] \\
2.\ \ \quad \s=\ab(\IE.\otruepast) \\
3.\ \ \quad \spred[t+p]=\pr(\s) \\
4.\ \ \quad \starget[t+p]=\text{weighted ave}(\{\langle w_{pred},\spred[t+p]\rangle\} \cup \\
\kern+7em\{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\otarget[\tau]) e^{-\n[ch].\lambda(t+p-\tau)}, \n[ch].\otarget[\tau]\rangle\ |\ \exists\n[ch].\otarget[\tau];\ \tau=\max [t-N_o:t+p];\ \forall\n[ch]\in\IE.\nchildren\}) \\
5.\ \ \quad\otarget[t+p]=\pol(\s,\starget[t+p]) \\
6.\ \ \mathbf{end\ for} \\
7.\ \ \mathbf{for}\ t\in[\IE.T_{start}+p,\IE.T_{start}+2p,\dots,\IE.T_{end}]: \\
8.\ \ \quad \loss_t=\underbrace{\beta_{\pol,D}\norm{\ln{D_{KL}\bigl(\starget[t+p]||\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)}}}_\text{target state achievement}-\underbrace{\beta_{\pol,\CER}\norm{\ln{\CER\bigl(\ab([\otruepast;\pol(\s,\starget[t+p])])\bigr)}}}_\text{resulting state certainty}-\underbrace{\beta_{\pol,I}\norm{\ln{\CER\bigl(\pol(\s,\starget[t+p])\bigr)}}}_\text{informative observations targeted} \\
9.\ \ \quad \tau=t \\
10.\ \quad \oimagpast=\otruepast \\
11.\ \quad \sblind[\tau]=\s \\
12.\ \quad \mathbf{while}\ \tau\le t+T_{rollout}\ \mathbf{and}\ \tau\le\IE.T_{end}-p:\qquad\rhd\text{compute rollout comparing against real data} \\
13.\ \qquad \simag=\ab(\oimagpast) \\
14.\ \qquad \spredimag[\tau+p]=\pr(\simag) \\
15.\ \qquad \stargetimag[\tau+p]=\text{weighted ave}(\{\langle w_{predimag},\spredimag[\tau+p]\rangle\} \cup \{\langle \n[ch].w_{\mathcal{Pa}_i}\CER(\n[ch].\o[\tau_{ch}])e^{-\n[ch].\lambda(\tau+p-\tau_{ch})}, \n[ch].\o[\tau_{ch}]\rangle \\
\kern+10em |\ \exists\n[ch].\o[\tau_{ch}];\ \tau_{ch}=\max [\tau-N_o:\tau+p];\ \forall\n[ch]\in\IE.\nchildren\}) \\
16.\ \qquad \oimag[\tau+p]=\pol(\simag,\stargetimag) \\
17.\ \qquad \sblind=\pr(\sblind[\tau]) \\
18.\ \qquad \loss_t=\loss_t+\gamma^{t-\tau}\Biggl[ \underbrace{\beta_{\pol,D}\norm{\ln{D_{KL}(\stargetimag[\tau]||\simag)}}}_\text{target state achievement} - \underbrace{\beta_{\pol,\CER}\norm{\ln{\CER(\simag)}}}_\text{resulting state certainty} - \underbrace{\beta_{\pol,I}\norm{\ln{\CER(\oimag[\tau+p])}}}_\text{informative observations targeted} \\
\kern+6em + \underbrace{\beta_{\pr,D}\norm{\ln{\bigl[D_{KL}(\sblind||\s[t+p])}}}_\text{blind prediction accuracy} - \underbrace{\beta_{\pr,\CER}\norm{\ln{ \CER(\sblind)}}}_\text{blind prediction certainty}\Biggr]\\
19.\ \qquad \tau = \tau + p \\
20.\ \ \quad \mathbf{end\ while} \\
21.\ \quad \underset{\theta_\ab,\theta_\pr,\theta_\pol}\min\loss_{t}-R_t(s)\qquad\rhd\text{minimize discounted sum loss and reward for timestep }t \\
22.\ \ \mathbf{end\ for} \end{equation}$

To minimize uncertainty, $\beta_{\pol,\CER}>1$, $\beta_{\pr,\CER}>1$. $R_t(s)$ encompasses any supervised rewards.

## Knowledge Transfer and Swarm Intelligence

AGI-0's node-based architecture allows different networks to share underlying nodes and individual nodes to transfer in knowledge from other neural networks. For example, a software automation agent and a writing agent might benefit by sharing nodes that process natural language. Some natural language processing nodes might also utilize existing language models like gpt for their abstractor and a conditional language modeling head for the policy. Nodes do not belong to any one agent but instead a *swarm cloud* which trains and serves individual nodes. *Information episodes* $\IE$ are collected from nodes after the agent's episode completes. This allows agents to synergize each other's heterogeneous architectures.

In the above comparison between a language modeling transformer and an information node, note that the predictor must be trained on supervised data. This could be implimented by initializing or even freezing the abstractor's weights with the transformer network, training the predictor to predict embedding transitions, and forcing the policy to produce the required next token from that predicted embedding (although top-down influence can change network output).

# Implementation

## Nodes

Since nodes are asychronous, they can be trained or deployed on different underlying architectures without significant issues resulting from latency. Following is a generic node class and then sensory, actuator, and information node implimentations.

**ACTUALLY, EMBED TO FILE IN GITHUB**

```python
class Node:

    self.children

    self.p

    self.o
    self.s

    def update():
        pass

```

### Sensory Node

```python
class Sensory_Node(node):

    def update():
        pass
```

### Actuator Node

```python
class Actuator_Node(node):

    def update():
        pass
```

### Information Node

```python
class Information_Episode:

```

```python
class Information_Node(node):

    self.parents

    self.otarget

    self.hparams

    def update():
        pass

    def compile_information_episode() -> Information_Episode:
        pass

    def train(information_episode: Information_Episode):
        pass
```

## Network

Brief overview of what's happening (maybe remove)

```python

```

# Experiments

Introduction

Observing Nodal RL in
- different environments
- different agents
- architecture deviations
- unsupervised/guided exploration

The following experiment parameters are explored:

### Generic Information Nodes:

- Dense Information Node
    - proof of concept node
    - dense/specialized abstractor
    - dense predictor
    - dense policy

- Sparse Information Node
    - sparse variational autoencoders ensembles competing to represent $\ab$ by lowest representation entropy
    - sparse transformer $\pr$
    - optionally penalizing state certainty decrease and rewarding entropic actions
    - directly minimizing entropy of binary valued sparse state vector quantization with annealing self organizing maps

- Energy-based Information Node
    - energy-based $\ab$ and $\pr$
    - $pol$ analagous to 'energy descent with a bias'

### Specialized Information Nodes

- language modeling
- machine vision
- audio

### Substructures

- Homogeneous Ensembles
- Heterogeneous Ensembles

### Training

- in-context supervised training and zero, one, and few shot test
- penalizing activation
- maximize $ln{[\CER(\s[t+p]) - \CER(\s)]}$

### agents/environments

- kinistetic
    - one legged hopper
    - spider
    - humanoid

- multimedia modeling
    - image
    - video
    - audio

- board games
    - checkers
    - chess
    - go

- software agent
    - bash REPL
    - python REPL
        - autoML
        - selfML

- Multi-Organism Systems
    - chatbots

- MALMO
    - single agent
    - multi agent (self-play)
    - multi-agent (human-play)

- Second Life

- engineering design

- more to come

### high-level conditioning
- none (open ended learning)
- RL agent
- realtime human control

Individual experiments are found in hyperlinked notebooks

# Results

# Analysis

# Discussion

# Conclusion

General conclusion here

## Questions

- **How generalized can Nodel RL agents become?** Will agent performance continue improving as it acquires skills specialized for increasingly diverse domains? Can a humanoid robot controlled by Nodal RL perform reliably in the general human activity domain?
- **Can an [imitator-imitator](https://arxiv.org/abs/1912.02875) framework be used to condition Nodel-RL agent behavior?**
- **How could transformer neural networks learn when to dream ($\NN_{transformer}(\opast)$) instead of employing a blind rolling average ($\text{Percieved Observation }\o$)?**
- **Can humans learn to dynamically condition agents to control their behavior?** Since high level conditioning data only provides a general handle on agent behavior, would a user commanding the agent in realtime be able to adjust his or her commands dynamically to more reliably achieve high level targets?
- **Will centralized shared-parameter multiagent training efficently improve performance?** Although parameter sharing in reinforcement learning [generally accelerates learning in MARL](https://arxiv.org/abs/2005.13625), is it worth the cost to centralize training these huge architectures?

# References

 - @book{Goodfellow-et-al-2016,
    title={Deep Learning},
    author={Ian Goodfellow and Yoshua Bengio and Aaron Courville},
    publisher={MIT Press},
    note={\url{http://www.deeplearningbook.org}},
    year={2016}
}

# Appendix


% PACKAGES INCLUDED HERE
% DO NOT NEED TO CHANGE
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}% AUTHOR NAMES AND TITLE GOES HERE
\title{Paper Title*\\
{\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and
should not be used}
\thanks{Identify applicable funding agency here. If none, delete this.}
}

\author{\IEEEauthorblockN{1\textsuperscript{st} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
\and
\IEEEauthorblockN{2\textsuperscript{nd} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
\and
\IEEEauthorblockN{3\textsuperscript{rd} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
\and
\IEEEauthorblockN{4\textsuperscript{th} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
\and
\IEEEauthorblockN{5\textsuperscript{th} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
\and
\IEEEauthorblockN{6\textsuperscript{th} Given Name Surname}
\IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
\textit{name of organization (of Aff.)}\\
City, Country \\
email address}
}

\maketitle% ABSTRACT AND KEYWORDS
\begin{abstract}
This document is a model and instructions for \LaTeX.
This and the IEEEtran.cls file define the components of your paper [title, text, heads, etc.]. *CRITICAL: Do Not Use Symbols, Special Characters, Footnotes,
or Math in Paper Title or Abstract.
\end{abstract}

\begin{IEEEkeywords}
component, formatting, style, styling, insert
\end{IEEEkeywords}% SECTIONS
\section{Introduction}
This document is a model and instructions for \LaTeX.
Please observe the conference page limits.

\section{Ease of Use}

\subsection{Maintaining the Integrity of the Specifications}

The IEEEtran class file is used to format your paper and style the text. All margins,
column widths, line spaces, and text fonts are prescribed; please do not
alter them. You may note peculiarities. For example, the head margin
measures proportionately more than is customary. This measurement
and others are deliberate, using specifications that anticipate your paper
as one part of the entire proceedings, and not as an independent document.
Please do not revise any of the current designations.

\section{Prepare Your Paper Before Styling}
Before you begin to format your paper, first write and save the content as a
separate text file. Complete all content and organizational editing before
formatting. Please note sections \ref{AA}--\ref{SCM} below for more information on
proofreading, spelling and grammar.

Keep your text and graphic files separate until after the text has been
formatted and styled. Do not number text heads---{\LaTeX} will do that
for you.

\subsection{Abbreviations and Acronyms}\label{AA}
Define abbreviations and acronyms the first time they are used in the text,
even after they have been defined in the abstract. Abbreviations such as
IEEE, SI, MKS, CGS, ac, dc, and rms do not have to be defined. Do not use
abbreviations in the title or heads unless they are unavoidable.

\subsection{Units}
\begin{itemize}
\item Use either SI (MKS) or CGS as primary units. (SI units are encouraged.) English units may be used as secondary units (in parentheses). An exception would be the use of English units as identifiers in trade, such as ``3.5-inch disk drive''.
\item Avoid combining SI and CGS units, such as current in amperes and magnetic field in oersteds. This often leads to confusion because equations do not balance dimensionally. If you must use mixed units, clearly state the units for each quantity that you use in an equation.
\item Do not mix complete spellings and abbreviations of units: ``Wb/m\textsuperscript{2}'' or ``webers per square meter'', not ``webers/m\textsuperscript{2}''. Spell out units when they appear in text: ``. . . a few henries'', not ``. . . a few H''.
\item Use a zero before decimal points: ``0.25'', not ``.25''. Use ``cm\textsuperscript{3}'', not ``cc''.)
\end{itemize}

\subsection{Equations}
Number equations consecutively. To make your
equations more compact, you may use the solidus (~/~), the exp function, or
appropriate exponents. Italicize Roman symbols for quantities and variables,
but not Greek symbols. Use a long dash rather than a hyphen for a minus
sign. Punctuate equations with commas or periods when they are part of a
sentence, as in:
\begin{equation}
a+b=\gamma\label{eq}
\end{equation}

Be sure that the
symbols in your equation have been defined before or immediately following
the equation. Use ``\eqref{eq}'', not ``Eq.~\eqref{eq}'' or ``equation \eqref{eq}'', except at
the beginning of a sentence: ``Equation \eqref{eq} is . . .''

\subsection{\LaTeX-Specific Advice}

Please use ``soft'' (e.g., \verb|\eqref{Eq}|) cross references instead
of ``hard'' references (e.g., \verb|(1)|). That will make it possible
to combine sections, add equations, or change the order of figures or
citations without having to go through the file line by line.

Please don't use the \verb|{eqnarray}| equation environment. Use
\verb|{align}| or \verb|{IEEEeqnarray}| instead. The \verb|{eqnarray}|
environment leaves unsightly spaces around relation symbols.

Please note that the \verb|{subequations}| environment in {\LaTeX}
will increment the main equation counter even when there are no
equation numbers displayed. If you forget that, you might write an
article in which the equation numbers skip from (17) to (20), causing
the copy editors to wonder if you've discovered a new method of
counting.

{\BibTeX} does not work by magic. It doesn't get the bibliographic
data from thin air but from .bib files. If you use {\BibTeX} to produce a
bibliography you must send the .bib files.

{\LaTeX} can't read your mind. If you assign the same label to a
subsubsection and a table, you might find that Table I has been cross
referenced as Table IV-B3.

{\LaTeX} does not have precognitive abilities. If you put a
\verb|\label| command before the command that updates the counter it's
supposed to be using, the label will pick up the last counter to be
cross referenced instead. In particular, a \verb|\label| command
should not go before the caption of a figure or a table.

Do not use \verb|\nonumber| inside the \verb|{array}| environment. It
will not stop equation numbers inside \verb|{array}| (there won't be
any anyway) and it might stop a wanted equation number in the
surrounding equation.

\subsection{Some Common Mistakes}\label{SCM}
\begin{itemize}
\item The word ``data'' is plural, not singular.
\item The subscript for the permeability of vacuum $\mu_{0}$, and other common scientific constants, is zero with subscript formatting, not a lowercase letter ``o''.
\item In American English, commas, semicolons, periods, question and exclamation marks are located within quotation marks only when a complete thought or name is cited, such as a title or full quotation. When quotation marks are used, instead of a bold or italic typeface, to highlight a word or phrase, punctuation should appear outside of the quotation marks. A parenthetical phrase or statement at the end of a sentence is punctuated outside of the closing parenthesis (like this). (A parenthetical sentence is punctuated within the parentheses.)
\item A graph within a graph is an ``inset'', not an ``insert''. The word alternatively is preferred to the word ``alternately'' (unless you really mean something that alternates).
\item Do not use the word ``essentially'' to mean ``approximately'' or ``effectively''.
\item In your paper title, if the words ``that uses'' can accurately replace the word ``using'', capitalize the ``u''; if not, keep using lower-cased.
\item Be aware of the different meanings of the homophones ``affect'' and ``effect'', ``complement'' and ``compliment'', ``discreet'' and ``discrete'', ``principal'' and ``principle''.
\item Do not confuse ``imply'' and ``infer''.
\item The prefix ``non'' is not a word; it should be joined to the word it modifies, usually without a hyphen.
\item There is no period after the ``et'' in the Latin abbreviation ``et al.''.
\item The abbreviation ``i.e.'' means ``that is'', and the abbreviation ``e.g.'' means ``for example''.
\end{itemize}
An excellent style manual for science writers is \cite{b7}.

\subsection{Authors and Affiliations}
\textbf{The class file is designed for, but not limited to, six authors.} A
minimum of one author is required for all conference articles. Author names
should be listed starting from left to right and then moving down to the
next line. This is the author sequence that will be used in future citations
and by indexing services. Names should not be listed in columns nor group by
affiliation. Please keep your affiliations as succinct as possible (for
example, do not differentiate among departments of the same organization).

\subsection{Identify the Headings}
Headings, or heads, are organizational devices that guide the reader through
your paper. There are two types: component heads and text heads.

Component heads identify the different components of your paper and are not
topically subordinate to each other. Examples include Acknowledgments and
References and, for these, the correct style to use is ``Heading 5''. Use
``figure caption'' for your Figure captions, and ``table head'' for your
table title. Run-in heads, such as ``Abstract'', will require you to apply a
style (in this case, italic) in addition to the style provided by the drop
down menu to differentiate the head from the text.

Text heads organize the topics on a relational, hierarchical basis. For
example, the paper title is the primary text head because all subsequent
material relates and elaborates on this one topic. If there are two or more
sub-topics, the next level head (uppercase Roman numerals) should be used
and, conversely, if there are not at least two sub-topics, then no subheads
should be introduced.

\subsection{Figures and Tables}
\paragraph{Positioning Figures and Tables} Place figures and tables at the top and
bottom of columns. Avoid placing them in the middle of columns. Large
figures and tables may span across both columns. Figure captions should be
below the figures; table heads should appear above the tables. Insert
figures and tables after they are cited in the text. Use the abbreviation
``Fig.~\ref{fig}'', even at the beginning of a sentence.

% CREATES TABLE
\begin{table}[htbp]
\caption{Table Type Styles}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Table}&\multicolumn{3}{|c|}{\textbf{Table Column Head}} \\
\cline{2-4}
\textbf{Head} & \textbf{\textit{Table column subhead}}& \textbf{\textit{Subhead}}& \textbf{\textit{Subhead}} \\
\hline
copy& More table copy$^{\mathrm{a}}$& &  \\
\hline
\multicolumn{4}{l}{$^{\mathrm{a}}$Sample of a Table footnote.}
\end{tabular}
\label{tab1}
\end{center}
\end{table}% CREATES IMAGE FIGURE
\begin{figure}[htbp]
\centerline{\includegraphics[width=\linewidth]{fig1.png}}
\caption{Example of a figure caption.}
\label{fig}
\end{figure}Figure Labels: Use 8 point Times New Roman for Figure labels. Use words
rather than symbols or abbreviations when writing Figure axis labels to
avoid confusing the reader. As an example, write the quantity
``Magnetization'', or ``Magnetization, M'', not just ``M''. If including
units in the label, present them within parentheses. Do not label axes only
with units. In the example, write ``Magnetization (A/m)'' or ``Magnetization
\{A[m(1)]\}'', not just ``A/m''. Do not label axes with a ratio of
quantities and units. For example, write ``Temperature (K)'', not
``Temperature/K''.

\section*{Acknowledgment}

The preferred spelling of the word ``acknowledgment'' in America is without
an ``e'' after the ``g''. Avoid the stilted expression ``one of us (R. B.
G.) thanks $\ldots$''. Instead, try ``R. B. G. thanks$\ldots$''. Put sponsor
acknowledgments in the unnumbered footnote on the first page.

\section*{References}

Please number citations consecutively within brackets \cite{b1}. The
sentence punctuation follows the bracket \cite{b2}. Refer simply to the reference
number, as in \cite{b3}---do not use ``Ref. \cite{b3}'' or ``reference \cite{b3}'' except at
the beginning of a sentence: ``Reference \cite{b3} was the first $\ldots$''

Number footnotes separately in superscripts. Place the actual footnote at
the bottom of the column in which it was cited. Do not put footnotes in the
abstract or reference list. Use letters for table footnotes.

Unless there are six authors or more give all authors' names; do not use
``et al.''. Papers that have not been published, even if they have been
submitted for publication, should be cited as ``unpublished'' \cite{b4}. Papers
that have been accepted for publication should be cited as ``in press'' \cite{b5}.
Capitalize only the first word in a paper title, except for proper nouns and
element symbols.

For papers published in translation journals, please give the English
citation first, followed by the original foreign-language citation \cite{b6}.
\bibliographystyle{IEEEtran}
\bibliography{References}\end{document}


```python
import sys
sys.stdout.write('hello world\n')
sys.stdout.flush()
for i in range(3):
    sys.stdout.write('%s\n' % i)
    sys.stdout.flush()
sys.stderr.write('output to stderr\n')
sys.stderr.flush()
sys.stdout.write('some more stdout text\n')
sys.stdout.flush()
```

    hello world
    0
    1
    2

    output to stderr

    some more stdout text

# Markdown Cell

$ e^{ \pm i\theta } = \cos \theta \pm i\sin \theta + \beta $

*It* **really** is!

```python
this is a syntax error
```

```python
print('test')
```

```python
from IPython.display import Latex
Latex('''The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\\begin{equation}
E=m
\\end{equation}''')
```

The mass-energy equivalence is described by the famous equation

$$E=mc^2$$

discovered in 1905 by Albert Einstein.
In natural units ($c$ = 1), the formula expresses the identity

\begin{equation}
E=m
\end{equation}

```python

```


# Overview of RNA-Seq analysis

## 0. Preliminaries

### Notation

A line beginning with a dollar sign (`$`) indicates the command line. It is meant to orient you that the command is to be performed in the terminal.

    $

This is not to be confused with something like `$HOME`, which means a variable named "HOME" that has the value (on my computer), shown by `echo` (which prints the output of commands). The dollar sign here means to access that variable. The line after the command line means the output of the command.

    $ echo $HOME
    /Users/olga

### Anatomy of a Unix command

We'll be using unix commands here and it's worth pausing and talking about what these commands are really doing. Let's pretend we have a program called `libraryprep`. We can call the program using its name:

    $ libraryprep

Let's say it has the option flag (aka "flag" aka "argument") `--stranded`. You'd tell the program this flag by putting it after the command:

    $ libraryprep --stranded

A lot of times there's a shortened version of the flag. For `--stranded` let's say it's `-s`. Then you could use the shorter command,

    $ libraryprep -s  # This is equal to "libraryprep --stranded"

Notice the hash "`#`". In all the languages we'll be using (bash, Python), this is a "comment" meaning its only for human reading and the computer can stop reading as soon as it sees the hash.

If a flag requires an input, you would put it immediately after the flag. For example, say we have a flag called `--input` and it requires the cell lysis file:

    $ libraryprep -s --input lysis

Now Let's say it has the options `-a`, `-s`, `-d`, `-f` We can put all of them sequentially:

    $ libraryprep -a -s -d -f

Or put them all in a row after a single dash. So the above command is equivalent to:

    $ libraryprep -asdf

But if you have a multi-character flag like `--stranded` then that has to be separate:

    $ libraryprep -adf --stranded  # correct
    $ libraryprep -adfstranded     # incorrect

Or if the flag requires an argument, then you have to keep it separate, unless the shortened version is at the end. For the purposes of this, let's say `-i` is short for `--input`.

    $ libraryprep -asdf --input lysis  # correct
    $ libraryprep -asdfi lysis         # correct
    $ libraryprep -asdfinput lysis     # incorrect
    $ libraryprep -asdif lysis         # incorrect

If you have a long command, you can separate it out onto multiple lines for readability (meaning just for you the human, not for the computer) by using the backslash "`\`" for each line that's supposed to continue on to the next one. The indentation is purely for readability and isn't required - it just looks nicer.

Fun fact: The way I remember that "`\`" backslash because the top is falling backwards, and "`/`" is frontslash because the top is falling forwards.

    $ # Correct
    $ libraryprep -adf \
        --input lysis \
        --stranded

 It is very important that there are no spaces or other characters after the slashes! Otherwise your program will throw a cryptic syntax error and you will be sad.

    $ # Incorrect
    $ libraryprep -adf \ # comments aren't allowed here
        --input lysis \ --flags-arent-okay-either
        --stranded

### Getting help

For most command-line programs, you can get help on how to actually use the program one of four ways. This is called finding the "usage" message. I've written them in priority order so, try the first one first, then the second, etc.

1. `$ programname`
    2. Without any arguments, this will call the program. Sometimes this will give you usage information, sometimes it will result in an error.
2. `$ programname -h`
    3. The `-h` is (generally) a universal short version flag for "help me!!! I don't know what's going on!!" Though sometimes that doesn't work either ...
3. `$ programname --help`
    4. The expanded version of `-h`
4. `$ man programname`
    5. `man` is for "manual" and will bring up the manual pages for that command

### Subcommands

Some programs we'll use will have subcommands. This means that we're calling a specific subprogram of a single program. For example, let's pretend we have a program called `microscopy`, and it has the subcommands `brightfield` and `fluorescence`. You wouldn't be able to call just `microscopy` by itself, because you need to specify the exact subprogram (in this analogy, the type of light inputs) that you want to use:

    $ microscopy
    Error: Must specify either "brightfield" or "fluorescence"

There will be flags like `--zoom` that would make sense for both subcommands:

    $ microscopy brightfield --zoom 10x

But only for the `fluorescence` subcommand does the flag `--wavelength` make any sense, since `brightfield` would show all the light.

    $ microscopy fluorescence --wavelength 488 --zoom 10x

### "Standard in" and "standard out"

Some of these commands use the greater than (`>`), which means that the output of the program should be saved a file. I like to think of it as the big input (the left side of the `>`) being squeezed into the small corner of the arrow (right side of the `>`).

For example, let's go back to our `libraryprep` example. We tell the `libraryprep` program we want a stranded library with our `lysis` input, and to save it as the file called `cdna`.

    $ libraryprep --stranded --input lysis > cdna

## 1. Library quality control

### `.fasta` file format

This is the simple, very commonly used file format in bioinformatics for storing sequences. An example of the p53 protein sequence is below.

    >sp|P04637|P53_HUMAN Cellular tumor antigen p53 OS=Homo sapiens GN=TP53 PE=1 SV=4
    MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP
    DEAPRMPEAAPPVAPAPAAPTPAAPAPAPSWPLSSSVPSQKTYQGSYGFRLGFLHSGTAK
    SVTCTYSPALNKMFCQLAKTCPVQLWVDSTPPPGTRVRAMAIYKQSQHMTEVVRRCPHHE
    RCSDSDGLAPPQHLIRVEGNLRVEYLDDRNTFRHSVVVPYEPPEVGSDCTTIHYNYMCNS
    SCMGGMNRRPILTIITLEDSSGNLLGRNSFEVRVCACPGRDRRTEEENLRKKGEPHHELP
    PGSTKRALPNNTSSSPQPKKKPLDGEYFTLQIRGRERFEMFRELNEALELKDAQAGKEPG
    GSRAHSSHLKSKKGQSTSRHKKLMFKTEGPDSD

### `.fastq` file format

The current standard of sequencing files is [`fastq`](https://en.wikipedia.org/wiki/FASTQ_format), which is derived from the `fasta` format. Check out an example `fastq` format below which shows 3 entries:

    $ head -n 12 projects/shalek2013/raw_data/S10.chr11.R1.fastq
    @SRR578577.1 C0HLAACXX120407:2:1107:16553:42649 length=101
    TCTCCTTAAATTTTAAGTAAATGTTTAAGGGATTTTACACCGGTCTATGGAGGTTTGCATGTGTAATTTTACCTCTAATTAATTATAAGGCCAGGACCAAA
    +SRR578577.1 C0HLAACXX120407:2:1107:16553:42649 length=101
    @@@FFDDDHHFHGIG?EAFDHHGHHHFHIEG;FDHIHIIIGIIDGHGHHIGIGGEHGGGGIFHIIGGGGGIGHHHECEDFFFFFFDEDEEE@BCBBBCB##
    @SRR578577.2 C0HLAACXX120407:2:1203:19118:72461 length=101
    CCTCTCCTTAAATTTTAAGTAAATGTTTAAGGGATTTTACACCGGTCTATGGAGGTTTGCATGTGTAATTTTACCTCTAATTAATTATAAGGCCAGGACCA
    +SRR578577.2 C0HLAACXX120407:2:1203:19118:72461 length=101
    ?@@DFFFDHGFDHEGHFHII<EHHAHHIIIIIIIIIIGEHEIIIIGGG<B<3BFGIEIGIIIGGC.7=CDCHCE??ACEBEE@>CACEC>6;;==BBBBBB
    @SRR578577.3 C0HLAACXX120407:2:1110:7566:40229 length=101
    ACCCTCTCCTTAAATTTTAAGTAAATGTTTAAGGGATTTTACACCGGTCTATGGAGGTTTGCATGTGTAATTTTACCTCTAATTAATTATAAGGCCAGGAC
    +SRR578577.3 C0HLAACXX120407:2:1110:7566:40229 length=101
    ???;:A;;C:D?AG<EFBDG<3CF:F><CCDHI>GFFEGEFFEGE:D0?6?D*/.8-;>AC=FDD=@CAEE=>?CAE@B@@BD>ABB;;>5>;;>?#####

Notice that each read takes up 4 lines, where

    @unique_identifier length=101
    [genomic sequence]
    +unique_identifier length=101
    [ASCII-encoded PHRED quality score]

Let's break the quality down a bit.

### Sequencing quality scores

The quality of base calling for each position in the read is encoded as a score $Q$ from 1 to 40. Defining $p$ as the probability of incorrect base calling, then

$
Q = -10 \log_{10}(p)
$

Given any score, we can back-calculate the probability of an incorrect base by using our understanding of logarithms.

$$
\begin{align}
40 &= -10 \log_{10}(p)\\
\frac{40}{-10} &= \frac{-10}{-10} \log_{10}(p)\\
-4 &= \log_{10}(p)\\
10^{-4} &= 10^{\log_{10}(p)}\\
10^{-4} &= p
\end{align}
$$

Thus the best score of 40 represents the lowest probability of incorrect base calling, $p=10^{-4}$.

#### Exercise 1

What is the probabilty of incorrect base calling for a score of 39? For 31?

### PHRED64 quality scores

Because "40" takes up two spaces, instead of using numbers the score is encoded using letters in a system called "ASCII." You already know that computers store information in zeros and ones, which can be turned into numbers, well, the letter "!" (exclamation point) represents the number 64 and acts as the lowest value for the quality scores, while

    !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHI

* Input: Raw `.fastq` files
* Program: FASTQC, SeQC
* Output: Report (`.html`) and plots (`.png`) of library quality

Example:

    $ fastqc $HOME/projects/differentiation/raw_data/sample01.R1.fastq -o $HOME/projects/differentiation/processed_data

## 2. Adapter trimming

* Input: Raw `.fastq` files
* Program: `cutadapt`, `trimgalore`
* Output: Adapter trimmed `.fastq` files

Example:

    cutadapt \
        # format of the input files \
        -f fastq  \
        # Maximum number of times an adapter sequence can be removed \
        --times 2 \
        # Maximum error rate in adapter \
        -e 0.0  \
        # Minimum overlap length \
        -O 5  \
        # Minimum sequencing quality \
        --quality-cutoff 6 \
        # Minimum read length \
        -m 18 \
        # Adapters that could appear anywhere in the sequencing read (5' end or 3' end) \
        -b TCGTATGCCGTCTTCTGCTTG \
        -b ATCTCGTATGCCGTCTTCTGCTTG \
        -b CGACAGGTTCAGAGTTCTACAGTCCGACGATC \
        -b GATCGGAAGAGCACACGTCTGAACTCCAGTCAC \
        -b AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA \
        -b TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT \
        # Read 1 output \
        -o $HOME/projects/shalek2013/processed_data/S10_R1.polyATrim.adapterTrim.fastq \
        # Read 2 output \
        -p $HOME/projects/shalek2013/processed_data/S10_R2.polyATrim.adapterTrim.fastq \
        # Read 1 input \
        $HOME/projects/shalek2013/raw_data/S10_R1.fastq.gz \
        # Read 2 input \
        $HOME/projects/shalek2013/raw_data/S10_R2.fastq.gz \
        # Statistics about how many adapters were removed
        > $HOME/projects/shalek2013/processed_data/S10_R2.polyATrim.adapterTrim.metrics

## 3. Map to repetitive elements

### `.sam` file format

"Sequence alignment map"

* Input: Trimmed `.fastq` files and repetitive element "genome"
* Program: `STAR`
* Output: Alignment `.sam` and unaligned `.fastq`

Example:

    STAR --genomeDir $HOME/genomes/RepBase18.05.fasta/STAR/ --readFilesIn \
        $HOME/projects/differentiation/raw_data/sample01.R1.fastq \
        $HOME/projects/differentiation/raw_data/sample01.R2.fastq \
        --outFileNamePrefix $HOME/projects/differentiation/processed_data/sample01.repetitive. \
        # Save unmapped files as fastq
        --outReadsUnmapped Fastx

## 4. Map to genome

* Input: `.fastq` of trimmed reads which didn't align to repetitive elements
* Program: `STAR`
* Output: Alignment `.sam`

Example:

    STAR --genomeDir $HOME/genomes/hg19/gencode/v20/star \
        --readFilesIn \
        $HOME/projects/differentiation/processed_data/sample01.repetitive.Unmapped.out.mate1 \
        $HOME/projects/differentiation/processed_data/sample01.repetitive.Unmapped.out.mate2 \
        --outFileNamePrefix \
        $HOME/projects/differentiation/processed_data/sample01.rmRepetitive.

### Exercise 2

Run STAR with the following parameters:

* genome directory: `$HOME/genomes/mm10/gencode/m8/star_chr11`
* Fastq files: `$HOME/projects/shalek2013/raw_data/S10.chr11.R1.fastq` and `$HOME/projects/shalek2013/raw_data/S10.chr11.R2.fastq`
* Output prefix: `$HOME/projects/shalek2013/processed_data/S10.chr11.`

#### Questions
1. What files were created?
2. In what file can you find the percentage of mapped reads?
3. In what file can you find the debugging information for the program, like the full command, the files used, and the values of all the parameters?

## 5. Create sorted, indexed `.bam` file

### `.bam` file format

The `.bam` file format is a compressed (aka "binary" - meaning not human readable) version of the `.sam` format.

All programs downstream of mapping require a sorted, indexed `.bam` file. You can also do these steps with your repetitive element-aligned `.sam` file if

Think of creating a sorted, indexed bam file like making a dictionary.

A raw aligned `.sam` file like a jumble of words in random order:

    single
    cell
    bioinformatics
    experiment
    beyonce
    sequencing
    library
    prep

Sorting puts the words into alphabet order:

    beyonce
    bioinformatics
    cell
    experiment
    library
    prep
    sequencing
    single

And indexing is like adding the alphabet tabs in a dictionary:

    B:
        beyonce
        bioinformatics
    C:
        cell
    E:
        experiment
    L:
        library
    P:
        prep
    S:
        sequencing
        single

Instead of letters, indexing adds "chromosome tabs" so each program will know where, say chromosome 7 (`chr7`) starts and can jump to it right away.

### 5.0 Convert `.sam` to `.bam`

* Input: `.sam` of aligned reads
* Program: `samtools view -b`
    * `-b` flag forces output to be `.bam`
* Output: Compressed `.bam` file

Example (`samtools v1.3.1`):

    samtools view -b sample01.sam > sample01.sorted.bam

#### Exercise 3

Convert your `.sam` file to `.bam`

### 5.1 Sort `.sam`

For deeply sequenced samples (>50 million reads), this will take quite some time

* Input: `.sam` of aligned reads
* Program: `samtools sort`
* Output: Sorted `.bam`

Example (`samtools v1.3.1`):

    samtools sort sample01.bam > sample01.sorted.bam

#### Exercise 4

Sort your `.bam` file

### 5.2 Index `.sam`

Indexing only works with sorted bam files

* Input: `.bam` of *sorted*, aligned reads
* Program: `samtools sort`
* Output: Sorted `.bam`

Example (`samtools v1.3.1`):

    samtools index sample01.sorted.bam

This creates the file `sample01.sorted.bam.bai`

#### Exercise 5

Index your `.bam` file

## 6. Quantify gene expression

### Units

TPM, FPKM, what? Check out this blog post on [What the FPKM?](https://haroldpimentel.wordpress.com/2014/05/08/what-the-fpkm-a-review-rna-seq-expression-units/) to learn about RNA-seq units in depth.

### File types

Gene annotation files are either `.gtf` or `.gff` format. We'll use the [GENCODE GTF](http://www.gencodegenes.org/data_format.html) format for our purposes. A subset of the gtf format is shown below. Notice the third column specifies the type of feature.

```
chr11   HAVANA  gene    3125904 3131004 .       +       .       gene_id "ENSMUSG00000082286.10"; gene_type "transcribed_unprocessed_pseudogene"; gene_status "KNOWN"; gene_name "Pisd-ps1"; level 2; havana_gene "OTTMUSG00000000759.2";
chr11   HAVANA  transcript      3125904 3129396 .       +       .       gene_id "ENSMUSG00000082286.10"; transcript_id "ENSMUST00000145164.7"; gene_type "transcribed_unprocessed_pseudogene"; gene_status "KNOWN"; gene_name "Pisd-ps1"; transcript_type "processed_transcript"; transcript_status "KNOWN"; transcript_name "Pisd-ps1-004"; level 2; tag "basic"; transcript_support_level "1"; havana_gene "OTTMUSG00000000759.2"; havana_transcript "OTTMUST00000086908.1";
chr11   HAVANA  exon    3125904 3126058 .       +       .       gene_id "ENSMUSG00000082286.10"; transcript_id "ENSMUST00000145164.7"; gene_type "transcribed_unprocessed_pseudogene"; gene_status "KNOWN"; gene_name "Pisd-ps1"; transcript_type "processed_transcript"; transcript_status "KNOWN"; transcript_name "Pisd-ps1-004"; exon_number 1; exon_id "ENSMUSE00000728833.1"; level 2; tag "basic"; transcript_support_level "1"; havana_gene "OTTMUSG00000000759.2"; havana_transcript "OTTMUST00000086908.1";
chr11   HAVANA  exon    3127479 3127644 .       +       .       gene_id "ENSMUSG00000082286.10"; transcript_id "ENSMUST00000145164.7"; gene_type "transcribed_unprocessed_pseudogene"; gene_status "KNOWN"; gene_name "Pisd-ps1"; transcript_type "processed_transcript"; transcript_status "KNOWN"; transcript_name "Pisd-ps1-004"; exon_number 2; exon_id "ENSMUSE00000708691.1"; level 2; tag "basic"; transcript_support_level "1"; havana_gene "OTTMUSG00000000759.2"; havana_transcript "OTTMUST00000086908.1";
```

### How to run the programs

* Input: `.bam` of *sorted*, aligned reads
* Program: `featureCounts`, Salmon, HTSeq, ...
* Output: some kind of table (comma- or tab-delimited file)

Example:

    featureCounts -a annotation.gtf -o sample01.featurecounts.txt sample1.sorted.bam

#### Exercise 5

Run `featureCounts` on your sorted, indexed `.bam` file

* Annotation file: `$HOME/genomes/mm10/gencode/m8/gencode.vM8.basic.annotation.chr11.gtf`

## 7. Create gene expression matrix

* Input: Individual cells' gene expression counts
* Program: R, Python, `awk`, `cut`, `paste`
* Output: A matrix of size (`n_samples` $\times$ `n_features`)

## 8. Cell-level QC

* Input: Gene expression matrix
* Program: R, Python, MATLAB
* Possible criteria:
    * Low number of reads mapped (indicates lowly sequenced)
    * Low percentage of reads mapped (indicates poor library)
    * High mitochondrial:genome ratio (indicates dying cell)
    * High spike-in:genome ratio (indicates low cellular mRNA content)
    * Low numbers of expressed genes (indicates low detection)
* Output: Gene expression matrix with "bad" cells removed

## 9. Gene-level QC

* Input: Gene expression matrix with "bad" cells removed
* Program: R, Python, MATLAB
* Possible criteria:
    * Low expression (e.g. $< 1$ TPM in $> 5$ cells)
* Output: Gene expression matrix with "bad" genes removed

## 10. Exploratory analysis

* Input: Gene expression matrix with "bad" genes removed
* Program: R, Python, MATLAB
* Possible algorithms:
    * Dimensionality reduction
        * PCA
        * ICA
        * MDS
        * t-SNE
* Output: plots

## 11. Subpopulation identification

* Input: Gene expression matrix with "bad" genes removed
* Program: R, Python, MATLAB
* Possible algorithms:
    * Clustering
        * Hierarchical clustering
        * $K$-means
    * Pseudo-time
        * Monocle
        * Waterfall
        * Wanderlust
        * Wishbone
* Output: plots, cluster identification

## 12. Subpopulation characterization

* Input: Gene expression matrix with "bad" genes removed, cluster or psuedotime ordering
* Program: R, Python, MATLAB
* Possible algorithms:
    * Differential expression
        * DESeq
        * scDE ("single-cell differential expression")
    * Classifiers
        * Support Vector Machines (SVM)
        * DecisionTree classifiers
    * Clustering
        * Hierarchical clustering on ordered gene expression
* Output: Lists of genes

## 13. Interpretation

* Input: Lists of genes
* Program: Biologist, Gene Ontology (e.g. DAVID) or other gene set enrichment analysis, PubMed
* Output: Biological description of populations