# Example

```python
!python3 -m pip install git+git://github.com/tensortrade-org/tensortrade.git
```

    Collecting git+git://github.com/tensortrade-org/tensortrade.git
      Cloning git://github.com/tensortrade-org/tensortrade.git to /private/var/folders/r9/1ph6hh194qj_w8209my99snc0000gn/T/pip-req-build-wu0s93p0
      Running command git clone -q git://github.com/tensortrade-org/tensortrade.git /private/var/folders/r9/1ph6hh194qj_w8209my99snc0000gn/T/pip-req-build-wu0s93p0
    Requirement already satisfied: numpy==1.16.4 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from tensortrade==0.1.0rc1) (1.16.4)
    Requirement already satisfied: pandas==0.25.0 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from tensortrade==0.1.0rc1) (0.25.0)
    Requirement already satisfied: gym==0.14.0 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from tensortrade==0.1.0rc1) (0.14.0)
    Requirement already satisfied: pyyaml==5.1.2 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from tensortrade==0.1.0rc1) (5.1.2)
    Requirement already satisfied: sympy==1.4 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from tensortrade==0.1.0rc1) (1.4)
    Requirement already satisfied: pytz>=2017.2 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from pandas==0.25.0->tensortrade==0.1.0rc1) (2019.3)
    Requirement already satisfied: python-dateutil>=2.6.1 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from pandas==0.25.0->tensortrade==0.1.0rc1) (2.8.0)
    Requirement already satisfied: cloudpickle~=1.2.0 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.1.0rc1) (1.2.2)
    Requirement already satisfied: pyglet<=1.3.2,>=1.2.0 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.1.0rc1) (1.3.2)
    Requirement already satisfied: six in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.1.0rc1) (1.12.0)
    Requirement already satisfied: scipy in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from gym==0.14.0->tensortrade==0.1.0rc1) (1.3.1)
    Requirement already satisfied: mpmath>=0.19 in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from sympy==1.4->tensortrade==0.1.0rc1) (1.1.0)
    Requirement already satisfied: future in /opt/anaconda3/envs/tensortrade/lib/python3.7/site-packages (from pyglet<=1.3.2,>=1.2.0->gym==0.14.0->tensortrade==0.1.0rc1) (0.17.1)
    Building wheels for collected packages: tensortrade
      Building wheel for tensortrade (setup.py) ... done
      Created wheel for tensortrade: filename=tensortrade-0.1.0rc1-cp37-none-any.whl size=79410 sha256=e9288cb7882925c90f42a7426e59bcfd7380f77ec50651d4f420b726338d7aef
      Stored in directory: /private/var/folders/r9/1ph6hh194qj_w8209my99snc0000gn/T/pip-ephem-wheel-cache-_vho_1jn/wheels/d4/5f/92/18e058e7f31d59deb14af7ecd3f907093cc894226b1c7abaa3
    Successfully built tensortrade
    Installing collected packages: tensortrade
    Successfully installed tensortrade-0.1.0rc1

```python
import ssl
import pandas as pd

ssl._create_default_https_context = ssl._create_unverified_context # Only used if pandas gives a SSLError

def fetch(exchange_name, symbol, timeframe):
    url = "https://www.cryptodatadownload.com/cdd/"
    filename = "{}_{}USD_{}".format(exchange_name, symbol, timeframe)
    df = pd.read_csv('https://www.cryptodatadownload.com/cdd/Coinbase_BTCUSD_1h.csv', skiprows=1)
    df = df[::-1]
    df = df.drop(["Symbol", "Volume BTC"], axis=1)
    df = df.rename({"Volume USD": "volume"}, axis=1)
    df.columns = [name.lower() for name in df.columns]
    df = df.set_index("date")
    df.columns = [symbol + ":" + name.lower() for name in df.columns]
    return df
```

## Historical Data

```python
coinbase_data = pd.concat([
    fetch("Coinbase", "BTC", "1h"),
    fetch("Coinbase", "ETH", "1h")
], axis=1)

bitstamp_data = pd.concat([
    fetch("Bitstamp", "BTC", "1h"),
    fetch("Bitstamp", "ETH", "1h"),
    fetch("Bitstamp", "LTC", "1h")
], axis=1)
```

```python
coinbase_data.head()
```

      BTC:open
      BTC:high
      BTC:low
      BTC:close
      BTC:volume
      ETH:open
      ETH:high
      ETH:low
      ETH:close
      ETH:volume

      date

      2017-07-01 11-AM
      2505.56
      2513.38
      2495.12
      2509.17
      287000.32
      2505.56
      2513.38
      2495.12
      2509.17
      287000.32

      2017-07-01 12-PM
      2509.17
      2512.87
      2484.99
      2488.43
      393142.50
      2509.17
      2512.87
      2484.99
      2488.43
      393142.50

      2017-07-01 01-PM
      2488.43
      2488.43
      2454.40
      2454.43
      693254.01
      2488.43
      2488.43
      2454.40
      2454.43
      693254.01

      2017-07-01 02-PM
      2454.43
      2473.93
      2450.83
      2459.35
      712864.80
      2454.43
      2473.93
      2450.83
      2459.35
      712864.80

      2017-07-01 03-PM
      2459.35
      2475.00
      2450.00
      2467.83
      682105.41
      2459.35
      2475.00
      2450.00
      2467.83
      682105.41

```python
bitstamp_data.head()
```

      BTC:open
      BTC:high
      BTC:low
      BTC:close
      BTC:volume
      ETH:open
      ETH:high
      ETH:low
      ETH:close
      ETH:volume
      LTC:open
      LTC:high
      LTC:low
      LTC:close
      LTC:volume

      date

      2017-07-01 11-AM
      2505.56
      2513.38
      2495.12
      2509.17
      287000.32
      2505.56
      2513.38
      2495.12
      2509.17
      287000.32
      2505.56
      2513.38
      2495.12
      2509.17
      287000.32

      2017-07-01 12-PM
      2509.17
      2512.87
      2484.99
      2488.43
      393142.50
      2509.17
      2512.87
      2484.99
      2488.43
      393142.50
      2509.17
      2512.87
      2484.99
      2488.43
      393142.50

      2017-07-01 01-PM
      2488.43
      2488.43
      2454.40
      2454.43
      693254.01
      2488.43
      2488.43
      2454.40
      2454.43
      693254.01
      2488.43
      2488.43
      2454.40
      2454.43
      693254.01

      2017-07-01 02-PM
      2454.43
      2473.93
      2450.83
      2459.35
      712864.80
      2454.43
      2473.93
      2450.83
      2459.35
      712864.80
      2454.43
      2473.93
      2450.83
      2459.35
      712864.80

      2017-07-01 03-PM
      2459.35
      2475.00
      2450.00
      2467.83
      682105.41
      2459.35
      2475.00
      2450.00
      2467.83
      682105.41
      2459.35
      2475.00
      2450.00
      2467.83
      682105.41

## Define Exchanges

An exchange needs a name, an execution service, and streams of price data in order to function properly.

The setups supported right now are the simulated execution service using simulated or stochastic data. More execution services will be made available in the future, as well as price streams so that live data and execution can be supported.

```python
from tensortrade.exchanges import Exchange
from tensortrade.exchanges.services.execution.simulated import execute_order
from tensortrade.data import Stream

coinbase = Exchange("coinbase", service=execute_order)(
    Stream("USD-BTC", list(coinbase['BTC:close'])),
    Stream("USD-ETH", list(coinbase['ETH:close']))
)

bitstamp = Exchange("bitstamp", service=execute_order)(
    Stream("USD-BTC", list(bitstamp['BTC:close'])),
    Stream("USD-ETH", list(bitstamp['ETH:close'])),
    Stream("USD-LTC", list(bitstamp['LTC:close']))
)
```

Now that the exchanges have been defined we can define our features that we would like to include, excluding the prices we have provided for the exchanges.

## Define External Data Feed

Here we will define the external feed to use whatever data you would like. From financial indicators to datasets that have nothing to do with instrument prices, they will all have to be defined and incorporated into the external data feed provided to the environment. There is also an internal data feed that is used to collect data on all the wallets and the net worth of the portfolio. You can choose to include this or not by using the `use_internal` parameter of the trading environment.

```python
import ta

from tensortrade.data import DataFeed, Namespace

# Add all features for coinbase bitcoin
coinbase_btc = coinbase_data.loc[:, [name.startswith("BTC") for name in coinbase_data.columns]]

ta.add_all_ta_features(
    coinbase_btc,
    colprefix="BTC:",
    **{k: "BTC:" + k for k in ['open', 'high', 'low', 'close', 'volume']}
)

nodes = []
for name in coinbase_btc.columns:
    nodes += [Stream(name, list(coinbase_btc[name]))]

# Namespaces are used to give context to features so they don't
# conflict with features of the same name that are being used for
# different data (e.g. different exchange)
coinbase_ns = Namespace("coinbase")(*nodes)

# Add all features for coinbase ethereum
bitstamp_eth = bitstamp_data.loc[:, [name.startswith("ETH") for name in bitstamp_data.columns]]
ta.add_all_ta_features(
    bitstamp_eth,
    colprefix="ETH:",
    **{k: "ETH:" + k for k in ['open', 'high', 'low', 'close', 'volume']}
)

nodes = []
for name in coinbase_btc.columns:
    nodes += [Stream(name, list(coinbase_btc[name]))]

bitstamp_ns = Namespace("coinbase")(*nodes)

feed = DataFeed([coinbase_ns, bitstamp_ns])
```

```python
feed.next()
```

    {'coinbase:/BTC:open': 2505.56,
     'coinbase:/BTC:high': 2513.38,
     'coinbase:/BTC:low': 2495.12,
     'coinbase:/BTC:close': 2509.17,
     'coinbase:/BTC:volume': 287000.32,
     'coinbase:/BTC:volume_adi': 154659.5371741516,
     'coinbase:/BTC:volume_obv': 287000.32,
     'coinbase:/BTC:volume_cmf': 0.5388828039430464,
     'coinbase:/BTC:volume_fi': nan,
     'coinbase:/BTC:volume_em': nan,
     'coinbase:/BTC:volume_sma_em': nan,
     'coinbase:/BTC:volume_vpt': -187654.4252942016,
     'coinbase:/BTC:volume_nvi': 1000.0,
     'coinbase:/BTC:volatility_atr': 0.0,
     'coinbase:/BTC:volatility_bbm': 2509.17,
     'coinbase:/BTC:volatility_bbh': 2509.17,
     'coinbase:/BTC:volatility_bbl': 2509.17,
     'coinbase:/BTC:volatility_bbw': 0.0,
     'coinbase:/BTC:volatility_bbhi': 0.0,
     'coinbase:/BTC:volatility_bbli': 0.0,
     'coinbase:/BTC:volatility_kcc': 2505.89,
     'coinbase:/BTC:volatility_kch': 2505.89,
     'coinbase:/BTC:volatility_kcl': 2487.6299999999997,
     'coinbase:/BTC:volatility_kchi': 0.0,
     'coinbase:/BTC:volatility_kcli': 0.0,
     'coinbase:/BTC:volatility_dcl': 2509.17,
     'coinbase:/BTC:volatility_dch': 2509.17,
     'coinbase:/BTC:volatility_dchi': 1.0,
     'coinbase:/BTC:volatility_dcli': 1.0,
     'coinbase:/BTC:trend_macd': nan,
     'coinbase:/BTC:trend_macd_signal': nan,
     'coinbase:/BTC:trend_macd_diff': nan,
     'coinbase:/BTC:trend_ema_fast': nan,
     'coinbase:/BTC:trend_ema_slow': nan,
     'coinbase:/BTC:trend_adx': 0.0,
     'coinbase:/BTC:trend_adx_pos': 0.0,
     'coinbase:/BTC:trend_adx_neg': 0.0,
     'coinbase:/BTC:trend_vortex_ind_pos': nan,
     'coinbase:/BTC:trend_vortex_ind_neg': nan,
     'coinbase:/BTC:trend_vortex_ind_diff': nan,
     'coinbase:/BTC:trend_trix': nan,
     'coinbase:/BTC:trend_mass_index': 0.0,
     'coinbase:/BTC:trend_cci': nan,
     'coinbase:/BTC:trend_dpo': 4750.493965940025,
     'coinbase:/BTC:trend_kst': -654.3682997212809,
     'coinbase:/BTC:trend_kst_sig': -654.3682997212809,
     'coinbase:/BTC:trend_kst_diff': 0.0,
     'coinbase:/BTC:trend_ichimoku_a': 2504.25,
     'coinbase:/BTC:trend_ichimoku_b': 2504.25,
     'coinbase:/BTC:trend_visual_ichimoku_a': 7247.049603784953,
     'coinbase:/BTC:trend_visual_ichimoku_b': 7236.058519236877,
     'coinbase:/BTC:trend_aroon_up': 4.0,
     'coinbase:/BTC:trend_aroon_down': 4.0,
     'coinbase:/BTC:trend_aroon_ind': 0.0,
     'coinbase:/BTC:trend_psar': 2509.17,
     'coinbase:/BTC:trend_psar_up': nan,
     'coinbase:/BTC:trend_psar_down': nan,
     'coinbase:/BTC:trend_psar_up_indicator': 0.0,
     'coinbase:/BTC:trend_psar_down_indicator': 0.0,
     'coinbase:/BTC:momentum_rsi': 100.0,
     'coinbase:/BTC:momentum_mfi': nan,
     'coinbase:/BTC:momentum_tsi': nan,
     'coinbase:/BTC:momentum_uo': nan,
     'coinbase:/BTC:momentum_stoch': 76.94414019715232,
     'coinbase:/BTC:momentum_stoch_signal': 76.94414019715232,
     'coinbase:/BTC:momentum_wr': -23.055859802847678,
     'coinbase:/BTC:momentum_ao': 0.0,
     'coinbase:/BTC:momentum_kama': nan,
     'coinbase:/BTC:momentum_roc': nan,
     'coinbase:/BTC:others_dr': -65.4368299721281,
     'coinbase:/BTC:others_dlr': nan,
     'coinbase:/BTC:others_cr': 0.0}

## Portfolio

Make the portfolio using the any combinations of exchanges and intruments that the exchange supports

```python
from tensortrade.instruments import USD, BTC, ETH, LTC
from tensortrade.wallets import Wallet, Portfolio

portfolio = Portfolio(USD, [
    Wallet(coinbase, 10000 * USD),
    Wallet(coinbase, 10 * BTC),
    Wallet(coinbase, 5 * ETH),
    Wallet(bitstamp, 1000 * USD),
    Wallet(bitstamp, 5 * BTC),
    Wallet(bitstamp, 20 * ETH),
    Wallet(bitstamp, 3 * LTC),
])
```

## Environment

```python
from tensortrade.environments import TradingEnvironment

env = TradingEnvironment(
    portfolio=portfolio,
    action_scheme='managed-risk',
    reward_scheme='simple',
    feed=feed,
    window_size=15,
    enable_logger=False
)
```

```python
env.feed.next()
```

    {'coinbase:/BTC:open': 2505.56,
     'coinbase:/BTC:high': 2513.38,
     'coinbase:/BTC:low': 2495.12,
     'coinbase:/BTC:close': 2509.17,
     'coinbase:/BTC:volume': 287000.32,
     'coinbase:/BTC:volume_adi': 154659.5371741516,
     'coinbase:/BTC:volume_obv': 287000.32,
     'coinbase:/BTC:volume_cmf': 0.5388828039430464,
     'coinbase:/BTC:volume_fi': nan,
     'coinbase:/BTC:volume_em': nan,
     'coinbase:/BTC:volume_sma_em': nan,
     'coinbase:/BTC:volume_vpt': -187654.4252942016,
     'coinbase:/BTC:volume_nvi': 1000.0,
     'coinbase:/BTC:volatility_atr': 0.0,
     'coinbase:/BTC:volatility_bbm': 2509.17,
     'coinbase:/BTC:volatility_bbh': 2509.17,
     'coinbase:/BTC:volatility_bbl': 2509.17,
     'coinbase:/BTC:volatility_bbw': 0.0,
     'coinbase:/BTC:volatility_bbhi': 0.0,
     'coinbase:/BTC:volatility_bbli': 0.0,
     'coinbase:/BTC:volatility_kcc': 2505.89,
     'coinbase:/BTC:volatility_kch': 2505.89,
     'coinbase:/BTC:volatility_kcl': 2487.6299999999997,
     'coinbase:/BTC:volatility_kchi': 0.0,
     'coinbase:/BTC:volatility_kcli': 0.0,
     'coinbase:/BTC:volatility_dcl': 2509.17,
     'coinbase:/BTC:volatility_dch': 2509.17,
     'coinbase:/BTC:volatility_dchi': 1.0,
     'coinbase:/BTC:volatility_dcli': 1.0,
     'coinbase:/BTC:trend_macd': nan,
     'coinbase:/BTC:trend_macd_signal': nan,
     'coinbase:/BTC:trend_macd_diff': nan,
     'coinbase:/BTC:trend_ema_fast': nan,
     'coinbase:/BTC:trend_ema_slow': nan,
     'coinbase:/BTC:trend_adx': 0.0,
     'coinbase:/BTC:trend_adx_pos': 0.0,
     'coinbase:/BTC:trend_adx_neg': 0.0,
     'coinbase:/BTC:trend_vortex_ind_pos': nan,
     'coinbase:/BTC:trend_vortex_ind_neg': nan,
     'coinbase:/BTC:trend_vortex_ind_diff': nan,
     'coinbase:/BTC:trend_trix': nan,
     'coinbase:/BTC:trend_mass_index': 0.0,
     'coinbase:/BTC:trend_cci': nan,
     'coinbase:/BTC:trend_dpo': 4750.493965940025,
     'coinbase:/BTC:trend_kst': -654.3682997212809,
     'coinbase:/BTC:trend_kst_sig': -654.3682997212809,
     'coinbase:/BTC:trend_kst_diff': 0.0,
     'coinbase:/BTC:trend_ichimoku_a': 2504.25,
     'coinbase:/BTC:trend_ichimoku_b': 2504.25,
     'coinbase:/BTC:trend_visual_ichimoku_a': 7247.049603784953,
     'coinbase:/BTC:trend_visual_ichimoku_b': 7236.058519236877,
     'coinbase:/BTC:trend_aroon_up': 4.0,
     'coinbase:/BTC:trend_aroon_down': 4.0,
     'coinbase:/BTC:trend_aroon_ind': 0.0,
     'coinbase:/BTC:trend_psar': 2509.17,
     'coinbase:/BTC:trend_psar_up': nan,
     'coinbase:/BTC:trend_psar_down': nan,
     'coinbase:/BTC:trend_psar_up_indicator': 0.0,
     'coinbase:/BTC:trend_psar_down_indicator': 0.0,
     'coinbase:/BTC:momentum_rsi': 100.0,
     'coinbase:/BTC:momentum_mfi': nan,
     'coinbase:/BTC:momentum_tsi': nan,
     'coinbase:/BTC:momentum_uo': nan,
     'coinbase:/BTC:momentum_stoch': 76.94414019715232,
     'coinbase:/BTC:momentum_stoch_signal': 76.94414019715232,
     'coinbase:/BTC:momentum_wr': -23.055859802847678,
     'coinbase:/BTC:momentum_ao': 0.0,
     'coinbase:/BTC:momentum_kama': nan,
     'coinbase:/BTC:momentum_roc': nan,
     'coinbase:/BTC:others_dr': -65.4368299721281,
     'coinbase:/BTC:others_dlr': nan,
     'coinbase:/BTC:others_cr': 0.0,
     'coinbase:/USD-BTC': 2509.17,
     'coinbase:/USD-ETH': 2509.17,
     'coinbase:/USD:/free': 10000,
     'coinbase:/USD:/locked': 0,
     'coinbase:/USD:/total': 10000,
     'coinbase:/BTC:/free': 10,
     'coinbase:/BTC:/locked': 0,
     'coinbase:/BTC:/total': 10,
     'coinbase:/BTC:/worth': 25091.7,
     'coinbase:/ETH:/free': 5,
     'coinbase:/ETH:/locked': 0,
     'coinbase:/ETH:/total': 5,
     'coinbase:/ETH:/worth': 12545.85,
     'bitstamp:/USD-BTC': 2509.17,
     'bitstamp:/USD-ETH': 2509.17,
     'bitstamp:/USD-LTC': 2509.17,
     'bitstamp:/USD:/free': 1000,
     'bitstamp:/USD:/locked': 0,
     'bitstamp:/USD:/total': 1000,
     'bitstamp:/BTC:/free': 5,
     'bitstamp:/BTC:/locked': 0,
     'bitstamp:/BTC:/total': 5,
     'bitstamp:/BTC:/worth': 12545.85,
     'bitstamp:/ETH:/free': 20,
     'bitstamp:/ETH:/locked': 0,
     'bitstamp:/ETH:/total': 20,
     'bitstamp:/ETH:/worth': 50183.4,
     'bitstamp:/LTC:/free': 3,
     'bitstamp:/LTC:/locked': 0,
     'bitstamp:/LTC:/total': 3,
     'bitstamp:/LTC:/worth': 7527.51,
     'net_worth': 118894.30999999998}


```python
import theme
theme.load_style()
```

# Introducing the IPython Notebook

This lecture is an adaptation of the lecture at [Teaching Numerics with Notebooks](http://nbviewer.ipython.org/github/ketch/teaching-numerics-with-notebooks/blob/master/Introducing%20the%20IPython%20Notebook.ipynb) and is covered by the following license

Creative Commons Attribution 4.0 International License.  All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

## What is this?

This is a gentle introduction to the IPython Notebook aimed at lecturers who wish to incorporate it in their teaching, written in an IPython Notebook.  This presentation adapts material from the [IPython official documentation](http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Notebook).

## What is an IPython Notebook?

An IPython Notebook is a:

**[A]** Interactive environment for writing and running code
**[B]** Weave of code, data, prose, equations, analysis, and visualization
**[C]** Tool for prototyping new code and analysis
**[D]** Reproducible workflow for scientific research
**[E]** All of the above

**[E]** All of the above

## How Do I Open an IPython Notebook?

Open a command prompt and execute

    ipython notebook
and an IPython Notebook server will start and launch a viewer in the default web browser.  Notebooks found in the working directory will show up in the list of notebooks.

## Example

For example, navigating to the course `Lessons` directory and executing `ipython notebook` will launch a browser window similar to that shown below

## Writing and Running Code

The IPython Notebook consists of an ordered list of cells.

There are three important cell types:

* **Code**
* **Markdown**
* **Raw NBConvert**

### Code Cells

```python
# This is a code cell made up of Python comments
# We can execute it by clicking on it with the mouse
# then clicking the "Run Cell" button
```

```python
# A comment is a pretty boring piece of code
# This code cell generates "Hello, World" when executed

print "Hello, World"
```

    Hello, World

### Code Cells

```python
# Code cells can also generate graphical output
%matplotlib inline
import matplotlib
matplotlib.pyplot.hist([0, 1, 2, 2, 3, 3, 3, 4, 4, 4, 10]);
```

## Modal editor

Starting with IPython 2.0, the IPython Notebook has a modal user interface. This means that the keyboard does different things depending on which mode the Notebook is in. There are two modes: edit mode and command mode.

### Edit mode

Edit mode is indicated by a green cell border and a prompt showing in the editor area:

When a cell is in edit mode, you can type into the cell, like a normal text editor.

Enter edit mode by pressing `enter` or using the mouse to click on a cell's editor area.

While in edit mode, tab-completion works for variables the kernel knows about from executing previous cells.

### Command mode

Command mode is indicated by a grey cell border:

When you are in command mode, you are able to edit the notebook as a whole, but not type into individual cells. Most importantly, in command mode, the keyboard is mapped to a set of shortcuts that let you perform notebook and cell actions efficiently. For example, if you are in command mode and you press `c`, you will copy the current cell - no modifier is needed.

Don't try to type into a cell in command mode; unexpected things will happen!

Enter command mode by pressing `esc` or using the mouse to click *outside* a cell's editor area.

## Mouse navigation

All navigation and actions in the Notebook are available using the mouse through the menubar and toolbar, which are both above the main Notebook area:

### Cell Clicking

The first idea of mouse based navigation is that **cells can be selected by clicking on them.** The currently selected cell gets a grey or green border depending on whether the notebook is in edit or command mode. If you click inside a cell's editor area, you will enter edit mode. If you click on the prompt or output area of a cell you will enter command mode.

If you are running this notebook in a live session (not on http://nbviewer.ipython.org) try selecting different cells and going between edit and command mode. Try typing into a cell.

### Current Selection

The second idea of mouse based navigation is that **cell actions usually apply to the currently selected cell**. Thus if you want to run the code in a cell, you would select it and click the "Play" button in the toolbar or the "Cell:Run" menu item. Similarly, to copy a cell you would select it and click the "Copy" button in the toolbar or the "Edit:Copy" menu item. With this simple pattern, you should be able to do most everything you need with the mouse.

### Markdown Rendering

Markdown and heading cells have one other state that can be modified with the mouse. These cells can either be rendered or unrendered. When they are rendered, you will see a nice formatted representation of the cell's contents. When they are unrendered, you will see the raw text source of the cell. To render the selected cell with the mouse, click the "Play" button in the toolbar or the "Cell:Run" menu item. To unrender the selected cell, double click on the cell.

## Keyboard Navigation

The modal user interface of the IPython Notebook has been optimized for efficient keyboard usage. This is made possible by having two different sets of keyboard shortcuts: one set that is active in edit mode and another in command mode.

The most important keyboard shortcuts are `enter`, which enters edit mode, and `esc`, which enters command mode.

In edit mode, most of the keyboard is dedicated to typing into the cell's editor. Thus, in edit mode there are relatively few shortcuts:

## Keyboard Navigation

In command mode, the entire keyboard is available for shortcuts.

Here the rough order in which the IPython Developers recommend learning the command mode shortcuts:

1. Basic navigation: `enter`, `shift-enter`, `up/k`, `down/j`
2. Saving the notebook: `s`
2. Cell types: `y`, `m`, `1-6`, `t`
3. Cell creation and movement: `a`, `b`, `ctrl+k`, `ctrl+j`
4. Cell editing: `x`, `c`, `v`, `d`, `z`, `shift+=`
5. Kernel operations: `i`, `0`

I suggest learning `h` first!

## The IPython Notebook Architecture

So far, we have learned the basics of using IPython Notebooks.

For simple demonstrations, the typical user doesn't need to understand how the computations are being handled, but to successfully write and present computational notebooks, **you** will need to understand how the notebook architecture works.

A *live* notebook is composed of an interactive web page (the front end), a running IPython session (the kernel or back end), and a web server responsible for handling communication between the two (the, err..., middle-end)

A *static* notebook, as for example seen on NBViewer, is a static view of the notebook's content.  The default format is HTML, but a notebook can also be output in PDF or other formats.

The centerpiece of an IPython Notebook is the "kernel", the IPython instance responsible for executing all code.  Your IPython kernel maintains its state between executed cells.

```python
x = 0
print x
```

    0

```python
x += 1
print x
```

    1

### Interacting with the Kernel

There are two important actions for interacting with the kernel.  The first is to interrupt it.  This is the same as sending a Control-C from the command line.  The second is to restart it.  This completely terminates the kernel and starts it anew.  None of the kernel state is saved across a restart.

## Markdown cells

Text can be added to IPython Notebooks using Markdown cells.  Markdown is a popular markup language that is a superset of HTML.  Its specification can be found here:

<http://daringfireball.net/projects/markdown/>

## Markdown basics

### Text formatting

You can make text *italic* or **bold**.

### Itemized Lists

* One
    - Sublist
        - This
  - Sublist
        - That
        - The other thing
* Two
  - Sublist
* Three
  - Sublist

### Enumerated Lists

1. Here we go
    1. Sublist
    2. Sublist
2. There we go
3. Now this

### Horizontal Rules

---

---

---

### Blockquotes

> To me programming is more than an important practical art. It is also a gigantic undertaking in the foundations of knowledge. -- Rear Admiral Grace Hopper

### Links

[IPython's website](http://ipython.org)

### Code

    def f(x):
        """a docstring"""
        return x**2

You can also use triple-backticks to denote code blocks.
This also allows you to choose the appropriate syntax highlighter.

```C
if (i=0; i<n; i++) {
  printf("hello %d\n", i);
  x += 4;
}
```

### Tables

Time (s) | Audience Interest
---------|------------------
 0       | High
 1       | Medium
 5       | Facebook

### Images

### YouTube

from IPython.display import YouTubeVideo
YouTubeVideo('vW_DRAJ0dtc')

### Other HTML

 Be Bold!

## Mathematical Equations

Courtesy of MathJax, you can beautifully render mathematical expressions, both inline:
$e^{i\pi} + 1 = 0$, and displayed:

$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$

### Equation Environments

You can also use a number of equation environments, such as `align`:

\begin{align}
  x &= 4 \\
y+z &= x
\end{align}

[A full list of available TeX and LaTeX commands is maintained by Dr. Carol Burns.](http://www.onemathematicalcat.org/MathJaxDocumentation/TeXSyntax.htm)

### Other Useful MathJax Notes

* inline math is demarcated by `$ $`, or `\( \)`
* displayed math is demarcated by `$$ $$` or `\[ \]`
* displayed math environments can also be directly demarcated by `\begin` and `\end`
* `\newcommand` and `\def` are supported, *within* areas MathJax processes (such as in a `\[ \]` block)
* equation numbering is not officially supported, but it can be indirectly enabled

## A Note about Notebook Security

By default, a notebook downloaded to a new computer is *untrusted*

* HTML and Javascript in Markdown cells is now *never* executed
* HTML and Javascript code outputs must be explicitly *re-executed*
* Some of these restrictions can be mitigrated through shared accounts (Sage MathCloud) and secrets

More information on notebook security is in the [IPython Notebook documentation](http://ipython.org/ipython-doc/stable/notebook/security.html)

## Magics

IPython kernels execute a superset of the Python language.  The extension functions, commonly referred to as *magics*, come in two variants.

### Line Magics

* A *line magic* looks like a command line call.  The most important of these is `%matplotlib inline`, which embeds all matplotlib plot output as images in the notebook itself.

```python
%matplotlib inline
```

```python
%whos
```

    Variable     Type      Data/Info
    --------------------------------
    matplotlib   module    <module 'matplotlib' from<...>matplotlib/__init__.pyc'>
    x            int       1

### Cell Magics

* A *cell magic* takes its entire cell as an argument.  Although there are a number of useful cell magics, you may find `%%timeit` to be useful for exploring code performance.

```python
%%timeit

import numpy as np
np.sum(np.random.rand(1000))
```

    The slowest run took 464.75 times longer than the fastest. This could mean that an intermediate result is being cached
    100000 loops, best of 3: 17.6 µs per loop

## Interacting with the Command Line

IPython supports one final trick, the ability to interact directly with your  shell by using the `!` operator.

```python
!ls
```

    Images                                       Lesson11_HigherOrderShapeFuncs.ipynb
    Lesson01_IPythonNotebookTutorial.ipynb       Lesson12_ThermalConvection.ipynb
    Lesson01_IPythonNotebookTutorial.slides.html Lesson13_AdvectionDiffusion.ipynb
    Lesson02_IntroductionToPython.ipynb          Lesson14_NonlinearMaterials.ipynb
    Lesson03_DirectMethods.ipynb                 Lesson15_ErrorAnalysis.ipynb
    Lesson04_PlaneTruss.ipynb                    Lesson16_MethodOfManufacturedSolutions.ipynb
    Lesson05_StrongForm.ipynb                    Lesson17_EulerBernouliBeamElement.ipynb
    Lesson06_VariationalForms.ipynb              Lesson20_GeneralizedBoundaryConditions.ipynb
    Lesson07_TheWeakForm.ipynb                   MathematicalPreliminaries.ipynb
    Lesson08_DerivationOf1DEquations.ipynb       styles
    Lesson09_ExamplesIn1D.ipynb                  theme.py
    Lesson10_NumericalIntegration.ipynb          theme.pyc

```python
x = !ls
```

```python
print x
```

    ['Images', 'Lesson01_IPythonNotebookTutorial.ipynb', 'Lesson01_IPythonNotebookTutorial.slides.html', 'Lesson02_IntroductionToPython.ipynb', 'Lesson03_DirectMethods.ipynb', 'Lesson04_PlaneTruss.ipynb', 'Lesson05_StrongForm.ipynb', 'Lesson06_VariationalForms.ipynb', 'Lesson07_TheWeakForm.ipynb', 'Lesson08_DerivationOf1DEquations.ipynb', 'Lesson09_ExamplesIn1D.ipynb', 'Lesson10_NumericalIntegration.ipynb', 'Lesson11_HigherOrderShapeFuncs.ipynb', 'Lesson12_ThermalConvection.ipynb', 'Lesson13_AdvectionDiffusion.ipynb', 'Lesson14_NonlinearMaterials.ipynb', 'Lesson15_ErrorAnalysis.ipynb', 'Lesson16_MethodOfManufacturedSolutions.ipynb', 'Lesson17_EulerBernouliBeamElement.ipynb', 'Lesson20_GeneralizedBoundaryConditions.ipynb', 'MathematicalPreliminaries.ipynb', 'styles', 'theme.py', 'theme.pyc']

## A Note about Notebook Version Control

The IPython Notebook is stored using canonicalized JSON for ease of use with version control systems.

There are two things to be aware of:

* By default, IPython embeds all content and saves kernel execution numbers.  You may want to get in the habit of clearing all cells before committing.

* As of IPython 2.0, all notebooks are signed on save.  This increases the chances of a commit collision during merge, forcing a manual resolution.  Either signature can be safely deleted in this situation.


---
author: "Marc Weber"
title: "Markdown template for scientific report using pandoc"
date: 2018.11.14
link-citations: "true"
#bibliography: "bibliography.bib"
bibliography: "/users/lserrano/mweber/Research_cloud/Bibliography/library.bib"

wordpress_site_url: "www.wordpress.org/site"
wordpress_username: "username"
wordpress_status: "draft"
wordpress_update_post_if_exists: "true"
wordpress_use_quicklatex: "true"
wordpress_toc: "true"

---

# Background

## Introduction

This document is a template for academic writing, science notebook records and reports. It is written in the Markdown format: it’s plain text, human-readable, and can easily be translated into other formats. It allows to write easily structured text with figures, mathematical formulas and bibliographic references without bothering about formatting. One of the biggest advantage is that by separating writing from formatting, I can keep a single source file in a lightweight text format and generate several different output documents in an automatic fashion. These output, for the moment, are: i) a PDF document generated by LaTeX, ii) a stand-alone HTML with integrated images and iii) a Wordpress post on my personal website.

Resources:

+ [Writing academic papers in plain text with Markdown and Jupyter notebook | Sylvain Deville](https://sylvaindeville.net/2015/07/17/writing-academic-papers-in-plain-text-with-markdown-and-jupyter-notebook/)
+ [Writing Workflow 2016, Part 1: Markdown Writing Environment · v & r](http://verifyandrepair.com/04-11-2016/writing-workflow-2016-markdown-environment/)

## Citations

### Citations

Citations are managed by the `pandoc-citeproc`  package (see [pandoc manual section about citations](http://pandoc.org/MANUAL.html#citations)). The use is as follows, now I want to insert an inline citation here: @Gillespie1977. A citation in square brackets format it with parenthesis: [@Miller1965]. Then, I could also use a grouped citations to multiple references like this: [@Danino2010;@Prindle2012]. We can also use citations in footnotes ^[Si2017].

Citations go inside square brackets and are separated by semicolons. Each citation must have a key, composed of ‘@’ + the citation identifier from the database, and may optionally have a prefix, a locator, and a suffix. To make your citations hyperlinks to the corresponding bibliography entries, add `link-citations: true` to your YAML metadata.

# Outline header 1

## Outline header 2

### Outline header 3

#### Outline header 4

##### Outline header 5

This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list. This is the outline text with a short list.

+ item 1
+ item 2
  + subitem 1
  + subitem 2

**Important note**: we need an empty line just *before* the list for pandoc markdown. In other markdown flavors this is not necessary.

We also test here the "citation" html tag: <cite>This is a short citation but we probably could use the quote tag for this, so it seems redundant.</cite>.

# Code highlighting

This is an inline code example in html `<pre class="lang:python">code example</pre>`{.html}.

This is a code block in python language:

```python
print("this is a test")
def new_function(var1=23):
    return [x for x in range(10) if x > 2]

raise SystemExit
```

This is a pre block:

```
print("this is a test")
def new_function(var1=23):
    return [x for x in range(10) if x > 2]

raise SystemExit
```

This is a code cell in jupyter notebook:

```python
# css: "/users/lserrano/mweber/Research_Dropbox/Python_mwTools/CSS_for_scientific_reports/pandoc.css"
# css: "/users/lserrano/mweber/Research_Dropbox/Python_mwTools/CSS_for_scientific_reports/github-pandoc.css"
print("test")
def function(arg):
    return list([x for x in range(0,10)])
```

# Latex math

Latex formulas are rendered by different plugins depending on the output file format. The html for wordpress output will use [QuickLatex](http://www.holoborodko.com/pavel/quicklatex/) wordpress plugin. The self-contained html output will use [MathJax](http://docs.mathjax.org/en/latest/index.html) script. And the pdf output will use the xelatex Latex engine.

#### How to write latex expressions in the markdown master file

In the *markdown master file*, we should wrap latex expression with tags that are understood by both the jupyter latex engine and the pandoc markdown math syntax, namely:

+ inline math `$ ... $`
+ display math `$$ ... $$`
+ math environments, e.g. `\begin{align} .. \end{align}`

Then, we will have to tweak pandoc to correctly convert these tags to tags compatible with the different latex engines of each output format.

#### Editing the markdown master file with Jupyter notebook

If using the jupyter notebook as an editing tool for the markdown master file, we would like to use latex tags that are recognized by the the jupyter markdown rendering engine. Note that in the case of editing the markdown master file with a simple text editor, this would not be important. As mentionned in the [documentation](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Working%20With%20Markdown%20Cells.html#LaTeX-equations), jupyter uses MathJax to render latex, however with some limitations that are not reported in the documentation.

+ inline math `$ ... $`
+ display math `$$ ... $$`
+ math environments (at least `align` and `multline`, probably all environments supported by MathJax)

#### Pandoc markdown to pdf using xelatex

From the pandoc [documentation](http://pandoc.org/MANUAL.html#math):

> Anything between two \$ characters will be treated as TeX math. The opening \$ must have a non-space character immediately to its right, while the closing \$ must have a non-space character immediately to its left, and must not be followed immediately by a digit. Thus, \$20,000 and \$30,000 won’t parse as math. If for some reason you need to enclose text in literal \$ characters, backslash-escape them and they won’t be treated as math delimiters.

+ inline math `$ ... $` with the whitespace restrictions.
+ display math `$$ ... $$`
+ math environments

#### Wordpress post with QuickLatex plugin

[QuickLatex](http://www.holoborodko.com/pavel/quicklatex/) supports many Latex features, as AMS math, and also custom preamble, which makes it very powerful. Also, all equations are rendered as SVG images, which display crisp at every zooming scale. Because all equations are rendered once and included as SVG images in the post, the load on the client browser is null resulting in fast download and display of the webpage, unlike MathJax script. The supported latex tags are the same as for MathJax:

+ inline math `\( ... \)` (`$ ... $` is also supported but is deactivated on my wordpress installation!).
+ display math `\[ ... \]` and `$$ ... $$`
+ math environments: `equation`, `align`, `displaymath`, `eqnarray`, `multline`, `flalign`, `gather`, and `alignat`.

Note: unfortunately quicklatex is parsing inside `<code></code>` block. This is why double dollar syntax was getting displayed as latex display equation with dots `...` inside. This could happend quite often when printing inline code with regex with escaped parenthesis, for example `re.sub(r'\(test\)', r'\1', "string")`. There is no problem in seperate code block because these get in a `pre` block, which QuickLatex does not parse. Solution is to enclose inline code in a `span` tag with `crayon-inline` class, which will enable basic syntax highlighting while preventing any latex parser to mess with it: `<span class="decode:true crayon-inline">re.sub(r'\(test\)', r'\1', "string")</span>`{.html}.

```python
import re
re.sub(r'\(test\)', r'\1', "string")
0 < 1
re.sub(r'$$abc$$', r'a$$abcd$$', string) = "this should not be parsed as latex expression"
```

Now we have to choose one particular syntax for inline and block code, because there is no standard in html5 about how to do that. People either use specific tags: `<codeblock>`, `<code>`{.html}, or the more general `<pre>` tag with class attribute that defines if the code is inline or in block. The Crayon plugin for syntax highlighting that I use on the wordpress site prefers 1) for code block the `<pre class="lang:python">` tag with the class defining the language, and 2) for inline code the `<span class="lang:python decode:true  crayon-inline ">` tag with the class giving the language attribute and the `crayon-inline` attribute which defines to insert the code inline. We will use these tags in the output for wordpress post.

#### Self-contained html with MathJax script

[MathJax](http://docs.mathjax.org/en/latest/index.html) preprocessor can look for whatever markers to define math delimiters. In the default configuration, the latex tags are

+ inline math `\( ... \)`. Important: `$ ... $` is **not** supported by default.
+ display math `$$ ... $$`
+ math environments
+ Need spaces around `<` and `>` symbols in order not to interfere with html syntax, example: `$x < y$`

I wish MathJax could render also all equations as SVG and include them as URI images in the self-contained html file. However, is seems [not possible at this time](https://github.com/jgm/pandoc/issues/682). Latex tags get correctly exported with the pandoc tool by passing the `--mathjax` option.

### Examples of LaTeX formulas in markdown master file

#### Inline math

inline latex expression examples enclosed by `$ .. $`: $\exp\left(e^{\tau}\right) < 1$

+ jupyter nbconvert to Markdown: `$\exp(e^{\tau)} < 1$` (unchanged)
+ pandoc markdown to pdf using xelatex: ok
+ self-contained html with MathJax script: ok. Converted to html code `<span class="math inline">\(\exp(e^{\tau)} < 1\)</span>`{.html}
+ wordpress post: ok, enclosed in span class and recognized by QuickLatex `<span class="math inline">\(\exp(e^{\tau)} < 1\)</span>`{.html}

Important: inline latex expression examples enclosed by `\( .. \)` is **not compatible** with Pandoc markdown math syntax and will result in an error in pandoc conversion to pdf using latex engine, for example.

#### Display math

display latex expression example enclosed by `$$ ... $$`: $$\exp\left(e^{\tau}\right) < 1$$

+ jupyter nbconvert to Markdown: `$$\exp(e^{\tau)} < 1$$` (unchanged)
+ pandoc markdown to pdf using xelatex: ok
+ self-contained html with MathJax script: ok. Converted to html code `<span class="math display">\[\exp(e^{\tau)} < 1\]</span>`{.html}
+ wordpress post: ok, enclosed in span class and recognized by QuickLatex `<span class="math display">\[\exp(e^{\tau)} < 1\]</span>`{.html}

#### Math environments

`align` environment:

\begin{align}
x + y &< 0 \\
\int_{i=0}^N e^{\alpha t} dt &= \pi
\end{align}

+ jupyter nbconvert to Markdown: (unchanged)
+ pandoc markdown to pdf using xelatex: ok
+ self-contained html with MathJax script: ok. Left unchanged in html code.
+ wordpress post: ok, Left unchanged in html code and recognized by QuickLatex.

`multline` environment:

\begin{multline}
x + y + c + a + b + d + e + f + g + x + y + c + a + b + d + \\
e + f + g + x + y + c + a + b + d + e + f + g + \\
+ 4 + a + b + d < \pi
\end{multline}

+ jupyter nbconvert to Markdown: (unchanged)
+ pandoc markdown to pdf using xelatex: ok
+ self-contained html with MathJax script: ok. Left unchanged in html code.
+ wordpress post: ok, Left unchanged in html code and recognized by QuickLatex.

#### Latex in code

The expressions in inline code and code blocks should **not** get parsed as latex expression. Examples inline: `code with latex-like tags inside $s = 1$ and $$x + y < 0$$`. And codeblock,

```python
print("code with latex-like tags inside")
$s = 1$
$$x + y < 0$$
```

+ jupyter editor: unfortunately, the latex expressions inside code blocks gets rendered as LateX expression.
+ jupyter nbconvert to Markdown: ok (unchanged)
+ pandoc markdown to pdf using xelatex: ok
+ self-contained html with MathJax script: ok.
+ wordpress post: ok

### Latex macros

New Latex commands can be defined by the `\newcommand` command inside the markdown file. They will be parsed and macros will be replaced every in the file when producing html output. When producing latex pdf output, the macros will be included in the latex header.

<!-- Latex commands -->
\newcommand{\ee}[1]{\cdot 10^{#1}\ }

# Images

Images can be inserted into Markdown documents in a variety of ways, each of one is not compatible with the external software that will convert markdown into PDF, html, or other document type. With Pandoc, the syntax is:

**Markdown syntax**

{#fig:isingmodel1 width=60%}[[ *Phase transition in a growing population of toggle switches interfaced by QS leading to the high $u$ state.* (A) Time evolution of concentration $u$ in color code for an exponentially growing population of toggle switches interfaced by QS.]]

**HTML syntax**

[[ *Phase transition in a growing population of toggle switches interfaced by QS leading to the high $u$ state.* (A) Time evolution of concentration $u$ in color code for an exponentially growing population of toggle switches interfaced by QS. ]]

Note that in the case of the markdown syntax, the figure is correctly rendered by the Jupyter notebook interface, **however the figure size directive is not taken into account**. The only way to change the image size in the jupyter interface, which uses standard markdown, is to use an HTML tags (see [python - How to Include image or picture in jupyter notebook - Stack Overflow](http://stackoverflow.com/questions/32370281/how-to-include-image-or-picture-in-jupyter-notebook)). However, images inserted by html tag will probably not get parsed correctly by pandoc (to verify).

We used a custom syntax for the figure captions, both for markdown and html, in order to be able to display the captions below the plot in a markdown rendered cell in the jupyter notebook. These captions will be converted to the pandoc markdown syntax in the parser script, such that:

```
{#fig:fig_id width=60%}[[figure caption text.]]
[[ figure caption text. ]]
```

displays the captions below the figure in the jupyter notebook. The markdown master file will be transformed into the correct pandoc markdown syntax:

```
{#fig:fig_id width=60%}
```

The figure number can be automatically referenced using the package `pandoc-fignos` (see [github webpage](https://github.com/tomduck/pandoc-fignos)) this is a reference to the figure @fig:isingmodel1.

##### Other markdown compatible syntaxes for inserting image

HTML tag:

``

General markdown:

``

pandoc style:

`{ width=50% }`

pandoc style with fignos package:

`{#fig:id}`

# Styling

The HTML outputs are styled with custom CSS sheet. I used the package MegaType (SCSS template) to create a responsive design and define all font sizes and line heights of the different elements in order to create a (almost perfect) vertical rhythm, aligning the text baseline on a baseline grid.

{width=80%}[[Baseline grid on desktop device, google chrome.]]

{width=50%}[[Baseline grid on mobile\ device, google chrome.]]

More information:

+ [Web typography is broken. Here’s how we can fix it – Thomas Bredin-Grey – Medium](https://medium.com/@tbredin/a-jolly-web-typesetting-adventure-42948ab0d1dd)
+ [StudioThick/megatype: Execute typographic structure with ease](https://github.com/StudioThick/megatype)
+ [cap-height calculator](http://codepen.io/elrumordelaluz/details/ZGpvvE) (useful to calculate the cap height of some common fonts).

# Bibliography

Note: the Bibliography header is not automatically included by pandoc.