# OPTIMADE and *pymatgen*

# What is *pymatgen*?

[*pymatgen*](https://pymatgen.org) is a materials science analysis code written in the Python programming language. It helps power the [Materials Project](https://materialsproject.org)'s high-throughput DFT workflows. It supports integration with a wide variety of simulation codes and can perform many analysis tasks such as the generation of phase diagrams or diffraction patterns.

# The motivation behind this tutorial

**This tutorial is aimed either at:**

* People who are already familiar with using *pymatgen* or the Materials Project
    * In particular, anyone already using the Materials Project API through the `MPRester`, and who would like to start using the OPTIMADE API in a similar way

* People who like using Python and think they might appreciate an interface like the one provided by *pymatgen*.
    * *pymatgen* provides a lot of input/output routines (such as conversion to CIF, POSCAR, etc.) and analysis tools (such as determination of symmetry, analysis of possible bonds, etc.) that can be performed directly on structures retrieved from OPTIMADE providers.

**What this tutorial is not:**

* This is not necessarily the way everyone should be accessing OPTIMADE providers!
    * This tool may be useful to you, or it may not be. There are a lot of good tools available in our community. You are encouraged to try out different tools and find the one that's most useful for your own work.

* It is not currently the best way to access OPTIMADE APIs for advanced users.
    * It is still under development.
    * It is unit tested against several OPTIMADE providers but **some do not work yet**.
    * It only currently supports information retrieval from `/v1/structures/` routes.

# Pre-requisites

This tutorial is aimed at people who already have a basic understanding of Python, including how to import modules, the use of basic data structures like dictionaries and lists, and how to intantiate and use objects.

If you do not have this understanding of Python, this tutorial may help you become familiar, but you are highly encouraged to follow a dedicated Python course such as those provided by [Software Carpentry](https://software-carpentry.org).

# Install pymatgen

This tutorial uses the Python programming language. It can be run on any computer with Python installed. For convenience, here we are running in Google's "Colaboratory" notebook environment.

Before we begin, we must install the `pymatgen` package:

```python
!pip install pymatgen pybtex
```

    Requirement already satisfied: pymatgen in /usr/local/lib/python3.7/dist-packages (2022.0.14)
    Requirement already satisfied: pybtex in /usr/local/lib/python3.7/dist-packages (0.24.0)
    Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (3.7.4.3)
    Requirement already satisfied: matplotlib>=1.5 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (3.2.2)
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pymatgen) (2.23.0)
    Requirement already satisfied: plotly>=4.5.0 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (5.3.1)
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen) (0.8.9)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pymatgen) (1.7.1)
    Requirement already satisfied: spglib>=1.9.9.44 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (1.16.2)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from pymatgen) (1.1.5)
    Requirement already satisfied: uncertainties>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (3.1.6)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (2.6.2)
    Requirement already satisfied: monty>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (2021.8.17)
    Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (1.7.1)
    Requirement already satisfied: numpy>=1.20.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (1.21.2)
    Requirement already satisfied: ruamel.yaml>=0.15.6 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (0.17.16)
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen) (3.3.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen) (2.8.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen) (2.4.7)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=1.5->pymatgen) (1.3.1)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib>=1.5->pymatgen) (1.15.0)
    Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly>=4.5.0->pymatgen) (8.0.1)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml>=0.15.6->pymatgen) (0.2.6)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from uncertainties>=3.1.4->pymatgen) (0.16.0)
    Requirement already satisfied: latexcodec>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from pybtex) (2.0.1)
    Requirement already satisfied: PyYAML>=3.01 in /usr/local/lib/python3.7/dist-packages (from pybtex) (3.13)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->pymatgen) (2018.9)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen) (2021.5.30)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen) (1.24.3)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pymatgen) (2.10)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pymatgen) (1.2.1)

Next, let us **verify the correct version of *pymatgen* is installed**. This is good practice to do before starting out! For this tutorial we need version 2022.0.17 or above. We also need the `pybtex` package installed.

```python
try:
    from importlib_metadata import version
except ImportError:
    from importlib.metadata import version
```

```python
version("pymatgen")
```

    '2022.0.14'

# Import and learn about the `OptimadeRester`

The `OptimadeRester` is a class that is designed to retrieve data from an OPTIMADE provider and automatically convert the data into *pymatgen* `Structure` objects. These `Structure` objects are designed as a good intermediate format for crystallographic structure analysis, transformation and input/output.

You can read documentation on the `OptimadeRester` here: https://pymatgen.org/pymatgen.ext.optimade.html

```python
from pymatgen.ext.optimade import OptimadeRester
```

The first step is to inspect the **documentation** for the `OptimadeRester`. We can run:

```python
OptimadeRester?
```

# Understanding "aliases" as shortcuts for accessing given providers

```python
OptimadeRester.aliases
```

    {'aflow': 'http://aflow.org/API/optimade/',
     'cod': 'https://www.crystallography.net/cod/optimade',
     'mcloud.2dstructures': 'https://aiida.materialscloud.org/2dstructures/optimade',
     'mcloud.2dtopo': 'https://aiida.materialscloud.org/2dtopo/optimade',
     'mcloud.curated-cofs': 'https://aiida.materialscloud.org/curated-cofs/optimade',
     'mcloud.li-ion-conductors': 'https://aiida.materialscloud.org/li-ion-conductors/optimade',
     'mcloud.optimade-sample': 'https://aiida.materialscloud.org/optimade-sample/optimade',
     'mcloud.pyrene-mofs': 'https://aiida.materialscloud.org/pyrene-mofs/optimade',
     'mcloud.scdm': 'https://aiida.materialscloud.org/autowannier/optimade',
     'mcloud.sssp': 'https://aiida.materialscloud.org/sssplibrary/optimade',
     'mcloud.stoceriaitf': 'https://aiida.materialscloud.org/stoceriaitf/optimade',
     'mcloud.tc-applicability': 'https://aiida.materialscloud.org/tc-applicability/optimade',
     'mcloud.threedd': 'https://aiida.materialscloud.org/3dd/optimade',
     'mp': 'https://optimade.materialsproject.org',
     'odbx': 'https://optimade.odbx.science',
     'omdb.omdb_production': 'http://optimade.openmaterialsdb.se',
     'oqmd': 'http://oqmd.org/optimade/',
     'tcod': 'https://www.crystallography.net/tcod/optimade'}

These aliases are useful since they can provide a quick shorthand for a given database without having to remember a full URL.

This list of aliases is updated periodically. However, new OPTIMADE providers can be made available and will be listed at https://providers.optimade.org. The `OptimadeRester` can query the OPTIMADE providers list to refresh the available aliases.

You can do this as follows, but be aware this might take a few moments:

```python
opt = OptimadeRester()
opt.refresh_aliases()
```

    Connecting to all known OPTIMADE providers, this will be slow. Please connect to only the OPTIMADE providers you want to query. Choose from: aflow, cod, mcloud.2dstructures, mcloud.2dtopo, mcloud.curated-cofs, mcloud.li-ion-conductors, mcloud.optimade-sample, mcloud.pyrene-mofs, mcloud.scdm, mcloud.sssp, mcloud.stoceriaitf, mcloud.tc-applicability, mcloud.threedd, mp, odbx, omdb.omdb_production, oqmd, tcod

    odbx: 100%|██████████| 54/54 [00:19<00:00, 26.75it/s]

# Connecting to one or more OPTIMADE providers

Let's begin by connecting to the Materials Project (`mp`) and Materials Cloud "2D Structures" (`mcloud.2dstructures`) databases.
By default pymatgen expects a server to reply within 5 seconds, some servers however require up to several minutes to process a querry.
You can therefore set the timeout to a different value (in seconds) if you get a "Read timed out" error.

```python
opt = OptimadeRester(["mp", "mcloud.2dstructures"],timeout=10)
```

We can find more information about the OPTIMADE providers we are connected to using the `describe()` method.

```python
print(opt.describe())
```

    OptimadeRester connected to:
    Provider(name='The Materials Project', base_url='https://optimade.materialsproject.org', description='The Materials Project OPTIMADE endpoint', homepage='https://materialsproject.org', prefix='mp')
    Provider(name='Materials Cloud', base_url='https://aiida.materialscloud.org/3dd/optimade', description='A platform for Open Science built for seamless sharing of resources in computational materials science', homepage='https://materialscloud.org', prefix='mcloud')

# Query for materials: binary nitrides case study

`OptimadeRester` provides an `get_structures` method. **It does not support all features of OPTIMADE filters** but is a good place to get started.

For this case study, we will search for materials containing nitrogen and that have two elements.

```python
results = opt.get_structures(elements=["N"], nelements=2)
```

    mp:   2%|▏         | 20/892 [00:00<?, ?it/s]

    mp:   4%|▍         | 40/892 [00:00<00:10, 80.59it/s]

    mp:   7%|▋         | 60/892 [00:00<00:16, 50.73it/s]

    mp:   9%|▉         | 80/892 [00:01<00:15, 52.98it/s]

    mp:  11%|█         | 100/892 [00:01<00:13, 57.95it/s]

    mp:  13%|█▎        | 120/892 [00:01<00:15, 50.05it/s]

    mp:  16%|█▌        | 140/892 [00:02<00:13, 54.44it/s]

    mp:  18%|█▊        | 160/892 [00:02<00:13, 54.04it/s]

    mp:  20%|██        | 180/892 [00:03<00:14, 47.85it/s]

    mp:  22%|██▏       | 200/892 [00:03<00:14, 46.13it/s]

    mp:  25%|██▍       | 220/892 [00:04<00:15, 44.63it/s]

    mp:  27%|██▋       | 240/892 [00:04<00:14, 43.67it/s]

    mp:  29%|██▉       | 260/892 [00:04<00:13, 46.71it/s]

    mp:  31%|███▏      | 280/892 [00:05<00:12, 49.17it/s]

    mp:  34%|███▎      | 300/892 [00:05<00:12, 46.48it/s]

    mp:  36%|███▌      | 320/892 [00:06<00:10, 52.24it/s]

    mp:  38%|███▊      | 340/892 [00:06<00:09, 56.87it/s]

    mp:  40%|████      | 360/892 [00:06<00:08, 59.89it/s]

    mp:  43%|████▎     | 380/892 [00:07<00:09, 52.13it/s]

    mp:  45%|████▍     | 400/892 [00:07<00:08, 55.45it/s]

    mp:  47%|████▋     | 420/892 [00:07<00:07, 59.55it/s]

    mp: 100%|██████████| 892/892 [07:30<00:00,  1.94it/s]
    mcloud.threedd: 100%|██████████| 87/87 [07:13<00:00,  6.02s/it]
    odbx: 100%|██████████| 54/54 [00:30<00:00,  1.11it/s]

    mp:  52%|█████▏    | 460/892 [00:08<00:08, 50.54it/s]

    mp:  54%|█████▍    | 480/892 [00:08<00:08, 48.59it/s]

    mp:  56%|█████▌    | 500/892 [00:09<00:07, 52.73it/s]

    mp:  58%|█████▊    | 520/892 [00:09<00:07, 52.49it/s]

    mp:  61%|██████    | 540/892 [00:09<00:06, 56.16it/s]

    mp:  63%|██████▎   | 560/892 [00:10<00:06, 55.00it/s]

    mp:  65%|██████▌   | 580/892 [00:10<00:05, 58.61it/s]

    mp:  67%|██████▋   | 600/892 [00:11<00:05, 57.08it/s]

    mp:  70%|██████▉   | 620/892 [00:11<00:04, 56.17it/s]

    mp:  72%|███████▏  | 640/892 [00:11<00:04, 58.87it/s]

    mp:  74%|███████▍  | 660/892 [00:12<00:04, 56.31it/s]

    mp:  76%|███████▌  | 680/892 [00:12<00:03, 58.23it/s]

    mp:  78%|███████▊  | 700/892 [00:12<00:03, 60.50it/s]

    mp:  81%|████████  | 720/892 [00:12<00:02, 62.29it/s]

    mp:  83%|████████▎ | 740/892 [00:13<00:02, 59.05it/s]

    mp:  85%|████████▌ | 760/892 [00:13<00:02, 55.66it/s]

    mp:  87%|████████▋ | 780/892 [00:14<00:01, 59.02it/s]

    mp:  90%|████████▉ | 800/892 [00:14<00:01, 61.37it/s]

    mp:  92%|█████████▏| 820/892 [00:14<00:01, 53.02it/s]

    mp:  94%|█████████▍| 840/892 [00:15<00:00, 59.87it/s]

    mp:  96%|█████████▋| 860/892 [00:15<00:00, 61.36it/s]

    mp:  99%|█████████▊| 880/892 [00:15<00:00, 62.68it/s]

    mcloud.threedd: 100%|██████████| 87/87 [00:02<00:00, 32.31it/s]

We see that the `OptimadeRester` does some of the hard work for us: it automatically retrieves multiple pages of results when many results are available, and also gives us a progress bar.

Let us inspect the `results`:

```python
type(results)  # this method returns a dictionary, so let's examine the keys of this dictionary...
```

    dict

```python
results.keys()  # we see that the results dictionary is keyed by provider/alias
```

    dict_keys(['mp', 'mcloud.threedd'])

```python
results['mp'].keys()  # and these are then keyed by that database's unique identifier
```

    dict_keys(['mp-1008610', 'mp-1019271', 'mp-1065394', 'mp-7790', 'mp-1015582', 'mp-1104441', 'mp-1019272', 'mp-1102235', 'mp-29973', 'mp-2245', 'mp-1014303', 'mp-1007824', 'mp-1245326', 'mp-973935', 'mp-834', 'mp-1018160', 'mp-1014298', 'mp-1214623', 'mp-1245013', 'mp-1244914', 'mp-27908', 'mp-32652', 'mp-1216472', 'mp-1014358', 'mp-1018733', 'mp-1009885', 'mp-1058019', 'mp-11660', 'mp-1209181', 'mp-510557', 'mp-1245132', 'mp-1009769', 'mvc-15478', 'mp-1865', 'mp-9981', 'mp-1077354', 'mp-10736', 'mp-1091417', 'mp-1245165', 'mp-1204550', 'mp-1245046', 'mp-661', 'mp-8101', 'mp-1001117', 'mp-2114', 'mp-1008833', 'mp-1008489', 'mp-1009483', 'mp-1188353', 'mp-1245244', 'mp-1077275', 'mp-1271276', 'mp-2828', 'mp-1078568', 'mp-1093992', 'mp-1213154', 'mp-2341', 'mvc-11235', 'mp-1014324', 'mp-1102114', 'mp-1244927', 'mp-634410', 'mp-567617', 'mp-1001826', 'mp-1002222', 'mp-1014995', 'mp-1072543', 'mp-1001916', 'mp-1064529', 'mp-754553', 'mp-1097011', 'mp-1096882', 'mp-1008921', 'mvc-13808', 'mp-1188721', 'mp-1933', 'mp-1097766', 'mp-754353', 'mp-986716', 'mp-1017550', 'mp-1096965', 'mp-1120744', 'mp-13480', 'mp-1180282', 'mp-1079947', 'mp-1212113', 'mp-1954', 'mp-510087', 'mp-27954', 'mp-999496', 'mp-476', 'mp-1018027', 'mp-1080201', 'mp-1096917', 'mp-1245010', 'mp-998899', 'mp-608071', 'mp-1014287', 'mp-1096962', 'mp-1218446', 'mp-1245263', 'mp-1009130', 'mp-1062817', 'mp-1220479', 'mp-251', 'mp-1018849', 'mp-1016048', 'mp-1079684', 'mp-1221793', 'mp-13175', 'mp-570604', 'mp-1009646', 'mp-1410', 'mp-973674', 'mp-1002207', 'mp-1188869', 'mp-1217167', 'mp-570471', 'mp-1060281', 'mp-1080202', 'mp-1179691', 'mp-2565', 'mp-504769', 'mvc-15427', 'mp-1047', 'mp-1096966', 'mp-1173453', 'mp-1019316', 'mp-1245258', 'mp-776470', 'mp-1077346', 'mp-1804', 'mp-2852', 'mp-637209', 'mp-1014264', 'mp-1014316', 'mp-1057758', 'mp-1208632', 'mvc-13729', 'mp-1016058', 'mp-1180215', 'mp-1180313', 'mp-804', 'mp-940', 'mp-984', 'mp-1097042', 'mp-1196421', 'mp-1245106', 'mp-530081', 'mp-1065265', 'mp-604884', 'mp-1008986', 'mp-1078389', 'mp-1015780', 'mp-754628', 'mp-1216464', 'mp-1940', 'mp-2131', 'mp-535', 'mvc-13927', 'mp-1180230', 'mp-1238814', 'mp-2599', 'mp-1014281', 'mp-1008826', 'mp-27488', 'mp-1244991', 'mp-570454', 'mp-779689', 'mp-684642', 'mp-1018734', 'mp-1014565', 'mp-1179882', 'mp-569461', 'mp-754556', 'mp-1009639', 'mp-1015065', 'mp-1096957', 'mp-1009640', 'mp-1077506', 'mp-510350', 'mp-973', 'mp-1017546', 'mp-10564', 'mp-2634', 'mp-568172', 'mp-1017532', 'mp-1013529', 'mp-1187906', 'mp-1200511', 'mp-1195863', 'mp-1271281', 'mp-13146', 'mp-685145', 'mp-1001844', 'mp-1209103', 'mp-1014379', 'mp-1102941', 'mp-1018852', 'mp-1056900', 'mp-1079760', 'mp-11607', 'mp-1193580', 'mp-1015026', 'mp-1097039', 'mp-13098', 'mp-1009649', 'mp-1096909', 'mp-8282', 'mp-1061865', 'mp-1580', 'mp-36891', 'mp-1180345', 'mp-616407', 'mp-883', 'mp-1080197', 'mp-1246153', 'mp-754629', 'mp-1008965', 'mp-1245175', 'mp-13117', 'mp-32590', 'mp-1008820', 'mp-1207010', 'mvc-13455', 'mp-1002229', 'mp-1009720', 'mp-1080242', 'mp-1080623', 'mp-1208477', 'mp-248', 'mp-2789', 'mp-6988', 'mp-971694', 'mp-1079585', 'mp-1204171', 'mp-1245228', 'mp-1096911', 'mp-1101174', 'mp-1181864', 'mp-570538', 'mp-999264', 'mp-1009596', 'mp-1184048', 'mp-1019052', 'mp-1069841', 'mp-1244984', 'mp-1244972', 'mp-32913', 'mp-1009734', 'mp-1222421', 'mp-607371', 'mp-1067139', 'mp-1245229', 'mp-676', 'mp-864647', 'mp-1066400', 'mp-1096924', 'mp-1225746', 'mp-925', 'mp-1014454', 'mp-1078828', 'mp-1105655', 'mp-755988', 'mp-1014214', 'mp-1080193', 'mp-1220310', 'mp-684592', 'mp-754434', 'mp-1009767', 'mp-1245130', 'mp-1244934', 'mp-569957', 'mp-1062345', 'mp-6933', 'mp-870', 'mp-999263', 'mp-1072167', 'mp-1080198', 'mp-1096961', 'mp-1245281', 'mp-999195', 'mp-1203253', 'mp-1205002', 'mp-1245322', 'mp-1078816', 'mp-999266', 'mp-1018948', 'mp-1104403', 'mp-1639', 'mp-581833', 'mp-1599', 'mp-1080203', 'mp-1220830', 'mp-20812', 'mp-504768', 'mp-601223', 'mp-8102', 'mp-1066162', 'mp-2033', 'mp-1208406', 'mp-2596', 'mp-999485', 'mp-1101901', 'mp-1193871', 'mp-18337', 'mp-999469', 'mp-1009219', 'mp-1009015', 'mp-1096918', 'mp-1201150', 'mp-1014478', 'mp-1018655', 'mp-999296', 'mp-1102429', 'mp-583712', 'mp-1018049', 'mp-1014345', 'mp-1077096', 'mp-1184774', 'mp-684692', 'mvc-13841', 'mp-1078791', 'mp-1719', 'mp-1002197', 'mp-637576', 'mp-999294', 'mp-2117', 'mp-567885', 'mp-571653', 'mp-1198545', 'mp-1009836', 'mp-1015612', 'mp-1244872', 'mp-973933', 'mp-980941', 'mp-22631', 'mp-6913', 'mp-999517', 'mp-2653', 'mp-1096890', 'mp-1014509', 'mp-1019239', 'mp-1180334', 'mp-1187056', 'mp-1189035', 'mp-1224501', 'mp-555136', 'mp-1002181', 'mp-1002182', 'mp-1014311', 'mp-1019086', 'mp-1104513', 'mp-12981', 'mp-1064647', 'mp-1097032', 'mp-1244946', 'mp-673635', 'mp-2701', 'mp-1018157', 'mp-1014347', 'mp-1014296', 'mp-1058586', 'mp-1180473', 'mp-1019083', 'mp-1079812', 'mp-1190322', 'mp-685209', 'mp-1225831', 'mp-1096899', 'mp-1078576', 'mp-1080172', 'mp-1096913', 'mp-1096900', 'mp-13173', 'mp-568293', 'mp-1180500', 'mp-569167', 'mp-972036', 'mp-1016073', 'mp-1212961', 'mp-1707', 'mp-1001834', 'mp-1330', 'mp-684887', 'mp-1008823', 'mp-1064119', 'mp-1066707', 'mp-1080199', 'mp-1009601', 'mp-1014342', 'mp-1014339', 'mp-1211954', 'mp-343', 'mp-672264', 'mp-1009078', 'mp-1077299', 'mp-1096893', 'mp-1188048', 'mp-982734', 'mp-1097757', 'mp-1180757', 'mp-1419', 'mp-1975', 'mp-998908', 'mp-1009237', 'mp-1096888', 'mp-1173647', 'mp-1183691', 'mp-33090', 'mp-641541', 'mp-1016069', 'mp-1202811', 'mp-1219567', 'mp-13035', 'mp-2118', 'mp-1009694', 'mp-563', 'mp-1009833', 'mp-1096935', 'mp-1244866', 'mp-1225758', 'mp-1096968', 'mp-1220726', 'mp-13099', 'mp-1776', 'mp-534', 'mp-1120742', 'mp-1190632', 'mp-1245149', 'mp-1244917', 'mp-20304', 'mp-680640', 'mp-1014332', 'mp-749', 'mp-1079708', 'mp-2493', 'mp-1195888', 'mp-1214748', 'mp-971684', 'mp-1014318', 'mp-1179538', 'mp-8046', 'mp-864918', 'mp-998903', 'mvc-11162', 'mvc-11231', 'mp-1014366', 'mp-1190039', 'mp-567290', 'mp-975606', 'mp-999357', 'mp-1009545', 'mp-1078609', 'mp-1096907', 'mp-40793', 'mp-620058', 'mp-1014910', 'mp-1019080', 'mp-1097803', 'mp-16031', 'mp-827', 'mp-1209413', 'mp-1245048', 'mp-1180438', 'mp-1245193', 'mp-1080190', 'mp-1245', 'mp-7927', 'mp-999495', 'mp-1057084', 'mp-1245002', 'mp-1206425', 'mp-1097059', 'mp-1245123', 'mp-685023', 'mp-1080194', 'mp-1192688', 'mp-641539', 'mp-999303', 'mp-1205065', 'mp-1245075', 'mp-13151', 'mp-754381', 'mp-1102', 'mp-22003', 'mp-999355', 'mp-1009548', 'mp-20340', 'mp-1642', 'mp-344', 'mp-651723', 'mp-11801', 'mp-1245172', 'mp-755880', 'mp-1014297', 'mp-1001', 'mp-1008600', 'mp-1014558', 'mp-1019077', 'mp-1095181', 'mp-1009657', 'mp-1245188', 'mp-1245295', 'mp-13148', 'mp-1194075', 'mp-998900', 'mp-1008918', 'mp-1096928', 'mp-1196867', 'mp-1245150', 'mp-2355', 'mvc-14553', 'mp-1096898', 'mp-1216242', 'mp-1245095', 'mp-13178', 'mp-13076', 'mp-2132', 'mp-1009831', 'mp-1007778', 'mp-1210764', 'mp-1096896', 'mp-1178600', 'mp-1218021', 'mp-13034', 'mp-15799', 'mp-988', 'mp-10196', 'mp-1058549', 'mp-1103129', 'mp-1182172', 'mp-1009766', 'mp-1096906', 'mp-1224686', 'mp-1019082', 'mp-467', 'mp-636839', 'mp-1008483', 'mp-1203301', 'mp-13132', 'mp-568867', 'mp-1019078', 'mp-1096914', 'mp-2659', 'mp-999197', 'mp-1058689', 'mp-1245301', 'mp-864675', 'mp-1244896', 'mp-2811', 'mp-567907', 'mp-6977', 'mp-1066', 'mp-1205986', 'mp-1245252', 'mp-1224383', 'mp-720515', 'mp-1009493', 'mp-1018947', 'mp-974435', 'mp-1014265', 'mp-1008798', 'mp-1064952', 'mp-1013524', 'mp-1009732', 'mp-1225203', 'mp-999117', 'mp-999292', 'mp-1096951', 'mp-568862', 'mp-1244916', 'mp-256', 'mp-1059612', 'mp-1080192', 'mp-1106142', 'mp-1185783', 'mp-20411', 'mp-985299', 'mp-1216967', 'mp-2251', 'mp-971657', 'mp-1016076', 'mp-1016071', 'mp-1097796', 'mp-7139', 'mp-981366', 'mp-991', 'mp-1009485', 'mp-34761', 'mp-643432', 'mp-1015076', 'mp-1173466', 'mp-1244943', 'mp-13150', 'mp-608366', 'mp-673174', 'mp-1096892', 'mp-1103420', 'mp-1245231', 'mp-1019319', 'mp-1245069', 'mp-1182484', 'mp-9460', 'mp-542738', 'mp-1059160', 'mp-1180220', 'mp-1057979', 'mp-1096919', 'mp-1103897', 'mp-1459', 'mp-1352', 'mp-1559', 'mp-235', 'mp-448', 'mp-1244977', 'mp-22777', 'mp-1009019', 'mp-1204579', 'mp-1216851', 'mp-1064161', 'mp-1245286', 'mp-21476', 'mp-1018142', 'mp-1019055', 'mp-1096889', 'mp-830', 'mp-1014286', 'mp-1096864', 'mp-1120827', 'mp-1700', 'mp-1221446', 'mp-29145', 'mp-1245233', 'mp-8780', 'mp-980940', 'mp-1203286', 'mp-570572', 'mp-1009005', 'mp-1009471', 'mp-1180808', 'mp-776343', 'mp-864757', 'mp-1077595', 'mp-1220519', 'mp-644751', 'mp-9410', 'mp-555', 'mp-1078219', 'mp-1244884', 'mp-571297', 'mp-685079', 'mp-1101173', 'mp-1102441', 'mp-999300', 'mp-1094090', 'mp-1097018', 'mp-22387', 'mp-685050', 'mp-1016078', 'mp-1245249', 'mp-999307', 'mp-1014373', 'mp-32584', 'mp-1095618', 'mp-32742', 'mp-415', 'mp-1056955', 'mp-1080032', 'mp-20839', 'mp-999293', 'mp-1066821', 'mp-1096908', 'mp-1245302', 'mp-1059879', 'mp-574424', 'mp-7234', 'mp-999473', 'mp-1009049', 'mp-1018716', 'mp-1203025', 'mp-236', 'mp-1009770', 'mp-1064706', 'mp-1198478', 'mp-1207164', 'mp-23922', 'mp-7991', 'mp-999317', 'mp-1105105', 'mp-1205447', 'mp-636056', 'mp-999086', 'mp-1244886', 'mp-27953', 'mp-542635', 'mvc-15387', 'mp-1016070', 'mp-1217984', 'mp-1009818', 'mp-1077232', 'mp-1080196', 'mp-1096886', 'mvc-13461', 'mp-569228', 'mp-971683', 'mp-1008929', 'mp-1009838', 'mp-1245148', 'mp-1245287', 'mp-684744', 'mvc-15664', 'mp-1013528', 'mp-1080200', 'mp-1206884', 'mp-1279', 'mp-2853', 'mp-973785', 'mp-1892', 'mp-1008987', 'mp-1009494', 'mp-1180208', 'mp-1244995', 'mp-1245181', 'mp-1014333', 'mp-1097738', 'mp-13174', 'mp-672289', 'mp-1018018', 'mp-1016081', 'mp-1147700', 'mp-1245030', 'mp-1071980', 'mp-1187591', 'mp-22205', 'mp-32994', 'mp-999503', 'mp-1009496', 'mp-1064272', 'mp-1096894', 'mp-1101868', 'mp-1014444', 'mp-1071531', 'mp-1080191', 'mp-1102074', 'mvc-15384', 'mp-27943', 'mp-2718', 'mp-1078930', 'mp-1244903', 'mp-569655', 'mp-971682', 'mp-1206907', 'mp-973835', 'mp-1188347', 'mp-1212209', 'mp-1007660', 'mp-1104073', 'mp-1008922', 'mp-1016063', 'mp-1067644', 'mp-1173210', 'mp-1103427', 'mp-1217048', 'mp-1008762', 'mp-1019238', 'mp-1070456', 'mvc-13303', 'mp-1008985', 'mp-1014160', 'mp-1002209', 'mp-866083', 'mp-1096901', 'mp-13116', 'mp-2075', 'mp-1096885', 'mp-1188944', 'mp-1224391', 'mp-674509', 'mp-2857', 'mp-31000', 'mp-1015567', 'mp-1014369', 'mp-1016059', 'mp-1071868', 'mp-13126', 'mp-999295', 'mp-1009488', 'mp-1096994', 'mp-1180489', 'mp-2639', 'mp-492', 'mp-1224698', 'mp-540572', 'mp-277', 'mp-743', 'mp-1017982', 'mp-1017533', 'mp-1096903', 'mp-1101178', 'mp-1079438', 'mp-1244975', 'mp-12857', 'mp-603694', 'mp-667338', 'mp-1072204', 'mp-1102371', 'mp-844', 'mp-1014993', 'mp-1200550', 'mp-1224388', 'mp-1771', 'mp-1008558', 'mp-1001828', 'mp-1015908', 'mp-1079251', 'mp-1096931', 'mp-12120', 'mp-1245297', 'mvc-13772', 'mp-1214163', 'mp-22762', 'mp-1096910', 'mp-1096912', 'mp-1245288', 'mp-21264', 'mp-1217441', 'mp-13036', 'mp-776321', 'mvc-11196', 'mp-1058151', 'mp-1101975', 'mp-1179622', 'mp-1220496', 'mp-27461', 'mp-13852', 'mp-2686', 'mp-1009750', 'mp-1188283', 'mp-33046', 'mp-971911', 'mp-1182030', 'mvc-14624', 'mp-1017987', 'mp-1080204', 'mp-1245014', 'mp-2247', 'mp-571281', 'mp-1180221', 'mp-1225892', 'mp-1008923', 'mp-1002117', 'mp-1018648', 'mp-557865', 'mp-1225717', 'mp-1245033', 'mp-505622', 'mp-1014365', 'mp-1188007', 'mp-1002105', 'mp-11661', 'mp-1221525', 'mp-1080195', 'mp-629015', 'mp-1009128', 'mp-1009495', 'mp-1094084', 'mp-1189239', 'mp-1008809', 'mp-1009221', 'mp-1059462', 'mp-999549', 'mp-1245138', 'mp-1985', 'mp-570155', 'mp-16730', 'mp-1497', 'mp-1063003', 'mp-1096922', 'mp-1100774', 'mp-1102681', 'mp-1244998', 'mp-1018851', 'mp-1064715', 'mp-1096934', 'mp-1226927', 'mp-19830', 'mp-1244870', 'mp-1205092', 'mp-866056', 'mp-1014460', 'mp-1219988'])

So let us inspect one structure as an example:

```python
example_structure = results['mp']['mp-804']
print(example_structure)
```

    Full Formula (Ga2 N2)
    Reduced Formula: GaN
    abc   :   3.216290   3.216290   5.239962
    angles:  90.000000  90.000000 120.000003
    Sites (4)
      #  SP           a         b        c
    ---  ----  --------  --------  -------
      0  Ga    0.666667  0.333333  0.49912
      1  Ga    0.333333  0.666667  0.99912
      2  N     0.666667  0.333333  0.87588
      3  N     0.333333  0.666667  0.37588

We can then use *pymatgen* to further manipulate these `Structure` objects, for example to calculate the spacegroup or to convert to a CIF:

```python
example_structure.get_space_group_info()
```

    ('P6_3mc', 186)

```python
print(example_structure.to(fmt="cif", symprec=0.01))
```

    # generated using pymatgen
    data_GaN
    _symmetry_space_group_name_H-M   P6_3mc
    _cell_length_a   3.21629007
    _cell_length_b   3.21629007
    _cell_length_c   5.23996200
    _cell_angle_alpha   90.00000000
    _cell_angle_beta   90.00000000
    _cell_angle_gamma   120.00000000
    _symmetry_Int_Tables_number   186
    _chemical_formula_structural   GaN
    _chemical_formula_sum   'Ga2 N2'
    _cell_volume   46.94282137
    _cell_formula_units_Z   2
    loop_
     _symmetry_equiv_pos_site_id
     _symmetry_equiv_pos_as_xyz
      1  'x, y, z'
      2  'x-y, x, z+1/2'
      3  '-y, x-y, z'
      4  '-x, -y, z+1/2'
      5  '-x+y, -x, z'
      6  'y, -x+y, z+1/2'
      7  'y, x, z+1/2'
      8  'x, x-y, z'
      9  'x-y, -y, z+1/2'
      10  '-y, -x, z'
      11  '-x, -x+y, z+1/2'
      12  '-x+y, y, z'
    loop_
     _atom_site_type_symbol
     _atom_site_label
     _atom_site_symmetry_multiplicity
     _atom_site_fract_x
     _atom_site_fract_y
     _atom_site_fract_z
     _atom_site_occupancy
      Ga  Ga0  2  0.33333333  0.66666667  0.99912000  1
      N  N1  2  0.33333333  0.66666667  0.37588000  1

# Data analysis

This section I will use some code I prepared earlier to summarize the `results` into a tabular format (`DataFrame`).

```python
import pandas as pd
```

```python
records = []
for provider, structures in results.items():
    for identifier, structure in structures.items():
        records.append({
            "provider": provider,
            "identifier": identifier,
            "formula": structure.composition.reduced_formula,
            "spacegroup": structure.get_space_group_info()[0],
            "a_lattice_param": structure.lattice.a,
            "volume": structure.volume,
        })
df = pd.DataFrame(records)
```

```python
df
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>provider</th>
      <th>identifier</th>
      <th>formula</th>
      <th>spacegroup</th>
      <th>a_lattice_param</th>
      <th>volume</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>mp</td>
      <td>mp-1008610</td>
      <td>AuN</td>
      <td>Fm-3m</td>
      <td>3.302053</td>
      <td>25.458750</td>
    </tr>
    <tr>
      <th>1</th>
      <td>mp</td>
      <td>mp-1019271</td>
      <td>TaN2</td>
      <td>P6/mmm</td>
      <td>5.459583</td>
      <td>72.277001</td>
    </tr>
    <tr>
      <th>2</th>
      <td>mp</td>
      <td>mp-1065394</td>
      <td>MoN</td>
      <td>P6_3/mmc</td>
      <td>2.870508</td>
      <td>40.722201</td>
    </tr>
    <tr>
      <th>3</th>
      <td>mp</td>
      <td>mp-7790</td>
      <td>Ti2N</td>
      <td>I4_1/amd</td>
      <td>5.322762</td>
      <td>76.524076</td>
    </tr>
    <tr>
      <th>4</th>
      <td>mp</td>
      <td>mp-1015582</td>
      <td>CrN2</td>
      <td>Pbcn</td>
      <td>4.405213</td>
      <td>118.349470</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>974</th>
      <td>mcloud.threedd</td>
      <td>25786</td>
      <td>N2O</td>
      <td>Pa-3</td>
      <td>5.842646</td>
      <td>199.447562</td>
    </tr>
    <tr>
      <th>975</th>
      <td>mcloud.threedd</td>
      <td>26371</td>
      <td>SbN9</td>
      <td>P-1</td>
      <td>7.350390</td>
      <td>322.370944</td>
    </tr>
    <tr>
      <th>976</th>
      <td>mcloud.threedd</td>
      <td>26467</td>
      <td>SmN</td>
      <td>Fm-3m</td>
      <td>3.559131</td>
      <td>31.879910</td>
    </tr>
    <tr>
      <th>977</th>
      <td>mcloud.threedd</td>
      <td>26627</td>
      <td>CsN</td>
      <td>Pm-3m</td>
      <td>3.821708</td>
      <td>55.817766</td>
    </tr>
    <tr>
      <th>978</th>
      <td>mcloud.threedd</td>
      <td>26808</td>
      <td>GdN</td>
      <td>Fm-3m</td>
      <td>3.527077</td>
      <td>31.026289</td>
    </tr>
  </tbody>
</table>
<p>979 rows × 6 columns</p>
</div>

To pick one specific formula as an example, we can use tools from `pandas` to show the spacegroups present for that formula:

```python
df[df["formula"] == "GaN"].spacegroup
```

    11     P6_3/mmc
    12           P1
    18           P1
    30           P1
    38           P1
    40           P1
    49           P1
    60           P1
    128          P1
    142      P6_3mc
    147          P1
    239      P6_3mc
    246          P1
    270          P1
    271          P1
    420          P1
    471          P1
    481          P1
    493          P1
    502          P1
    503          P1
    510          P1
    515          P1
    549          P1
    571          P1
    601          P1
    621          P1
    626       F-43m
    633          P1
    649          P1
    673          P1
    693          P1
    708          P1
    716       Fm-3m
    722          P1
    723          P1
    731          P1
    749          P1
    801          P1
    824          P1
    855          P1
    872          P1
    881          P1
    969      P6_3mc
    Name: spacegroup, dtype: object

Here, we see that there are a few common high-symmetry spacegroups (such as $P6_3mc$) there are also many low-symmetry structures ($P1$).

I know that in this instance, this is because the $P1$ structures are actually amorphous and not crystalline. This highlights the importance of doing appropraiate **data cleaning** on retrieved data.

### Plotting data

As a quick example, we can also plot information in our table:

```python
import plotly.express as px
```

```python
px.bar(df, x="spacegroup", facet_row="provider")
```

<html>
<head><meta charset="utf-8" /></head>
<body>
    <div>
                        <div id="a12b656a-4ddc-4b53-bd4d-4dbcc6f61374" class="plotly-graph-div" style="height:525px; width:100%;"></div>                    </div>
</body>
</html>

**Remember, there is no single "best database" to use. Every database might be constructed for a specific purpose, subject to different biases, with different data qualities and sources.**

The ideal database for one scientist with one application in mind may be different to the ideal database for another scientist with a different application.

**The power of OPTIMADE is that you can query across multiple databases!**

# Advanced usage: querying using the OPTIMADE filter grammar

You can also query using an OPTIMADE filter as defined in the OPTIMADE specification and publication.

**This is recommended** for advanced queries to use the full power of OPTIMADE.

For example, the above query could have equally been performed as:

```python
results = opt.get_structures_with_filter('(elements HAS ALL "N") AND (nelements=2)')
```

    mp:   2%|▏         | 20/892 [00:00<?, ?it/s]
    mp:   4%|▍         | 40/892 [00:00<00:11, 72.90it/s]
    mp:   7%|▋         | 60/892 [00:00<00:11, 72.88it/s]
    mp:   9%|▉         | 80/892 [00:00<00:11, 72.95it/s]
    mp:  11%|█         | 100/892 [00:01<00:12, 65.69it/s]
    mp:  13%|█▎        | 120/892 [00:01<00:14, 54.45it/s]

    mp: 100%|██████████| 892/892 [00:27<00:00, 47.21it/s]
    mp:  16%|█▌        | 140/892 [00:02<00:13, 54.42it/s]
    mp:  18%|█▊        | 160/892 [00:02<00:13, 54.52it/s]
    mp:  20%|██        | 180/892 [00:02<00:12, 57.95it/s]
    mp:  22%|██▏       | 200/892 [00:02<00:11, 61.80it/s]
    mp:  25%|██▍       | 220/892 [00:03<00:10, 63.65it/s]
    mp: 100%|██████████| 892/892 [00:29<00:00, 29.79it/s]
    mcloud.threedd: 100%|██████████| 87/87 [00:12<00:00,  5.91it/s]

    mp:  29%|██▉       | 260/892 [00:03<00:10, 57.87it/s]
    mp:  31%|███▏      | 280/892 [00:04<00:10, 57.30it/s]
    mp:  34%|███▎      | 300/892 [00:04<00:09, 60.09it/s]
    mp:  36%|███▌      | 320/892 [00:04<00:09, 62.86it/s]
    mp:  38%|███▊      | 340/892 [00:05<00:08, 64.44it/s]
    mp:  40%|████      | 360/892 [00:05<00:08, 64.95it/s]
    mp:  43%|████▎     | 380/892 [00:05<00:07, 70.98it/s]
    mp:  45%|████▍     | 400/892 [00:06<00:06, 70.61it/s]
    mp:  47%|████▋     | 420/892 [00:06<00:06, 70.85it/s]
    mp:  49%|████▉     | 440/892 [00:06<00:06, 68.00it/s]
    mp:  52%|█████▏    | 460/892 [00:06<00:06, 69.75it/s]
    mp:  54%|█████▍    | 480/892 [00:07<00:06, 63.08it/s]
    mp:  56%|█████▌    | 500/892 [00:07<00:06, 64.69it/s]
    mp:  58%|█████▊    | 520/892 [00:07<00:05, 65.02it/s]
    mp:  61%|██████    | 540/892 [00:08<00:05, 67.29it/s]
    mp:  63%|██████▎   | 560/892 [00:08<00:05, 62.68it/s]
    mp:  65%|██████▌   | 580/892 [00:08<00:04, 62.63it/s]
    mp:  67%|██████▋   | 600/892 [00:09<00:04, 58.89it/s]
    mp:  70%|██████▉   | 620/892 [00:09<00:04, 57.19it/s]
    mp:  72%|███████▏  | 640/892 [00:09<00:04, 58.50it/s]
    mp:  74%|███████▍  | 660/892 [00:10<00:03, 62.53it/s]
    mp:  76%|███████▌  | 680/892 [00:10<00:03, 54.97it/s]
    mp:  78%|███████▊  | 700/892 [00:10<00:03, 58.01it/s]
    mp:  81%|████████  | 720/892 [00:11<00:02, 60.39it/s]
    mp:  83%|████████▎ | 740/892 [00:11<00:03, 49.75it/s]
    mp:  85%|████████▌ | 760/892 [00:12<00:02, 51.13it/s]
    mp:  87%|████████▋ | 780/892 [00:12<00:02, 55.38it/s]
    mp:  90%|████████▉ | 800/892 [00:12<00:01, 58.72it/s]
    mp:  92%|█████████▏| 820/892 [00:13<00:01, 53.50it/s]
    mp:  94%|█████████▍| 840/892 [00:13<00:00, 56.96it/s]
    mp:  96%|█████████▋| 860/892 [00:13<00:00, 63.27it/s]
    mp:  99%|█████████▊| 880/892 [00:14<00:00, 64.92it/s]
    mcloud.threedd: 100%|██████████| 87/87 [00:02<00:00, 31.86it/s]

# Advanced usage: retrieving provider-specific property information

The OPTIMADE specification allows for providers to include database-specific information in the returned data, prefixed by namespace.

To access this information with *pymatgen* we have to request "snls" (`StructureNL`) instead of "structures". A `StructureNL` is a `Structure` with additional metadata included, such as the URL it was downloaded from and any of this additional database-specific information.

```python
results_snls = OptimadeRester("odbx").get_snls(nelements=2, additional_response_fields=["_odbx_thermodynamics"])
```

    odbx:  37%|███▋      | 20/54 [00:00<?, ?it/s]

    odbx:  74%|███████▍  | 40/54 [00:00<00:00, 31.84it/s]

    odbx: 100%|██████████| 54/54 [00:01<00:00, 27.29it/s]

```python
example_snl = results_snls['odbx']['odbx/2']
```

```python
example_snl.data['_optimade']['_odbx_thermodynamics']
```

    {'enthalpy': -816.9762466666666,
     'formation_energy': -0.39885422222209854,
     'hull_distance': 0.0004641666667737354,
     'relative_enthalpy': None,
     'total_energy': -816.9727734428334}

This extra data provided differs from every database, and sometimes from material to material, so some exploration is required!

# When Things Go Wrong and How to Get Help

Bugs may be present! The `OptimadeRester` is still fairly new.

If it does not work it is likely because of either:

* A bug in the *pymatgen* code. This may be reported directly to Matthew Horton at mkhorton@lbl.gov or an issue can be opened in the *pymatgen* code repository. Matt apologises in advance if this is the case!

* An issue with a provider. This may be because the provider does not yet fully follow the OPTIMADE specification, because the provider is suffering an outage, or because the filters are not yet optimized with that provider.

    * If this happens, you may try to first increase the `timeout` value to something larger. The default is too low for some providers.

    * Otherwise, you may want to contact the provider directly, or create a post at the OPTIMADE discussion forum: https://matsci.org/optimade

# How to Get Involved

New developers are very welcome to add code to *pymatgen*! If you want to get involved, help fix bugs or add new features, your help would be very much appreciated. *pymatgen* can only exist and be what it is today thanks to the many efforts of its [development team](https://pymatgen.org/team.html).


## Utils

---

You are currently looking at **version 1.0** of this notebook.
Project/objective:
Date: 24-may-2018
Author: Frank Ebbers
Contributors:
License: MIT https://opensource.org/licenses/MIT

---

## Changelog

-

Useful resources:

https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

http://blog.juliusschulz.de/blog/ultimate-ipython-notebook

http://nbviewer.jupyter.org/github/jupyter/nbconvert/blob/master/docs/source/customizing.ipynb

```python
# %pylab inline         # imports numpy, pandas, scipy, matplotlib inline
# %matplotlib inline    # shows plots in notebook
# %matplotlib notebook  # same, interactive but slower
```

### Markdown on Output

```python
from IPython.display import display, Markdown, Latex
text = '**some markdown** `truly nice` $\phi$ {}'.format(' what the hack')
display(Markdown(text))
```

```python
def printmd(text):
    """Prints out as markdown
    Example:
    text = '**some markdown** `truly nice` $\phi$ {}'.format(' what the hack')
    """
    from IPython.display import display, Markdown, Latex
    return display(Markdown(text))
printmd(text)
```

```python
def printmd(text):
    """Prints out as markdown
    Examples:
    md_text = '** green::some markdown ** `truly nice` $\phi$ {}'.format(' red::hack')
    colored_text = 'red::green'
    """
    import re
    from IPython.display import display, Markdown, Latex

    try:
        for col_item in re.findall(r"(\w+::\S+)", text):
            col, item = col_item.split('::')
            text = re.sub(col_item, '<span style="color:{}">{}</span>'.format(col, item), text)
    except: pass
    return display(Markdown(text))

md_text = '** green::some markdown ** `truly nice` $\phi$ {}'.format(' red::hack')
printmd(md_text)
printmd('** {}::{} **'.format('green', 'bold_green_text'))
printmd('<green::red> more text')
printmd('red::red, black:orange::orange : blue: :green::c')

```

```python
# in flask env
!touch requirements.txt && pip freeze > requirements.txt
```

```python
# Running servers
!jupyter notebook list
```

```python
# Automatically reload modules - for working with own modules
# https://ipython.org/ipython-doc/3/config/extensions/autoreload.html
%load_ext autoreload
%autoreload 2
```

```python
!pwd
```

```python
# Ipython.__file__
```

```python
%psearch?
```

```python
%config
# https://ipython.org/ipython-doc/3/interactive/magics.html
```

### Load aliases
from my_lib/aliases.py

```python
%alias aliases \%load my_lib/aliases.py
```

```python
%aliases
```

```python
# %load my_lib/aliases.py
# Generic aliases
%alias show echo
%alias mop echo Monty Python
%alias tree pwd && tree . %s
%alias find find ~ | grep -i %s
%alias find.. find .. | grep -i %s
%alias find. find . | grep -i %s
%alias findx find ~ | grep -Ei %s
%alias read cat %s
%alias aug ls -al | grep Aug | grep -v 2018

# !brew install cowsay
%alias say cowsay
%alias_magic t timeit

# Create directory with packages for importing
%alias make_lib mkdir my_lib && touch my_lib/__init__.py
%alias add_lib cp %s my_lib/my_module.py
%alias to_lib mkdir my_lib && touch my_lib/__init__.py && cp %s my_lib/my_module.py
```

---
## Setup notebook
---

### Import the generic libraries used in this notebook

```python
import numpy as np
import pandas as pd

%matplotlib inline
# import matplotlib
import matplotlib.pyplot as plt

from collections import OrderedDict, Counter
import string
import re
import datetime
import requests
import json
import pprint
```

### Import interactive widgets library

```python
from ipywidgets import interact, fixed
from IPython.html import widgets
from IPython.display import HTML, IFrame
```

### Toggle code dispay

```python
HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Show/hide code"></form>''')
```

### Manage warnings

```python
import warnings
warnings.filterwarnings("ignore")
# warnings.filterwarnings("once") # always will print every occurance
```

### Set defaults and constants

```python
# seed the random number generator so we all get the same results
rand_seed = 2018
np.random.seed(rand_seed)
epsy = 1e-10
```

```python
matplotlib.rc('xtick', labelsize=14)
matplotlib.rc('ytick', labelsize=14)
```

### Check current working directory and file structure

```python
%%!
# directories with token
ls -FGlAhp | grep 'ml'
```

```python
today = datetime.now().strftime(r'%Y-%m-%d')
dirname = os.path.join('../_data/news-{}'.format(today, '_name'))
os.makedirs(dirname, exist_ok=True)
```

```python
def sys_info():
    !echo "You are logged on:$NC "; scutil --get ComputerName
    !echo "The current datetime is:$NC " ; date
    !echo  "\nMachine stats:$NC " ; uptime; macstats
    !echo  "\nDisk usage:$NC "; df -l
    !echo "\nPublic facing IP address:$NC "; myip
    !echo
```

```python
sys_info()
```

```python
# !brew install cowsay
!cowsay 'Hello Awesome Notebook'
```

```python
# !brew install tree
!tree . | head -10
```

```python
!find .. | grep -i edit
```

```python
!pwd
!ls data | head -5
```

```python
# Find file
def get_file_path(filename, first=5):
    !pwd
    !find . | grep '$filename' | head -'$first'
    return None
```

```python
filename = 'full_set.txt'
get_file_path(filename, 10)
```

### Run helper notebooks

```python
%run 'organ_helper.py'
```

---
## Utils
---

```python
# Mute terminal Output / Out[]:

_ = 3
3;
```

### Debug, test,

```python
%timeit
```

### Save as Excel spreadsheet

```python
writer = pd.ExcelWriter('data/excel_test.xlsx')
df = pd.DataFrame(np.ones(10).reshape(-1, 5))

# index startrow/startcol N means value is inserted at N+1
df.to_excel(writer, 'sheet_name', startrow=5, startcol=5, header=False, index=False)
writer.save()
```

### Print output at same position

```python
print('\b\b\b{}'.format(i), sep='', end='', flush=True)
# or
print('{}{}'.format('\b'*len(str(len(text))), len(text)), end='')
```

### Download files from list

```python
!cat url-list.txt | xargs wget –c
```

### Show all Out from cell - iso only the last

```python
from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"
```

```python
# !find ~ | grep .ipython/profile_default
```

#### Run all nodes interactively for all instances of Jupyter (Notebook and Console)

 - simply create a file ~/.ipython/profile_default/ipython_config.py with the lines below.

```python
%%writefile ~/.ipython/profile_default/ipython_config.py
get_config()

# Run all nodes interactively
c.InteractiveShell.ast_node_interactivity = "all"
```

```python
!cat ~/.ipython/profile_default/ipython_config.py
```

### Open TextEdit to edit file

```python
!open -e $ipython_config
```

### RISE slideshow

```python
!pip install RISE
```

```python
!jupyter-nbextension install rise --py --sys-prefix
```

```python
!jupyter-nbextension enable rise --py --sys-prefix
```

```python
# matplotlib legend without repeat
```

```python
# Legend without repeat
handles, labels = plt.gca().get_legend_handles_labels()
by_label = OrderedDict(zip(labels, handles))
plt.legend(by_label.values(), by_label.keys())
```

```python
plt.rcParams.find_all
```

```python
%%writefile ../../_data/standard_import.txt

%matplotlib inline
plt.style.use('seaborn-white')

import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
# %load ../_data/standard_import.txt

%matplotlib inline
plt.style.use('seaborn-white')

import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
```

```python

# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.set_option.html
# pd.set_option('display.notebook_repr_html', False)
```

 **Code**
```python
# test function
testy_L2 = NN_L2(trainx, trainy, testx)
print( type( testy_L2) )
print( len(testy_L2) )
print( testy_L2[40:50] )
```

 **Output**
```
<class 'numpy.ndarray'>
62
[ 2.  2.  1.  0.  0.  2.  0.  0.  0.  0.]
```

```python
train_digits, train_counts = np.unique(train_labels, return_counts=True)
print("Training set distribution:")
print(dict(zip(train_digits, train_counts)))
```

```python
def tree_plot(clf, features, labels=None):
    """Plot decision tree in Notebook"""
    from sklearn import tree
    import graphviz
    from pydotplus import graph_from_dot_data

    dot_data = tree.export_graphviz(clf, out_file=None, feature_names=features, class_names=labels,
                             filled=True, rounded=True, special_characters=True, impurity=True, proportion=False)
    return Image(graph_from_dot_data(dot_data).create_png())
```

```python
# nxn similarity matrix
x = np.arange(-5, 5.1, 0.1)
y = np.arange(-4, 4.1, 0.1)

K = np.zeros((x.shape[0], y.shape[0]))
for (i,j), _ in np.ndenumerate(K):
    K[i,j] = (x[i] - y[j])
K
```

```python

import numpy as np
import matplotlib.pyplot as plt

# Create all possible pairs (x, y)
x = np.arange(-1, 1.1, 0.1)
y = np.arange(-2, 2.1, 0.1)
xx, yy = np.meshgrid(x, y, sparse=False)
xx[:3, :3]
yy[:3, :3]
[(x.round(1), y.round(1)) for x, y in zip(xx.ravel(), yy.ravel())][:25]

z = np.sin(xx**2 + yy**2) / (xx**2 + yy**2)

print(x.shape, y.shape, xx.shape, yy.shape, z.shape)
h = plt.contourf(x, y, z)
```

```python
def xy_grid(x, y, ax_pad=0, density=0.1):
    """returns grid of (xx, yy) pairs, w.r.t. density and padding"""
    xmin, xmax = min(x)-ax_pad, max(x)+ax_pad
    ymin, ymax = min(y)-ax_pad, max(y)+ax_pad
    xx, yy = np.meshgrid(np.arange(xmin, xmax+density, density), np.arange(ymin, ymax+density, density))
    return np.c_[xx.ravel(), yy.ravel()]

x = np.arange(-1, 1.1)
y = np.arange(-2, 2.1)
xy_grid(x, y, ax_pad=0, density=0.05)
```

```python
def xy_grid(x, y, ax_pad=0, density=0.1):
    """returns grid of (xx, yy) pairs, w.r.t. density and padding"""
    xmin, xmax = min(x)-ax_pad, max(x)+ax_pad
    ymin, ymax = min(y)-ax_pad, max(y)+ax_pad
    xx, yy = np.meshgrid(np.arange(xmin, xmax+density, density), np.arange(ymin, ymax+density, density))
    return {'array': np.c_[xx.ravel(), yy.ravel()],
            'xx': xx, 'yy': yy,
            'xmin': xmin, 'xmax': xmax,
            'ymin': ymin, 'ymax': ymax}
```

```python
def confusion_df(y_true, y_pred):
    cm = confusion_matrix(y_true, y_pred)
    cm_df = pd.DataFrame(cm.T, index=svc.classes_, columns=svc.classes_)
    cm_df.index.name = 'Predicted'
    cm_df.columns.name = 'True'
    return cm_df
```

```python

```

```python
# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)
cm
# Accuracy
np.sum(np.eye(2) * cm) / np.sum(cm)
```

```python
# confusion matrix/pivot table
def pivot(df, ftr1, ftr2):
    return df.groupby([ftr1,ftr2]).size().unstack(ftr2).T
```

```python
def pivot(y, y_pred):
    df = pd.DataFrame({'True label': y, 'Predicted label': y_pred})
    return df.groupby(['True label', 'Predicted label']).size().unstack('Predicted label').T
```

```python
def pivot(y, y_pred, labels=['Negative', 'Positive']):
    dict_labels = {k:v for k, v in zip(np.unique(y), labels)}
    df = pd.DataFrame({y.name: y, y_pred.name: y_pred})
    df.replace(to_replace=dict_labels, inplace=True)
    return df.groupby([y.name, y_pred.name]).size().unstack(y_pred.name).T
```

```python
def pivot(y, y_pred, labels=['Negative', 'Positive']):
    dict_labels = {k:v for k, v in zip(np.unique(y), labels)}
    try:
        y_name, y_pred_name = y.name, y_pred.name
    except:
        y_name, y_pred_name = 'True label', 'Predicted label'
    df = pd.DataFrame({y_name: y, y_pred_name: y_pred})
    df.replace(to_replace=dict_labels, inplace=True)
    return df.groupby([y_name, y_pred_name]).size().unstack(y_pred_name)
```

```python
# Workaround to fix bug in statsmodels .summary()
# https://github.com/statsmodels/statsmodels/issues/3931
from scipy import stats
stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)
```

## Bug fixes - statsmodels not compatible with current stable version of scipy/pandas

```python
# Statsmodels bug fix:
from pandas.core import datetools

import statsmodels.api as sm
import statsmodels.formula.api as smf

# Workaround to fix bug in statsmodels .summary() - missing stats.chisqprob function
# https://github.com/statsmodels/statsmodels/issues/3931
from scipy import stats
stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)
```

```python
# parse datetime
df = pd.read_csv('../../_data/Smarket.csv', usecols=range(1, 10), index_col=0, parse_dates=True)
# select columns
pd.read_csv('../../_data/Advertising.csv', usecols=[1,2,3,4])
pd.read_csv('../../_data/Credit.csv', usecols=list(range(1,12)))
pd.read_csv('../../_data/Auto.csv', na_values='?').dropna()
```

```python
# Probability counter with threshold
thres = 0.9
band = [0.4, 0.6]

pred_p = clf.predict_proba(X_test)
np.unique(pred_p[:, 1]>thres, return_counts=True)
np.unique(band[0] > pred_p[:, 1] > band[-1], return_counts=True)
```

```python
margin = 0.03
np.unique(pred_p[:, 1] < 0.5+margin, return_counts=True), 'smaller than .5+margin'
np.unique(0.5-margin < pred_p[:, 1], return_counts=True), 'bigger than .5-margin'
'Predictions between margin({}) [false true] : {}'.format(margin,
np.unique(pred_p[:, 1] < 0.5+margin, return_counts=True)[1] + np.unique(0.5-margin < pred_p[:, 1], return_counts=True)[1])
```

```python
np.set_printoptions(suppress=True, precision=3, threshold=10)
# np.set_printoptions()  # formatter gets reset
```

```python
# Output numpy as laTex
import IPython
ip = IPython.core.getipython.get_ipython()
from IPython.display import display, Math, Latex
import sympy as sp
from IPython.display import Latex, display, display_latex
from sympy.interactive import printing
printing.init_printing(use_latex='png')

from sympy import Matrix as spx
sp.init_printing(use_unicode=True)
# sp.init_printing(use_latex='png')
ip.display_formatter.formatters['text/latex'].enabled = True

spx([[1, -1], [3, 4], [0, 2]])
```

```python
# http://docs.sympy.org/0.7.2/tutorial.html#printing-tutorial
from sympy.printing import print_latex as plx
from sympy.printing import print_mathml as plm

from sympy.printing import latex as slx
from sympy.printing import mathml as slm

ip.display_formatter.formatters['text/latex'].enabled = True

plx([[1, -1], [3, 4], [0, 2]])
ltx = slx([[1, -1],
           [3, 4],
           [0, 2]])
display(Latex('${}$'.format(ltx)))

display(Latex(slx([[1, -1], [3, 4], [0, 2]])))

```

```python
import IPython
ip = IPython.core.getipython.get_ipython()

# Sympy are setting:
ip.display_formatter.formatters['text/latex'].enabled = False
# to disable all latex output. You can revert this with
ip.display_formatter.formatters['text/latex'].enabled = True
```

```python
from IPython.display import Latex, display, display_latex
from sympy.interactive import printing
printing.init_printing(use_latex='png')
display(Latex('$\\alpha^2 + \\eta$'))

display(Latex(slx([[1, -1], [3, 4], [0, 2]])))
```

```python
# set precision
pd.set_option('float_format', '{:.2f}'.format)
```

```python
# qgrid.version_info
!jupyter notebook --version
!jupyter lab --version
!pip list | grep widgets
!pip list | grep jupyter
!pip list | grep qgrid
```

```python
N = 5
Z = np.zeros(N*N).reshape(-1, N)
Z = [distance(MNIST.data[i], MNIST.data[j]) for (i,j), v in np.ndenumerate(np.zeros(N*N).reshape(-1, N))]
Z = np.array(Z).reshape(-1, N)
Z
```

```python
N = 5
Z = np.zeros(N*N).reshape(-1, N)
for (i,j), v in np.ndenumerate(Z):
    Z[i,j] = distance(MNIST.data[i], MNIST.data[j])
Z
```

```python
def pairwise_map(data, fn, N=0):
    if N == 0: N = len(data)
    Z = np.zeros(N*N).reshape(-1, N)
    for (i,j), v in np.ndenumerate(Z):
        Z[i,j] = fn(data[i], data[j])
    return Z
```

```python
import numpy as np
# Peak to peak - (max - min)
b1 = np.array([2,1,1])
b2 = np.array([1,3,4])
b3 = np.array([1,4,9])

B = np.stack((b1, b2, b3), axis=0)
np.ptp(B, axis=0)
np.ptp(B, axis=1)
```

```python
# Piecewise binning and clipping
x = np.arange(50)
x
y = 20
np.piecewise(x, [x>0 , x>10, x>20], [0, 10, y])
np.piecewise(x, [x<10, ((10<=x) & (x<40)), x>=40], [-1, lambda x: x, 1])
```

```python
def cut_outliers(pd_series, z_value=1.96):
    X = pd_series
    lb = np.mean(X) - z_value * np.std(X)
    ub = np.mean(X) + z_value * np.std(X)
    return np.piecewise(X, [X < lb, ((lb <= X) & (X < ub)), X >= ub], [lb, lambda X: X, ub])
```

```python
def cut_min_max(pd_series, mini, maxi):
    X = pd_series
    lb = mini
    ub = maxi
    return np.piecewise(X, [X < lb, ((lb <= X) & (X < ub)), X >= ub], [lb, lambda X: X, ub])
```

#### DataFrame to dictionary - key is index

Drops NaN's from index!

```python
cities = pd.read_excel('../_data/global-city-population-estimates.xlsx', 'CITIES-OVER-300K')
cities.sample(5)
cities_dict = cities[['Urban Agglomeration', '2015']].set_index('Urban Agglomeration').to_dict()['2015']
cities_dict
```

```python
df.to_dict
df.to_dict('index')
df.to_dict('index').items()
airport_dict = airports[['lon', 'lat']].to_dict('index')
```

```python
https://www.census.gov/popclock/world/nl
```

Next we can use a lambda to pull out the outcome from the attributes dictionary.

```python
df['outcome'] = df['outcome'].map(lambda x: x['outcome'])
df.head()
```

```python
def extract_dict_value(x, ftr):
    try: return x[ftr]
    except: return 0

df['score'] = df['score_weight'].map(lambda x: extract_dict_value(x, 'score'))
```

```python
# [:] copies the slice
```

```python
X_train.select_dtypes('float')
```

### open notebook link
[Open Notebook](./nlp_part2_sentiment_topic_similarity_classification_.ipynb)

```python

```

```python
file = '../_credentials/twitter_credentials.txt'
pickle_file = '../_credentials/twitter_credentials.pkl'
```

```python
%%writefile {file}
p6mTTVJjjBi0G59xIbpkooJXH
Y44ZH5ME8SndWYSCk7WpegQKwyPecnmLOUX2UAIFEGaC3TZayR
27198609-TDq5p3hTA8VTNJtwUASxUT21nnTs6jOxYlDONgkeQ
1wDvWkmZnoxeQnzaylH00FIhgNBhlX4DOy5vWLBUIow53
```

```python
def recursive_keys(json, i=2):
    """Recursively get keys and list items"""
    if type(json) == dict:
        for key, value in json.items():
#                 print('-'*i, key)
                if type(json[key]) == str:
                    print('--', key)
                    continue
                elif type(json[key]) == list:
                    print('list', key)
                    recursive_keys(json[key]) # [0]
                elif type(json[key]) == dict:
                    print('dict', key)
                    recursive_keys(json[key])

    elif type(json) == list:
        for i, item in enumerate(json):
            print(i, end=' ')
            recursive_keys(json[i])
```

```python

```

```python
def xml2dict(xml):
    """Convert XML to dictionary format."""
    import plistlib
    return plistlib.loads(str.encode(xml), fmt=plistlib.FMT_XML)
```

```python
def wifi_positioning(saved_map=False):
    """Return current location based on nearby wifi-hotspots."""

    payload = wifi_hotspots()

    if payload is None:
        print('WIFI positioning failed, returning IP location from provider')
        _, lat, lon, radius = ip_location()
        latlon = '{}, {}'.format(lat, lon)
    else:
        api_url_locate = 'https://pos.cit.api.here.com/positioning/v1/locate'
        json_result = geo_location(api_url_locate, payload)
        lat, lon = json_result['location']['lat'], json_result['location']['lng']
        latlon = '{}, {}'.format(lat, lon)
        radius = json_result['location']['accuracy']

    # Print address and lat, lon, radius and open/show location on map
#     print('\ngeoPy Nominatim location: ', geolocator.reverse(latlon))
#     rev_geocoder(lat, lon, radius)
#     here_maps = 'https://wego.here.com/directions/mix/{},{}/?map={},{},15'.format(lat, lon, lat, lon)
#     google_maps = 'https://www.google.com/maps/search/{},{}/@{},{},17z'.format(lat, lon, lat, lon)
#     !open -a Safari $here_maps; open -a Safari $google_maps
#     if saved_map: mia_saved_map(lon, lat, radius)

    return lat, lon
```

```python
import quilt
# https://quiltdata.com
quilt.install("ResidentMario/geoplot_data")
```

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices


Introduction to respy

### What is **respy**?

**respy** is an open source framework written in Python for the simulation and estimation of some finite-horizon discrete choice dynamic programming models. The group of models which can be currently represented in **respy** are called Eckstein-Keane-Wolpin models.

### Where do I find more information?

**GitHub:** The source code is available at https://github.com/OpenSourceEconomics/respy.

**Online Documentation:** The online documentation with tutorials, guides, and more is available at https://respy.readthedocs.io.

**Lectures:** Lectures on selected topics like Maximum Likilhood estimation using **respy** can be found here: https://ekw-lectures.readthedocs.io.

### Why should I use the package?

What makes **respy** powerful is that it allows to build and solve structural models in weeks or months whose development usually takes years.

### Tutorial Outline

The upcoming tutorial provides an example on how you can utilize **respy** to set up your own model. As an example, we implement a Robinson Crusoe economy as a discrete choice dynamic programming model. The tutorial is structured as follows:

 1.  Installation
 2.  Introductory Tutorial
  2.1. Theoretical Framework
  2.2. Specifying a Model in Respy
  2.3. Simulating Data
  2.4. Extending the Model
 3. Example Models in respy
 4. Further and upcoming functionalities in respy and other OpenSourceEconomics packages

1. Installation

The package can be installed via conda as follows

```
$ conda config --add channels conda-forge
$ conda install -c opensourceeconomics respy
```

After doing so, you can import respy alongside all other packages you are using.

```python
import io
import copy
import yaml
import numpy as np
import pandas as pd
import respy as rp
import seaborn as sns
import matplotlib.pyplot as plt
from IPython.display import Image
from pathlib import Path

from auxiliary.auxiliary import *
```

2. Introductory Tutorial

2.1. Theoretical Framework: Robinson Crusoe on an island

> After setting sail against his parents' wishes, being captured by pirates, escaping from them, building a plantation, and setting sail again, [Robinson Crusoe](https://en.wikipedia.org/wiki/Robinson_Crusoe) stranded on a small island. He is alone with one dog, two cats, and only some supplies. He goes fishing to make ends meet and if he is too tired he will relax in his hammock. But, he cannot relax to often as storing food is a difficult task on a tropical island.

Summarizing this economic problem:

- Robinson chooses every period $t = 0, \dots, T$ to either go fishing, $a = 0$, or spend the day in the hammock, $a = 1$.

```python
Image(filename="figures/tree_small.jpg")
```

- If Robinson chooses to go fishing, he gains one additional unit of experience in the next period. Experience starts at zero.

- The utility of a choice, $U(s_t, a_t)$, depends on the state $s_t$, which contains information on the individual's characteristics, and the chosen alternative $a_t$.

- Robinson's utility for any given option can be denoted by

$$\begin{align}
    U(s_t, a_t) = \underbrace{W(s_t, a_t)}_{wage} + \underbrace{N(s_t, a_t)}_{non-pecuniary}
\end{align}$$

**Working alternatives**

- For working alternatives like fishing, utility consists of two components, a wage and a non-pecuniary component. The wage is defined as

$$\begin{align}
        W(s_t, a_t)  &= r_a \exp\{x^w_{at} \beta^w_a + \epsilon_{at}\}\\
    \ln(W(s_t, a_t)) &= \ln(r_a) + x^w_{at} \beta^w_a + \epsilon_{at}
\end{align}$$

- It consists of several components:
    - $r_a$ is a market rental price for skill units.
    - $x^w_{at}$ and $\beta^w_a$ are the choice- and time-dependent covariates and returns related to the wage signaled by superscript $w$.
    - $\epsilon_{at}$ is a choice-specific random shock from the shock vector $\epsilon_t \sim \mathcal{N}(0, \Sigma)$ for all choices.

- The non-pecuniary rewards for working alternatives are a vector dot product of covariates $x_t^w$ and parameters $\beta^w$. The superscript $w$ signals that the components belong to working alternatives.

$$\begin{align}
    N^w(s_t, a_t) = x_t^w\beta^w
\end{align}$$

**Non-working alternatives**

- For non-working alternatives like the hammock, $W(s_t, a_t) = 0$. The non-pecuniary reward for non-working alternatives is very similar to the working alternative except that the shocks enter the equation additively. Superscript $n$ stands for non-pecuniary.

$$
    N^n(s_t, a_t) = x_t^n\beta^n + \epsilon_{at}
$$

- Robinson's choice set thus consists of a "working" alternative which awards him a pecuniary compensation or wage and a "leisure" or non-working alternative which he derives a utility from, but no experience or wage. Experience in this basic model starts at zero and increases by one for every period $t$ in $1, ..., T$ where he chooses to go fishing.

**Robinson's choice problem**

- During his time on the island, Robinson has to make choices about which actions to take. The general assumption is that Robinson is forward-looking and maximizes the expected present value of utility over the remaining lifetime which is achieved by selecting the optimal sequence of choices $\{a_t\}^T_{t = 0}$.

- Robinsons decision problem can be expressed as a Bellman equation where the value of a given state consists of the current and future state values associated with it.

$$\begin{align}
    V(s_{t})&= \max_{a_t} \, \{\underbrace{U(s_t, a_t)}_{\text{flow utility}}+ \delta \underbrace{\text{E}[ V(s_{t+1})]\}}_{\text{continuation value}}
\end{align}$$

- For each period, the value of a given choice consists of its flow utility and its continuation value. Since the realization of shocks becomes known in each period before Robinson makes his choice, the flow utility can be extracted from the expectation operator. As shocks in period $t + 1$ are unknown to the individual in period $t$, utility must be maximized given the joint distribution of shocks in period $t + 1$ which is a maximization problem over a two-dimensional integral. Denote the non-stochastic part of a state as $s^-$. Then, Robinson maximizes

$$\begin{equation}
    V(s_t) = \max_{a_t}\{
        U(s_t, a_t) + \delta \int_{\epsilon_{1, t + 1}} \int_{\epsilon_{0, t + 1}}
        \max_{a_{t + 1}} V_{a_{t + 1}}(s^-_{t + 1}, \epsilon_{t + 1})
        f_\epsilon(\epsilon_{t + 1})
        d_{\epsilon_{0, t + 1}}, d_{\epsilon_{1, t + 1}}
    \}
\end{equation}$$

2.2. Specifying the Model in respy

We can use **respy** to implement the model specified above. In order to process a model, **respy** requires two main inputs: `params` and `options`.

### params

The `params` object acts as the parameter vector of the model. Most of its values translate directly to parameters you can see in the equations.

Parameters are specified as a [MultiIndex pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html) with two index levels. A *category* which indicates certain parameters groups and a *name* which indicates a specific parameter. There are certain naming conventions in place that help **respy** recognize different parameters of the model.

We can read in parameters from a .csv-file. Here we define the parameter data in a string format to showcase the specification. Usually, the more convenient way would be to specify the parameters outside this notebook and to just import them from a separate file.

```python
params = """category,name,value
delta,delta,0.95
wage_fishing,exp_fishing,0.1
nonpec_fishing,constant,-1
nonpec_hammock,constant,2.5
nonpec_hammock,not_fishing_last_period,-1
shocks_sdcorr,sd_fishing,1
shocks_sdcorr,sd_hammock,1
shocks_sdcorr,corr_hammock_fishing,-0.2
lagged_choice_1_hammock,constant,1
"""
```

```python
params_basic = pd.read_csv(
    io.StringIO(params), sep=",", index_col=["category", "name"]
)
```

```python
params_basic
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      nonpec_fishing
      constant
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      -0.20

      lagged_choice_1_hammock
      constant
      1.00

### options

The second object required to set up a model are the `options`. The are specified as a dictionary which contains additional information like the number of periods in the model or the size of the simulated sample.

```python
options_basic = {
    "n_periods": 10,
    "simulation_agents": 1000,
    "simulation_seed": 123,
    "covariates": {
        "constant": "1",
        "not_fishing_last_period": "lagged_choice_1 != 'fishing'",
    },
}
```

```python
options_basic
```

    {'n_periods': 10,
     'simulation_agents': 1000,
     'simulation_seed': 123,
     'covariates': {'constant': '1',
      'not_fishing_last_period': "lagged_choice_1 != 'fishing'"}}

2.3. Simulating Data

Using the `params` and `options` we can set up a simulator using the function `get_simulate_func` and subsequently simulate a data for our selected parametrization.

```python
simulate = rp.get_simulate_func(params_basic, options_basic)
df_basic = simulate(params_basic)
```

```python
df_basic.head(5)
```

      Experience_Fishing
      Lagged_Choice_1
      Shock_Reward_Fishing
      Meas_Error_Wage_Fishing
      Shock_Reward_Hammock
      Meas_Error_Wage_Hammock
      Dense_Key
      Core_Index
      Choice
      Wage
      ...
      Nonpecuniary_Reward_Fishing
      Wage_Fishing
      Flow_Utility_Fishing
      Value_Function_Fishing
      Continuation_Value_Fishing
      Nonpecuniary_Reward_Hammock
      Wage_Hammock
      Flow_Utility_Hammock
      Value_Function_Hammock
      Continuation_Value_Hammock

      Identifier
      Period

      0
      0
      0
      hammock
      0.428556
      1
      0.427466
      1
      0
      1
      fishing
      1.535039
      ...
      -1
      1.535039
      0.535039
      19.006626
      19.443776
      1.5
      NaN
      1.833118
      18.704274
      17.759111

      1
      1
      fishing
      0.780145
      1
      1.850937
      1
      1
      0
      hammock
      NaN
      ...
      -1
      2.411250
      1.411250
      19.104983
      18.624982
      2.5
      NaN
      4.157512
      20.273393
      16.964086

      2
      1
      hammock
      2.063393
      1
      0.708209
      1
      2
      0
      fishing
      8.700606
      ...
      -1
      8.700606
      7.700606
      23.415469
      16.541961
      1.5
      NaN
      1.781221
      16.030649
      14.999397

      3
      2
      fishing
      -1.190336
      1
      -2.028002
      1
      3
      4
      fishing
      0.371452
      ...
      -1
      0.371452
      -0.628548
      13.904880
      15.298346
      2.5
      NaN
      0.751039
      13.874853
      13.814541

      4
      3
      fishing
      -0.049084
      1
      0.607816
      1
      4
      5
      hammock
      NaN
      ...
      -1
      1.285202
      0.285202
      13.405278
      13.810606
      2.5
      NaN
      3.105353
      14.884157
      12.398742

5 rows × 22 columns

Using our simulated data, we can now inspect the choices of Robinson.

```python
plot_choice_shares(df_basic)
```

Inspecting the persistence in choices shows that Robinson rarely chooses the same option multiple times in a row.

```python
data = pd.crosstab(df_basic.Lagged_Choice_1, df_basic.Choice, normalize=True)
sns.heatmap(data, cmap="Blues", annot=True)
```

2.4. Extending the Model

As already stated in the introduction, one of the main advantages of **respy** is that the package is flexible regarding the model structure. We will illustrate this by extending the previously introduced model framework in several different ways.

2.4.1. Extension: Covariates

Next we want to add a covariate. As before, we concentrate on a simple example, but this can be extended in any direction. Note, that in the basic model we had two covariates:

1. The constant, which is the simplest covariate
2. The indicator whether Robinson went fishing last period.

Covariates allow to specify advanced return structures to each choice. In the following example, Robinson's return to fishing decreases as he gets older. We assume that he arrives at the island with 16 years. Then his age in each period, which we assume for now is one year, increases by one. So in each period his age is given by `period + 16`. We define the covariate and his value in each period in the option file.

```python
options_age = copy.deepcopy(options_basic)
options_age["covariates"].update({"age": "period + 16"})
options_age
```

    {'n_periods': 10,
     'simulation_agents': 1000,
     'simulation_seed': 123,
     'covariates': {'constant': '1',
      'not_fishing_last_period': "lagged_choice_1 != 'fishing'",
      'age': 'period + 16'}}

In a second step, we specify the return for the choice for this covariate.

```python
params_age = params_basic.copy()
params_age.loc[("wage_fishing", "age"), "value"] = -0.015
params_age
```

      value

      category
      name

      delta
      delta
      0.950

      wage_fishing
      exp_fishing
      0.100

      nonpec_fishing
      constant
      -1.000

      nonpec_hammock
      constant
      2.500

      not_fishing_last_period
      -1.000

      shocks_sdcorr
      sd_fishing
      1.000

      sd_hammock
      1.000

      corr_hammock_fishing
      -0.200

      lagged_choice_1_hammock
      constant
      1.000

      wage_fishing
      age
      -0.015

Having adjusted the parameters and options accordingly, we can now simulate data that incorporates the new age covariate. As the plot below shows, including the negative effect of age reduces the proportion of individuals that choose fishing, increasingly so in later periods.

```python
simulate = rp.get_simulate_func(params_age, options_age)
df_age = simulate(params_age)
```

```python
plot_choice_shares(df_age)
```

2.4.2. Extension: Adding Choices

Lets return to the basic example and alter the choice set of Robinson. Let's assume after some time on the island, Robinson meets Friday we can help him improve his fishing.

We thus add the option `"friday"`, which affects the utility of fishing. The choice should be available only once starting with the third period, and only after Robinson has been fishing before. As a first step, we need to alter the parameter vector. Here we just import a pre-defined parameter vector from a .csv-file. There have been three main changes to the base parameter vector:

1. `wage_fishing` now includes a return to contemplation with Friday.
2. There is a new category,`nonpec_friday`, which captures the non-pecuniary reward for of spending a period with Friday. Since talking to Friday is purely non-pecuniary option, Robinson does not earn a wage when choosing it.
3. The shock shock variance-covariance matrix has been altered to include the additional option.

```python
params_friday = pd.read_csv(
    "params_files/robinson_crusoe_friday.csv", index_col=["category", "name"]
)
params_friday
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      contemplation_with_friday
      0.40

      nonpec_fishing
      constant
      -1.00

      nonpec_friday
      constant
      -1.00

      not_fishing_last_period
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_friday
      1.00

      sd_hammock
      1.00

      corr_friday_fishing
      0.00

      corr_hammock_fishing
      0.00

      corr_hammock_friday
      0.00

      lagged_choice_1_hammock
      constant
      1.00

Turning to the `options`, we can see that the new covariate `"contemplation_with_friday"` is only affecting utility if Robinson is experienced in fishing and only for one interaction with Friday. This naturally limits the interaction with Friday.

```python
options_friday = yaml.safe_load(open("option_files/options_friday.yml"))
options_friday["covariates"]
```

    {'constant': '1',
     'contemplation_with_friday': 'exp_friday == 1 and exp_fishing >= 1',
     'not_fishing_last_period': "lagged_choice_1 != 'fishing'"}

The key `"negative_choice_set"` can be used to restrict the choice Friday to the third and following periods. The first key matches a choice.

```python
options_friday["negative_choice_set"]
```

    {'friday': ['period < 2', 'exp_fishing == 0']}

We additionally need to define `core_state_space_filters` now that we added Friday. They are specified as a list of strings that define certain events which shall be avoided. If the string evaluates to `True` for a potential state, the corresponding state is eliminated from the core state space. These options thus compliment the `negative_choice_set`.

Some states are impossible from a model perspective, but eliminating them using the `core_state_space_filters` is still worthwhile to eliminate them to ease the computational burden of the model. See for instance the string `"period > 0 and exp_fishing + exp_friday == period and lagged_choice_1 == 'hammock'"`. This expression  essentially states hat whenever individuals accumulated as many years of experience as there are periods, the previous choice cannot be a choice without experience accumulation. This seems trivial, but states for which this expression evaluates to `True` will still be included in the state space if we do not eliminate them using the filters.

```python
options_friday["core_state_space_filters"]
```

    ["period > 0 and exp_fishing + exp_friday == period and lagged_choice_1 == 'hammock'",
     'period <= 2 and exp_friday != 0',
     'period >= 3 and period - exp_friday < 2',
     'exp_friday > 0 and exp_fishing == 0',
     "exp_friday > 0 and exp_fishing == 1 and lagged_choice_1 == 'fishing'",
     "period - exp_friday == 2 and lagged_choice_1 != 'friday' and period > 2",
     "exp_{choices_w_exp} == 0 and lagged_choice_1 == '{choices_w_exp}'"]

We can now simulate our new model including Friday.

```python
simulate = rp.get_simulate_func(params_friday, options_friday)
df_friday = simulate(params_friday)
```

```python
plot_choice_shares(df_friday, friday=True)
```

2.4.3. Extension: Initial Experience

Again returning to the basic example, we will next demonstrate how to include heterogeneity in initial conditions in `respy`: In period $t=0$, Robinson will have $0$, $1$, or $2$ periods of experience in fishing. We can thus analyze what happens to Robinson's choices if he lands on the island with previous experience in fishing attained before being stranded.

To allow Robinson to have nonzero experience in fishing in the first period, we need to modify `params`: We specify the distribution of initial experience via **probability mass functions**, as the probabilities do not depend on any information.
The keyword is `"initial_exp_fishing_*"` in the category-level of the index, where the asterisk needs to be replaced with the experience level. In the name-level, use `"probability"` to signal that the float in `"value"` is a probability.

The new parameter specification is below: Robinson has equal probability to start out with 0, 1 or 2 periods of experience. However, one probability is set to 0.34 so that all probabilities sum to one. If that is not the case, `respy` will emit a warning and normalize probabilities to a uniform distribution.

```python
params_initial_exp = params_basic.copy()
params_initial_exp.loc[
    ("initial_exp_fishing_0", "probability"), "value"
] = 0.33
params_initial_exp.loc[
    ("initial_exp_fishing_1", "probability"), "value"
] = 0.33
params_initial_exp.loc[
    ("initial_exp_fishing_2", "probability"), "value"
] = 0.34
params_initial_exp
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      nonpec_fishing
      constant
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      -0.20

      lagged_choice_1_hammock
      constant
      1.00

      initial_exp_fishing_0
      probability
      0.33

      initial_exp_fishing_1
      probability
      0.33

      initial_exp_fishing_2
      probability
      0.34

```python
simulate = rp.get_simulate_func(params_initial_exp, options_basic)
df_exp = simulate(params_initial_exp)
```

We can the plot the choice frequencies for both scenarios to compare Robinson's actions.

```python
plot_choice_prob_and_exp_level(df_basic)
```

Comparing the plots shows that the share of engagement in fishing increases compared to the baseline scenario, as two thirds of Robinsons land on the island with previous experience in fishing.

```python
plot_choice_prob_and_exp_level(df_exp)
```

2.4.4. Extension: Observables

Next add one observable characteristic to the baseline model, `"Fishing_Grounds"`: Now Robinson can end up, with a certain probability, on the side of the island which has `"poor"` or `"rich"` fishing grounds. Experiencing rich fishing grounds affects the **non-pecuniary reward** for fishing.

To specify the observable, we add three additional rows to `params`:

- The probability with which Robinson will find himself in rich and in poor fishing grounds;

- The return for being in a rich fishing ground, which here is set to be positive and constant.

`respy` allows for complex probability distributions of observables, which may for instance depend on other covariates. However, we will assume that the observable's probability distribution does not depend on any other information. We assume that each Robinson is randomly assigned to a certain side of the island, according to the float specified under value in the name-level probability.

Note that all probabilities sum to one. If that is not the case, **respy** will emit a warning and normalize probabilities.

```python
params_fish_ground = params_basic.copy()
params_fish_ground.loc[
    ("observable_fishing_grounds_rich", "probability"), "value"
] = 0.5
params_fish_ground.loc[
    ("observable_fishing_grounds_poor", "probability"), "value"
] = 0.5
params_fish_ground.loc[
    ("nonpec_fishing", "rich_fishing_grounds"), "value"
] = 0.5
params_fish_ground
```

      value

      category
      name

      delta
      delta
      0.95

      wage_fishing
      exp_fishing
      0.10

      nonpec_fishing
      constant
      -1.00

      nonpec_hammock
      constant
      2.50

      not_fishing_last_period
      -1.00

      shocks_sdcorr
      sd_fishing
      1.00

      sd_hammock
      1.00

      corr_hammock_fishing
      -0.20

      lagged_choice_1_hammock
      constant
      1.00

      observable_fishing_grounds_rich
      probability
      0.50

      observable_fishing_grounds_poor
      probability
      0.50

      nonpec_fishing
      rich_fishing_grounds
      0.50

We also need to overwrite the covariates section of `options` to include which level of the observable is associated with a higher nonpecuniary reward for fishing:

```python
options_fish_ground = options_basic.copy()
options_fish_ground["covariates"] = {
    "constant": "1",
    "not_fishing_last_period": "lagged_choice_1 != 'fishing'",
    "rich_fishing_grounds": "fishing_grounds == 'rich'",
}
options_fish_ground
```

    {'n_periods': 10,
     'simulation_agents': 1000,
     'simulation_seed': 123,
     'covariates': {'constant': '1',
      'not_fishing_last_period': "lagged_choice_1 != 'fishing'",
      'rich_fishing_grounds': "fishing_grounds == 'rich'"}}

```python
simulate = rp.get_simulate_func(params_fish_ground, options_fish_ground)
df_fish_ground = simulate(params_fish_ground)
```

As can be expected, Robinson engages in fishing more often if he is stranded on the side of the island with rich fishing grounds.

```python
plot_fishing_grounds(df_fish_ground)
```

3. Example Models in respy

The story of Robinson is an illustrative introductory example to `respy`'s modeling capabilities. What may not be evident at first sight, is that it in this context is a story of human capital accumulation which constitutes a central topic in economic research. `respy` is designed to be used for much larger and more complex models that are capable of addressing real world policy questions and are calibrated to empirical data. To illustrate the scalability of `respy` models, the interface includes a number of example models that are taken from actual economic publications.

They can be accessed as ```params, options, data = rp.get_example_model(<model>)```

Below are the example models that are currently available.

### Toy models

The package provides users with two toy models. These models can be used to acquaintance oneself with **respy**'s functionalities and can be encountered in the [tutorials](https://respy.readthedocs.io/en/latest/tutorials/index.html) in the documentation. The models are called:

- `robinson_crusoe_basic`
- `robinson_crusoe_extended`

We have gotten to know the toy models in this tutorial: we started off with a version of the most basic model in which has Robinson either fishing or relaxing in the hammock and, amongst other things, extended the model to `robinson_crusoe_extended` where he can additionally talk to Friday to improve his fishing.

These models are excellent examples to use for learning and prototyping because they are very computationally feasible due to the small number of available choices and low number of periods in the models.

#### Basic Robinson Model

```python
_, _, data = rp.get_example_model("robinson_crusoe_basic")
```

```python
plot_choice_shares(data)
```

#### Extended Robinson Model

```python
_, _, data = rp.get_example_model("robinson_crusoe_extended")
```

```python
plot_choice_shares(data, friday=True)
```

### Keane and Wolpin (1994)

Aside from toy models, **respy** also provides several models that stem from the economic literature on dynamic life-cycle models. The most simple examples are a group of models based on the following publication:

- Keane, M. P., & Wolpin, K. I. (1994). The Solution and Estimation of Discrete Choice Dynamic Programming Models by Simulation and Interpolation: Monte Carlo Evidence. *The Review of Economics and Statistics*, 648-672.

The study is mainly concerned with the computational difficulties that accompany the solution of discrete choice dynamic programming (DCDP) problems such as the ones encountered in dynamic life-cycle models of educational and occupational choice. In the study, Keane and Wolpin (1994) develop an approximate solution method which consists of Monte Carlo integration with simulation and an interpolation approach to ease the computational burden of solving the DCDP model. They utilize one model with three different parametrizations to assess their solution method. This model and its three parametrizations are example models that are part of the **respy** interface. They are called:

- `kw_94_one`
- `kw_94_two`
- `kw_94_three`

The model consists of four mutually exclusive alternatives that individuals can choose in each period. Agents can either choose to work in one of two sectors `a` or `b`, invest in education, or stay home. The work alternatives award a wage and experience, while school only awards experience. In the home option, individuals gain neither a wage nor experience. The plot below shows the choice patterns for the three parametrizations. The model considers a time horizon of 40 periods.

```python
plot_choices_kw(example="1994")
```

### Keane and Wolpin (1997)

A more advanced group of examples are given by the models developed by Keane and Wolpin (1997). In this study, the authors implement an empirical structural life-cycle model of occupational choice and human capital investment. They estimate their models on data from the National Longitudinal Survey of Youth (NLSY).

* Keane, M. P., & Wolpin, K. I. (1997). The Career Decisions of Young Men. *Journal of Political Economy*, 105(3), 473-522.

**respy** supports both the basic and extended model from the paper. They are named:

- `kw_97_basic`
- `kw_97_extended`

However, the parametrization from the paper returns different life-cycle patterns for **respy** than presented in the paper. You can thus also access our estimates based for the models that are based on the same empirical data by adding `_respy` to the model name.

The modelsnumber consists of three occupational choices (*white collar*, *blue collar*, and *military*), one educational choice (*school*), and a *home* option. Both models consider a life-cycle of 50 periods. These models are decidedly larger than the toy models explored in this tutorial and require a considerable amount of computation power to solve, especially the extended model. The estimated life-cycle patterns are shown below.

```python
plot_choices_kw(example="1997")
```

#### How do the models compare to empirical data?

Below are the model predictions plotted against the observed choices. These plots correspond to Figure 1-5 (p.492-494) in the paper.

```python
plot_kw_97_comparison()
```

### Keane and Wolpin (2000)

Another example model provided in the respy interface is the model presented in Keane and Wolpin (2000). The model incorporates an observable charactistic to account for race aiming to analyze the effects of monetary incentive schemes designed to reduce racial differences in school attainment and earnings.

- Keane, M. P., & Wolpin, K. I. (2000). Eliminating Race Differences in School Attainment and Labor Market Success. *Journal of Labor Economics*, 18(4), 614-652.

The model is named

- `kw_2000`

The model is very similar to the extended model specification in Keane and Wolpin (1997).

4. Further and upcoming functionalities in respy and other OpenSourceEconomics packages

As `respy` is an open source project multiple teams work on extensions of the existing flexibility. The next major advancements are:

- Exogenous processes (childbirth, health shocks etc.)

- Consumption and saving decision.

A very recent and big step for `respy` was the introduction of state specific choice sets. As documentation is still in development, there are not tutorials yet for every aspect. The current documentation can be found [here](https://respy.readthedocs.io/en/latest/).

Besides simulation of economic models, `respy` of course offers the possibility to estimate your model on data. Furthermore you have the option to fix certain parameters and only estimate others. For the estimation, the following techniques are available:

- Maximum likelihood estimation (ML)
- Method of simulated moments (MSM)

For both techniques you can find details in our [ekw-lectures](https://github.com/OpenSourceEconomics/ekw-lectures/tree/master/lectures).

Note that `respy`, does not have any optimizers implemented. For a set of given parameters `respy` creates the corresponding criterion function value. It is best optimized by the OpenSourceEconomics estimation package `estimagic`. Further examples of OSE software packages designed to interact with each other are:

- [`ruspy`](https://ruspy.readthedocs.io/en/latest/) is an open-source Python package for the simulation and estimation of a prototypical infinite-horizon dynamic discrete choice model based on Rust (1987).

- [`grmpy`](https://grmpy.readthedocs.io/en/latest/) is an open-source Python package for the simulation and estimation of the generalized Roy model.

- [`econsa`](https://econsa.readthedocs.io/en/latest/) is an open-source Python package that facilitates uncertainty propagation and global sensitivity of computational economic models.analysis.

- [`robupy`](https://robupy.readthedocs.io/en/latest/) is an open-source Python package for robust decision making.

- [`gettsim`](https://gettsim.readthedocs.io/en/stable/) is an open-source Python package developed by OSE members providing a depiction of the German Taxes and Transfers System.

Check out the OpenSourceEconomics homepage for more information on all packages https://open-econ.org.