# Welcome to the second MAST-ML tutorial notebook,

# Data Import and Data Cleaning with MAST-ML!

## In this notebook, we will learn different ways to download and import data into a MAST-ML run:

1. [Set up MAST-ML on Colab and begin session](#task1)
2. [Import model datasets from scikit-learn](#task2)
3. [Conduct different data cleaning methods](#task3)
4. [Import and prepare a real dataset that is stored locally](#task4)
5. [Download data from various materials databases](#task5)

## Task 1: Set up MAST-ML on Colab and begin session

If you are working on Google Colab and need to install MAST-ML,
begin by pip installing MAST-ML to the Colab session
and install the needed dependencies:

```python
!pip install mastml
```

    Collecting mastml
      Downloading https://files.pythonhosted.org/packages/cf/40/e094142e31fee2cdb86e7ac6676bdc023917d811f870a7969724c5984e41/mastml-3.0.0-py3-none-any.whl (121kB)
         |████████████████████████████████| 122kB 5.6MB/s
    Collecting mdf-toolbox
      Downloading https://files.pythonhosted.org/packages/f5/44/45ed3b256d744891384d13088c12a23baaa2e395b341c3ab03c18001f715/mdf_toolbox-0.5.7-py2.py3-none-any.whl
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from mastml) (1.19.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from mastml) (1.4.1)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from mastml) (0.22.2.post1)
    Collecting citrination-client
      Downloading https://files.pythonhosted.org/packages/61/49/c0af91084172f6a6aa7d625651ec366c85a4fd717c5b4fa0e014d1953d6e/citrination-client-6.5.1.tar.gz (54kB)
         |████████████████████████████████| 61kB 6.8MB/s
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from mastml) (3.2.2)
    Requirement already satisfied: mlxtend in /usr/local/lib/python3.7/dist-packages (from mastml) (0.14.0)
    Collecting globus-sdk
      Downloading https://files.pythonhosted.org/packages/92/a4/57b628cc5509eeb8361eb87506a3aea2078ca9c4e4ffaebc88280cdf7f40/globus_sdk-2.0.1-py2.py3-none-any.whl (85kB)
         |████████████████████████████████| 92kB 7.7MB/s
    Collecting sphinx-automodapi
      Downloading https://files.pythonhosted.org/packages/16/08/83a3ee2cd420043538d195f024caac76dc2567a266361d95bb9344ab5594/sphinx_automodapi-0.13-py3-none-any.whl (75kB)
         |████████████████████████████████| 81kB 10.7MB/s
    Requirement already satisfied: openpyxl in /usr/local/lib/python3.7/dist-packages (from mastml) (2.5.9)
    Collecting pymatgen==2021.3.9
      Downloading https://files.pythonhosted.org/packages/18/06/8f4a5288ea5d940773d9c8651aee59535006353b1b592ac1e61802b0749f/pymatgen-2021.3.9.tar.gz (3.0MB)
         |████████████████████████████████| 3.0MB 35.3MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
      Installing backend dependencies ... done
        Preparing wheel metadata ... done
    Collecting scikit-optimize
      Downloading https://files.pythonhosted.org/packages/8b/03/be33e89f55866065a02e515c5b319304a801a9f1027a9b311a9b1d1f8dc7/scikit_optimize-0.8.1-py2.py3-none-any.whl (101kB)
         |████████████████████████████████| 102kB 13.4MB/s
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from mastml) (1.1.5)
    Collecting mdf-forge
      Downloading https://files.pythonhosted.org/packages/6f/0c/697e48caaabc83e071c222dc6d25ec0a924c69d4cf8f990964f47091b8d2/mdf_forge-0.7.6-py2.py3-none-any.whl
    Collecting dlhub-sdk
      Downloading https://files.pythonhosted.org/packages/1e/cd/02ad247cf7df4467b63dec57e2c8d8f5fe64330bf6da61e6ac6cddcde149/dlhub_sdk-0.9.4-py2.py3-none-any.whl (41kB)
         |████████████████████████████████| 51kB 7.9MB/s
    Collecting globus-nexus-client
      Downloading https://files.pythonhosted.org/packages/52/ca/a0e2c03aeea3e4b3b3256ab309e24fb5227ebaf92aabca56b6dfc3cc758a/globus_nexus_client-0.3.0-py2.py3-none-any.whl
    Collecting matminer
      Downloading https://files.pythonhosted.org/packages/18/67/319db03366448bf367f6239598da2da0021389b02a7f874380ee3c193890/matminer-0.6.5.tar.gz (5.8MB)
         |████████████████████████████████| 5.8MB 18.1MB/s
    Requirement already satisfied: requests>=2.18.4 in /usr/local/lib/python3.7/dist-packages (from mdf-toolbox->mastml) (2.23.0)
    Collecting fair-research-login>=0.1.5
      Downloading https://files.pythonhosted.org/packages/89/a6/3239efa5150d9cb52af90d246ca46a405eefdaee503e77bf84023dbd2752/fair_research_login-0.2.0-py2.py3-none-any.whl
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->mastml) (1.0.1)
    Collecting pypif
      Downloading https://files.pythonhosted.org/packages/1e/04/a59e5ebacaed2800678467d9fcf511d249bac9b1b692a8fba0072cb09a49/pypif-2.1.2.tar.gz
    Requirement already satisfied: six<2 in /usr/local/lib/python3.7/dist-packages (from citrination-client->mastml) (1.15.0)
    Collecting pyyaml>=5.1.2
      Downloading https://files.pythonhosted.org/packages/7a/a5/393c087efdc78091afa2af9f1378762f9821c9c1d7a22c5753fb5ac5f97a/PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636kB)
         |████████████████████████████████| 645kB 46.1MB/s
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mastml) (1.3.1)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mastml) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mastml) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->mastml) (2.8.1)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from mlxtend->mastml) (54.2.0)
    Collecting pyjwt[crypto]<2.0.0,>=1.5.3
      Downloading https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl
    Requirement already satisfied: sphinx>=1.7 in /usr/local/lib/python3.7/dist-packages (from sphinx-automodapi->mastml) (1.8.5)
    Requirement already satisfied: jdcal in /usr/local/lib/python3.7/dist-packages (from openpyxl->mastml) (1.4.1)
    Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.7/dist-packages (from openpyxl->mastml) (1.0.1)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pymatgen==2021.3.9->mastml) (1.7.1)
    Collecting ruamel.yaml>=0.15.6
      Downloading https://files.pythonhosted.org/packages/29/4e/c3105bbbbc662f6a671a505f00ec771e93b5254f09fbb06002af9087071a/ruamel.yaml-0.17.4-py3-none-any.whl (101kB)
         |████████████████████████████████| 102kB 13.3MB/s
    Requirement already satisfied: palettable>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2021.3.9->mastml) (3.3.0)
    Collecting plotly>=4.5.0
      Downloading https://files.pythonhosted.org/packages/1f/f6/bd3c17c8003b6641df1228e80e1acac97ed8402635e46c2571f8e1ef63af/plotly-4.14.3-py2.py3-none-any.whl (13.2MB)
         |████████████████████████████████| 13.2MB 54.1MB/s
    Requirement already satisfied: typing-extensions>=3.7.4.3; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from pymatgen==2021.3.9->mastml) (3.7.4.3)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from pymatgen==2021.3.9->mastml) (2.5)
    Collecting monty>=3.0.2
      Downloading https://files.pythonhosted.org/packages/41/47/e6f045cd69f24df0b4ddd55fab329c079b7c9ce978a32431dce904f6a1d6/monty-2021.3.3-py3-none-any.whl (63kB)
         |████████████████████████████████| 71kB 10.5MB/s
    Collecting spglib>=1.9.9.44
      Downloading https://files.pythonhosted.org/packages/4b/98/fa4760b9c71e2eace5a60b85dbf36dece49c379f8291cf1203056f287766/spglib-1.16.1-cp37-cp37m-manylinux2010_x86_64.whl (296kB)
         |████████████████████████████████| 296kB 38.3MB/s
    Collecting uncertainties>=3.1.4
      Downloading https://files.pythonhosted.org/packages/45/41/fc7e7b73b603e7c2c9e040b7aa8caf4a88d74b6faa567601ed82b6f0d8e1/uncertainties-3.1.5-py2.py3-none-any.whl (246kB)
         |████████████████████████████████| 256kB 43.5MB/s
    Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pymatgen==2021.3.9->mastml) (0.8.9)
    Collecting pyaml>=16.9
      Downloading https://files.pythonhosted.org/packages/15/c4/1310a054d33abc318426a956e7d6df0df76a6ddfa9c66f6310274fb75d42/pyaml-20.4.0-py2.py3-none-any.whl
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->mastml) (2018.9)
    Requirement already satisfied: tqdm>=4.19.4 in /usr/local/lib/python3.7/dist-packages (from mdf-forge->mastml) (4.41.1)
    Collecting jsonschema>=3.2.0
      Downloading https://files.pythonhosted.org/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl (56kB)
         |████████████████████████████████| 61kB 9.7MB/s
    Collecting funcx>=0.0.5
      Downloading https://files.pythonhosted.org/packages/eb/1c/79404da8ff1aabe8d1619905cbfe376d4128bfdd0904fba992c7f732e86a/funcx-0.2.0-py3-none-any.whl
    Requirement already satisfied: pymongo>=3.10.1 in /usr/local/lib/python3.7/dist-packages (from matminer->mastml) (3.11.3)
    Collecting pint>=0.11
      Downloading https://files.pythonhosted.org/packages/33/de/53a77b82553579affab7438d299f850acbc1c4dd741c5ce52594513cb0ef/Pint-0.17-py2.py3-none-any.whl (204kB)
         |████████████████████████████████| 215kB 57.0MB/s
    Collecting future>=0.18.2
      Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
         |████████████████████████████████| 829kB 38.4MB/s
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->mdf-toolbox->mastml) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->mdf-toolbox->mastml) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->mdf-toolbox->mastml) (2020.12.5)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.18.4->mdf-toolbox->mastml) (1.24.3)
    Collecting cryptography>=1.4; extra == "crypto"
      Downloading https://files.pythonhosted.org/packages/b2/26/7af637e6a7e87258b963f1731c5982fb31cd507f0d90d91836e446955d02/cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2MB)
         |████████████████████████████████| 3.2MB 51.5MB/s
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (20.9)
    Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (2.6.1)
    Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (2.11.3)
    Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (2.9.0)
    Requirement already satisfied: sphinxcontrib-websupport in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (1.2.4)
    Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (0.7.12)
    Requirement already satisfied: docutils>=0.11 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (0.16)
    Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (2.1.0)
    Requirement already satisfied: imagesize in /usr/local/lib/python3.7/dist-packages (from sphinx>=1.7->sphinx-automodapi->mastml) (1.2.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pymatgen==2021.3.9->mastml) (1.2.1)
    Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10"
      Downloading https://files.pythonhosted.org/packages/5e/6e/f652c56bbb2c3d3fca252ffc7c0358597f57a1bbdf484dac683054950c63/ruamel.yaml.clib-0.2.2-cp37-cp37m-manylinux1_x86_64.whl (547kB)
         |████████████████████████████████| 552kB 45.7MB/s
    Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly>=4.5.0->pymatgen==2021.3.9->mastml) (1.3.3)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.2->pymatgen==2021.3.9->mastml) (4.4.2)
    Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.2.0->dlhub-sdk->mastml) (20.3.0)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.2.0->dlhub-sdk->mastml) (3.8.1)
    Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.2.0->dlhub-sdk->mastml) (0.17.3)
    Requirement already satisfied: pyzmq>=22.0.0 in /usr/local/lib/python3.7/dist-packages (from funcx>=0.0.5->dlhub-sdk->mastml) (22.0.3)
    Collecting configobj
      Downloading https://files.pythonhosted.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz
    Collecting parsl>=1.1.0a0
      Downloading https://files.pythonhosted.org/packages/41/2e/33f0b66ee8afe2af0cffd103cffb487c7616a16a2e77f0e1635238b69dfd/parsl-1.1.0a1-py3-none-any.whl (449kB)
         |████████████████████████████████| 450kB 28.1MB/s
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from funcx>=0.0.5->dlhub-sdk->mastml) (0.3.3)
    Collecting typer>=0.3.0
      Downloading https://files.pythonhosted.org/packages/90/34/d138832f6945432c638f32137e6c79a3b682f06a63c488dcfaca6b166c64/typer-0.3.2-py3-none-any.whl
    Collecting texttable
      Downloading https://files.pythonhosted.org/packages/06/f5/46201c428aebe0eecfa83df66bf3e6caa29659dbac5a56ddfd83cae0d4a4/texttable-1.6.3-py2.py3-none-any.whl
    Collecting python-daemon
      Downloading https://files.pythonhosted.org/packages/b1/cc/2ab0d910548de45eaaa50d0372387951d9005c356a44c6858db12dc6b2b7/python_daemon-2.3.0-py2.py3-none-any.whl
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.4; extra == "crypto"->pyjwt[crypto]<2.0.0,>=1.5.3->globus-sdk->mastml) (1.14.5)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=2.3->sphinx>=1.7->sphinx-automodapi->mastml) (1.1.1)
    Requirement already satisfied: sphinxcontrib-serializinghtml in /usr/local/lib/python3.7/dist-packages (from sphinxcontrib-websupport->sphinx>=1.7->sphinx-automodapi->mastml) (1.1.4)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=3.2.0->dlhub-sdk->mastml) (3.4.1)
    Requirement already satisfied: tblib in /usr/local/lib/python3.7/dist-packages (from parsl>=1.1.0a0->funcx>=0.0.5->dlhub-sdk->mastml) (1.7.0)
    Collecting paramiko
      Downloading https://files.pythonhosted.org/packages/95/19/124e9287b43e6ff3ebb9cdea3e5e8e88475a873c05ccdf8b7e20d2c4201e/paramiko-2.7.2-py2.py3-none-any.whl (206kB)
         |████████████████████████████████| 215kB 56.3MB/s
    Collecting psutil>=5.5.1
      Downloading https://files.pythonhosted.org/packages/84/da/f7efdcf012b51506938553dbe302aecc22f3f43abd5cffa8320e8e0588d5/psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296kB)
         |████████████████████████████████| 296kB 53.0MB/s
    Collecting typeguard>=2.10
      Downloading https://files.pythonhosted.org/packages/06/1f/c10ad900a10e1421b85d20f1c9d1748469ef5a34296693a02be887af5f95/typeguard-2.12.0-py3-none-any.whl
    Requirement already satisfied: click<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer>=0.3.0->funcx>=0.0.5->dlhub-sdk->mastml) (7.1.2)
    Collecting lockfile>=0.10
      Downloading https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.4; extra == "crypto"->pyjwt[crypto]<2.0.0,>=1.5.3->globus-sdk->mastml) (2.20)
    Collecting pynacl>=1.0.1
      Downloading https://files.pythonhosted.org/packages/9d/57/2f5e6226a674b2bcb6db531e8b383079b678df5b10cdaa610d6cf20d77ba/PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961kB)
         |████████████████████████████████| 962kB 55.3MB/s
    Collecting bcrypt>=3.1.3
      Downloading https://files.pythonhosted.org/packages/26/70/6d218afbe4c73538053c1016dd631e8f25fffc10cd01f5c272d7acf3c03d/bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63kB)
         |████████████████████████████████| 71kB 11.0MB/s
    Building wheels for collected packages: pymatgen
      Building wheel for pymatgen (PEP 517) ... done
      Created wheel for pymatgen: filename=pymatgen-2021.3.9-cp37-cp37m-linux_x86_64.whl size=3793998 sha256=c4d6f1a9700ecbc03eb41147d1daf2bdf34a435c717c7988af894186786a437d
      Stored in directory: /root/.cache/pip/wheels/8b/f9/ce/1d72ca15aa8d8de3ae81ae333b3eabb49eeb5f294c25ad128b
    Successfully built pymatgen
    Building wheels for collected packages: citrination-client, matminer, pypif, future, configobj
      Building wheel for citrination-client (setup.py) ... done
      Created wheel for citrination-client: filename=citrination_client-6.5.1-py2.py3-none-any.whl size=116097 sha256=f8af0b0c2e2c1c5720a9f59ab8119fb815364ea3b54c0cf62beacf81c31ee5e1
      Stored in directory: /root/.cache/pip/wheels/6c/88/f5/5a3d5759228aabc63cb3baf2d767369ac99d18f3a941d381e7
      Building wheel for matminer (setup.py) ... done
      Created wheel for matminer: filename=matminer-0.6.5-cp37-none-any.whl size=1189789 sha256=fff862a6427e6ebf17dcde23db5464de0e7f447b41901bc67708ff3328dfffe3
      Stored in directory: /root/.cache/pip/wheels/b9/d9/ca/b03a29a28fb675c253a5792995097c48c25b8d409ddac4e2c1
      Building wheel for pypif (setup.py) ... done
      Created wheel for pypif: filename=pypif-2.1.2-py2.py3-none-any.whl size=29506 sha256=e94bbd245d2d13c1df0e623cd3e9af50cf85b8a172d90811ea513e9acbe8c063
      Stored in directory: /root/.cache/pip/wheels/0d/19/de/9d99bb147337e90a209a44c89e67b3aaadcac4f6f8d106c13d
      Building wheel for future (setup.py) ... done
      Created wheel for future: filename=future-0.18.2-cp37-none-any.whl size=491058 sha256=5ccac7ca500898cc0e427199fe86ed3b16b8658ba921931afbf652a8ff71b4bb
      Stored in directory: /root/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
      Building wheel for configobj (setup.py) ... done
      Created wheel for configobj: filename=configobj-5.0.6-cp37-none-any.whl size=34547 sha256=15d71a6c96670b3dff070f4e84ce111e0da3f0a69eb02958c62a90d409c9148b
      Stored in directory: /root/.cache/pip/wheels/f1/e4/16/4981ca97c2d65106b49861e0b35e2660695be7219a2d351ee0
    Successfully built citrination-client matminer pypif future configobj
    ERROR: nbclient 0.5.3 has requirement jupyter-client>=6.1.5, but you'll have jupyter-client 5.3.5 which is incompatible.
    ERROR: pymatgen 2021.3.9 has requirement numpy>=1.20.1, but you'll have numpy 1.19.5 which is incompatible.
    ERROR: pymatgen 2021.3.9 has requirement scipy>=1.5.0, but you'll have scipy 1.4.1 which is incompatible.
    ERROR: dlhub-sdk 0.9.4 has requirement requests>=2.24.0, but you'll have requests 2.23.0 which is incompatible.
    ERROR: matminer 0.6.5 has requirement scikit-learn>=0.23.1, but you'll have scikit-learn 0.22.2.post1 which is incompatible.
    ERROR: matminer 0.6.5 has requirement tqdm>=4.46.0, but you'll have tqdm 4.41.1 which is incompatible.
    Installing collected packages: cryptography, pyjwt, globus-sdk, globus-nexus-client, fair-research-login, mdf-toolbox, pypif, pyyaml, citrination-client, sphinx-automodapi, ruamel.yaml.clib, ruamel.yaml, plotly, monty, spglib, future, uncertainties, pymatgen, pyaml, scikit-optimize, mdf-forge, jsonschema, configobj, pynacl, bcrypt, paramiko, psutil, typeguard, parsl, typer, texttable, lockfile, python-daemon, funcx, dlhub-sdk, pint, matminer, mastml
      Found existing installation: PyYAML 3.13
        Uninstalling PyYAML-3.13:
          Successfully uninstalled PyYAML-3.13
      Found existing installation: plotly 4.4.1
        Uninstalling plotly-4.4.1:
          Successfully uninstalled plotly-4.4.1
      Found existing installation: future 0.16.0
        Uninstalling future-0.16.0:
          Successfully uninstalled future-0.16.0
      Found existing installation: jsonschema 2.6.0
        Uninstalling jsonschema-2.6.0:
          Successfully uninstalled jsonschema-2.6.0
      Found existing installation: psutil 5.4.8
        Uninstalling psutil-5.4.8:
          Successfully uninstalled psutil-5.4.8
      Found existing installation: typeguard 2.7.1
        Uninstalling typeguard-2.7.1:
          Successfully uninstalled typeguard-2.7.1
    Successfully installed bcrypt-3.2.0 citrination-client-6.5.1 configobj-5.0.6 cryptography-3.4.7 dlhub-sdk-0.9.4 fair-research-login-0.2.0 funcx-0.2.0 future-0.18.2 globus-nexus-client-0.3.0 globus-sdk-2.0.1 jsonschema-3.2.0 lockfile-0.12.2 mastml-3.0.0 matminer-0.6.5 mdf-forge-0.7.6 mdf-toolbox-0.5.7 monty-2021.3.3 paramiko-2.7.2 parsl-1.1.0a1 pint-0.17 plotly-4.14.3 psutil-5.8.0 pyaml-20.4.0 pyjwt-1.7.1 pymatgen-2021.3.9 pynacl-1.4.0 pypif-2.1.2 python-daemon-2.3.0 pyyaml-5.4.1 ruamel.yaml-0.17.4 ruamel.yaml.clib-0.2.2 scikit-optimize-0.8.1 spglib-1.16.1 sphinx-automodapi-0.13 texttable-1.6.3 typeguard-2.12.0 typer-0.3.2 uncertainties-3.1.5

Sync your Google drive to Colab so that we can save MAST-ML results to our Google
Drive. If we save to the Colab session, the data will be deleted when the session
ends.

```python
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
```

    Mounted at /content/drive

Here we import the MAST-ML modules used in this tutorial

```python
from mastml.mastml import Mastml
from mastml.datasets import SklearnDatasets, LocalDatasets, MatminerDatasets, FigshareDatasets, FoundryDatasets
from mastml.data_cleaning import DataCleaning
import numpy as np
from copy import copy
import mastml
import os
data_path = os.path.join(mastml.__path__._path[0], 'data')
```

Here, we set the name of the savepath to save MAST-ML results to, and initialize
our MAST-ML run

When the above command is run, a new folder with the name designated SAVEPATH is created.
This is where all of the output for the current MAST-ML run will be saved to.
Note that you can perform multiple runs with the same folder name, and the current datetime
will be appended to the name so that no data is lost or overwritten.

```python
SAVEPATH = 'drive/MyDrive/MASTML_tutorial_2_DataImport'

mastml = Mastml(savepath=SAVEPATH)
savepath = mastml.get_savepath
```

## Task 2: Import model datasets from scikit-learn

Let's begin by first showing how to load some common model datasets that
come with the scikit-learn package. As in the first tutorial, we use the SklearnDatasets module
to load in the data. There are a few regression datasets that come with scikit-learn, such as
the Boston housing data, the diabetes dataset, and the Friedman data set. For now,
let's once again load in the Boston housing dataset.

```python
X, y = SklearnDatasets(as_frame=True).load_boston()
```

Let's quickly examine the X data so we know what we're dealing with:

```python
X
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      69.1
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      76.7
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      91.0
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      89.3
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      80.8
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

506 rows × 13 columns

## Task 3: Conduct different data cleaning method

We have 506 data points (houses) and 13 columns (features). From inspecting the dataset,
we can see there are no missing values in the dataset. However, many datasets in the real
world aren't this neat and tidy, and may contain values that are missing. To illustrate
how we can go about cleaning our data, let's make the first few rows of values in the AGE
column not-a-number (NaN), which is what they will be if you import a dataset with missing values:

```python
X['AGE'][0:4] = np.nan
X
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      NaN
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      NaN
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      NaN
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      NaN
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      69.1
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      76.7
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      91.0
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      89.3
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      80.8
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

506 rows × 13 columns

We can now see that the first four rows of the 'AGE' column are NaN instead
of their original values. Let's clean the data using a few different methods
in the DataCleaning class. First, let's copy the X data so we can more easily
see the changes we make to each dataset

```python
X1row = X1col = X2 = X3 = X4 = copy(X)
```

There are few different ways we can clean the data: (1) we can simply
remove the data points that have missing values, (2) we can perform
data imputation to replace the missing values with the mean (or median) value

Let's start with simply removing the problematic data points:

```python
X1row, y1 = DataCleaning().remove(X=X1row,
                                  y=y,
                                  axis=0)
X1row
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      5
      0.02985
      0.0
      2.18
      0.0
      0.458
      6.430
      58.7
      6.0622
      3.0
      222.0
      18.7
      394.12
      5.21

      6
      0.08829
      12.5
      7.87
      0.0
      0.524
      6.012
      66.6
      5.5605
      5.0
      311.0
      15.2
      395.60
      12.43

      7
      0.14455
      12.5
      7.87
      0.0
      0.524
      6.172
      96.1
      5.9505
      5.0
      311.0
      15.2
      396.90
      19.15

      8
      0.21124
      12.5
      7.87
      0.0
      0.524
      5.631
      100.0
      6.0821
      5.0
      311.0
      15.2
      386.63
      29.93

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      69.1
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      76.7
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      91.0
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      89.3
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      80.8
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

502 rows × 13 columns

What we can see is that we now have 502 rows instead of the original 506.
The four data points that had missing values in the AGE column have been removed.

Note that we could have instead removed the column (feature) containing
the missing values, as in our case all of the missing values were part of
a single feature:

```python
X1col, y1 = DataCleaning().remove(X=X1col,
                                  y=y,
                                  axis=1)
X1col
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

506 rows × 12 columns

What we can see is that we now have all 506 of our original data points,
but we now just have 12 of the 13 features as the AGE feature has been removed.

There are smarter things we can do besides simply removing the problematic
data points. Here, let's use the imputation process to replace the missing
values with the mean value of the feature column

```python
X2, y2 = DataCleaning().imputation(X=X2,
                                   y=y,
                                   strategy='mean')
X2
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      68.621315
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      68.621315
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      68.621315
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      68.621315
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.200000
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      69.100000
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      76.700000
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      91.000000
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      89.300000
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      80.800000
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

506 rows × 13 columns

What we can see is that we now have all 506 of our original data points and
all 13 of our feature columns, and the first four rows of the AGE feature
now contain the average value for that feature.

Finally, let's do imputation again but use the median instead of
the mean value

```python
X3, y3 = DataCleaning().imputation(X=X3,
                                   y=y,
                                   strategy='median')
X3
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      77.7
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      77.7
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      77.7
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      77.7
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      501
      0.06263
      0.0
      11.93
      0.0
      0.573
      6.593
      69.1
      2.4786
      1.0
      273.0
      21.0
      391.99
      9.67

      502
      0.04527
      0.0
      11.93
      0.0
      0.573
      6.120
      76.7
      2.2875
      1.0
      273.0
      21.0
      396.90
      9.08

      503
      0.06076
      0.0
      11.93
      0.0
      0.573
      6.976
      91.0
      2.1675
      1.0
      273.0
      21.0
      396.90
      5.64

      504
      0.10959
      0.0
      11.93
      0.0
      0.573
      6.794
      89.3
      2.3889
      1.0
      273.0
      21.0
      393.45
      6.48

      505
      0.04741
      0.0
      11.93
      0.0
      0.573
      6.030
      80.8
      2.5050
      1.0
      273.0
      21.0
      396.90
      7.88

506 rows × 13 columns

What we can see is that we now have all 506 of our original data points and
all 13 of our feature columns, and the first four rows of the AGE feature
now contain the median value for that feature.

MAST-ML provides a convenient method for evaluating a data cleaning routine,
and saving the output to a dedicated folder in your MAST-ML output directory
Let's do imputation again on our final remaining dataset and examine what's
contained in the saved output:

```python
X4, y4 = DataCleaning().evaluate(X=X4,
                                 y=y,
                                 method='imputation',
                                 savepath=savepath,
                                 strategy='mean')
```

Check your MAST-ML save directory: you'll see a new folder starting with the name "DataCleaning_..."
In here, there are numerous Excel data files and a histogram plot. Below is a brief overview
of what is contained in these files. One useful visualization is the histogram of the target data
you are trying to fit machine learning models to

**data_cleaned.xlsx:** the cleaned dataset

**data_columns_with_strings**: listed column names that contain string entries. Empty for our dataset

**data_original.xlsx**: the original dataset prior to cleaning

**data_outliers_all.xlsx**: a summary of the possible outlier points for each feature column

**data_outliers_summary.xlsx**: a summary showing, for each data point, the number of feature values which may be outliers

**histogram_target_values_statistics.xlsx**: key summary statistics of the distribution of y target data

**histogram_target_values.png**: histogram plot of the y target data with some basic statistics included

**histogram_target_values.xlsx**: the raw data used to make the histogram plot of the same name

### DataCleaning directory with files:

### Histogram of target values:

## Task 3: Import and prepare a real dataset that is stored locally

Next, we want to move away from the model datasets contained in scikit-learn and learn how to import
a real materials science dataset. As part of MAST-ML, we have included a dataset in the mastml/data
folder. The data is contained there as diffusion_data_selectfeatures.xlsx. This dataset contains
calculated migration energies of solute alloys in metal host materials.

Here, we use the LocalDatasets module to load in the above mentioned diffusion dataset.

We first need to denote the column name of the target (y-data)

In addition, there are columns in the data file not used as features or target. We need to
list them here in the parameter extra_columns

```python
target = 'E_regression'

extra_columns = ['Material compositions 1', 'Material compositions 2', 'Hop activation barrier']
```

Here, we make an instance of our LocalDatasets class. It needs a few parameters:

**file_path**: where the data is stored

**target**: the column name of the y-data

**extra_columns**: list containing extra columns in the data file not used for fitting

**group_column**: column name denoting group labels (only used for LeaveOutGroup CV)

**testdata_columns**: column names denoting left-out data to evaluate using best
     model from CV tests. This is manual way to leave out data. Can also be done
     automatically using nested CV (we will do this in later tutorials)

**as_frame**: whether to return data as dataframe. Want this to be true.

We load the data with the load_data() method

```python
d = LocalDatasets(file_path=data_path+'/diffusion_data_allfeatures.xlsx',
                  target=target,
                  extra_columns=extra_columns,
                  group_column='Material compositions 1',
                  testdata_columns=None,
                  as_frame=True)

data_dict = d.load_data()
```

    WARNING: feature_names not specified but target was specified. Assuming all columns except target and extra columns are features

Let's look at the contents of the loaded data_dict

We see there are 5 keys:

**X**: the X feature matrix (used to fit the ML model)

**y**: the y target data vector (true values)

**X_extra**: matrix of extra information not used in fitting (i.e. not part of X or y)

**groups**: vector of group labels (here, a list of host elements)

**X_testdata**: matrix or vector of left out data. Empty for our current example.

```python
data_dict.keys()
```

    dict_keys(['X', 'y', 'groups', 'X_extra', 'X_testdata'])

Let's assign each data object to its respective name

```python
X = data_dict['X']
y = data_dict['y']
X_extra = data_dict['X_extra']
groups = data_dict['groups']
X_testdata = data_dict['X_testdata']
```

Let's have a look at the X feature matrix. It contains 287 elemental features.
There are a total of 408 diffusion activation barriers used for fitting

```python
X
```

      valence_composition_average
      phi_composition_average
      NdValence_composition_average
      MiracleRadius_composition_average
      GSestFCClatcnt_composition_average
      SecondIonizationEnergy_composition_average
      IonizationEnergy_composition_average
      n_ws^third_composition_average
      ThermalConductivity_composition_average
      CovalentRadius_composition_average
      Column_composition_average
      NsValence_composition_average
      NUnfilled_composition_average
      CovalentRadii_composition_average
      Density_composition_average
      BCCenergy_pa_composition_average
      BCCvolume_padiff_composition_average
      BCCvolume_pa_composition_average
      MeltingT_composition_average
      SpaceGroupNumber_composition_average
      NValance_composition_average
      ElectricalConductivity_composition_average
      FirstIonizationEnergy_composition_average
      GSvolume_pa_composition_average
      Row_composition_average
      Polarizability_composition_average
      AtomicNumber_composition_average
      ElectronAffinity_composition_average
      BCCfermi_composition_average
      GSestBCClatcnt_composition_average
      HHIr_composition_average
      HHIp_composition_average
      GSenergy_pa_composition_average
      ThirdIonizationEnergy_composition_average
      BulkModulus_composition_average
      BoilingT_composition_average
      ICSDVolume_composition_average
      HeatVaporization_composition_average
      IonicRadii_composition_average
      NdUnfilled_composition_average
      ...
      Site2_NdValence
      Site2_MiracleRadius
      Site2_GSestFCClatcnt
      Site2_IonizationEnergy
      Site2_ThermalConductivity
      Site2_CovalentRadius
      Site2_Column
      Site2_NUnfilled
      Site2_CovalentRadii
      Site2_Density
      Site2_BCCenergy_pa
      Site2_BCCvolume_padiff
      Site2_BCCvolume_pa
      Site2_MeltingT
      Site2_SpaceGroupNumber
      Site2_NValance
      Site2_ElectricalConductivity
      Site2_FirstIonizationEnergy
      Site2_GSvolume_pa
      Site2_Row
      Site2_Polarizability
      Site2_AtomicNumber
      Site2_BCCfermi
      Site2_GSestBCClatcnt
      Site2_GSenergy_pa
      Site2_BoilingT
      Site2_ICSDVolume
      Site2_HeatVaporization
      Site2_IonicRadii
      Site2_NdUnfilled
      Site2_MendeleevNumber
      Site2_ElasticModulus
      Site2_Electronegativity
      Site2_AtomicWeight
      Site2_HeatFusion
      Site2_SpecificHeatCapacity
      Site2_AtomicRadii
      Site2_ThermalExpansionCoefficient
      Site2_BCCefflatcnt
      Site2_AtomicVolume

      0
      2.0
      4.350
      10.0
      144.0
      4.027313
      21.4900
      731.00
      1.360
      429.00
      145.0
      11.0
      1.0
      1.0
      1.340
      10490.0
      -2.719979
      -0.1300
      16.200
      1234.930
      225.0
      11.0
      62.90
      7.576230
      16.3300
      5.0
      6.990
      47.0
      125.70
      4.052760
      3.196481
      1400
      1200
      -2.765394
      34.8300
      100.0
      2435.0
      17.00
      250.630
      1.150
      0.0
      ...
      10
      144
      4.027313
      731.0
      429.0
      145
      11
      1
      1.34
      10490
      -2.719979
      -0.130
      16.20
      1234.93
      225
      11
      62.9
      7.57623
      16.330
      5
      6.99
      47
      4.052760
      3.196481
      -2.765394
      2435.0
      17.0
      250.63
      1.15
      0
      65
      80.0
      1.93
      107.868200
      11.30
      0.235
      1.444
      18.9
      6.375951
      17.075648

      1
      2.5
      4.725
      8.5
      134.5
      3.737485
      19.2750
      744.50
      1.555
      264.50
      135.5
      10.0
      1.5
      2.0
      1.250
      9695.0
      -4.850033
      0.0325
      13.320
      1501.465
      209.5
      10.0
      40.40
      7.728620
      13.2875
      4.5
      7.245
      37.0
      97.85
      5.096587
      2.966444
      2050
      2150
      -4.922891
      34.1650
      140.0
      2817.5
      14.05
      311.965
      0.900
      1.5
      ...
      7
      125
      3.447656
      758.0
      100.0
      126
      9
      3
      1.16
      8900
      -6.980087
      0.195
      10.44
      1768.00
      194
      9
      17.9
      7.88101
      10.245
      4
      7.50
      27
      6.140415
      2.736407
      -7.080389
      3200.0
      11.1
      373.30
      0.65
      3
      58
      208.0
      1.88
      58.933195
      16.20
      0.421
      1.253
      13.0
      5.507318
      10.995861

      2
      4.0
      4.500
      7.5
      137.0
      3.788936
      18.9950
      691.90
      1.545
      261.35
      142.0
      8.5
      1.0
      3.5
      1.260
      8815.0
      -6.112393
      -0.0750
      13.685
      1707.465
      227.0
      8.5
      35.40
      7.171370
      13.7600
      4.5
      9.295
      35.5
      94.85
      6.014901
      3.007280
      2750
      2150
      -6.135555
      32.8950
      130.0
      2689.5
      14.50
      295.065
      0.885
      2.5
      ...
      5
      130
      3.550559
      652.8
      93.7
      139
      6
      6
      1.18
      7140
      -9.504808
      -0.020
      11.17
      2180.00
      229
      6
      7.9
      6.76651
      11.190
      4
      11.60
      24
      7.977043
      2.818080
      -9.505717
      2944.0
      12.0
      339.50
      0.62
      5
      49
      259.0
      1.66
      51.996100
      21.00
      0.449
      1.249
      4.9
      5.632801
      12.092937

      3
      2.0
      4.400
      10.0
      135.0
      3.782567
      20.8910
      738.20
      1.415
      415.00
      138.5
      11.0
      1.0
      1.0
      1.255
      9705.0
      -3.181433
      -0.0600
      13.640
      1296.350
      225.0
      11.0
      61.80
      7.651305
      13.7000
      4.5
      6.570
      38.0
      122.00
      4.215750
      3.002225
      1450
      1400
      -3.223024
      35.8300
      120.0
      2817.5
      14.40
      275.565
      0.940
      0.0
      ...
      10
      126
      3.537821
      745.4
      401.0
      132
      11
      1
      1.17
      8920
      -3.642887
      0.010
      11.08
      1357.77
      225
      11
      60.7
      7.72638
      11.070
      4
      6.15
      29
      4.378740
      2.807970
      -3.680655
      3200.0
      11.8
      300.50
      0.73
      0
      64
      124.0
      1.90
      63.546000
      13.60
      0.385
      1.278
      16.5
      5.617632
      11.829942

      4
      4.0
      4.640
      8.0
      134.5
      3.764269
      18.8350
      745.15
      1.565
      254.60
      138.5
      9.5
      1.5
      2.5
      1.255
      9182.0
      -5.502732
      -0.0650
      13.465
      1522.965
      227.0
      9.5
      37.05
      7.739315
      13.5300
      4.5
      7.695
      36.5
      74.85
      5.252127
      2.987702
      1400
      1800
      -5.525439
      32.7405
      135.0
      2784.5
      14.35
      300.065
      0.850
      2.0
      ...
      6
      125
      3.501224
      759.3
      80.2
      132
      8
      4
      1.17
      7874
      -8.285484
      0.000
      10.73
      1811.00
      229
      8
      11.2
      7.90240
      10.730
      4
      8.40
      26
      6.451493
      2.778923
      -8.285484
      3134.0
      11.7
      349.50
      0.55
      4
      55
      211.0
      1.83
      55.845000
      13.81
      0.449
      1.241
      11.8
      5.557847
      11.777365

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      403
      4.5
      3.750
      3.0
      150.5
      4.350534
      13.7250
      662.00
      1.525
      38.20
      169.5
      4.5
      1.5
      7.5
      1.395
      7540.5
      -9.280298
      -0.2225
      20.465
      2439.000
      211.5
      4.5
      4.45
      6.696375
      20.6875
      5.0
      16.800
      40.5
      73.00
      4.564936
      3.453021
      5700
      5950
      -9.319640
      24.0150
      85.0
      4849.5
      20.65
      640.300
      0.740
      7.0
      ...
      4
      143
      4.173989
      664.0
      53.7
      164
      5
      7
      1.34
      8570
      -10.093322
      0.000
      18.18
      2750.00
      229
      5
      6.6
      6.75885
      18.180
      5
      15.70
      41
      4.901822
      3.312897
      -10.093322
      5017.0
      18.1
      690.10
      0.64
      6
      47
      104.0
      1.60
      92.906380
      30.00
      0.265
      1.429
      7.3
      6.625794
      18.002133

      404
      4.5
      3.750
      2.5
      151.5
      4.348236
      6.5650
      710.50
      1.520
      40.10
      172.5
      4.5
      2.0
      7.5
      1.395
      11580.5
      -10.159337
      -0.2225
      20.435
      2709.000
      211.5
      11.5
      5.20
      7.091735
      20.6575
      5.5
      15.500
      56.5
      55.00
      4.832147
      3.451197
      3700
      2850
      -10.198679
      11.4950
      100.0
      5206.5
      20.40
      663.750
      0.740
      7.5
      ...
      3
      145
      4.169392
      761.0
      57.5
      170
      5
      7
      1.34
      16650
      -11.851400
      0.000
      18.12
      3290.00
      229
      19
      8.1
      7.54957
      18.120
      6
      13.10
      73
      5.436244
      3.309249
      -11.851400
      5731.0
      17.6
      737.00
      0.64
      7
      48
      183.0
      1.50
      180.947880
      36.57
      0.140
      1.430
      6.3
      6.618497
      18.046730

      405
      4.0
      3.625
      2.0
      150.0
      4.291886
      13.3550
      659.00
      1.465
      22.30
      167.5
      4.0
      2.0
      8.0
      1.385
      5509.0
      -8.064271
      -0.1475
      19.795
      2034.500
      194.0
      4.0
      2.45
      6.731010
      19.9425
      4.5
      16.250
      31.0
      35.00
      3.602101
      3.406472
      2100
      2250
      -8.160593
      25.2405
      55.0
      4121.0
      20.40
      507.850
      0.725
      8.0
      ...
      2
      142
      4.056693
      658.0
      21.9
      160
      4
      8
      1.32
      4507
      -7.661268
      0.150
      16.84
      1941.00
      194
      4
      2.6
      6.82812
      16.690
      4
      14.60
      22
      2.976151
      3.219799
      -7.775227
      3560.0
      17.6
      425.20
      0.61
      8
      43
      110.0
      1.54
      47.867000
      14.15
      0.523
      1.448
      8.6
      6.458833
      17.636317

      406
      4.0
      3.525
      2.0
      158.0
      4.494239
      14.0275
      651.00
      1.430
      22.85
      175.0
      4.0
      2.0
      8.0
      1.445
      9910.5
      -9.120962
      -0.3875
      22.310
      2317.000
      194.0
      11.0
      2.85
      6.729485
      22.6975
      5.5
      17.050
      56.0
      25.00
      4.480361
      3.567080
      2600
      3400
      -9.250129
      23.1550
      55.0
      4779.0
      22.80
      625.785
      0.835
      8.0
      ...
      2
      158
      4.461398
      642.0
      23.0
      175
      4
      8
      1.44
      13310
      -9.774649
      -0.330
      21.87
      2506.00
      194
      18
      3.4
      6.82507
      22.200
      6
      16.20
      72
      4.732671
      3.541014
      -9.954299
      4876.0
      22.4
      661.07
      0.83
      8
      45
      139.0
      1.30
      178.490000
      27.20
      0.140
      1.564
      5.9
      7.046762
      22.268711

      407
      4.0
      3.800
      6.0
      156.5
      4.816669
      13.8810
      684.30
      1.325
      44.65
      157.0
      9.0
      2.0
      6.0
      1.430
      6910.5
      -6.181100
      -4.2500
      23.990
      1316.540
      167.5
      9.0
      5.50
      6.988910
      28.2400
      5.0
      12.480
      45.0
      85.50
      6.639299
      3.822993
      2100
      3000
      -6.253503
      26.7460
      29.0
      3778.5
      25.10
      440.435
      0.775
      4.0
      ...
      10
      155
      5.106259
      708.6
      66.6
      139
      14
      4
      1.41
      7310
      -3.894925
      -8.055
      25.23
      505.08
      141
      14
      8.7
      7.34392
      33.285
      5
      7.06
      50
      9.050547
      4.052841
      -3.961047
      2875.0
      27.0
      290.37
      0.71
      0
      80
      50.0
      1.96
      118.710000
      7.15
      0.228
      1.620
      22.0
      7.390589
      26.966785

408 rows × 287 columns

Let's look at the groups list. The groups denote the element identity of the host metal

```python
groups
```

    0      Ag
    1      Ag
    2      Ag
    3      Ag
    4      Ag
           ..
    403    Zr
    404    Zr
    405    Zr
    406    Zr
    407    Zr
    Name: Material compositions 1, Length: 408, dtype: object

Let's examine the extra data not used in fitting. Note that the y-data we are
fitting to are equal to the E_regression values listed here, minus the E_regression
value of the pure material. Material compositions 1 are the host elements, which
are the same values used in the groups list. Material compositions 2 are the solute
elements (i.e. the diffusing impurity in the host material)

```python
X_extra
```

      Material compositions 1
      Material compositions 2
      Hop activation barrier

      0
      Ag
      Ag
      -0.365590

      1
      Ag
      Co
      -0.675263

      2
      Ag
      Cr
      -0.047690

      3
      Ag
      Cu
      -0.428459

      4
      Ag
      Fe
      0.081529

      ...
      ...
      ...
      ...

      403
      Zr
      Nb
      -0.478007

      404
      Zr
      Ta
      -0.054726

      405
      Zr
      Ti
      -0.022476

      406
      Zr
      Hf
      -0.104201

      407
      Zr
      Sn
      -0.045931

408 rows × 3 columns

Finally, we can look at our manually left-out data set. For this tutorial we
have left this blank, but we will make use of it in a future tutorial

```python
X_testdata
```

## Task 4: Download and import data from various materials databases

Next, let's download some datasets from remote materials databases, then import
them for use in MAST-ML

Here, we will download a materials dataset of steel alloy yield strengths from the
matminer datasets repository:

```python
df = MatminerDatasets().download_data(name='steel_strength')
```

    Fetching steel_strength.json.gz from https://ndownloader.figshare.com/files/13354691 to /usr/local/lib/python3.7/dist-packages/matminer/datasets/steel_strength.json.gz

The output of downloading this data is a pandas dataframe. If we know which columns
correspond to X, y, X_extra, and so on, we can designate those fields here through
pandas commands. Alternatively, we can import the just-saved steel_strength.xlsx file
using the LocalDatasets class just like we did in the previous task:

```python
target = 'yield strength'
extra_columns = ['formula', 'tensile strength', 'elongation']

d = LocalDatasets(file_path='steel_strength.xlsx',
                  target=target,
                  extra_columns=extra_columns,
                  group_column=None,
                  testdata_columns=None,
                  as_frame=True)

# Load the data with the load_data() method
data_dict = d.load_data()

# Designate the X, y and X_extra data
X = data_dict['X']
y = data_dict['y']
X_extra = data_dict['X_extra']
```

    WARNING: feature_names not specified but target was specified. Assuming all columns except target and extra columns are features

Let's have a look at the X feature matrix for this data.
The X features are the atom fractions of different elements
in each steel alloy

```python
X
```

      c
      mn
      si
      cr
      ni
      mo
      v
      n
      nb
      co
      w
      al
      ti

      0
      0.02
      0.05
      0.05
      0.01
      19.70
      2.95
      0.01
      0.00
      0.01
      15.00
      0.00
      0.15
      1.55

      1
      0.18
      0.01
      0.01
      13.44
      0.01
      3.01
      0.46
      0.04
      0.01
      19.46
      2.35
      0.04
      0.00

      2
      0.00
      0.01
      0.01
      8.67
      13.45
      0.82
      0.01
      0.00
      0.01
      13.90
      0.00
      0.39
      0.57

      3
      0.01
      0.05
      0.05
      0.01
      17.70
      3.95
      0.01
      0.00
      0.01
      15.00
      0.00
      0.13
      1.47

      4
      0.01
      0.05
      0.05
      0.01
      19.40
      1.45
      0.01
      0.00
      0.01
      14.90
      0.00
      0.13
      1.55

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      307
      0.38
      0.18
      0.01
      7.27
      0.01
      3.77
      0.96
      0.00
      0.01
      4.90
      0.00
      0.03
      0.00

      308
      0.00
      0.06
      0.05
      5.15
      10.20
      3.20
      0.01
      0.00
      0.01
      0.01
      0.00
      0.03
      0.09

      309
      0.37
      0.17
      0.01
      5.20
      0.01
      5.84
      1.05
      0.00
      0.91
      4.88
      0.00
      0.03
      0.00

      310
      0.41
      0.19
      0.01
      6.99
      0.01
      5.84
      0.92
      0.00
      0.01
      0.02
      0.00
      0.03
      0.00

      311
      0.27
      0.27
      0.01
      0.41
      8.28
      0.49
      0.07
      0.00
      0.01
      3.90
      0.00
      0.03
      0.00

312 rows × 13 columns

Let's have a look at the y target data. They data are yield strengths (in MPa) for each alloy

```python
y
```

    0      2411.5
    1      1123.1
    2      1736.3
    3      2487.3
    4      2249.6
            ...
    307    1722.5
    308    1019.0
    309    1860.3
    310    1812.1
    311    1139.7
    Name: yield strength, Length: 312, dtype: float64

There are many other datasets besides this steel strength dataset
available on the matminer repository. Information on these other
datasets, and the names used to query them, can be output using:

```python
MatminerDatasets().get_available_datasets()
```

    boltztrap_mp: Effective mass and thermoelectric properties of 8924 compounds in The  Materials Project database that are calculated by the BoltzTraP software package run on the GGA-PBE or GGA+U density functional theory calculation results. The properties are reported at the temperature of 300 Kelvin and the carrier concentration of 1e18 1/cm3.

    brgoch_superhard_training: 2574 materials used for training regressors that predict shear and bulk modulus.

    castelli_perovskites: 18,928 perovskites generated with ABX combinatorics, calculating gllbsc band gap and pbe structure, and also reporting absolute band edge positions and heat of formation.

    citrine_thermal_conductivity: Thermal conductivity of 872 compounds measured experimentally and retrieved from Citrine database from various references. The reported values are measured at various temperatures of which 295 are at room temperature.

    dielectric_constant: 1,056 structures with dielectric properties, calculated with DFPT-PBE.

    double_perovskites_gap: Band gap of 1306 double perovskites (a_1-b_1-a_2-b_2-O6) calculated using Gritsenko, van Leeuwen, van Lenthe and Baerends potential (gllbsc) in GPAW.

    double_perovskites_gap_lumo: Supplementary lumo data of 55 atoms for the double_perovskites_gap dataset.

    elastic_tensor_2015: 1,181 structures with elastic properties calculated with DFT-PBE.

    expt_formation_enthalpy: Experimental formation enthalpies for inorganic compounds, collected from years of calorimetric experiments. There are 1,276 entries in this dataset, mostly binary compounds. Matching mpids or oqmdids as well as the DFT-computed formation energies are also added (if any).

    expt_gap: Experimental band gap of 6354 inorganic semiconductors.

    flla: 3938 structures and computed formation energies from "Crystal Structure Representations for Machine Learning Models of Formation Energies."

    glass_binary: Metallic glass formation data for binary alloys, collected from various experimental techniques such as melt-spinning or mechanical alloying. This dataset covers all compositions with an interval of 5 at. % in 59 binary systems, containing a total of 5959 alloys in the dataset. The target property of this dataset is the glass forming ability (GFA), i.e. whether the composition can form monolithic glass or not, which is either 1 for glass forming or 0 for non-full glass forming.

    glass_binary_v2: Identical to glass_binary dataset, but with duplicate entries merged. If there was a disagreement in gfa when merging the class was defaulted to 1.

    glass_ternary_hipt: Metallic glass formation dataset for ternary alloys, collected from the high-throughput sputtering experiments measuring whether it is possible to form a glass using sputtering. The hipt experimental data are of the Co-Fe-Zr, Co-Ti-Zr, Co-V-Zr and Fe-Ti-Nb ternary systems.

    glass_ternary_landolt: Metallic glass formation dataset for ternary alloys, collected from the "Nonequilibrium Phase Diagrams of Ternary Amorphous Alloys,’ a volume of the Landolt– Börnstein collection. This dataset contains experimental measurements of whether it is possible to form a glass using a variety of processing techniques at thousands of compositions from hundreds of ternary systems. The processing techniques are designated in the "processing" column. There are originally 7191 experiments in this dataset, will be reduced to 6203 after deduplicated, and will be further reduced to 6118 if combining multiple data for one composition. There are originally 6780 melt-spinning experiments in this dataset, will be reduced to 5800 if deduplicated, and will be further reduced to 5736 if combining multiple experimental data for one composition.

    heusler_magnetic: 1153 Heusler alloys with DFT-calculated magnetic and electronic properties. The 1153 alloys include 576 full, 449 half and 128 inverse Heusler alloys. The data are extracted and cleaned (including de-duplicating) from Citrine.

    jarvis_dft_2d: Various properties of 636 2D materials computed with the OptB88vdW and TBmBJ functionals taken from the JARVIS DFT database.

    jarvis_dft_3d: Various properties of 25,923 bulk materials computed with the OptB88vdW and TBmBJ functionals taken from the JARVIS DFT database.

    jarvis_ml_dft_training: Various properties of 24,759 bulk and 2D materials computed with the OptB88vdW and TBmBJ functionals taken from the JARVIS DFT database.

    m2ax: Elastic properties of 223 stable M2AX compounds from "A comprehensive survey of M2AX phase elastic properties" by Cover et al. Calculations are PAW PW91.

    matbench_dielectric: Matbench v0.1 test dataset for predicting refractive index from structure. Adapted from Materials Project database. Removed entries having a formation energy (or energy above the convex hull) more than 150meV and those having refractive indices less than 1 and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_expt_gap: Matbench v0.1 test dataset for predicting experimental band gap from composition alone. Retrieved from Zhuo et al. supplementary information. Deduplicated according to composition, removing compositions with reported band gaps spanning more than a 0.1eV range; remaining compositions were assigned values based on the closest experimental value to the mean experimental value for that composition among all reports. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_expt_is_metal: Matbench v0.1 test dataset for classifying metallicity from composition alone. Retrieved from Zhuo et al. supplementary information. Deduplicated according to composition, ensuring no conflicting reports were entered for any compositions (i.e., no reported compositions were both metal and nonmetal). For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_glass: Matbench v0.1 test dataset for predicting full bulk metallic glass formation ability from chemical formula. Retrieved from "Nonequilibrium Phase Diagrams of Ternary Amorphous Alloys,’ a volume of the Landolt– Börnstein collection. Deduplicated according to composition, ensuring no compositions were reported as both GFA and not GFA (i.e., all reports agreed on the classification designation). For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_jdft2d: Matbench v0.1 test dataset for predicting exfoliation energies from crystal structure (computed with the OptB88vdW and TBmBJ functionals). Adapted from the JARVIS DFT database. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_log_gvrh: Matbench v0.1 test dataset for predicting DFT log10 VRH-average shear modulus from structure. Adapted from Materials Project database. Removed entries having a formation energy (or energy above the convex hull) more than 150meV and those having negative G_Voigt, G_Reuss, G_VRH, K_Voigt, K_Reuss, or K_VRH and those failing G_Reuss <= G_VRH <= G_Voigt or K_Reuss <= K_VRH <= K_Voigt and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_log_kvrh: Matbench v0.1 test dataset for predicting DFT log10 VRH-average bulk modulus from structure. Adapted from Materials Project database. Removed entries having a formation energy (or energy above the convex hull) more than 150meV and those having negative G_Voigt, G_Reuss, G_VRH, K_Voigt, K_Reuss, or K_VRH and those failing G_Reuss <= G_VRH <= G_Voigt or K_Reuss <= K_VRH <= K_Voigt and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_mp_e_form: Matbench v0.1 test dataset for predicting DFT formation energy from structure. Adapted from Materials Project database. Removed entries having formation energy more than 3.0eV and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_mp_gap: Matbench v0.1 test dataset for predicting DFT PBE band gap from structure. Adapted from Materials Project database. Removed entries having a formation energy (or energy above the convex hull) more than 150meV and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_mp_is_metal: Matbench v0.1 test dataset for predicting DFT metallicity from structure. Adapted from Materials Project database. Removed entries having a formation energy (or energy above the convex hull) more than 150meV and those containing noble gases. Retrieved April 2, 2019. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_perovskites: Matbench v0.1 test dataset for predicting formation energy from crystal structure. Adapted from an original dataset generated by Castelli et al. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_phonons: Matbench v0.1 test dataset for predicting vibration properties from crystal structure. Original data retrieved from Petretto et al. Original calculations done via ABINIT in the harmonic approximation based on density functional perturbation theory. Removed entries having a formation energy (or energy above the convex hull) more than 150meV. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    matbench_steels: Matbench v0.1 test dataset for predicting steel yield strengths from chemical composition alone. Retrieved from Citrine informatics. Deduplicated. For benchmarking w/ nested cross validation, the order of the dataset must be identical to the retrieved data; refer to the Automatminer/Matbench publication for more details.

    mp_all_20181018: A complete copy of the Materials Project database as of 10/18/2018. mp_all files contain structure data for each material while mp_nostruct does not.

    mp_nostruct_20181018: A complete copy of the Materials Project database as of 10/18/2018. mp_all files contain structure data for each material while mp_nostruct does not.

    phonon_dielectric_mp: Phonon (lattice/atoms vibrations) and dielectric properties of 1296 compounds computed via ABINIT software package in the harmonic approximation based on density functional perturbation theory.

    piezoelectric_tensor: 941 structures with piezoelectric properties, calculated with DFT-PBE.

    steel_strength: 312 steels with experimental yield strength and ultimate tensile strength, extracted and cleaned (including de-duplicating) from Citrine.

    wolverton_oxides: 4,914 perovskite oxides containing composition data, lattice constants, and formation + vacancy formation energies. All perovskites are of the form ABO3. Adapted from a dataset presented by Emery and Wolverton.

As one last example, let's look at how to download datasets that are
hosted on Figshare. To use the Figshare data import, we need to
first install the python package to query Figshare datasets:

```python
!git clone https://github.com/cognoma/figshare.git
import sys
sys.path.append('figshare')
```

    Cloning into 'figshare'...
    remote: Enumerating objects: 18, done.
    remote: Counting objects: 100% (18/18), done.
    remote: Compressing objects: 100% (17/17), done.
    remote: Total 53 (delta 6), reused 9 (delta 1), pack-reused 35
    Unpacking objects: 100% (53/53), done.

Note that you may need to restart your runtime (if on Colab) so that Colab can find the Figshare module

Now we can download a dataset from Figshare. The diffusion dataset
we examined earlier is hosted on Figshare, with article ID number 7418492.
Let's download it to our working environment:

```python
from mastml.datasets import FigshareDatasets
FigshareDatasets().download_data(article_id='7418492')
```

    /usr/local/lib/python3.7/dist-packages/distributed/config.py:20: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      defaults = yaml.load(f)

We can now see the new folder "figshare_7418492" has been downloaded,
and within it is the data file "All_Model_Data.xlsx". This is the same
file we imported earlier from mastml/data when demonstrating the use
of the LocalDatasets class. As with this file or other datasets downloaded
from Figshare or other remote datasets, after downloading the data files
you can import them into MAST-ML using the LocalDatasets class as we
learned in this tutorial.

You've now completed your second MAST-ML tutorial notebook! Now that you're more familiar with working with datasets
within MAST-ML, the next thing for us to do is look at feature engineering in more detail.

The next example in this notebook series is titled MASTML_Tutorial_3_FeatureEngineering.ipynb, and will guide you through the process
of generating, preprocessing, and selecting features for a particular model and data split test.

```python

```


# An Overview of Scramjet Physics & Modeling

## 2021-06-14 · N E Davis for [CEESD](https://ceesd.illinois.edu/)

## Program

1. Make sure you know how to submit your time worked (if applicable).
2. Be in contact with your mentor.
3. Put together a summer workplan this week (overall target + some intermediate targets) and let Carl and me know.
4. COMMUNICATE COMMUNICATE COMMUNICATE

## Objectives

- Describe the qualitative behavior of a scramjet and distinguish it from other jet engines.
- Identify quantities in governing equations and list consequences of MIRGE-Com's assumptions.
- Distinguish experimental data collection from computer-based simulation data collection.

## Jet Propulsion Systems

1. Chemical rocket
2. Turbojet engine
3. Ramjet engine
4. Scramjet engine

## Newton's Third Law

$$
\mathbf{F}_\text{A}
=
-\mathbf{F}_\text{B}
$$

Action A equals reaction B; consider also conservation of momentum.

## Chemical Rocket

- Self-contained, not air-breathing

## Turbojet

- Air-breathing
- Stators and rotors accelerate and compress

## Ramjet

- Air-breathing
- No moving parts
- High-speed (transonic or supersonic)
- Requires external propulsion to get up to operating speed $>0.5 \text{Ma}$

## Scramjet

- Air-breathing
- No moving parts
- High-speed (supersonic or hypersonic)
- Requires external propulsion to get up to operating speed $> 1 \text{Ma}$

## Scramjet

## Scramjet

Steady-State Temperature Distribution:

Supersonic inviscid flow exhibits sharp shockwaves starting at surfaces and propagating throughout the medium.  Streamlines can change discontinuously, making them relatively mathematically intractable.

## Scramjet

> In a supersonic flow, because the local flow velocity is greater than the speed of sound, disturbances created at some point in the flow cannot work their way upstream (in contrast to subsonic flow). This property is one of the most significant physical differences between subsonic and supersonic flows. It is the basic reason why shock waves occur in supersonic flows but do not occur in steady subsonic flow.  [[Anderson, "Flight (Aerodynamics)"](https://www.sciencedirect.com/science/article/pii/B0122274105009157)]

## Scramjet

### Warm-Up Phase

## Scramjet

### Steady-State Phase

- Diamonds are “radical farms” and “ignition hotspots”.
- $x \propto t$

## Governing Equations

### Navier-Stokes Equations

$$
\frac{ \partial \left( \rho \varphi \right) }{ \partial t }
+ \nabla \cdot \left( \rho \vec{v} \varphi \right)
= \nabla \cdot \left( \Gamma ^{\varphi} \nabla \varphi \right)
+ Q ^{\varphi} \text{,}
$$

## Governing Equations

### Navier-Stokes Equations

$$
\int_{V_C} dV\, \nabla \cdot \left( \rho \vec{v} \varphi \right)
= \int_{V_C} dV\, \nabla \cdot \left( \Gamma ^{\varphi} \nabla \varphi \right)
+ \int_{V_C} dV\, Q ^{\varphi} \text{.}
$$

## Governing Equations

### Navier-Stokes Equations

$$
\oint_{\partial V_C} d\vec{S} \cdot \nabla \cdot \left( \rho \vec{v} \varphi \right)
= \oint_{\partial V_C} d\vec{S} \cdot \left( \Gamma ^{\varphi} \nabla \varphi \right)
+ \int_{V_C} dV\, Q ^{\varphi} \text{.}
$$

## Governing Equations

## Momentum & Mass Conservation

$$
\begin{align}
  \rho \left(\frac{\partial u_x}{\partial t} + u_x \frac{\partial u_x}{\partial x} + u_y \frac{\partial u_x}{\partial y} + u_z \frac{\partial u_x}{\partial z}\right)
    =& -\frac{\partial p}{\partial x} + \mu \left(\frac{\partial^2 u_x}{\partial x^2} + \frac{\partial^2 u_x}{\partial y^2} + \frac{\partial^2 u_x}{\partial z^2}\right) \\
    & - \mu \frac{\partial}{\partial x} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_x \\
  \rho \left(\frac{\partial u_y}{\partial t} + u_x \frac{\partial u_y}{\partial x} + u_y \frac{\partial u_y}{\partial y}+ u_z \frac{\partial u_y}{\partial z}\right)
    =& -\frac{\partial p}{\partial y} + \mu \left(\frac{\partial^2 u_y}{\partial x^2} + \frac{\partial^2 u_y}{\partial y^2} + \frac{\partial^2 u_y}{\partial z^2}\right) \\
    & - \mu \frac{\partial}{\partial y} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_y \\
  \rho \left(\frac{\partial u_z}{\partial t} + u_x \frac{\partial u_z}{\partial x} + u_y \frac{\partial u_z}{\partial y}+ u_z \frac{\partial u_z}{\partial z}\right)
    =& -\frac{\partial p}{\partial z} + \mu \left(\frac{\partial^2 u_z}{\partial x^2} + \frac{\partial^2 u_z}{\partial y^2} + \frac{\partial^2 u_z}{\partial z^2}\right) \\
    & - \mu \frac{\partial}{\partial z} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_z.
\end{align}
$$

$$
{\partial \rho \over \partial t} + {\partial (\rho u_x ) \over \partial x} + {\partial (\rho u_y) \over \partial y} + {\partial (\rho u_z) \over \partial z} = 0.
$$

## Governing Equations

## Momentum & Mass Conservation

$$
\begin{align}
  \rho \left(\frac{\partial u_x}{\partial t} + u_x \frac{\partial u_x}{\partial x} + u_y \frac{\partial u_x}{\partial y} + u_z \frac{\partial u_x}{\partial z}\right)
    =& -\frac{\partial p}{\partial x} + \mu \left(\frac{\partial^2 u_x}{\partial x^2} + \frac{\partial^2 u_x}{\partial y^2} + \frac{\partial^2 u_x}{\partial z^2}\right) \\
    & - \mu \frac{\partial}{\partial x} \left( \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z} \right) + \rho g_x \\
\end{align}
$$

$$
{\partial \rho \over \partial t} + {\partial (\rho u_x ) \over \partial x} + {\partial (\rho u_y) \over \partial y} + {\partial (\rho u_z) \over \partial z} = 0.
$$

- Which quantities appear in the Navier–Stokes equations?
- Which quantities in the Navier–Stokes equations are conservative?
- Compare equations to the [canonical MIRGE-Com formulation](https://mirgecom.readthedocs.io/en/latest/fluid.html).

## Other Relationships

**Non-dimensional thrust equation**:

$$
\frac{T}{\dot{m}_o a_o} = M_o\left(\frac{U_e}{U_o}-1\right)
$$

**Ratio of inlet to outlet velocity**:

$$
\frac{U_e}{U_o} = \frac{M_e}{M_o} \sqrt{\frac{T_e}{T_o}}
$$

**Ramjet specific thrust** (across burner 3→4):

$$
\frac{T}{\dot{m}_o a_o} = M_o\left(\sqrt{\frac{T_{t4}}{T_{t3}}}-1\right)
$$

**Energy balance across burner**:

$$
\dot{m}_f h = \dot{m}_o c_p \left(T_{t4}-T_{t3}\right)
$$

## Other Relationships

**Specific impulse**:

$$
I_\text{sp} = \frac{T}{\dot{m}_f g}
$$

**Thrust specific fuel consumption**:

$$
\text{TSFC} = \frac{\dot{m}_f}{T}
$$

**Overall efficiency**:

$$
\eta = \frac{T U_o}{\dot{m}_f h}
$$

## Experimental Data

Scramjet observational data are obtained two ways:  from wind tunnel experiments and from actual flights.

Data which can be obtained experimentally include:

- Temperature and pressure distributions along various lines
- Thrust and performance under various conditions
- Fuel composition at sample points.

## Challenges

> While scramjets are conceptually simple, actual implementation is limited by extreme technical challenges. Hypersonic flight within the atmosphere generates immense drag, and temperatures found on the aircraft and within the engine can be much greater than that of the surrounding air. Maintaining combustion in the supersonic flow presents additional challenges, as the fuel must be injected, mixed, ignited, and burned within milliseconds. Fuel mixing, along with the configuration and positioning of the injectors and the boundary conditions, play a key role in combustion efficiency. [[Gerdroodbary, _Scramjets: Fuel Mixing and Injection Systems_](https://www.sciencedirect.com/book/9780128211380/scramjets)]

Practical challenges to producing and operating scramjets include:

- autoignition of the internal flow ("like lighting a match in a hurricane")
- fuel–oxidizer mixing
- developing heat-dissipative refractory materials
- obtaining experimental data
  - acquiring high-grade internal flow state
  - visualizing phenomena such as shock waves

## Exercises

Complete these using a graphing calculator, Python, MATLAB, or Excel.

1.  Plot $\frac{T}{\dot{m_a}}$ and $\frac{T}{\dot{m_a}a_0}$ v. Mach number $M$ from 1 to 5.
2.  Plot $\text{TSFC}$ v. Mach number $M$ from 1 to 5.
3.  Plot $I_\text{sp}$ v. Mach number $M$ from 1 to 5.
4.  Plot thermal, propulsive, and overall efficiency v. Mach number $M$ from 1 to 5.

given data:

- $Q_R = 48,000 \,\text{kJ}/\text{kg}$ for _n_-decane
- $Q_R = 55,500 \,\text{kJ}/\text{kg}$ for methane
- $Q_R = 141,800 \,\text{kJ}/\text{kg}$ for hydrogen
- maximum combustion $T = 2000 \,\text{K}$

(Based on an exercise by [D R Kirk, "Hypersonic Airbreathing Engines"](https://slideplayer.com/slide/14223195/).)