```python
"""
Please run notebook locally (if you have all the dependencies and a GPU).
Technically you can run this notebook on Google Colab but you need to set up microphone for Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Set up microphone for Colab
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg portaudio19-dev
!pip install unidecode
!pip install pyaudio

# ## Install NeMo
BRANCH = 'v1.0.0'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[asr]

## Install TorchAudio
!pip install torchaudio>=0.6.0 -f https://download.pytorch.org/whl/torch_stable.html

## Grab the config we'll use in this example
!mkdir configs
```

    Requirement already satisfied: wget in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (3.2)

    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    'apt-get' is not recognized as an internal or external command,
    operable program or batch file.

    Requirement already satisfied: unidecode in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (1.1.2)

    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)

    Requirement already satisfied: pyaudio in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (0.2.11)

    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)

    Collecting nemo_toolkit[asr]
      Cloning https://github.com/NVIDIA/NeMo.git (to revision v1.0.0) to c:\users\hapir\appdata\local\temp\pip-install-wvlmdsim\nemo-toolkit_03d2625ee8c249faaeaf8e6a92cf3297
    Requirement already satisfied: numpy>=1.18.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.19.2)
    Requirement already satisfied: onnx>=1.7.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.8.1)
    Requirement already satisfied: pytorch-lightning<=1.2.10,>=1.2.8 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.2.10)
    Requirement already satisfied: python-dateutil in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (2.8.1)
    Requirement already satisfied: torch in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.8.1)
    Requirement already satisfied: wget in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (3.2)
    Requirement already satisfied: wrapt in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.12.1)
    Requirement already satisfied: ruamel.yaml in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.16.12)
    Requirement already satisfied: scikit-learn in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.24.1)
    Requirement already satisfied: omegaconf<2.1.0,>=2.0.5 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (2.0.6)
    Requirement already satisfied: hydra-core>=1.0.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.0.6)
    Requirement already satisfied: transformers>=4.0.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (4.1.1)
    Requirement already satisfied: sentencepiece<1.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.1.95)
    Requirement already satisfied: webdataset>=0.1.48 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.1.62)
    Requirement already satisfied: tqdm>=4.41.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (4.51.0)
    Requirement already satisfied: opencc in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.1.1)
    Requirement already satisfied: pangu in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (4.0.6.1)
    Requirement already satisfied: jieba in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.42.1)
    Requirement already satisfied: numba in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.52.0)
    Requirement already satisfied: braceexpand in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.1.7)
    Requirement already satisfied: editdistance in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.5.3)
    Requirement already satisfied: frozendict in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (2.0.2)
    Requirement already satisfied: inflect in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (5.0.2)
    Requirement already satisfied: kaldi-io in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.9.4)
    Requirement already satisfied: librosa in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.8.0)
    Requirement already satisfied: marshmallow in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (3.12.1)
    Requirement already satisfied: packaging in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (20.9)
    Requirement already satisfied: soundfile in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.10.3.post1)
    Requirement already satisfied: sox in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.4.1)
    Requirement already satisfied: unidecode in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.1.2)
    Requirement already satisfied: kaldi-python-io in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.2.2)
    Requirement already satisfied: kaldiio in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (2.17.2)
    Requirement already satisfied: scipy in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.6.2)
    Requirement already satisfied: pandas in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (1.2.3)
    Requirement already satisfied: g2p_en in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (2.1.0)
    Requirement already satisfied: pydub in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.25.1)
    Requirement already satisfied: pyannote.core in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (4.1)
    Requirement already satisfied: pyannote.metrics in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (3.0.1)
    Requirement already satisfied: pyamg in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (4.1.0)
    Requirement already satisfied: torch-stft in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (0.1.4)
    Requirement already satisfied: matplotlib in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nemo_toolkit[asr]) (3.4.1)
    Requirement already satisfied: antlr4-python3-runtime==4.8 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from hydra-core>=1.0.4->nemo_toolkit[asr]) (4.8)
    Requirement already satisfied: importlib-resources in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from hydra-core>=1.0.4->nemo_toolkit[asr]) (5.1.4)
    Requirement already satisfied: typing-extensions in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from omegaconf<2.1.0,>=2.0.5->nemo_toolkit[asr]) (3.7.4.3)
    Requirement already satisfied: PyYAML>=5.1.* in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from omegaconf<2.1.0,>=2.0.5->nemo_toolkit[asr]) (5.3.1)
    Requirement already satisfied: six in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from onnx>=1.7.0->nemo_toolkit[asr]) (1.15.0)
    Requirement already satisfied: protobuf in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from onnx>=1.7.0->nemo_toolkit[asr]) (3.14.0)
    Requirement already satisfied: fsspec[http]>=0.8.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.8.7)
    Requirement already satisfied: torchmetrics==0.2.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.2.0)
    Requirement already satisfied: tensorboard!=2.5.0,>=2.2.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (2.4.1)
    Requirement already satisfied: future>=0.17.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.18.2)
    Requirement already satisfied: requests in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (2.25.1)
    Requirement already satisfied: aiohttp in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (3.7.4.post0)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.4.2)
    Requirement already satisfied: absl-py>=0.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.11.0)
    Requirement already satisfied: google-auth<2,>=1.6.3 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.28.0)
    Requirement already satisfied: setuptools>=41.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (57.0.0)
    Requirement already satisfied: wheel>=0.26 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.36.2)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.7.0)
    Requirement already satisfied: grpcio>=1.24.3 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.34.1)
    Requirement already satisfied: werkzeug>=0.11.15 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.0.1)
    Requirement already satisfied: markdown>=2.6.8 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (3.3.3)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.2.8)
    Requirement already satisfied: rsa<5,>=3.1.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (4.6)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (4.2.2)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.3.0)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (0.4.8)
    Requirement already satisfied: certifi>=2017.4.17 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from requests->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (2021.5.30)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from requests->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.26.4)
    Requirement already satisfied: idna<3,>=2.5 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from requests->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (2.10)
    Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from requests->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (4.0.0)
    Requirement already satisfied: oauthlib>=3.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard!=2.5.0,>=2.2.0->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (3.1.0)
    Requirement already satisfied: tokenizers==0.9.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from transformers>=4.0.1->nemo_toolkit[asr]) (0.9.4)
    Requirement already satisfied: sacremoses in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from transformers>=4.0.1->nemo_toolkit[asr]) (0.0.43)
    Requirement already satisfied: filelock in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from transformers>=4.0.1->nemo_toolkit[asr]) (3.0.12)
    Requirement already satisfied: regex!=2019.12.17 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from transformers>=4.0.1->nemo_toolkit[asr]) (2020.11.13)
    Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (3.0.1)
    Requirement already satisfied: attrs>=17.3.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (20.3.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (5.1.0)
    Requirement already satisfied: yarl<2.0,>=1.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from aiohttp->fsspec[http]>=0.8.1->pytorch-lightning<=1.2.10,>=1.2.8->nemo_toolkit[asr]) (1.6.3)
    Requirement already satisfied: distance>=0.1.3 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from g2p_en->nemo_toolkit[asr]) (0.1.3)
    Requirement already satisfied: nltk>=3.2.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from g2p_en->nemo_toolkit[asr]) (3.6.2)
    Requirement already satisfied: joblib in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nltk>=3.2.4->g2p_en->nemo_toolkit[asr]) (1.0.1)
    Requirement already satisfied: click in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from nltk>=3.2.4->g2p_en->nemo_toolkit[asr]) (7.1.2)
    Requirement already satisfied: zipp>=3.1.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from importlib-resources->hydra-core>=1.0.4->nemo_toolkit[asr]) (3.4.1)
    Requirement already satisfied: decorator>=3.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from librosa->nemo_toolkit[asr]) (4.4.2)
    Requirement already satisfied: audioread>=2.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from librosa->nemo_toolkit[asr]) (2.1.9)
    Requirement already satisfied: pooch>=1.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from librosa->nemo_toolkit[asr]) (1.3.0)
    Requirement already satisfied: resampy>=0.2.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from librosa->nemo_toolkit[asr]) (0.2.2)
    Requirement already satisfied: llvmlite<0.36,>=0.35.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from numba->nemo_toolkit[asr]) (0.35.0)
    Requirement already satisfied: appdirs in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pooch>=1.0->librosa->nemo_toolkit[asr]) (1.4.4)
    Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from scikit-learn->nemo_toolkit[asr]) (2.1.0)
    Requirement already satisfied: cffi>=1.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from soundfile->nemo_toolkit[asr]) (1.14.5)
    Requirement already satisfied: pycparser in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from cffi>=1.0->soundfile->nemo_toolkit[asr]) (2.20)
    Requirement already satisfied: pillow>=6.2.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from matplotlib->nemo_toolkit[asr]) (8.1.2)
    Requirement already satisfied: pyparsing>=2.2.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from matplotlib->nemo_toolkit[asr]) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from matplotlib->nemo_toolkit[asr]) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from matplotlib->nemo_toolkit[asr]) (1.3.1)
    Requirement already satisfied: pytz>=2017.3 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pandas->nemo_toolkit[asr]) (2021.1)
    Requirement already satisfied: pytest>=2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyamg->nemo_toolkit[asr]) (6.2.4)
    Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (0.13.1)
    Requirement already satisfied: toml in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (0.10.2)
    Requirement already satisfied: colorama in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (0.4.4)
    Requirement already satisfied: atomicwrites>=1.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (1.4.0)
    Requirement already satisfied: iniconfig in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (1.1.1)
    Requirement already satisfied: py>=1.8.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pytest>=2->pyamg->nemo_toolkit[asr]) (1.10.0)
    Requirement already satisfied: sortedcontainers>=2.0.4 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.core->nemo_toolkit[asr]) (2.3.0)
    Requirement already satisfied: simplejson>=3.8.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.core->nemo_toolkit[asr]) (3.17.2)
    Requirement already satisfied: sympy>=1.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.metrics->nemo_toolkit[asr]) (1.7.1)
    Requirement already satisfied: pyannote.database>=4.0.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.metrics->nemo_toolkit[asr]) (4.1)
    Requirement already satisfied: docopt>=0.6.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.metrics->nemo_toolkit[asr]) (0.6.2)
    Requirement already satisfied: tabulate>=0.7.7 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.metrics->nemo_toolkit[asr]) (0.8.9)
    Requirement already satisfied: typer[all]>=0.2.1 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from pyannote.database>=4.0.1->pyannote.metrics->nemo_toolkit[asr]) (0.3.2)
    Requirement already satisfied: mpmath>=0.19 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from sympy>=1.1->pyannote.metrics->nemo_toolkit[asr]) (1.1.0)
    Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from typer[all]>=0.2.1->pyannote.database>=4.0.1->pyannote.metrics->nemo_toolkit[asr]) (1.4.0)
    Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in c:\users\hapir\anaconda3\envs\bigai\lib\site-packages (from ruamel.yaml->nemo_toolkit[asr]) (0.2.2)

    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
      Running command git clone -q https://github.com/NVIDIA/NeMo.git 'C:\Users\hapir\AppData\Local\Temp\pip-install-wvlmdsim\nemo-toolkit_03d2625ee8c249faaeaf8e6a92cf3297'
      Running command git checkout -b v1.0.0 --track origin/v1.0.0
      Branch 'v1.0.0' set up to track remote branch 'v1.0.0' from 'origin'.
      Switched to a new branch 'v1.0.0'
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    WARNING: Ignoring invalid distribution -illow (c:\users\hapir\anaconda3\envs\bigai\lib\site-packages)
    A subdirectory or file configs already exists.

This notebook demonstrates automatic speech recognition (ASR) from a microphone's stream in NeMo.

It is **not a recommended** way to do inference in production workflows. If you are interested in
production-level inference using NeMo ASR models, please sign-up to Jarvis early access program: https://developer.nvidia.com/nvidia-jarvis

The notebook requires PyAudio library to get a signal from an audio device.
For Ubuntu, please run the following commands to install it:
```
sudo apt-get install -y portaudio19-dev
pip install pyaudio
```

This notebook requires the `torchaudio` library to be installed for MatchboxNet. Please follow the instructions available at the [torchaudio Github page](https://github.com/pytorch/audio#installation) to install the appropriate version of torchaudio.

If you would like to install the latest version, please run the following command to install it:

```
conda install -c pytorch torchaudio
```

```python
import numpy as np
import pyaudio as pa
import os, time
```

```python
import nemo
import nemo.collections.asr as nemo_asr
```

    [NeMo W 2021-06-09 18:34:09 optimizers:46] Apex was not found. Using the lamb optimizer will error out.
    [NeMo W 2021-06-09 18:34:14 nemo_logging:349] C:\Users\hapir\anaconda3\envs\BIGAI\lib\site-packages\torchaudio\extension\extension.py:13: UserWarning: torchaudio C++ extension is not available.
          warnings.warn('torchaudio C++ extension is not available.')

    ################################################################################
    ### WARNING, path does not exist: KALDI_ROOT=/mnt/matylda5/iveselyk/Tools/kaldi-trunk
    ###          (please add 'export KALDI_ROOT=<your_path>' in your $HOME/.profile)
    ###          (or run as: KALDI_ROOT=<your_path> python <your_script>.py)
    ################################################################################

    [NeMo W 2021-06-09 18:34:15 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.

```python
# sample rate, Hz
SAMPLE_RATE = 16000
```

## Restore the model from NGC

```python
asr_model = nemo_asr.models.EncDecCTCModel.from_pretrained('stt_es_quartznet15x5')
```

    [NeMo I 2021-06-09 18:34:31 cloud:56] Found existing object C:\Users\hapir\.cache\torch\NeMo\NeMo_1.0.0\stt_es_quartznet15x5\a65f9c865cfd58f57bfba25a7e44e8e2\stt_es_quartznet15x5.nemo.
    [NeMo I 2021-06-09 18:34:31 cloud:62] Re-using file from: C:\Users\hapir\.cache\torch\NeMo\NeMo_1.0.0\stt_es_quartznet15x5\a65f9c865cfd58f57bfba25a7e44e8e2\stt_es_quartznet15x5.nemo
    [NeMo I 2021-06-09 18:34:31 common:675] Instantiating model from pre-trained checkpoint

    [NeMo W 2021-06-09 18:34:32 modelPT:137] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
        Train config :
        manifest_filepath: /raid/noneval.json
        sample_rate: 16000
        labels:
        - ' '
        - a
        - b
        - c
        - d
        - e
        - f
        - g
        - h
        - i
        - j
        - k
        - l
        - m
        - 'n'
        - o
        - p
        - q
        - r
        - s
        - t
        - u
        - v
        - w
        - x
        - 'y'
        - z
        - ''''
        - á
        - é
        - í
        - ó
        - ú
        - ñ
        - ü
        batch_size: 16
        trim_silence: true
        max_duration: 16.7
        shuffle: true
        is_tarred: false
        tarred_audio_filepaths: null
        num_workers: 8
        pin_memory: true

    [NeMo W 2021-06-09 18:34:32 modelPT:144] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
        Validation config :
        manifest_filepath: /raid/dev.json
        sample_rate: 16000
        labels:
        - ' '
        - a
        - b
        - c
        - d
        - e
        - f
        - g
        - h
        - i
        - j
        - k
        - l
        - m
        - 'n'
        - o
        - p
        - q
        - r
        - s
        - t
        - u
        - v
        - w
        - x
        - 'y'
        - z
        - ''''
        - á
        - é
        - í
        - ó
        - ú
        - ñ
        - ü
        batch_size: 32
        shuffle: false
        num_workers: 8
        pin_memory: true

    [NeMo I 2021-06-09 18:34:32 features:252] PADDING: 16
    [NeMo I 2021-06-09 18:34:32 features:269] STFT using torch
    [NeMo I 2021-06-09 18:34:35 modelPT:434] Model EncDecCTCModel was successfully restored from C:\Users\hapir\.cache\torch\NeMo\NeMo_1.0.0\stt_es_quartznet15x5\a65f9c865cfd58f57bfba25a7e44e8e2\stt_es_quartznet15x5.nemo.

## Observing the config of the model

```python
from omegaconf import OmegaConf
import copy
```

```python
# Preserve a copy of the full config
cfg = copy.deepcopy(asr_model._cfg)
print(OmegaConf.to_yaml(cfg))
```

    preprocessor:
      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
      normalize: per_feature
      window_size: 0.02
      sample_rate: 16000
      window_stride: 0.01
      window: hann
      features: 64
      n_fft: 512
      frame_splicing: 1
      dither: 1.0e-05
      stft_conv: false
    spec_augment:
      _target_: nemo.collections.asr.modules.SpectrogramAugmentation
      rect_freq: 50
      rect_masks: 5
      rect_time: 120
    encoder:
      _target_: nemo.collections.asr.modules.ConvASREncoder
      feat_in: 64
      activation: relu
      conv_mask: true
      jasper:
      - filters: 256
        repeat: 1
        kernel:
        - 33
        stride:
        - 2
        dilation:
        - 1
        dropout: 0.0
        residual: false
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 33
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 33
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 33
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 39
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 39
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 256
        repeat: 5
        kernel:
        - 39
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 51
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 51
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 51
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 63
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 63
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 63
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 75
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 75
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 5
        kernel:
        - 75
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: true
        separable: true
      - filters: 512
        repeat: 1
        kernel:
        - 87
        stride:
        - 1
        dilation:
        - 2
        dropout: 0.0
        residual: false
        separable: true
      - filters: 1024
        repeat: 1
        kernel:
        - 1
        stride:
        - 1
        dilation:
        - 1
        dropout: 0.0
        residual: false
    decoder:
      _target_: nemo.collections.asr.modules.ConvASRDecoder
      feat_in: 1024
      num_classes: 35
      vocabulary:
      - ' '
      - a
      - b
      - c
      - d
      - e
      - f
      - g
      - h
      - i
      - j
      - k
      - l
      - m
      - 'n'
      - o
      - p
      - q
      - r
      - s
      - t
      - u
      - v
      - w
      - x
      - 'y'
      - z
      - ''''
      - á
      - é
      - í
      - ó
      - ú
      - ñ
      - ü
    optim:
      name: novograd
      lr: 0.0012
      betas:
      - 0.8
      - 0.5
      weight_decay: 0.001
      sched:
        name: CosineAnnealing
        warmup_steps: 500
        warmup_ratio: null
        min_lr: 0.0
        last_epoch: -1
    target: nemo.collections.asr.models.ctc_models.EncDecCTCModel
    train_ds:
      manifest_filepath: /raid/noneval.json
      sample_rate: 16000
      labels:
      - ' '
      - a
      - b
      - c
      - d
      - e
      - f
      - g
      - h
      - i
      - j
      - k
      - l
      - m
      - 'n'
      - o
      - p
      - q
      - r
      - s
      - t
      - u
      - v
      - w
      - x
      - 'y'
      - z
      - ''''
      - á
      - é
      - í
      - ó
      - ú
      - ñ
      - ü
      batch_size: 16
      trim_silence: true
      max_duration: 16.7
      shuffle: true
      is_tarred: false
      tarred_audio_filepaths: null
      num_workers: 8
      pin_memory: true
    validation_ds:
      manifest_filepath: /raid/dev.json
      sample_rate: 16000
      labels:
      - ' '
      - a
      - b
      - c
      - d
      - e
      - f
      - g
      - h
      - i
      - j
      - k
      - l
      - m
      - 'n'
      - o
      - p
      - q
      - r
      - s
      - t
      - u
      - v
      - w
      - x
      - 'y'
      - z
      - ''''
      - á
      - é
      - í
      - ó
      - ú
      - ñ
      - ü
      batch_size: 32
      shuffle: false
      num_workers: 8
      pin_memory: true

### Modify preprocessor parameters for inference

```python
# Make config overwrite-able
OmegaConf.set_struct(cfg.preprocessor, False)

# some changes for streaming scenario
cfg.preprocessor.dither = 0.0
cfg.preprocessor.pad_to = 0

# spectrogram normalization constants
normalization = {}
normalization['fixed_mean'] = [
     -14.95827016, -12.71798736, -11.76067913, -10.83311182,
     -10.6746914,  -10.15163465, -10.05378331, -9.53918999,
     -9.41858904,  -9.23382904,  -9.46470918,  -9.56037,
     -9.57434245,  -9.47498732,  -9.7635205,   -10.08113074,
     -10.05454561, -9.81112681,  -9.68673603,  -9.83652977,
     -9.90046248,  -9.85404766,  -9.92560366,  -9.95440354,
     -10.17162966, -9.90102482,  -9.47471025,  -9.54416855,
     -10.07109475, -9.98249912,  -9.74359465,  -9.55632283,
     -9.23399915,  -9.36487649,  -9.81791084,  -9.56799225,
     -9.70630899,  -9.85148006,  -9.8594418,   -10.01378735,
     -9.98505315,  -9.62016094,  -10.342285,   -10.41070709,
     -10.10687659, -10.14536695, -10.30828702, -10.23542833,
     -10.88546868, -11.31723646, -11.46087382, -11.54877829,
     -11.62400934, -11.92190509, -12.14063815, -11.65130117,
     -11.58308531, -12.22214663, -12.42927197, -12.58039805,
     -13.10098969, -13.14345864, -13.31835645, -14.47345634]
normalization['fixed_std'] = [
     3.81402054, 4.12647781, 4.05007065, 3.87790987,
     3.74721178, 3.68377423, 3.69344,    3.54001005,
     3.59530412, 3.63752368, 3.62826417, 3.56488469,
     3.53740577, 3.68313898, 3.67138151, 3.55707266,
     3.54919572, 3.55721289, 3.56723346, 3.46029304,
     3.44119672, 3.49030548, 3.39328435, 3.28244406,
     3.28001423, 3.26744937, 3.46692348, 3.35378948,
     2.96330901, 2.97663111, 3.04575148, 2.89717604,
     2.95659301, 2.90181116, 2.7111687,  2.93041291,
     2.86647897, 2.73473181, 2.71495654, 2.75543763,
     2.79174615, 2.96076456, 2.57376336, 2.68789782,
     2.90930817, 2.90412004, 2.76187531, 2.89905006,
     2.65896173, 2.81032176, 2.87769857, 2.84665271,
     2.80863137, 2.80707634, 2.83752184, 3.01914511,
     2.92046439, 2.78461139, 2.90034605, 2.94599508,
     2.99099718, 3.0167554,  3.04649716, 2.94116777]

cfg.preprocessor.normalize = normalization

# Disable config overwriting
OmegaConf.set_struct(cfg.preprocessor, True)
```

## Setup preprocessor with these settings

```python
asr_model.preprocessor = asr_model.from_config_dict(cfg.preprocessor)
```

    [NeMo I 2021-06-09 18:34:46 features:252] PADDING: 0
    [NeMo I 2021-06-09 18:34:46 features:269] STFT using torch

```python
# Set model to inference mode
asr_model.eval();
```

```python
asr_model = asr_model.to(asr_model.device)
```

## Setting up data for Streaming Inference

```python
from nemo.core.classes import IterableDataset
from nemo.core.neural_types import NeuralType, AudioSignal, LengthsType
import torch
from torch.utils.data import DataLoader
```

```python
# simple data layer to pass audio signal
class AudioDataLayer(IterableDataset):
    @property
    def output_types(self):
        return {
            'audio_signal': NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),
            'a_sig_length': NeuralType(tuple('B'), LengthsType()),
        }

    def __init__(self, sample_rate):
        super().__init__()
        self._sample_rate = sample_rate
        self.output = True

    def __iter__(self):
        return self

    def __next__(self):
        if not self.output:
            raise StopIteration
        self.output = False
        return torch.as_tensor(self.signal, dtype=torch.float32), \
               torch.as_tensor(self.signal_shape, dtype=torch.int64)

    def set_signal(self, signal):
        self.signal = signal.astype(np.float32)/32768.
        self.signal_shape = self.signal.size
        self.output = True

    def __len__(self):
        return 1
```

```python
data_layer = AudioDataLayer(sample_rate=cfg.preprocessor.sample_rate)
data_loader = DataLoader(data_layer, batch_size=1, collate_fn=data_layer.collate_fn)
```

```python
# inference method for audio signal (single instance)
def infer_signal(model, signal):
    data_layer.set_signal(signal)
    batch = next(iter(data_loader))
    audio_signal, audio_signal_len = batch
    audio_signal, audio_signal_len = audio_signal.to(asr_model.device), audio_signal_len.to(asr_model.device)
    log_probs, encoded_len, predictions = model.forward(
        input_signal=audio_signal, input_signal_length=audio_signal_len
    )
    return log_probs
```

```python
# class for streaming frame-based ASR
# 1) use reset() method to reset FrameASR's state
# 2) call transcribe(frame) to do ASR on
#    contiguous signal's frames
class FrameASR:

    def __init__(self, model_definition,
                 frame_len=2, frame_overlap=2.5,
                 offset=10):
        '''
        Args:
          frame_len: frame's duration, seconds
          frame_overlap: duration of overlaps before and after current frame, seconds
          offset: number of symbols to drop for smooth streaming
        '''
        self.vocab = list(model_definition['labels'])
        self.vocab.append('_')

        self.sr = model_definition['sample_rate']
        self.frame_len = frame_len
        self.n_frame_len = int(frame_len * self.sr)
        self.frame_overlap = frame_overlap
        self.n_frame_overlap = int(frame_overlap * self.sr)
        timestep_duration = model_definition['AudioToMelSpectrogramPreprocessor']['window_stride']
        for block in model_definition['JasperEncoder']['jasper']:
            timestep_duration *= block['stride'][0] ** block['repeat']
        self.n_timesteps_overlap = int(frame_overlap / timestep_duration) - 2
        self.buffer = np.zeros(shape=2*self.n_frame_overlap + self.n_frame_len,
                               dtype=np.float32)
        self.offset = offset
        self.reset()

    def _decode(self, frame, offset=0):
        assert len(frame)==self.n_frame_len
        self.buffer[:-self.n_frame_len] = self.buffer[self.n_frame_len:]
        self.buffer[-self.n_frame_len:] = frame
        logits = infer_signal(asr_model, self.buffer).cpu().numpy()[0]
        # print(logits.shape)
        decoded = self._greedy_decoder(
            logits[self.n_timesteps_overlap:-self.n_timesteps_overlap],
            self.vocab
        )
        return decoded[:len(decoded)-offset]

    @torch.no_grad()
    def transcribe(self, frame=None, merge=True):
        if frame is None:
            frame = np.zeros(shape=self.n_frame_len, dtype=np.float32)
        if len(frame) < self.n_frame_len:
            frame = np.pad(frame, [0, self.n_frame_len - len(frame)], 'constant')
        unmerged = self._decode(frame, self.offset)
        if not merge:
            return unmerged
        return self.greedy_merge(unmerged)

    def reset(self):
        '''
        Reset frame_history and decoder's state
        '''
        self.buffer=np.zeros(shape=self.buffer.shape, dtype=np.float32)
        self.prev_char = ''

    @staticmethod
    def _greedy_decoder(logits, vocab):
        s = ''
        for i in range(logits.shape[0]):
            s += vocab[np.argmax(logits[i])]
        return s

    def greedy_merge(self, s):
        s_merged = ''

        for i in range(len(s)):
            if s[i] != self.prev_char:
                self.prev_char = s[i]
                if self.prev_char != '_':
                    s_merged += self.prev_char
        return s_merged
```

# Streaming Inference

Streaming inference depends on a few factors, such as the frame length and buffer size. Experiment with a few values to see their effects in the below cells.

```python
# duration of signal frame, seconds
FRAME_LEN = 1.0
# number of audio channels (expect mono signal)
CHANNELS = 1

CHUNK_SIZE = int(FRAME_LEN*SAMPLE_RATE)
asr = FrameASR(model_definition = {
                   'sample_rate': SAMPLE_RATE,
                   'AudioToMelSpectrogramPreprocessor': cfg.preprocessor,
                   'JasperEncoder': cfg.encoder,
                   'labels': cfg.decoder.vocabulary
               },
               frame_len=FRAME_LEN, frame_overlap=2,
               offset=4)
```

```python
asr.reset()

p = pa.PyAudio()
print('Available audio input devices:')
input_devices = []
for i in range(p.get_device_count()):
    dev = p.get_device_info_by_index(i)
    if dev.get('maxInputChannels'):
        input_devices.append(i)
        print(i, dev.get('name'))

if len(input_devices):
    dev_idx = -2
    while dev_idx not in input_devices:
        print('Please type input device ID:')
        dev_idx = int(input())

    empty_counter = 0

    def callback(in_data, frame_count, time_info, status):
        global empty_counter
        signal = np.frombuffer(in_data, dtype=np.int16)
        text = asr.transcribe(signal)
        if len(text):
            print(text,end='')
            empty_counter = asr.offset
        elif empty_counter > 0:
            empty_counter -= 1
            if empty_counter == 0:
                print(' ',end='')
        return (in_data, pa.paContinue)

    stream = p.open(format=pa.paInt16,
                    channels=CHANNELS,
                    rate=SAMPLE_RATE,
                    input=True,
                    input_device_index=dev_idx,
                    stream_callback=callback,
                    frames_per_buffer=CHUNK_SIZE)

    print('Listening...')

    stream.start_stream()

    # Interrupt kernel and then speak for a few more words to exit the pyaudio loop !
    try:
        while stream.is_active():
            time.sleep(0.1)
    finally:
        stream.stop_stream()
        stream.close()
        p.terminate()

        print()
        print("PyAudio stopped")

else:
    print('ERROR: No audio input device found.')
```

```python

```

```python

```


# On the viability of crowdsourcing NLP annotations in healthcare

```python
__author__  = 'Bruno Godefroy'
```

__TL;DR__ For most annotation tasks in healthcare, we must rely on trained experts, who are always in high-demand, creating a bottleneck for system development. Can we address this limitation by crowdsourcing the easier annotation tasks in this space? We report on an experiment addressing this question, with a focus on how to identify problematic workers and infer reliable annotations from noisy crowd work.

## Overview

Obtaining high-quality annotations is a bottleneck for all natural language processing applications. The challenges are especially severe in healthcare, where we rely on annotators who have expertise in the practice of medicine and in understanding medical texts, and who are authorized to access sensitive data. Such annotators are in high-demand and thus demand high prices. The question then arises whether we can ease the burden on these annotators by crowdsourcing the less demanding annotation tasks involving publicly-available data, reserving the experts for where they are truly needed.

This post reports on a crowdsourcing experiment we ran to explore this issue. We defined a reasonably nuanced span-identification task, and launched it on [Figure Eight](https://www.figure-eight.com). As expected, the output was noisy, as a result of the highly variable pool of annotators we tapped into. To infer quality labels from this noisy process, we used a straightforward application of Expectation-Maximization, with quite good results, suggesting that crowdsourcing is an effective tool for obtaining annotations for at least some NLP problems in healthcare.

## Crowdsourcing task definition

The publicly available [FDA Drug Labels data set](https://labels.fda.gov) is a rich source of information about how drugs relate to disease states (among other things). Since it's a public dataset, we don't have to address privacy concerns, which are of course another limiting factor when it comes to annotating data in healthcare.

For our pilot task, we decided to focus on developing annotations to facilitate automatic extraction of the core drug–disease relationships expressed in these labels, as exemplified in figure 1. This is [a problem we've worked on before](https://roamanalytics.com/2018/02/02/retrofitting-distributional-embeddings-to-knowledge-graphs-with-functional-relations/), so we have a good sense for what information should be extracted.

Figure 1: Annotated sentences from a drug label.

We need our NLP model to identify the disease mentions in these texts and determine the sense for those mentions. Our own high-level exploration of the data led us to the label set _prevents_, _treats_, and _is contraindicated for_ as the right set of high-level categories for the mentions. This annotation design is motivated by recent experiments which gave encouraging results on similar tasks [1, 2].

For crowdsourcing, we break this down into two separate tasks: identifying disease mentions spans, and assigning categories to identified spans. (Breaking the task down in this way isn't strictly speaking necessary, but it eases the burden on individual annotators). Since the span identification task is more open-ended and challenging, it's the one we explore in this post.

We developed a short annotation manual with examples, had an in-house expert label 50 sentences for quality control, and used the Figure Eight platform to design a simple annotation interface (figure 2).

Figure 2: The interface for our first task. Crowdworkers are asked to select disease, symptoms and injury mentions in a sentence. Some drug and disease mentions are automatically underlined using fixed lexicons to help the workers understand the texts.

## Who completed our task on Figure Eight?

We launched our task on Figure Eight with 10,000 sentences. It was completed within a few days. The job was done by 451 people from 45 countries, the majority from India and Venezuela. No special qualifications were imposed.

Most workers made just a few contributions in a short period of time, as figures 3 and 4 show. Half of the sessions (continuous periods of work) lasted less than 20 minutes; at the upper end, 7% were more than an hour and a half. This is expected given recent studies of crowdworkers' behaviors [3, 4].

Figure 3: Number of work sessions per contributor.

Figure 4: Work session durations.

## Assessment against gold labels

Given our large and diverse pool of workers, we expect some of them to be unreliable, perhaps due to a lack of expertise or a lack of attention. With Figure Eight, we can supply our own labeled examples for a subset of cases, to help identify and filter out unreliable workers. Figure 5 summarizes the work of 100 annotators who were rejected from our task based on their performance on this gold data.

Figure 5: Main reason for failing out of our gold-label assessment. Results come from the manual study of the output from 100 workers who didn't pass the test.

It seems that several workers attempted cheating strategies, such as selecting no text segments at all. Our gold labels help us identify such work. We also see evidence that some workers used auxiliary tools to translate the full Figure Eight interface into another language, including the sentences to annotate, with the result that the responses they submitted were in that language!

## From noisy judgments to crowd truth

Should we blindly trust all the judgments of all the workers who passed our gold-label assessment? Probably not. Some errors are inevitable even for careful workers, and some malicious workers are likely to slip past our assessment against our gold examples. Furthermore, there are bound to be cases that are ambiguous or open to interpretation, leading to multiple right answers that we ourselves might not have fully appreciated, as in [figure 6](#fig6).

Figure 6: An ambiguous case. Should "feelings of sadness related to winter" be selected as a disease? Or a symptom?

The most basic step we can take to address these concerns is to collect multiple judgments from many different annotators, in the hope that a consensus emerges. We could define consensus in various ways – e.g., the majority label, or at least 70% agreement on a label (leaving some cases without a label at all). However, these approaches are clearly suboptimal because they make the implicit assumption that all workers are equally trustworthy, by giving equal weight to all of them. To address this, we use a simple application of the [Expectation-Maximization (EM) method](https://en.wikipedia.org/wiki/Expectation–maximization_algorithm) to our matrix of worker judgments.

Our guiding idea is that "*disagreement is not noise but signal*" [[5](#ref_aroyo)]. Assuming the majority is usually right, it's probably worth giving more credit to workers who often agree with each other. Following this intuition, Dawid and Skene [[6](#ref_dawidSkene)] propose a simple model which uses EM to estimate response confidence and workers' reliability in an efficient way. [Figure 7](#fig7) provides an overview of this algorithm, which we state in full in an [appendix](#appendix).

Figure 7: The Expectation-Maximization algorithm for inferring labels from crowd work.

Let's consider a small example where we want to find the responses to five boolean questions using a crowd of four workers. For each question, we collect three True/False (T/F) judgments.

        Worker 1
        Worker 2
        Worker 3
        Worker 4
        True response

        Question 1
        T
        T
        T

        T

        Question 2

        F
        F
        T
        F

        Question 3
        T
        T
        F

        T

        Question 4
        F

        F
        T
        F

        Question 5
        F
        T

        T
        F

Table 1: Dummy collected judgments and true response (that we don't have), for each question.

Worker 1 looks highly reliable: where it contributed annotations, it always agrees with our true response. Workers 2 and 4 have low *precision* in the sense that they sometimes say T when the correct label is F. Conversely, Worker 3 has low *recall* in the sense that he sometimes says F when the correct label is T.

Of course, we crowdsource precisely because we don't have these true responses – we have to try to uncover them given only the matrix of worker judgments. With EM, we do this by jointly estimating the reliability of each worker and the response that maximizes the likelihood of the observed judgments.

[Table 2](#tab2) gives the output of the algorithm for this simple example. The *crowd response* is the one derived from the EM algorithm; T when the maximum likelihood estimate is larger than 0.5, F otherwise. Our implementation ([available on Github](https://github.com/roaminsight/analysis_notebooks/blob/crowdsourcing-blogpost/Blog_posts/develop/2018-07-02-crowdsourcing_annotation/EM_example.py)) converges after 8 iterations.

        Maximum likelihood estimate
        Derived crowd response
        True response

        Question 1
        0.99
        T
        T

        Question 2
        0.01
        F
        F

        Question 3
        0.98
        T
        T

        Question 4
        0.08
        F
        F

        Question 5
        0.28
        F
        F

Table 2: Estimated maximum likelihood responses.

The algorithm does well here and shows how weighting contributions, using inter-worker agreement, can be valuable for judgment aggregation. All the responses derived from the crowd are correct according to our gold labels. For the first four questions, the model has high confidence in its estimate. For question 5, though the majority voted T, the model correctly infers F as the label because worker 1's judgments are given more weight than the others' judgments.

The individual workers' performance estimates, [table 3](#tab3), look good too. Despite the small amount of data, the algorithm successfully figured out that worker 1 was very reliable, workers 2 and 4 had low precision and worker 3 had low recall.

        Estimated precision
        True precision
        Estimated recall
        True recall

        Worker 1
        0.99
        1.0
        0.84
        1.0

        Worker 2
        0.75
        0.67
        0.99
        1.0

        Worker 3
        0.99
        1.0
        0.48
        0.5

        Worker 4
        0.12
        0.0
        0.99
        /

Table 3: Workers' performance estimates.

Several extensions of this and other methods [7, 8, 9, 10] have been proposed for this problem. Zhang et al. [10], for example, address the issue of getting trapped in a local optimum with EM by computing initial parameter estimates with a spectral method, rather than initializing them randomly. Other approaches focus on modeling micro-task difficulty along with worker reliability [8].

## The wisdom of our crowd

There is evidence that, in many settings, a crowd of non-experts can collectively offer estimates that match or exceed those of individual experts [11, 12, 13, 14]. Is this true of our crowd of Figure Eight workers?

To address this question, we applied the EM algorithm as described above. Since many of our disease spans are multi-word expressions, we do this at the token level: for each token selected by at least one worker, its probability to be a disease mention is estimated.

To begin, we hope that most tokens end up with a probability close to 0 or 1, that is, a high confidence of being part of an entity of interest or not. [Figure 8](#fig8) shows that this is the case.

Figure 8: The distribution of probabilities estimated with EM, after 1, 2 and 10 iterations (convergence).

Next, we randomly selected 1,000 tokens from our task and presented them (in their linguistic context) to an in-house expert for labeling. We then compared these expert-provided labels with the ones inferred from the crowd work. [Figure 9](#fig9) summarizes this evaluation, which uses the standard classification metrics of [precision, recall, and F1](https://en.wikipedia.org/wiki/Precision_and_recall).

Figure 9: Precision, recall and F1 scores at various confidence thresholds for each aggregation method.

We see that the crowd does well. At a threshold of 0.5, precision and recall are above 0.8, which is comparable to results claimed by MacLean et al. in a similar entity extraction task [1].

In this experiment, it appears that most workers have high precision, but recall is very diverse (see [our last section](#Facing-the-crowd)). Therefore, weighting contributions with EM does not really impact the precision of the crowd; majority voting (orange curve) and EM (blue curve) give similar results. For recall, however, since there is high variance among contributors, giving more weight to high-recall workers significantly improves the crowd response.

Assuming a majority of workers are mostly "noise", one could also consider, for each sentence, only the judgment from the most reliable worker according to EM (green curve). In this experiment, this still performs better than majority voting but does not beat the weighted aggregation of all available judgments (blue curve).

## Facing the crowd

EM estimates individual worker reliability and can therefore help us understand individual behaviors in the crowd. To that end, [figure 10](#fig10) shows the timeline of each work session (a continuous period of work for one contributor).

Figure 10: Precision and recall against time during work sessions. Clusters are represented with distinct colors.

As we noted above, most workers have high precision, while recall is very diverse. That is, quality issues mostly come from missing relevant mentions rather than selecting some wrong ones. This is evident in the figure: most of the timelines are near the top of the plot, but they are spread out over the recall axis.

One could think that aggregation could easily fix this bad recall: by combining multiple high-precision, low-recall judgments, we would end up with high-precision, high-recall annotations. We see however that there is a high correlation between workers' behavior; they tend to miss the same text segments. In our task, this has a likely explanation: to maximize profit, workers might choose to increase their judgment rate at the cost of reducing work quality. By reading sentences superficially, they would identify the most obvious relevant mentions while missing more demanding ones. This strategy could go unnoticed during judgment collection because our gold-label assessment deliberately avoids difficult cases and ambiguity.

Using dynamic time warping (DTW) to measure similarity between these time series (see [this blog post for discussion of DTW](https://roamanalytics.com/2016/11/28/shape-matching-with-time-series-data/)), we identified 4 distinct clusters of work sessions. These correspond to the different colors in [figure 10](#fig10):

* Light blue (1,050 timelines): The largest cluster, consisting mostly of short sessions.

* Yellow (6 timelines): Very low precision and very low recall. These workers unfortunately slipped through our gold-label assessment.

* Orange (15 timelines): High precision and very low recall, probably from workers who selected almost no text segments.

* Dark blue (31 timelines): Long sessions consisting of reliable annotations.

This analysis highlights the diversity of the crowd and strengthens our intuition that weighting contributions with individual reliability, estimated using inter-worker agreement, is valuable. A judgment from a dark blue session in [figure 10](#fig10) should be given way more credit than one from the orange cluster.

Although, these findings also reveal that a large group of workers with similar biases might lead to misleading results with EM. For example, a large group of consistently wrong workers would appear to be reliable and might even drown out the work of a smaller, more reliable group. Any unsupervised method will have weaknesses like this, which highlights the importance of having at least some expert annotators reviewing the work.

## Looking ahead

The above findings are reassuring – we might in fact be able to relieve some of the annotation burden from trained experts by using crowdsourcing. It remains to be seen where the boundary lies between work the crowd can do and work that requires experts, especially as we find ways to break down very challenging tasks into simpler component parts that crowdworkers can succeed at.

## Appendix: A closer look to Expectation-Maximization

Let's have a closer look to the EM algorithm. We consider an experiment where we want to find the response $r_i$ to questions $i$, using some noisy judgments $J_i$ collected from workers $k$. For each question, we collect boolean judgments $j_{ik}$ (T/F, or None if the worker did not respond to question $i$). Workers reliability is modeled with $\Theta = \theta_1, \ldots, \theta_k$.

We assume judgments are independent between questions for a given worker (no gain of experience) and between workers for a given question (no inter-worker communication).

We describe below the main steps of the algorithm (see [figure 7](#fig7) above).

### Initialization
Since we have no prior knowledge of workers' trustworthiness, their reliability parameters $\Theta$ are initialized to reasonable values based on what we might expect from workers in pools like ours.

### Expectation
During the *expectation* step, workers' performance is re-estimated, given the output from the *maximization* step. Here, we will model workers quality with the true positive, false positive, true negative and false negative rates.

### Maximization
In this step, we calculate, for each question $i$, the maximum likelihood estimate that the true response $r_i$ is positive, given the collected judgments $J_i$: $p(r_i = \text{T} \mid J_i, \Theta)$.

Using Bayes' theorem:

$$
\begin{align}
    p(r_i = \text{T} \mid J_i, \Theta) &= \frac{p(J_i \mid r_i = \text{T}, \Theta) p(r_i = \text{T})}{p(J_i \mid \Theta)}
\end{align}
$$

We know that

$$
\begin{align}
    p(J_i \mid \Theta) =
    p(J_i \mid r_i=\text{T}, \Theta)p(r_i=\text{T}) +
    p(J_i \mid r_i=\text{F}, \Theta)p(r_i=\text{F})
\end{align}
$$

By substitution, assuming the T and F response outcomes are equiprobable, i.e. $p(r_i = \text{T}) = p(r_i = \text{F})$, we obtain

$$
\begin{align}
p(r_i= \text{T} \mid J_i, \Theta) &=
\frac{
  p(J_i \mid, r_i=\text{T}, \Theta)
}{
 p(J_i \mid r_i = \text{T}, \Theta) +
 p(J_i \mid r_i = \text{F}, \Theta)
} \\
\end{align}
$$

Because we assume worker independence, the collected judgment probability for a given question equals the product of individual judgment probabilities:

$$
p(J_i \mid r_i, \Theta) = \prod_{k} p(j_{ik} \mid r_i, \theta_k)
$$

All in all,

$$
\begin{align}
    p(r_i = \text{T} \mid J_i, \Theta) =
    \prod_{k}
    \frac{
      p(j_k \mid r_i = \text{T}, \theta_k)
    }{
      p(j_k \mid r_i = \text{T}, \theta_k) +
      p(j_k \mid r_i = \text{F}, \theta_k)
    }
\end{align}
$$

This probability could be computed from worker's reliability parameters $\Theta$ (estimated in the expectation step). Indeed, $p(j_k \mid r_i = \text{T}, \theta_k)$ is the true positive rate if $j_k = \text{T}$ and the false negative rate if $j_k = \text{F}$. Similarly, for $p(j_k \mid r_i = \text{F}, \theta_k)$, we use the false positive rate if $j_k = \text{T}$ and the true negative rate if $j_k = \text{F}$.

Since it maximizes a non-convex log-likelihood function, this approach has no theoretical guarantee of performance, though empirical studies show that it usually converges to good estimates.

## References

[1] D. MacLean, J. Heer. 2013. Identifying medical terms in patient-authored text: A crowdsourcing-based approach. *Journal of the American Medical Informatics Association (JAMIA)*.

[2] J. Tenuto. 2015. [How scientists are using CrowdFlower to create a massive biomedical database](https://www.figure-eight.com/citizen-science/). Figure Eight's Artificial Intelligence Resource Center.

[3] H. Zhai, T. Lingren, L. Deleger, Q. Li, M. Kaiser, L Stoutenborough, I Solti. 2013. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. *Journal Medical Internet Research*.

[4] S. Amer-Yahia, S. Basu Roy. 2016. Toward worker-centric crowdsourcing. *IEEE Data Engineering Bulletin*.

[5] L. Aroyo, C. Welty. 2013. Measuring crowd truth for medical relation extraction. In AAAI Fall Symposium on Semantics for Big Data.

[6] A. P. Dawid, A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C.

[7] V. Raykar et al. 2009. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. *International Conference on Machine Learning (ICML)*.

[8] J. Whitehill, T. Wu, J. Bergsma, J.R. Movellan, P.L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. *Neural Information Processing Systems (NIPS)*.

[9] D.R. Karger, S. Oh, D. Shah. 2011. Iterative learning for reliable crowdsourcing systems. *Neural Information Processing Systems (NIPS)*

[10] Y. Zhang, X. Chen, D. Zhou, M. I. Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. *Neural Information Processing Systems (NIPS)*

[11] J. Surowiecki. 2005. *The Wisdom of Crowds*. Anchor Books.

[12] W. Antweiler, M. Z. Frank. 2004. Is all that talk just noise? The information content of Internet stock message boards. *The Journal of Finance*.

[13] H. Chen, P. De, Y. Hu, B.-H. Hwang. 2014. Wisdom of crowds: The value of stock opinions transmitted through social media. *Review of Financial Studies*.

[14] M. Nofer, O. Hinz. 2014. Are crowds on the Internet wiser than experts? The case of a stock prediction community. *Journal of Business Economics*.


# Homework 3
### Due Date:  Tuesday, September 20th at 11:59 PM

---
## Problem 1:  `Git` and recovering from a mistake
You will do this problem in the Jupyter notebook so I can see your output.  Once again, you will work with your playground repository.

**NOTE:** At the beginning of each cell, you MUST type `%%bash`.  If you don't do that then you will not be able to work with the necessary bash commands.

Follow the following steps for this problem:

1. First cell:
  1. Type `cd /tmp` to enter the temporary directory
  2. git clone url_to_your_playground_repo
2. Second cell:
  1. Go into your local `playground` directory (`cd /tmp/playground`)
  2. Type `git pull origin mybranch1`
  3. `ls`
3. Third cell:
  1. Go into your local `playground` directory (`cd /tmp/playground`)
  2. Type `git status`
4. Fourth cell:
  1. Go into your local `playground` directory (`cd /tmp/playground`)
  2. Type `git reset --hard origin/master`
  3. `ls`
5. Fifth cell:
  1. Go into your local `playground` directory (`cd /tmp/playground`)
  2. Type `git status`

The whole point of this problem was to show you how to get your local repo back to an earlier state.  In this exercise, you accidentally merged something to `master` that you didn't want.  Rather than starting to delete things all over the place, you can simply reset your `HEAD` to a previous commit.

```bash
%%bash
cd /tmp
rm -rf playground
git clone https://github.com/xuwd11/playground
```

    Cloning into 'playground'...

```bash
%%bash
cd /tmp/playground
git pull origin mybranch1
ls
```

    Already up-to-date.
    books.md
    feature.txt
    intro.md
    world.md

    From https://github.com/xuwd11/playground
     * branch            mybranch1  -> FETCH_HEAD

```bash
%%bash
cd /tmp/playground
git status
```

    On branch master
    Your branch is up-to-date with 'origin/master'.

    nothing to commit, working tree clean

```bash
%%bash
cd /tmp/playground
git reset --hard origin/master
ls
```

    HEAD is now at 9621d1b Merge branch 'master' of https://github.com/xuwd11/playground
    books.md
    feature.txt
    intro.md
    world.md

```bash
%%bash
cd /tmp/playground
git status
```

    On branch master
    Your branch is up-to-date with 'origin/master'.

    nothing to commit, working tree clean

## Problem 2:  `Git` and checking out a single file
Sometimes you don't want to merge an entire branch from the upstream but just one file from it. There is a direct use case for such a situation. Suppose I've made an error in this homework (or a lecture) and want to correct it. I fix the mistake in the upstream repo. In the meantime you have edited some other files and you really don't want to manually ignore my older copies of those files. Rather, you want to fix just one file from this new branch. This is how you do it.

As usual, be sure to type in `%%bash` before you write any `bash` commands in a cell.

**Note:** The steps below assume that you have already cloned the `playground` repo in this notebook.

1. First cell:
  1. Go into the `playground` repo and fetch the changes from the master branch of the `course` remote.
2. Second cell:
  1. `git checkout course/master -- README.md`.  The `--` means that `README.md` is a file (as opposed to a `branch`).
  2. `cat README.md`.  This just looks at the updated file.
3. Third cell:
  1. `git status`
  2. Commit the changes to your local repo with an appropriate commit message.
  3. `git status`
  4. Push the changes to your remote repo.

```bash
%%bash
cd /tmp/playground
git remote add course https://github.com/dsondak/playground.git
git fetch course master
```

    From https://github.com/dsondak/playground
     * branch            master     -> FETCH_HEAD
     * [new branch]      master     -> course/master

```bash
%%bash
cd /tmp/playground
git checkout course/master -- README.md
cat README.md
```

    # Playground Repo

    For practicing.

```bash
%%bash
cd /tmp/playground
git status
git commit -m "Updated README.md"
git status
git push
```

    On branch master
    Your branch is up-to-date with 'origin/master'.

    Changes to be committed:
      (use "git reset HEAD <file>..." to unstage)

    	new file:   README.md

    [master 468e3dc] Updated README.md
     1 file changed, 3 insertions(+)
     create mode 100644 README.md
    On branch master
    Your branch is ahead of 'origin/master' by 1 commit.
      (use "git push" to publish your local commits)

    nothing to commit, working tree clean

    To https://github.com/xuwd11/playground
       9621d1b..468e3dc  master -> master

## Problem 3
This problem is related to the Lecture 4 exercises.
1. Open the `languages.txt` file.  This file contains all the languages that students listed as their primary language in the course survey.
2. Load the language strings from the file into a list.
3. Use the `Counter` method from the `collections` library to count the number of occurrences of each element of the list.
  + **NOTE:**  It is not necessary to use the `most_common()` method here.
4. Create a bar plot to display the frequency of each language.  Be sure to label the x-axis!
  + Remember, to create plots in the notebook you must put the line `%matplotlib inline` at the beginning of your notebook.
  + Be sure to import matplotlib:  `import matplotlib.pyplot as plt`.
  + To generate the bar plot write `plt.bar(x_coords, freqs)`.  You need to define `x_coords` and `freqs`.
  + **Hint:** You may want to use the `numpy` `arange` function to create `x_coords`.  Remember, `x_coords` is the x-axis and it should have points for each distinct language.
  + **Hint:** To get `freqs`, you may want to use the `values()` method on your result from step 3.  That is, `freqs = result_from_3.values()`.
  + **Hint:** To label the x-axis you should use `plt.xticks(x_coords, labels)` where labels can be accessed through the `keys()` method on your result from step 3.

```python
%matplotlib inline
from collections import Counter
import matplotlib.pyplot as plt
import numpy as np
```

```python
f = open('languages.txt', 'r')
languages = f.read().strip().split()
f.close()
languages_count = Counter(languages)
languages_count_sorted = sorted(languages_count.items(), key=lambda x:x[1], reverse=True)
```

```python
x_coords = range(len(languages_count_sorted))
labels = [x[0] for x in languages_count_sorted]
freqs = [x[1] for x in languages_count_sorted]
plt.bar(x_coords, freqs);
plt.xticks(x_coords, labels);
plt.xlabel('Language');
plt.ylabel('Frequency');
plt.show()
```

---
## Problem 4
In chemical kinetics, the reaction rate coefficient for a given reaction depends on the temperature of the system.  The functional relationship between the reaction rate coefficient and temperature is given by the Arrhenius rate:
\begin{align}
k\left(T\right) = A T^{b}\exp\left(-\frac{E}{RT}\right)
\end{align}
where $A$, $b$, and $E$ are parameters, $R = 8.314 \dfrac{\textrm{J}}{\textrm{mol} \textrm{ K}}$  is the universal gas constant, and $T$ is the temperature.

Write a function which returns $k\left(T\right)$ given $A$, $b$, $E$, and $T$.  Here are a few requirements:
* The function should test for exceptions where necessary.
* Pass the parameters $A$, $b$, and $E$ in as a list.
* Make $R$ a keyword argument to the function.

```python
def Arrhenius_rate(T, parameters, R=8.314):

    # Check temperature
    # T can be either a numpy array or a number
    try:
        try:
            if all(T > 0):
                pass
            else:
                raise ValueError
        except:
            if T <= 0:
                raise ValueError
    except:
        print('Invalid temperature! Please check and set a valid temperature (in K)!')
        return None

    # Check parameters
    try:
        if 3 != len(parameters):
            raise ValueError
        parameters = [float(p) for p in parameters]
    except:
        print('Invalid parameters! Please pass the parameters A, b, E in as a list and make sure they are valid numbers!')
        return None

    # Check the universal gas constant R
    try:
        R = float(R)
        if R <= 0:
            raise ValueError
    except:
        print('Invalid universal gas constant R! Please set a valid R (in J/mol/K)!')
        return None

    return parameters[0] * T**parameters[1] * np.exp(-parameters[2]/(R*T))

```

```python
# Test 1 (invalid temperature)
Arrhenius_rate(-10, [100,4,1])
Arrhenius_rate([10], [100,4,1])
Arrhenius_rate('T', [100,4,1])
```

    Invalid temperature! Please check and set a valid temperature (in K)!
    Invalid temperature! Please check and set a valid temperature (in K)!
    Invalid temperature! Please check and set a valid temperature (in K)!

```python
# Test 2 (invalid parameters)
Arrhenius_rate(273, [100,4,1,10])
Arrhenius_rate(273, 100)
```

    Invalid parameters! Please pass the parameters A, b, E in as a list and make sure they are valid numbers!
    Invalid parameters! Please pass the parameters A, b, E in as a list and make sure they are valid numbers!

```python
# Test 3 (invalid R)
Arrhenius_rate(273, [100,4,1], -10)
Arrhenius_rate(273, [100,4,1], 'R')
```

    Invalid universal gas constant R! Please set a valid R (in J/mol/K)!
    Invalid universal gas constant R! Please set a valid R (in J/mol/K)!

```python
# Test 4 (valid input)
Arrhenius_rate(273, [100,2,50000])
```

    0.0020192769398056079

---
## Problem 5
Using `numpy` arrays, plot $k\left(T\right)$ for $T\in\left(0, 5000\right]$ for three different sets of parameters $\left\{A, b, E\right\}$.  Make sure all three lines are on the same figure and be sure to label each line.  You may use the function from Problem 2.  You may want to play with the parameters a little bit to get some nice curves but you won't lose points for ugly curves either (as long as they're correct!).

```python
para1 = [100, 2, 40000]
para2 = [100, 2, 55000]
para3 = [100, 2, 80000]
T = np.arange(1, 5001)
k1 = Arrhenius_rate(T, para1)
k2 = Arrhenius_rate(T, para2)
k3 = Arrhenius_rate(T, para3)
plt.plot(T, k1, label='A = {}, b = {}, E = {}'.format(para1[0], para1[1], para1[2]));
plt.plot(T, k2, label='A = {}, b = {}, E = {}'.format(para2[0], para2[1], para2[2]));
plt.plot(T, k3, label='A = {}, b = {}, E = {}'.format(para3[0], para3[1], para3[2]));
plt.legend();
plt.xlabel('Temperature (K)');
plt.ylabel('Reaction rate (mol/L/s)');
```