```python
from IPython.display import Image
```

# CNTK 203: Reinforcement Learning Basics

Reinforcement learning (RL) is an area of machine learning inspired by behaviorist psychology, concerned with how [software agents](https://en.wikipedia.org/wiki/Software_agent) ought to take [actions](https://en.wikipedia.org/wiki/Action_selection) in an environment so as to maximize some notion of cumulative reward. In machine learning, the environment is typically formulated as a [Markov decision process](https://en.wikipedia.org/wiki/Markov_decision_process) (MDP) as many reinforcement learning algorithms for this context utilize [dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming) techniques.

In some machine learning settings, we do not have immediate access to labels, so we cannot rely on supervised learning techniques. If, however, there is something we can interact with and thereby get some feedback that tells us occasionally, whether our previous behavior was good or not, we can use RL to learn how to improve our behavior.

Unlike in supervised learning, in RL, labeled correct input/output pairs are never presented and sub-optimal actions are never explicitly corrected. This mimics many of the online learning paradigms which involves finding a balance between exploration (of conditions or actions never learnt before) and exploitation (of already learnt conditions or actions from previous encounters). Multi-arm bandit problems is one of the category of RL algorithms where exploration vs. exploitation trade-off have been thoroughly studied. See figure below for [reference](http://www.simongrant.org/pubs/thesis/3/2.html).

```python
# Figure 1
Image(url="https://cntk.ai/jup/polecart.gif", width=300, height=300)
```

**Problem**

We will use the [CartPole](https://gym.openai.com/envs/CartPole-v0) environment from OpenAI's [gym](https://github.com/openai/gym) simulator to teach a cart to balance a pole. As described in the link above, in the CartPole example, a pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center. See figure below for reference.

**Goal**
Our goal is to prevent the pole from falling over as the cart moves with the pole in upright position (perpendicular to the cart) as the starting state. More specifically if the pole is less than 15 degrees from vertical while the cart is within 2.4 units of the center we will collect reward. In this tutorial, we will train till we learn a set of actions (policies) that lead to an average reward of 200 or more over last 50 batches.

In, RL terminology, the goal is to find _policies_ $a$, that maximize the _reward_ $r$ (feedback) through interaction with some environment (in this case the pole being balanced on the cart). So given a series of experiences $$s \xrightarrow{a} r, s'$$ we then can learn how to choose action $a$ in a given state $s$ to maximize the accumulated reward $r$ over time:
\begin{align}
Q(s,a) &= r_0 + \gamma r_1 + \gamma^2 r_2 + \ldots \newline
&= r_0 + \gamma \max_a Q^*(s',a)
\end{align}
where $\gamma \in [0,1)$ is the discount factor that controls how much we should value reward that is further away. This is called the [*Bellmann*-equation](https://en.wikipedia.org/wiki/Bellman_equation).

In this tutorial we will show how to model the state space, how to use the received reward to figure out which action yields the highest future reward.

We present two different popular approaches here:

**Deep Q-Networks**: DQNs have become famous in 2015 when they were successfully used to train how to play Atari just form raw pixels. We train neural network to learn the $Q(s,a)$ values (thus _Q-Network _). From these $Q$ functions values we choose the best action.

**Policy gradient**: This method directly estimates the policy (set of actions) in the network. The outcome is a learning of an ordered set of actions which leads to maximize reward by probabilistically choosing a subset of actions. In this tutorial, we learn the actions using a gradient descent approach to learn the policies.

In this tutorial, we focus how to implement RL in CNTK. We choose a straight forward shallow network. One can extend the approaches by replacing our shallow model with deeper networks that are introduced in other CNTK tutorials.

Additionally, this tutorial is in its early stages and will be evolving in future updates.

## Before we start...
Please run the following cell from the menu above or select the cell below and hit `Shift + Enter` to ensure the environment is ready. Verify that the following imports work in your notebook.

```python
from __future__ import print_function
from __future__ import division
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
import pandas as pd
import seaborn as sns

style.use('ggplot')
%matplotlib inline
```

We use the following construct to install the OpenAI gym package if it is not installed. For users new to Jupyter environment, this construct can be used to install any python package.

```python
try:
    import gym
except:
    !pip install gym
    import gym
```

    Collecting gym
      Downloading gym-0.8.0.tar.gz (150kB)
    Requirement already satisfied: numpy>=1.10.4 in c:\anaconda\3\lib\site-packages (from gym)
    Requirement already satisfied: requests>=2.0 in c:\anaconda\3\lib\site-packages (from gym)
    Requirement already satisfied: six in c:\anaconda\3\lib\site-packages (from gym)
    Collecting pyglet>=1.2.0 (from gym)
      Downloading pyglet-1.2.4-py3-none-any.whl (964kB)
    Building wheels for collected packages: gym
      Running setup.py bdist_wheel for gym: started
      Running setup.py bdist_wheel for gym: finished with status 'done'
      Stored in directory: C:\Users\ebarsoum.REDMOND\AppData\Local\pip\Cache\wheels\21\8e\8c\9aa1b43491bd9950fa2bea8a470e1af3b48b0cedcf964a4452
    Successfully built gym
    Installing collected packages: pyglet, gym
    Successfully installed gym-0.8.0 pyglet-1.2.4

### Select the notebook run mode

There are two run modes:
- *Fast mode*: `isFast` is set to `True`. This is the default mode for the notebooks, which means we train for fewer iterations or train / test on limited data. This ensures functional correctness of the notebook though the models produced are far from what a completed training would produce.

- *Slow mode*: We recommend the user to set this flag to `False` once the user has gained familiarity with the notebook content and wants to gain insight from running the notebooks for a longer period with different parameters for training.

```python
isFast = True
```

# CartPole: Data and Environment

We will use the [CartPole](https://gym.openai.com/envs/CartPole-v0) environment from OpenAI's [gym](https://github.com/openai/gym) simulator to teach a cart to balance a pole. Please follow the links to get more details.

In every time step, the agent
 * gets an observation $(x, \dot{x}, \theta, \dot{\theta})$, corresponding to *cart position*, *cart velocity*, *pole angle with the vertical*, *pole angular velocity*,
 * performs an action `LEFT` or `RIGHT`, and
 * receives
  * a reward of +1 for having survived another time step, and
  * a new state $(x', \dot{x}', \theta', \dot{\theta}')$

The episode ends, if
 * the pole is more than 15 degrees from vertical and/or
 * the cart is moving more than 2.4 units from center.

The task is considered done, if
 * the agent achieved and averaged reward of 200 over the last 50 episodes (if you manage to get a reward of 200 averaged over the last 100 episode you can consider submitting it to OpenAI).

In fast mode these targets are relaxed.

# Part 1: DQN

After a transition $(s,a,r,s')$, we are trying to move our value function $Q(s,a)$ closer to our target $r+\gamma \max_{a'}Q(s',a')$, where $\gamma$ is a discount factor for future rewards and ranges in value between 0 and 1.

DQNs
 * learn the _Q-function_ that maps observation (state, action) to a `score`
 * use memory replay (previously recorded $Q$ values corresponding to different $(s,a)$ to decorrelate experiences (sequence state transitions)
 * use a second network to stabilize learning (*not* part of this tutorial)

### Setting up the model
\begin{equation}
l_1 = relu( x W_1 + b_1) \\
Q(s,a) = l_1 W_2 + b_2 \\
\end{equation}

We will start with a slightly modified version for Keras, https://github.com/jaara/AI-blog/blob/master/CartPole-basic.py, published by Jaromír Janisch in his [AI blog](https://jaromiru.com/2016/09/27/lets-make-a-dqn-theory/), and will then incrementally convert it to use CNTK.

We use a simple two-layer densely connected network, for simpler illustrations. More advance networks can be substituted.

**CNTK** concepts: The commented out code is meant to be an illustration of the similarity of concepts between CNTK API/abstractions against Keras.

```python
import random, numpy, math, os

#from keras.models import Sequential
#from keras.layers import *
#from keras.optimizers import *
from cntk import *
from cntk.layers import *
# Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        device.try_set_default_device(device.cpu())
    else:
        device.try_set_default_device(device.gpu(0))
```

STATE_COUNT = 4 (corresponding to $(x, \dot{x}, \theta, \dot{\theta})$),

ACTION_COUNT = 2 (corresponding to `LEFT` or `RIGHT`)

```python
env = gym.make('CartPole-v0')

STATE_COUNT  = env.observation_space.shape[0]
ACTION_COUNT = env.action_space.n

STATE_COUNT, ACTION_COUNT
```

    [2017-03-15 21:53:27,258] Making new env: CartPole-v0

    (4, 2)

Note: in the cell below we highlight how one would do it in Keras. And a marked similarity with CNTK. While CNTK allows for more compact representation, we present a slightly verbose illustration for ease of learning.

Additionally, you will note that, CNTK model doesn't need to be compiled explicitly and is implicitly done when data is processed during training.

CNTK effectively uses available memory on the system between minibatch execution. Thus the learning rates are stated as **rates per sample** instead of **rates per minibatch** (as with other toolkits).

```python
# Targetted reward
REWARD_TARGET = 30 if isFast else 200
# Averaged over these these many episodes
BATCH_SIZE_BASELINE = 20 if isFast else 50

H = 64 # hidden layer size

class Brain:
    def __init__(self):
        self.params = {}
        self.model, self.trainer, self.loss = self._create()
        # self.model.load_weights("cartpole-basic.h5")

    def _create(self):
        observation = input_variable(STATE_COUNT, np.float32, name="s")
        q_target = input_variable(ACTION_COUNT, np.float32, name="q")

        # model = Sequential()
        # model.add(Dense(output_dim=64, activation='relu', input_dim=STATE_COUNT))
        # model.add(Dense(output_dim=ACTION_COUNT, activation='linear'))

        # Following a style similar to Keras
        l1 = Dense(H, activation=relu)
        l2 = Dense(ACTION_COUNT)
        unbound_model = Sequential([l1, l2])
        model = unbound_model(observation)

        self.params = dict(W1=l1.W, b1=l1.b, W2=l2.W, b2=l2.b)

        lr = 0.00025
        # opt = RMSprop(lr=0.00025)
        # model.compile(loss='mse', optimizer=opt)

        # loss='mse'
        loss = reduce_mean(square(model - q_target), axis=0)
        meas = reduce_mean(square(model - q_target), axis=0)

        # optimizer=opt
        lr_schedule = learning_rate_schedule(lr, UnitType.minibatch)
        learner = sgd(model.parameters, lr_schedule, gradient_clipping_threshold_per_sample=10)
        trainer = Trainer(model, (loss, meas), learner)

        # CNTK: return trainer and loss as well
        return model, trainer, loss

    def train(self, x, y, epoch=1, verbose=0):
        #self.model.fit(x, y, batch_size=64, nb_epoch=epoch, verbose=verbose)
        arguments = dict(zip(self.loss.arguments, [x,y]))
        updated, results =self.trainer.train_minibatch(arguments, outputs=[self.loss.output])

    def predict(self, s):
        return self.model.eval([s])
```

The `Memory` class stores the different states, actions and rewards.

```python
class Memory:   # stored as ( s, a, r, s_ )
    samples = []

    def __init__(self, capacity):
        self.capacity = capacity

    def add(self, sample):
        self.samples.append(sample)

        if len(self.samples) > self.capacity:
            self.samples.pop(0)

    def sample(self, n):
        n = min(n, len(self.samples))
        return random.sample(self.samples, n)
```

The `Agent` uses the `Brain` and `Memory` to replay the past actions to choose optimal set of actions that maximize the rewards.

```python
MEMORY_CAPACITY = 100000
BATCH_SIZE = 64

GAMMA = 0.99 # discount factor

MAX_EPSILON = 1
MIN_EPSILON = 0.01 # stay a bit curious even when getting old
LAMBDA = 0.0001    # speed of decay

class Agent:
    steps = 0
    epsilon = MAX_EPSILON

    def __init__(self):
        self.brain = Brain()
        self.memory = Memory(MEMORY_CAPACITY)

    def act(self, s):
        if random.random() < self.epsilon:
            return random.randint(0, ACTION_COUNT-1)
        else:
            return numpy.argmax(self.brain.predict(s))

    def observe(self, sample):  # in (s, a, r, s_) format
        self.memory.add(sample)

        # slowly decrease Epsilon based on our eperience
        self.steps += 1
        self.epsilon = MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * math.exp(-LAMBDA * self.steps)

    def replay(self):
        batch = self.memory.sample(BATCH_SIZE)
        batchLen = len(batch)

        no_state = numpy.zeros(STATE_COUNT)

        # CNTK: explicitly setting to float32
        states = numpy.array([ o[0] for o in batch ], dtype=np.float32)
        states_ = numpy.array([(no_state if o[3] is None else o[3]) for o in batch ], dtype=np.float32)

        p = agent.brain.predict(states)
        p_ = agent.brain.predict(states_)

        # CNTK: explicitly setting to float32
        x = numpy.zeros((batchLen, STATE_COUNT)).astype(np.float32)
        y = numpy.zeros((batchLen, ACTION_COUNT)).astype(np.float32)

        for i in range(batchLen):
            s, a, r, s_ = batch[i]

            # CNTK: [0] because of sequence dimension
            t = p[0][i]
            if s_ is None:
                t[a] = r
            else:
                t[a] = r + GAMMA * numpy.amax(p_[0][i])

            x[i] = s
            y[i] = t

        self.brain.train(x, y)
```

### Brain surgery

As any learning experiences, we expect to see the initial state of actions to be wild exploratory and over the iterations the system learns the range of actions that yield longer runs and collect more rewards. The tutorial below implements the [$\epsilon$-greedy](https://en.wikipedia.org/wiki/Reinforcement_learning) approach.

```python
def plot_weights(weights, figsize=(7,5)):
    '''Heat map of weights to see which neurons play which role'''
    sns.set(style="white")
    f, ax = plt.subplots(len(weights), figsize=figsize)
    cmap = sns.diverging_palette(220, 10, as_cmap=True)

    for i, data in enumerate(weights):
        axi = ax if len(weights)==1 else ax[i]
        if isinstance(data, tuple):
            w, title = data
            axi.set_title(title)
        else:
            w = np.asarray(data)
        sns.heatmap(w, cmap=cmap, square=True, center=True, #annot=True,
                    linewidths=.5, cbar_kws={"shrink": .25}, ax=axi)
```

### Exploration - exploitation trade-off

Note that the initial $\epsilon$ is set to 1 which implies we are entirely exploring but as steps increase we reduce exploration and start leveraging the learnt space to collect rewards (a.k.a. exploitation) as well.

```python
def epsilon(steps):
    return MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * np.exp(-LAMBDA * steps)
plt.plot(range(10000), [epsilon(x) for x in range(10000)], 'r')
plt.xlabel('step');plt.ylabel('$\epsilon$')
```

We are now ready to train our agent using **DQN**. Note this would take anywhere between 2-10 min and we stop whenever the learner hits the average reward of 200 over past 50 batches. One would get better results if they could train the learner until say one hits a reward of 200 or higher for say larger number of runs. This is left as an exercise.

```python
TOTAL_EPISODES = 2000 if isFast else 3000

def run(agent):
    s = env.reset()
    R = 0

    while True:
        # Uncomment the line below to visualize the cartpole
        # env.render()

        # CNTK: explicitly setting to float32
        a = agent.act(s.astype(np.float32))

        s_, r, done, info = env.step(a)

        if done: # terminal state
            s_ = None

        agent.observe((s, a, r, s_))
        agent.replay()

        s = s_
        R += r

        if done:
            return R

agent = Agent()

episode_number = 0
reward_sum = 0
while episode_number < TOTAL_EPISODES:
    reward_sum += run(agent)
    episode_number += 1
    if episode_number % BATCH_SIZE_BASELINE == 0:
        print('Episode: %d, Average reward for episode %f.' % (episode_number,
                                                               reward_sum / BATCH_SIZE_BASELINE))
        if episode_number%200==0:
            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))
        if reward_sum / BATCH_SIZE_BASELINE > REWARD_TARGET:
            print('Task solved in %d episodes' % episode_number)
            plot_weights([(agent.brain.params['W1'], 'Episode %i $W_1$'%episode_number)], figsize=(14,5))
            break
        reward_sum = 0
agent.brain.model.save('dqn.mod')
```

If you run it, you should see something like
```
Episode: 20, Average reward for episode 20.700000.
Episode: 40, Average reward for episode 20.150000.
Episode: 60, Average reward for episode 21.100000.
...
Episode: 960, Average reward for episode 20.150000.
Episode: 980, Average reward for episode 26.700000.
Episode: 1000, Average reward for episode 32.900000.
Task solved in 1000 episodes
```

#### Task 1.1
Rewrite the model without using the layer lib.
#### Task 1.2
Play with different [learners](https://cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner). Which one works better? Worse? Think about which parameters you would need to adapt when switching from one learner to the other.

### Running the DQN model

```python
import cntk as C
env = gym.make('CartPole-v0')

num_episodes = 10  # number of episodes to run

modelPath = 'dqn.mod'
root = C.load_model(modelPath)

for i_episode in range(num_episodes):
    print(i_episode)
    observation = env.reset()  # reset environment for new episode
    done = False
    while not done:
        if not 'TEST_DEVICE' in os.environ:
            env.render()
        action = np.argmax(root.eval([observation.astype(np.float32)]))
        observation, reward, done, info  = env.step(action)
```

    [2017-03-15 21:54:44,454] Making new env: CartPole-v0

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9

# Part 2: Policy gradient
**Goal:**
\begin{equation}\text{maximize } E [R | \pi_\theta]
\end{equation}

**Approach:**
1. Collect experience (sample a bunch of trajectories through $(s,a)$ space)
2. Update the policy so that _good_ experiences become more probable

**Difference to DQN: **
 * we don't consider single $(s,a,r,s')$ transitions, but rather use whole episodes for the gradient updates
 * our parameters directly model the policy (output is an action probability), whereas in DQN they model the value function (output is raw score)

#### Rewards:
Remember, we get +1 reward for every time step, in which we still were in the game.

The problem: we normally do not know, which action led to a continuation of the game, and which was actually a bad one. Our simple heuristic: actions in the beginning of the episode are good, and those towards the end are likely bad (they led to losing the game after all).

```python
def discount_rewards(r, gamma=0.999):
    """Take 1D float array of rewards and compute discounted reward """
    discounted_r = np.zeros_like(r)
    running_add = 0
    for t in reversed(range(0, r.size)):
        running_add = running_add * gamma + r[t]
        discounted_r[t] = running_add
    return discounted_r
```

```python
discounted_epr = discount_rewards(np.ones(10))
f, ax = plt.subplots(1, figsize=(5,2))
sns.barplot(list(range(10)), discounted_epr, color="steelblue")
```

We normalize the rewards so that they tank below zero towards the end. gamma controls how late the rewards tank.

```python
discounted_epr_cent = discounted_epr - np.mean(discounted_epr)
discounted_epr_norm = discounted_epr_cent/np.std(discounted_epr_cent)
f, ax = plt.subplots(1, figsize=(5,2))
sns.barplot(list(range(10)), discounted_epr_norm, color="steelblue")
```

```python
discounted_epr = discount_rewards(np.ones(10), gamma=0.5)
discounted_epr_cent = discounted_epr - np.mean(discounted_epr)
discounted_epr_norm = discounted_epr_cent/np.std(discounted_epr_cent)
f, ax = plt.subplots(2, figsize=(5,3))
sns.barplot(list(range(10)), discounted_epr, color="steelblue", ax=ax[0])
sns.barplot(list(range(10)), discounted_epr_norm, color="steelblue", ax=ax[1])
```

### Setting up the model
\begin{equation}
l_1 = relu( x W_1 + b_1) \\
l_2 = l_1 W_2 + b_2 \\
\pi(a|s) = sigmoid(l_2)
\end{equation}

Note: in policy gradient approach, the output of the dense layer is mapped into to a 0-1 range via the sigmoid function.

```python
import cntk as C

TOTAL_EPISODES = 2000 if isFast else 10000

D = 4  # input dimensionality
H = 10 # number of hidden layer neurons

observations = C.input_variable(STATE_COUNT, np.float32, name="obs")

W1 = C.parameter(shape=(STATE_COUNT, H), init=C.glorot_uniform(), name="W1")
b1 = C.parameter(shape=H, name="b1")
layer1 = C.relu(C.times(observations, W1) + b1)

W2 = C.parameter(shape=(H, ACTION_COUNT), init=C.glorot_uniform(), name="W2")
b2 = C.parameter(shape=ACTION_COUNT, name="b2")
score = C.times(layer1, W2) + b2
# Until here it was similar to DQN

probability = C.sigmoid(score, name="prob")
```

**Policy Search**: The optimal policy search can be carried out with either gradient free approaches or by computing gradients over the policy space ($\pi_\theta$) which is parameterized by $\theta$. In this tutorial, we use the classic forward (`loss.forward`) and back (`loss.backward`) propagation of errors over the parameterized space $\theta$. In this case, $\theta = \{W_1, b_1, W_2, b_2\}$, our model parameters.

```python
input_y = C.input_variable(1, np.float32, name="input_y")
advantages = C.input_variable(1, np.float32, name="advt")

loss = -C.reduce_mean(C.log(C.square(input_y - probability) + 1e-4) * advantages, axis=0, name='loss')

lr = 0.001
lr_schedule = learning_rate_schedule(lr, UnitType.sample)
sgd = C.sgd([W1, W2], lr_schedule)

gradBuffer = dict((var.name, np.zeros(shape=var.shape)) for var in loss.parameters if var.name in ['W1', 'W2', 'b1', 'b2'])

xs, hs, label, drs = [], [], [], []
running_reward = None
reward_sum = 0
episode_number = 1

observation = env.reset()

while episode_number <= TOTAL_EPISODES:
    x = np.reshape(observation, [1, STATE_COUNT]).astype(np.float32)

    # Run the policy network and get an action to take.
    prob = probability.eval(arguments={observations: x})[0][0][0]
    action = 1 if np.random.uniform() < prob else 0

    xs.append(x)  # observation
    # grad that encourages the action that was taken to be taken

    y = 1 if action == 0 else 0  # a "fake label"
    label.append(y)

    # step the environment and get new measurements
    observation, reward, done, info = env.step(action)
    reward_sum += float(reward)

    # Record reward (has to be done after we call step() to get reward for previous action)
    drs.append(float(reward))

    if done:
        # Stack together all inputs, hidden states, action gradients, and rewards for this episode
        epx = np.vstack(xs)
        epl = np.vstack(label).astype(np.float32)
        epr = np.vstack(drs).astype(np.float32)
        xs, label, drs = [], [], []  # reset array memory

        # Compute the discounted reward backwards through time.
        discounted_epr = discount_rewards(epr)
        # Size the rewards to be unit normal (helps control the gradient estimator variance)
        discounted_epr -= np.mean(discounted_epr)
        discounted_epr /= np.std(discounted_epr)

        # Forward pass
        arguments = {observations: epx, input_y: epl, advantages: discounted_epr}
        state, outputs_map = loss.forward(arguments, outputs=loss.outputs,
                                          keep_for_backward=loss.outputs)

        # Backward psas
        root_gradients = {v: np.ones_like(o) for v, o in outputs_map.items()}
        vargrads_map = loss.backward(state, root_gradients, variables=set([W1, W2]))

        for var, grad in vargrads_map.items():
            gradBuffer[var.name] += grad

        # Wait for some batches to finish to reduce noise
        if episode_number % BATCH_SIZE_BASELINE == 0:
            grads = {W1: gradBuffer['W1'].astype(np.float32),
                     W2: gradBuffer['W2'].astype(np.float32)}
            updated = sgd.update(grads, BATCH_SIZE_BASELINE)

            # reset the gradBuffer
            gradBuffer = dict((var.name, np.zeros(shape=var.shape))
                              for var in loss.parameters if var.name in ['W1', 'W2', 'b1', 'b2'])

            print('Episode: %d. Average reward for episode %f.' % (episode_number, reward_sum / BATCH_SIZE_BASELINE))

            if reward_sum / BATCH_SIZE_BASELINE > REWARD_TARGET:
                print('Task solved in: %d ' % episode_number)
                break

            reward_sum = 0

        observation = env.reset()  # reset env
        episode_number += 1
probability.save('pg.mod')
```

    Episode: 20. Average reward for episode 19.400000.
    Episode: 40. Average reward for episode 16.200000.
    Episode: 60. Average reward for episode 20.450000.
    Episode: 80. Average reward for episode 17.600000.
    Episode: 100. Average reward for episode 17.200000.
    Episode: 120. Average reward for episode 18.550000.
    Episode: 140. Average reward for episode 19.250000.
    Episode: 160. Average reward for episode 19.550000.
    Episode: 180. Average reward for episode 18.600000.
    Episode: 200. Average reward for episode 21.600000.
    Episode: 220. Average reward for episode 16.750000.
    Episode: 240. Average reward for episode 17.800000.
    Episode: 260. Average reward for episode 18.450000.
    Episode: 280. Average reward for episode 17.800000.
    Episode: 300. Average reward for episode 21.800000.
    Episode: 320. Average reward for episode 20.150000.
    Episode: 340. Average reward for episode 19.450000.
    Episode: 360. Average reward for episode 20.500000.
    Episode: 380. Average reward for episode 20.200000.
    Episode: 400. Average reward for episode 20.800000.
    Episode: 420. Average reward for episode 19.200000.
    Episode: 440. Average reward for episode 21.600000.
    Episode: 460. Average reward for episode 20.150000.
    Episode: 480. Average reward for episode 23.850000.
    Episode: 500. Average reward for episode 19.400000.
    Episode: 520. Average reward for episode 20.250000.
    Episode: 540. Average reward for episode 19.950000.
    Episode: 560. Average reward for episode 22.250000.
    Episode: 580. Average reward for episode 21.800000.
    Episode: 600. Average reward for episode 17.000000.
    Episode: 620. Average reward for episode 17.600000.
    Episode: 640. Average reward for episode 19.700000.
    Episode: 660. Average reward for episode 21.200000.
    Episode: 680. Average reward for episode 23.650000.
    Episode: 700. Average reward for episode 23.700000.
    Episode: 720. Average reward for episode 24.350000.
    Episode: 740. Average reward for episode 19.650000.
    Episode: 760. Average reward for episode 27.050000.
    Episode: 780. Average reward for episode 25.700000.
    Episode: 800. Average reward for episode 24.500000.
    Episode: 820. Average reward for episode 28.050000.
    Episode: 840. Average reward for episode 31.550000.
    Task solved in: 840

# Solutions
#### Solution 1.1

```python

```

```python
observation = input_variable(STATE_COUNT, np.float32, name="s")

W1 = parameter(shape=(STATE_COUNT, H), init=glorot_uniform(), name="W1")
b1 = parameter(shape=H, name="b1")
layer1 = relu(times(observation, W1) + b1)
W2 = parameter(shape=(H, ACTION_COUNT), init=glorot_uniform(), name="W2")
b2 = parameter(shape=ACTION_COUNT, name="b2")
model = times(layer1, W2) + b2
W1.shape, b1.shape, W2.shape, b2.shape, model.shape
```

    ((4, 10), (10,), (10, 2), (2,), (2,))


# QHack 2022 Open Hackathon Project: Matrix-Model Simulations using VQE
Team Name: anonymousr007<br>
Team Member: [Rishabh Singh](https://github.com/anonymousr007)

# Introduction

Matrix quantum mechanics plays various important roles in theoretical physics, such as a holographic description of quantum black holes.Understanding quantum black holes and the role of entanglement in a holographic setup is of paramount importance for the development of better quantum algorithms (quantum error correction codes) and for the realization of a quantum theory of gravity. Quantum computing offer us potentially useful approaches to study the dynamics of matrix quantum mechanics. Gauge/gravity duality translates difficult (or intractable) problems in quantum gravity to well-defined problems in non-gravitational quantum theories. Although it originated from string/M-theory, connections to various other fields including quantum information theory, condensed matter theory, and cosmology, have been recognized by now with research programs actively pursued. Quantum mechanics with matrix degrees of freedom (matrix QM, or matrix models in the following) play important roles in gauge/gravity duality. More specifically, Yang–Mills-type matrix models can provide us with a nonperturbative formulation of superstring/M-theory. In certain regions of the parameter space (i.e., strong coupling and large-N), weakly-coupled gravity with small stringy corrections can be described by matrix models.Therefore, matrix models can offer us an ideal setup to study quantum corrections to gravity. By solving the matrix models, it should be possible to study fascinating problems such as the microscopic mechanism of black hole evaporation.

## Quantum Computer

- On a quantum computer, quantum states are directly realized.
- Quantum Computing does not suffer from the sign problem
- The Hamiltonian time evolution can be simulated straightforwardly.
- Some research directions that could benefit from direct quantum simulations are the formation and evaporation of a black hole, or a wormhole, and quantum teleportation.
- The biggest challenge is that we do not yet have a reliable large-scale quantum computer.
- Although we can emulate a quantum computer on a classical computer, these simulations are restricted to systems with only a small number of qubits.
- Furthermore, the effects of the regularization needed to map a theory on a quantum device are not well understood yet.

## Variational Quantum Eigensolver (VQE)

- The Variational Quantum Eigensolver (VQE) is expected to be a useful quantum-classical hybrid algorithm in the Noisy Intermediate Scale Quantum (NISQ) era.
- Being a variational method, the VQE can have systematic errors: a priori, we do not know if we can get the right answer, because the simulation might be
trapped in a local optimum.
- This is a general problem for variational methods.
- Quantum computing represents the computation in terms of qubits and quantum gates which can represent larger Hamiltonians and Hilbert spaces on quantum hardware than classical computers that are limited by memory constraints.
- Also, quantum computing represents fermions straightforwardly in a similar way to how they represent bosons and, as mentioned above, are not susceptible to the sign problem like classical methods using Monte Carlo algorithms.
- Some of the disadvantages of NISQ devices include the lack of quantum error correction for quantum gates and the presence of noise which can lead to inaccurate quantum gates and decoherence of the qubits during the computation.
- For hybrid quantum-classical computations that we discuss here, the advantages are that many parts of a hybrid quantum-classical variational calculation, such as optimization, can be done efficiently on a classical computer, while expectation values can be computed on a quantum computer.
- This leads to a lower depth for the circuits, making them less susceptible to decoherence.
- However, these methods include complicated state-preparation protocols and variational ansatze for the wave function that need to be specified
to have strong overlaps with the true ground state. Such wave functions may need a high-depth quantum circuit which is difficult to implement on a NISQ hardware, and it is prone to decoherence.

# Abstract

This project aims to use a truncated (regularized) Hamiltonian for the matrix quantum mechanics models. This Hamiltonian is constructed by considering a truncated Hilbert space in the Fock basis. The truncated Hilbert space is constructed starting from the individual matrix degrees of freedom.

Two types of matrix quantum mechanics models are used

- A Yang-Mills-type bosonic 2-matrix model with SU(2) gauge group, which has 6 bosonic degrees of freedom in total.
- A Supersymmetric 2-matrix model with SU(2) gauge group which corresponds to with the minimal number of degrees of freedom 6 bosons and 3 fermions.

Quantum mechanics with matrix degrees of freedom plays an important role in gauge-gravity duality. Gauge-gravity duality translates difficult problems in quantum gravity to well-defined problems in non-gravitational quantum theories. Although it originated from string–M-theory, connections to various other fields, including

- Quantum Information Theory
- Condensed Matter Theory
- Cosmology
- Holographic simulation of Quantum Black Holes
- Complex high-dimensional supergravity theories

We use the **Variational Quantum EigenSolver (VQE)** to estimate the low-energy spectrum As for the VQE, the specific architecture that we use does not show a satisfactory performance at strong coupling, perhaps due to the variational forms parametrized by the quantum circuits not adequately probing the full gauge-invariant Hilbert space. This result shows that going beyond the VQE and using more complicated or fully quantum algorithms is not the correct way to approach matrix quantum mechanics for now, because they would require even deeper quantum circuits that are more prone to noise on actual quantum hardware.

```python
#@title Check the box below if you have a GPU

HAS_GPU = True #@param {type: "boolean"}
if HAS_GPU:
  print("Installing qiskit with GPU support")
  !pip3 -q install qiskit[visualization] qiskit-aer-gpu pylatexenc
else:
  print("Installing qiskit")
  !pip3 -q install qiskit[visualization] pylatexenc
```

    Installing qiskit with GPU support

```python
!pip install qiskit
```

    Requirement already satisfied: qiskit in /usr/local/lib/python3.7/dist-packages (0.34.2)
    Requirement already satisfied: qiskit-ibmq-provider==0.18.3 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.18.3)
    Requirement already satisfied: qiskit-terra==0.19.2 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.19.2)
    Requirement already satisfied: qiskit-aer==0.10.3 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.10.3)
    Requirement already satisfied: qiskit-ignis==0.7.0 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.7.0)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.7.3)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.21.5)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.1.0)
    Requirement already satisfied: websocket-client>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.2.3)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Requirement already satisfied: retworkx>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (0.11.0)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.7.1)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (5.4.8)
    Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (3.11)
    Requirement already satisfied: tweedledum<2.0,>=1.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.1.1)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.3.4)
    Requirement already satisfied: stevedore>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (3.5.0)
    Requirement already satisfied: python-constraint>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.4.0)
    Requirement already satisfied: symengine>=0.8 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.9.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.5.0)
    Requirement already satisfied: cryptography>=1.3 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (36.0.1)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (4.11.1)
    Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (5.8.1)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.7.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.10.0.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.2->qiskit) (1.2.1)

# Libraries

```python
# Let us begin by importing necessary libraries.
import time
import sys
import numpy as np
import matplotlib.pyplot as plt
import warnings
%matplotlib inline

# qiskit
from qiskit import Aer
from qiskit.algorithms import VQE, NumPyEigensolver
from qiskit.algorithms.optimizers import SLSQP, SPSA
from qiskit.circuit.library import EfficientSU2
from qiskit.opflow import MatrixOp
from qiskit.utils import algorithm_globals, QuantumInstance

# scipy
from scipy.sparse import diags, identity, kron
from scipy.sparse.linalg import eigsh
```

# Basis operators for a single boson

We define the annihilation operator in a $N \times N$ representation for each bosonic matrix, and we define the identity operator.
Combining them for all bosonic matrices will allow us to construct the Hamiltonian.

```python
n = 2  # cutoff for Fock space. The number of qubits used will be K = log2(N)
annihilation_operator = np.array(np.diagflat(np.sqrt(np.linspace(1,n-1,n-1)),k=1))
identity = np.identity(n)
```

```python
print(annihilation_operator)
```

    [[0. 1.]
     [0. 0.]]

```python
print(identity)
```

    [[1. 0.]
     [0. 1.]]

```python
bosonic_dof = 6  # number of bosonic dof as matrices: SU(2) -> 2^2-1=6 generators
bosonList = [annihilation_operator]
for bosons in range(0, bosonic_dof-1):
  bosonList.append(identity)
with np.printoptions(precision=3, suppress=True, linewidth=120, threshold=sys.maxsize):
  for i in bosonList:
    print(f"{i}\n")
```

    [[0. 1.]
     [0. 0.]]

    [[1. 0.]
     [0. 1.]]

    [[1. 0.]
     [0. 1.]]

    [[1. 0.]
     [0. 1.]]

    [[1. 0.]
     [0. 1.]]

    [[1. 0.]
     [0. 1.]]

```python
# This for loop takes the appropriate Kronecker products for each boson.
for i in range(0, bosonic_dof):
  for j in range(0, bosonic_dof - 1):
    # For the nth boson, the nth Kronecker product is with the annihilation operator.
    if j == i-1 and i != 0:
        bosonList[i] = np.kron(bosonList[i], annihilation_operator)
    # Else, the nth Kronecker product is with the identity matrix.
    else:
        bosonList[i] = np.kron(bosonList[i], identity)
```

```python
[x.shape for x in bosonList]
```

    [(64, 64), (64, 64), (64, 64), (64, 64), (64, 64), (64, 64)]

```python
fig, ax = plt.subplots(ncols=6, figsize=(18, 6))
fig.subplots_adjust(left=0.02, bottom=0.06, right=0.95, top=0.94, wspace=0.5, hspace=1)
for i,axi in enumerate(np.ravel(ax)):
  pos = axi.imshow(bosonList[i], cmap='jet')
  fig.colorbar(pos, ax=axi)
  axi.set_title(f"Boson {i+1}")
fig.suptitle('Operators', fontsize=16)
fig.subplots_adjust(top=0.9)
```

# Creating the Bosons and the Quartic Hamiltonians

The following cell defines a function which creates the bosons, then it creates the simple quartic and full quartic SU2 Hamiltonians. The function recieves the number of bosons and the size of the bosons as arguments. For example, for 6 bosons that are 8x8, the function call would look like bosonHamiltonians(6, 8). The function returns the two Hamiltonians.

First, an annihilation operator and identity matrix is created. Each of the 6 bosons is constructed by taking the Kronecker product of the annihilation operator with the identity matrix, in a specific order. For the $n^{th}$ boson, the annhiliation operator will be in the $n^{th}$ position of the Kronecker product. For example, the first boson's Kronecker product would look like $$\hat{a} \otimes I \otimes I \otimes I \otimes I \otimes I$$ where $\hat{a}$ is the annihilation operator and $I$ is the identity matrix. The 2nd boson would look like $$I \otimes \hat{a} \otimes I \otimes I \otimes I \otimes I$$ and so on.

```python
def bosonHamiltonians(num_boson, bosonMatrixDim, coupling=0.1):
    # This function takes the number of bosons (num_boson) and the size of the bosons (bosonMatrixDim)
    # as arguments. If the size of the boson is 2 x 2, for example, then bosonMatrixDim = 2.
    # coupling is lambda/2 and lambda is the 'tHooft coupling'

    # Create the n x n annhiliation operator. Here, we create a list with all the integers from
    # the square root of 1 to the square root of n-1, where n x n is the size of the boson.
    # This list is injected as the upper diagonal in an array of zeros.
    ann_op = np.array(np.diagflat(np.sqrt(np.linspace(1,bosonMatrixDim-1,bosonMatrixDim-1)),k=1))

    # Create the n x n identity matrix.
    id = np.identity(bosonMatrixDim)

    # Create a list which holds the six bosons. bosonList[0] is the first boson. bosonList[5] is the 6th boson.
    # For the first boson, the Kronecker product starts with the annihilation operator.
    bosonList = [ann_op]

    # This for loop creates the list of bosons before the Kronecker products are taken. The first one was
    # already created in the previous line.
    for bosons in range(0,num_boson-1):
        bosonList.append(id)

    # This for loop takes the appropriate Kronecker products for each boson.
    for i in range(0,num_boson):
        for j in range(0,num_boson-1):

            # For the nth boson, the nth Kronecker product is with the annihilation operator.
            if j == i-1 and i != 0:
                bosonList[i] = np.kron(bosonList[i], ann_op)
            # Else, the nth Kronecker product is with the identity matrix.
            else:
                bosonList[i] = np.kron(bosonList[i], id)

    # Create the position operators. Normalized as in the notes
    x = []
    for r in range(0, num_boson):
        x.append((1/np.sqrt(2))*(bosonList[r] + np.transpose(np.conjugate(bosonList[r]))))

    # Create the simple quadratic Hamiltonian.
    H2MM = 0

    for i in range(0,num_boson):
        # The @ symbol is a shorthand for matrix multiplication. It's equivalent to using np.matmul().
        H2MM = H2MM + (np.transpose(np.conjugate(bosonList[i])) @ bosonList[i])

    H2MM = H2MM + 0.5*num_boson*np.identity(bosonMatrixDim**(num_boson))

    # Create the full quartic SU(2) Hamiltonian.
    x_sq = []

    for i in x:
        x_sq.append(i @ i)

    H4MM1 = (H2MM + coupling*((x_sq[2] @ x_sq[3])
                              + (x_sq[2] @ x_sq[4])
                              + (x_sq[1] @ x_sq[3])
                              + (x_sq[1] @ x_sq[5])
                              + (x_sq[0] @ x_sq[4])
                              + (x_sq[0] @ x_sq[5])
                              - 2*((x[0] @ x[2]) @ (x[3] @ x[5]))
                              - 2*((x[0] @ x[1]) @ (x[3] @ x[4]))
                              - 2*((x[1] @ x[2]) @ (x[4] @ x[5]))))
    return H2MM, H4MM1

# Set the number of bosons here. For the proper full quartic SU(2) Hamiltonian, this has to be 6.
num_boson = 6

# Set the size of the bosons here. For an n x n boson, bosonMatrixDim = n.
# 5 crashes this colab with 12Gb of RAM ->  tries 6 of these tcmalloc: large alloc 1953128448 bytes
bosonMatrixDim = 2

# the function actually creates intermediate matrices x and x2 so the memory footprint is even higher...for N=5 it requires ~50GB of RAM
H2MM, H4MM1 = bosonHamiltonians(num_boson, bosonMatrixDim, coupling=0.1)  # we can change the coupling by adding coupling=0.5 to the args

# Set the Hamiltonian that you want to run the VQE algorithm with here.
hamiltonian = H4MM1

# Use numpy to find the eigenvalues of the simple quartic Hamiltonian
# this can take a long time for large matrices
val,vec=np.linalg.eig(hamiltonian)
z = np.argsort(val)
z = z[0:len(hamiltonian)]
energies=(val[z])

if hamiltonian.all() == H4MM1.all():
    # Sometimes there are negligible imaginary parts (e.g. order of 10^-16), so I take the real parts only.
    print('Full Quartic SU2 Eigenvalues:\n\n', np.real(energies))
else:
    print('Simple Quartic Eigenvalues:\n\n', energies)
```

    Full Quartic SU2 Eigenvalues:

     [3.14807787 4.14674965 4.14674965 4.14674965 4.14674965 4.14674965
     4.14674965 5.04817155 5.09501244 5.09501244 5.09501244 5.09501244
     5.09501244 5.09501244 5.2        5.2        5.2        5.2
     5.2        5.2        5.2        5.2        6.         6.
     6.10024938 6.10024938 6.10024938 6.10024938 6.10024938 6.10024938
     6.2        6.2        6.2        6.2        6.2        6.2
     6.2        6.2        6.2        6.2        6.2        6.2
     7.05192213 7.10498756 7.10498756 7.10498756 7.10498756 7.10498756
     7.10498756 7.2        7.2        7.2        7.2        7.2
     7.2        7.2        7.2        8.15300098 8.15300098 8.15300098
     8.15300098 8.15300098 8.15300098 9.15182845]

# Convert the Hamiltonian
Now, we take the Hamiltonian matrix and map it onto a qubit operator. There may be better ways to do this now that `qiskit` has added new ways to map Hamiltonians to qubit operators.

In fact, we should skip the creation of the Hamiltonian as a matrix and directly used operators in `qiskit`

```python
start_time = time.time()
warnings.filterwarnings("ignore")

qubit_operator = MatrixOp(primitive=hamiltonian)
print("Size of Hamiltonian:", np.shape(hamiltonian))

end_time = time.time()

runtime = end_time-start_time
print('Program runtime: ',runtime)
```

    Size of Hamiltonian: (64, 64)
    Program runtime:  0.0030336380004882812

```python
print(qubit_operator)
```

    Operator([[3.15+0.j, 0.  +0.j, 0.  +0.j, ..., 0.  +0.j, 0.  +0.j,
               0.  +0.j],
              [0.  +0.j, 4.15+0.j, 0.  +0.j, ..., 0.  +0.j, 0.  +0.j,
               0.  +0.j],
              [0.  +0.j, 0.  +0.j, 4.15+0.j, ..., 0.  +0.j, 0.  +0.j,
               0.  +0.j],
              ...,
              [0.  +0.j, 0.  +0.j, 0.  +0.j, ..., 8.15+0.j, 0.  +0.j,
               0.  +0.j],
              [0.  +0.j, 0.  +0.j, 0.  +0.j, ..., 0.  +0.j, 8.15+0.j,
               0.  +0.j],
              [0.  +0.j, 0.  +0.j, 0.  +0.j, ..., 0.  +0.j, 0.  +0.j,
               9.15+0.j]],
             input_dims=(2, 2, 2, 2, 2, 2), output_dims=(2, 2, 2, 2, 2, 2))

```python
print("Total number of qubits: ",qubit_operator.num_qubits)
print("Type of qubitOp: ",type(qubit_operator).__name__)
```

    Total number of qubits:  6
    Type of qubitOp:  MatrixOp

```python
ansatz = EfficientSU2(qubit_operator.num_qubits, su2_gates=['ry'], entanglement="full", reps=1)
ansatz.draw(output='mpl')
```

```python
np.log2(hamiltonian.shape[0])
```

    6.0

# Looping the VQE algorithm over several optimizers

## Statevector simulator

```python
warnings.filterwarnings("ignore")
seed = 0
algorithm_globals.random_seed = seed
if HAS_GPU:
  backend = Aer.get_backend("statevector_simulator", method="statevector_gpu")
else:
  backend = Aer.get_backend("statevector_simulator", max_parallel_threads=6, max_parallel_experiments=0)
quantum_instance = QuantumInstance(backend, seed_transpiler = seed, seed_simulator = seed)
```

```python
counts = []
values = []

def store_intermediate_result(eval_count, parameters, mean, std):
    counts.append(eval_count)
    values.append(mean)

optimizer = SLSQP(maxiter=1000)

vqe = VQE(ansatz=ansatz, optimizer=optimizer, quantum_instance=quantum_instance, callback=store_intermediate_result)
result = vqe.compute_minimum_eigenvalue(qubit_operator)
vqe_result = np.real(result.eigenvalue)
print(f"VQE gs energy: {vqe_result}")
```

    VQE gs energy: 3.150000100711781

```python
# Plot all of the optimizers in a single convergence plot. If you see that the optimizer isn't converging,
# increase the maximum iterations or maximum function evaluations in the optimizer list above.

labelList = ["SLSQP"]

plt.plot(counts, values)
plt.axhline(y=np.real(energies[0]),xmin=0,xmax=counts[-1],color='k',linestyle='--',label="Exact Energy")
plt.xlabel('Eval count')
plt.ylabel('Energy')
plt.title('Energy convergence')
plt.legend(loc='upper right', labels=labelList)
```

## QASM Simulator

```python
warnings.filterwarnings("ignore")

backendQasm = Aer.get_backend('qasm_simulator')
quantum_instance = QuantumInstance(backend=backendQasm, shots=1024)
optimizer = SPSA(maxiter=10000)

counts = []
values = []

def store_intermediate_result(eval_count, parameters, mean, std):
    counts.append(eval_count)
    values.append(mean)

vqe = VQE(ansatz=ansatz, optimizer=optimizer, quantum_instance=quantum_instance, callback=store_intermediate_result)

result = vqe.compute_minimum_eigenvalue(qubit_operator)
vqe_result = np.real(result.eigenvalue)
print(f"VQE gs energy (QASM): {vqe_result}")
```

    VQE gs energy (QASM): 3.15

```python
# Plot convergence plot for the SPSA optimizer
plt.plot(counts, values, label="SPSA")
plt.plot(counts,[np.real(energies[0])]*len(counts), 'k--',label="Exact Energy")
plt.xlabel('Eval count')
plt.ylabel('Energy')
plt.title('Energy convergence of SPSA optimizer (QASM)')
plt.legend(loc='upper right')
```

```python
import qiskit.tools.jupyter
%qiskit_version_table
```

<h3>Version Information</h3><table><tr><th>Qiskit Software</th><th>Version</th></tr><tr><td><code>qiskit-terra</code></td><td>0.19.2</td></tr><tr><td><code>qiskit-aer</code></td><td>0.10.3</td></tr><tr><td><code>qiskit-ignis</code></td><td>0.7.0</td></tr><tr><td><code>qiskit-ibmq-provider</code></td><td>0.18.3</td></tr><tr><td><code>qiskit</code></td><td>0.34.2</td></tr><tr><th>System information</th></tr><tr><td>Python version</td><td>3.7.12</td></tr><tr><td>Python compiler</td><td>GCC 7.5.0</td></tr><tr><td>Python build</td><td>default, Jan 15 2022 18:48:18</td></tr><tr><td>OS</td><td>Linux</td></tr><tr><td>CPUs</td><td>1</td></tr><tr><td>Memory (Gb)</td><td>12.686653137207031</td></tr><tr><td colspan='2'>Fri Feb 25 20:28:37 2022 UTC</td></tr></table>


# Lecture 1: What is Sound?
Audio Processing, MED4, Aalborg University, 2021

Jesper Kjær Nielsen (jekjaer@gmail.com) and Cumhur Erkut (cer@create.aau.dk), Aalborg University

Table of Contents

1  Sound and simple vibrations

1.1  What is sound?

1.2  Human hearing

1.3  Sinusoids

1.4  Summary

2  Complex numbers

2.1  The need for complex numbers

2.2  The complex number

2.3  Summary

2.4  Additional information on complex numbers

2.5  Exit Question

3  Phasors

3.1  The polar (or exponential) form of a complex number

3.2  The phasor

3.3  Summary

## Sound and simple vibrations
In the next 20 minutes, you will learn
- What sound is and how it propagates
- How humans perceive sound
- What a sinusoid is
- that striking a bar creates a sinusoidal sound

### What is sound?

```python
from IPython.display import display, YouTubeVideo, Image
YouTubeVideo('GkNJvZINSEY')
```

Sound is a **vibration** which travels through a medium as
- **longitudinal** waves: gasses (e.g., air), liquids (e.g., water), and solids (e.g., concrete)
- **transversal** sound waves: solids (e.g., concrete)

The **speed of sound** depends on the medium. In, e.g., air at room temperature, it is approximately 343 m/s.

Sound is normally divided into three types:
1. Infrasound: Sound with frequencies up to 20 Hz
2. Audible sound: Sound with frequencies in range 20 Hz - 20 kHz (this is often referred to as **audio**)
3. Ultra sound: Sound with frequencies above 20 kHz

### Human hearing

```python
YouTubeVideo('eQEaiZ2j9oc')
```

The human ear consists of the following parts:
- **Outer ear**: Everything on the outside of the ear drum, including the pinna
- **Middle ear**: The three bones (Malleus, Incus, Stapes) acting as a mechanical transducer between the ear drum and the oval window in the cochlea
- **Inner ear**: The cochlea which is a coiled, liqued-filled tube. Subsided in the liqued is the basilar membrane which vibrates when sound propagates through the fluid. These vibrations are converted into electrical signals by hair cells attached to the basilar membrane.

The human ear
- does not hear all frequencies equally well
- is most sensitive to frequencies around 4 kHz
- is tuned to speech
- has a really large dynamic range of up to ~120 dB (i.e., we can hear sound intensities up to ~10^12 the quietest sounds)

### Sinusoids
A sinusoid (or a sine wave) is given by
$$
    x(t) = A \cos(\Omega t + \Phi)
$$
where
- $A\geq0$ is the **amplitude**
- $\Omega$ is the **frequency** measured in radians pr. second (SI symbol **rad/s**). Is related to the frequency $f$ measured in cycles pr. second (SI symbol **Hz**) via $\Omega = 2\pi f$.
- $t$ is the **time** measured in seconds (SI symbol **s**)
- $\Phi$ is the **initial phase** measured in radians (SI symbol **rad**)

The above form of the sinusoid is often referred to as the **polar form**. By using the angle addition formula for a cosine, i.e.,
$$
    \cos(\theta+\phi) = \cos(\theta)\cos(\phi)-\sin(\theta)\sin(\phi)\ ,
$$
a sinusoid can also be written in a **rectangular form** as
$$
    x(t) = a\cos(\Omega t) + b\sin(\Omega t)
$$
where a and b are scalers given by
\begin{align}
    a &= A\cos(\Phi)\\
    b &= -A\sin(\Phi)\ .
\end{align}

#### Numpy example: A sinusoid

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

samplingFreq = 44100 # Hz
nData = 2000
time = np.arange(0,nData).T/samplingFreq # s

# Generate a sinusoid
amp = 1;
freq = 100 # Hz
initPhase = np.pi/2 # rad
sinusoid = amp*np.cos(2*np.pi*freq*time+initPhase)

# Plot the sinusoids
plt.plot(time, sinusoid, lineWidth=2)
plt.xlim((time[0],time[nData-1]))
plt.ylim((-1,1))
plt.xlabel('Time [s]')
plt.ylabel('Amplitude [.]')
plt.grid(True)
```

#### Example: Generation of a sinusoid from a vibrating bar

Assume that the act of striking a bar is modelled as **compressing a spring** in one dimension. From Hooke's law, this compresssion can be written as
$$
    F(t) = -k x(t)
$$
where
- $F(t)$ is the **restoring force** measured in Newton (SI unit **N**)
- $x(t)$ is the **displacement** measured in meters (SI unit **m**) of the string from its resting position
- $k$ is the **spring constant** measured in N/m

From **Newton's second law**, the force can also be expressed as
$$
    F(t) = ma(t)
$$
where
- $m$ is the **mass** of the string measured in kilogram (SI unit **kg**)
- $a(t)$ is the **acceleration** measured in m/s^2.
The acceleration is related to the displacement $x(t)$ as
$$
    a(t) = \frac{dv(t)}{dt} = \frac{d^2 x(t)}{d t^2}
$$
where $v(t)$ is the **velocity** measured in m/s.

Combining these three equations gives
$$
    -k x(t) = F(t) = ma(t) = m \frac{d^2 x(t)}{dt^2}
$$
which can be rewritten as
$$
    \frac{d^2 x(t)}{dt^2} = -\frac{k}{m} x(t)\ .
$$
This is a constant-coefficient second-order differential equation.

Let us check if our sinusoid
$$
    x(t) = A\cos(\Omega t + \Phi)
$$
is a solution to the above differential equation. Since
\begin{align}
    \frac{dx(t)}{dt} &= -\Omega A\sin(\Omega t + \Phi)\\
    \frac{d^2 x(t)}{d t^2} &= -\Omega^2 A\cos(\Omega t + \Phi) = -\Omega^2 x(t)\ ,
\end{align}
we obtain
$$
    -\Omega^2x(t) = -\frac{k}{m} x(t)\ .
$$
Thus, striking a bar will make it vibrate sinusoidally with the frequency
$$
    \Omega = \sqrt{k/m}\ .
$$
This frequency can be changed by changing the spring constant and mass.

### Summary
- Sound is a vibration travelling through a medium.
- Sound waves are longitudal waves (and also transversal waves when travelling through a solid).
- The human ear converts pressure variations in the air to
 1. mechanical movement (interface is the eardrum)
 2. vibrations in a liquid (interface is the oval window)
 3. electrical signal to the brain (interface is the haircells attached to the basilar membrane)
- A sinusoid (or sine wave) is given by
$$
    x(t) = A\cos(\Omega t + \Phi)\ ,
$$
and it an extremely important building block (or atom) in analysing and manipulating sound.
- Assuming that striking a bar can be modelled as compressing a spring, the bar will vibrate sinusoidally.

## Complex numbers
In the next 20 minutes, you will learn
- that the equation
$$
    x^2+1=0
$$
has two solutions
- what a complex number is
- how you add and multiply complex numbers

### The need for complex numbers
While the **linear** equation
$$
    x + 1 = 0
$$
can easily be solved, the simple **quadratic** equation
$$
    x^2 + 1 = 0
$$
was in high school said to have **no** solution since its descriminant was negative.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

nData = 100
x = np.linspace(-2,2,nData)
y = x**2+1
plt.plot(x,y,linewidth=2)
plt.xlabel('x')
plt.ylabel('y')
plt.ylim((-1,5))
plt.grid(True);
```

However, the quadratic equation can in fact be solved by using **complex numbers**.

```python
YouTubeVideo('T647CGsuOVU')
```

Rearranging our simple **quadratic** equation gives
$$
    x^2 = -1
$$
which allows us to write the solution as
$$
    x = \pm\sqrt{-1} = \pm j
$$
where
$$
    j = \sqrt{-1}
$$
is the **imaginary unit**. This unit also satisfies that
$$
    j^2 = \sqrt{-1}^2 = -1\ .
$$

Note that
- **engineers** normally use the symbol $j$ for the imaginary unit
- **mathematicians** normally use the symbol $i$ for the imaginary unit.

Let us now consider the quadratic equation
$$
    x^2 + 2x + 5 = 0\ .
$$

We know from high school that the solutions to the general quadratic
$$
    ax^2 + bx + c = 0\ ,\qquad\text{for }a\neq0
$$
have the form
$$
    x = \frac{-b\pm\sqrt{d}}{2a}
$$
where $d$ is the **discriminant** given by
$$
    d = b^2-4ac\ .
$$

We obtain
$$
    d = 4-20 = -16
$$
so that
$$
    x = \frac{-2\pm\sqrt{-16}}{2} = -1\pm\frac{1}{2}\sqrt{-1\cdot 4^2} = -1\pm 2\sqrt{-1} = -1\pm 2j\ .
$$
Thus, the **complex numbers** $-1+2j$ and $-1-2j$ are the solutions.

### The complex number
A **complex number** can be written as
$$
    z = a + jb
$$
where
- $a = \text{Re}\{z\}$ is the **real** part
- $b = \text{Im}\{z\}$ is the **imaginary** part.

A complex number can be depicted in the **complex plane** which is a 2D coordinate system.

#### The complex conjugate
The **complex conjugate** of a complex number $z$ is
$$
    z^* = a - jb\ .
$$
Thus, the conjugation operator ${}^*$ changes the sign of imaginary part, but not the real part.

#### Addition of complex numbers
Assume we have the two complex numbers
\begin{align}
    z_1 &= a_1+jb_1\\
    z_2 &= a_2+jb_2\ .
\end{align}

The **sum** of these two numbers is then
$$
    z = z_1 + z_2 = a_1+jb_1 + a_2+jb_2 = (a_1+a_2) + j(b_1+b_2).
$$

Thus, the real and imaginary part of of $z=a+jb$ are simply
\begin{align}
    a &= a_1 + a_2\\
    b &= b_1 + b_2\ .
\end{align}

Note that
\begin{align}
    z_1 + z_1^* &= 2a_1 + 0j = 2\text{Re}(z_1)\\
    z_1 - z_1^* &= 0 + 2jb_1 = 2\text{Im}(z_1)\ .
\end{align}

#### Multiplication of complex numbers
Assume we have the two complex numbers
\begin{align}
    z_1 &= a_1+jb_1\\
    z_2 &= a_2+jb_2\ .
\end{align}

The **product** of these two numbers is then
$$
    z = z_1z_2 = (a_1+jb_1)(a_2+jb_2) = (a_1a_2-b_1b_2) + j(a_1b_2+b_1a_2).
$$

Thus, the real and imaginary part of of $z=a+jb$ are
\begin{align}
    a &= (a_1a_2-b_1b_2)\\
    b &= (a_1b_2+b_1a_2)\ .
\end{align}

Note that
$$
    z_1z_1^* = (a_1a_1-b_1(-b_1)) +j(a_1b_1-b_1a_1) = a_1^2+b_1^2 = \text{Re}(z_1)^2+\text{Im}(z_1)^2
$$

### Summary
- Complex numbers were originally invented to solve algebraic equations (e.g., the cubic equation)
- The imaginary unit is $j=\sqrt{-1}$
- A **complex number** $z$ consists of a real part $a$ and imaginary part $b$, and is written as
$$
    z = a+jb\ .
$$
- The **complex conjugate** of $z$ is
$$
    z^* = a-jb\ .
$$
- It is much easier to add two complex numbers than it is to multiply them.

### Additional information on complex numbers
If you want to know more about complex numbers (e.g., its history), you can find some nice videos here:

https://www.youtube.com/playlist?list=PLiaHhY2iBX9g6KIvZ_703G3KJXapKkNaF

### Active 5 minutes break
Let
\begin{align}
    z_1 &= a_1+jb_1 =  2+3j\\
    z_2 &= a_2+jb_2 = -1-2j\ .
\end{align}

By hand, please calculate
\begin{align}
    z_1 + z_2 &= \\
    z_1 - z_2 &= \\
    z_1 + z_1^* &= \\
    z_2 - z_2^*+2z_1 &= \\
    z_1z_2^* &=\\
    z_1^2+z_2^*z_1 &=
\end{align}
Check the results with your neighbours.

---
**Tip:** Use the rules
\begin{align}
    z_1 + z_2 &= (a_1+a_2) + j(b_1+b_2)\\
     z_1z_2 &= (a_1a_2-b_1b_2) + j(a_1b_2+b_1a_2)\ .
\end{align}

## Phasors
In the next 20 minutes, you will learn
- how a complex number can be written in a **polar form**
- why the polar form makes multiplications much easier
- what a **phasor** is
- how a phasor is related to a **real sinusoid**

### The polar (or exponential) form of a complex number
As for 2D vectors, we can also write a complex number in terms of its **magnitude** $r$ and **angle** $\psi$. We have
\begin{align}
    a &= r\cos\psi\\
    b &= r\sin\psi\ .
\end{align}
Thus,
$$
    z = a + jb = r\left(\cos\psi + j\sin\psi\right) = r\mathrm{e}^{j\psi}
$$
where the last equality follows from **Euler's formula**.

#### Euler's formula
Given by
$$
    \mathrm{e}^{j\psi} = \cos\psi + j\sin\psi\ .
$$
- A very important formula used everywhere in science and engineering
- Simplifies notation and mathematical manipulations
- Its real and imaginary parts are a cosine and a sine, respectively, i.e.,
\begin{align}
    \text{Re}(\mathrm{e}^{j\psi}) &= \cos\psi\\
    \text{Im}(\mathrm{e}^{j\psi}) &= \sin\psi\ .
\end{align}

#### The complex conjugate
The **complex conjugate** of a complex number
$$
    z=r \mathrm{e}^{j\psi}
$$
is
$$
    z^* = r \mathrm{e}^{-j\psi}\ .
$$
Thus, the conjugation operator ${}^*$ changes the sign of the angle, but not the magnitude.

#### Multiplication of complex numbers
Multiplication of complex numbers is much easier when the polar form is used. Let
\begin{align}
    z_1 &= a_1+jb_1 = r_1 \mathrm{e}^{j\psi_1}\\
    z_2 &= a_2+jb_2 = r_2 \mathrm{e}^{j\psi_2}\ .
\end{align}

The **product** of these two numbers is then
$$
    z = z_1z_2 = r_1 \mathrm{e}^{j\psi_1}r_2 \mathrm{e}^{j\psi_2} = r_1 r_2 \mathrm{e}^{j\psi_1}r_2 \mathrm{e}^{j\psi_2} = r_1 r_2 \mathrm{e}^{j(\psi_1+\psi_2)}
$$
where we used $a^na^m = a^{n+m}$ to get the last equation.

Thus, to multiply two complex numbers we
- multiply their magnitudes
- add their angles

Note that **divisions** can be calculated as multiplications since
$$
    \frac{z_1}{z_2} = z_1\frac{1}{z_2} = z_1 z_2^{-1}
$$
and
$$
    z_2^{-1} = \frac{1}{r_2}\mathrm{e}^{-j\psi_2}\ .
$$

#### Converting between the rectangular and polar forms
We have seen that a complex number $z$ can be written as
$$
    z = a+jb = r\mathrm{e}^{j\psi}\ .
$$

We can convert from the polar coordinates $(r,\psi)$ to the rectangular coordinates $(a,b)$ via
\begin{align}
    a &= r\cos\psi\\
    b &= r\sin\psi\ .
\end{align}

We can convert from the rectangular coordinates $(a,b)$ to the polar coordinates $(r,\psi)$ via
\begin{align}
    r &= \sqrt{a^2+b^2}\\
    \psi &= \mathrm{arctan2}(b,a)\ .
\end{align}

### The phasor
We have previously looked at the sinusoid
$$
    x(t) = A\cos(\Omega t + \Phi)\ .
$$

Based on what we know about Euler's formula and complex numbers, we can now also write $x(t)$ as
$$
    x(t) = \text{Re}\left[A\exp(j(\Omega t +\Psi))\right]
$$
since (from Euler's formula)
$$
    A\exp(j(\Omega t +\Psi)) = A\cos(\Omega t +\Psi)+jA\sin(\Omega t +\Psi)\ .
$$
This time-varying complex number is called a **phasor** or a **complex sinusoid**.

Note that
- using the phasor instead of the real sinusoid makes life much easier (you will see this later in the course)
- even though we work with the phasor, we can always come back to the real sinusoid by taking the real part of the phasor

### Summary
- The **polar form** of a complex number $z=a+jb$ is
$$
    z = r\mathrm{e}^{j\psi}
$$
where the magnitude $r$ and angle $\psi$ are given by
\begin{align}
    r &= \sqrt{a^2+b^2}\\
    \psi &= \mathrm{arctan2}(b,a)\ .
\end{align}
- Multiplications (and divisions) are much easier when using the polar form.
- A **phasor** is a complex sinusoid given by
$$
    z(t) = A\exp(j(\Omega t +\Psi))\ ,
$$
and its real part is a real sinusoid, i.e.,
$$
    x(t) = \text{Re}(z(t)) = A\cos(\Omega t +\Psi)\ .
$$


# Encoding tests for UTF-8 text

This file tries to understand how to encode universally all UTF-8 possible characters.
The idea is to create a basic encoding that could be reused not by domain, like current word encodings, but that can handle all current languages and symbols.

~~The idea is to go to a context based encoding, for this a lot of training will be needed, but if this works, this can be game changer.~~

As this file tries to encode all the characters possible by utf-8 we have to check the feasible number so:

From [Wikipedia utf-8](https://en.wikipedia.org/wiki/UTF-8)

UTF-8 is a variable width character encoding capable of encoding all 1,112,064.

$$17×2^{16} = 1114112 $$ code points minus 2,048 technically-invalid surrogate code points

As for the number of letters, we can check a few places:

- [The Bible](https://wordcounter.net/blog/2015/12/08/10975_how-many-words-bible.html) 783,137 words, 3,116,480 characters

- [10 longest novels](http://mentalfloss.com/article/18661/quick-10-10-longest-novels-ever)  The Blah Story by Nigel Tomm. 3,277,227 words

the longest book about 5 times more words than the bible, so about 15 Million characters should deal with almost all the english text. I imagine that almost all languages could be around that for the most characters.

```python
import numpy as np
```

```python
from encoders import *
from helpers import *
```

```python
utf_enc = Lin2WaveEncoder(0,1114112, neg_allowed=False)
```

```python
utf_enc.coefficients
```

    [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40]

```python
len(utf_enc.coefficients)
```

    20

```python
2**utf_enc.n_high
```

    2097152

We can encode almost double the elements ... which is OK, better to be safe than sorry

```python
char_pos_enc= Lin2WaveEncoder(0,15000000, neg_allowed=False)
len(char_pos_enc.coefficients)
```

    23

```python
2**char_pos_enc.n_high
```

    16777216

It seems that with about 43 elements we can encode all the things that we want, so for the moment what I'll do is add a few tets to check that every encoding is correct

```python
utf_enc.coefficients
```

    [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40]

```python
utf_enc.periods
```

    [2,
     4,
     8,
     16,
     32,
     64,
     128,
     256,
     512,
     1024,
     2048,
     4096,
     8192,
     16384,
     32768,
     65536,
     131072,
     262144,
     524288,
     1048576]

```python
# nums = np.array(list(range(1114113)))
# enc_nums = utf_enc.encode(nums)
# dec_nums = utf_enc.decode(enc_nums)
```

```python
# diffs = dec_nums - nums
```

```python
# diffs.max()
```

```python
# diffs.min()
```

```python
for i in range(10000, 10010):  # range(1114113):
    ei = utf_enc.encode(np.array([i]))
    di = utf_enc.decode(ei)
    print(di - np.array([i]), di, [i], ei.T)

```

    [-7614.99628525] [2385.00371475] [10000] [[0.00601678 0.17493624 0.32673182 0.58800814 0.00195542 0.13115829
      0.70151093 0.98928396 0.8150547  0.33285724 0.00724349 0.82218012
      0.96967038 0.78657779 0.65023042 0.57599823 0.53810998 0.51906886
      0.50953616 0.5047683 ]]
    [-7615.37501024] [2385.62498976] [10001] [[1.03564933e-01 2.79033554e-01 3.86558448e-01 5.57094223e-01
      8.18281119e-04 1.36477680e-01 6.97929851e-01 9.89682423e-01
      8.15812401e-01 3.32397127e-01 7.28495169e-03 8.22086763e-01
      9.69691306e-01 7.86602798e-01 6.50244973e-01 5.76005771e-01
      5.38113780e-01 5.19070767e-01 5.09537118e-01 5.04768776e-01]]
    [-7615.52474358] [2386.47525642] [10002] [[2.98174216e-01 3.96869493e-01 4.48155289e-01 5.25957358e-01
      1.68588183e-04 1.41885822e-01 6.94336695e-01 9.90073413e-01
      8.16568902e-01 3.31937175e-01 7.32653409e-03 8.21993385e-01
      9.69712229e-01 7.86627804e-01 6.50259527e-01 5.76013312e-01
      5.38117583e-01 5.19072673e-01 5.09538072e-01 5.04769253e-01]]
    [-7615.55751564] [2387.44248436] [10003] [[5.42197489e-01 5.21117588e-01 5.10561150e-01 4.94719130e-01
      6.97213906e-06 1.47381393e-01 6.90731677e-01 9.90456925e-01
      8.17324195e-01 3.31477384e-01 7.36823395e-03 8.21899987e-01
      9.69733146e-01 7.86652809e-01 6.50274080e-01 5.76020852e-01
      5.38121387e-01 5.19074579e-01 5.09539025e-01 5.04769730e-01]]
    [-7615.61038605] [2388.38961395] [10004] [[7.75889346e-01 6.44052693e-01 5.72802208e-01 4.63501524e-01
      3.33590802e-04 1.52963050e-01 6.87115019e-01 9.90832953e-01
      8.18078278e-01 3.31017753e-01 7.41005127e-03 8.21806570e-01
      9.69754056e-01 7.86677813e-01 6.50288633e-01 5.76028393e-01
      5.38125190e-01 5.19076485e-01 5.09539979e-01 5.04770207e-01]]
    [-7615.81518888] [2389.18481112] [10005] [[0.94203387 0.7580313  0.63390721 0.43242644 0.00114813 0.15862943
      0.68348694 0.99120149 0.81883115 0.33055828 0.00745199 0.82171313
      0.96977496 0.78670282 0.65030319 0.57603593 0.53812899 0.51907839
      0.50954093 0.50477068]]
    [-7616.26952678] [2389.73047322] [10006] [[0.99995308 0.85596677 0.69292264 0.40161524 0.00244978 0.16437915
      0.67984766 0.99156254 0.8195828  0.33009898 0.00749404 0.82161968
      0.96979585 0.78672782 0.65031774 0.57604347 0.5381328  0.5190803
      0.50954189 0.50477116]]
    [-7617.0159332] [2389.9840668] [10007] [[0.93546635 0.93176995 0.74892757 0.37118822 0.00423728 0.17021081
      0.67619741 0.99191608 0.82033323 0.32963983 0.00753621 0.8215262
      0.96981674 0.78675282 0.65033229 0.57605102 0.5381366  0.5190822
      0.50954284 0.50477164]]
    [-7618.0341349] [2389.9658651] [10008] [[0.76436226 0.98072777 0.80104807 0.34126421 0.00650889 0.17612298
      0.6725364  0.99226212 0.82108245 0.32918085 0.00757849 0.82143271
      0.96983762 0.78677782 0.65034684 0.57605856 0.5381404  0.51908411
      0.50954379 0.50477211]]
    [-7619.24816254] [2389.75183746] [10009] [[0.52853307 0.99979626 0.84847082 0.31196006 0.00926239 0.18211423
      0.66886486 0.99260064 0.82183043 0.32872203 0.0076209  0.8213392
      0.9698585  0.78680282 0.6503614  0.5760661  0.53814421 0.51908601
      0.50954475 0.50477259]]

```python
l2f = Lin2Fourier(0, 1114112, terms=5)
```

```python
c0 = l2f.coefficients[0]
```

```python

```

```python
l2f.factors[0](10)
```

    5.639635246590041e-05

```python
ten = np.array(range(10))
```

```python
eten = l2f.encode(ten)
```

```python
l2f.fs.subs(abc.x,10)
```

    -1114112*sin(5*pi/278528)/pi - 557056*sin(5*pi/139264)/pi - 1114112*sin(15*pi/278528)/(3*pi) - 278528*sin(5*pi/69632)/pi - 1114112*sin(25*pi/278528)/(5*pi)

```python
l2f.fs.subs(abc.x,4).evalf()
```

    -39.9999999626816

```python
lf = lambdify(abc.x, l2f.fs, "numpy")
```

```python
lf(4)
```

    -39.999999962681564

```python
l2f.fs
```

    -1114112*sin(pi*x/557056)/pi - 557056*sin(pi*x/278528)/pi - 1114112*sin(3*pi*x/557056)/(3*pi) - 278528*sin(pi*x/139264)/pi - 1114112*sin(5*pi*x/557056)/(5*pi)

```python
# l2f.decode(eten)
```

```python
c = [c for c in l2f.factors][3]
```

```python
c
```

    <function _lambdifygenerated(x)>

```python
# for i in range(10):  # range(1114113):
#     ei = l2f.encode(np.array([i]))
#     di = l2f.decode(ei)
#     print(di - np.array([i]), di, [i], ei.T)

```

```python
import numpy as np
numpy = np  # FIXME, somewhere the reference is missing
import sympy
from sympy import abc
from sympy import lambdify
from sympy import fourier_series, pi
```

```python
min_val = 0
max_val = 1114112
terms = 20
fs = fourier_series(abc.x, (abc.x, min_val, max_val))  # .truncate(terms)
```

```python
fs
```

    FourierSeries(x, (x, 0, 1114112), (0, SeqFormula(0, (_k, 1, oo)), SeqFormula(Piecewise((-620622774272*cos(2*_n*pi)/(_n*pi) + 310311387136*sin(2*_n*pi)/(_n**2*pi**2), (_n > -oo) & (_n < oo) & Ne(_n, 0)), (0, True))*sin(_n*pi*x/557056)/557056, (_n, 1, oo))))

```python
fs_args = fs.args
```

```python
# coef_fact = [ (lambdify(abc.x, (np.product(f.args[:-1]), "numpy")[0]),  # evaluate  sympy.pi to numeric
#                       lambdify(abc.x, f.args[-1], "numpy")) for f in fs_args  # create lambda functions
#                      ]
```

```python
# coefficients, factors = zip(*coef_fact)
```

```python
# coefficients = np.stack(coefficients)
```

```python
# coefficients
```

```python
a = 'a'
b = 'b'
s = "this is a text"
s1 = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHI'
```

```python
ea = a.encode('utf-8')
```

```python
int.from_bytes(ea, byteorder='big')
```

    97

```python
import commpy
```

```python
commpy.
```


# Control volume analysis

This module covers developing our governing equations (conservation of mass, momentum, and energy) by applying control volume analysis.
It also introduces the concept of the stagnation reference state.

Our analysis requires us to make a series of assumptions about the fluid flow:

 1. In general, flow is three-dimensional and unsteady, so $V = f(t, x, y, z)$
 2. If we assume a flow is two-dimensional and unsteady, then $V = f(t, x, y)$
 3. We can next assume the flow is unsteady and one-dimensional:  $V = f(t, x)$
 4. Finally, we can assume one-dimensional steaedy flow: $V = f(x)$

However, one-dimensional steady flow is **not** the same as *unidirectional flow*,
since the overall flow direction can change.

## Total derivative

For a control mass, the **total/material/substantial derivative** describes the time rate at which a property changes as we follow it around:

$$
\frac{D (\cdot)}{Dt} \text{ or } \frac{d(\cdot)}{dt}
$$

The total derivative = convective derivative (changes due to movement of fluid to a new location) + local/partial derivative (changes with time at a given location). For example, the total derivative for pressure is

$$
\frac{dp}{dt} = \frac{\partial p}{\partial x} \frac{dx}{dt} + \frac{\partial p}{\partial y} \frac{dy}{dt} + \frac{\partial p}{\partial z} \frac{dz}{dt} + \frac{\partial p}{\partial t} \;,
$$

where $\frac{dx}{dt}$, $\frac{dy}{dt}$, and $\frac{dz}{dt}$ are the components of flow velocity.

If fluid properties are independent of time, then we can assume steady flow and thus

$$
\frac{\partial (\cdot)}{\partial t} = 0
$$

:::{note}
How good is the assumption of one-dimensional flow in a duct?

For *laminar* flow, the velocity forms a parabolic shape, with the peak in the center and zero at the walls. Thus, $\dot{m} = \rho A \frac{V_{\max}}{2} = \rho A V$, where *V* is the average velocity. Clearly, the velocity varies significantly at a given axial location, and so the assumption of one-dimensional flow is not good.

For *turbulent* flow, the larger inertial forces lead to more mixing, and so the velocity profile is much more uniform: $V_{\max} \approx V$, so the one-dimensional assumption is reasonable.

Since most applications in gas dynamics will be at high-enough Reynolds numbers for the flow to be fully turbulent, the one-dimensional assumption is pretty good!
:::

## Total derivative → control volume

With fluids, it is much more convenient to work with control volumes, so we need a way to convert from equations governing a control mass (written with a total derivative) to those that apply to control volumes.

We can work with any extensive property *N*, where

$$
N = \int \eta \, dm \equiv \iiint \rho \eta \, d \tilde{v} = \int_V \rho \eta \, d\tilde{v} \;,
$$

where *N* is the total amount of the property in a given mass and $\eta$ is the
amount per unit mass (the intensive version of this property).
The overall change in the property *N* is then $\frac{dN}{dt}$.

Let's follow a given control mass of fluid from time $t$ to time $t + \Delta t$,
as it moves through space. The volume of space that the control mass occupied
initially is our **control volume**, which the mass crosses as it moves.
For a small time increment $\Delta t$, the volumes occupied by the fluid at time $t$
and time $t + \Delta t$ will overlap.

We can write the overall change in *N* with time as

$$
\begin{align}
\frac{dN}{dt} &\equiv \lim_{\Delta t \to 0} \frac{ (\text{final value of } N)_{t+\Delta t} - (\text{initial value of } N)_t }{\Delta t} \\
&= \lim_{\Delta t \to 0} \frac{ (N_2 + N_3)_{t+\Delta t} - (N_1 + N_2)_t}{\Delta t} \;,
\end{align}
$$
where 1+2 represents the original volume and 2+3 represents the new region, and 2 represents the overlap between the two spaces.

Let's focus on each of the terms present in that total derivative.

The quantity

$$
\lim_{\Delta t \to 0} \frac{N_3 (t+\Delta t)}{\Delta t}
$$

represents the amount of property $N$ lost by the fluid out of the control volume.
(After all, region 3 is *formed* by the fluid moving out.)
We can relate this to the fluid that crosses the surface of the control volume:

$$
\int_{S_{\text{out}}} \eta \rho \left( \vec{V} \cdot \hat{n} \right) \, dA \, \Delta t \approx \text{amount of $N$ in region 3} \;,
$$

where $S_{\text{out}}$ is the area where fluid leaves the control volume,
$\vec{V}$ is the velocity vector of the fluid, and
$\hat{n}$ is the unit normal vector of the surface area.
So, then

$$
\therefore \lim_{\Delta t \to 0} \frac{N_3 (t+\Delta t)}{\Delta t} = \int_{S_{\text{out}}} \eta \rho \left( \vec{V} \cdot \hat{n} \right) \, dA
$$

and this represents the flux of $N$ out of the control volume.

We can do similar work for the flux of $N$ into the control volume:

$$
\lim_{\Delta t \to 0} \frac{N_1 (t)}{\Delta t} = \int_{S_{\text{in}}} \eta \rho \left( \vec{V} \cdot \hat{n}^{\prime} \right) \, dA \;,
$$

where $\hat{n}^{\prime} = -\hat{n}$ is the unit normal vector pointing into the region.

Finally, the remaining terms are

$$
\begin{align}
\lim_{\Delta t \to 0} \frac{ N_2 (t+\Delta t) - N_2 (t) }{\Delta t} &= \frac{\partial N_2}{\partial t} \\
&= \frac{\partial N_{cv}}{\partial t} \\
&= \frac{\partial}{\partial t} \int_{cv} \rho \eta \, d\tilde{v} \;,
\end{align}
$$

because $N_2 \to N_{cv}$ as $\Delta t \to 0$.

## Reynolds transport theorem

Now, we can rewrite the total derivative with these quantities based on the control volume:

$$
\frac{dN}{dt} = \frac{\partial}{\partial t} \int_{cv} \rho \eta \, d\tilde{v} + \int_{S_{\text{out}}} \eta \rho \left( \vec{V} \cdot \hat{n} \right) \, dA - \int_{S_{\text{in}}} \eta \rho \left( \vec{V} \cdot \hat{n}^{\prime} \right) \, dA \;,
$$

but as $\Delta t \to 0$, and thus the region of overlap becomes the control volume itself,
$\hat{n} = -\hat{n}^{\prime}$, and the areas where the fluid leave and enter the control volume
are the *control surface* (CS), and so

$$
\frac{dN}{dt} = \frac{\partial}{\partial t} \int_{CV} \rho \eta \, d\tilde{v} + \int_{CS} \eta \rho \left( \vec{V} \cdot \hat{n} \right) \, dA \;,
$$ (eq:reynolds)

which is the **Reynolds transport theorem**.
This expresses that the rate of change of an extensive property $N$
for a given mass as it moves around is equal to the rate of change of $N$
inside the control volume *plus* the net efflux (flow out - flow in) of $N$
from the control volume.

## Conservation of mass

Conservation of mass states that, for a control mass, $\frac{d (\text{mass})}{dt} = 0$.
We can apply this to a control volume by recognizing that $N = \text{mass}$ and $\eta = 1$.
Then, applying this to Reynolds transport theorem {eq}`eq:reynolds` we get the continuity
equation for a control volume:

$$
0 = \frac{\partial}{\partial t} \int_{CV} \rho \, d\tilde{v} + \int_{CS} \rho (\vec{V} \cdot \hat{n}) \, dA
$$

This applies to a general flow.

For steady one-dimensional flow we can simplify this further.
First, $\frac{\partial}{\partial t} = 0$, and for a given inlet/outlet we can write

$$
\int \rho (\vec{V} \cdot \hat{n}) \, dA = \rho \vec{V} \cdot \hat{n} \int dA = \rho V A
$$

Thus, for a steady one-dimensional flow, conservation of mass says that

$$
0 = \sum \rho V A
$$

over all inlets and outlets. For flow with a single inlet and single outlet,
this simplifies to

$$
\dot{m} = \rho A V = \text{constant}
$$

Alternately, we can express conservation of mass by differentiating this expression:

$$
\begin{gather*}
d (\rho A V) = 0 = A V \, d\rho + \rho V \, dA + \rho A \, dV \\
\frac{d \rho}{\rho} + \frac{dA}{A} + \frac{dV}{V} = 0 \,
\end{gather*}
$$

which is a useful expression for flow through a duct, channel, or streamline.
Also, it shows us that if $dA = 0$ (i.e., the area is constant), then
increasing flow velocity means decreasing flow density, and vice versa.

## Conservation of energy

For a control mass, conservation of energy (i.e., the first law of thermodynamics) says

$$
\begin{align}
Q &= W + \Delta E \\
\frac{\delta Q}{dt} &= \frac{\delta W}{dt} + \frac{dE}{dt}
\end{align}
$$

where the first two terms are the instantaneous rates of heat and work transfer
between the system and its surroundings, and $\frac{dE}{dt}$ is the total derivative
of energy. In this case, $N = E$, and $\eta = e = u + \frac{V^2}{2} + gz$,
so applying Reynolds transport theorem {eq}`eq:reynolds` gives:

$$
\begin{align}
\frac{dE}{dt} &= \frac{\partial}{\partial t} \int_{CV} e \rho \, d\tilde{v} + \int_{CS} e \rho (\vec{V} \cdot \hat{n}) \, dA \\
\rightarrow \frac{\delta Q}{dt} &= \frac{\delta W}{dt} + \frac{\partial}{\partial t} \int_{CV} e \rho \, d\tilde{v} + \int_{CS} e \rho (\vec{V} \cdot \hat{n}) \, dA
\end{align}
$$

For one-dimensional flow, we can simplify the integral over the control surface
to consider just the inlets and outlets separately:

$$
\int_{CS} e \rho (\vec{V} \cdot \hat{n}) \, dA = \sum \dot{m} e \;,
$$

where contributions are positive where fluid leaves the control volume (outlet) and
negative where fluid enters the control volume (inlet).

### Example: pressure-volume and shaft work

Consider a system with pressure-volume work from a piston and shaft work.
We should choose the control surface appropriately so that there is no fluid motion
at the boundary, except where
 1. fluid enters and leaves the control volume, and
 2. a device (e.g., shaft) crosses the boundary.

There is no pressure/shear work along the sidewalls, since the fluid velocity at the wall is zero.

The pressure work is then

$$
\delta W' = \vec{F} \cdot d\vec{x} = p A \, dx = p A V \, dt
$$

and the rate of this work is

$$
\frac{\delta W'}{dt} = p A V = \dot{m} p v \;,
$$

which represents the work done to push the fluid out of the control volume.
The total work is then

$$
\frac{\delta W}{dt} = \frac{\delta W_s}{dt} + \sum \dot{m} p v
$$

and we can rewrite conservation of energy as

$$
\frac{\delta Q}{dt} = \frac{\delta W_s}{dt} + \frac{\partial}{\partial t} \int_{CV} e \rho \, d\tilde{v} + \sum \dot{m} (e + pv)
$$

For *steady* one-dimensional flow, this simplifies to

$$
\frac{\delta Q}{dt} = \frac{\delta W_s}{dt} + \sum \dot{m} (e + pv) \;,
$$

and for a single inlet, single outlet duct, where
$\dot{m}_{\text{in}} = \dot{m}_{\text{out}} = \dot{m}$,
we have

$$
q = w_s + \left( e + p v \right)_{\text{out}} - \left( e + p v \right)_{\text{in}}
$$

where $q = \frac{1}{\dot{m}} \frac{\delta Q}{dt}$ and
$w_s = \frac{1}{\dot{m}} \frac{\delta W_s}{dt}$.

For example, for a system with one inlet, one outlet, shaft work, and heat transfer,
conservation of energy simplifies down to the usual

$$
\begin{gather*}
q = w_s + \left( e_2 + p_2 v_2 \right) - \left( e_1 + p_1 v_1 \right) \\
h_1 + \frac{V_1^2}{2} + g z_1 + q = h_2 + \frac{V_2^2}{2} + g z_2 + w_s
\end{gather*}
$$

### Entropy

We can also develop a useful expression that relates pressure, energy, and entropy.
Recall that $\Delta S \equiv \int \frac{\delta Q_R}{T}$.

We can express changes in entropy as $dS = dS_e + dS_i$,
separating into changes caused by actual heat transfer ($dS_e$)
and changes caused by irreversibilities ($dS_i$).
By definition,

$$
dS_e = \frac{\delta Q}{T} \;,
$$

where $dS_e = 0$ for adiabatic processes.
Then, $dS_i \geq 0$, and this comes from irreversible effects
(e.g., temperature or pressure gradients, friction). For a reversible process, $dS_i = 0$.

Let's consider a cyclic integral of entropy (meaning, integrate over a cycle):

$$
\oint dS = \oint dS_e + \oint dS_i = 0 \;,
$$

by definition, since entropy is a property and we are integrating over a cycle
(which ends back at the original state).
But,

$$
\begin{align}
\oint dS_i &\geq 0 \\
0 &= \oint dS_e + (\geq 0) \\
\therefore \oint \frac{\delta Q}{T} &\leq 0 \\
\end{align}
$$

which is the **inequality of Clausius**.

### Pressure-energy equation

We can obtain a pressure-energy equation by integrating our property relation
into conservation of energy, taking advantage of this approach to considering entropy:

$$
\begin{align}
T \, ds &= dh - v \, dp \\
T \, ds_e + T \, ds_i &= dh - \frac{dp}{\rho} \\
dh &= T \, ds_e + T \, ds_i + \frac{dp}{\rho}
\end{align}
$$

Incorporate this into the energy equation:

$$
\begin{gather*}
\delta q = \delta w_s + \left( T \, ds_e + T \, ds_i + \frac{dp}{\rho} \right) + \frac{dV^2}{2} + g \, dz \\
\frac{dp}{\rho} + \frac{dV^2}{2} + g \, dz + \delta w_s + T \, ds_i = 0 \;,
\end{gather*}
$$

which is the **pressure-energy equation**.

```{note}
We can obtain Euler's equation from the pressure-energy equation by applying it to a reversible process with no shaft work:

$$
\frac{dp}{\rho} + \frac{dV^2}{2} + g \, dz = 0
$$
```

## Stagnation state

Up until this point, we have exclusively referred to **static properties**, which are measured as though we are moving along with the fluid.

A useful concept is the **stagnation state**, which is a reference state associated with the fluid
being brought to rest and at zero potential, *isentropically*, and with no work done.
This means that there is no energy exchange with surroundings ($Q = W = 0$, and $dS_e = 0$),
there are no losses ($dS_i = 0$).

We can apply the energy equation from any point in the flow *a* to the stagnation reference point *b*:

$$
\begin{align}
h_a + \frac{V_a^2}{2} + g z_a + q &= h_b + \frac{V_b^2}{2} + g z_b + w_s \\
\rightarrow h_b &= h_a + \frac{V_a^2}{2} + g z_a
\end{align}
$$

and thus we can define the **stagnation enthlapy**, or **total enthalpy**, as

$$
h_t = h + \frac{V^2}{2} + g z
$$

```{important}

The stagnation state is a reference state, and may or may not actually exist physically in a flow system.
```

The one-dimensional steady energy equation can then be written as

$$
h_{t1} + q = h_{t2} + w_s \;,
$$

or $\delta q = \delta w_s + d h_t$.

So, in an adiabatic, no-work system ($\delta q = \delta w_s = 0$),
**stagnation enthalpy remains constant**.

We can also consider a **stagnation pressure**, which is the pressure at the stagnation state.
Applying our property relationship to the stagnation state:

$$
\begin{align}
T \, ds &= dh - v \, dp \\
\rightarrow T_t \, ds_t &= d h_t - v_t \, d p_t
\end{align}
$$

where $d s_t = ds = ds_e + ds_i$, since the stagnation process is isentropic.
We can combine this with the (stagnation) energy equation:

$$
\begin{gather*}
T_t \left( ds_e + ds_i \right) = dh_t - v_t \, dp_t \\
\delta q = \delta w_s + d h_t \\
\delta q = \delta w_s + T_t \left( ds_e + ds_i \right) + v_t \, dp_t \\
\rightarrow \frac{dp_t}{\rho_t} + ds_e (T_t - T) + T_t \, ds_i + \delta w_s = 0 \;,
\end{gather*}
$$

which is the **stagnation pressure-energy equation**.
This looks complicated, but tells us something useful.
If we consider a process with no shaft work ($\delta w_s = 0$),
no heat transfer ($\delta q = 0$), and no losses ($ds_i = 0$),
then

$$
\frac{d p_t}{\rho_t} = 0 = d p_t \;,
$$

or stagnation pressure is constant.

Losses will be present in most real systems, and so the more realistic version of this equation is

$$
\frac{d p_t}{\rho_t} + T_t \, ds_i = 0 \,
$$

which shows us that losses cause the stagnation pressure to decrease (since $ds_i > 0$).

```{important}

The stagnation state, as well as stagnation enthalpy and stagnation pressure, are very important concepts that we will use throughout this course. You should always consider whether these quantities are constant or changing in a given scenario.
```

## Conservation of momentum

For a control mass, conservation of momentum comes from Newton's second law:

$$
\sum \vec{F} = \frac{ d (\vec{\text{momentum}})}{dt} \;.
$$

We can apply Reynolds transport theorem {eq}`eq:reynolds` to momentum, where
$\vec{N}$ is momentum and $\eta = \vec{V}$:

$$
\frac{ d (\vec{\text{momentum}})}{dt} = \frac{\partial}{\partial t} \int_{CV} \vec{V} \rho \, d\tilde{v} + \int_{CS} \vec{V} \rho (\vec{V} \cdot \hat{n} ) \, dA
$$

For one-dimensional flow, the area integral simplifies to

$$
\int_{CS} \vec{V} \rho (\vec{V} \cdot \hat{n} ) \, dA = \sum \dot{m} \vec{V} \;,
$$

and for steady, one-dimensional flow, we can write conservation of momentum as

$$
\begin{align}
\sum \vec{F} &= \sum \dot{m} \vec{V} \\
\sum \vec{F} &= \dot{m} \left( \vec{V}_{\text{out}} - \vec{V}_{\text{in}} \right)
\end{align}
$$

where the latter form applies if we have one inlet and one outlet.

For example, consider flow through a duct, which includes friction at the wall.
We can focus on the $x$-component of the momentum equation:

$$
\begin{align}
\sum F_x &= \dot{m} \left( V_{\text{out}, x} - V_{\text{in}, x} \right) \\
\rightarrow p_1 A_1 - p_2 A_2 - F_f &= \dot{m} (V_2 - V_1)
\end{align}
$$