#### _Speech Processing Labs 2020: Signals: Module 2_

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 3 Impulse Source, IIR Filter

### Learning Outcomes
* Be able to explain the difference between an FIR and an IIR filter
* Understand that IIR filters can be applied to impulse trains to produce different spectral envelopes

### Need to know
* Topic Videos: Impulse Response, Source Filter Model
* [Building the source: impulse trains](sp-m2-1-impulse-as-source.ipynb)
* [FIR filters](sp-m2-2-fir-filters.ipynb)

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 3.0 Filters again

The previous notebook explored Finite Impulse Response (FIR) filters which took a
 an input sequence $x[n]$ to give us some desired output sequence $y[n]$.  We saw that these FIR filters basically applied a weighted average across the input sequence and that this could be characterised in terms of convolution.
Now, we'll look at Infinite Impulse Response (IIR) filters, which allow us to model more complex feedback behaviour that's closer to what we might expect from the vocal tract.

## 3.1 Definition of an Infinite Impulse Response filter

Unlike FIR filters, IIR filters use outputs from previous time steps to calculate the next output.

$$ y[n] = \sum_{k=0}^N b[k] x[n-k] + \sum_{k=0}^M a[k] y[n-k]$$

The 'infinite' in the name comes from the fact that you can define a filter in this way so that a single impulse results in an infinitely long non-zero output sequence.

For example

$$ y[n] =  x[n] + y[n-1] $$

if $x[n] = 1$ for $n=1$ and 0 otherwise, then

* $ y[1] = 1 + 0  = 1$
* $ y[2] = 0 + 1 = 1$
* $ y[3] = 0 + 1 = 1$

i.e. $y[n] = 1$ for all $n$ to infinity!

These can be much more efficient than using an FIR filter: we can get the same effect with fewer parameters. (e.g. in some cases you can define a 4th order FIR filter that does the same job as a 19th order FIR filter). The flip side is that they're harder to analyze.

Unfortunately, there's no direct way to compute the a(k)'s and b(k)'s from the impulse response for an IIR filter, but we can use the **z Transform** to do this (we won't go into it in this class).  Instead we'll focus on how we can use an IIR filter to create a speech shaped sound.

## 3.2 Using an IIR filter to shape the spectral envelope

### Let's start with an impulse train

```python
N=400
f_s = 8000
t_s = 1/f_s

## Set the impulse train frequency in Hz
frequency = 100

## Generate N samples of an impulse train with frequency=frequency given a sampling rate of f_s
x_imp, time_steps = make_impulse_train(sample_rate=f_s, frequency=frequency, n_samples=N)

## plot and label
fig, ax = plt.subplots(figsize=(16, 4))
ax.set(ylim=(-1, 2))
ax.scatter(time_steps, x_imp)
ax.plot(time_steps, x_imp)
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("An impulse train")
```

### Now let's define a function to calculate the output of an IIR filter given feedback coefficients `a` and feedforward coefficients `b`.

```python
# iir_filter applies the IIR filter defined by feedforward coefficients, b, and feedback coefficients, a,
# where the coefficients are input as ordered lists
def iir_filter(x, a=[], b=[]):

    ## Get the size of the input
    ## Note the size of N will depend on how we deal with edge cases

    N = len(x)

    ## Initialize our output array, y, to all zeros
    y = np.zeros(N)

    ## step through each input index n
    for n in range(N):
        y[n] = 0

        ## get the contribution from the inputs
        for k in range(len(b)):
            if n-k >= 0:
                ## multiply the previous inputs with filter coefficients, b, and add them up
                ## just like an FIR filter
                y[n] = y[n] + (b[k]*x[n-k])

        ## get contribution from the existing outputs
        for l in range(len(a)):
            ## Note: We don't want to include the current output in this calculation!

            if n-l > 0:
                ## multiply the previous outputs with filter coefficients, a, and add them up
                y[n] = y[n] + (a[l]*y[n-l-1])

    return y

```

### Apply the IIR filter to our impulse train

```python
## An impulse train with amplified impulse to make things easier to see.
## It doesn't change the shape of the frequency response
x = x_imp * 10

## Note coefficients opposite sign to what Simon has in his slides.  Maybe a matlab thing?
## Or a notation thing to make transfer function look nicer?
a = [3.22666099, -4.3967485, 3.03596532, -0.88529281]
b = [1]

## You might like to try this Low pass filter from Lyon's DSP textbook
#a=[1.194, -0.436]
#b=[0.0605, 0.121, 0.0605]

y_iir = iir_filter(x, a=a, b=b)

```

### Plot the filtered output in the time domain

```python
## Plot the filtered output
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(range(N), y_iir)
print(N, np.max(y_iir))
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("Impulse train transformed by an IIR filter")

## Starting to look more speech...ish
```

Now we're starting to make something that looks like a speech waveform!

### Now let's look at the DFT of the filter's output

```python
## Get the DFT outputs of the IIR filter's output
mags, phases = get_dft_mag_phase(y_iir, N)

## Plot the DFT frequencies rather than DFT output indices
dft_freqs = (f_s/N) * np.arange(N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])
ax.set_xlabel("Frequency (Hz))")
ax.set_ylabel("Magnitude")
ax.set_title("Frequency response of IIR filter output (sample rate: %d samples/sec)" % f_s)

```

### Exercise

* What happens to the filtered signal if we alter the filter coefficients:
    * Try out the filter coefficients in the cell below and describe what sort of effect they have.
    * What sort of frequencies do they boost and which do they dampen?
    * Do any look like low-pass or high-pass filters, for example?
    * Do all of them produce outputs that look like vocal pulses?

* If you play around the coefficients it's quite easy to end up with an output that doesn't 'fade out' over time.  What happens to the DFT in this case?

```python
## Try out these different sets of IIR filter coefficients
## ex 1
a = [0.8]
b = [1]

## ex 2
a = [-0.8]
b = [1,-1]

## ex 3:
a = [0.566, -0.16017]
b = [1, 1.414, 1]

## ex 4:
a = [0.566, -0.16017]
b = [1, 0.414, -0.414, -1]

## ex 5:
a = [1.272, -0.80899]
b = [1, 0.414, -0.414, -1]

## ex 6: unstable (unbounded) but still gives us a frequency response like the one previous
a = [1.554, -1.207]
b = [1, 0.414, -0.414, -1]

```

```python
## set which coeffients you want to use here.
## e.g. ex 1
a = [0.8]
b = [1]

## the impulse train
x = x_imp * 10
y_iir = iir_filter(x, a=a, b=b)

## Plot the filtered output
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(range(N), y_iir)
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("Impulse train transformed by an IIR filter: a=%s, b=%s" % (repr(a), repr(b)))

#print(N, np.max(y_iir))

## Plot the DFT frequencies rather than DFT output indices
mags, phases = get_dft_mag_phase(y_iir, N)
dft_freqs = (f_s/N) * np.arange(N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
#ax.scatter(dft_freqs[0:int(N/2)],mags[0:int(N/2)])
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])
ax.set_xlabel("Frequency (Hz))")
ax.set_ylabel("Magnitude")
ax.set_title("Frequency response of IIR filter output: a=%s, b=%s" % (repr(a), repr(b)))

```

### Notes

```python

```

---

## 3.3 (Optional Extras) Designing a filter using poles and zeros

This section is very much optional extension. The following gives a very, very terse overview of how you can design an IIR filter frequency response using the concept of poles and filters. This requires a fair grip on solving polynomials and complex numbers, but once you've got that it's actually quite nice!

If you're interested in learning more about this (or signal processing in general), it's probably time to pick up a Digital Signal Processing textbook. Rick Lyon's [Understanding Digital Signal Processing](https://www.dsprelated.com/books/6.php) is probably the most accessible I've come across. The example filters above were taken from that book. The end of this section basically gives a excessively detailed derivation going from poles and filters to coefficients for those examples.

Previous SLP students have also recommended [The Scientist and Engineer's Guide to
Digital Signal Processing](http://www.dspguide.com/pdfbook.htm), though the free online version renders a bit weirdly!.

IIR filters have the following form:

$$ y[n] = \sum_{k=0}^N b(k) x[n-k] + \sum_{k=0}^M a(k) y[n-k]$$

Unlike an FIR filter, we can't just put an impulse in and find out what the frequency response of the filter is.  Instead, to understand the properties of an IIR filter, we use the **z-transform**.  The z-transform is the discrete version of the continuous Laplace transform.  We won't get into the details here, but if you do read more about this you'll see that you can think of the Laplace transform as a generalization of the Fourier Transform. So the intuitions you've developed about how the Fourier Transform works (e.g. finding correlations between inputs and basis functions) will help understand the Laplace Transform too.

We use the z-transform to convert filter difference equations to an equivalent form in the z-domain (z being a complex number).  We rewrite our difference equation in terms of the z-transformed input $x[n]$ and output $y[n]$ functions, which we denote as $X(z)$ and $Y(z)$ respectively.  Some algebra then allows us to define a **transfer function** $H(z)$:

$$ Y(z) = H(z)X(z)$$

So, $H(z)$ is what our filter does to our input to get our output in the z-domain. You can think of it as the filter part of the source filter model (the source being our impulse train and/or white noise).

Unsurprisingly, we can actually define $H(z)$ in terms of our the feedforward coefficients $b(k)$ and the feedback coefficients $a(k)$. What's surprising is that we can define it in a relatively simple way:

$$ H(z) = \frac{Y(z)}{X(z)} = \frac{\sum_{k=0}^N b(k)z^{-k}}{1- \sum_{k=1}^M a(k)z^{-k}}\\ $$

You can think of  $z^{-k}$ as referring to a position in the sequence $k$ steps back in time.

We can analyze $H(z)$ to figure out how the filter is going to behave. To do this we need to factorize the numerator and denominator polynomials of $H(z)$.

$$
\begin{align}
H(z) &= \frac{\sum_{k=0}^N b(k)z^{-k}}{1- \sum_{k=1}^M a(k)z^{-k}}\\
&= G\frac{(z-z_0)(z-z_1)...(z-z_N)}{(z-p_0)(z-p_1)...(z-p_M)}
\end{align}
$$

In the equation above, $G$ is an arbitrary gain constant which doesn't affect the frequency response or filter stability. The polynomial roots, $z_n$ and $p_m$ are called *zeros* and *poles* respectively.  These are the (complex) values of $z$ which cause the numerator (zeros) or the denominator (poles) to equal 0.  We can interpret them as follows:

* **Zeros:** when the numerator goes to zero, the transfer function $H(z)$ goes to zero, and we see decrease (attenuation) in the frequency magnitude response.
* **Poles:** when the denominator goes to zero, the transfer function $H(z)$ goes to infinity, leads to an increase (gain) the frequency magnitude response (like vowel formants!)

It may not look like it right now, but it's way easier to analyze IIR filters in the z-domain than in the time domain! Once we know the poles and the zeros of the transfer function, we basically know what the frequency response will look like and whether the filter is stable or not.  The main things to note are:

* If a pole falls inside the unit-circle (of the complex z-plane), the filter output will be **stable**.
    * Given an impulse input, $y[n]$ it will eventually go to zero as $n$ increases.
* If a pole falls outside the unit circle, the output will be **unstable**
    * Given an impulse input, $y[n]$ will keep increasing in magnitude as more time passes (definitely not the behaviour of the vocal tract!).
* If a pole lies exactly on the unit circle, the output will oscillate between finite values determined by the pole locations forever.

An IIR filter is only as stable as it's weakest link - if any poles sit outside the unit circle, the filter as a whole will be unstable.  Zeros don't affect filter stability.

Another property of poles (and zeros) is that, if the denominator polynomial has real-valued coefficients (i.e. the $a(k)$ values), poles (and zeros) either come in pairs of complex conjugates (i.e. if $p=a+ib$ is a pole, so is $p^*=a-ib$), or it's a value on the real axis of the z-plane.  We can use these facts to design filters using the factorized form of the transfer function, which will then allow us to derived the filter coefficients $a(k)$ and $b(k)$. We'll see some examples this below.

The frequency response of the filter is $H(z)$ evaluated at $e^{j\omega}$, where $-\pi \lt \omega \lt \pi$ represents frequencies in terms of radians per sample ($\pi$ radians/sample $\mapsto$ a frequency matching half the sampling frequency (i.e. $f_s/2$).

$$ H(\omega) = H(e^{j\omega}) = \frac{\sum_{k=0}^N b(k)e^{-jk\omega}}{1 -\sum_{k=0}^M a(k)e^{-jk\omega}} $$

These example IIR filters from Richard J Lyon's book
[Understanding Digital Signal Processing](https://www.dsprelated.com/books/6.php), which is a very nice introduction to the topic.

Working through from the factorized form to the coefficients is not hard, but it is very annoying and it's very easy to make mistakes, so I've given all the steps expanding out the factorized forms of the filter examples he gives in his book below.

### Ex 1:

$$
\begin{aligned}
H_1(z) &= \frac{G}{z-0.8}\\
       &= G\frac{1}{1-(0.8z^{-1})}
\end{aligned}
$$

* One pole on the real axis, inside the unit circle, stable peak at $\omega=0$ rad/sample
* no zeros

```python
a = [0.8]
b = [1]
```

### Ex 2:

$$
\begin{aligned}
H_1(z) &= G\frac{(z-1)}{z-(-0.8)}\\
       &= G\frac{1 - z^{-1}}{1 -  (-0.8z^{-1})}
\end{aligned}
$$

* pole on real axis at -0.8, inside the unit circle, stable peak at $\omega=-\pi$ rad/sample, i.e. $f_s/2$ Hz which is 8000/2 = 4000 Hz in our setup above.
* one zero at z=1, this causes attenuation at $\omega = 0$ rads/sample (0 Hz)

```python
a = [-0.8]
b = [1,-1]
```

### Ex 3
$$
\begin{align}
H_3(z) &= G\frac{[z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.283+j0.283)][z-(0.283-j0.283)]} \\
&= G\frac{z^2 -z(-0.707 - j0.707) - z(-0.707 + j0.707) + (-0.707 + j0.707)(-0.707 - j0.707) }{z^2 - z(0.283-j0.283) -z(0.283+j0.283) + (0.283+j0.283)0.283-j0.283)} \\
&= G\frac{z^2 +z0.707 + j0.707z + z0.707 - j0.707z + 1 }{z^2 - 0.283z + j0.283z - 0.283z - j0.283z + 0.16017} \\
&= G\frac{z^2 +z[0.707  + 0.707] + 1 }{z^2 - 0.283z - 0.283z  + 0.16017} \\
&= G\frac{z^2 + 1.414z + 1 }{z^2 - 0.566z + 0.16017} \\
&= G\frac{1 + 1.414z^{-1} + 1z^{-2} }{1 - 0.566z^{-1} + 0.16017z^{-2}} \\
\end{align}
$$

* The poles are inside the unit circle, this produces a low pass filter  (pass band is 'bordered' by +/- $\omega = \pi/4$ rad/sample, 1000 Hz)
    * $(0.283+j0.283) \mapsto 0.400e^{j\pi/4}$
* zeros on the unit circle, attenuation at $3\pi/4$ rad/sample = 3000 Hz.
    * $(-0.707+j0.707) \mapsto 1e^{j3\pi/4} $

```python
a = [0.566, -0.16017]
b = [1, 1.414, 1]

```

### Ex 4:
$$
\begin{align}
H_4(z) &= G\frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.283+j0.283)][z-(0.283-j0.283)]} \\
&= G\frac{[z-1] [z^2 + 1.414z + 1] }{z^2 - 0.566z + 0.16017}\\
&= G\frac{z^3 + 1.414z^2 + 1z -z^2 - 1.414z - 1}{z^2 - 0.566z + 0.16017}\\
&= G\frac{z^3 + 0.414z^2 - 0.414z - 1}{z^2 - 0.566z + 0.16017}\\
&= G\frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 0.566z^{-1} + 0.16017z^{-2}}\\
\end{align}
$$

* We've added an extra zero at $z=0$ compared to example 3.  This means that we get a valley (attenuation) in the frequency response at $\omega = 0$ (0 Hz).

```python
a = [0.566, -0.16017]
b = [1, 0.414, -0.414, -1]
```

### Ex 5:
$$
\begin{align}
H_5(z) &= G\frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.636+j0.636)][z-(0.636-j636)]} \\
&= G\frac{[z-1] [z^2 + 1.414z + 1] }{z^2 - 1.272z + 0.808992}\\
&= G\frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 1.272z^{-1} +  0.808992z^{-2}}\\
\end{align}
$$

* poles are nearer to the unit circle (magnitude 0.80899), so the peaks associated with them (+/- $\omega = \pi/4$ rad/sec (1000 Hz with our sample rate) are sharper.

```python
a = [1.272, -0.80899]
b = [1, 0.414, -0.414, -1]
```

### Ex 6:
$$
\begin{align}
H_6(z) &= G \frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.777+j0.777)][z-(0.777-777)]} \\
&= G \frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 1.554z^{-1} +  1.207z^{-2}}\\
\end{align}
$$

* poles are now outside the unit circle (magnitude 1.207), so the filter is now unstable: as n increases, the value of $y[n]$ eventually goes to infinity.
* It still has the same shape frequency response though!

```python
a = [1.554, -1.207]
b = [1, 0.414, -0.414, -1]
```


# Cariberation (TOP LEFT -> Bottom Right)

```python
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands

# text cv2 puttext

font                   = cv2.FONT_HERSHEY_SIMPLEX
location = (100,50)
fontScale              = 1
fontColor              = (255,255,255)
lineType               = 2
cali=[]

# For webcam input:
hands = mp_hands.Hands(
    min_detection_confidence=0.75, min_tracking_confidence=0.75)
cap = cv2.VideoCapture(0)
countdown=100
while cap.isOpened():
    success, image = cap.read()
    if not success:
        break

  # Flip the image horizontally for a later selfie-view display, and convert
  # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
  # To improve performance, optionally mark the image as not writeable to
  # pass by reference.
    image.flags.writeable = False
    results = hands.process(image)

  # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
        cv2.putText(image,str(countdown/5), location, font, fontScale,fontColor,lineType)
        for hand_landmarks in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
            if countdown==0:
                land = hand_landmarks
                val = land.landmark
                cali.append([val[8].x,val[8].y])
                countdown=100
    else:
        cv2.putText(image,'No Hands!', location, font, fontScale,fontColor,lineType)
    cv2.imshow('MediaPipe Hands', image)
    key = cv2.waitKey(1)
    countdown-=1
    if key == 27:
        break
    if len(cali)==2:
        break
hands.close()
cap.release()
cv2.destroyAllWindows()
```

# Calculations

```python
print(cali)
from sympy import symbols, Eq, solve

x, y = symbols('x,y')

# defining equations
eq1 = Eq((x*cali[0][0]+y), 0)
eq2 = Eq((x*cali[1][0]+y), 1920)

e1=solve((eq1, eq2), (x, y))
l1=list(e1.values())

x, y = symbols('x,y')

# defining equations
eq1 = Eq((x*cali[0][1]+y), 0)
eq2 = Eq((x*cali[1][1]+y), 1080)
e2=solve((eq1, eq2), (x, y))
l2=list(e2.values())

print(l1)
print(l2)
```

    [[0.557029664516449, 0.21512927114963531], [0.7416685819625854, 0.47747597098350525]]
    [10398.6744861636, -5792.37016044346]
    [4116.68986377151, -885.620489942256]

# Cursur Movement (No click)

```python
import pyautogui
pyautogui.FAILSAFE= False
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands

# text cv2 puttext

font                   = cv2.FONT_HERSHEY_SIMPLEX
location = (100,50)
fontScale              = 1
fontColor              = (255,255,255)
lineType               = 2

# For webcam input:
hands = mp_hands.Hands(
    min_detection_confidence=0.75, min_tracking_confidence=0.75)
cap = cv2.VideoCapture(0)
while cap.isOpened():
    success, image = cap.read()
    if not success:
        break

  # Flip the image horizontally for a later selfie-view display, and convert
  # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
  # To improve performance, optionally mark the image as not writeable to
  # pass by reference.
    image.flags.writeable = False
    results = hands.process(image)

  # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
            land = hand_landmarks
            val = land.landmark
            x=val[8].x*l1[0]+l1[1]
            if(x<0):
                x=0
            if(x>1920):
                x=1920
            y=val[8].y*l2[0]+l2[1]
            if(y<0):
                y=0
            if(y>1080):
                y=1080
            pyautogui.moveTo(x, y)
    else:
        cv2.putText(image,'No Hands!', location, font, fontScale,fontColor,lineType)
    cv2.imshow('MediaPipe Hands', image)
    key = cv2.waitKey(1)
    if key == 27:
        break
hands.close()
cap.release()
cv2.destroyAllWindows()
```

```python
import pyautogui
print(pyautogui.size())
```

    Size(width=1920, height=1080)

# Mouse

```python
from keras.models import load_model
import numpy as np
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
model=load_model("curmodel")
import pyautogui
pyautogui.FAILSAFE= False
```

```python
Char="0123456789DOU$"

font        = cv2.FONT_HERSHEY_SIMPLEX
location    = (100,50)
fontScale   = 1
fontColor   = (255,255,255)
lineType    = 2

hands = mp_hands.Hands(
    min_detection_confidence=0.5, min_tracking_confidence=0.5)
cap = cv2.VideoCapture(0)
ccount=20
while cap.isOpened():
    success, image = cap.read()
    if not success:
        break

    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    image.flags.writeable = False
    results = hands.process(image)
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
        L=[]
        for hand_landmarks in results.multi_hand_landmarks:
            mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
            land = hand_landmarks
            val = land.landmark
            first=val[0];
            for j in val:
                L.append(j.x-first.x);
                L.append(j.y-first.y);
                L.append(j.z-first.z);
            break
        L=L[3::]
        L=np.array(L)
        x = np.expand_dims(L,0)
        x = np.expand_dims(x,-1)
        fi=int(np.argmax(model.predict(x)))
        if fi==1:
                for hand_landmarks in results.multi_hand_landmarks:
                    land = hand_landmarks
                    val = land.landmark
                    x=val[8].x*l1[0]+l1[1]
                    if(x<0):
                        x=0
                    if(x>1920):
                        x=1920
                    y=val[8].y*l2[0]+l2[1]
                    if(y<0):
                        y=0
                    if(y>1080):
                        y=1080
                    pyautogui.moveTo(x, y)
        if fi==6:
            ccount-=1
            for hand_landmarks in results.multi_hand_landmarks:
                land = hand_landmarks
                val = land.landmark
                x=val[8].x*l1[0]+l1[1]
                if(x<0):
                    x=0
                if(x>1920):
                    x=1920
                y=val[8].y*l2[0]+l2[1]
                if(y<0):
                    y=0
                if(y>1080):
                    y=1080
                if(ccount==0):
                    ccount=20
                    print("Single click")
                    pyautogui.click(x, y)
        if fi==2:
            ccount-=1
            for hand_landmarks in results.multi_hand_landmarks:
                land = hand_landmarks
                val = land.landmark
                x=val[8].x*l1[0]+l1[1]
                if(x<0):
                    x=0
                if(x>1920):
                    x=1920
                y=val[8].y*l2[0]+l2[1]
                if(y<0):
                    y=0
                if(y>1080):
                    y=1080
                if(ccount==0):
                    ccount=20
                    print("Double clicked")
                    pyautogui.click(x, y,clicks=2)
    else:
        cv2.putText(image,'No Hands!', location, font, fontScale,fontColor,lineType)

    cv2.imshow('MediaPipe Hands', image)
    key = cv2.waitKey(1)
    if key == 27:
        break
hands.close()
cap.release()
cv2.destroyAllWindows()
```

    Single click
    Double clicked
    Single click
    Single click
    Single click
    Single click
    Single click

```python

```


# RoadMap 11 - Torch NN Layers - Loss Functions

    1.   torch.nn.L1Loss                     - L1 Loss Function
    2.   torch.nn.MSELoss                    - Mean Square Error Loss Function
    3.   torch.nn.CrossEntropyLoss           - Cross entropy loss function
    4.   torch.nn.NLLLoss                    - Negative Log Likelihood loss function
    5.   torch.nn.PoissonNLLLoss             - Poisson Negative Log Likelihood loss function
    6.   torch.nn.nn.KLDivLoss               - Kullback Leiber Divergence loss function
    7.   torch.nn.BCELoss                    - Binary cross entropy loss function
    8.   torch.BCEWithLogitsLoss             - Binary cross entropy loss with logits function
    9.   torch.nn.MarginRankingLoss          - Margin ranking loss function
    10.  torch.nn.HingeEmbeddingLoss         - Hinge Embedding Loss function
    11.  torch.nn.MultiLabelMarginLoss       - Multi Label Margin loss function
    12.  torch.nn.SmoothL1Loss               - Smooth L1 Loss function
    13.  torch.nn.MultiLabelSoftMarginLoss   - Multi Label Soft Margin Loss function
    14.  torch.nn.CosineEmbeddingLoss        - Cosine Embedding loss function
    15.  torch.nn.MultiMarginLoss            - Multi Margin loss function
    16.  torch.nn.TripletMarginLoss          - Triplet Margin Loss Function

```python
import os
import sys
import torch
import numpy as np
```

```python
import torch.nn as nn
from torchvision import transforms, datasets
from PIL import Image
import cv2
import matplotlib.pyplot as plt
import torchvision

# FUNCTIONAL modules - Implementing each module as functions
import torch.nn.functional as F
```

```python

```

```python

```

```python

```

## Extra Blog Layers

1. https://medium.com/udacity-pytorch-challengers/a-brief-overview-of-loss-functions-in-pytorch-c0ddb78068f7

2. https://machinelearningmastery.com/loss-and-loss-functions-for-training-deep-learning-neural-networks/

3. https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/

```python

```

```python

```

```python

```

```python
# Input
image_name = "dog.jpg"
image_pil = Image.open(image_name)

transform = transforms.Compose([transforms.ToTensor()])
image = transform(image_pil).float()
image_nchw = torch.unsqueeze(image, 0)
print(image_nchw.size())
plt.imshow(image_pil)
```

```python

```

### L1 loss Function

1. nn.L1Loss - Creates a criterion that measures the mean absolute value of the element-wise difference between input x and target y
            - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
            - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
            - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function

$$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad
l_n = \left| x_n - y_n \right|$$

```python
print("Module implementation")
loss = nn.L1Loss()
input_data = torch.randn(10, requires_grad=True)
target_data = torch.randn(10)
output_loss = loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.l1_loss(input_data, target_data, reduction='elementwise_mean')
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([ 1.6689,  0.9617, -0.8708,  1.7215, -1.7089, -1.4635,  0.6810,  0.9599,
             0.8776, -0.3141], requires_grad=True)
    Output Loss =  tensor(1.4486, grad_fn=<L1LossBackward>)

    Functional implementation
    Output Loss =  tensor(1.4486, grad_fn=<L1LossBackward>)

    /home/abhi/.virtualenvs/study_pytorch_test/lib/python3.6/site-packages/torch/nn/_reduction.py:13: UserWarning: reduction='elementwise_mean' is deprecated, please use reduction='mean' instead.
      warnings.warn("reduction='elementwise_mean' is deprecated, please use reduction='mean' instead.")

```python

```

### Mean Square Error Loss

2. nn.MSELoss - Creates a criterion that measures the mean squared error between n elements in the input x and target y.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:
$$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad
l_n = \left( x_n - y_n \right)^2$$

```python
print("Module implementation")
loss = nn.MSELoss()
input_data = torch.randn(10, requires_grad=True)
target_data = torch.randn(10)
output_loss = loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.mse_loss(input_data, target_data, reduction='elementwise_mean')
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([-0.6542, -1.0031, -0.0445, -0.2803, -0.3421, -1.1031,  0.9784,  0.1492,
             0.8774, -2.4233], requires_grad=True)
    Output Loss =  tensor(1.7418, grad_fn=<MseLossBackward>)

    Functional implementation
    Output Loss =  tensor(1.7418, grad_fn=<MseLossBackward>)

```python

```

### Cross Entropy Loss

3. nn.CrossEntropyLoss - This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class. It is useful when training a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.
    - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C
    - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
    - ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets.
    - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
    - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

      Function:

$$ \text{loss}(x, class) = weight[class] \left(-x[class] + \log\left(\sum_j \exp(x[j])\right)\right) $$

```python
print("Module Implementation")
loss = nn.CrossEntropyLoss()
softmax = nn.Softmax()
input_data = torch.randn(3, 4, requires_grad=True)
input_data = softmax(input_data)
target_data = torch.tensor([0, 1, 2])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.cross_entropy(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")

```

    Module Implementation
    Input Data =  tensor([[0.0514, 0.0519, 0.6677, 0.2290],
            [0.1202, 0.2288, 0.3351, 0.3158],
            [0.1132, 0.3875, 0.4544, 0.0449]], grad_fn=<SoftmaxBackward>)

    Target Data =  tensor([0, 1, 2])

    Output Loss =  tensor(1.4089, grad_fn=<NllLossBackward>)

    Functional implementation
    Output Loss =  tensor(1.4089, grad_fn=<NllLossBackward>)

    /home/abhi/.virtualenvs/study_pytorch_test/lib/python3.6/site-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
      """

```python

```

### Negative log likelihood loss

4. nn.NLLLoss - The negative log likelihood loss. It is useful to train a classification problem with C classes. If provided, the optional argument weight should be a 1D Tensor assigning weight to each of the classes. This is particularly useful when you have an unbalanced training set.

    - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
    - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
    - ignore_index (int, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. When size_average is True, the loss is averaged over non-ignored targets.
    - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
    - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\begin{split}\ell(x, y) = \begin{cases}
    \sum_{n=1}^N \frac{1}{\sum_{n=1}^N w_{y_n}} l_n, & \text{if}\;
    \text{size_average} = \text{True},\\
    \sum_{n=1}^N l_n,  & \text{if}\;
    \text{size_average} = \text{False}.
\end{cases}\end{split}
$$

```python
print("Module Implementation")
m = nn.LogSoftmax()
loss = nn.NLLLoss()
# input is of size N x C = 3 x 5
input_data = torch.randn(3, 10, requires_grad=True)
input_data = m(input_data)
# each element in target has to have 0 <= value < C
target_data = torch.tensor([1, 0, 4])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.nll_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module Implementation
    Input Data =  tensor([[-1.5806, -1.6066, -3.2434, -3.1549, -2.9530, -2.7948, -1.4620, -2.3547,
             -3.3739, -3.2791],
            [-1.8911, -4.4236, -2.4025, -2.9850, -2.2202, -3.2712, -2.4602, -2.8455,
             -2.5018, -1.1268],
            [-2.0087, -4.0166, -2.9455, -2.8539, -3.3092, -2.3496, -2.0006, -1.1939,
             -3.0565, -2.1174]], grad_fn=<LogSoftmaxBackward>)

    Target Data =  tensor([1, 0, 4])

    Output Loss =  tensor(2.2690, grad_fn=<NllLossBackward>)

    Functional implementation
    Output Loss =  tensor(2.2690, grad_fn=<NllLossBackward>)

    /home/abhi/.virtualenvs/study_pytorch_test/lib/python3.6/site-packages/ipykernel_launcher.py:6: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.

```python

```

### Poisson Negative log likelihood loss

5. nn.PoissonNLLLoss - Negative log likelihood loss with Poisson distribution of target.
        - log_input (bool, optional) – if True the loss is computed as exp(input)−target∗input, if False the loss is input−target∗log(input+eps)
        - full (bool, optional) – whether to compute full loss, i. e. to add the Stirling approximation term target∗log(target)−target+0.5∗log(2πtarget).
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - eps (float, optional) – Small value to avoid evaluation of log(0), when log_input == False. Default: 1e-8
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\begin{align}\begin{aligned}\text{target} \sim \mathrm{Poisson}(\text{input})\\\text{loss}(\text{input}, \text{target}) = \text{input} - \text{target} * \log(\text{input})
                            + \log(\text{target!})\end{aligned}\end{align}$$

```python
print("Module Implementation")
loss = nn.PoissonNLLLoss()
input_data = torch.randn(5, 2, requires_grad=True)
target_data = torch.randn(5, 2)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.poisson_nll_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module Implementation
    Input Data =  tensor([[-0.3775,  1.5565],
            [-0.2749,  0.6706],
            [ 0.4687,  1.6879],
            [ 0.6063,  0.6137],
            [-0.3487, -0.9277]], requires_grad=True)

    Target Data =  tensor([[ 0.6550,  0.6313],
            [-1.0439,  0.6372],
            [-1.3411, -1.0727],
            [-0.1483,  1.3046],
            [ 1.8552, -1.4957]])

    Output Loss =  tensor(1.9469, grad_fn=<MeanBackward0>)

    Functional implementation
    Output Loss =  tensor(1.9469, grad_fn=<MeanBackward0>)

```python

```

### Kullback Leiber Divergence loss

6. nn.KLDivLoss - KL divergence is a useful distance measure for continuous distributions and is often useful when performing direct regression over the space of (discretely sampled) continuous output distributions.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

```python
print("Module Implementation")
loss = nn.KLDivLoss(size_average=False)
log_softmax = nn.LogSoftmax(1)
softmax = nn.Softmax(1)

batch_size = 2

input_data = torch.randn(batch_size, 3, requires_grad=True)
log_input_data = log_softmax(input_data)

target_data = torch.randn(batch_size, 3, requires_grad=True)
softmax_target_data = softmax(target_data)

output_loss = loss(log_input_data, softmax_target_data) / batch_size
output_loss.backward(retain_graph=True)

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.kl_div(log_input_data, softmax_target_data, size_average=False) / batch_size
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module Implementation
    Input Data =  tensor([[ 1.0913,  0.8709, -0.3859],
            [-0.8672,  0.2453,  0.2501]], requires_grad=True)

    Target Data =  tensor([[ 2.6353e-01, -2.5150e-01,  4.9396e-04],
            [ 4.4976e-01, -5.5277e-01,  5.8091e-01]], requires_grad=True)

    Output Loss =  tensor(0.2249, grad_fn=<DivBackward0>)

    Functional implementation
    Output Loss =  tensor(0.2249, grad_fn=<DivBackward0>)

    /home/abhi/.virtualenvs/study_pytorch_test/lib/python3.6/site-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
      warnings.warn(warning.format(ret))

```python

```

### Binary Cross Entripy Loss

7. nn.BCELoss - Creates a criterion that measures the Binary Cross Entropy between the target and the output
        - weight (Tensor, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size “nbatch”.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

   Function:

$$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad
l_n = - w_n \left[ y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right]$$

```python
print("Module Implementation")
loss = nn.BCELoss()
sigmoid = nn.Sigmoid()

input_data = torch.randn(5, requires_grad=True)
sigmoid_input_data = sigmoid(input_data)

target_data = torch.empty(5).random_(2) # Binary Target

output_loss = loss(sigmoid_input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.binary_cross_entropy(sigmoid_input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")

```

    Module Implementation
    Input Data =  tensor([-1.4087, -1.0918,  0.0857,  0.7139,  0.2992], requires_grad=True)

    Target Data =  tensor([0., 1., 1., 0., 1.])

    Output Loss =  tensor(0.7837, grad_fn=<BinaryCrossEntropyBackward>)

    Functional implementation
    Output Loss =  tensor(0.7837, grad_fn=<BinaryCrossEntropyBackward>)

```python

```

### Binary Cross Entropy with Logits Loss

8. nn.BCEWithLogitsLoss - This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.
        - weight (Tensor, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size “nbatch”.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’
        - pos_weight – a weight of positive examples. Must be a vector with length equal to the number of classes.

    Function

$$\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad
l_n = - w_n \left[ t_n \cdot \log \sigma(x_n)
+ (1 - t_n) \cdot \log (1 - \sigma(x_n)) \right]$$

```python
print("Module implementation")
loss = nn.BCEWithLogitsLoss()
sigmoid = nn.Sigmoid()

input_data = torch.randn(5, requires_grad=True)
sigmoid_input_data = sigmoid(input_data)

target_data = torch.empty(5).random_(2) # Binary Target

output_loss = loss(sigmoid_input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.binary_cross_entropy_with_logits(sigmoid_input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([-0.2327, -1.6082, -0.4721, -1.4554,  0.7109], requires_grad=True)

    Target Data =  tensor([0., 0., 0., 0., 0.])

    Output Loss =  tensor(0.8996, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)

    Functional implementation
    Output Loss =  tensor(0.8996, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)

```python

```

### Margin Ranking Loss

9. nn.MarginRankingLoss - Creates a criterion that measures the loss given inputs x1, x2, two 1D mini-batch Tensor's, and a label 1D mini-batch tensor 'y with values (1 or -1). If y == 1 then it assumed the first input should be ranked higher (have a larger value) than the second input, and vice-versa for y == -1.
        - margin (float, optional) – Has a default value of 0.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\text{loss}(x, y) = \max(0, -y * (x1 - x2) + \text{margin})$$

```python
print("Module implementation")
loss = nn.MarginRankingLoss()
input_data1 = torch.randn(3, requires_grad=True)
input_data2 = torch.randn(3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data1, input_data2, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data1 = ", input_data1)
print("\n")
print("Input Data2 = ", input_data2)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.margin_ranking_loss(input_data1, input_data2, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data1 =  tensor([-0.0064, -0.0055, -0.2032], requires_grad=True)

    Input Data2 =  tensor([-0.1289, -0.2205,  0.9442], requires_grad=True)

    Target Data =  tensor([-1.,  1.,  1.])

    Output Loss =  tensor(0.4233, grad_fn=<MeanBackward0>)

    Functional implementation
    Output Loss =  tensor(0.4233, grad_fn=<MeanBackward0>)

```python

```

### Hinge Embedding Loss

10. nn.HingeEmbeddingLoss - Measures the loss given an input tensor x and a labels tensor y containing values (1 or -1). This is usually used for measuring whether two inputs are similar or dissimilar, e.g. using the L1 pairwise distance as x, and is typically used for learning nonlinear embeddings or semi-supervised learning.
        - margin (float, optional) – Has a default value of 1.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:
$$\begin{split}l_n = \begin{cases}
    x_n, & \text{if}\; y_n = 1,\\
    \max \{0, \Delta - x_n\}, & \text{if}\; y_n = -1,
\end{cases}\end{split}$$

```python
print("Module implementation")
loss = nn.HingeEmbeddingLoss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.hinge_embedding_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([ 0.0038, -1.3820,  0.4059], requires_grad=True)

    Target Data =  tensor([-1.,  1.,  1.])

    Output Loss =  tensor(0.0067, grad_fn=<MeanBackward0>)

    Functional implementation
    Output Loss =  tensor(0.0067, grad_fn=<MeanBackward0>)

```python

```

### Multi Label Margin Loss

11. nn.MultiLabelMarginLoss - Creates a criterion that optimizes a multi-class multi-classification hinge loss (margin-based loss) between input x (a 2D mini-batch Tensor) and output y (which is a 2D Tensor of target class indices). For each sample in the mini-batch
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\text{loss}(x, y) = \sum_{ij}\frac{\max(0, 1 - (x[y[j]] - x[i]))}{\text{x.size}(0)}$$

```python
print("Module implementation")
loss = nn.MultiLabelMarginLoss()
softmax = nn.Softmax()

input_data = torch.randn(3, requires_grad=True)
input_data = softmax(input_data)

target_data = torch.tensor([0, 1, 2])
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multilabel_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([0.4973, 0.4560, 0.0467], grad_fn=<SoftmaxBackward>)

    Target Data =  tensor([0, 1, 2])

    Output Loss =  tensor(0., grad_fn=<MultilabelMarginLossBackward>)

    Functional implementation
    Output Loss =  tensor(0., grad_fn=<MultilabelMarginLossBackward>)

    /home/abhi/.virtualenvs/study_pytorch_test/lib/python3.6/site-packages/ipykernel_launcher.py:6: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.

```python

```

### Smooth L1 Loss

12. nn.SmoothL1Loss - Creates a criterion that uses a squared term if the absolute element-wise error falls below 1 and an L1 term otherwise. It is less sensitive to outliers than the MSELoss and in some cases prevents exploding gradients (e.g. see “Fast R-CNN” paper by Ross Girshick).
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\text{loss}(x, y) = \frac{1}{n} \sum_{i} z_{i}$$

Where, $$z_{i}\text{ is }$$

\begin{split}z_{i} =
\begin{cases}
0.5 (x_i - y_i)^2, & \text{if } |x_i - y_i| < 1 \\
|x_i - y_i| - 0.5, & \text{otherwise }
\end{cases}\end{split}

```python
print("Module implementation")
loss = nn.SmoothL1Loss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.randn(3, requires_grad=False)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph=True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.smooth_l1_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([ 0.4350,  0.6409, -0.5428], requires_grad=True)

    Target Data =  tensor([-0.2078,  1.8307,  1.2295])

    Output Loss =  tensor(0.7229, grad_fn=<SmoothL1LossBackward>)

    Functional implementation
    Output Loss =  tensor(0.7229, grad_fn=<SmoothL1LossBackward>)

```python

```

### Multi Label Soft Margin loss

13. nn.MultiLabelSoftMarginLoss - Creates a criterion that optimizes a multi-label one-versus-all loss based on max-entropy, between input x and target y of size (N, C)
        - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$loss(x, y) = - \sum_i y[i] * \log((1 + \exp(-x[i]))^{-1})
                 + (1-y[i]) * \log\left(\frac{\exp(-x[i])}{(1 + \exp(-x[i]))}\right)
$$

```python
print("Module implementation")
loss = nn.MultiLabelSoftMarginLoss()
input_data = torch.randn((1, 3), requires_grad=True)
target_data = torch.randn((1, 3), requires_grad=True)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multilabel_soft_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Input Data =  tensor([[ 0.3408,  2.0571, -1.4630]], requires_grad=True)

    Target Data =  tensor([[0.6857, 0.0263, 0.2031]], requires_grad=True)

    Output Loss =  tensor(1.0910, grad_fn=<MeanBackward0>)

    Functional implementation
    Output Loss =  tensor(1.0910, grad_fn=<MeanBackward0>)

```python

```

### Cosine Embedding Loss

14. nn.CosineEmbeddingLoss - Creates a criterion that measures the loss given input tensors x1, x2 and a Tensor label y with values 1 or -1. This is used for measuring whether two inputs are similar or dissimilar, using the cosine distance, and is typically used for learning nonlinear embeddings or semi-supervised learning.
        - margin (float, optional) – Should be a number from -1 to 1, 0 to 0.5 is suggested. If margin is missing, the default value is 0.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\begin{split}\text{loss}(x, y) =
\begin{cases}
1 - \cos(x_1, x_2), & \text{if } y == 1 \\
\max(0, \cos(x_1, x_2) - \text{margin}), & \text{if } y == -1
\end{cases}\end{split}
$$

```python
print("Module Implementation")
loss = nn.CosineEmbeddingLoss()

input_data1 = torch.randn(3, 3, requires_grad=True)
input_data2 = torch.randn(3, 3, requires_grad=True)
target_data = torch.tensor([-1., 1., 1.])
output_loss = loss(input_data1, input_data2, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data1 = ", input_data1)
print("\n")
print("Input Data2 = ", input_data2)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.cosine_embedding_loss(input_data1, input_data2, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module Implementation
    Input Data1 =  tensor([[-0.7491, -0.3227,  1.2011],
            [-0.1215, -0.1149, -0.5620],
            [ 1.2559,  0.6572, -1.7037]], requires_grad=True)

    Input Data2 =  tensor([[ 0.9953, -1.0401,  1.2301],
            [-0.6587, -0.4275, -1.5131],
            [ 1.1152, -1.9418, -1.4029]], requires_grad=True)

    Target Data =  tensor([-1.,  1.,  1.])

    Output Loss =  tensor(0.3263, grad_fn=<MeanBackward0>)

    Functional implementation
    Output Loss =  tensor(0.3263, grad_fn=<MeanBackward0>)

```python

```

### Multi margin loss

15. nn.MultiMarginLoss - Creates a criterion that optimizes a multi-class classification hinge loss (margin-based loss) between input x (a 2D mini-batch Tensor) and output y (which is a 1D tensor of target class indices, 0≤y≤x.size(1))
        - p (int, optional) – Has a default value of 1. 1 and 2 are the only supported values
        - margin (float, optional) – Has a default value of 1.
        - weight (Tensor, optional) – a manual rescaling weight given to each class. If given, it has to be a Tensor of size C. Otherwise, it is treated as if having all ones.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$\text{loss}(x, y) = \frac{\sum_i \max(0, \text{margin} - x[y] + x[i]))^p}{\text{x.size}(0)}$$

```python
print("Model Implementation")
loss = nn.MultiMarginLoss()
input_data = torch.randn(3, requires_grad=True)
target_data = torch.tensor([1, 2, 3], requires_grad=False)
output_loss = loss(input_data, target_data)
output_loss.backward(retain_graph = True)    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.

print("Input Data = ", input_data)
print("\n")
print("Target Data = ", target_data)
print("\n")
print("Output Loss = ", output_loss)
print("\n")
print("\n")

print("Functional implementation")
output_loss = F.multi_margin_loss(input_data, target_data)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Model Implementation
    Input Data =  tensor([-0.3182, -0.4905, -0.2097], requires_grad=True)

    Target Data =  tensor([1, 2, 3])

    Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)

    Functional implementation
    Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)

```python

```

### Triplet Margin Loss

16. nn.TripletMarginLoss - Creates a criterion that measures the triplet loss given an input tensors x1, x2, x3 and a margin with a value greater than 0. This is used for measuring a relative similarity between samples.
        - margin (float, optional) – Default: 1.
        - p (int, optional) – The norm degree for pairwise distance. Default: 2.
        - swap (float, optional) – The distance swap is described in detail in the paper Learning shallow convolutional feature descriptors with triplet losses by V. Balntas, E. Riba et al. Default: False.
        - size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True
        - reduce (bool, optional) – Deprecated (see reduction). By default, the losses are averaged or summed over observations for each minibatch depending on size_average. When reduce is False, returns a loss per batch element instead and ignores size_average. Default: True
        - reduction (string, optional) – Specifies the reduction to apply to the output: ‘none’ | ‘elementwise_mean’ | ‘sum’. ‘none’: no reduction will be applied, ‘elementwise_mean’: the sum of the output will be divided by the number of elements in the output, ‘sum’: the output will be summed. Note: size_average and reduce are in the process of being deprecated, and in the meantime, specifying either of those two args will override reduction. Default: ‘elementwise_mean’

    Function:

$$L(a, p, n) = \max \{d(a_i, p_i) - d(a_i, n_i) + {\rm margin}, 0\}
\text{where,}
d(x_i, y_i) = \left\lVert {\bf x}_i - {\bf y}_i \right\rVert_p$$

```python
print("Module implementation")
triplet_loss = nn.TripletMarginLoss(margin=1.0, p=2)
input_data1 = torch.randn(100, 128, requires_grad=True)
input_data2 = torch.randn(100, 128, requires_grad=True)
input_data3 = torch.randn(100, 128, requires_grad=True)
output = triplet_loss(input_data1, input_data2, input_data3)
output.backward(retain_graph = True)

print("Output Loss = ", output_loss)
print("\n")

print("Functional implementation")
output_loss = F.triplet_margin_loss(input_data1, input_data2, input_data3, margin=1.0, p=2)
output_loss.backward()    # loss.backward() computes dloss/dx for every parameter x which has requires_grad=True.
print("Output Loss = ", output_loss)
print("\n")
```

    Module implementation
    Output Loss =  tensor(0.8177, grad_fn=<MultiMarginLossBackward>)

    Functional implementation
    Output Loss =  tensor(1.0206, grad_fn=<MeanBackward0>)

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## Author - Tessellate Imaging - https://www.tessellateimaging.com/

## Monk Library - https://github.com/Tessellate-Imaging/monk_v1

    Monk is an opensource low-code tool for computer vision and deep learning

### Monk features
- low-code
- unified wrapper over major deep learning framework - keras, pytorch, gluoncv
- syntax invariant wrapper

### Enables
- to create, manage and version control deep learning experiments
- to compare experiments across training metrics
- to quickly find best hyper-parameters

### At present it only supports transfer learning, but we are working each day to incorporate
- GUI based custom model creation
- various object detection and segmentation algorithms
- deployment pipelines to cloud and local platforms
- acceleration libraries such as TensorRT
- preprocessing and post processing libraries

## To contribute to Monk AI or Pytorch RoadMap repository raise an issue in the git-repo or dm us on linkedin
 - Abhishek - https://www.linkedin.com/in/abhishek-kumar-annamraju/
 - Akash - https://www.linkedin.com/in/akashdeepsingh01/

```python

```

K) RoadMap 11 - Torch NN 5 - Loss Functions.ipynb