# FastEMRIWaveforms Tutorial
## ICERM Workshop: Waveform acceleration with machine learning and GPUs

### Michael Katz, Max Planck Institute for Gravitational Physics (Albert Einstein Institute), Lead developer for FastEMRIWaveforms

In this tutorial, you will learn the basics of building an accelereated EMRI waveform. We encourage participants to see our paper ([arxiv.org/2008.06071](https://arxiv.org/abs/2008.06071)) and the FastEMRIWaveforms [package documentation](https://bhptoolkit.org/FastEMRIWaveforms/) for more information, as well as our forthcoming paper that will describe the waveform in much more detail than our PRL submission.

Import packages:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np

import h5py

from few.amplitude import romannet
from few.utils.utility import check_for_file_download, p_to_y
from few.trajectory.flux import RunSchwarzEccFluxInspiral
from few.amplitude.romannet import RomanAmplitude
from few.amplitude.interp2dcubicspline import Interp2DAmplitude
from few.waveform import FastSchwarzschildEccentricFlux
```

## A Quick Introduction to FastEMRIWaveforms

### Collaborators: Alvin Chua, Niels Warburton, Scott Hughes, Lorenzo Speri

```python
generate = FastSchwarzschildEccentricFlux()

M = 1e6  # large mass
mu = 1e1  # small mass
p0 = 12.0  # separation
e0 = 0.4  # eccentricity
theta = np.pi/3.  # polar viewing angle
phi = np.pi/4.  # azimuthal viewing angle
dist = 1.0  # distance in Gpc

T = 1/365.  # in years
dt = 10.0  # time spacing of data stream

wave = generate(M, mu, p0, e0, theta, phi, dist, T=T, dt=dt)

plt.plot(wave.real)
plt.plot(wave.imag)
```

FastEMRIWaveforms is the first fully relativistic template generation tool for extreme mass ratio inspirals. We think of it more as a framework than any specific waveform. In this tutorial, we will discuss the main pieces of this framework and how they relate to our first fully relativistic waveform model shown above.

As a basic primer, our five key points we want to achieve with this framework are:

* Accuracy: Our fast waveforms must be accurate when compared to slow and accurate waveforms generated by the waveform modeling community.

* Modularity: These waveforms are to be built out of a set of modules. These modules are to be easily interchangeable, as well as stand alone tools available for more in-depth analysis.

* Flexibility: This framework must be easily adaptable to new computational methods and/or improvements in EMRI physics.

* Easy User Interface: All modules and complete waveforms are to have a front-facing python interface with clear and extensive documentation, as well as many examples.

* Parallelization: These waveforms must take advantage of parallelization techniques such as OpenMP, as well as accelerator hardware such as GPUs.

## Basics of an EMRI Waveform

The guiding equation for building an EMRI waveform is given by

\begin{equation}
h_+-ih_x = \frac{1}{r}\sum_{lmkn}\left(-\frac{Z_{lmkn}}{\omega_{mkn}^2}\right)\left(S_{lmkn}(\theta)e^{-im\phi}\right)e^{i\Phi_{mkn}} = \frac{1}{r}\sum_{lmkn}A_{lmkn}\Theta_{lmkn}e^{i\Phi_{mkn}}.
\end{equation}

Here we are concerned with generating fast and accurate waveforms. These waveforms are created with a sequence of modules. We will discuss the three main modules used to produce these waveforms. The first module is the Trajectory module which takes initial parameters and produces arrays for all of the evolving quantities of concern. These include the phase evolution, {$\Phi_\varphi,\Phi_\theta, \Phi_r$}, from the start to the end of the EMRI orbit. With these quantities, we can produce:

\begin{equation}
\Phi_{mkn} = m\Phi_\varphi + k\Phi_\theta + n\Phi_r.
\end{equation}

The trajectory also produces orbital quantities over time. These include the separation ($p$), eccentricity ($e$), and the inclination angle of the orbit ($\iota$). With these arrays containing the orbital evolution of these values, we generate the amplitudes, $A_{lmkn}$. While the evolution is over time, the various phasing and amplitude computations are produced within a frequency decomposition into $(l,m,k,n)$ modes.

With phases and amplitdues in hand, we combine this with the angular harmonic information ($\Theta_{lmkn}$) to produce the final waveform. In this step, we calculate the sum of all modes at each time point in the template data stream.

## Overall Waveform Strategy

* We calculate our trajectories and amplitudes as sparse arrays in time by using a large adaptive stepping integrator. This produces arrays with $\sim100$ points. All of these quantities vary slowly and smoothly. After these calculations are complete, we scale this up to the actual data stream cadence.

* The speed of EMRI waveforms is strongly determined by the amount of harmonic content. Higher eccentricities require more modes to produce a high fidelity waveform. In order to make our waveforms as efficient as possible, we perform an online mode content calculation that removes modes from consideration if they do not contribute to the waveform power determined by a user-defined threshold.

## Current Waveform Model: Schwarzschild Eccentric

* No $k$ modes
* Orbit is equatorial
* $S_{lmkn}(\theta)e^{-im\phi}$ reduces to $_{-2}Y_{lm}(\theta,\phi)$
* $l:\{2,10\}$, $m:\{-l,l\}$, $n:\{-30,30\}$ $\rightarrow$ 3843 modes.

## Fast Trajectories: $\{p, e, \Phi_\varphi, \Phi_r\}$

We are not going to spend too much time on the trajectory part. However, we need to generate it in order to build the rest of our waveform. To build the trajectory, we integrate with large steps using an RK8 integrator.

```python
traj = RunSchwarzEccFluxInspiral()
```

```python
p0 = 16.0  # initial separation
e0 = 0.4  # initial eccentricity
mu = 180.  # iniital small mass in solar masses, produces approximately 1 yr waveform
M = 1e6   # initial large mass in solar masses
dt = 10.0 # sets initial step size
T = 1.0 # in years

t, p, e, Phi_phi, Phi_r, flux = traj(M, mu, p0, e0, T=T, dt=dt)
print("length:", len(t), "duration:", t[-1])
```

```python
fig, axes = plt.subplots(2, 3)
plt.subplots_adjust(wspace=0.3)
fig.set_size_inches(14, 8)
axes = axes.ravel()

ylabels = [r'$e$', r'$p$', r'$e$', r'$\Phi_\phi$', r'$\Phi_r$', r'Flux']
xlabels = [r'$p$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$', r'$t$']
ys = [e, p, e, Phi_phi, Phi_r, flux]
xs = [p, t, t, t, t, t]

for i, (ax, x, y, xlab, ylab) in enumerate(zip(axes, xs, ys, xlabels, ylabels)):
    ax.plot(x, y, lw=0.5)
    ax.scatter(x, y, s=5)
    ax.set_xlabel(xlab, fontsize=16)
    ax.set_ylabel(ylab, fontsize=16)
```

## RomanNet Amplitudes: $A_{lmn}$

To generate the amplitudes, we use a RomanNet ([arXiv:1811.05491](https://arxiv.org/abs/1811.05491)). Roman stands for Reduced Order Modelling with Artificial Neurons. When training a neural network, it can be crucial to hand it data that is distilled to highlight the most pertinent information. A nice tool for doing this is Reduced Order Modeling. Reduced order modelling projects the information with lossless compression down to a lower dimensional space. We take our complex mode amplitude vectors containing 3843 modes and project this down to a real-valued space with 198 values. The neural network is then trained with inputs given by the $p$ and $e$ values and outputs given by the reduced order coefficients ($\alpha_i$):

\begin{equation}
A_{lmn}\in\mathbb{C}^{3843}\xrightarrow{\mathit{ROM}}\alpha_i\in\mathbb{R}^{198}\xrightarrow{\mathit{train}} f(p,e)=\alpha_i
\end{equation}

The neural network itself is extremely simple. It is a fully connected network with a LeakyReLU activation on all layers but the final layer. This means the neural network can be built simply with a sequence of linear matrix multiplications followed by a pass through the activation function. Once the neural network is trained, we will have a set of weights. This is where we will start in this tutorial.

During online evaluation of the waveform, we perform the reverse process:

\begin{equation}
f(p,e)\xrightarrow{\mathit{eval}}\alpha_i\in\mathbb{R}^{198}\xrightarrow{\mathit{project}}A_{lmn}\in\mathbb{C}^{3843}
\end{equation}

This method has pros and cons:

Pros:

* This is more of a global fit, rather than individual fits to given modes. This generally means storage of less information in memory. If we do individual fits of every mode, the memory necessary to store this information would scale badly with mode content. We are currently working in Schwarzschild Eccentric. As we go to generic Kerr, we expect the number of modes to increase by a factor of $\sim10$.

* Due to its global fit nature, this method is generally faster to evaluate than individual interpolants.

* Since it is a neural network and a linear projection, this is very suitable to GPUs.

* We expect this method, or methods similar to this, to scale better with dimensionality as we move towards the end goal of generic Kerr orbits.

Cons:

* For extremely quiet modes, this method can be less accurate. However, as these modes are quiet, this does not result in a significant loss of accuracy in the final waveform. Caution must be taken when using this method to analyze individual mode amplitudes. There is an approximate floor in the amplitude values at $\sim10^{-5}$. For reference, the loudest modes at a given $p$ and $e$ value are usually $\sim0.1-1.0$. (These amplitudes are not scaled for distance.|)

* Training these neural networks can be more of an art than a science. It takes a lot of trial and error to get this right.

### Constructing the neural network from trained weights

```python

# prepare to load the weights

path_to_few_dir = romannet.__file__[:-25]

weight_file = fp = "SchwarzschildEccentricInput.hdf5"
check_for_file_download(fp, path_to_few_dir)

weights = []
bias = []
dim1 = []
dim2 = []

# get highest layer number
num_layers = 0

# extract all necessary information from the file
with h5py.File(path_to_few_dir + "few/files/" + weight_file, "r") as fp:

    # get basic information
    num_teuk_modes = fp.attrs["num_teuk_modes"]
    transform_factor = fp.attrs["transform_factor"]
    break_index = fp.attrs["break_index"]

    # determine layer arrangement
    for key, value in fp.items():
        if key == "reduced_basis":
            continue

        layer_num = int(key[1:])

        if layer_num > num_layers:
            num_layers = layer_num

    # get weights and bias
    for i in range(1, num_layers + 1):
        temp = {}
        for let in ["w", "b"]:
            mat = fp.get(let + str(i))[:]
            temp[let] = np.asarray(mat)

        weights.append(temp["w"])
        bias.append(temp["b"])
        dim1.append(temp["w"].shape[0])
        dim2.append(temp["w"].shape[1])

    # get the post network transform matrix
    transform_matrix = np.asarray(fp["reduced_basis"])

# activation function
# we use a factor of 0.2 for negative values
def LeakyReLU(x):
    out = (x >= 0.0) * x +  (x < 0.0) * 0.2*x
    return out

# build the neural network
def RomanNet(p, e):

    p = np.atleast_1d(p)
    e = np.atleast_1d(e)

    # convert from the p coordinate to a special y coordinate
    # see the documentation for more details
    y = p_to_y(p, e)

    # prepare input
    x = np.array([y, e])

    # basic fully connected network
    for layer_i in range(num_layers):

        # linear transformation
        x = np.dot(weights[layer_i].T, x) + bias[layer_i][:, np.newaxis]

        # do not want to activate last layer
        if layer_i < num_layers - 1:
            # non-linear activatation
            x = LeakyReLU(x)

    # separate real and imaginary
    x = x[:break_index] + 1j * x[break_index:]

    # project back to amplitude basis
    out = np.dot(transform_matrix.T, x)/transform_factor
    return out.T

```

```python
# test it
p_test = np.array([11.0, 10.0, 10.0])
e_test = np.array([0.2, 0.3, 0.1])

RomanNet(p_test, e_test)
```

### Produce amplitudes associated with our trajectories

```python
# generate amplitudes with roman net
amps = RomanNet(p, e)

# check against actual code
RomanNetTrue = RomanAmplitude()
amps_check_1 = RomanNetTrue(p, e)

assert np.allclose(amps, amps_check_1)
```

### Check our RomanNet global fit against accurate values

```python
# get accurate values
# each mode is fitted with a bicubic spline
BicubicAmps = Interp2DAmplitude()
amps_check_2 = BicubicAmps(p, e)
```

#### Print vectors

```python
print("romannet:", amps_check_1[0][0:10], "\nBicubic Spline:", amps_check_2[0][0:10])
```

We see if we take a quick examination of our results (a small subset) we see the problem with the global fit. It cannot handle modes with a very small amplitude. Therefore, you might naively think that this method might not work.

#### Compare via the cosine between the vectors

```python
cos = (np.dot(amps_check_2.conj()[0].T, amps_check_1[0]) /
       np.sqrt(np.dot(amps_check_2.conj()[0], amps_check_2[0])
               * np.dot(amps_check_1.conj()[0], amps_check_1[0]))).real

print('Cos:', cos)
```

We now see that if we compare the results as a whole, the results match very well. If we look at individual modes that have high power, they are likely to strongly match when the bicubic spline is compared to the RomanNet method.

## GPU-accelerated Waveform Build: $\sum_{lmn}$

### Quick primer on GPUs

GPUs run code in parallel in a configuration of grids, blocks, and threads.

* **Threads** are the the actual software units that run the code. Threads run independently of one another. Threads are referenced in code using `threadIdx.x`.
* **Blocks** are an array of threads. This array can be 1, 2, or 3 dimensions. In more than one dimension, thread indices are referenced using `x`, `y`, and `z` (e.g. `threadIdx.z`). Blocks, similar to threads, are referenced using `blockIdx.x`, `blockIdx.y`, or `blockIdx.z`. The size of a block, i.e. the number of threads along a given dimension is given as `blockDim.x`. **Note**: In applications I have worked on, I rarely ever use more than 1 dimension of threads.
* **Grids** are an array of blocks. This array can be 1, 2, or 3 dimensions. There is no reference for grids as they efffectively represent the entire GPU kernel. The size of a grid, i.e. the number of blocks along a dimension, is determined with `gridDim.x`.

Organizing your code properly into blocks and threads is a key component of maximizing your efficiency.

Another equally important aspect to the maximization of efficiency is the proper use of the GPU memory structures. There are three main GPU memory structures to consider (there are a few more but they are for more specific uses). These are global memory, shared memory, and local memory.

* **Global memory** is contained on the host (or off-chip). Global memory is accessible by every block and thread in the entire grid. It contains many GBs of RAM. (The GPUs I use currently, which are Tesla V100s, have ~16GB. The new A100s have 40-80GB.) Because this memory is contained off-chip, it is slow to access. A key to accessing global memory efficiently is to use so-called "memory coalescence." This effectively means that neighboring threads access neighboring addresses in memory. This allows the compiler to make up to 32 memory reads at the same time, rather than 32 separate memory reads. We will see this in a simple case below.

* **Local memory** is also contained on the host. Local memory consists of any arrays allocated within the kernel specific to each thread. Therefore, this memory is only accessible by the thread it is created on. Usually, around 512 KB are available for local memory for each thread. Since, this memory is off-chip, it is also slow to access. Local memory is, however, always accessed in a memory coalescing manner.

* **Shared memory** is different. Shared memory is located on-chip. It is, therefore, much faster to read from (~100x faster). Shared memory is accessible by all the threads on a given block. The catch is that only ~48 KB are available for shared memory. Therefore, leveraging shared memory effectively is key to the efficiency of GPU code.

To sum up, there are two main points to consider when beginning to program on GPUs:

* Layout your grid effectively for your given problem.
* Use memory effectively: leverage the availability of shared memory and read from global memory in a coalescing fashion.

Some parting thoughts on GPUs:

* For maximal efficiency and stability, I usually code in C++/CUDA so that everything is precompiled and tested.
* There are python libraries that leverage the power of GPUs. Check out numba, CuPy, PyTorch, Tensorflow, PyCUDA.
* Lately, I have really focused on writing CPU/GPU agnostic code. This means that the source code is ~99% the same between the two. On the python side I usually sub in CuPy for Numpy. We will see some basic pointers on this below. In C++, I use short compiler directives to make minimal changes.
* Generally speaking it is optimal to store all quantities in a 1 dimensional array when working with GPUs. Let's say you have a two dimensional array that has dimensions (dim1, dim2) and is referenced with (i, j). You can turn this into a 1D array that references each value with (i * dim2 + j).

### Why are GPUs important for EMRI Waveforms?

The waveform summation is the key bottleneck. This is an operation that is uniquely suited to GPUs. You can see the improvement below.

### Basic example coded in python

We are going to write code in python that will reflect what we will do on the GPU. It will obviously still be on the CPU, but will give you a chance to see how to write some basic GPU code and understand how it works. **Make sure to read the comments in the code.**

We will do two examples.

#### Multiply two arrays

Here we will multiply two arrays as we would on the GPU.

```python
# setup our GPU quantities that would come in if we were actually on GPUs.

NUM_THREADS = 64  # needs to be a multiple of 32
blockDim = NUM_THREADS  # blockDim.x
```

```python

# this initial piece is a basic of CPU/GPU agnostic code in python
try:
    import cupy as xp

except ModuleNotFoundError:
    import numpy as xp

# __global__
def multiply_arrays(array_out, array1, array2, n):
    """
    // what this would like like in CUDA

    // if (threadIdx.x + blockDim.x * blockIdx.x >= n) return;
    for (int i = threadIdx.x + blockDim.x * blockIdx.x; i < n; i += blockDim.x * gridDim.x)
    {
        array_out[i] = array1[i] * array2[i];
    }
    """
    # The loops here are to simulate the GPU.
    # In reality the GPU will run all threads and blocks in parallel

    # begin simulation
    for block in range(num_blocks):
        for thread in range(NUM_THREADS):
    # end simulation

            # get the overall index in the grid
            # based on the thread and block
            i = thread + block * blockDim

            # since the GPU runs threads in parallel
            # we need to make sure the GPU does not index a value
            # on the last block that goes over the array length
            if i >= n:
                continue

            # use our index to get values out of the array
            # notice this simple statement uses memory coalescence
            # neighboring threads will have consecutive indices
            # therefore, they will access consecutive addresses in each array
            array_out[i] = array1[i] * array2[i]

length = int(2 ** 14)

# initialize arrays
array1 = xp.random.rand(length)
array2 = xp.random.rand(length)

# prepare output array
array_out = xp.zeros_like(array1)

# get the number of blocks
# the number of blocks multiplied by the number of threads per block
# must be greater than the length of the array
num_blocks = int(np.ceil((length + NUM_THREADS -1)/NUM_THREADS))

# this would actually be called as
"""
multiply_arrays<<<num_blocks, NUM_THREADS>>>(array_out, array1, array2, length);
cudaDeviceSynchronize();
"""

multiply_arrays(array_out, array1, array2, length)

# confirm it
assert np.allclose(array_out, array1 * array2)
```

#### Linear Interpolation to scale up an array

For our second example it will be a bit more complicated. This way we can see how to use shared memory.

Here we will scale up the size of an array using linear interpolation. The key here is that we need every block to read in the original array to shared memory. From there, we perform the interpolation. This allows us to achieve memory coalescence with all global reads and ensure that all reads when actually interpolating are done from shared memory.

```python
# in CUDA, when you statically allocate shared arrays,
# you cannot use a variable length
# therefore, in C++/CUDA you need to declare a max value up front

#define MAX_INPUT 1000

def linear_interpolation(array_out, array_in, n_out, n_in, dx_out, dx_in):

    # we want to read in to shared memory
    # want to do this with memory coalescence
    # since this is only on a specific block
    # we need to use only the threads on this block

    """
    // the below simulated code block in CUDA would look like this:

    // in CUDA, you have to declare shared arrays

    __shared__ double shared_array_in[MAX_INPUT];

    for (int i = threadIdx.x; i < n_in; i += blockDim.x)
    {
        // memory coalescence only needed from array_in (global memory)
        shared_array_in[i] = array_in[i];
    }

    // we need to make sure the threads all finish this operation before we move on
    // Therefore, we force the threads to synchronize

    __syncthreads();

    """

    # we will just do the following to simulate in python easily
    shared_array_in = xp.zeros(n_in)

    for thread in range(NUM_THREADS):
        i = thread

        # this while statement simulates the above commented loop
        while (i < n_in):
            # here is the memory coalesced reads
            shared_array_in[i] = array_in[i]

            # just for simulation
            i += blockDim

    """
    // this is what the below would really like it in CUDA
    // if (threadIdx.x + blockDim.x * blockIdx.x >= n_out) return;
    for (int i = threadIdx.x + blockDim.x * blockIdx.x; i < n_out; i += blockDim.x * gridDim.x)
    {
        // get the new out value assuming equal spacing
        double x_new = dx_out * i;

        // get index of the point in the original array below the new point
        int ind_in = (int) (x_new / dx_in);

        // get the below x value
        double x_old = ind_in * dx_in;

        // slope of segment
        double m = (shared_array_in[ind_in + 1] - shared_array_in[ind_in])/dx_in;

        // interpolate
        double new_value = m * (x_new - x_old) + shared_array_in[ind_in];
        array_out[i] = new_value;
    }

    """
    # The loops here are to simulate the GPU.
    # In reality the GPU will run all threads and blocks in parallel

    # begin simulation
    for block in range(num_blocks):
        for thread in range(NUM_THREADS):
    # end simulation

            # get the overall index in the grid
            # based on the thread and block
            i = thread + block * blockDim

            # since the GPU runs threads in parallel
            # we need to make sure the GPU does not index a value
            # on the last block that goes over the array length
            if i >= n_out:
                continue

            x_new = dx_out * i

            ind_in = int(x_new / dx_in)
            x_old = ind_in * dx_in

            m = (shared_array_in[ind_in + 1] - shared_array_in[ind_in])/dx_in

            new_value = m * (x_new - x_old) + shared_array_in[ind_in]
            array_out[i] = new_value

# original length
length_in = 100

# prepare input arrays
x_in = xp.arange(length_in)
y_in = (x_in ** 2).astype(xp.float64)

# set length out
length_out = int(2 ** 14)

# setup the new x_values
x_new = xp.linspace(x_in[0], x_in[-1], length_out + 1)[:-1]

# change in original x values
dx_in = 1.0

# we will actually only use the spacing to find new points
dx_out = x_new[1] - x_new[0]

y_out = xp.zeros(length_out)

num_blocks = int(np.ceil((length_out + NUM_THREADS -1)/NUM_THREADS))

# this would actually be called as
# linear_interpolation<<<num_blocks, NUM_THREADS>>>(y_out, y_in, length_out, length_in, dx_out, dx_in);
# cudaDeviceSynchronize();
linear_interpolation(y_out, y_in, length_out, length_in, dx_out, dx_in)

# confirm it
plt.plot(x_new, y_out, lw=6, label='out')
plt.plot(x_in, y_in, '.', label='in')
```

### Comment on CPU/GPU Agnostic code in C++/CUDA and Python

Making CPU/GPU agnostic code consists of 3 main parts:

* Sub CuPy for NumPy
* Compiler directives in C++/CUDA
* Easy transition from Python to C++ through an augmented Cython process

Above, we saw an example of how to deal with the Python side. The Cython functionality is effectively a decorator function that gets the pointer of a Numpy or Cupy array and sends that into the C++ code. Below is a basic example of how to use basic compiler directives to make your code more CPU/GPU agnostic.

**Note**: I generally argue that we should build codes for GPUs and then adapt them to CPUs, not the other way around which is the typical direction. In my experience, the CPU codes adapted from GPU codes are just as fast or within a small percentage of the speed of CPU designed codes. And, generally, it is much harder to optimize going from CPU->GPU rather than GPU->CPU.

## Future Plans

* Build fast trajectories in Kerr under the NIT framework
* Generate amplitudes in Kerr regime
* Further analyze how to determine best methods for mode content inclusion

We need your help! If anyone is interested in working on these types of issues, please let myself or any of my collaborators (listed above) know!


# 참고 문서
https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py
https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

Deep Learning with PyTorch

What is PyTorch?
================

It’s a Python-based scientific computing package targeted at two sets of
audiences:

-  A replacement for NumPy to use the power of GPUs
-  a deep learning research platform that provides maximum flexibility
   and speed

Getting Started
---------------

Tensors
^^^^^^^

Tensors are similar to NumPy’s ndarrays, with the addition being that
Tensors can also be used on a GPU to accelerate computing.

```python
from __future__ import print_function
import torch
```

```python
print(torch.cuda.is_available())
```

    True

 **주의) 위 결과가 False이면 Colab 메뉴의 런타임->런타임 유형변경에서 하드웨어 가속기를 GPU로 선택하고 다시 시작**

```python
x = torch.empty(5, 3) # 5행, 3열의 텐서. 값은 모두 garbage
print(x)
```

    tensor([[-3.0226e+35,  3.0672e-41,  3.3631e-44],
            [ 0.0000e+00,         nan,  0.0000e+00],
            [ 1.1578e+27,  1.1362e+30,  7.1547e+22],
            [ 4.5828e+30,  1.2121e+04,  7.1846e+22],
            [ 9.2198e-39,  7.0374e+22,  0.0000e+00]])

```python
print(x.size()) # size, shape
```

    torch.Size([5, 3])

```python
print(x.shape)
```

    torch.Size([5, 3])

Uninitialized vs random

An uninitialized matrix is declared,
    but does not contain definite known
    values before it is used. When an
    uninitialized matrix is created,
    whatever values were in the allocated
    memory at the time will appear as the initial values.

---

torch.empty는 초기화하지 않은 값(할당받은 메모리에 있는 값)이 들어있다

Construct a 5x3 matrix, uninitialized:

즉 값이 garbage임. random이 아님.

```python
x = torch.empty(5, 3)
print(x)
```

    tensor([[-3.0226e+35,  3.0672e-41,  3.3631e-44],
            [ 0.0000e+00,         nan,  0.0000e+00],
            [ 4.4721e+21,  1.5956e+25,  4.7399e+16],
            [ 3.7293e-08,  3.9664e+28,  6.9397e+22],
            [ 1.7260e+25,  2.2856e+20,  5.0948e-14]])

```python
print(torch.get_default_dtype())
```

    torch.float32

```python
print(x.dtype)
```

    torch.float32

torch.empty는 torch의 global default dtype 이면서 garbage 값을 가지는 원소가 채워진 텐서를 생성.

즉 디폴트 타입을 갖는 빈 텐서를 생성.

```python
x = torch.FloatTensor(5,3)
print(x)
```

    tensor([[-3.0226e+35,  3.0672e-41,  3.7835e-44],
            [ 0.0000e+00,         nan,  7.1547e+22],
            [ 1.3733e-14,  6.4069e+02,  4.3066e+21],
            [ 1.1824e+22,  4.3066e+21,  6.3828e+28],
            [ 3.8016e-39,  0.0000e+00,  0.0000e+00]])

주의) torch.Tensor와 torch.tensor 사용 예.

torch.Tensor는 Tensor 클래스를 instantiation. 즉 torch.empty와 같은 동작.

torch.tensor는 인자로 받는 데이터 값을 복제. 인자는 리스트, 튜플, ndarray, 스칼라 값 등 다양한 형태. 텐서를 만드는 가장 일반적 형태임.

```python
x = torch.Tensor(5,3)
print(x, x.dtype) # 의미없는 값.
```

    tensor([[-3.0226e+35,  3.0672e-41,  2.3694e-38],
            [ 9.2196e-41, -3.0226e+35,  3.0672e-41],
            [-3.0478e+35,  3.0672e-41, -3.1655e+35],
            [ 3.0672e-41, -3.1656e+35,  3.0672e-41],
            [-3.1655e+35,  3.0672e-41,  0.0000e+00]]) torch.float32

```python
x = torch.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])
print(x, x.dtype)
```

    tensor([[0.1000, 1.2000],
            [2.2000, 3.1000],
            [4.9000, 5.2000]]) torch.float32

```python
x = torch.tensor([0, 1]) # dtype을 인자의 값에서 유추하여 설정
print(x, x.dtype)
```

    tensor([0, 1]) torch.int64

```python
y = torch.tensor([0.0, 1.0])
print(y, y.dtype)
```

    tensor([0., 1.]) torch.float32

```python
try:
    z = torch.tensor([[0.11111, 0.222222, 0.3333333]], dtype=torch.float64, device=torch.device('cuda:0'))  # creates a torch.cuda.DoubleTensor
    print(z, z.dtype)
except Exception as e:
    print('Error:', e)
```

    tensor([[0.1111, 0.2222, 0.3333]], device='cuda:0', dtype=torch.float64) torch.float64

CPU 환경에선 위 문장이 오류 발생.

GPU 환경으로 런타임 변경 후 실행.

```python
x = torch.tensor(3.14159)  # Create a scalar (zero-dimensional tensor)
print(x, x.dtype)
```

    tensor(3.1416) torch.float32

```python
x = torch.tensor([])  # Create an empty tensor (of size (0,))
print(x, x.dtype)
```

    tensor([]) torch.float32

```python
try:
    x = torch.tensor(3,2) # 인자가 매치하지 않으므로 오류가 나야 함.
    print(x, x.dtype)
except Exception as e:
    print('Error:', e)
```

    Error: tensor() takes 1 positional argument but 2 were given

---

Random

Construct a randomly initialized matrix:

```python
x = torch.rand(5, 3)
print(x)
```

    tensor([[0.6106, 0.1358, 0.2298],
            [0.4580, 0.9274, 0.1602],
            [0.1059, 0.0834, 0.8331],
            [0.0217, 0.5579, 0.4679],
            [0.8884, 0.5640, 0.5379]])

Construct a matrix filled zeros and of dtype long:

```python
x = torch.zeros(5, 3, dtype=torch.long)
print(x)
```

    tensor([[0, 0, 0],
            [0, 0, 0],
            [0, 0, 0],
            [0, 0, 0],
            [0, 0, 0]])

데이터를 인자로 주어 텐서 생성

Construct a tensor directly from data:

```python
x = torch.tensor([5.5, 3])
print(x, x.dtype)
```

    tensor([5.5000, 3.0000]) torch.float32

or create a tensor based on an existing tensor. These methods
will reuse properties of the input tensor, e.g. dtype, unless
new values are provided by user

x와 같은 타입의 새 텐서 생성

```python
x  = x.new_ones(5, 3)      # new_* methods take in sizes
print(x, x.dtype)
```

    tensor([[1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.]]) torch.float32

```python
x  = x.new_ones(5, 3, dtype=torch.double)      # new_* methods take in sizes
print(x, x.dtype)
```

    tensor([[1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.]], dtype=torch.float64) torch.float64

```python
x = torch.randn_like(x, dtype=torch.float)    # override dtype!
print(x, x.dtype)                                      # result has the same size
```

    tensor([[ 0.8815, -0.4184,  0.0153],
            [-0.1281, -0.1178,  0.4411],
            [ 1.3856,  0.2023, -0.7654],
            [-1.6897,  0.8962,  0.8546],
            [ 0.8858, -1.1012,  0.8380]]) torch.float32

Get its size: size() or shape

```python
print(x.size())
print(x.shape)
```

    torch.Size([5, 3])
    torch.Size([5, 3])

Note

``torch.Size`` is in fact a tuple, so it supports all tuple operations.

Operations
^^^^^^^^^^
There are multiple syntaxes for operations. In the following
example, we will take a look at the addition operation.

텐서 더하기

Addition: syntax 1

```python
y = torch.rand(5, 3)
print('x\n', x)
print('y\n', y)
print('x+y\n', x + y)
```

    x
     tensor([[ 0.8815, -0.4184,  0.0153],
            [-0.1281, -0.1178,  0.4411],
            [ 1.3856,  0.2023, -0.7654],
            [-1.6897,  0.8962,  0.8546],
            [ 0.8858, -1.1012,  0.8380]])
    y
     tensor([[0.2691, 0.3013, 0.0288],
            [0.3602, 0.8997, 0.1451],
            [0.0238, 0.5176, 0.7644],
            [0.1169, 0.9153, 0.7058],
            [0.3358, 0.9711, 0.0666]])
    x+y
     tensor([[ 1.1506e+00, -1.1708e-01,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

Addition: syntax 2

```python
print(torch.add(x, y))
```

    tensor([[ 1.1506e+00, -1.1708e-01,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

Addition: providing an output tensor as argument

```python
result = torch.empty(5, 3)
print(result)
```

    tensor([[ 2.8641e+08,  3.0673e-41,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

```python
torch.add(x, y, out=result)
print(result)
```

    tensor([[ 1.1506e+00, -1.1708e-01,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

```python
res = x + y
print(res)
```

    tensor([[ 1.1506e+00, -1.1708e-01,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

Addition: in-place

```python
# adds x to y
y.add_(x)
print(y)
```

    tensor([[ 1.1506e+00, -1.1708e-01,  4.4037e-02],
            [ 2.3207e-01,  7.8194e-01,  5.8625e-01],
            [ 1.4095e+00,  7.1991e-01, -1.0388e-03],
            [-1.5728e+00,  1.8116e+00,  1.5604e+00],
            [ 1.2216e+00, -1.3014e-01,  9.0465e-01]])

# **_로 끝나는 operation은 in-place**

Note

Any operation that mutates a tensor in-place is post-fixed with an ``_``.
    For example: ``x.copy_(y)``, ``x.t_()``, will change ``x``.

You can use standard NumPy-like indexing with all bells and whistles!

```python

```

# autograd

학습에서 error backpropagation을 위한 그래디언트 계산을 자동화한 PyTorch 기능

```python
import torch

a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
```

```python
a
```

    tensor([2., 3.], requires_grad=True)

```python
b
```

    tensor([6., 4.], requires_grad=True)

```python
Q = 3*a**3 - b**2
```

\begin{align}Q=3a^{3}-b^{3}\end{align}

```python
Q
```

    tensor([-12.,  65.], grad_fn=<SubBackward0>)

\begin{align}\frac{\partial Q}{\partial a} = 9a^2\end{align}

\begin{align}\frac{\partial Q}{\partial b} = -2b\end{align}

\begin{align}\frac{dQ}{dQ} = 1\end{align}

```python
external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)
```

```python
Q
```

    tensor([-12.,  65.], grad_fn=<SubBackward0>)

```python
a.grad, b.grad
```

    (tensor([36., 81.]), tensor([-12.,  -8.]))

```python
# check if collected gradients are correct
print(9*a**2 == a.grad)
print(-2*b == b.grad)
```

    tensor([True, True])
    tensor([True, True])

```python

```

# 일반적으로는 Numpy 배열과 유사한 신택스를 가진다

# Indexing tensors

## Python slicing 과 유사

https://drive.google.com/file/d/1qYbFGElZiXcB4VgUiolBp0f2YZS1pzGw/view?usp=sharing

```python
some_list = list(range(6))
some_list[:]
```

    [0, 1, 2, 3, 4, 5]

```python
print(some_list[1:4])
print(some_list[1:])
print(some_list[:4])
print(some_list[:-1])
print(some_list[1:4:2]) # 1 to 3 ([1,2,3]), step 2 ([1,3])
```

    [1, 2, 3]
    [1, 2, 3, 4, 5]
    [0, 1, 2, 3]
    [0, 1, 2, 3, 4]
    [1, 3]

```python
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
print(points)
print(points.shape)

```

    tensor([[4., 1.],
            [5., 3.],
            [2., 1.]])
    torch.Size([3, 2])

```python
print(points[1:])
```

    tensor([[5., 3.],
            [2., 1.]])

```python
print(points[1:, :])
```

    tensor([[5., 3.],
            [2., 1.]])

```python
print(points[1:, 0])
```

    tensor([5., 2.])

```python
print(points[None])
```

    tensor([[[4., 1.],
             [5., 3.],
             [2., 1.]]])

```python
print(points[None].size())
```

    torch.Size([1, 3, 2])

```python
points.shape
```

    torch.Size([3, 2])

```python
points
```

    tensor([[4., 1.],
            [5., 3.],
            [2., 1.]])

# unsqueeze - dimension 확장. dim으로 지정된 디멘션을 하나 확장

```python
print(points.unsqueeze(dim=0))
print(points.unsqueeze(dim=0).size())
```

    tensor([[[4., 1.],
             [5., 3.],
             [2., 1.]]])
    torch.Size([1, 3, 2])

```python
unsq_points = points.unsqueeze(dim=0)
print(unsq_points)
print(unsq_points.size())
```

    tensor([[[4., 1.],
             [5., 3.],
             [2., 1.]]])
    torch.Size([1, 3, 2])

# squeeze - dimension 축소
### 원소 1개짜리 dimension을 제거

```python
sq_points = unsq_points.squeeze()
print(sq_points)
print(sq_points.size())
```

    tensor([[4., 1.],
            [5., 3.],
            [2., 1.]])
    torch.Size([3, 2])

---

Resizing: If you want to resize/reshape tensor, you can use ``torch.view``:

x는 (4,4), y는 (16,), z는 (inferred, 8)인데 전체가 16 원소이므로 inferred는 2가 되어 (2,8)

```python
x = torch.randn(4, 4)
y = x.view(16)
z = x.view(-1, 8)  # the size -1 is inferred from other dimensions
print('x.size()=', x.size(), '\ny.size()=', y.size(), '\nz.size()=', z.size())
print(x, '\n', y, '\n', z)
```

    x.size()= torch.Size([4, 4])
    y.size()= torch.Size([16])
    z.size()= torch.Size([2, 8])
    tensor([[-0.3112, -0.8315,  2.0023, -0.8920],
            [-0.7311, -0.1806, -1.3087, -0.4823],
            [ 1.4226, -2.2820, -0.0773, -2.7146],
            [ 1.2870,  1.0097,  0.1546,  1.3778]])
     tensor([-0.3112, -0.8315,  2.0023, -0.8920, -0.7311, -0.1806, -1.3087, -0.4823,
             1.4226, -2.2820, -0.0773, -2.7146,  1.2870,  1.0097,  0.1546,  1.3778])
     tensor([[-0.3112, -0.8315,  2.0023, -0.8920, -0.7311, -0.1806, -1.3087, -0.4823],
            [ 1.4226, -2.2820, -0.0773, -2.7146,  1.2870,  1.0097,  0.1546,  1.3778]])

---
# transpose

```python
points = torch.tensor([[3.0, 1.0, 2.0], [4.0, 1.0, 7.0]])
print(points)
```

    tensor([[3., 1., 2.],
            [4., 1., 7.]])

## 서로 바꿀 두 디멘션을 지정

```python
points_tr = points.transpose(0, 1)
```

```python
points_tr
```

    tensor([[3., 4.],
            [1., 1.],
            [2., 7.]])

```python
points
```

    tensor([[3., 1., 2.],
            [4., 1., 7.]])

```python
points_t = points.t() # 2D or less dimension
print(points_t)
```

    tensor([[3., 4.],
            [1., 1.],
            [2., 7.]])

## Multidimensional

```python
some_t = torch.ones(3, 4, 5)
transpose_t = some_t.transpose(0, 2) # 0번째와 2번째 디멘션을 transpose
some_t.shape
```

    torch.Size([3, 4, 5])

```python
some_t
```

    tensor([[[1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.]],

            [[1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.]],

            [[1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.],
             [1., 1., 1., 1., 1.]]])

```python
transpose_t.shape
```

    torch.Size([5, 4, 3])

# Contiguous

가장 오른쪽 차원이 순차적으로 저장된 텐서

```python
some_t.is_contiguous()
```

    True

continous한 텐서의 transpose는 실제 데이터를 옮기는 것이 아니라 접근하는 방법만 다르게 하기 때문에 continous하지 않음

```python
transpose_t.is_contiguous()
```

    False

## Contiguous로 변환하기

```python
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
```

```python
points_t = points.t()
```

```python
points_t
```

    tensor([[4., 5., 2.],
            [1., 3., 1.]])

```python
points_t.is_contiguous()
```

    False

실제 데이터를 contiguous하게 재배치

```python
points_t_cont = points_t.contiguous()
```

```python
points_t_cont
```

    tensor([[4., 5., 2.],
            [1., 3., 1.]])

```python
points_t_cont.contiguous()
```

    tensor([[4., 5., 2.],
            [1., 3., 1.]])

## tensor view()

같은 storage 에서 보는 관점 (shape)만 다르게 함

```python
t = torch.rand(4, 4)
b = t.view(2, 8)
print(t, t.shape)
print(b, b.shape)
```

    tensor([[0.4407, 0.9571, 0.6890, 0.5465],
            [0.2858, 0.9516, 0.4148, 0.6201],
            [0.9849, 0.9962, 0.1261, 0.0170],
            [0.2510, 0.7078, 0.5342, 0.2367]]) torch.Size([4, 4])
    tensor([[0.4407, 0.9571, 0.6890, 0.5465, 0.2858, 0.9516, 0.4148, 0.6201],
            [0.9849, 0.9962, 0.1261, 0.0170, 0.2510, 0.7078, 0.5342, 0.2367]]) torch.Size([2, 8])

```python
t.is_contiguous()
```

    True

```python
b.is_contiguous()
```

    True

## view 같은 operation은 contiguous tensor에서만 동작

```python
x = torch.tensor([[1,2,3], [4,5,6]])
x
```

    tensor([[1, 2, 3],
            [4, 5, 6]])

```python
x.shape
```

    torch.Size([2, 3])

```python
x_t = x.t()
x_t
```

    tensor([[1, 4],
            [2, 5],
            [3, 6]])

```python
y = x.view(3,2) # transpose가 아님
y
```

    tensor([[1, 2],
            [3, 4],
            [5, 6]])

```python
y.is_contiguous()
```

    True

```python
x_t.is_contiguous()
```

    False

contiguous하지 않은 텐서로 view를 부르면 오류가 발생

```python
try:
    x_t.view(2, 3) # error
except Exception as e:
    print('Error:', e)
```

    Error: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

```python
x_t_cont = x_t.contiguous()
x_t_cont
```

    tensor([[1, 4],
            [2, 5],
            [3, 6]])

```python
x_t_cont.view(2, 3)
```

    tensor([[1, 4, 2],
            [5, 3, 6]])

```python
x_t
```

    tensor([[1, 4],
            [2, 5],
            [3, 6]])

x_t와 x_t_cont는 표시되는 값은 같으나 내부의 데이터 순서가 다르게 배치되어 있음

```python
x = torch.randn(2,3,4)
x_t = x.transpose(0, 2)
print(x.shape)
print(x_t.shape)

```

    torch.Size([2, 3, 4])
    torch.Size([4, 3, 2])

```python
x
```

    tensor([[[-0.0203,  1.4557,  0.6741, -0.9552],
             [ 0.5372,  1.0151,  1.3597,  0.4710],
             [-1.3028, -0.1712, -0.4442, -0.3842]],

            [[ 0.8422, -0.2642,  0.1464, -0.7503],
             [-0.7967,  1.5664, -1.2226, -1.3853],
             [ 0.3374, -1.1332, -0.7648,  0.3682]]])

```python
x.view(2, 12)
```

    tensor([[-0.0203,  1.4557,  0.6741, -0.9552,  0.5372,  1.0151,  1.3597,  0.4710,
             -1.3028, -0.1712, -0.4442, -0.3842],
            [ 0.8422, -0.2642,  0.1464, -0.7503, -0.7967,  1.5664, -1.2226, -1.3853,
              0.3374, -1.1332, -0.7648,  0.3682]])

```python
try:
    x_t.view(4,6) # Error
except Exception as e:
    print('Error:', e)
```

    Error: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

## reshape()

```python
y = x.reshape((2, 12))
```

```python
print(y.size())
y
```

    torch.Size([2, 12])

    tensor([[-0.0203,  1.4557,  0.6741, -0.9552,  0.5372,  1.0151,  1.3597,  0.4710,
             -1.3028, -0.1712, -0.4442, -0.3842],
            [ 0.8422, -0.2642,  0.1464, -0.7503, -0.7967,  1.5664, -1.2226, -1.3853,
              0.3374, -1.1332, -0.7648,  0.3682]])

```python
z = x.reshape((6, -1)) # -1 은 알아서 맞추라는 의미
```

```python
z
```

    tensor([[-0.0203,  1.4557,  0.6741, -0.9552],
            [ 0.5372,  1.0151,  1.3597,  0.4710],
            [-1.3028, -0.1712, -0.4442, -0.3842],
            [ 0.8422, -0.2642,  0.1464, -0.7503],
            [-0.7967,  1.5664, -1.2226, -1.3853],
            [ 0.3374, -1.1332, -0.7648,  0.3682]])

```python
z = x.reshape((3, -1))
```

```python
z
```

    tensor([[-0.0203,  1.4557,  0.6741, -0.9552,  0.5372,  1.0151,  1.3597,  0.4710],
            [-1.3028, -0.1712, -0.4442, -0.3842,  0.8422, -0.2642,  0.1464, -0.7503],
            [-0.7967,  1.5664, -1.2226, -1.3853,  0.3374, -1.1332, -0.7648,  0.3682]])

```python
y = x.reshape((-1, 2))
```

```python
y
```

    tensor([[-0.0203,  1.4557],
            [ 0.6741, -0.9552],
            [ 0.5372,  1.0151],
            [ 1.3597,  0.4710],
            [-1.3028, -0.1712],
            [-0.4442, -0.3842],
            [ 0.8422, -0.2642],
            [ 0.1464, -0.7503],
            [-0.7967,  1.5664],
            [-1.2226, -1.3853],
            [ 0.3374, -1.1332],
            [-0.7648,  0.3682]])

NumPy Bridge
------------

Converting a Torch Tensor to a NumPy array and vice versa is a breeze.

## The Torch Tensor and NumPy array will share their underlying memory locations (if the Torch Tensor is on CPU), and changing one will change the other.

Converting a Torch Tensor to a NumPy Array
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

```python
a = torch.ones(5)
print(a)
print(type(a))
```

    tensor([1., 1., 1., 1., 1.])
    <class 'torch.Tensor'>

```python
b = a.numpy()
print(b)
print(type(b))
```

    [1. 1. 1. 1. 1.]
    <class 'numpy.ndarray'>

Converting NumPy Array to Torch Tensor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
See how changing the np array changed the Torch Tensor automatically

```python
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)
```

    [2. 2. 2. 2. 2.]
    tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

All the Tensors on the CPU except a CharTensor support converting to
NumPy and back.

CUDA Tensors
------------

Tensors can be moved onto any device using the ``.to`` method.

```python
# let us run this cell only if CUDA is available
# We will use ``torch.device`` objects to move tensors in and out of GPU
if torch.cuda.is_available():
    device = torch.device("cuda")          # a CUDA device object
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.dtype, z.device)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!
    print(z.device)
```

    tensor([[[ 0.9797,  2.4557,  1.6741,  0.0448],
             [ 1.5372,  2.0151,  2.3597,  1.4710],
             [-0.3028,  0.8288,  0.5558,  0.6158]],

            [[ 1.8422,  0.7358,  1.1464,  0.2497],
             [ 0.2033,  2.5664, -0.2226, -0.3853],
             [ 1.3374, -0.1332,  0.2352,  1.3682]]], device='cuda:0')
    torch.float32 cuda:0
    tensor([[[ 0.9797,  2.4557,  1.6741,  0.0448],
             [ 1.5372,  2.0151,  2.3597,  1.4710],
             [-0.3028,  0.8288,  0.5558,  0.6158]],

            [[ 1.8422,  0.7358,  1.1464,  0.2497],
             [ 0.2033,  2.5664, -0.2226, -0.3853],
             [ 1.3374, -0.1332,  0.2352,  1.3682]]], dtype=torch.float64)
    cuda:0

```python
print(z.to("cpu").dtype)
```

    torch.float32

```python
print(z.to("cpu").device)
```

    cpu

```python

```