```
from google.colab import drive
drive.mount('/content/drive')
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```
current_path = '/content/drive/My Drive/Colab Notebooks/Project_GANder/gankook_cuda/semantic-segmentation-pytorch/pytorch_room_layout/'

import os
os.chdir(current_path)
!ls
```

    build		      features		     segment_lsun.py
    cv2_requirements.txt  info.json		     segment_rl.py
    datasets	      lsun.pth.tar	     segment_st.py
    data_transforms.py    lu_vp_detect.egg-info  sunrgbd.pth.tar
    dist		      pics		     test_result
    drn_d_105_024_val     __pycache__	     Vanish_Point_Detection
    drn_d_105_024_val_ms  README.md		     visualize.m
    drn.py		      requirements.txt	     XiaohuLuVPDetection

### Get Vanishing point, inliers

```
# !python XiaohuLuVPDetection/setup.py build
# !python XiaohuLuVPDetection/setup.py install
!pip install -r cv2_requirements.txt
```

    Collecting opencv-python==4.0.0.21
      Downloading https://files.pythonhosted.org/packages/37/49/874d119948a5a084a7ebe98308214098ef3471d76ab74200f9800efeef15/opencv_python-4.0.0.21-cp36-cp36m-manylinux1_x86_64.whl (25.4MB)
         |████████████████████████████████| 25.4MB 1.4MB/s
    Collecting opencv-contrib-python==4.0.0.21
      Downloading https://files.pythonhosted.org/packages/1f/c0/0463d91f297521b2e15e3d682d7077557fe773db72a03a0d2dba899ab8a5/opencv_contrib_python-4.0.0.21-cp36-cp36m-manylinux1_x86_64.whl (31.2MB)
         |████████████████████████████████| 31.2MB 195kB/s
    Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python==4.0.0.21->-r cv2_requirements.txt (line 1)) (1.18.5)
    ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
    Installing collected packages: opencv-python, opencv-contrib-python
      Found existing installation: opencv-python 4.1.2.30
        Uninstalling opencv-python-4.1.2.30:
          Successfully uninstalled opencv-python-4.1.2.30
      Found existing installation: opencv-contrib-python 4.1.2.30
        Uninstalling opencv-contrib-python-4.1.2.30:
          Successfully uninstalled opencv-contrib-python-4.1.2.30
    Successfully installed opencv-contrib-python-4.0.0.21 opencv-python-4.0.0.21

```
import cv2
cv2.__version__
```

    '4.0.0'

### pytorch.room.layout

```
# !pip install -r requirements.txt
# !apt install octave
```

```
# import torch
# torch.__version__
```

```
# !python segment_rl.py train -d datasets/lsun/ -c 4 -s 480 --arch drn_d_105 --batch-size 32 --random-scale 1.75 --random-rotate 15 --epochs 100 --lr 0.01 --momentum 0.9 --lr-mode poly --pretrained sunrgbd.pth.tar
```

```
# !python segment_st.py test -d datasets/sunrgbd/ -c 37 --arch drn_d_105 -s 480 --resume sunrgbd.pth.tar --phase val --batch-size 1 --ms
```

```
# !python segment_lsun.py test -d ../mask_data/wall/ -o ../mask_layout_data/wall/ -c 4 --arch drn_d_105 -s 320 --resume lsun.pth.tar --phase test --batch-size 1 --ms
```

### Functions

```
#             Detect Intersection             #
import math
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.cluster import KMeans
from sympy import Line
from sympy.geometry import Ray

def drawLines(img, lines, color=(255,255,255)):
    """
    Draw lines on an image
    """
    for line in lines:
        for rho,theta in line:
          a = np.cos(theta)
          b = np.sin(theta)
          x0 = a*rho
          y0 = b*rho
          x1 = int(x0 + 1000*(-b))
          y1 = int(y0 + 1000*(a))
          x2 = int(x0 - 1000*(-b))
          y2 = int(y0 - 1000*(a))

          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))
          # if abs(angle) < 70:
          cv2.line(img, (x1,y1), (x2,y2), color, 1)

def line(p1, p2):
    A = (p1[1] - p2[1])
    B = (p2[0] - p1[0])
    C = (p1[0]*p2[1] - p2[0]*p1[1])
    return A, B, -C

def intersection2(L1, L2):
    D  = L1[0] * L2[1] - L1[1] * L2[0]
    Dx = L1[2] * L2[1] - L1[1] * L2[2]
    Dy = L1[0] * L2[2] - L1[2] * L2[0]
    if D != 0:
        x = Dx / D
        y = Dy / D
        return x,y
    else:
        return False

def regression(img, x, y, color=(0, 0, 0), axis=1):

  if axis == 1:
    y_at_border = np.array([0, img.shape[0]])
    p = np.polyfit(y, x, deg=1)
    x_at_border = np.poly1d(p)(y_at_border)
  else:
    x_at_border = np.array([0, img.shape[1]])
    p = np.polyfit(x, y, deg=1)
    y_at_border = np.poly1d(p)(x_at_border)

  cv2.line(img, (int(x_at_border[0]), int(y_at_border[0])), (int(x_at_border[1]), int(y_at_border[1])), color, 2)

  return x_at_border, y_at_border

def find_reg_points(img, lines, color=(255,0,0), drawlines=False):

  centroids = list()
  r_xs = list()
  r_ys = list()
  for line_ in lines:
      for rho,theta in line_:
          a = np.cos(theta)
          b = np.sin(theta)
          x0 = a*rho
          y0 = b*rho
          x1 = int(x0 + 1000*(-b))
          y1 = int(y0 + 1000*(a))
          x2 = int(x0 - 1000*(-b))
          y2 = int(y0 - 1000*(a))

          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))
          if abs(angle) > 80:
            # print(img.shape[1])
            h_layout = line((0, 0), (img.shape[1], 0))
            h_layout_lower = line((0, img.shape[0]), (img.shape[1], img.shape[0]))
            r = intersection2(h_layout, line((x1, y1), (x2, y2)))
            r_lower = intersection2(h_layout_lower, line((x1, y1), (x2, y2)))
            # cv2.line(img, (int(x1), int(y1)), (int(x2), int(y2)), color, 2)

            center_p = (int((r[0] + r_lower[0]) / 2), int((r[1] + r_lower[1])/ 2))
            centroids.append(center_p)

            r_xs.append((r[0], r_lower[0]))
            r_ys.append((r[1], r_lower[1]))

            if drawlines:
              cv2.line(img, (int(r[0]), int(r[1])), (int(r_lower[0]), int(r_lower[1])), color, 2)
              cv2.circle(img, center_p, 10, (255, 0, 255), -1)

  if drawlines:
    cv2.line(img, (int(0), int(0)), (int(0), int(img.shape[0])), color, 2)
    cv2.line(img, (int(img.shape[1]), int(0)), (int(img.shape[1]), int(img.shape[0])), color, 2)
    cv2.circle(img, (0, int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
    cv2.circle(img, (img.shape[1], int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
  centroids.append((0, int(img.shape[0] / 2)))
  centroids.append((img.shape[1], int(img.shape[0] / 2)))

  return r_xs, r_ys, centroids

from scipy.spatial import distance as sci_dist

def order_points(pts):

  xSorted = pts[np.argsort(pts[:, 0]), :]
  leftMost = xSorted[:2, :]
  rightMost = xSorted[2:, :]
  leftMost = leftMost[np.argsort(leftMost[:, 1]), :]
  (tl, bl) = leftMost
  D = sci_dist.cdist(tl[np.newaxis], rightMost, "euclidean")[0]
  (br, tr) = rightMost[np.argsort(D)[::-1], :]

  return np.array([tl, tr, br, bl], dtype="float32")

```

```
from PIL import Image
import os
import matplotlib.pyplot as plt
import numpy as np
from XiaohuLuVPDetection.lu_vp_detect.vp_detection import VPDetection
import time
# import os
import cv2
# import pylab as pl
from skimage import morphology as mp
import sys
from Vanish_Point_Detection import wrapper

length_thresh = 50
principal_point = None
focal_length = 1300 # 1102.79
seed = 1300
vpd = VPDetection(length_thresh, principal_point, focal_length, seed)

org_path = 'datasets/lsun/images/' # -> original image
layout_path = 'drn_d_105_024_val_ms/images/' # -> layout image (result image frome above model file)
refer_path = '../refer_data/wall/myxkehu1kfzggursepnk0tfiyps8zbs5umvzv8d92r6hhxejgawebwsufssgov5q-.jpg'

object_ = 'wall'
mask_path = '../mask_data/' + object_ + '/'
# layout_path = '../test_layout_result/' + object_ + '/'
layout_path = '../mask_layout_data/' + object_ + '/'
# refer_path = './refer_data/' + object_ + '/'

val_image_list = os.listdir(layout_path)

one_iter = False
start_index = 0
for image_index, image in enumerate(val_image_list):

  if image_index < start_index:
    continue
  image = '720x480_20200131153356598_gnIdiy35s8.png'
  print('image_index :', val_image_list.index(image))
  one_iter = True

  # print(image)
  img = Image.open(layout_path + image)
  # print(type(img))
  img_np = np.invert(np.asarray(img))
  # print(img_np.max(), img_np.min())
  ret, thr = cv2.threshold(img_np, 254, 255, cv2.THRESH_BINARY_INV)

  org = Image.open(mask_path + image)
  org_color_np, org_np = np.split(np.asarray(org), 2, axis=1)
  # org_np = np.asarray(org)
  # gray = cv2.cvtColor(org_np, cv2.COLOR_RGB2GRAY)
  # print(org_np.shape)
  # ret, thr = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY_INV)

  print(image)
  # plt.subplot(131)
  # plt.imshow(org)
  # plt.show()
  img_size = (org_np.shape[1], org_np.shape[0])

  scale_factor = 6
  print('scale_factor :', scale_factor)

  refer = Image.open(refer_path)
  refer = np.asarray(refer)

  #     refer_size와 img_size가 동일하거나 refer_size가 작은 경우를 고려해야한다.     #
  refer = np.tile(refer, (scale_factor, scale_factor, 1))
  size_ratio = math.floor(min((refer.shape[0] / (org_np.shape[0] * 1.5)), (refer.shape[1] / (org_np.shape[1] * 1.5))))
  refer = Image.fromarray(refer).resize((int(refer.shape[1] / size_ratio), int(refer.shape[0] / size_ratio)))

  tun = thr
  skl = mp.medial_axis(tun).astype(np.uint8) * 255
  rho = 1
  theta = np.pi/180
  thresh = 50
  # print(skl.shape, skl.dtype)

  # lines = cv2.HoughLines(org_np.astype(np.uint8), rho, theta, thresh)
  lines = cv2.HoughLines(skl, rho, theta, thresh)

  # print(lines)
  # plt.imshow(skl)
  # plt.show()

  # Draw all Hough lines in red
  # img_with_all_lines = np.copy(skl)
  # img_with_all_lines = cv2.cvtColor(img_with_all_lines, cv2.COLOR_GRAY2RGB)
  # plt.imshow(img_with_all_lines)
  # plt.show()
  org_color_np2 = org_color_np.copy()
  org_np = org_np.astype(np.uint8)
  org_np2 = org_np.copy()
  org_np3 = org_np.copy()

  start = time.time()

  r_x_list, r_y_list, centroids_list = find_reg_points(org_np, lines,)
  # drawLines(img_with_all_lines, lines)

  #     Edit Your Mask     #
  kernel = np.ones((10, 10), np.uint8)

  # plt.figure(figsize=(20, 15))
  # for plot_i in range(5):
  #   plt.subplot(1, 5, plot_i+1)
  #   if plot_i == 0:
  #     plt.imshow(org_np2)
  #   elif plot_i == 1: #      Dilation     #
  #     plt.imshow(cv2.dilate(org_np2, kernel, iterations=1))
  #   elif plot_i == 2: #      Closing      #
  #     plt.imshow(cv2.morphologyEx(org_np2, cv2.MORPH_CLOSE, kernel))
  #   elif plot_i == 3: #     Erode     #
  #     plt.imshow(cv2.erode(org_np2, kernel, iterations=1))
  #   elif plot_i == 4: #     Opening   #
  #     plt.imshow(cv2.morphologyEx(org_np2, cv2.MORPH_OPEN, kernel))
  # plt.show()

  org_np2_morp = cv2.morphologyEx(org_np2, cv2.MORPH_CLOSE, kernel)
  org_np2_copy = org_np2_morp.copy()
  org_np2_copy_gray = cv2.cvtColor(org_np2_copy, cv2.COLOR_RGB2GRAY)

  vpd.find_vps(org_np2_morp)
  vps = vpd.vps_2D
  vl_org_img, vl_list = vpd.create_debug_VP_image()

  vpd.find_vps(org_color_np)
  vps = vpd.vps_2D
  vl_color_img, _ = vpd.create_debug_VP_image()

  edge_org_np2 = cv2.Canny(org_np2_morp, 30, 300)
  edge_org_np2_copy = edge_org_np2.copy()
  edge_org_np2_copy2 = cv2.Canny((org_np2_copy / 255. * org_np2_copy).astype(np.uint8), 30, 300)

  vpd.find_vps(edge_org_np2)
  # vps = vpd.vps_2D
  edge_vl_img, vl_list = vpd.create_debug_VP_image()

  #             Hough Lines Detection           #
  # print('org_np2.dtype :', org_np2.dtype)
  # print('org_np2.shape :', org_np2.shape)
  # plt.imshow(org_np2_copy_gray)
  # plt.show()
  # lines = cv2.HoughLines(org_np2_copy_gray, rho, theta, thresh)
  # drawLines(org_np2_copy_gray, lines)
  # plt.imshow(edge_org_np2_copy)
  # plt.show()
  edge_lines = cv2.HoughLines(edge_org_np2_copy, rho, theta, thresh)
  drawLines(edge_org_np2_copy, edge_lines)
  # plt.imshow(edge_org_np2_copy2)
  # plt.show()
  edge_lines2 = cv2.HoughLines(edge_org_np2_copy2, rho, theta, thresh)
  drawLines(edge_org_np2_copy2, edge_lines2)
  # vl_color_img, vps, vpssum = wrapper.dealAImage(org_color_np, True, True, True, True, Ostandard=True)

  # print(vps)
  # print('vps :', vps)
  # right_vp = vps[0]
  # color = (255, 0, 255)
  # cv2.circle(vl_color_img, (int(right_vp[0]), int(right_vp[1])), 50, (255, 0, 255), -1)
  # # cv2.line(vl_org_img, (0, 0), (int(right_vp[0]), int(right_vp[1])), color, 5)
  # # cv2.line(vl_org_img, (0, vl_org_img.shape[0]), (int(right_vp[0]), int(right_vp[1])), color, 5)
  # left_vp = vps[1]
  # color = (255, 0, 0)
  # cv2.circle(vl_color_img, (int(left_vp[0]), int(left_vp[1])), 50, (255, 0, 255), -1)
  # # cv2.line(vl_org_img, (vl_org_img.shape[1], 0), (int(left_vp[0]), int(left_vp[1])), color, 5)
  # # cv2.line(vl_org_img, (vl_org_img.shape[1], vl_org_img.shape[0]), (int(left_vp[0]), int(left_vp[1])), color, 5)

  plt.figure(figsize=(15, 10))
  plt.subplot(141)
  plt.imshow(vl_org_img)
  plt.title('vl_org_img')

  plt.subplot(142)
  # plt.imshow(org_color_np)
  plt.imshow(vl_color_img)

  plt.subplot(143)
  plt.imshow(edge_org_np2_copy)

  plt.subplot(144)
  plt.imshow(edge_org_np2_copy2)
  plt.show()

  centroids_data = np.array(centroids_list)[:, [0]]
  print('len(r_x_list), len(r_y_list) :', len(r_x_list), len(r_y_list))
  print('centroids_data.shape :', centroids_data.shape)
  mms = MinMaxScaler()
  cen_data =mms.fit_transform(centroids_data)
  # print('cen_data :', cen_data)
  K = range(2, 6)
  s_dist = list()
  for k in K:
    if cen_data.shape[0] < k:
      break
    km = KMeans(n_clusters=k)
    km = km.fit(cen_data)
    inertia = km.inertia_
    s_dist.append(inertia)

    # if inertia <= 0:
    #   print('vertical line number =', k)
    #   break
  mms2 = MinMaxScaler()
  mms_s_dist = mms2.fit_transform(np.array(s_dist).reshape(-1, 1))
  k_thresh = 0.1
  # plt.plot(range(len(mms_s_dist)), mms_s_dist)
  # plt.show()

  if cen_data.shape[0] > 2:
    for i in range(len(mms_s_dist)):
      if mms_s_dist[i] < k_thresh:
        k = i + 2
        break
  else:
    k = 2

  #     k Confirmation : Comparing k.cluster_centers_ dist  #
  while True:

    km = KMeans(n_clusters=k)
    km = km.fit(cen_data)
    if k <= 2:
      break

    cluster_centroids = km.cluster_centers_
    # print('cluster_centroids :', cluster_centroids)
    error_exist = False
    for i in range(len(cluster_centroids) - 1):
      for j in range(i + 1, len(cluster_centroids)):
        if abs(cluster_centroids[i] - cluster_centroids[j]) < 0.05:
          error_exist = True
    if error_exist:
      k -= 1
    else:
      break

  print('k is', k)

  predict_cen = km.predict(cen_data)
  print(predict_cen)

  keys = list(range(k))
  for rm in predict_cen[-2:]:
    keys.remove(rm)
  print('keys :', keys)

  skl_rgb = cv2.cvtColor(skl, cv2.COLOR_GRAY2RGB)
  skl_copy = np.copy(skl_rgb)

  # black_plane = np.zeros(skl_rgb.shape).astype(np.uint8)
  black_plane2 = np.zeros(skl_rgb.shape).astype(np.uint8)

  reg_xs = list()
  reg_ys = list()
  for key in keys:
    temp_rx = tuple()
    temp_ry = tuple()
    for pred_key, r_x, r_y in zip(predict_cen[:-2], r_x_list, r_y_list):
      if key == pred_key:
        temp_rx += r_x
        temp_ry += r_y

    #       Regression Multiple Lines     #
    border_x, border_y = regression(org_np3, temp_rx, temp_ry)
    # regression(black_plane, temp_rx, temp_ry)
    # print(border_x, border_y)

    reg_xs.append(border_x)
    reg_ys.append(border_y)

  print('reg_xs :', reg_xs)

  center_xs = list()
  for reg_x in reg_xs:
    #     all_closest_inter가 정렬되지 않았다면 inters type = list()      #
    # sum_x = 0
    # for inter in inters:
    #   sum_x += inter[0] + inter[2]
    center_xs.append(np.mean(reg_x))
  sorted_center_xs = sorted(center_xs)

  sorted_index = list()
  for center_x in sorted_center_xs:
    # print('center_x :', center_x)
    sorted_index.append(center_xs.index(center_x))
  print('sorted_index :', sorted_index)
  #       sorted_center_xs 순서로 all_closest_inter를 정렬한다.     #
  #       index list를 추출해서 for 문으로 all_closest_inter의 inters를 추출해
  #       sorted_all ... 에 append 시킨다.      #

  sorted_reg_xs = list()
  sorted_reg_ys = list()
  for s_index in sorted_index:
    sorted_reg_xs.append(reg_xs[s_index])
    sorted_reg_ys.append(reg_ys[s_index])
  reg_xs = sorted_reg_xs
  reg_ys = sorted_reg_ys

  print('reg_xs :', reg_xs)

  #           VL Segmentation             #
  # Angle between line1 and x-axis
  # print(len(vl_list))

  # print('len(v_lines) :', len(v_lines))
  # print('len(h_lines) :', len(h_lines))

  #         Extend v_lines and draw h_lines       #
  h_border = line((0, 0), (skl_copy.shape[1], 0))
  h_border_lower = line((0, skl_copy.shape[0]), (skl_copy.shape[1], skl_copy.shape[0]))

  #     vline 이 존재하지 않는 경우도 고려해야한다.     #
  if len(reg_xs) == 0:
    reg_xs.append([0, 0])
    reg_ys.append([0, org_np2.shape[0]])

  #         Divide Session by vline       #
  for reg_index, (reg_x, reg_y) in enumerate(zip(reg_xs, reg_ys)):

    #     In a Session     #
    #     1.  find vanishing point    #
    #     2.  Find 3 points (vp, top & bottom points)     #
    #     3.  Do warfine and attach to the black plane      #

    print('reg_index, reg_x, reg_y :', reg_index, reg_x, reg_y)

    #     0.  crop by vline     #
          #   Find Max_x, min_x, (Max_y, min_y = org.shape[0], 0)
          #   1.    우편에 한해서 max_x = 우편 vline max_x & min_x = 현재 vline min_x
          #   2.    좌편에 한해서 max_x = 현재 Max_x & min_x = 좌편 vline min_x

    #      vline 별로 양옆으로 작업을 하면 len(vline) = 1의 작업을 반복할 필요가 없어진다.    #
    iter = False
    while True:

      #                   We need Max, min x & y                #
      # four_inters = list()
      find_pair = True
      # centroid_inters = all_centroid_inters[inters_i]

      if not iter:
        #       vline 우측 session      #
        # print(final_xs)
        # four_inters.append([final_xs[0], final_ys[0]])
        # four_inters.append(centroid_inters[0])
        # print(four_inters)

        #       오른쪽 끝 vline 이면        #
        if reg_index == len(reg_xs) - 1:
          print('reg_index = len(reg_xs) - 1')
          #   1.    우편에 한해서 max_x = 우편 vline max_x & min_x = 현재 vline min_x
          max_x = org_np.shape[1]
          min_x = np.min(reg_x)

        else:
          print('middle vline')
          next_reg_x = reg_xs[reg_index + 1]
          max_x = np.max(next_reg_x)
          min_x = np.min(reg_x)

      #     i = 0 에 한해서만 왼쪽으로도 refering 진행, 나머지는 오른쪽으로만     #
      else:
          #   2.    좌편에 한해서 max_x = 현재 Max_x & min_x = 좌편 vline min_x
          max_x = np.max(reg_x)
          min_x = 0

      print('max_x, min_x :', max_x, min_x)
      # max_y = org_np.shape[0]
      # min_y = 0

      #     Make Session      #
      session = org_np2[:, int(min_x):int(max_x)]

      vpd.find_vps(session)
      # vps = vpd.vps_2D
      vl_img, vl_list = vpd.create_debug_VP_image(show_vl=True)

      #       Closing Morphing by OpenCV      #
      kernel = np.ones((10, 10), np.uint8)
      session = cv2.morphologyEx(session, cv2.MORPH_CLOSE, kernel)

      vpd.find_vps(session)
      # vps = vpd.vps_2D
      vl_session_img, vl_session_list = vpd.create_debug_VP_image(show_vl=True)

      #     Feed Binary inv Threshold     #
      # gray = cv2.cvtColor(session, cv2.COLOR_RGB2GRAY)
      ret, thr_session = cv2.threshold(session, 127, 255, cv2.THRESH_BINARY)
      thr_session = cv2.morphologyEx(thr_session, cv2.MORPH_CLOSE, kernel)
      thr_session_copy = thr_session.copy()
      vpd.find_vps(thr_session_copy)
      # vps = vpd.vps_2D
      vl_thresh_img, vl_thresh_list = vpd.create_debug_VP_image(show_vl=True)

      plt.imshow(vl_thresh_img)
      plt.title('vl_thresh_img')
      plt.show()

      #       Edge Session      #
      #     Multiply mask map with Edge => Erase Unnecessary vl line    #
      edge_session = cv2.Canny(session, 20, 60)

      #     Before Canny    #
      session_prob = session / 255.

      #   multiply one time   #
      session2 = session_prob * (session)
      edge_session2 = cv2.Canny(session2.astype(np.uint8), 20, 60)

      #       two time      #
      # session2 = session_prob**2 * (session)
      # edge_session2 = cv2.Canny(session2.astype(np.uint8), 20, 60)

      vpd.find_vps(edge_session)
      # vps = vpd.vps_2D
      vl_edge_img, vl_edge_list = vpd.create_debug_VP_image(show_vl=True)

      vpd.find_vps(edge_session2)
      # vps = vpd.vps_2D
      vl_edge_img2, vl_edge_list2 = vpd.create_debug_VP_image(show_vl=True)

      # session = wrapper.dealAImage(session,True,True,True)

      # #             color             #
      # session = org_color_np[:, int(min_x):int(max_x)]

      #       Use made vl_img (get more vn_lines)     #
      # session = vl_org_img[:, int(min_x):int(max_x)]

      plt.figure(figsize=(15, 10))
      plt.subplot(161)
      plt.imshow(vl_img)
      plt.title('vl_session_img')

      plt.subplot(162)
      plt.imshow(vl_thresh_img)
      plt.title('vl_thresh_img')

      plt.subplot(163)
      plt.imshow(vl_session_img)
      plt.title('vl_session_img morphed')

      plt.subplot(164)
      plt.imshow(edge_session2)
      plt.title('edge_session2')

      plt.subplot(165)
      plt.imshow(vl_edge_img)
      plt.title('vl_edge_img')

      plt.subplot(166)
      plt.imshow(vl_edge_img2)
      plt.title('vl_edge_img2')
      # plt.imshow(vl_img)
      # plt.title('vl_img')
      plt.show()

      v_lines = list()
      h_lines = list()
      # print('vps :', vps)

      top_vl = list()
      bottom_vl = list()

      left_vl_list = list()
      right_vl_list = list()

      left_angle = list()
      right_angle = list()

      #         What is best vl_list      #
      vl_list = vl_thresh_list

      for vl in vl_list:
        x0, y0, x1, y1 = vl
        slope = (y1 - y0) / float(x1 - x0)
        angle = math.degrees(math.atan(slope))
        # print(angle)
        # if abs(angle) > 80:
        #   # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (255, 0, 0), 3,
        #   #                  cv2.LINE_AA)
        #   v_lines.append(vl)

        if abs(angle) < 70:
          if (y0 + y1) / 2 < vl_img.shape[0] / 2:
            # print(y0, y1)
            top_vl.append(vl)
            if angle < 0:
              left_vl_list.append(vl)
              left_angle.append(abs(angle))
            else:
              right_vl_list.append(vl)
              right_angle.append(abs(angle))
          else:
            bottom_vl.append(vl)
            if angle > 0:
              left_vl_list.append(vl)
              left_angle.append(abs(angle))
            else:
              right_vl_list.append(vl)
              right_angle.append(abs(angle))
          # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 2,
          #                  cv2.LINE_AA)
          h_lines.append(vl)
        #   regression(vl_img, (x0, x1), (y0, y1), color=(0,255,255),axis=1)
      print('#        Original vl list        #')
      print('len(left_vl_list) :', len(left_vl_list))
      print('len(right_vl_list) :', len(right_vl_list))
      print()

      #             Delete over standarded angle vl             #
      if len(left_vl_list) != 0:
        copy_left_vl_list = left_vl_list.copy()
        copy_left_angle = left_angle.copy()
        mean_left_angle = np.array(left_angle).mean()
        print('mean_left_angle :', mean_left_angle)
        print('left_angle :', left_angle)

        for angle_index, angle in enumerate(copy_left_angle):
          if abs(angle - mean_left_angle) > 20:
            left_vl_list.remove(copy_left_vl_list[angle_index])
            left_angle.remove(copy_left_angle[angle_index])
        # standardScaler = StandardScaler()
        # std_left_angle = standardScaler.fit_transform(np.array(left_angle).reshape(-1, 1))
        # print('standardScaler.fit_transform(left_angle) :', std_left_angle)
        # print('left_angle :', left_angle)

        # for std_index, std_angle in enumerate(std_left_angle):
        #   if abs(std_angle) > 2:
        #     left_vl_list.remove(copy_left_vl_list[std_index])
            # left_angle.remove(copy_left_angle[std_index])

        # print('left_angle :', left_angle)

      if len(right_vl_list) != 0:
        copy_right_vl_list = right_vl_list.copy()
        copy_right_angle = right_angle.copy()
        mean_right_angle = np.array(right_angle).mean()
        print('mean_right_angle :', mean_right_angle)
        print('right_angle :', right_angle)

        for angle_index, angle in enumerate(copy_right_angle):
          if abs(angle - mean_right_angle) > 20:
            right_vl_list.remove(copy_right_vl_list[angle_index])
            right_angle.remove(copy_right_angle[angle_index])

      # top_condition = False
      # bot_condition = False
      # for vl in left_vl_list:
      #   if vl in top_vl:
      #     top_condition = True
      #   elif vl in bottom_vl:
      #     bot_condition = True
      # if top_condition and bot_condition:
      #   pass
      # else:
      #   left_vl_list = list()
      #   print('left top bot condition False')
      #   print('top_condition :', top_condition)
      #   print('bot_condition :', bot_condition)

      # top_condition = False
      # bot_condition = False
      # for vl in right_vl_list:
      #   # print('vl :', vl)
      #   if vl in top_vl:
      #     top_condition = True
      #   elif vl in bottom_vl:
      #     bot_condition = True
      # if top_condition and bot_condition:
      #   pass
      # else:
      #   right_vl_list = list()
      #   print('right top bot condition False')
      #   print('top_condition :', top_condition)
      #   print('bot_condition :', bot_condition)

      # print('-- after --')
      # print('len(left_vl_list) :', len(left_vl_list))
      # print('len(right_vl_list) :', len(right_vl_list))
      # print()

      # for vl in vl_list:
      #   x0, y0, x1, y1 = vl

      #   # if right_gap < 30:
      #   #   left_vl_list.append(vl)
      #   #   print('right_angle :', angle)
      #   if vl in left_vl_list:
      #     regression(vl_img, (x0, x1), (y0, y1), color=(0,0,255),axis=1)
      #   # elif left_gap < 30:
      #   elif vl in right_vl_list:
      #     # right_vl_list.append(vl)
      #     # print('left_angle :', angle)
      #     regression(vl_img, (x0, x1), (y0, y1), color=(255,0,0),axis=1)
      if len(left_vl_list) == 0 and len(right_vl_list) == 0:
          #     i != 0 인 경우 break     #
        if reg_index == 0 and not iter:
          iter = True
          print('iter :', iter)
          continue

        else:
          break

      #                       Find External vn_line                     #
      h_border_top = Line((0, 0), (vl_img.shape[1], 0))
      h_border_bot = Line((0, vl_img.shape[0]), (vl_img.shape[1], vl_img.shape[0]))
      top_down_ratio = 0.66
      left_margin = 0.33
      right_margin = 1 - left_margin

      ex_left_top_vl = list()
      ex_left_bot_vl = list()
      ex_right_top_vl = list()
      ex_right_bot_vl = list()

      l2 = Line((0, 0), (0, vl_img.shape[0]))
      l3 = Line((vl_img.shape[1], 0), (vl_img.shape[1], vl_img.shape[0]))

      for vl in left_vl_list:
        x0, y0, x1, y1 = vl
        l1 = Line((x0, y0), (x1, y1))
        #     v_border line     #
        [left_point] = l1.intersection(l2)
        # print('left_point :', left_point)
        [right_point] = l1.intersection(l3)

        top_bot_pass = False
        try:
          [top_point] = l1.intersection(h_border_top)
          [bot_point] = l1.intersection(h_border_bot)
        except:
          top_bot_pass = True
        # print('top_point :', int(top_point[0]), int(top_point[1]))
        # print('bot_point :', int(bot_point[0]), int(bot_point[1]))

        #      Figure out current vl is top / bottom vl     #
        extend = False
        if vl in top_vl:
          #      line should have same height (y)     #
          if (left_point[1] < vl_img.shape[0] * top_down_ratio and right_point[1] < vl_img.shape[0] * top_down_ratio):
            #     h_border 와의 교차점 조건    #
            # if not top_bot_pass:
            # if top_point[0] >= vl_img.shape[1] * right_margin and bot_point[0] <= vl_img.shape[1] * left_margin:
              # extend = True
              ex_left_top_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

        elif vl in bottom_vl:
          # print('vl in bottom at left :', vl)
          #      line should have same height (y)     #
          if (left_point[1] > vl_img.shape[0] * (1 - top_down_ratio) and right_point[1] > vl_img.shape[0] * (1 - top_down_ratio)):
            #     h_border 와의 교차점 조건    #
            # if not top_bot_pass:
            # if top_point[0] <= vl_img.shape[1] * left_margin and bot_point[0] >= vl_img.shape[1] * right_margin:
              # extend = True
              ex_left_bot_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

      for vl in right_vl_list:
        x0, y0, x1, y1 = vl
        l1 = Line((x0, y0), (x1, y1))
        #     v_border line     #

        [left_point] = l1.intersection(l2)
        # print('left_point :', left_point)
        [right_point] = l1.intersection(l3)

        top_bot_pass = False
        try:
          [top_point] = l1.intersection(h_border_top)
          [bot_point] = l1.intersection(h_border_bot)
        except:
          top_bot_pass = True
        # print('top_point :', int(top_point[0]), int(top_point[1]))
        # print('bot_point :', int(bot_point[0]), int(bot_point[1]))

        #      Figure out current vl is top / bottom vl     #
        extend = False
        if vl in top_vl:
          #      line should have same height (y)     #
          if (left_point[1] < vl_img.shape[0] * top_down_ratio and right_point[1] < vl_img.shape[0] * top_down_ratio):
            #     h_border 와의 교차점 조건    #
            # if top_point[0] <= vl_img.shape[1] * left_margin and bot_point[0] >= vl_img.shape[1] * right_margin:
              # extend = True
              ex_right_top_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

        elif vl in bottom_vl:
          # print('vl in bottom at right :', vl)
          #      line should have same height (y)     #
          if left_point[1] > vl_img.shape[0] * (1 - top_down_ratio) and right_point[1] > vl_img.shape[0] * (1 - top_down_ratio):
            #     h_border 와의 교차점 조건    #
            # if top_point[0] >= vl_img.shape[1] * right_margin and bot_point[0] <= vl_img.shape[1] * left_margin:
              # extend = True
              ex_right_bot_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

        # if extend:
        #   ex_right_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))
        # print('ex_left_vl[-1] :', ex_left_vl[-1])

      print()
      print('#          Extended vl list left / right  TB Condition        #')
      print('len(ex_left_top_vl) :', len(ex_left_top_vl))
      print('len(ex_left_bot_vl) :', len(ex_left_bot_vl))
      print('len(ex_right_top_vl) :', len(ex_right_top_vl))
      print('len(ex_right_bot_vl) :', len(ex_right_bot_vl))
      print()

      #       find min max Line     #
      max_y = 0
      right_bottest_vl = None
      min_y = vl_img.shape[0]
      right_toppest_vl = None

      for vl in ex_right_bot_vl:
        if vl[1] > max_y:
          right_bottest_vl = vl
          max_y = vl[1]

      for vl in ex_right_top_vl:
        if vl[1] < min_y:
          right_toppest_vl = vl
          min_y = vl[1]

      max_y = 0
      left_bottest_vl = None
      min_y = vl_img.shape[0]
      left_toppest_vl = None

      for vl in ex_left_bot_vl:
        if vl[3] > max_y:
          left_bottest_vl = vl
          max_y = vl[3]

      for vl in ex_left_top_vl:
        if vl[3] < min_y:
          left_toppest_vl = vl
          min_y = vl[3]

      #       Figure out min_y & max_y of thr_session       #
      # print('thr_session.shape :', thr_session.shape)

      #     바닥은 오히려 하면 안된다.    #
      thr_session_gray = cv2.cvtColor(thr_session, cv2.COLOR_RGB2GRAY)
      print('thr_session_gray.shape :', thr_session_gray.shape)

      top_white_cnt = 0
      bot_white_cnt = 0
      white_min_x = thr_session_gray.shape[1]
      white_max_x = 0
      white_min_y = thr_session.shape[0]
      white_max_y = 0
      temp_min_y = white_min_y
      temp_max_y = white_max_y
      top_parallel = False
      bot_parallel = False
      for i in range(thr_session_gray.shape[1]):

        #     Check Top Condition     #
        for j in range(int(thr_session_gray.shape[0] / 10)):
          # if thr_session_gray[j][i] == 255 and j < thr_min_y:
            # thr_min_y = j
            # thr_min_y_coord = (i, j)
          if thr_session_gray[j][i] == 255:
            top_white_cnt += 1
            if i > white_max_x:
              white_max_x = i
              max_x_coord = (i, j)
            if i < white_min_x:
              white_min_x = i
              min_x_coord = (i, j)
            if j < temp_min_y:
              temp_min_y = j
              temp_min_y_coord = (i, j)

        for j in range(int(thr_session_gray.shape[0] / 10), int(thr_session_gray.shape[0] * 9/10)):
          if thr_session_gray[j][i] == 255:
            if j > white_max_y:
              white_max_y = j
              max_y_coord = (i, j)
            if j < white_min_y:
              white_min_y = j
              min_y_coord = (i, j)

        #     Check Bot Condition   #
        for j in range(int(thr_session_gray.shape[0] * 9/10), thr_session_gray.shape[0]):
          # if thr_session_gray[j][i] == 255 and j < thr_min_y:
            # thr_min_y = j
            # thr_min_y_coord = (i, j)
          if thr_session_gray[j][i] == 255:
            bot_white_cnt += 1
            if j > temp_max_y:
              temp_max_y = j
              temp_max_y_coord = (i, j)
            # if i > white_max_x:
            #   white_max_x = i
            #   max_x_coord = (i, j)
            # if i < white_min_x:
            #   white_min_x = i
            #   min_x_coord = (i, j)

      if top_white_cnt / (thr_session_gray.shape[1] * int(thr_session_gray.shape[0] / 10)) > 0.5:
        top_parallel = True
        min_y_coord = temp_min_y_coord

      if bot_white_cnt / (thr_session_gray.shape[1] * int(thr_session_gray.shape[0] / 10)) > 0.5:
        bot_parallel = True
        max_y_coord = temp_max_y_coord

          # elif thr_session_gray[j][i] == 1 and j > thr_max_y:
          #   thr_max_y = j
          #   thr_max_y_coord = (j, i)
      # print('thr_min_y :', thr_min_y)
      # print('thr_min_y_coord :', thr_min_y_coord)
      try:
        max_x_coord, min_x_coord
        print('max_x_coord, min_x_coord :', max_x_coord, min_x_coord)
      except:
        pass

      #     None 이라면 대칭이동을 통해 만들어주어야한다.     #
      if len(ex_right_bot_vl) == 0:
        if right_toppest_vl is not None:
          x0, y0, x1, y1 = right_toppest_vl
          slope = (y1 - y0) / float(x1 - x0)
          angle = - math.degrees(math.atan(slope))

          #       white_max_y 를 - toppest_vl angle 로 지나는 직선      #
          print('max_y_coord :', max_y_coord)
          print('angle :', angle)
          endy = 100 * math.sin(math.radians(angle))
          endx = 100 * math.cos(math.radians(angle))
          temp_vl = Line(max_y_coord, (max_y_coord[0] + endx, max_y_coord[1] + endy))
          [left_point] = temp_vl.intersection(l2)
          [right_point] = temp_vl.intersection(l3)
          # print('left_point :', left_point)
          right_bottest_vl = (float(left_point[0]), float(left_point[1]), float(right_point[0]), float(right_point[1]))
          print('right_bottest_vl :', right_bottest_vl)

      elif len(ex_right_top_vl) == 0:
        if right_bottest_vl is not None:
          x0, y0, x1, y1 = right_bottest_vl
          slope = (y1 - y0) / float(x1 - x0)
          angle = - math.degrees(math.atan(slope))

          #       white_max_y 를 - toppest_vl angle 로 지나는 직선      #
          print('min_y_coord :', min_y_coord)
          print('angle :', angle)
          endy = 100 * math.sin(math.radians(angle))
          endx = 100 * math.cos(math.radians(angle))
          temp_vl = Line(min_y_coord, (min_y_coord[0] + endx, min_y_coord[1] + endy))
          [left_point] = temp_vl.intersection(l2)
          [right_point] = temp_vl.intersection(l3)
          # print('left_point :', left_point)
          right_toppest_vl = (float(left_point[0]), float(left_point[1]), float(right_point[0]), float(right_point[1]))
          print('right_toppest_vl :', right_toppest_vl)

      try:
        x1, y1, x0, y0 = right_toppest_vl
        cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)
        x1, y1, x0, y0 = right_bottest_vl
        cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)
      except:
        pass

      if len(ex_left_bot_vl) == 0:
        if left_toppest_vl is not None:
          x0, y0, x1, y1 = left_toppest_vl
          slope = (y1 - y0) / float(x1 - x0)
          angle = - math.degrees(math.atan(slope))

          #       white_max_y 를 - toppest_vl angle 로 지나는 직선      #
          print('max_y_coord :', max_y_coord)
          print('angle :', angle)
          endy = 100 * math.sin(math.radians(angle))
          endx = 100 * math.cos(math.radians(angle))
          temp_vl = Line(max_y_coord, (max_y_coord[0] + endx, max_y_coord[1] + endy))
          [left_point] = temp_vl.intersection(l2)
          [right_point] = temp_vl.intersection(l3)
          # print('left_point :', left_point)
          left_bottest_vl = (float(left_point[0]), float(left_point[1]), float(right_point[0]), float(right_point[1]))
          print('left_bottest_vl :', left_bottest_vl)

      elif len(ex_left_top_vl) == 0:
        if left_bottest_vl is not None:
          x0, y0, x1, y1 = left_bottest_vl
          slope = (y1 - y0) / float(x1 - x0)
          angle = - math.degrees(math.atan(slope))

          #       white_max_y 를 - toppest_vl angle 로 지나는 직선      #
          print('min_y_coord :', min_y_coord)
          print('angle :', angle)
          endy = 100 * math.sin(math.radians(angle))
          endx = 100 * math.cos(math.radians(angle))
          temp_vl = Line(min_y_coord, (min_y_coord[0] + endx, min_y_coord[1] + endy))
          [left_point] = temp_vl.intersection(l2)
          [right_point] = temp_vl.intersection(l3)
          # print('left_point :', left_point)
          left_toppest_vl = (float(left_point[0]), float(left_point[1]), float(right_point[0]), float(right_point[1]))
          print('left_toppest_vl :', left_toppest_vl)

      try:
        x1, y1, x0, y0 = left_bottest_vl
        cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (255, 255, 0), 3, cv2.LINE_AA)
        x1, y1, x0, y0 = left_toppest_vl
        cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (255, 255, 0), 3, cv2.LINE_AA)
      except:
        pass

      # 일단은 위아래 둘다 있는 조건 성립 #
      #   있는 쪽만 사용, 양방향 소실점 다 존재하면 소실선 개수가 더 많은 쪽 사용   #
      # left_condition = left_bottest_vl is not None and left_toppest_vl is not None
      # right_condition = right_bottest_vl is not None and right_toppest_vl is not None

      # if left_condition or right_condition:
        # print('left_bottest_vl', left_bottest_vl)
      if len(left_vl_list) >= len(right_vl_list):
        #       toppest & bottest vl should be on the min_y & max_y of white mask       #
        # x1, y1, x0, y0 = left_bottest_vl
        # cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)
        x1, y1, x0, y0 = left_toppest_vl
        # cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)

        # if thr_min_y == 0:
        #   thr_min_x = thr_session_gray.shape[1]
        #   for i in range(thr_session_gray.shape[1]):
        #       if thr_session_gray[0][i] == 255 and i < thr_min_x:
        #         thr_min_x = i
        #         thr_min_y_coord = (i, 0)
        #         break
        #   print('thr_min_y_coord :', thr_min_y_coord)
        if top_parallel:
          top_line = Line((x1, y1), (x0, y0))
          print('top_line.slope :', float(top_line.slope))
          if abs(top_line.slope) < 0.05:
              max_x_coord = (0, 0)
          parallel_tl = top_line.parallel_line(min_x_coord)
          [tl] = parallel_tl.intersection(l2)
          [tr] = parallel_tl.intersection(l3)
          br, bl = left_bottest_vl[2:], left_bottest_vl[:2]
        else:
          tl, tr, br, bl = left_toppest_vl[:2], left_toppest_vl[2:], left_bottest_vl[2:], left_bottest_vl[:2]

      else:
        # x1, y1, x0, y0 = right_bottest_vl
        # cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)
        x1, y1, x0, y0 = right_toppest_vl
        # cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)

        # if thr_min_y == 0:
        #   thr_max_x = 0
        #   for i in range(thr_session_gray.shape[1] - 1, 0, -1):
        #       print('i :', i)
        #       if thr_session_gray[0][i] == 255 and i > thr_max_x:
        #         thr_max_x = i
        #         thr_min_y_coord = (i, 0)
        #         break
        #   print('thr_min_y_coord :', thr_min_y_coord)
        if top_parallel:
          top_line = Line((x1, y1), (x0, y0))
          print('top_line.slope :', float(top_line.slope))
          if abs(top_line.slope) < 0.05:
              max_x_coord = (thr_session_gray.shape[1], 0)
          parallel_tl = top_line.parallel_line(max_x_coord)
          [tl] = parallel_tl.intersection(l2)
          [tr] = parallel_tl.intersection(l3)
          br, bl = right_bottest_vl[2:], right_bottest_vl[:2]
        else:
          tl, tr, br, bl = right_toppest_vl[:2], right_toppest_vl[2:], right_bottest_vl[2:], right_bottest_vl[:2]

      #       Refering      #
      refered = np.asarray(refer)[:vl_img.shape[0], :vl_img.shape[1]]

      # plt.imshow(refered)
      # plt.title('refered')
      # plt.show()

      #   tl, tr, br, bl    #
      #     refer를 위해 src_x => 0 으로 맞춰준다.    #
      src = np.array([
          [0, 0],
          [vl_img.shape[1], 0],
          [vl_img.shape[1], vl_img.shape[0]],
          [0, vl_img.shape[0]]], dtype = "float32")
      dst = np.array([list(tl),
                      list(tr),
                      list(br),
                      list(bl)], dtype = "float32")

      print()
      print("#          Warping Points        #")
      print('src :', src)
      print('dst :', dst)

      # compute the perspective transform matrix and then apply it
      matrix = cv2.getPerspectiveTransform(src, dst)
      refered = cv2.warpPerspective(refered, matrix, (refered.shape[1], refered.shape[0]))

      plt.subplot(121)
      plt.imshow(vl_img)
      plt.title('Session_vl_img')
      # plt.xlim(np.min(vps[:, [0]]), np.max(vps[:, [0]]))
      # plt.ylim(np.min(vps[:, [1]]), np.max(vps[:, [1]]))
      plt.subplot(122)
      plt.imshow(refered)
      plt.show()

      org_color_np2[:, int(min_x):int(max_x)] = refered

      #     i != 0 인 경우 break     #
      if reg_index == 0 and np.sum(reg_xs) != 0 and not iter:
        iter = True
        print('iter :', iter)

      else:
        break

  #         Refer의 검은 부분은 original image로 채운다.       #
  for i in range(org_color_np2.shape[1]):
    for j in range(org_color_np2.shape[0]):
      if sum(org_color_np2[j][i]) == 0:
        org_color_np2[j][i] = org_color_np[j][i]

  print('elapsed time :', time.time() - start)
  print()

  plt.figure(figsize=(15, 10))
  plt.subplot(131)
  plt.imshow(org_np3)

  plt.subplot(132)
  plt.imshow(org_color_np2)
  # plt.show()

  # print('np.max(org_np) :', np.max(org_np))
  org_np = org_np/ 255.
  refer = org_color_np2

  #     1.wall_mask * reference color image + (1 - wall_mask) * original_image)    #
  syn_ = org_np * (refer) + (1 - org_np) * org_color_np
  syn = org_np * syn_ + (1 - org_np) * org_color_np
  plt.subplot(133)
  plt.imshow(syn.astype(np.uint8))
  plt.show()

  if one_iter:
    break
  # break
```

```

  # v_border = line((0, 0), (0, skl_copy.shape[0]))
  # v_border_lower = line((skl_copy.shape[1], 0), (skl_copy.shape[1], skl_copy.shape[0]))
  # for h_line in h_lines:

  #     #       Find Intersection between v_line, h_line      #
  #     vh_intersections_x = list()
  #     vh_intersections_y = list()

  #     hline = line(h_line[:2], h_line[2:])
  #     for reg_x, reg_y in zip(reg_xs, reg_ys):
  #       vline = line((reg_x[0], reg_y[0]), (reg_x[1], reg_y[1]))

  #       #     Extract only x - coordination     #
  #       vh_intersections_x.append(intersection2(vline, hline)[0])
  #       vh_intersections_y.append(intersection2(vline, hline)[1])

  #     # h_x = np.array([h_line[0], h_line[2]])
  #     # h_y = np.array([h_line[1], h_line[3]])
  #     # ex_h_x, ex_h_y = extended(h_x, h_y, 500)
  #     # ex_h_line = line((int(h_x[0]), int(h_y[0])), (int(h_x[1]), int(h_y[1])))
  #     r = intersection2(v_border, hline)
  #     r_lower = intersection2(v_border_lower, hline)

  #     vh_intersections_x.append(r[0])
  #     vh_intersections_y.append(r[1])
  #     vh_intersections_x.append(r_lower[0])
  #     vh_intersections_y.append(r_lower[1])

  #     sorted_vh_inter_x = sorted(vh_intersections_x)
  #     # print('vh_intersections_x :', vh_intersections_x)
  #     # print('sorted_vh_inter_x :', sorted_vh_inter_x)

  #     center_h_x = (h_line[0] + h_line[2]) / 2
  #     # print('center_h_x :', center_h_x)

  #     #       hline 상의 교차점을 찾아내서 범위 내에서 연결한다.      #
  #     for i in range(1, len(sorted_vh_inter_x)):

  #       if sorted_vh_inter_x[i - 1] <= center_h_x <= sorted_vh_inter_x[i]:
  #         lx, ly = sorted_vh_inter_x[i - 1], vh_intersections_y[vh_intersections_x.index(sorted_vh_inter_x[i - 1])]
  #         rx, ry = sorted_vh_inter_x[i], vh_intersections_y[vh_intersections_x.index(sorted_vh_inter_x[i])]
  #         # print('lx, ly, rx, ry :', lx, ly, rx, ry)
  #         #       이곳의 lx, ly, rx, ry 는 close_p 다. 좌우로 나뉘어진 lx, ly가 아니다.     #

  #           #       정제된 교차점을 만드는 hline만 사용해야한다.        #
  #         for inters in all_closest_inters:
  #           for inter in inters:
  #             if lx in inter or rx in inter:
  #               h_intersections.append((lx, ly, rx, ry))
  #               # cv2.line(black_plane, (int(lx), int(ly)), (int(rx), int(ry)), (0, 0, 255), 1, cv2.LINE_AA)

  #       vline 기준으로 구획해야한다. -> 가장 왼쪽 / 오른쪽 vline + 구역 먼저 찾아 작업하기     #
  #       vline 별로 구획 나누기      #
  # h_intersections = list(set(h_intersections))
  # print('np.array(all_closest_inters).shape :', np.array(all_closest_inters).shape)
  # all_closest_inters = np.array(all_closest_inters)
  # print('len(all_closest_inters) :', len(all_closest_inters))

  # center_xs = list()
  # for inters in all_closest_inters:
  #   #     all_closest_inter가 정렬되지 않았다면 inters type = list()      #
  #   sum_x = 0
  #   for inter in inters:
  #     sum_x += inter[0] + inter[2]
  #   center_xs.append(sum_x / len(inters) * 2)
  # sorted_center_xs = sorted(center_xs)

  # sorted_index = list()
  # for center_x in sorted_center_xs:
  #   # print('center_x :', center_x)
  #   sorted_index.append(center_xs.index(center_x))
  # # print('sorted_index :', sorted_index)
  # #       sorted_center_xs 순서로 all_closest_inter를 정렬한다.     #
  # #       index list를 추출해서 for 문으로 all_closest_inter의 inters를 추출해
  # #       sorted_all ... 에 append 시킨다.      #
  # sorted_all_closest_inters = list()
  # sorted_all_centroid_inters = list()
  # for s_index in sorted_index:
  #   sorted_all_closest_inters.append(all_closest_inters[s_index])
  #   sorted_all_centroid_inters.append(all_centroid_inters[s_index])
  # all_closest_inters = sorted_all_closest_inters
  # all_centroid_inters = sorted_all_centroid_inters

```

```
#                                   Refering                                 #
      #     Find Top left / right & Bottom left / right       #
      #     ->  tl, tr, bl, br     #
      four_inters = np.array(four_inters)
      [tl, tr, br, bl] = order_points(four_inters)
      # print(four_inters.shape)
      top_length = math.hypot(tl[0] - tr[0], tl[1] - tr[1])
      bottom_length = math.hypot(bl[0] - br[0], bl[1] - br[1])
      max_hlength = max(top_length, bottom_length)

      #     1. Compare left / right height     #
      l_height = bl[1] - tl[1]
      r_height = br[1] - tr[1]

      #     2. Extend Shorter Height      #
      if l_height <= r_height:
        shorter_points = [bl, tl]
        longer_points = [br, tr]
      else:
        shorter_points = [br, tr]
        longer_points = [bl, tl]

      ex_shorter_points = np.zeros_like(shorter_points)
      ex_longer_points = np.zeros_like(longer_points)

      print('shorter_points :', shorter_points)
      x = (shorter_points[0][0], shorter_points[1][0])
      y = (shorter_points[0][1], shorter_points[1][1])
      print('shorter x, y :', x, y)

      y_ext = np.array([org_np.shape[0], 0])
      p = np.polyfit(y, x , deg=1)
      x_ext = np.poly1d(p)(y_ext)

      ex_shorter_points[0] = [x_ext[0], y_ext[0]]
      ex_shorter_points[1] = [x_ext[1], y_ext[1]]

      #     Find intersection between (longer + shorter points)'s parallel line and longer line   #
      long_short_lower = Line(longer_points[0], shorter_points[0])
      long_short = Line(longer_points[1], shorter_points[1])
      p_line_lower = long_short_lower.parallel_line(ex_shorter_points[0])
      p_line = long_short.parallel_line(ex_shorter_points[1])
      longer_line = Line(longer_points[0], longer_points[1])

      # print('ex_shorter_points[1] in p_line ? :', ex_shorter_points[1] in p_line)
      # print('p_line_lower.intersection(longer_line) :', p_line_lower.intersection(longer_line)[0][0])
      ex_longer_points[0] = [p_line_lower.intersection(longer_line)[0][0], p_line_lower.intersection(longer_line)[0][1]]
      ex_longer_points[1] = [p_line.intersection(longer_line)[0][0], p_line.intersection(longer_line)[0][1]]

      if l_height <= r_height:
        ex_bl, ex_tl = ex_shorter_points
        ex_br, ex_tr = ex_longer_points
      else:
        ex_br, ex_tr = ex_shorter_points
        ex_bl, ex_tl = ex_longer_points

      for ex_p in [ex_bl, ex_br, ex_tl, ex_tr]:
        print(ex_p)
        # cv2.circle(black_plane2, (int(ex_p[0]), int(ex_p[1])), 10, (255, 0, 255), -1)

      min_x, max_x = min(ex_bl[0], ex_tl[0]), max(ex_br[0], ex_tr[0])
      min_y, max_y = min(ex_tl[1], ex_tr[1]), max(ex_bl[1], ex_br[1])
      src_x = max((max_x - min_x), max_hlength) * math.sqrt(2)
      print('src_x :',  src_x)

      refered = np.asarray(refer)[:int(max_y - min_y), :int(src_x)]

      # plt.imshow(refered)
      # plt.title('refered')
      # plt.show()

      #   tl, tr, br, bl    #
      #     refer를 위해 src_x => 0 으로 맞춰준다.    #
      src = np.array([
          [0, 0],
          [src_x, 0],
          [src_x, max_y - min_y],
          [0, max_y - min_y]], dtype = "float32")
      dst = np.array([
                      [ex_tl[0] - min_x, ex_tl[1] - min_y],
                      [ex_tr[0] - min_x, ex_tr[1] - min_y],
                      [ex_br[0] - min_x, ex_br[1] - min_y],
                      [ex_bl[0] - min_x, ex_bl[1] - min_y]], dtype = "float32")

      print('src :', src)
      print('dst :', dst)

      # compute the perspective transform matrix and then apply it
      matrix = cv2.getPerspectiveTransform(src, dst)
      refered = cv2.warpPerspective(refered, matrix, (refered.shape[1], refered.shape[0]))

      # print('refered.min() :', refered.min())
      # print(refered[[0], [0]])
      # print('refered.shape :', refered.shape)
      # plt.imshow(refered)
      # plt.title('warfed')
      # plt.xlim(min_x, max_x)
      # plt.xlim(0, max_x - min_x)
      # plt.ylim(max_y, min_y)
      # plt.xlim(-1000, 1000)
      # plt.ylim(1000, -1000)
      # plt.show()

      refer_crop = refered[int(abs(min_y)): int(abs(min_y)) + org_np.shape[0], :]
      # print('refer_crop.shape :', refer_crop.shape)

      # plt.imshow(refer_crop)
      # plt.title('cropped refer')
      # plt.show()

      #     Crop Section From Cropped Refer     #
      #         src_x => 0으로 맞춰줬던 것을 되돌린다.      #
      print('int(min_x) :', int(min_x))
      print('int(min_y) :', int(min_y))

      for i in range(refer_crop.shape[1]):
        for j in range(refer_crop.shape[0]):
          if sum(refer_crop[j][i]) != 0:
            if i + int(min_x) < black_plane2.shape[1]:
              black_plane2[j][i + int(min_x)] = refer_crop[j][i]
```

Layout Estimation by vanishing line Func.ipynb


```python
import theme
theme.load_style()
```

# Lesson 5: Formulations for 1D Problems
## The Strong Form

This lecture by Tim Fuller is licensed under the
Creative Commons Attribution 4.0 International License.  All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

# Topics

- [The Strong Form](#strong-form)
  - [Axially Loaded Elastic Bar](#axibar)
  - [Heat Conduction](#heatcond)
  - [Electrical Conduction](#eleccond)
- [Flux-Potential Relationship](#fluxpotrel)

# The Strong Form[](#top)

## Axially Loaded Elastic Bar

Consider the tapered bar

Summing forces on the differential volumes gives

$$-p(x) + b\left(x+\frac{\Delta x}{2}\right)\Delta x + p\left(x+\Delta x\right) = 0$$

and taking the limit as $\Delta x \rightarrow 0$ leads to the following differential relationship

$$\lim_{\Delta x \rightarrow 0} \frac{p\left(x + \Delta x\right) -p\left(x\right)}{\Delta x} + b\left(x+\frac{\Delta x}{2}\right) = \frac{dp}{dx} + b(x) = 0$$

Solution of the differential equation requires a *constitutive equation* relating $p(x)$ to $u(x)$.  Recalling that

$$\sigma(x) = \frac{p(x)}{A(x)} \Rightarrow p(x) = \sigma(x) A(x)$$

and, for a simple linear elastic material, the stress $\sigma$ is given by Hooke's law of linear elasticity†

$$\sigma(x)=E(x)\epsilon(x)$$

where the strain $\epsilon$ is given by

$$\epsilon(x) = \frac{\Delta L}{L} = \lim_{\Delta x\rightarrow 0} \frac{u(\Delta x) - u(x+\Delta x)}{\Delta x} = \frac{du}{dx}$$

Substituting the above expressions in to the differential relationship leads to

$$\frac{d}{dx}\left(A(x)E(x)\frac{du}{dx}\right)+b(x)=0$$

The above is a second-order ordinary differential equation. In the above equation, $u(x)$ is the dependent variable, which is the unknown function, and $x$ is the independent variable.

To solve the strong form of the governing equation, boundary conditions at each end of the bar must be prescribed.  For the purpose of illustration, we will consider the following specific boundary conditions: at $x=0$, the displacement, $u(x=0)$, is prescribed; at $x=L$, the force per unit area, or traction, denoted by 􏰢$t$, is prescribed. These conditions are written as

$$
\begin{align}
u(x=0) &= \overline{u} \\
\sigma(x=L) &= \left(E\frac{du}{dx}\right)_{x=L}=\frac{F(L)}{A(L)} \equiv \overline{t}
\end{align}
$$

The traction $􏰢t$ has the same units as stress (force/area), but its sign is positive when it acts in the positive $x$-direction regardless of which face it is acting on, whereas the stress is positive in tension and negative in compression, so that on a negative face a positive stress corresponds to a negative traction.

The differential equation governing the response of the elastic bar along with the associated boundary conditions is called the **strong form** of the governing equation and is summarized here

The strong form

$$
\begin{split}
\frac{d}{dx}\left(A(x)E(x)\frac{du}{dx}\right)+b(x)=0 \\
u(x=0) = \overline{u} \\
\left(E\frac{du}{dx}\right)_{x=L} = \overline{t}
\end{split}
$$

**Note**
- The problem data are $\overline{u}$, $\overline{t}$, and $b(x)$ and are given
- Either the load or the displacement can be specified at a boundary point, but not both.

†For thermo-elastic materials, the constitutive model is modified to include thermal effects $\sigma(x) = E(x)\left(\epsilon(x) - \alpha(x)T(x)\right)$

Remark

In the previous example there are two types of boundary conditions: one in which the dependent variable ($u$) was specified on the boundary and at the other boundary its derivative was $\left(E du/dx\right)$.  These boundary conditions are classified as Dirichlet and Neummann, respectively.

## Heat Conduction

Heat flow in a body occurs when there is a temperature difference within the body. Heat transferred in this way is referred to as conduction.  The heat flow through the wall of a heated room in the winter is an example of conduction.  In this section, the strong form of the differential equation governing heat transfer through conduction will be developed.  Other forms of heat transfer such as convection will be discussed in later chapters.

Consider a body as shown above. The objective is to determine the temperature distribution. Let $A(x)$ be the area normal to the direction of heat flow and let $s(x)$ be the heat generated per unit thickness of the body, denoted by l. This is often called a heat source. A common example of a heat source is the heat generated in an electric wire due to resistance. In the one-dimensional case, the rate of heat generation is measured in units of energy per time; in SI units, the units of energy are joules (J) per unit length (meters, m) and time (seconds, s). Recall that the unit of power is watts (1 W = 1 J/s). A heat source $s(x)$ is considered positive when heat is generated, i.e. added to the system, and negative when heat is withdrawn from the system. Heat flux, denoted by $q(x)$, is defined as a the rate of heat flow across a surface. Its units are heat rate per unit area; in SI units, W/m$^2$. It is positive when heat flows in the positive x-direction. We will consider a steady-state problem, i.e. a system that is not changing with time.

To establish the differential equation that governs the system, we consider energy balance in a control volume of the wall. Energy balance requires that the rate of heat energy ($qA$) that is generated in the control volume must equal the heat energy leaving the control volume, as the temperature, and hence the energy in the control volume, is constant in a steady-state problem. The heat energy leaving the control volume is the difference between the flow in at on the left-hand side, $qA$, and the flow out on the right-hand side, $q(x + \Delta x)A(x+\Delta x)$. Thus, energy balance for the control volume can be written as

$$s\left(x+\frac{\Delta x}{2}\right)\Delta x + q(x)A(x) -  q\left(x+\Delta x\right)A(x+ \Delta x) = 0$$

Rearranging and taking the limit as $\Delta x \rightarrow 0$ leads to the following differential relationship

$$\lim_{\Delta x \rightarrow 0} \frac{q\left(x + \Delta x\right)A(x+\Delta x) -q\left(x\right)A(x)}{\Delta x} - s\left(x+\frac{\Delta x}{2}\right) = \frac{d(qA)}{dx} - s = 0$$

The constitutive equation for heat flow, which relates the heat flux to the temperature, is known as Fourier’s law and is given by

$$
q = -k \frac{dT}{dx}
$$

where $T$ is the temperature and $k$ is the thermal conductivity (which must be positive); in SI units, the dimensions of thermal conductivity are W/m$\cdot$oC. A negative sign appears because heat flows from high (hot) to low temperature (cold), i.e. opposite to the direction of the gradient of the temperature field.  Substituting Fourier's law gives

$$\frac{d}{dx}\left(Ak\frac{dT}{dx}\right) + s = 0, \quad 0 < x < L$$

Either the flux or the temperature must be prescribed on each end of the bar; these are the boundary conditions. We consider the specific boundary conditions of the prescribed temperature $\overline{T}$ at $x=0$ and prescribed flux $\overline{q}$ at $x=L$. The prescribed flux $\overline{q}$ is positive if heat (energy) flows out of the bar, i.e. $q(x=L)=-\overline{q}$. The strong form for the heat conduction problem is then given by

The strong form

$$\begin{split}
\frac{d}{dx}\left(Ak\frac{dT}{dx}\right) + s = 0, \quad 0 < x < L \\
T=\overline{T} \text{ on } x=0 \\
-q=k\frac{dT}{dx} = \overline{q} \text{ on } x=L\\
\end{split}
$$

##  Electrical Conduction

Consider the tapered bar with electrical potential as shown

Conservation of charge requires that

$$i(x) - i\left(x+\Delta x\right) = 0$$

rearranging and taking the limit as $\Delta x \rightarrow 0$ leads to the following differential relationship

$$\lim_{\Delta x \rightarrow 0} \frac{i\left(x\right) -i\left(x + \Delta x\right)}{\Delta x} = -\frac{di}{dx} = 0$$

Ohm's law gives the constitutive equation relating voltagel and current

$$\begin{split}
V(x) - V(x+\Delta x) = i \frac{\rho(\Delta x)}{A\left(x+\frac{\Delta x}{2}\right)} \\
-\frac{1}{\rho}A(x)\frac{dV}{dx} = i
\end{split}$$

where $\rho$ is the resistivity.  Substituting the above expressions in to the differential relationship leads to

$$\frac{d}{dx}\left(\frac{1}{\rho}A(x)\frac{dV}{dx}\right)=0$$

which is the strong from of the governing equation.

Boundary conditions take the form of either a prescribed voltage or current at either end.  The strong form for electrical conduction with voltage prescribed at $x=0$ and current prescribed at $x=L$ is given by

The Strong Form

$$
\begin{split}
\frac{d}{dx}\left(\frac{1}{\rho}A(x)\frac{dV}{dx}\right)=0 \\
V(x=0) = \overline{V} \\
\left(\frac{1}{\rho}A\frac{dV}{dx}\right)_{x=L} = \overline{i}
\end{split}
$$

Remark

To find the resistance of the entire element

 Prescribe a current BC $i(0) = \overline{i} = 1A$

 Prescribe a voltage BC $V(L) = 0$

 Solve for V(0)

 Sove for resistance $R=\frac{V(0) - V(L)}{i(0)} = \frac{V(0)}{1 A}$

# Flux Potential Relationships[](#top)

## What is being fluxed?

Flux is the rate of "something" per unit area.  To determine what is being fluxed, multiply flux by area to identify the rate of the thing being fluxed

### Heat

The flux, $-qn$,  has units W/m$^2$.  Rate of the “thing” fluxed has units W.
A watt, W, has units of Joules/s.  The thing fluxed has units of Joules.

 The thing fluxed is energy!

The heat equation is often called "energy balance"

Steady state: rate of energy is zero!

### Elasticity

The flux $\bar{t}$ has units N/m$^2$.  Rate of the “thing” fluxed has units N.
A Newton, N, has units kg m/s$^2$. The thing fluxed has units of kg m/s.

That’s mass times velocity!

The thing fluxed is momentum!

The elasticity equation is often called “momentum balance”

Equilibrium: rate of momentum is zero!


```python
import theme
theme.load_style()
```

# Lesson 7: The Weak Form

This lecture by Tim Fuller is licensed under the
Creative Commons Attribution 4.0 International License.  All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

#  Topics
- [Introduction](#intro)
- [Continuity](#continuity)
- [Weighted Integral Formulations](#weight-forms)
  - [Three Steps to Develop the Weak Form](#three-steps)
    - [Step 1](#step-1)
    - [Step 2](#step-2)
    - [Step 3](#step-3)
- [Summary](#summary)
- [Examples](#examples)
  - [Example 1](#ex-1)
  - [Example 2](#ex-2)

#  Introduction[](#top)
> The weak form is the most intellectually challenging part in the development of finite elements, so a student may encounter some difficulties in understanding this concept

> Understanding how a solution to a differential equation can be obtained by this rather abstract statement, and why it is a useful solution, is not easy. It takes most students considerable thought and experience to comprehend the process.

> - Fish and Belytschko, 2007

The weighted integral formulations of the previous lesson allowed finding approximate solutions to the accompanying governing differential equation.  Similarly, the finite element method is a technique for constructing approximate solutions to differential equations.  To understand the foundations of the method, it is necessary to understand the weighted-integral formulations and the *weak* formulation of the differential equations.  A weak form of a differential equation is equivalent to the governing equation and boundary conditions, i.e. the strong form.  It allows classifying boundary conditions as essential and natural.

# Continuity[](#top)

Before developing the weak form, let us discuss the concept of smoothness, or continuity.  In general, a $k^{\rm th}$ order ODE

$$
 au^{(k)} + a_{k-1}u^{(k-1)}+\cdots++ a_{2}u''+ a_{1}u' + a_0=0
$$

requires that

$$
 u\in C^{k-1}
$$

which means that $u$ has $k-1$ continuous derivatives. For example, if

$$
 \frac{d^2u}{dx^2}=0
$$

then $u\in C^1$, or the first derivative of $u$ must be continuous. Let's see if this is true.

$$
 \frac{d^2u}{dx^2} = 0 \Rightarrow u=c_1x + c_2
$$

and

$$
 u'=c_1
$$

sure enough, $u'$ is continuous and $u\in C^1$

A function is a $C^k$ function if its derivatives of order $j$ for $0 \leq j \leq k$ exist and are continuous functions in the entire domain.  In finite element analysis, we are concerned mostly with $C^0$, $C􏰂^{-1}$ and $C^1$ functions

As can be seen, a $C^0$ function is piecewise continuously differentiable, i.e. its first derivative is continuous except at selected points. The derivative of a $C^0$ function is a $C^{-􏰂1}$ function. So for example, if the displacement is a $C^0$ function, the strain is a $C^{-1}$ function.  In general, the derivative of a $C^n$ function is $C^{n-􏰂1}$.

#  Weighted-Integral and Weak Formulations[](#top)

Consider the following differential equation

$$
-\frac{d}{dx}\left(a(x)\frac{du}{dx}\right) = q(x) \quad
\text{for } 0 < x < L
$$

The solution $u(x)$ is subject to the boundary conditions

$$
u(0) = u_0, \quad \left.\left(a\frac{du}{dx}\right)\right|_{x=L}=Q_0
$$

where $a$ and $q$ are known functions of $x$, $u_0$ and $Q_0$ are known values, and $L$ is the length of the domain.  These are the problem data.  The solution $u$ is the dependent variable of the problem.  When $u_0\neq0$ and/or $Q_0\neq0$, the boundary conditions are nonhomogeneous and homogeneous otherwise.

The sole purpose of developing the weighted-integral statement was to be able to obtain $N$ linearly independent algebraic relations among the coefficients of the approximation

$$
u\approx U_N = \sum_{i=1}^N c_i\phi_i(x) + \phi_0(x)
$$

##  Three Steps to Form the Weak Form

Reddy describes the following three steps to developing the weak form, if it exists, to a differnital equation:

1. Multiply the equations by a weight function $w$ and integrate over the problem domain.
2. Distribute differentiation among the dependent variable and weight function.
3. Impose the boundary conditions.

###  Step 1

Move all expressions to one side, multiply by the weight function, and integrate over the domain

$$
\int_0^L w\left(-\frac{d}{dx}\left(a\frac{du}{dx}\right)-q\right)dx = 0, \quad \forall w
$$

The expression in the outer parenthesis is not zero if $u$ is replaced by an approximation.  Thus, the above expression is a statement that the error in the differential equation due to the approximation is zero in the weighted-integral sense.

The weight function $w$ can be any nonzero, integrable function.  The weighted-integral statement is equivalent only to the differential equation, and it does not include any boundary conditions.

The statement that the weighted integral equation must hold for all $w$ ($\forall w$) is not arbitrary.  If it were not for that statement, none of the following would hold true.

### Step 2

With the weighted-integral statement, $N$ algebraic relations among the $c_i$ for $N$ different choices of $w$ can be obtained.  But, the approximation functions $\phi_i$ must still be chosen such that $U_N$ is differentiable as many times as required by the original differential equation and satisfies the original boundary conditions.  If the differentiation is distributed between the approximate solution $U_N$ and the weight function $w$, the resulting integral form will require weaker continuity on $\phi_i$ and, hence, the weighted-integral statement is called the *weak form*.

Starting with the integral statement, we integrate the first term in the expression by parts to obtain

$$
\begin{align}
0 &= \int_0^L w\left(-\frac{d}{dx}\left(a\frac{du}{dx}\right)-q\right)dx \\
&= \int_0^L w\left(\frac{dw}{dx}a\frac{du}{dx}-wq\right)dx - \left[wa\frac{du}{dx}\right]_0^L\\
\end{align}
$$

The integration by parts formula is given by
$$
\int_0^L wdv = -\int_0^Lvdw + \left[wv\right]_0^L
$$

Note that the weight function is required to be differentiable at least once.

From the above expression, we can identify two types of boundary conditions: *natural* and *essential*.  The following rule is used to identify the natural boundary conditions: after completing Step 2, the coefficients of the weight function and its derivatives in the boundary expressions are called the *secondary variables*.  Specification of the secondary variables on the boundary constitutes the *natural boundary conditions*.  For this case, the coefficient of the weight function (the secondary variable) is $adu/dx$.

The secondary variables always have a physical meaning.  In the case of the axial bar problem, the secondary variable represents a traction $t$.  The secondary variable will be denoted by

$$
Q = \left(a\frac{du}{dx}\right)n_x
$$

where $n_x$ is the direction cosine:

$$
\begin{align}
n_x = &\text{cosine of the angle between the } x \\
      & \text{ axis and the normal to the boundary}
\end{align}
$$

For 1D problems, $n_x=-1$ on the left end and $n_x=1$ at the right end of the domain.

The dependent variable of the problem, expressed in the same form as the weight function appearing in the boundary term is called the *primary variable*.  Its specification on the boundary consitutes the *essential boundary condition*.  For our example, the dependent variable $u$ is the primary variable and the essential boundary condition is the specification of $u$ at the boundary points.

The number of primary and secondary variables is always the same, and each primary variable has an associated secondary variable (e.g., temperature and heat, displacement and force, etc.).  Only one of the pair may be specified at a point on a boundary.

In light of the above, a given problem can have boundary conditions in one of three categories:

i) all boundary conditions are essential
ii) some of the boundary conditions are essential with the rest being natural
iii) all boundary conditions are natural

With our notation, the weak form is written

$$
0 = \int_0^L\left(a\frac{dw}{dx}\frac{du}{dx}-wq\right)dx - (wQ)_0 - (wQ)_L
$$

"Weak" refers to the reduced (weakened) continuity of $u$ which is now required only to be once differentiable, instead of twice.

### Step 3

We now impose the boundary conditions.  We require that the weight function $w$ vanish at boundary points where the essential boundary conditions are specified.  In other words, $w$ is required to satisfy the homogenous form of the specified essential boundary conditions of the problem.  Why is this the case?  In the weak formulation, the weight unction has the meaning of a *virtual* change of the primary variable.  When the primary variable is prescribed at a point, the virtual change must be 0.  For our example problem, the essential boundary condition is $u=u_0$; thus, the weight function $w$ is required to satisfy

$$
w(0) = 0, \quad \text{because } u(0) = u_0
$$

Since $w(0) = 0$ and

$$
Q(L) = \left(a\frac{du}{dx}n_x\right)\Bigg|_{x=L}=
\left(a\frac{du}{dx}\right)\Bigg|_{x=L} = Q_0
$$

our example problem reduces to

$$
0 = \int_0^L w\left(a\frac{dw}{dx}\frac{du}{dx}-wq\right)dx - w(L)Q_0
$$

which is the weak form equivalent to the original differential equation with the associated natural boundary condition.

#  Summary[](#top)

1. The weak form of a differential equation is a weighted integral statement that is equivalent to the strong form *and* the specified natural boundary conditions.
2. The weak form is developed by multiplying the differential equation by an arbitrary weight function, distributing differentiation among the primary variable and the weight function through integration by parts, and finally applying the boundary conditions.
3. After integrating by parts, the boundary terms are used to identify the forms of the primary and secondary variables.
4. Because of the restrictions placed on the weight function in the weak form, it must belong to the same space of functions as the approximation functions ($w \sim \phi_i$).

# Examples[](#top)

##  Example 1

Construct the weak and identify the primary and secondary variables for the nonlinear equation

$$
-\frac{d}{dx}\left(u\frac{du}{dx}\right) + f = 0 \quad \text{for} \quad
0 < x < L
$$
$$
\left(u\frac{du}{dx}\right)\Bigg|_{x=0}=0, \quad u(1)=\sqrt{2}
$$

### Solution

Following the three step procedure, we write

$$
\begin{align}
0 &= \int_0^Lw\left(-\frac{d}{dx}\left(u\frac{du}{dx}\right) + f\right)dx\\
&=\int_0^L\left(u\frac{dw}{dx}\frac{du}{dx}
+ wf\right)dx-\left(w\left(u\frac{du}{dx}\right)\right)\Bigg|_0^L
\end{align}
$$

Observation of the boundary term identifies $u$ as the primary variable and $u(du/dx)$ as the secondary variable.

From the boundary conditions, since $w(1) = 0$ (why?) and $du/dx=0$ at $x=0$, we obtain for the weak form

$$
0 = \int_0^L \left(u\frac{dw}{dx}\frac{du}{dx} + wf\right)dx
$$

The weak form does not contain an expression that is linear in both $u$ and $v$.

## Example 2

Construct the weak form and identify the primary and secondary variables of the following nonlinear differential equation

$$
-\frac{d}{dx}\left(\left(1 + 2x^2\right)\frac{du}{dx}\right) + u = x^2
$$
$$
u(0) = 1, \quad \left(\frac{du}{dx}\right)_{x=1}=2
$$

### Solution

Following the three step procedure, we write

$$
\begin{align}
(1) \quad & 0 = \int_0^1w\left(-\frac{d}{dx}\left(\left(1 + 2x^2\right)\frac{du}{dx}\right) + u - x^2\right)dx
\\
(2) \quad & 0 = \int_0^1\left(\left(1 + 2x^2\right)\frac{dw}{dx}\frac{du}{dx} + wu - wx^2\right)dx - \left(w\left(1+2x^2\right)\frac{du}{dx}\right)\Bigg|_0^1
\end{align}
$$

From the boundary term, $u$ is the primary variable and $\left(1+2x^2\right)du/dx$ is the secondary variable.  Since $du/dx=2$ at $x=1$ and $w=0$ at $x=0$, the weak form is

$$
0 = \int_0^1\left(\left(1 + 2x^2\right)\frac{dw}{dx}\frac{du}{dx} + wu\right)dx - \int_0^1wx^2dx - 6w(1)
$$


# Rich Output

In Python, objects can declare their textual representation using the `__repr__` method.  IPython expands on this idea and allows objects to declare other, rich representations including:

* HTML
* JSON
* PNG
* JPEG
* SVG
* LaTeX

A single object can declare some or all of these representations; all are handled by IPython's *display system*. This Notebook shows how you can use this display system to incorporate a broad range of content into your Notebooks.

## Basic display imports

The `display` function is a general purpose tool for displaying different representations of objects. Think of it as `print` for these rich representations.

```python
from IPython.display import display
```

A few points:

* Calling `display` on an object will send **all** possible representations to the Notebook.
* These representations are stored in the Notebook document.
* In general the Notebook will use the richest available representation.

If you want to display a particular representation, there are specific functions for that:

```python
from IPython.display import (
    display_pretty, display_html, display_jpeg,
    display_png, display_json, display_latex, display_svg
)
```

## Images

To work with images (JPEG, PNG) use the `Image` class.

```python
from IPython.display import Image
```

```python
i = Image(filename='./images/python-logo.png')
```

Returning an `Image` object from an expression will automatically display it:

```python
i
```

Or you can pass an object with a rich representation to `display`:

```python
display(i)
```

An image can also be displayed from raw data or a URL.

```python
Image(url='http://python.org/images/python-logo.gif')
```

SVG images are also supported out of the box.

```python
from IPython.display import SVG
SVG(filename='./images/python-logo.svg')
```

### Embedded vs non-embedded Images

By default, image data is embedded in the notebook document so that the images can be viewed offline. However it is also possible to tell the `Image` class to only store a *link* to the image. Let's see how this works using a webcam at Berkeley.

```python
from IPython.display import Image
img_url = 'http://www.lawrencehallofscience.org/static/scienceview/scienceview.berkeley.edu/html/view/view_assets/images/newview.jpg'

# by default Image data are embedded
Embed      = Image(img_url)

# if kwarg `url` is given, the embedding is assumed to be false
SoftLinked = Image(url=img_url)

# In each case, embed can be specified explicitly with the `embed` kwarg
# ForceEmbed = Image(url=img_url, embed=True)
```

Here is the embedded version. Note that this image was pulled from the webcam when this code cell was originally run and stored in the Notebook. Unless we rerun this cell, this is not todays image.

```python
Embed
```

Here is today's image from same webcam at Berkeley, (refreshed every minutes, if you reload the notebook), visible only with an active internet connection, that should be different from the previous one. Notebooks saved with this kind of image will be smaller and always reflect the current version of the source, but the image won't display offline.

```python
SoftLinked
```

Of course, if you re-run this Notebook, the two images will be the same again.

## HTML

Python objects can declare HTML representations that will be displayed in the Notebook. If you have some HTML you want to display, simply use the `HTML` class.

```python
from IPython.display import HTML
```

```python
s = """<table>
<tr>
<th>Header 1</th>
<th>Header 2</th>
</tr>
<tr>
<td>row 1, cell 1</td>
<td>row 1, cell 2</td>
</tr>
<tr>
<td>row 2, cell 1</td>
<td>row 2, cell 2</td>
</tr>
</table>"""
```

```python
h = HTML(s)
```

```python
display(h)
```

You can also use the `%%html` cell magic to accomplish the same thing.

```python
%%html
<table>
<tr>
<th>Header 1</th>
<th>Header 2</th>
</tr>
<tr>
<td>row 1, cell 1</td>
<td>row 1, cell 2</td>
</tr>
<tr>
<td>row 2, cell 1</td>
<td>row 2, cell 2</td>
</tr>
</table>
```

## JavaScript

The Notebook also enables objects to declare a JavaScript representation. At first, this may seem odd as  output is inherently visual and JavaScript is a programming language. However, this opens the door for rich output that leverages the full power of JavaScript and associated libraries such as [d3.js](http://d3js.org) for output.

```python
from IPython.display import Javascript
```

Pass a string of JavaScript source code to the `JavaScript` object and then display it.

```python
js = Javascript('alert("hi")');
```

```python
display(js)
```

The same thing can be accomplished using the `%%javascript` cell magic:

```javascript
%%javascript

alert("hi");
```

Here is a more complicated example that loads `d3.js` from a CDN, uses the `%%html` magic to load CSS styles onto the page and then runs ones of the `d3.js` examples.

```python
Javascript(
    """$.getScript('//cdnjs.cloudflare.com/ajax/libs/d3/3.2.2/d3.v3.min.js')"""
)
```

```python
%%html
<style type="text/css">

circle {
  fill: rgb(31, 119, 180);
  fill-opacity: .25;
  stroke: rgb(31, 119, 180);
  stroke-width: 1px;
}

.leaf circle {
  fill: #ff7f0e;
  fill-opacity: 1;
}

text {
  font: 10px sans-serif;
}

</style>
```

```javascript
%%javascript

// element is the jQuery element we will append to
var e = element.get(0);

var diameter = 600,
    format = d3.format(",d");

var pack = d3.layout.pack()
    .size([diameter - 4, diameter - 4])
    .value(function(d) { return d.size; });

var svg = d3.select(e).append("svg")
    .attr("width", diameter)
    .attr("height", diameter)
  .append("g")
    .attr("transform", "translate(2,2)");

d3.json("data/flare.json", function(error, root) {
  var node = svg.datum(root).selectAll(".node")
      .data(pack.nodes)
    .enter().append("g")
      .attr("class", function(d) { return d.children ? "node" : "leaf node"; })
      .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; });

  node.append("title")
      .text(function(d) { return d.name + (d.children ? "" : ": " + format(d.size)); });

  node.append("circle")
      .attr("r", function(d) { return d.r; });

  node.filter(function(d) { return !d.children; }).append("text")
      .attr("dy", ".3em")
      .style("text-anchor", "middle")
      .text(function(d) { return d.name.substring(0, d.r / 3); });
});

d3.select(self.frameElement).style("height", diameter + "px");
```

## LaTeX

The IPython display system also has builtin support for the display of mathematical expressions typeset in LaTeX, which is rendered in the browser using [MathJax](http://mathjax.org).

You can pass raw LaTeX test as a string to the `Math` object:

```python
from IPython.display import Math
Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx')
```

With the `Latex` class, you have to include the delimiters yourself.  This allows you to use other LaTeX modes such as `eqnarray`:

```python
from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")
```

Or you can enter LaTeX directly with the `%%latex` cell magic:

```latex
%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```

## Audio

IPython makes it easy to work with sounds interactively. The `Audio` display class allows you to create an audio control that is embedded in the Notebook. The interface is analogous to the interface of the `Image` display class. All audio formats supported by the browser can be used. Note that no single format is presently supported in all browsers.

```python
from IPython.display import Audio
Audio(url="http://www.nch.com.au/acm/8k16bitpcm.wav")
```

A NumPy array can be auralized automatically. The `Audio` class normalizes and encodes the data and embeds the resulting audio in the Notebook.

For instance, when two sine waves with almost the same frequency are superimposed a phenomena known as [beats](https://en.wikipedia.org/wiki/Beat_%28acoustics%29) occur. This can be auralised as follows:

```python
import numpy as np
max_time = 3
f1 = 220.0
f2 = 224.0
rate = 8000
L = 3
times = np.linspace(0,L,rate*L)
signal = np.sin(2*np.pi*f1*times) + np.sin(2*np.pi*f2*times)

Audio(data=signal, rate=rate)
```

## Video

More exotic objects can also be displayed, as long as their representation supports the IPython display protocol.  For example, videos hosted externally on YouTube are easy to load:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('sjfsUzECqK0')
```

Using the nascent video capabilities of modern browsers, you may also be able to display local
videos.  At the moment this doesn't work very well in all browsers, so it may or may not work for you;
we will continue testing this and looking for ways to make it more robust.

The following cell loads a local file called  `animation.m4v`, encodes the raw video as base64 for http
transport, and uses the HTML5 video tag to load it. On Chrome 15 it works correctly, displaying a control bar at the bottom with a play/pause button and a location slider.

```python
from IPython.display import HTML
from base64 import b64encode
video = open("./images/animation.m4v", "rb").read()
video_encoded = b64encode(video).decode('ascii')
video_tag = '<video controls alt="test" src="data:video/x-m4v;base64,{0}">'.format(video_encoded)
HTML(data=video_tag)
```

## External sites

You can even embed an entire page from another site in an iframe; for example this is today's Wikipedia
page for mobile users:

```python
from IPython.display import IFrame
IFrame('http://jupyter.org', width='100%', height=350)
```

## Links to local files

IPython provides builtin display classes for generating links to local files. Create a link to a single file using the `FileLink` object:

```python
from IPython.display import FileLink, FileLinks
FileLink('Cell Magics.ipynb')
```

Alternatively, to generate links to all of the files in a directory, use the `FileLinks` object, passing `'.'` to indicate that we want links generated for the current working directory. Note that if there were other directories under the current directory, `FileLinks` would work in a recursive manner creating links to files in all sub-directories as well.

```python
FileLinks('.')
```

## Rich output and security

The IPython Notebook allows arbitrary code execution in both the IPython kernel and in the browser, though HTML and JavaScript output. More importantly, because IPython has a JavaScript API for running code in the browser, HTML and JavaScript output can actually trigger code to be run in the kernel. This poses a significant security risk as it would allow IPython Notebooks to execute arbitrary code on your computers.

To protect against these risks, the IPython Notebook has a security model that specifies how dangerous output is handled. Here is a short summary:

* When you run code in the Notebook, all rich output is displayed.
* When you open a notebook, rich output is only displayed if it doesn't contain security vulberabilities, ...
* ... or if you have trusted a notebook, all rich output will run upon opening it.

A full description of the IPython security model can be found on [this page](http://ipython.org/ipython-doc/dev/notebook/security.html).

## Rich output and nbviewer

Much of the power of the Notebook is that it enables users to share notebooks with each other using http://nbviewer.ipython.org, without installing IPython locally. As of IPython 2.0, notebooks rendered on nbviewer will display all output, including HTML and JavaScript. Furthermore, to provide a consistent JavaScript environment on the live Notebook and nbviewer, the following JavaScript libraries are loaded onto the nbviewer page, *before* the notebook and its output is displayed:

* [jQuery](http://jquery.com/)
* [RequireJS](http://requirejs.org/)

Libraries such as [mpld3](http://mpld3.github.io/) use these capabilities to generate interactive visualizations that work on nbviewer.

```python

```