Face Mesh Application using OpenCV and dlib

Snapchat: Snapchat is one of the leading apps which is a fun-loving application for today’s generation in which we have seen multiple filters applied on our faces, Similarly this feature can also be added to Snapchat or other leading software of a similar genre to attract more users which in turn help in getting more downloads.

Augmented reality software: AR/VR can also use this particular functionality in some of their use cases to demonstrate and make it more creative.

import cv2
import numpy as np
import dlib
import requests  
from PIL import Image

Installation of the dlib library

This is a bit tricky you have to follow some steps:

1. Install visual studio (latest version)- refer to this link.
2. In visual studio, one needs to install the CMake package.
3. After installation from Visual studio, we have to install it again using – pip install CMake.
4. Here comes the last part now, we have to install dlib by – pip install dlib.

Download the pre-trained model shape_predictor

https://github.com/tzutalin/dlib-android/raw/master/data/shape_predictor_68_face_landmarks.dat

Now we will create a function to extract the index from the NumPy array.

# Extracting index from array
def extract_index_nparray(nparray):
    index = None
    for num in nparray[0]:
        index = num
        break
    return index

Next, we will load our source image from the internet using a URL and resize it.

# Reading source image form url
image1 = Image.open(requests.get('https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSx8Pu1tW1uCiZPfj9K1EL6uHxbg3bOKO9XkA&usqp=CAU', stream=True).raw)
image1 = image1.resize((300,300))
image1

Output:

 

Here we will load our destination image from the internet using a URL and resize it.

# Reading destination image form url
image2 = Image.open(requests.get('https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTYX1dyl9INRo5cbvDeTILRcZVzfcMsCsE0kg&usqp=CAU', stream=True).raw)
image2 = image2.resize((300,300))
image2

Output:

Now we will convert our images into a NumPy array and use cv2 to convert them into grayscale. We will create an empty image or mask similar to our source image with zeros.

# Converting image to array and converting them to grayscale
img = np.array(image1)
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
mask = np.zeros_like(img_gray)
img2 = np.array(image2)
img2_gray = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

For creating empty images with zeros, we will first load the Face detector and Face landmarks predictor using dlib and then we will find the height, width, channels.

# Initalizing frontal face detector and shape predictor
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
height, width, channels = img2.shape
img2_new_face = np.zeros((height, width, channels), np.uint8)

Triangulation of images for Face Mesh Application

 

First, we need to pass the image to the detector then that object will be used to extract landmarks using predictor. Afterward, storing extracted landmarks (x and y) into the landmarks (list). We’re going to mesh up the face into triangles. This step is the core of our face-swapping. Here we will interchange each triangle with the correspondent triangle of the destination image. The triangulation of the destination image needs to have the same pattern just like that of the triangulation of the source image. This means that the connection of the connecting symbols has to be the same. So after we do the triangulation of the source image, from that triangulation we take the indexes of the (x and y) so that we can replicate the same triangulation on the destination image. Once we have the triangles indexes we loop through them and we triangulate the destination face.

# Face 1
faces = detector(img_gray)
for face in faces:
    landmarks = predictor(img_gray, face)
    landmarks_points = []
    for n in range(0, 68):
        x = landmarks.part(n).x
        y = landmarks.part(n).y
        landmarks_points.append((x, y))

    points = np.array(landmarks_points, np.int32)

    convexhull = cv2.convexHull(points)
    cv2.fillConvexPoly(mask, convexhull, 255)    
    
    face_image_1 = cv2.bitwise_and(img, img, mask=mask)

    # Delaunay triangulation

    rect = cv2.boundingRect(convexhull)
    subdiv = cv2.Subdiv2D(rect)
    subdiv.insert(landmarks_points)
    triangles = subdiv.getTriangleList()
    triangles = np.array(triangles, dtype=np.int32)
    
    indexes_triangles = []
    for t in triangles:
        pt1 = (t[0], t[1])
        pt2 = (t[2], t[3])
        pt3 = (t[4], t[5])

        index_pt1 = np.where((points == pt1).all(axis=1))
        index_pt1 = extract_index_nparray(index_pt1)

        index_pt2 = np.where((points == pt2).all(axis=1))
        index_pt2 = extract_index_nparray(index_pt2)

        index_pt3 = np.where((points == pt3).all(axis=1))
        index_pt3 = extract_index_nparray(index_pt3)
        
        if index_pt1 is not None and index_pt2 is not None and index_pt3 is not None:
            triangle = [index_pt1, index_pt2, index_pt3]
            indexes_triangles.append(triangle)

Triangulation pattern – Delaunay Triangulation

Once we have cut and wrapped all the triangles we need to link them together. Then we have to re-build the face using the triangulation pattern, with the only difference that this time we put the wrapped triangle.

# Face will be swapped sucessfully by convex hull partioning
img2_face_mask = np.zeros_like(img2_gray)
img2_head_mask = cv2.fillConvexPoly(img2_face_mask, convexhull2, 255)
img2_face_mask = cv2.bitwise_not(img2_head_mask)

The face is now ready to be replaced.

So we take the new face, and the destination image without face and we link them together.

img2_head_noface = cv2.bitwise_and(img2, img2, mask=img2_face_mask)
result = cv2.add(img2_head_noface, img2_new_face)

Finally, the faces are correctly swapped and now we have to check on colors so that the source image fits the destination image.

On Opencv we have a built-in function called “seamless clone” that does this operation automatically. We need to take the new face (created on the 6th step), take the original destination image, and its a mask to cut out the face, we need to get the middle of the face.

Creating the seamless clone of two faces

(x, y, w, h) = cv2.boundingRect(convexhull2)
center_face2 = (int((x + x + w) / 2), int((y + y + h) / 2))
seamlessclone = cv2.seamlessClone(result, img2, img2_head_mask, center_face2, cv2.NORMAL_CLONE)

Finally, we will visualize the output NumPy image by converting it into the Image object of Pillow.

# Converting array to image
Image.fromarray(seamlessclone)

Output: