Creating an artificial artist: Color your photos using Neural Networks
Art has always transcended eons of human existence. We can see its traces from pre-historic time as the Harappan art in the Indus Valley Civilization to the contemporary art in modern times. Mostly, art has been a means to express one’s creativity, viewpoints of how we perceive the world. As legendary Leonardo Da Vinci has said,
“Painting is poetry that is seen rather than felt”.
What we sometimes forget that most of the art follows a pattern. A pattern that pleases us and makes sense in our brain. The next time you see a painting, try to notice the brush strokes in it. You will see a pattern arising out of the painting. We as humans are skilled in recognising these patterns. Our neural mechanisms has developed to be exceptionally great over the years recognising patterns in the wild.
Now you may ask why I am ranting away about art and patterns? This is because I will show you how to create art with the help of artificial brains! In this article, we will build an artificial neural network which will extract style from one image and replicate it on the other. So are you ready?
Table of Contents
- What is Neural Art?
- Familiarizing with the crux
- Coding it up
- Where to go from here?
- Additional Resources
What is Neural Art?
Let’s try to understand this topic with an example.
The above image is the famous “The Starry Night” by Vincent Van Gogh. Just look at the painting for a few minutes. What do you see? Do you notice the bush strokes? Do you see the curves and edges that define each and every object which makes it so easy for you to recognise them?
Now let’s do a quick assignment. Try to remember the patterns you see. Just cram your brain with every little detail. Done? Ok now take a look at the next image.
This is a photograph taken of a town called “Tubingen” located in Germany. For the next step of the assignment, just close your eyes and try to replicate the style of a starry night with this image. Ask yourself, if you are Van Gogh (hypothetically of course!) and are asked to draw this photograph keeping in mind the styles you memorized before, how would you do it?
Did you do it? Great! You just made a neural art!
Want to see what an artificial neural network can do?
You may ask how did a machine accomplish such a task. It’s simple once you get the gist of it!
What neural network does is, it tries to extract the “important points” from the both the images, that is it tries to recognize which attributes define the picture and learns from it. These learned attributes are an internal representation of the neural network, which can be seen as below.
So you got to know the theoretical concepts involved in neural art, now let’s get to know the practical aspects of implementing it.
Getting into the brain of artificial artist:
Neural Art works in the following way:
- We first define the loss functions necessary to generate our result, namely the style loss, content loss and the total variational loss
- We define our optimization function, i.e. back propagation algorithm. Here we use L-BFGS because it’s faster and more efficient for smaller data.
- Then we set our style and content attributes of our model
- Then we pass an image to our model (preferably our base image) and optimize it to minimize all the losses we defined above.
We will get to know some of the important points you ought to know before we jump in. While most of the fundamentals of Neural Networks are covered in this article, I will reiterate some of them and explain a few extra things.
- What is a loss function? Loss function is a function that calculates the difference of the predicted values from the original values. It essentially says how much error has occurred in a calculation. In any machine learning algorithm, loss function is used to estimate how the model performs on data. This is especially helpful in case of neural networks, where you iteratively try to make your model perform better. When implementing neural art, you have to keep check of three loss functions, namely:
- Content loss, i.e. the difference between “content” of resulting image and base image. This is done to ensure that your model does not get much deviated from the base image
- Style loss, i.e. the difference between “style” of resulting image and base image. To do this, you have to first calculate gram matrix of both the images and then find their difference. Gram matrix is nothing but finding covariance of an image with itself. This is done to maintain style in the resulting image.
- Total Validation loss, i.e. the difference between a pixel of resulting image with its neighbouring pixel. This is done so that the image remains visually coherent.
- What is an optimization function? When we have calculated the loss function, we try to minimize our losses by changing parameters. Optimization function helps us find out how much change is required so that our model is better “optimized”. Here we implement a method of optimization called Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS). BFGS is a variant of gradient descent algorithm in which you do a second order differentiation to find the local minimum. Read this article to get a mathematical perspective of the algorithm.
Now that we’ve understood what our flow will be to build a neural art, let’s get down and start hacking stuff!
Coding it up!
This Diwali was an interesting one for me. I decided to do some research on neural art and how India illuminates during the Diwali day. I came across this image “India on Diwali night”. And I thought of creating something similar on the same lines. To do that, we will be combining the two images below with the help of neural art.
So first we will first set the groundworks.
Step 0: Install Keras and its dependencies . For this, we will be using a Theano backend. Change your backend by following the steps mentioned here. Also additionally you have to set the proper ordering for image. In the keras.json file, where you have changed the backend, replace image_dim_ordering with ‘tr’. So it should look like this,
Step 1: Then go to your working directory and set your directory structure as below
|-- keras_NeuralStyle # this is your working directory | |-- base_image.jpg # this is your base image | |-- reference_image.jpg # this is your reference image
Step 2: Start a jupyter notebook in your working directory by typing
jupyter notebook and implement the following code. I will just provide you a step by step overview of what each block does.
- First, you have to import all the modules necessary to implement the code
- Then set the paths of the images you want to carry out the project on.
- Define the necessary variables and give them values as below. Note that these values can be changed but that may change the output drastically. Also, make sure the value of img_nrows variable is same as img_ncols. This is necessary for gram matrix to work.
- Then we define a helper functions. These are responsible for handling image preprocessing.
- Create input placeholders to pass images to the model
- Load a pre-trained neural network model (If you don’t know what pre-training is, go through this discussion)
- Print the model summary to see what the model is
- Store the names of all the layers of the neural network as a dictionary along with their outputs
- As defined above, we set the loss functions
- We then set the content and style attributes …
- And set the gradients and final output function for neural art
- We define the functions to calculate loss and gradients
- Now we take the base image as input and iterate through it to get our final image. On my local machine, it takes a minute to get the output in one iteration. Based on your resources (and patience) it would take at most 5 minutes to get the output. You can also increase the number of iterations to more optimize the result.
- And after a long wait, we will get this beautiful image!
NOTE: The code file can be viewed on github here.
Where to go from here?
We have seen a small demo of a significant discovery in the art world. There have been many modifications done to this method to make it aesthetically pleasing. For example, I really like this implementation in which they have taken different styles and applied them to different regions.
The first two images are the masks, which help to set which part should be stylized. The next two images represent the styles to be used. The last image is the base image that has to be stylized.
Below is the output that is generated by neural art.
Looks awesome, doesn’t it? I am sure like me, you are also fascinated to try your hands on neural art. To help you get started with it, I have covered the basics of neural art and how can you create your first image. I am sure you are eager to explore more and hence I am adding some additional resources only for you.
These are some of the best resources I have come across on neural art. Go ahead and enter the fascinating world of neural art.
- Neural Art implementation using Torch
- Neural Art implementation in Tensorflow
- Another noteworthy Neural Art implementation in Keras
- Deep Forger: A twitter bot that generates Neural Art on demand
- Neural Art on Videos
- DEEPART.io: A webapp for Neural Art
- Prisma-ai: A mobile app for Neural Art
Image Sources https://www.wikiart.org/en/vincent-van-gogh/the-starry-night-1889  https://arxiv.org/abs/1508.06576  Google
I hope you found this article inspiring. Now, it’s time for you to go through it and make art yourself! If you create an art do share it with the community. If you have any doubts, I’d love to interact with you in comments. And to gain expertise in working in neural network don’t forget to try out our deep learning practice problem – Identify the Digits.