Learn everything about Analytics

A Practical Implementation of the Faster R-CNN Algorithm for Object Detection (Part 2 – with Python codes)

Introduction

Which algorithm do you use for object detection tasks? I have tried out quite a few of them in my quest to build the most precise model in the least amount of time. And this journey, spanning multiple hackathons and real-world datasets, has usually always led me to the R-CNN family of algorithms.

It has been an incredible useful framework for me, and that’s why I decided to pen down my learnings in the form of a series of articles. The aim behind this series is to showcase how useful the different types of R-CNN algorithms are. The first part received an overwhelmingly positive response from our community, and I’m thrilled to present part two!

In this article, we will first briefly summarize what we learned in part 1, and then deep dive into the implementation of the fastest member of the R-CNN family – Faster R-CNN. I highly recommend going through this article if you need to refresh your object detection concepts first: A Step-by-Step Introduction to the Basic Object Detection Algorithms (Part 1).

We will work on a very interesting dataset here, so let’s dive right in!

 

Table of Contents

  1. A Brief Overview of the Different R-CNN Algorithms for Object Detection
  2. Understanding the Problem Statement
  3. Setting up the System
  4. Data Exploration
  5. Implementing Faster R-CNN

 

A Brief Overview of the Different R-CNN Algorithms for Object Detection

Let’s quickly summarize the different algorithms in the R-CNN family (R-CNN, Fast R-CNN, and Faster R-CNN) that we saw in the first article. This will help lay the ground for our implementation part later when we will predict the bounding boxes present in previously unseen images (new data).

R-CNN extracts a bunch of regions from the given image using selective search, and then checks if any of these boxes contains an object. We first extract these regions, and for each region, CNN is used to extract specific features. Finally, these features are then used to detect objects. Unfortunately, R-CNN becomes rather slow due to these multiple steps involved in the process.

R-CNN

Fast R-CNN, on the other hand, passes the entire image to ConvNet which generates regions of interest (instead of passing the extracted regions from the image). Also, instead of using three different models (as we saw in R-CNN), it uses a single model which extracts features from the regions, classifies them into different classes, and returns the bounding boxes.

All these steps are done simultaneously, thus making it execute faster as compared to R-CNN. Fast R-CNN is, however, not fast enough when applied on a large dataset as it also uses selective search for extracting the regions.

Fast R-CNN

Faster R-CNN fixes the problem of selective search by replacing it with Region Proposal Network (RPN). We first extract feature maps from the input image using ConvNet and then pass those maps through a RPN which returns object proposals. Finally, these maps are classified and the bounding boxes are predicted.

Faster R-CNN

I have summarized below the steps followed by a Faster R-CNN algorithm to detect objects in an image:

  1. Take an input image and pass it to the ConvNet which returns feature maps for the image
  2. Apply Region Proposal Network (RPN) on these feature maps and get object proposals
  3. Apply ROI pooling layer to bring down all the proposals to the same size
  4. Finally, pass these proposals to a fully connected layer in order to classify any predict the bounding boxes for the image

What better way to compare these different algorithms than in a tabular format? So here you go!

Algorithm Features Prediction time / image Limitations
CNN Divides the image into multiple regions and then classifies each region into various classes. Needs a lot of regions to predict accurately and hence high computation time.
R-CNN Uses selective search to generate regions. Extracts around 2000 regions from each image. 40-50 seconds High computation time as each region is passed to the CNN separately. Also, it uses three different models for making predictions.
Fast R-CNN Each image is passed only once to the CNN and feature maps are extracted. Selective search is used on these maps to generate predictions. Combines all the three models used in R-CNN together. 2 seconds Selective search is slow and hence computation time is still high.
Faster R-CNN Replaces the selective search method with region proposal network (RPN) which makes the algorithm much faster. 0.2 seconds Object proposal takes time and as there are different systems working one after the other, the performance of systems depends on how the previous system has performed.

 

Now that we have a grasp on this topic, it’s time to jump from the theory into the practical part of our article. Let’s implement Faster R-CNN using a really cool (and rather useful) dataset with potential real-life applications!

 

Understanding the Problem Statement

We will be working on a healthcare related dataset and the aim here is to solve a Blood Cell Detection problem. Our task is to detect all the Red Blood Cells (RBCs), White Blood Cells (WBCs), and Platelets in each image taken via microscopic image readings. Below is a sample of what our final predictions should look like:

The reason for choosing this dataset is that the density of RBCs, WBCs and Platelets in our blood stream provides a lot of information about the immune system and hemoglobin. This can help us potentially identify whether a person is healthy or not, and if any discrepancy is found in their blood, actions can be taken quickly to diagnose that.

Manually looking at the sample via a microscope is a tedious process. And this is where Deep Learning models play such a vital role. They can classify and detect the blood cells from microscopic images with impressive precision.

The full blood cell detection dataset for our challenge can be downloaded from here. I have modified the data a tiny bit for the scope of this article:

  • The bounding boxes have been converted from the given .xml format to a .csv format
  • I have also created the training and test set split on the entire dataset by randomly picking images for the split

Note that we will be using the popular Keras framework with a TensorFlow backend in Python to train and build our model.

 

Setting up the System

Before we actually get into the model building phase, we need to ensure that the right libraries and frameworks have been installed. The below libraries are required to run this project:

  • pandas
  • matplotlib
  • tensorflow
  • keras – 2.0.3
  • numpy
  • opencv-python
  • sklearn
  • h5py

Most of the above mentioned libraries will already be present on your machine if you have Anaconda and Jupyter Notebooks installed. Additionally, I recommend downloading the requirement.txt file from this link and use that to install the remaining libraries. Type the following command in the terminal to do this:

pip install -r requirement.txt

Alright, our system is now set and we can move on to working with the data!

 

Data Exploration

It’s always a good idea (and frankly, a mandatory step) to first explore the data we have. This helps us not only unearth hidden patterns, but gain a valuable overall insight into what we are working with. The three files I have created out of the entire dataset are:

  1. train_images: Images that we will be using to train the model. We have the classes and the actual bounding boxes for each class in this folder.
  2. test_images: Images in this folder will be used to make predictions using the trained model. This set is missing the classes and the bounding boxes for these classes.
  3. train.csv: Contains the name, class and bounding box coordinates for each image. There can be multiple rows for one image as a single image can have more than one object.

Let’s read the .csv file (you can create your own .csv file from the original dataset if you feel like experimenting) and print out the first few rows. We’ll need to first import the below libraries for this:

# importing required libraries
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import patches

 

# read the csv file using read_csv function of pandas
train = pd.read_csv(‘train.csv’)
train.head()

There are 6 columns in the train file. Let’s understand what each column represents:

  1. image_names: contains the name of the image
  2. cell_type: denotes the type of the cell
  3. xmin: x-coordinate of the bottom left part of the image
  4. xmax: x-coordinate of the top right part of the image
  5. ymin: y-coordinate of the bottom left part of the image
  6. ymax: y-coordinate of the top right part of the image

Let’s now print an image to visualize what we’re working with:

# reading single image using imread function of matplotlib
image = plt.imread('images/1.jpg')
plt.imshow(image)

This is what a blood cell image looks like. Here, the blue part represents the WBCs, and the slightly red parts represent the RBCs. Let’s look at how many images, and the different type of classes, there are in our training set.

# Number of unique training images
train['image_names'].nunique()

So, we have 254 training images.

# Number of classes
train['cell_type'].value_counts()

We have three different classes of cells, i.e., RBC, WBC and Platelets. Finally, let’s look at how an image with detected objects will look like:

fig = plt.figure()

#add axes to the image
ax = fig.add_axes([0,0,1,1])

# read and plot the image
image = plt.imread('images/1.jpg')
plt.imshow(image)

# iterating over the image for different objects
for _,row in train[train.image_names == "1.jpg"].iterrows():
    xmin = row.xmin
    xmax = row.xmax
    ymin = row.ymin
    ymax = row.ymax
    
    width = xmax - xmin
    height = ymax - ymin
    
    # assign different color to different classes of objects
    if row.cell_type == 'RBC':
        edgecolor = 'r'
        ax.annotate('RBC', xy=(xmax-40,ymin+20))
    elif row.cell_type == 'WBC':
        edgecolor = 'b'
        ax.annotate('WBC', xy=(xmax-40,ymin+20))
    elif row.cell_type == 'Platelets':
        edgecolor = 'g'
        ax.annotate('Platelets', xy=(xmax-40,ymin+20))
        
    # add bounding boxes to the image
    rect = patches.Rectangle((xmin,ymin), width, height, edgecolor = edgecolor, facecolor = 'none')
    
    ax.add_patch(rect)

This is what a training example looks like. We have the different classes and their corresponding bounding boxes. Let’s now train our model on these images. We will be using the keras_frcnn library to train our model as well as to get predictions on the test images.

 

Implementing Faster R-CNN

For implementing the Faster R-CNN algorithm, we will be following the steps mentioned in this Github repository. So as the first step, make sure you clone this repository. Open a new terminal window and type the following to do this:

git clone https://github.com/kbardool/keras-frcnn.git

Move the train_images and test_images folder, as well as the train.csv file, to the cloned repository. In order to train the model on a new dataset, the format of the input should be:

filepath,x1,y1,x2,y2,class_name

where,

  • filepath is the path of the training image
  • x1 is the xmin coordinate for bounding box
  • y1 is the ymin coordinate for bounding box
  • x2 is the xmax coordinate for bounding box
  • y2 is the ymax coordinate for bounding box
  • class_name is the name of the class in that bounding box

We need to convert the .csv format into a .txt file which will have the same format as described above. Make a new dataframe, fill all the values as per the format into that dataframe, and then save it as a .txt file.

data = pd.DataFrame()
data['format'] = train['image_names']

# as the images are in train_images folder, add train_images before the image name
for i in range(data.shape[0]):
    data['format'][i] = 'train_images/' + data['format'][i]

# add xmin, ymin, xmax, ymax and class as per the format required
for i in range(data.shape[0]):
    data['format'][i] = data['format'][i] + ',' + str(train['xmin'][i]) + ',' + str(train['ymin'][i]) + ',' + str(train['xmax'][i]) + ',' + str(train['ymax'][i]) + ',' + train['cell_type'][i]

data.to_csv('annotate.txt', header=None, index=None, sep=' ')

What’s next?

Train our model! We will be using the train_frcnn.py file to train the model.

cd keras-frcnn
python train_frcnn.py -o simple -p annotate.txt

It will take a while to train the model due to the size of the data. If possible, you can use a GPU to make the training phase faster. You can also try to reduce the number of epochs as an alternate option. To change the number of epochs, go to the train_frcnn.py file in the cloned repository and change the num_epochs parameter accordingly.

Every time the model sees an improvement, the weights of that particular epoch will be saved in the same directory as “model_frcnn.hdf5”. These weights will be used when we make predictions on the test set.

It might take a lot of time to train the model and get the weights, depending on the configuration of your machine. I suggest using the weights I’ve got after training the model for around 500 epochs. You can download these weights from here. Ensure you save these weights in the cloned repository.

So our model has been trained and the weights are set. It’s prediction time! Keras_frcnn makes the predictions for the new images and saves them in a new folder. We just have to make two changes in the test_frcnn.py file to save the images:

  1. Remove the comment from the last line of this file:
    cv2.imwrite(‘./results_imgs/{}.png’.format(idx),img)
  2. Add comments on the second last and third last line of this file:
    # cv2.imshow(‘img’, img)
    # cv2.waitKey(0)

Let’s make the predictions for the new images:

python test_frcnn.py -p test_images

Finally, the images with the detected objects will be saved in the “results_imgs” folder. Below are a few examples of the predictions I got after implementing Faster R-CNN:

Result 1

Result 2

Result 3

Result 4

 

End Notes

R-CNN algorithms have truly been a game-changer for object detection tasks. There has suddenly been a spike in recent years in the amount of computer vision applications being created, and R-CNN is at the heart of most of them.

Keras_frcnn proved to be an excellent library for object detection, and in the next article of this series, we will focus on more advanced techniques like YOLO, SSD, etc.

If you have any query or suggestions regarding what we covered here, feel free to post them in the comments section below and I will be happy to connect with you!

You can also read this article on Analytics Vidhya's Android APP Get it on Google Play

78 Comments

  • sset says:

    Thanks for article.
    Most of object detection algorithms fail if size of object to be detected is very small and with varying size. For example detection of small cracks on metal surface. What is your view on this? Any advice and best possible approach?

    • Pulkit Sharma says:

      Hi,

      I believe Faster RCNN works good enough for small objects as well. Currently, I am working on YOLO and SSD and will share my learnings on how they deal with small objects.

      • manoj singh says:

        hi pulkit ,
        i have trained my data but while predicting i am getting images with no bounding boxex, can you tell me the reason??

        • Pulkit Sharma says:

          Hi Manoj,

          Which dataset are you using to train the model? Are the annotations and classes correctly defined? And what is the number of epochs that you have used? Also please share the dataset so that I can look into it and help you in a better way.

  • Vishnu says:

    Hey Pulkit,

    I am a Freshman at UIUC studying CS and one of my projects is in the same domain. Would it be possible to connect with you and talk more about this?

    • Pulkit Sharma says:

      Hi Vishnu,

      You can ask any query related to this project here. I will try my best to respond to them.

      • ping wan says:

        Hi, Pulkit,

        Thanks for this wonderful tutorial !

        I have downloaded the BCCD dataset, but there is no train_images and test_images folder, as well as the train.csv file in it, shall I generate them by myself?

        Thank you!

  • Toni says:

    The dataset that you have provided, it doesn’t correspond with your example. Can you provide the right source?

    • Pulkit Sharma says:

      Hi Toni,

      The original dataset is available on GitHub, link for the same is provided in the article. I have made some changes in that dataset and have mentioned the changes as well. You just have to convert the dataset in csv format.

  • saurabh says:

    I am having issues reading the data which is in .rec format

    • Pulkit Sharma says:

      Hi Saurabh,

      You can convert the coordinates to csv format and then read it in python.

      • Sanjoy Datta says:

        Been able to follow your blog so far. Now stuck at preparing input data. Need more help to convert data as in github to familiar csv formats.

        Without it not able to proceed with the notebook

        • Pulkit Sharma says:

          Hi Sanjoy,

          I have used the below given code to convert the bounding boxes into csv file:

          %pylab inline

          import os, sys, random
          import xml.etree.ElementTree as ET
          from glob import glob
          import pandas as pd
          from shutil import copyfile

          annotations = glob('BCCD/Annotations/*.xml')

          df = []
          cnt = 0
          for file in annotations:
          prev_filename = file.split('/')[-1].split('.')[0] + '.jpg'
          filename = str(cnt) + '.jpg'
          row = []
          parsedXML = ET.parse(file)
          for node in parsedXML.getroot().iter('object'):
          blood_cells = node.find('name').text
          xmin = int(node.find('bndbox/ymin').text)
          xmax = int(node.find('bndbox/ymax').text)
          ymin = int(node.find('bndbox/xmin').text)
          ymax = int(node.find('bndbox/xmax').text)

          row = [prev_filename, filename, blood_cells, xmin, xmax,
          ymin, ymax]
          df.append(row)
          cnt += 1

          data = pd.DataFrame(df, columns=['prev_filename', 'filename', 'cell_type',
          'xmin', 'xmax', 'ymin', 'ymax'])

          data[['filename', 'cell_type', 'xmin', 'xmax', 'ymin', 'ymax']].to_csv('blood_cell_detection.csv', index=False)

          • Sanjoy Datta says:

            Dear Pulkit,
            Thanks a ton for the code. Ran it. But it is not populating the variable based on
            ‘annotations = glob(…..)’.
            Leading to:
            len(annotations) = 0.
            Hence the blood_cell_detection.csv it is creating is with only the header. No data is coming in.

            I must be doing some silly mistake. But not able to catch it. Got stuck again. What am I doing wrong?

            Sorry for bothering you again.
            Regards
            Sanjoy.

            • Pulkit Sharma says:

              Hi Sanjoy,

              Make sure to give the correct location of .xml files inside glob(). Replace ‘BCCD/Annotations/*.xml’ with the location where your .xml files are saved.

              • Sanjoy Datta says:

                Thank you Pulkit for prompt response again.

                Problem is elsewhere now. In short, do not have the xml and jpeg files yet at local folder.

                Have downloaded train.rec and val.rec. Not been able to extract .xml and .jpeg files out of them.

                So problem is stated as:
                1. How to extract xml and jpeg files from .rec files?
                2. Or, is there a way to download files directly from respective github folders? Been searching stackoverflow. But not found a way

                It appears from other posts, solution will help a few others like me too.

                Thank you in advance
                Sanjoy

  • laurent cesaro says:

    Hi,
    Thanks for this tutorial !

    Do you have installed the last package of tensorflow here ?

    Thanks
    Laurent cESARO

  • shravankumar P says:

    Very well documented all your learnings, thanks for sharing. Keep it going, all the best.

  • Michael Thomas says:

    Hi,
    I’m wondering if it’s possible to load existing weights from a pre-trained model such as Imagenet?
    Also how does the faster RCNN model compare to the Mask-RCNN model?
    Cheers
    Michael

    • Pulkit Sharma says:

      Hi Michael,

      We can use weights from any of the model which is trained for object detection.
      Faster RCNN predicts the bounding box coordinates whereas, Mask RCNN is used for pixel-wise predictions. So, it totally depends on the type of problem that you want to solve. In my opinion, both of these algorithms are good and can be used depending on the type of problem in hand.

  • Shilpa says:

    Hi Pulkit,

    Can you please also share the config file generated after training the model? Test_frcnn is looking for config file as well.

  • Sonal says:

    Thanks for the article. I am a beginner in ML / DS field.

    Can I use this code to detect the Region of Interest (ROI) from Glaucoma images?

    • Pulkit Sharma says:

      Hi Sonal,

      Yes, you can use this for your own dataset as well. You just have to make a .txt file in the same format as I have used.

  • Sanjoy Datta says:

    Thank you Pulkit. It is working finally. Thanks for all your help.

  • Hareesh Kumar says:

    Hi ,
    Thanks for the article.
    I am not able to train model on tensorflow-gpu. I get error
    “ValueError: Shape must be rank 1 but is rank 0 for ‘bn_conv1_7/Reshape_4’ (op: ‘Reshape’) with input shapes: [1,1,1,64], [].”
    Keras version on server : 2.2.4
    Tensorflow version on server: 1.11.0

    The training works fine without any issues on cpu machine.
    Please help.

  • sset says:

    How do we know when to stop training? For very small object detections any parameters to tune? Any anchor sizes, RPN to change? Thanks

    • Pulkit Sharma says:

      Hi,

      You can make a validation set from the training data and if the validation accuracy stops to increase or if it starts to decrease, you can say that the model is overfitting and you can stop the training at that point.
      Regarding the hyperparameter tuning, you can always try different set of hyperparameters to check whether they improve the model performance. There is no clear-cut answer to what hyperparameters should be used. You always have to try a range of hyperparameters and select the one which gives highest performance. Same goes with anchor sizes and RPN.

  • Shilpa says:

    Thanks !!

  • Hareesh Kumar says:

    Hi ,
    Can you tell us how did you arrive at the img_channel_mean, classifier_regr_std values and whats is the need of it?
    # image channel-wise mean to subtract
    self.img_channel_mean = [103.939, 116.779, 123.68]
    # scaling the stdev
    Similarly for self.classifier_regr_std = [8.0, 8.0, 4.0, 4.0]

    • Pulkit Sharma says:

      Hi Hareesh,

      These are the pre-processing steps which make the training faster. These are used to normalize the data. The values are taken by the people who have created the keras_frcnn library. They must have done experiments using different values and found this set as the best. You can try to change these values as well.

  • Asad Rauf Khan says:

    >>>train_images: Images that we will be using to train the model. We have the classes and the actual bounding boxes for each class in this folder.
    test_images: Images in this folder will be used to make predictions using the trained model. This set is missing the classes and the bounding boxes for these classes.
    train.csv: Contains the name, class and bounding box coordinates for each image. There can be multiple rows for one image as a single image can have more than one object.
    >>>

    Even tough you have provided the code to convert xml to csv, I’m not sure if I have reliably run it and I have a valid data. Can you please provide the exact data set with the exact train/test ratio so I can get results results identical to yours?

    • Pulkit Sharma says:

      Hi Asad,

      Once you have the complete csv file, you can divide it randomly in a ratio of say 70:30 (train:test) or 80:20 or any other ratio you want. Then you will have two different csv files, say train.csv and test.csv. Finally, for the images in train.csv, put them in train folder and images in test.csv file should be copied to test folder. Then you can run the model.

      • Pankaj says:

        Hi Pulkit,
        Can you please upload all the data files which you have exactly used for this and share?
        I am facing issues with splitting the datasets to train/test dataset. Thanks !

        • Pulkit Sharma says:

          Hi Pankaj,

          Can you please tell me what is the issue that you facing in splitting the data? Once you have the csv file (which you will get after running the code provided in the previous comments) you can randomly pick few images for the training and rest for the test. Please let me know what is the issue that you are facing so that I can help you to overcome it.

  • Pankaj says:

    Hi Pulkit,
    The command ran : python test_frcnn.py -p test_images successfully , but did not detect any bounding boxes in the image. Can you please suggest what is wrong ?

    Thanks,
    Pankaj

    • Pulkit Sharma says:

      Hi Pankaj,

      Make sure you have used the weights provided in the article. Or else you can train your model again and then make predictions for test_images.

  • Otavio Souza says:

    Hi!

    Great article!

    About weights that you make available, you know what the loss, the accuracy, time of the treining and you hardware that you used ?

    Thank you !

    • Pulkit Sharma says:

      Hi Otavio,

      The overall loss when I trained the model was around 2.3 The model was trained for more than 2 hours for 1000 epochs and I was using a GPU with 12 cores and 16 GB RAM

      • Otavio Souza says:

        I don’t understand, I am using a 2 GPU’s (Titan X 12G and Geforce GTX 1080 ti) and I took around 4 hours for 20 epochs.

        Did you change anything in files like config.py or train_frccn,py?

        I don’t know where I am going wrong

        I would like to ask, if you can post the complete codes that you used for train this model in github or in other plataform, because I’m not knowing how train the model very well

        Thank you!

      • RomDos says:

        Hi,

        Thanks for this article it is great !
        As I use the model on big images (4000×3000), the training is very long. I am using one Tesla K80 GPU, but I could find a way to train on multiple GPUs. Is there a way to do that ? How should the code be changed ?
        I have an other question about validation set. A validation set is created in train_frcnn but never I can not see where it is used after in the code. Am I wrong or is there a problem with that part ?

        Regards

        • Pulkit Sharma says:

          Hi,

          I am not yet aware of how to use multiple GPUs for training the model. I will look for it and will let you know how to do that.

          Regarding the validation set, they have passed the test images into the validation set as they have true labels and bounding boxes for test images. Whereas, in our case we do not have true labels for test images so we have not passed it in validation set.

  • Otavio Souza says:

    Hi!

    Great Article!

    How I load your weights for training more epochs?

    Thanks

    • Pulkit Sharma says:

      Hi Otavio,

      I think you have to retrain the model from the beginning. I am not aware of how to use the model weights and then start training on top of them. I will search for it and will let you know if any solution is found.

  • Tanuj Misra says:

    When I am running the code:
    python3 train_frcnn.py -o simple -p annotate3.txt;
    following error is coming:

    Using TensorFlow backend.
    Traceback (most recent call last):
    File “train_frcnn.py”, line 15, in
    from keras_frcnn import config, data_generators
    ModuleNotFoundError: No module named ‘keras_frcnn’

    Also showing ‘keras_frcnn’ is not available when trying to install/import it explicitly.

  • AMM says:

    Hi sir, thank you for this article, pleas can i apply this code on facial component detection?

    • Pulkit Sharma says:

      Hi,

      Yes! You can apply this algorithm for facial component detection. For that you first have to train your model on facial components. Weights used in this model will not help in detecting facial components as this model is trained on different dataset. So, first train your model on facial components and then you can use those weights to detect facial components in new set of images.

  • Jyoti says:

    Hi Pulkit – thanks for this – great article. I wanted to understand if the training set can be extended. I wanted to train some of the classes from the “Luminoth” library – would that be possible – where can I add the extra class labels?

    Thanks.

    • Pulkit Sharma says:

      Hi Jyoti,

      You can make changes in the csv file. You can add classes in them and then pass those labels while training the model.

  • aaditya says:

    Hey pulkit,
    I ran this for epoch length=100 and epoch =5 still i am not getting any output at least some wrong output should be there. dont know the reason why?

    • Pulkit Sharma says:

      Hi Aaditya,

      Make sure you are giving the correct bounding box coordinates for their corresponding images while training the model. If you give incorrect coordinates, you will not get results. So, I would suggest before passing the coordinates and images for training, first plot some images with their corresponding bounding boxes and check whether the data is in correct form.

  • Sajjad says:

    Hi and thanks for the tutorial.
    I have a problem with bounding box detection in python. I use this tutorial on Stanford cars dataset.
    http://ai.stanford.edu/~jkrause/cars/car_dataset.html

    I provide train_annotate.txt and it runs successfully on a few epochs with loss value of 2.
    it’s natural because of the small number of epochs.
    but when I run test_frcnn.py on the test_images folder it saves pictures with no bounding box.
    I expected at least a wrong bounding box appear on pictures.
    would you please help me? thank you.
    and my other question is, how can i evaluate the accuracy of the trained model on test set?

    here is the test log for 5 test picture:

    physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
    000001.jpg
    Elapsed time = 6.0193772315979
    []
    000146.jpg
    Elapsed time = 1.3050158023834229
    []
    000150.jpg
    Elapsed time = 1.2876217365264893
    []
    000160.jpg
    Elapsed time = 0.6831655502319336
    []
    000162.jpg
    Elapsed time = 0.76279616355896
    []

    • Pulkit Sharma says:

      Hi Sajjad,

      Make sure that the training and testing images have similar shape, also you have done similar pre-processing on these images. If the distribution of train and test images is different, you will not get fair results. Also, try to run the model for more number of epochs so that it can learn the patterns and can become better.

      Regarding your other question, you can evaluate the performance of model on test images only if you have the true labels for the test images, i.e. the class of the objects and their actual bounding boxes. If you do not have these values, you will not be able to evaluate the model on test images.

  • Shivangi says:

    Hi Pulkit,

    I have two questions to ask
    1. During training, we are only training on one image at a time if I understood the code correctly right. So if I have to say train on 2500 training images then to complete one epoch in one go, then I have to set `epoch_length=2500`.
    2. Could you please explain the outputs losses `rpn_cls` and `rpn_reg`? I know they are related to RPN output predictions. What I got from the code that we are generating ground truth for RPN using numpy operations and comparing RPN network output and training. But why is it actually required?
    3. What exactly is metric bounding box accuracy? Could you explain in more detail.
    4. Also for me all the 4 losses are not going down simultaneously, all of them are like really fluctuating. I am using VGG-16, Any suggestions on that

    • Pulkit Sharma says:

      Hi Shivangi,

      1. 1 epoch is when we have gone through all the images once. So, epoch_length has nothing to do with the training images.

      2. The overall loss is calculated based on these losses. Here, one loss is for classification and other for regression. rpn_cls will tell us how accurately we have classified the correct class for the identified object and rpn_reg will tell how accurate our bounding boxes are.

      3. It is similar to rmse, not exactly same but conveys the similar message. It tells us how accurate our predicted bounding boxes are.

      4. The losses can fluctuate. Ideally they all should be synced but practically when we implement things, there can be fluctuations.

      Hope this helps!!

      • Shivangi says:

        Hi Pulkit,

        1. I know what epoch is, but if you look at the code you have used variable `epoch_length`. So my question is different. I want to know if I have to train on 2500 training images then to complete one epoch in one go, then I have to set `epoch_length=2500` because we are only training on one image at a time.

        2. If `rpn_clas` and `rpn_reg` are related to classification (say WBC, RBC etc) the what is classifier outputs `detector_cls` and `detector_regr` related to? I think they rpns are not related to classifier output, they are something else. Correct me if I am wrong

        3. Still, I can’t really make out what it is . Could you explain it a bit more?

        • Pulkit Sharma says:

          Hi Shivangi,

          1. I checked the codes and all the images are used in every epoch. Can you please tell in which part of the code you are having doubts?

          2. For detail on various losses, please refer to the losses.py which is inside the keras_frcnn folder of the github repository. Link for the repository is given in the article.

          3. I checked the codes and found out that this is the accuracy of classes and not the bounding boxes. It can be seen in this code: print(‘Classifier accuracy for bounding boxes from RPN: {}’.format(class_acc))

          • Shivangi says:

            1. Let me frame it differently, what is variable `epoch_length` doing in the code?

            2. Yes, `rpn_clas` and `rpn_reg` are NOT related to classification (WBC, RBC). They are related to object presence or not.

            3. Got it

            • Pulkit Sharma says:

              Hi Shivangi,

              1. If you look at this line of code:
              if iter_num == epoch_length:
              loss_rpn_cls = np.mean(losses[:, 0])
              loss_rpn_regr = np.mean(losses[:, 1])
              loss_class_cls = np.mean(losses[:, 2])
              loss_class_regr = np.mean(losses[:, 3])
              class_acc = np.mean(losses[:, 4])

              Suppose you define epoch_number as 1000, so after every 1000 iterations, all the losses will be calculated and the weights will be updated accordingly. In single epoch, we can have multiple iterations. This is also known as Mini Batch Gradient Descent. So, instead of taking single image for training and updating the weights, we take batches of images and train the model on them and update the weights accordingly. Once the model has seen all the images once, one epoch is complete.

      • Shivangi says:

        Could you please un-delete the comment which you deleted about the follow ups on your reply.

  • Md Ashraful Haque says:

    Hey, Pulkit, thanks for your article. Can you guide me on how to annotate the images with bounding box?? I am confused with this part?? plzz help me..
    Thanks

    • Pulkit Sharma says:

      Hi,

      You have to manually annotate the images if you are using a new dataset. Or else, you can look for open source datasets which have their corresponding bounding boxes.

  • qinjian says:

    Thanks.I want to know how to set GPU config.

  • Martin says:

    Hello !

    First of all, thanks you for this amazing tuto.

    I am having just a little problem :

    Using the weights you provided ( my PC is way too slow for getting good ones myself) as well as the option file, when i use test_frcnn.py the labels on WBC and RBC are inverted. The cells themselfs are very-well recognised, it is just that every single WBC is labelled as RBC and every single RBC is labelled as WBC … Any idea of what may have caused than ?

    • Pulkit Sharma says:

      Hi Martin,

      Can you please share the screenshot of some of the results that you are getting? That would help me to clarify your doubt in a better way.

%d bloggers like this:







Download Brochure
Join 250,000+ Data Scientists in our Community

Receive awesome tips, guides, infographics and become expert at:




 P.S. We only publish awesome content. We will never share your information with anyone.

Subscribe!
%d bloggers like this:







Download Brochure
Join 250,000+ Data Scientists in our Community

Receive awesome tips, guides, infographics and become expert at:




 P.S. We only publish awesome content. We will never share your information with anyone.

Subscribe!