It’s a Record-Breaking Crowd! A Must-Read Tutorial to Build your First Crowd Counting Model using Deep Learning

Pulkit Sharma 04 Jan, 2021 • 9 min read

Introduction

Artificial Intelligence and Machine Learning is going to be our biggest helper in coming decade!

Today morning, I was reading an article which reported that an AI system won against 20 lawyers and the lawyers were actually happy that AI can take care of repetitive part of their roles and help them work on complex topics. These lawyers were happy that AI will enable them to have more fulfilling roles.

Today, I will be sharing a similar example – How to count number of people in crowd using Deep Learning and Computer Vision? But, before we do that – let us develop a sense of how easy the life is for a Crowd Counting Scientist.

 

Act like a Crowd Counting Scientist

Let’s start!

Can you help me count / estimate number of people in this picture attending this event?

Ok – how about this one?

Crowd image

Source: ShanghaiTech Dataset

You get the hang of it. By end of this tutorial, we will create an algorithm for Crowd Counting with an amazing accuracy (compared to humans like you and me). Will you use such an assistant?

 

P.S. This article assumes that you have a basic knowledge of how convolutional neural networks (CNNs) work. You can refer to the below post to learn about this topic before you proceed further:

 

Table of Contents

  1. What is Crowd Counting?
  2. Why is Crowd Counting required?
  3. Understanding the Different Computer Vision Techniques for Crowd Counting
  4. The Architecture and Training Methods of CSRNet
  5. Building your own Crowd Counting model in Python

This article is highly inspired by the paper – CSRNet : Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes.

 

What is Crowd Counting?

Crowd Counting is a technique to count or estimate the number of people in an image. Take a moment to analyze the below image:

Crowd image

Source: ShanghaiTech Dataset

Can you give me an approximate number of how many people are in the frame? Yes, including the ones present way in the background. The most direct method is to manually count each person but does that make practical sense? It’s nearly impossible when the crowd is this big!

Crowd scientists (yes, that’s a real job title!) count the number of people in certain parts of an image and then extrapolate to come up with an estimate. More commonly, we have had to rely on crude metrics to estimate this number for decades.

Surely there must be a better, more exact approach?

Yes, there is!

While we don’t yet have algorithms that can give us the EXACT number, most computer vision techniques can produce impressively precise estimates. Let’s first understand why crowd counting is important before diving into the algorithm behind it.

 

Why is Crowd Counting useful?

Let’s understand the usefulness of crowd counting using an example. Picture this – your company just finished hosting a huge data science conference. Plenty of different sessions took place during the event.

You are asked to analyze and estimate the number of people who attended each session. This will help your team understand what kind of sessions attracted the biggest crowds (and which ones failed in that regard). This will shape next year’s conference, so it’s an important task!

Image result for conference

There were hundreds of people at the event – counting them manually will take days! That’s where your data scientist skills kick in. You managed to get photos of the crowd from each session and build a computer vision model to do the rest!

There are plenty of other scenarios where crowd counting algorithms are changing the way industries work:

  • Counting the number of people attending a sporting event
  • Estimating how many people attended an inauguration or a march (political rallies, perhaps)
  • Monitoring of high-traffic areas
  • Helping with staffing allocation and resource allotment

Can you come up with some other use cases? Let me know in the comments section below! We can connect and try to figure out how we can use crowd counting techniques in your scenario.

 

Understanding the Different Computer Vision Techniques for Crowd Counting

Broadly speaking, there are currently four methods we can use for counting the number of people in a crowd:

1. Detection-based methods

Here, we use a moving window-like detector to identify people in an image and count how many there are. The methods used for detection require well trained classifiers that can extract low-level features. Although these methods work well for detecting faces, they do not perform well on crowded images as most of the target objects are not clearly visible.

2. Regression-based methods

We were unable to extract low level features using the above approach. Regression-based methods come up trumps here. We first crop patches from the image and then, for each patch, extract the low level features.

3. Density estimation-based methods

We first create a density map for the objects. Then, the algorithm learn a linear mapping between the extracted features and their object density maps. We can also use random forest regression to learn non-linear mapping.

4. CNN-based methods

Ah, good old reliable convolutional neural networks (CNNs). Instead of looking at the patches of an image, we build an end-to-end regression method using CNNs. This takes the entire image as input and directly generates the crowd count. CNNs work really well with regression or classification tasks, and they have also proved their worth in generating density maps.

CSRNet, a technique we will implement in this article, deploys a deeper CNN for capturing high-level features and generating high-quality density maps without expanding the network complexity. Let’s understand what CSRNet is before jumping to the coding section.

 

Understanding the Architecture and Training Method of CSRNet

CSRNet uses VGG-16 as the front end because of its strong transfer learning ability. The output size from VGG is ⅛th of the original input size. CSRNet also uses dilated convolutional layers in the back end.

But what in the world are dilated convolutions? It’s a fair question to ask. Consider the below image:

Dilated Convolutions

The basic concept of using dilated convolutions is to enlarge the kernel without increasing the parameters. So, if the dilation rate is 1, we take the kernel and convolve it on the entire image. Whereas, if we increase the dilation rate to 2, the kernel extends as shown in the above image (follow the labels below each image). It can be an alternative to pooling layers.

 

Underlying Mathematics (Recommended, but optional)

I’m going to take a moment to explain how the mathematics work. Note that this isn’t mandatory to implement the algorithm in Python, but I highly recommend learning the underlying idea. This will come in handy when you need to tweak or modify your model.

Suppose we have an input x(m,n), a filter w(i,j), and the dilation rate r. The output y(m,n) will be:

Dilation output

We can generalize this equation using a (k*k) kernel with a dilation rate r. The kernel enlarges to:

([k + (k-1)*(r-1)] * [k + (k-1)*(r-1)])

So the ground truth has been generated for each image. Each person’s head in a given image is blurred using a Gaussian kernel. All the images are cropped into 9 patches, and the size of each patch is ¼th of the original size of the image. With me so far?

The first 4 patches are divided into 4 quarters and the other 5 patches are randomly cropped. Finally, the mirror of each patch is taken to double the training set.

That, in a nutshell, are the architecture details behind CSRNet. Next, we’ll look at its training details, including the evaluation metric used.

Stochastic Gradient Descent is used to train the CSRNet as an end-to-end structure. During training, the fixed learning rate is set to 1e-6. The loss function is taken to be the Euclidean distance in order to measure the difference between the ground truth and estimated density map. This is represented as:

Loss function

where N is the size of the training batch. The evaluation metric used in CSRNet is MAE and MSE, i.e., Mean Absolute Error and Mean Square Error. These are given by:

MAE and MSE

Here, Ci is the estimated count:

Estimated count

L and W are the width of the predicted density map.

Our model will first predict the density map for a given image. The pixel value will be 0 if no person is present. A certain pre-defined value will be assigned if that pixel corresponds to a person. So, calculating the total pixel values corresponding to a person will give us the count of people in that image. Awesome, right?

And now, ladies and gentlemen, it’s time to finally build our own crowd counting model!

 

Building your own Crowd Counting model

Ready with your notebook powered up?

We will implement CSRNet on the ShanghaiTech dataset. This contains 1198 annotated images of a combined total of 330,165 people. You can download the dataset from here.

Use the below code block to clone the CSRNet-pytorch repository. This holds the entire code for creating the dataset, training the model and validating the results:

git clone https://github.com/leeyeehoo/CSRNet-pytorch.git

Please install CUDA and PyTorch before you proceed further. These are the backbone behind the code we’ll be using below.

Now, move the dataset into the repository you cloned above and unzip it. We’ll then need to create the ground truth values. The make_dataset.ipynb file is our savior. We just need to make minor changes in that notebook:

 

 

#setting the root to the Shanghai dataset you have downloaded
# change the root path as per your location of dataset
root = '/home/pulkit/CSRNet-pytorch/'

Now, let’s generate the ground truth values for images in part_A and part_B:

 

Generating the density map for each image is a time taking step. So go brew a cup of coffee while the code runs.

So far, we have generated the ground truth values for images in part_A. We will do the same for the part_B images. But before that, let’s see a sample image and plot its ground truth heatmap:

plt.imshow(Image.open(img_paths[0]))

ShanghaiTech part_A train image

Things are getting interesting!

gt_file = h5py.File(img_paths[0].replace('.jpg','.h5').replace('images','ground-truth'),'r')
groundtruth = np.asarray(gt_file['density'])
plt.imshow(groundtruth,cmap=CM.jet)

Let’s count how many people are present in this image:

np.sum(groundtruth)

270.32568

Similarly, we will generate values for part_B:

Now, we have the images as well as their corresponding ground truth values. Time to train our model!

We will use the .json files available in the cloned directory. We just have to change the location of the images in the json files. To do this, open the .json file and replace the current location with the location where your images are located.

Note that all this code is written in Python 2. Make the following changes if you’re using any other Python version:

  1. In model.py, change xrange in line 18 to range
  2. Change line 19 in model.py with: list(self.frontend.state_dict().items())[i][1].data[:] = list(mod.state_dict().items())[i][1].data[:]
  3. In image.py, replace ground_truth with ground-truth

Made the changes? Now, open a new terminal window and type the following commands:

cd CSRNet-pytorch
python train.py part_A_train.json part_A_val.json 0 0

Again, sit back because this will take some time. You can reduce the number of epochs in the train.py file to accelerate the process. A cool alternate option is to download the pre-trained weights from here if you don’t feel like waiting.

Finally, let’s check our model’s performance on unseen data. We will use the val.ipynb file to validate the results. Remember to change the path to the pretrained weights and images.

#defining the image path
img_paths = []
for path in path_sets:
    for img_path in glob.glob(os.path.join(path, '*.jpg')):
       img_paths.append(img_path)

 

model = CSRNet()

 

#defining the model
model = model.cuda()

 

#loading the trained weights
checkpoint = torch.load('part_A/0model_best.pth.tar')
model.load_state_dict(checkpoint['state_dict'])

Check the MAE (Mean Absolute Error) on test images to evaluate our model:

We got an MAE value of 75.69 which is pretty good. Now let’s check the predictions on a single image:

CSRNet outputWow, the original count was 382 and our model estimated there were 384 people in the image. That is a very impressive performance!

Congratulations on building your own crowd counting model!

 

End Notes

I encourage you to try out this approach on different images and share your results in the comments section below. Crowd counting has so many diverse applications and is already seeing adoption by organizations and government bodies.

It is a useful skill to add to your portfolio. Quite a number of industries will be looking for data scientists who can work with crowd counting algorithms. Learn it, experiment with it, and give yourself the gift of deep learning!

Did you find this article useful? Feel free to leave your suggestions and feedback for me below, and I’ll be happy to connect with you.

You should also check out the below resources to learn and explore the wonderful world of computer vision:

Pulkit Sharma 04 Jan 2021

My research interests lies in the field of Machine Learning and Deep Learning. Possess an enthusiasm for learning new skills and technologies.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Saurabh Singh
Saurabh Singh 18 Feb, 2019

Nicely explained the complex concept of Crowd counting . And its very helpful.

Preeti
Preeti 18 Feb, 2019

Very nicely explained. Good one Pulkit

sandesh
sandesh 18 Feb, 2019

Great learning through your article .

vivek
vivek 20 Feb, 2019

The zip file with pre-trained weights is corrupt , https://drive.google.com/file/d/1KY11yLorynba14Sg7whFOfVeh2ja02wm/view?usp=sharing Could you provide the correct file please? "A cool alternate option is to download the pre-trained weights from here if you don’t feel like waiting."

Mohan
Mohan 23 Feb, 2019

Hi, I'm getting an issue, when trying to replicate the concept. I kept the screenshots here about the issue as well as the part of the code where it is occurring. https://www.dropbox.com/sh/vkyqzogbhvzdj8s/AABC337K_nUxcA1y78OpCswja?dl=0

Yogesh
Yogesh 23 Feb, 2019

I thought I won't be able to deal with computer vision. But after reading this article i am amazed.

Nimish
Nimish 03 Mar, 2019

Hi Sir! I tried this method using CPU version of torch, and I modified the code accordingly to make it work. But after one picture has een evaluated, the RAM usage suddenly booms and freezes my system entirely. Is this a memory leak on my part, or does the model require a lot of RAM? (I have 8GB RAM)

Aditya Ayyagari
Aditya Ayyagari 19 Mar, 2019

How do we give the path in Windows? UNABLE TO GENERATE GROUND TRUTHS! Does this code run only on Linux? What changes must be made to replicate this code on Windows considering that the system has Cuda, Anaconda with Python3.7 and PyTorch ?

Anjalina
Anjalina 22 Mar, 2019

Any option to bypass this cuda since my machine has nvidea GPU 410M only which is not supporting the cuda9.X

Beenu
Beenu 22 Mar, 2019

First of all I am very new to this. Is there any alternative for cuda. My machine support only nvidea GE 410M, which is of cuda capacity 3.X only.

yuriy
yuriy 23 Mar, 2019

Hey, thanks for the article I'm running CUDA on my laptop with 16gb ram and 2gb video memory but alway caught the "CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 2.00 GiB total capacity; 927.71 MiB already allocated; 51.00 MiB free; 355.54 MiB cached)" what is the requirements for training? or how can I fix this?

Mr. T
Mr. T 29 Mar, 2019

Hi Pulkit, I love the tutorial, but the theory about making ground truth and density map were a bit hard for me. Could you give some sources to find an explanation of how to generate ground truth without counting the number of people in images manually? Rgs, Mr. T

beenu
beenu 02 Apr, 2019

please correct me if I am wrong. I am very new to this field.This might be a simple question for you. From this article what I could understand is evenif I supply any image from my local collection, the prediction.py will provide the crowed count. With this assumption when I tried, for example I have added IMG5000.jpg in the images and tried to run prediction.py, its complaining about the h5 and mat files since its not there in the dataset. mat = io.loadmat(img_path.replace('.jpg','.mat').replace('images','ground-truth').replace('IMG_','GT_IMG_')) is complaning about this.

esther
esther 17 Apr, 2019

How do you get the .mat files for each image? Thanks.

Esther Mead
Esther Mead 21 Apr, 2019

Thank you for this. How can I use it to count the number of people in unlabeled images?

JJ
JJ 21 Apr, 2019

I highly appreciate this post Sir Pulkit Sharma. Thank you very much! But Sir, I have a question. The official CSRNet paper states that they did different kernel values for different datasets such as UCF_CC_50. Correct me Sir If I'm wrong about it. How can correcly implement those fixed kernel values on this code? Thank you very much Sir!

Esther Mead
Esther Mead 23 Apr, 2019

Pulkit, thank you so much for this. I've gotten everything to work using google colab with GPU acceleration. Now, how can I use this to test new images? Thanks!

Arun
Arun 25 Apr, 2019

Your article is very helpful. Thanks a lot for explaining this in a better way. But when I tested the "part_B_final/test_data_backup/images/IMG_1.jpg" with the above code(reply to Esther), I could see the count that we are getting is 54 , actual the count is only less than 30. Similarly getting some weird counts if I try images with very less count , say 1, 2 or three. May I know whether this issue is with the the way that I am executing this code or this code is not considering all those cases

Aiswarya
Aiswarya 30 Apr, 2019

Hi Pulkit, Awesome doc. Thanks you for posting this. This doc definitely would be helpful to all those who are interested to explore in this area. I am facing an issue while running this code. It would be great if you could suggest a solution for the same. I have 14GB RAM and the following is the nvidia-smi console output. +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce MX110 Off | 00000000:01:00.0 Off | N/A | | N/A 49C P0 N/A / N/A | 443MiB / 2004MiB | 3% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1480 G /usr/lib/xorg/Xorg 172MiB | | 0 1651 G /usr/bin/gnome-shell 118MiB | | 0 5000 G ...quest-channel-token=8194962126415318391 146MiB | | 0 6268 G /snap/pycharm-community/123/jre64/bin/java 2MiB | I am getting frequent Runtimeerror, even-if I try to predict the crowed count of 150 kb file. The error I am getting is as follows: RuntimeError: CUDA out of memory. Tried to allocate 174.38 MiB (GPU 0; 1.96 GiB total capacity; 1.31 GiB already allocated; 65.50 MiB free; 1.12 MiB cached). The following the code base that I am running: import PIL.Image as Image import numpy as np from matplotlib import pyplot as plt from image import * from model import CSRNet import torch import gc; gc.collect() torch.cuda.empty_cache() from torchvision import datasets, transforms transform=transforms.Compose([ transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ]) model = CSRNet() model = model.cuda() #loading the trained weights checkpoint = torch.load('/home/abc/Downloads/Shanghai/part_A_final/0model_best.pth.tar') model.load_state_dict(checkpoint['state_dict']) from matplotlib import cm as c img = transform(Image.open("/home/abc/Downloads/Shanghai/xyz.jpg").convert('RGB')).cuda() output = model(img.unsqueeze(0)) print("Predicted Count : ",int(output.detach().cpu().sum().numpy())) temp = np.asarray(output.detach().cpu().reshape(output.detach().cpu().shape[2],output.detach().cpu().shape[3])) plt.imshow(temp,cmap = c.jet) plt.show()

Bharth singh
Bharth singh 30 Apr, 2019

Hi Pulkit, The '0model_best.pth.tar' contain the pretrained weight of the 1198 images(ie the train and test images of part_A and part_B) right?

mishal
mishal 01 May, 2019

Hi Pulkit, Excellent explanation, keep on the good work! I am running the code on Anaconda python 3 on windows and it did not like the following line if gt_count > 1: in the "# function to create density maps for images" Line 20 it complained about the semicolon. is there a problem in the syntax ? Also I had to install openCV which was not mentioned in the explanation but needed by the code.

Beenu
Beenu 02 May, 2019

Thanks a lot for this doc. For the crowed prediction, DO you think it has some dependency with the image size and pixels. I could see the trian images are with a dimensions of 1024*768 . Do you think if the image size and dimension is bigger then the accuracy will also be more fine tuned.

Nihed
Nihed 16 May, 2019

Hi! great article! I have stumped into a little problem and I do not know how to fix it. This is the error I am receiving: IndexError Traceback (most recent call last) in () 1 #now see a sample from ShanghaiA ----> 2 plt.imshow(Image.open(img_paths[0])) IndexError: list index out of range The location of my root is this: #set the root to the Shanghai dataset you download root = '/Desktop/CSRNet-pytorch-master/ShanghaiTech' #now generate the ShanghaiA's ground truth part_A_train = os.path.join(root,'part_A/train_data','images') part_A_test = os.path.join(root,'part_A/test_data','images') part_B_train = os.path.join(root,'part_B/train_data','images') part_B_test = os.path.join(root,'part_B/test_data','images') path_sets = [part_A_train,part_A_test] I do not understand why I am receiving the error Could someone help me?

Andrea
Andrea 17 May, 2019

Any chance to run it without cuda?? i just want to use my cpu

Nihed
Nihed 20 May, 2019

Hi Pulkit! I do not have an GPU processor that works for this.. How do I use Google Colab? OR could you maybe send me the right pre weights because your I am unable to open your tar. file.. I am working on a MacBook Pro Kind regards, Nihed

Aayush Jain
Aayush Jain 22 May, 2019

Hi Pulkit , Can you share the drive link for pre trained weights again ? it seems as if it is damaged because i tried open it in both linux and windows but failed to do so . It will be really helpful !

rasool
rasool 23 May, 2019

can i do this code by using goggle colab? if yes how can execute the command of terminal in colab?

Rahul Sharma
Rahul Sharma 26 May, 2019

I am new in this field, i know only basics of machine learning. What are the pre-requisite to know this article completely. If you have source, then please send me their links. Thank you .

Jarvah
Jarvah 27 May, 2019

Hi! This is a great article! Can this framework be used to do people detection?

Mike
Mike 06 Jun, 2019

How long does it take (approximately) to determine the number of people in a crowd? Can this be used At the time of an event or is it more of an after the event

Bharath
Bharath 13 Jun, 2019

Hi, I am new to this field. Sorry if it is a very simple question, In the cool method. i am getting an error in this line gt_file = h5py.File(img_paths[i].replace('.jpg','.h5').replace('images','ground-truth'),'r'). Saying OS error:File Signature not found. Is this because file could be corrupted or is it not in HDF5 format? can you please help me. Thanks

Vignesh
Vignesh 14 Jun, 2019

As, I am very new to machine learning. Kindly, clear me this error. I cant find .h5 file inside gt_file='part_A/test_data/ground-truth/IMG_100.h5',r in Part-B Cool Method .Thank you

Cecilia
Cecilia 20 Jun, 2019

Hi Pulkit, Im trying to generate ground truth for my own dataset, is there any easy way to do that ?

Ahmed
Ahmed 21 Jun, 2019

Hi Pulkit, Thank you very much for this tutorial. I am trying to extract that .tar file with pre-trained weights on windows and i am getting error that file is not archive. Can you provide other file format or do you know solution to my problem. Thanks

Minoru Aikawa
Minoru Aikawa 26 Jun, 2019

This is really great work! very impressed! I'd like to know whether I can use these scripts for commercial use.

Trumand
Trumand 26 Jun, 2019

This is really great work! congratulation! Do you have the progress of network training?

somayah
somayah 22 Jul, 2019

Thank for the great tutorial. How I can visualize the counted heads? either by boundary boxes or points on the considered head? Thank you

Rommy Shrivastav
Rommy Shrivastav 22 Jul, 2019

Hello Sir,I just wanted to know what to do If I want to run it through my CPU because its giving me Assertion Error.Aren't there any other possible ways? I have 8GB RAM and 2GB graphics card..!!!

Suraj Dakua
Suraj Dakua 06 Aug, 2019

Hello Pulkit Sharma, Can we merge Part A and Part B images instead of splitting in two parts? Thankyou in advance.

harsh
harsh 22 Aug, 2019

Hello Pulkit, I am not able to execute the code. The system setup is pycharm professional 2019 (trial) python 3.6.6 cuda cudnn tensorflow pytorch I have installed jupyter in pycharm, and trying to run, but getting no output. nor that tar file is getting opened. please help with how to execute

ruchika
ruchika 23 Aug, 2019

how long does it take to train the network

Divyakant Tahlyan
Divyakant Tahlyan 18 Sep, 2019

Was the pre-trained model that you provided trained on part A dataset or part B dataset. If it was for part A, can you please provide the one trained on part B as well.

Terry
Terry 20 Sep, 2019

Hi Pulkit, Thanks very much for sharing your work. When I tried the following code: for img_path in img_paths: print (img_path) mat = io.loadmat(img_path.replace('.jpg','.mat').replace('images','ground-truth').replace('IMG_','GT_IMG_')) img= plt.imread(img_path) k = np.zeros((img.shape[0],img.shape[1])) gt = mat["image_info"][0,0][0,0][0] for i in range(0,len(gt)): if int(gt[i][1])<img.shape[0] and int(gt[i][0])<img.shape[1]: k[int(gt[i][1]),int(gt[i][0])]=1 k = gaussian_filter_density(k) with h5py.File(img_path.replace('.jpg','.h5').replace('images','ground-truth'), 'w') as hf: hf['density'] = k I got a ValueError: not enough values to unpack (expected 2, got 0) Do you know what's the problem that cause the error?

James Ryan
James Ryan 08 Oct, 2019

Hi Pulkit, I only want to be able to predict the number of people in a given image, nothing else. With that being said, do I need to install CUDA and PyTorch? Also, what parts of your code would I require for just this functionality? Thank you very much.

Baldev sharma
Baldev sharma 15 Oct, 2019

1. Is it compulsory to use NVIDIA graphic card? 2. Is google colab support NVIDIA card only? 3. can we use anaconda 3 version ?

Akbar
Akbar 27 Jun, 2022

Hi Pulkit, the link to weights file does not exist. Can you share the weights file location. Thanks

Thinh
Thinh 06 Dec, 2023

Hi your pre train link does not work can u update the link thanks alot

Computer Vision
Become a full stack data scientist