How to Use Progress Bars in Python?

Hours of hard work would go into vain if our program becomes unresponsive during execution. Often we come across large datasets or longer loops that takes a long while to complete such as in Data Scraping.

While these commands are executing and massive loops are being processed behind the screen, it seems like an eternity of waiting time till the whole process is completed. Thus, Creating a Progress Bar would solve this problem.

Progress Bars would help us to look at the progress of our execution and to manage our anxiety levels. In this article, we will learn how to build Progress Bars in Python using Jupyter Notebook.

Tables of Contents

1. Introduction
2. Using tqdm()
3. Using tqdm_notebook()
4. Using tqdm() and tqdm_notebook() with Pandas
5. Comparison: Using tqdm() and tqdm_notebook() on Nested Loops
6. Conclusion

Introduction

tqdm is a library in Python which is used for creating Progress Meters or Progress Bars. tqdm got its name from the Arabic name taqaddum which means ‘progress’.

Implementing tqdm can be done effortlessly in our loops, functions or even Pandas. Progress bars are pretty useful in Python because:

1. One can see if the Kernal is still working

2. Progress Bars are visually appealing to the eyes

3. It gives Code Execution Time and Estimated Time for the code to complete which would help while working on huge datasets

Further down we will look at how to use tqdm(), tqdm_notebook() and their application In Pandas in Jupyter Notebook.

Using tqdm( )

tqdm gives a console kind of progress bar for our processes. Using tqdm is a straightforward process explained in the following steps:

1. Install the Requirements

First, we need to install our required library tqdm. Open a New Jupyter Notebook and execute:

!pip install tqdm
!pip install time

This will complete the installation of tqdm. Restart the Kernel to start using the tqdm.

2. Import the Libraries 

Import the newly installed tqdm and time library

from tqdm import tqdm
import time

3. Using tqdm() 

Now we will use the function tqdm() on a simple program with a  for loop.

for i in tq(range(20)):
    time.sleep(0.5)

Here i is the variable that takes a value of the number 0 to 19 during each iteration. During each iteration, the system will sleep for 0.5 seconds before moving to the next iteration.

The complete code would look like this:

from tqdm import tqdm
import time
for i in tq(range(20)):
    time.sleep(0.5)

On Completion of Code Execution, we get:

We can also give attributes to tqdm() such as desc, which takes a string and will get added as a prefix before the progress bar. Thus,

from tqdm import tqdm
import time
for i in tqdm(range(20), desc = 'tqdm() Progress Bar'):
    time.sleep(0.5)

On Completion of Code Execution, we get:

Image Source – Personal Computer

Apart from the progress bar, tqdm gives additional information such as the number of iterations completed out of the total number of iterations, Total Elapsed Time, Estimated Time to Complete the whole loop, and the speed of the loop in iterations per second (or it/s).

Using tqdm_notebook( )

Unlike the tqdm(), tqdm_notebook() gives a coloured version of progress bars. It has 3 sets of colour by default.

A moving Blue Bar shows for a process undergoing, a stable Green Bar shows that the process is completed, A Red Bar shows that process is being stopped. Interestingly, like the tqdm(), tqdm_notebook() too has a straightforward way of implementation.

1. Import the Libraries

from tqdm.notebook import tqdm_notebook
import time

2. Using tqdm_notebook()

for i in tqdm_notebook(range(10)):
    time.sleep(0.5)

Here i is the variable that takes a value of the number 0 to 19 during each iteration. During each iteration, the system will sleep for 0.5 seconds before moving to the next iteration.

The complete code would look like this:

from tqdm.notebook import tqdm_notebook
import time
for i in tqdm_notebook(range(10)):
    time.sleep(0.5)

On Executing the Code, we get:

Image Source – Personal Computer

On Completion of Code execution, we get:

Image Source – Personal Computer

If the Code execution is Terminated:

Image Source – Personal Computer

Notice the colour of the Progress Bar.

We can give additional arguments into tqdm_notebook() such as desc which add a prefix to the Progress Bar. The Code would look like this:

from tqdm.notebook import tqdm_notebook
import time
for i in tqdm_notebook(range(10), desc = 'Progress using tqdm_notebook()'):
    time.sleep(0.5)

On Executing the Code, we get:

Using tqdm_notebook() in Pandas

Both tqdm() and tqdm_notebook() can be used in Pandas. One way is to use them with for loops with Pandas Series which works the same as with the loops we have seen earlier. Another way is to use them in Pandas .apply() method. To use tqdm() or tqdm_notebook() for .apply(), the .apply() needs to be replaced by .progress_apply().

For example, let’s take a dataset from Kaggle:

import pandas as pd
df = pd.read_csv("WA_Fn-UseC_-Telco-Customer-Churn.csv")
df.head()
tqdm_notebook.pandas()
df['Churn'] = df['Churn'].progress_apply(lambda x: 1 if x == 'Yes' else 0)

On Completion of Code Execution, we get:

Image Source – Personal Computer

The tqdm_notebook.pandas() is responsible for displaying the progress bar. The progress bar shows the apply function being applied on all values of the Churn column. It is noteworthy that, the progress.apply() works the same as the .apply() method of Pandas.

Comparison: Using tqdm_notebook() and tqdm() on Nested                                    Loops

First, Let’s see an example of using tqdm() on nested loops:

from tqdm import tqdm
for i in tqdm(range(2), desc = 'Loop 1'):
    for j in tqdm(range(20,25), desc = 'Loop 2'):
        time.sleep(0.5)

On Completion of Code Execution, we get:

Image Source – Personal Computer

Now, let’s see tqdm_notebook( ) on nested loops:

from tqdm.notebook import tqdm_notebook
for i in tqdm_notebook(range(2), desc = 'Loop 1'):
    for j in tqdm_notebook(range(20,25), desc = 'Loop 2'):
        time.sleep(0.5)

On Completion of Code Execution, we get:

Image Source – Personal Computer

It is evident that tqdm() has more progress bars. For each iteration of Loop 1, it shows a separate progress bar and in each iteration of Loop 1, it shows individual progress bars for each iteration of Loop 2.

Thus, for the first progress bar of Loop 1, it shows 5 progress bars of Loop 2, then again for the second progress bar of Loop 1, it shows 5 separate progress bars of Loop 2 and so on.

In contrast to tqdm(), Progress Bars in tqdm_notebook() are very intuitive. It shows only one progress bar for Loop 1 and for each iteration of Loop 1, one progress bar of Loop 2. It might sound difficult to read but it becomes understandable once you run the code.

Conclusion

Thus, tracking the progresses helps a lot of time and saves us from the state of confusion. Apart from the desc attribute of both tqdm() and tqdm_notebook(), it takes more attributes that depend upon the end-user and its need.

Using tqdm() has one more advantage that it has very detailed documentation which would help one to refer anytime. There is numerous customization possible to the progress bars and more might come in the future updates.

Apart from tqdm and tqdm_notebook, one can track progress using progressbar, progressbar2, and Alive progress.

About the Author

Connect with me on LinkedIn Here.

Check out my other Articles Here and on Medium

You can provide your valuable feedback to me on LinkedIn.

Thanks for giving your time!

The media shown in this article are not owned by Analytics Vidhya and is used at the Author’s discretion.