Ashish — Published On June 21, 2021 and Last Modified On December 5th, 2022
Beginner Libraries Pandas Python Unstructured Data

This article was published as a part of the Data Science Blogathon


Data Extraction is the process of extracting data from various sources such as CSV files, web, PDF, etc. Although in some files, data can be extracted easily as in CSV, while in files like unstructured PDFs we have to perform additional tasks to extract data from PDF Python.

There are a couple of Python libraries using which you can extract data from PDFs. For example, you can use the PyPDF2 library for extracting text from PDFs where text is in a sequential or formatted manner i.e. in lines or forms. You can also extract tables in PDFs through the Camelot library. In all these cases data is in structured form i.e. sequential, forms or tables.

However, in the real world, most of the data is not present in any of the forms & there is no order of data. It is present in unstructured form. In this case, it is not feasible to use the above python libraries since they will give ambiguous results. To analyze unstructured data, we need to convert it to a structured form.

As such, there is no specific technique or procedure for extracting data from unstructured PDFs since data is stored randomly & it depends on what type of data you want to extract from PDF.

Here, I will show you a most successful technique & a python library through which you can extract data from bounding boxes in unstructured PDFs and then performing data cleaning operation on extracted data and converting it to a structured form.


I have used the PyMuPDF library for this purpose. This library provided many applications such as extracting images from PDF, extracting texts from different shapes, making annotations, draw a bounded box around the texts along with the features of libraries like PyPDF2.

Now, I will show you how I extracted data from the bounding boxes in a PDF with several pages.

Here are the PDF and the red bounding boxes from which we need to extract data.


Data extraction from pdf
Data extraction | image 2
Data extraction | image 3

I have tried many python libraries like PyPDF2, PDFMiner, pikepdf, Camelot, and tabulat. However, none of them worked except PyMuPDF to extract data from PDF using Python.

Before going into the code it’s important to understand the meaning of 2 important terms which would help in understanding the code.

Word: Sequence of characters without space. Ex – ash, 23, 2, 3.

Annots: An annotation associates an object such as a note, image, or bounding box with a location on a page of a PDF document, or provides a way to interact with the user using the mouse and keyboard. The objects are called annots.

Please note that in our case the bounding box, annots, and rectangles are the same thing. Therefore, these terms would be used interchangeably.

First, we will extract text from one of the bounding boxes. Then we will use the same procedure to extract data from all the bounding boxes of pdf.


import fitz
import pandas as pd 
doc ='Mansfield--70-21009048 - ConvertToExcel.pdf')
page1 = doc[0]
words = page1.get_text("words")

Firstly, we import the fitz module of the PyMuPDF library and pandas library. Then the object of the PDF file is created and stored in doc and 1st page of pdf is stored on page1. page.get_text() extracts all the words of page 1. Each word consists of a tuple with 8 elements.

In words variable, the First 4 elements represent the coordinates of the word, 5th element is the word itself, 6th,7th, 8th elements are block, line, word numbers respectively.


output 1


Extract the coordinates of the first object :




#Information of words in first object is stored in mywords

mywords = [w for w in words if fitz.Rect(w[:4]) in rec]

ann= make_text(mywords)


This function selects the words contained in the box, sort the words and return in form of a string : 

def make_text(words):

    line_dict = {} 

    words.sort(key=lambda w: w[0])

    for w in words:  

        y1 = round(w[3], 1)  

        word = w[4] 

        line = line_dict.get(y1, [])  


        line_dict[y1] = line  

    lines = list(line_dict.items())


    return "n".join([" ".join(line[1]) for line in lines])



output 2


page.first_annot() gives the first annot i.e. bounding box of the page.

.rect gives coordinates of a rectangle.

Now, we got the coordinates of the rectangle and all the words on the page. We then filter the words which are present in our bounding box and store them in mywords variable.

We have got all the words in the rectangle with their coordinates. However, these words are in random order. Since we need the text sequentially and that only makes sense, we used a function make_text() which first sorts the words from left to right and then from top to bottom. It returns the text in string format.

Hurrah! We have extracted data from one annot. Our next task is to extract data from all annots of the PDF which would be done in the same approach.

 Extracting each page of the document and all the annots/rectanges :

for pageno in range(0,len(doc)-1):

    page = doc[pageno]

    words = page.get_text("words")

    for annot in page.annots():

        if annot!=None:


            mywords = [w for w in words if fitz.Rect(w[:4]) in rec]

            ann= make_text(mywords)


 all_annots, a list is initialized to store the text of all annots in the pdf.

The function of the outer loop in the above code is to go through each page of PDF, while that of the inner loop is to go through all annots of the page and performing the task of adding texts to all_annots list as discussed earlier.

Printing all_annots provides us the text of all annots of the pdf which you can see below.



output 3 | Data extraction

Finally, we have extracted the texts from all the annots/ bounding boxes.

Its time to clean the data and bring it in an understandable form.

Data Cleaning and Data Processing

Splitting to form column name and its values :


for i in range(0,len(all_annots)):



Removing unnecessary symbols *,#,:


for i in range(0,len(cont)):


    for j in cont[i]:








Spliting into keys and values and removing spaces in the values which only contain digits :



for i in liss:



for i in range(0, len(values)):

    for j in range(0,len(values[i])):

        if values[i][j]>='A' and values[i][j]<='Z':


    if j==len(values[i])-1:
       values[i]=values[i].replace(' ','')


We split each string based on a new line (n) character to separate the column name from its values. By further cleaning unnecessary symbols like (*, #, 🙂 are removed. Spaces between digits are removed.

With the key-value pairs, we create a dictionary which is shown below:

 Converting to dictionary :






for local in dic:






    for i in range(0,len(local)-1):

        if local[i+1]>='0' and local[i+1]<='9':





    for i in li:

        if i[0] in lii:




    for i in li:

        if i[0]==k:


report['CRASH SEVERITY']=val_after[2]



output 3 | Data extraction


Lastly, dictionary is converted to dataframe with the help of pandas.

Converting to DataFrame  and exporting to CSV: 




Now, we can perform analysis on our structured data or export it to excel.

I hope that you have enjoyed reading this blog and it has given you an intuition of dealing with unstructured data.


Source of the featured image: Real Python

PyMuPDF documentation :

About the Author:

Hi! I am Ashish Choudhary. I am pursuing B.Tech from the JC Bose University of Science & Technology. Data Science is my passion and feels proud to write interesting blogs related to it. Feel free to contact me on Linkedin

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

About the Author

Our Top Authors

Download Analytics Vidhya App for the Latest blog/Article

3 thoughts on "Data Extraction from Unstructured PDFs"

D S BALA KRISHNA SARAN says: October 28, 2022 at 5:09 am
I have tried to execute /practice the same below code : import fitz import pandas as pd doc ='FlipKart_Invoice_2022_3.pdf') page1 = doc[0] words = page1.get_text("words") first_annots=[] rec=page1.first_annot.rect mywords = [w for w in words if fitz.Rect(w[:4]) in rec] ann= make_text(mywords) first_annots.append(ann) but, I am facing the below error : rec=page1.first_annot.rect AttributeError: 'NoneType' object has no attribute 'rect' Could you help me here ? Reply
bala krishna
bala krishna says: October 28, 2022 at 7:02 am
I am facing AttributeError: 'NoneType' object has no attribute 'rect'. Could you please help me here ? Reply
Manoj says: January 30, 2023 at 11:29 pm
Hi, great code by the way. However, when I tried, if I draw the bounding boxes using '' website the code is working but if I draw boxes using 'pypdf2' the boxes are not recognized by the code. Any sugessions! Reply

Leave a Reply Your email address will not be published. Required fields are marked *