KAVITA MALI — July 23, 2021
Beginner Data Exploration Libraries Pandas Programming Python

This article was published as a part of the Data Science Blogathon

Introduction:

 Pandas is a library generally used for data manipulation and data analysis. Pandas is used to handle tabular data. In particular, it provides the data structure as well as functionality for managing numerical tables and time series. The name ‘Pandas’ is derived from the term “panel data”, which means an econometrics term for data sets. One shall use Pandas heavily for data manipulation, visualization, building machine learning models, etc. There are two sorts of data structures in Pandas –

  1.  Series
  2. Dataframes.

Installing Pandas:

Installing pandas using the following command on the conda command prompt.

(base)$ source virtualenv_name/bin/activate

(virtualenv_name) $ pip install pandas

 

Getting Started with Pandas:

Once Pandas is successfully installed in your system, you can import Pandas simply by,

import pandas as pd

where pd is an alias used for Pandas so that the Pandas package can be referred to as pd instead of pandas.

Creating Pandas Series:

# A pandas series 
pd.Series( [ 12, 13, 1, 3, 5 ] )
# A pandas series with explicit indexing 
pd.Series( [ 2,  4,  6],  index = [ 'ad',  'bd',  'cd' ] )

 

Pandas series

Creating Pandas Dataframe:

# A pandas dataframe 
pd.DataFrame({'name': ['Akash', 'Shivam', 'Neil', 'Manish'], 'age': [21, 23, 24, 28],'occupation': ['data scientist',  'doctor',  'data analyst', 'engineer']})

 

pandas dataframe

Sorting in Pandas:

The below code will set the specified column as an index of the dataframe.

# setting a  column as index 
df.set_index('col_name' ,  inplace = True)

The below code will sort  the dataframe

# sorting the dataframe by index
df.sort_index(axis = 0,  ascending = False)

The below code will sort the dataframe based on column values specified.

 # sorting the data frame by a column
df.sort_values(by = 'col_name', ascending = False)

Inspection of Dataframe: 

The below code will return the total number of rows and columns as a tuple.

#Checking the shape of a dataframe
df.shape

dataframe.head(n): This function will return the first n rows of the data frame. By default, it gives the first 5 rows if n is not given explicitly.

#Viewing top 5 rows of the data frame
df.head() 
#One can view any number of top rows by
df.head(3)
df.head(15)

dataframe.tail(n): This function will return the last n rows of the data frame. By default, it gives the last 5 rows if n is not given explicitly.

#viewing bottom 5 rows
df.tail()
#One can view any number of bottom number
df.tail(10)

 

dataframe.info(): This method prints all the information about a DataFrame which includes the index dtype and columns, non-null values for each column, and memory usage for the overall data frame.

Let’s check the info for the dataframe that we created earlier

# metadata summary 
df.info()
data info

dataframe.describe() includes Descriptive statistics includes summary like central tendency, min, max, and shape of a dataset’s distribution, excluding null values for the numerical column. And summary like count, frequency, etc for categorical columns.

Let’s check the output of this function for the data frame we created earlier.

# statistical summary 
df.describe()
pandas df.describe

 

#Returns the memory consumed by columns 
df.memory_usage()
#Returns the columns of the data frame as a array
df.columns
#Returns the values as an array 
data.values

Extracting columns in Pandas:

# Extract a specific column as a series
df[ 'col_name' ] 
df.col_name
extracting column in pandas

 

#Extracting multiple columns at a time 
df[ [ 'col_1',  'col_2',  'col_3' ] ]
getting multiple columns in pandas

Let’s define a data frame so that we can implement the following codes

df = pd.DataFrame( {
    'Name' : ['Neil', 'Aishwarya', 'Rahul', 'Shivani','Namrata','Pawan', 'Karan','Krish'],
    'Age' : [23, 21, 22, 21, 20, 22, 22, 23],
    'University' : ['BHU', 'JNU', 'DU', 'BHU', 'IIT', 'MU', 'JNU', 'MU'],
    'Specialization' : ['Mathematics' , 'Biology', 'Physics' , 'Chemistry', 'Accountancy', 'Mathematics', 'Bio-Chemistry', 'Astrophysics']
creating a pandas dataframe

Index-based slicing:

# Extracting a particular column and row from dataframe. dataframe.iloc[m,n] where m is row index and n is column index 
 df.iloc[3, 5]   # single element 4throw and 6th column

Output:

index based slicing

#Extracting a specific row with all columns 
df.iloc[5, :] # single row and all columns

Output:

 using iloc function pandas

 

#Extracting multiple rows at a time by providing list of indices
df.iloc[ [2, 5, 7] ]
df.iloc[ [2, 5, 7] , : ] 
df.iloc[ [2, 5, 7] ,  ]

Output:

extracting multiple rows

 

#Extracting particular range of rows
df.iloc[2 : 8] 
df.iloc[2 : 8 ,  : ]
df.iloc[2 : 8 , ]

Output:

extracting range of rows

#Extracting  a particular column by index number
df.iloc[ : ,  2 ]

Output:

extracting a column pandas

#Extracting particular range of columns
df.iloc[ : , 1 : 3 ]

Output:

extracting range of columns

 

#Extracting rows and columns of a particular range
df.iloc[ 3 : 6 , 1 : 3 ]

Output:

extracting rows and columns

 

Label based slicing:

#Extracting rows and columns based on column or row labels
df.loc[2, 'col_name'] # single element row label 2 and column sales

output:

label based slicing

 

df.loc[5] # single row with label 5 
df.loc[5, :] # single row with label 5 and all columns

Output:

label based row slicing pandas

 

#Extracting multiple rows provided the list of labels
df.loc[ [2, 5, 7] ] # rows with label 5, 7, 8 
# Extracting multiple rows with label 3, 7, 8 and all columns
df.loc[ [2, 5, 7],  : ]

Output:

extracting multiple rows with given labels pandas

 

#Extracting the rows of a particular range of labels
df.loc[ 2 : 8 ] # multiple rows using a range of labels 
df.loc[ 2 : 8 ,  ]
# multiple rows using a range of labels and all columns
df.loc[ 2 : 8 ,  : ]

Output:

multiple rows with range of labels

 

# Extracting multiple rows using labels and columns
df.loc[ [1,  2] ,  'col_1' : 'col_2']

Output:

column extractions pandas

Condition-based slicing:

Here we can extract specific columns or rows satisfying a particular condition.

#Extracting all the values in column whose value is greater than 22
df.loc[ df.Age>  22] 
df[ df.Age> 22]

Output:

condition based slicing

 

#Extracting all the rows for which age in column are equal to 22
df.loc[ (df.Age== 22) ,  : ]

Output:

select all rows satisfying a condition

 

 #Extracting all the rows for which age in column are not equal to 22
df.loc[(df.Age != 22), :]

Output:

Another example pandas

Multiple conditioning 

#Extracting  all rows where 21< Age< 23 and University = MU
df.loc[ ( df.Age > 21) & ( df.Age < 23) & ( df.University == 'MU' ) ,  : ]

Output:

multiple conditions pandas

 

#Extract all rows where 22 > Age or University = 'MU'
df.loc[(df.Age < 22) | (df.University = 'MU'), :]

Output:

example multiple condition

 

#Extracting the all the rows where column having the values in list
df.loc[ df[ 'University' ].isin( ['MU', 'IIT' ) ,  : ]

Output:

columns in value list pandas

 

# Extract all rows where 21< age and University = ‘MU’ and only selected columns
df.loc[ ( df.Age > 21 ) & ( df.University == ‘MU’ ), [ ‘Name’, ‘Specialization’] ]

Output:

and conditions pandas

Various Operations:

This will merge the DataFrame or pandas Series objects with a database-style join.

Let’s take an example to understand this. First, we will create data frames and then join them with the inner intersection. There are more ways to join the data frames you can check that on official documentation.

data_1 = pd.DataFrame(
       {
            "key_1": ["K0", "K0", "K1", "K2"],
            "key_2": ["K0", "K1", "K0", "K1"],
            "A": ["A0", "A1", "A2", "A3"],
            "B": ["B0", "B1", "B2", "B3"],
       }
    )
data_2 = pd.DataFrame(
       {
            "key_1": ["K0", "K1", "K1", "K2"],
            "key_2": ["K0", "K0", "K0", "K0"],
            "C": ["C0", "C1", "C2", "C3"],
            "D": ["D0", "D1", "D2", "D3"],
        }
    )
#Merging the two dataframesby inner join on particular columns
pd.merge( data_1,  data_2,  how = 'inner' ,  on = ['col_1',  'col_2' ] )

Output:

merging pandas dataframe

Concatenating pandas objects along the specified axis. you can get additional information from the link given above.

# concatenating dataframes one on top of the other
 pd.concat( [ data_1,  data_2], axis = 0)

Output:

concatinating dataframes

# concatenate two dataframes side by side
 pd.concat( [ data_1, data_2] , axis = 1 )

Output:

concatenate pandas dataframes side by side

 

  • Creating new columns
# adding a new column
df['new_col'] = any_value

Output:

adding a new column

 

# add a new column by using existing columns
df[ 'new_col' ] = df[ 'col_1' ] / df[ 'col_2' ]

Output:

add new column using existing columns

 

  • Grouping the data
# grouping the data by specific columns
df.groupby( [ 'col_1' ,  'col_2' ] ) 
#Grouping the data by specific columns and aggregating 
df.groupby( [ 'col_1' ,  'col_2' ] )['col_3'].sum() 
df.groupby( [ 'col_1' ,  'col_2' ] )['col_3'].mean()

Output:

groupby

 

  • Applying a function on a column that can be inbuilt also
df[ 'col_1' ].apply( lambda x: function(x) ) 
df[ 'col_1' ].apply( lambda x: np.log(x) ) #Invuilt function

Output:

apply function

Pivot table:

Creating a spread-sheet like pivot table as a Dataframe

df.pivot_table( values = 'aggregation_col', index = 'group_by_row', columns = 'group_by_col', aggfunc = 'mean')

Output:

pivot table

EndNotes

There are many more pandas operations that one can try or apply. I hope u enjoyed the article and got an idea of pandas.

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

About the Author

Our Top Authors

  • Analytics Vidhya
  • Guest Blog
  • Tavish Srivastava
  • Aishwarya Singh
  • Aniruddha Bhandari
  • Abhishek Sharma
  • Aarshay Jain

Download Analytics Vidhya App for the Latest blog/Article

Leave a Reply Your email address will not be published. Required fields are marked *