How to Implement Data Engineering in Practice?

Andre Vianna 01 Dec, 2021 • 6 min read
This article was published as a part of the Data Science Blogathon.
Data Engineering
Image Source: GitHub

Table of Contents

  1. What is Data Engineering?
  2. Components of Data Engineering
  3. Object Storage
  4. Object Storage MinIO
  5. Install Object Storage MinIO
  6. Data Lake with Buckets
  7. Demo Data Lake Management
  8. Conclusion
  9. References

What is Data Engineering?

Initially, we have the definition of Software Engineering, as the branch of engineering capable of specifying, developing, executing, and maintaining programs, systems, and applications. That solves corporate business problems, helping companies to increase their competitiveness.
Within Software Engineering we have several specializations, such as social networks, ERP, Project Management, Mobile Apps, Networking Management, and Data Science, among others. We can understand Data Engineering as a subset of Data Science, which receives data from Big Data and prepares them to be used in a Machine Learning Model.

Data Engineering 


Data Engineering | Software Engineering

Image Source: GitHub

Data Engineering Components


Data Engineering Components

Image Source:GitHub

Data Engineering is formed by a wide range of software, which is executed from a flow, created by the Data Scientist.

  • Data Discovery
  • Observability
  • Governance
  • MLOps
  • Workflow
  • Notebooks
  • Formats
  • Data Lifecycle
  • Virtualization
  • Compute
  • Object Storage
  • Meta stores
  • Orchestration
  • Analytics Engine
  • Ingest – Data
  • Ingest – Application

In this article, we will implement Data Engineering, applying an Object Storage component, with data storage functionalities after extracting an external Big Data, later after data manipulation and processing, the data is stored in this Object component. Storage is already in a structured form to be consumed by a Machine Learning model.

Components of Data Engineering

Data Engineering can be classified by several components, which can be executed in isolation, or integrated into a data flow pipeline. These components can be run on cloud infrastructure or on-premises servers.

For the identification of new data, we have the Data Discovery component, for the capture and processing of logs we have the Observability component, for data management we have the governance components.

Data Engineering and its' practices

Image Source: GitHub

We can understand MLOps as an extension of DevOps applied to Data Science applications, where we can visualize two flow layers, the first the Data layer and the second the Machine Learning layer.

In the Data Layer, the Data Scientist executes the deployment process, through a continuous and cyclic pipeline, which starts with the Data Collection, which can be understood by the Classic ETL Extraction process. Then, we have the Data Ingestion job, which loads the data and indexes the pipeline, the Data Analysis job curates and selects the data, Data Labeling adds a tag to the data, to facilitate the future classification process, the job Data Validation checks the usability of the data, to finally be prepared for the Machine Learning layer.

In the figure below, we can see the flow of the Machine Learning layer, which receives the data prepared from the previous layer and starts the machine learning training. In the validation of the machine learning model, the model with the highest performance is adopted for use in the dataset. In Machine Learning validation, the validation is done with test data, after this last verification the Machine Learning model is exported to a cloud infrastructure, such as Google Cloud Platform, Microsoft Azure Machine Learning, or AWS Machine Learning Sage Marker.

Data Engineering | Machine Learning Operations

Image Source: GitHub


As a result of MLOps, after executing the Data layer and Machine Learning layer flows, the Data Scientist offers the user, business specialist, a customizable dashboard, with the insights and predictions generated by the MLOps architecture.

 Machine Learning Operations 2

Image Source: GitHub

Object Storage in Data Engineering

Object Storage is a fundamental component in Data Engineering, with unlimited data storage capacity, guaranteed to Big Data, flexible and highly scalable data storage.

Currently, we have AWS S3 as the main Object Storage available in the Cloud, having a large number of companies, which host data from various applications.

We have services with equivalent functionality in Oracle Cloud, Google Cloud Platform, Microsoft Azure, and IBM Cloud.

In this article, we will detail the open-source Minio software, which can be run on an on-premise server, or a virtual cloud server like AWS EC2.


Below we have the main features of the object storage software:

  • Directed to Big Data
  • Complete Data Lifecycle
  • Data Durability
  • Data Availability
  • Unlimited Scalability
  • High Transfer Fee
  • Integrated with Identity Management
  • Autocorrect
  • data protection
  • Server-Side Encryption
  • elastic sizing
  • Flexible Connectivity Options
  • Easy Integration
  • Fault Tolerance

Object Storage – MinIO for Data Engineering

Object Storage MinIO is an Open Source Software, a good alternative to AWS S3, it can be easily installed on a Linux server, and customized similarly to AWS S3.

We have a complete dashboard where we create Buckets, grant access authorization per user, create groups, define IAM policies, view logs, and monitor access. These features are available in the free version.

Object Storage | Minio

Image Source: Minio

Install Object Storage – MinIO

  • Installing and Configuring Data Lake with Minio Server
  • Inside the DataScience directory create the data lake directory.
  • If you are using Windows open the Powershell terminal and run the command:
docker run -d -p 9000:9000 -p 9001:9001 -v "$PWD/datalake:/data" minio/minio server /data --console-address ":9001" --name minio
[username]: minioadmin
[password]: minioadmin

Data Lake in Data Engineering

We implement Data Lake, through the Object Storage MinIO installed, in the ingest data stream from data extraction from an external source, the data is stored in the first bucket named Curated Bucket, later we have the second Bucket named Processing Bucket, and ending we have the third Bucket called Landing Bucket, in which the Machine Learning Model will access the data sets, for training and testing the model.

 Data Lake

Image Source: GitHub

Bucket Management

Data Engineering | Bucket Management

Image Source: GitHub

Below is the Data Lake repository on GitHub:

Curated Bucket #1

 Curated Bucket #1

Image Source: GitHub

Below is the Curated Bucket repository on GitHub:

Processing Bucket #2

Data Engineering | Bucket Processing

Image Source: GitHub

Below is the Processing Bucket repository on GitHub:

Landing Bucket #3

 Landing Bucket

Below is the Landing Bucket repository on GitHub:

In Landing Bucket, we have the output of the Machine Learning model, after training, testing, and validation of the model.

Follow the output of the model:


Demo Data Lake Management

Demo Data Lake Management

A brief demonstration of our Data Lake created, using the MinIO Object Storage resources, with three Buckets, the last being Landing Bucket will be accessed by the Machine Learning model.


This article aimed to show Data Science professionals how to use Data Engineering components in an agile way, to generate Machine Learning models with good performance and accuracy.

We show several software components, which are part of Data Engineering, with their main functionalities. We detail the Object Storage technically, we implement a Data Lake, with three buckets, which is integrated into a Machine Learning model.

In this way, I believe that it is good, to use the Object Storage MinIO, in the creation of a Data Lake for companies that are starting the journey in Machine Learning.



Author Reference:

  1. Github
  2. Twitter
  3. Medium

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Andre Vianna 01 Dec 2021

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers


Isac 01 Dec, 2021

Good Article ! Congrats !

  • [tta_listen_btn class="listen"]