Simplify Data Migration with AWS DMS
This article was published as a part of the Data Science Blogathon.
Introduction
DMS is a service that makes it easy to migrate on-premise databases into the cloud with minimal or no downtime. It can even monitor the changes in the original database and apply them to the new database. DMS supports all databases like MySQL, PostgreSQL Oracle, SQL Server, Microsoft SQL Server, and Amazon Aurora. The advantage is that the original and new databases don’t have to match. Hence it is easy to alter one type of database into another. DMS takes care of all the tedious tasks of exporting the data from the old database and importing it into the new database in lesser time. It supports data migration tasks like database development, consolidation, and testing. While migrating, the source database will be fully operational, minimizing downtime to applications that rely on the database. This article covers an overview of DMS, how it migrates data, its process, and why one should consider DMS for migration.
What is AWS DMS?
AWS Database Migration Service (AWS DMS) helps migrate databases to AWS quickly. It allows migrations of several databases like relational, nonrelational databases, and other types of commercial and open-source databases. It continuously monitors the source and target databases besides network connectivity. If any interruption arises that halts the replication task, it is automatically rectified by DMS. It can consolidate disparate databases into a multi-petabyte data warehouse by streaming data to AWS Redshift and Amazon S3.
Why Consider AWS DMS for Data Migration?
-
Highly resilient:
DMS can automate the backup snapshots and restore them very quickly. If the primary replication server fails, the backup replication server will take over; hence there is no service interruption.
- Minimal downtime:
While migrating databases through DMS, the source database will be fully operational, minimizing downtime to applications that rely on the database. So users can change databases whenever required without a need to shut down.
- Low cost :
DMS provides a free migration service for moving databases to Redshift, Aurora, DynamoDB, or DocumentDB. Users must pay for migrating to other databases depending on the log storing volume.
- Comprehensive Database Support:
DMS supports homogeneous migrations such as Oracle to Oracle and heterogeneous migrations Oracle to Aurora platforms using the SCT tool. It supports open-source and commercial databases helping in convenient database migration.
- Failover support :
DMS are deployed in a multi-AZ environment which helps in failover support and easy availability. In a Multi-AZ deployment, DMS automatically maintains and provisions a synchronous standby replica of the replication instance in a different Availability Zone.
AWS DMS Process- How it Works
- Connect endpoints:
First, connect DMS to both source and target endpoints. The primary requirement for DMS is that one user’s endpoints must be hosted on AWS service. For performing the migration, endpoint credentials should successfully connect to the databases with the essential privileges.
- Task scheduling:
Now schedule a task that runs on the server to migrate the data. DMS reads the data from the source database and prepares it for the target database. Full-load migration occurs when source data is transferred to the target database.
- Sync databases:
If any changes are made at the source database, it is cached on the replication server. Soon after the full-load migration is completed, these cached changes are applied to the target database to keep both databases in sync.
Types of Supported Databases
Source: https://www.logicata.com/blog/aws-database-migration-service
Using AWS DMS with other AWS services. It is possible to use DMS with other AWS services, which include the following:
- Amazon EC2 and RDS:
Users can use Amazon EC2 or RDS DB instances as a target for data migration. EC2 is the virtual machine representing a physical server that provides secure and scalable computing capacity. RDS refers to an SQL database service allowing access to files and databases anywhere.
- AWS Schema Conversion Tool:
AWS Schema Conversion Tool or SCT automatically converts source schema to a compatible format for your target database. With SCT, users can migrate database schema, including stored procedures, views, functions, etc., to compatible target databases.
- Amazon S3:
For storage purposes or while migrating large amounts of data, Amazon S3 can be used in the intermediate step. Users can set up a Database Migration Task, which transfers specific data to the S3 bucket. - AWS CloudFormation :
AWS CloudFormation can set up AWS resources for infrastructure deployment or management. Using CloudFormation templates, users can create, delete, and modify the entire stack (collection of AWS resources) as a single unit or stacks across multiple AWS accounts and regions without managing resources individually.
Components of AWS Database Migration Service
There are mainly three DMS components which include:
- Replication instance
- Source & Target Endpoints
- Replication tasks
Replication instance:
The Amazon EC2 (Elastic Compute Cloud) managed instance hosts one or more replication jobs. It deals with compute-intensive workloads, heterogeneous migrations, and replications like migrating from Oracle to PostgreSQL. Memory-optimized replication instances like R4/R5 help in midsize in-memory databases, real-time big data analytics, and enterprise applications.
Source & Target Endpoints:
Endpoints link both source and target databases helping in the transport of data. All the endpoints need information like Server name, Port number, Endpoint type, Engine type, Credentials, etc. Before using endpoints for a database migration task, users should test the endpoint connection.
Replication tasks:
It is the final step before beginning a migration which transports data from the source endpoint to the destination endpoint. It denotes what data is being transferred between the target & source endpoints and when the migration has to be performed.
Use cases
Homogeneous database migration:
Source: https://www.heptabit.com/blog/database-migration-to-aws-with-no-downtime
In this migration, both the source and the target database engines should match and be interoperable. Examples are Microsoft SQL Server to Amazon RDS for SQL Server, Oracle to Amazon RDS for Oracle, MySQL to Amazon RDS for MySQL, MySQL to Amazon Aurora, etc.
Heterogeneous database migration:
In this migration, the database engines of the source and the target do not complement each other. Here a schema and code transformation step is needed to address the differences in database code, data types, and schema structure between the source and target databases. Examples are Oracle to PostgreSQL, Oracle to Amazon Aurora, Microsoft SQL Server to MySQL migrations, etc.
Building a migration plan:
DMS allows users to assess and keep track of the progress of their database migrations. DMS dashboard displays migration status regardless of which regions users move their databases. It can detect and resolve troubles arising out of migration.
Continuous Data Replication:
DMS helps in the replication of data continuously with high availability. Through continuous data replication, DMS helps synchronize development and testing environment locations and Disaster Recovery instance locations.
Conclusion
Ultimately, we can conclude that AWS DMS is a great tool that makes it easier to migrate database workloads to AWS along with continuous data replication. It’s easy to set up, reliable, supports a variety of databases, and performs constant network connectivity monitoring of target databases.
Key Takeaways:
- AWS DMS helps migrate databases to AWS quickly and monitors source and target databases.
- Deployed in a multi-AZ environment for easy availability.
- AWS Schema Conversion Tool or AWS SCT can be used with DMS to automatically convert source schema to a compatible format for your target database.
- Mainly three components – Replication instance, Source & Target Endpoints, and Replication tasks.
- Support both homogeneous and heterogeneous database migrations.
- Endpoints refer to the two data stores, i.e., the source and the target.
- DMS helps in the replication of data continuously and synchronizes development and testing environment locations.
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.
