Object Migration using ECS-Sync

As mentioned in my previous post covering how to prevent objects from being accidentally deleted, this one will cover object migrations. 

Reasons for object migrations

There are a couple of reasons for object migrations. Obvious ones like if you need to change server-side encryption or if your data needs to be put into another replication group. Less obvious reasons like migration from an HDD based ECS to an all flash ECS is the reason we came across the request to migrate existing content again using ECS-Sync.

Features like Data Mobility that require a metadata search index on the Last Modified date which has not been configured when creating the original bucket may also lead to the request to copy data to a new bucket.

Why ECS-Sync

We have found ECS-Sync to be a reliable and free tool to migrate not only S3 and CAS objects but also from or to other storage devices. Check out the storage plugin documentation for more details. Everything can be found here. 

Using ECS-Sync, we have migrated several hundred pools from Centera to ECS which resulted in Billions of objects and and many terabytes transferred. What we have come to like is, that if the hardware resources are properly sized, ECS-Sync does a great job to copy objects. All without having to pay a license for terabytes migrated or anything else. Just the work needs to be done and we want to help you and avoid the pitfalls.

Overview of the Install Guide available

It all starts with creating a virtual machine based on a minimal CentOS 7 iso image. Be sure to increase the minimum CPU and Memory sizes as well as to add an large enough disk. Once the OS is installed go to GitHub and download the lastest version of ECS-Sync. Make sure Java uses all the memory by updating the configuration file followed by a restart of the ecs-sync service.

If you migrate CAS objects, install the Centera SDK and JCASScript. Using JCASScript to create a cliplist may be faster than the ECS-Sync UI query and the cliplist can easily be referenced by the migration job.

Once done, update all components and reboot.

Then log in to the UI and start creating migration jobs. Be sure to specify a table name to prevent accidental deletion of the tables. Also choose the option to store the extended fields like source and target MD5 values to prove the migration did work properly.

Migration documentation

ECS-Sync creates a table in the MariaDB database for each job. It documents source and target IDs and MD5 values. All timestamps are logged and the table can be queried to generate the required migration documents from the UI or the database directly.

ECS-Sync Installation Guide

Check out the more detailed ECS-Sync Installation guide. 

Upcoming Blog Posts

Upcoming topic will be to describe how we implement ECS S3 Identity and Access Management with its policies. We focus on namespaces with applications that need to be limited which buckets they may access while still being able to issue the list all my buckets command without receiving an error. Capabilities like being able to create buckets or enabling object lock may lead to policies not being followed and these are discussed as well. 

Looking for other ECS related information?

Do not hesitate to reach out to Backup ONE AG using the contact form in our website. We’ll be happy to assist. Ideas for additional posts are welcome 😊.