The Good Tech Companies - Data Migration from Dell ECS to MinIO

Episode Date: September 18, 2024

This story was originally published on HackerNoon at: https://hackernoon.com/data-migration-from-dell-ecs-to-minio. Dell ECS clusters allow you to migrate your data to any S3 compatible store. Dell ECS calls this feature “Data Movement”, also called copy-to-cloud. Check more stories related to cloud at: https://hackernoon.com/c/cloud. You can also check exclusive content about #minio, #minio-blog, #dell-ecs, #data-movement, #data-migration, #s3-compatible-storage, #cloud-computing, #good-company, and more. This story was written by: @minio. Learn more about this writer by checking @minio's about page, and for more stories, please visit hackernoon.com. Dell ECS clusters allow you to migrate your data to any S3 compatible store. Dell ECS calls this feature “Data Movement”, also called copy-to-cloud. It's a feature introduced in ECS 3.8.0.1 that allows you to copy objects from Dell E CS to MinIO.

This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Data Migration from Dell ECS to Minio, by Minio. Dell ECS clusters allow you to migrate your data to any S3-compatible store. Dell ECS calls this feature, Data Movement, also called Copy to Cloud. It's a feature introduced in ECS 3.8.0, one that allows you to copy objects from Dell ECS-TO Minio which is rather popular with customers and prospects who are modernizing their storage stack to support their AI data infrastructure requirements. The data movement is built atop of the ECS Sync open source tool which provides the capability to copy the data in

parallel. In this overview we'll show you how to migrate data from Dell ECS to Minio by specifically focusing on the following configuring source and target buckets. Setting up data movement policies, monitoring and logging the migration to Minio, configure source and target buckets. Before we can start creating the policy to migrate the data, let's ensure the source and target buckets are configured for data movement. Configure Dell ECS source bucket internally. The data movement policy scans the source bucket to enumerate all objects for data movement using metadata MD search. For data movement to actually move data, you need to ensure MD search is enabled in the Dell ECS source bucket and include last modified as an indexed field.

Next let's configure Minio, the target bucket. Configure Minio target bucket in order for the data to be transferred to Minio. We need to create the following resource in Minio beforehand access and secret keys. Bucket name, IAM policy. When creating the bucket, be sure to enable bucket versioning, unless the target bucket in Minio is dedicated just for the data movement policy. Be sure to make note of the above details after creating them as their values are needed later. The IAM policy should allow the following APIs S3, list bucket. S3, get object. S3, put object. S3, delete object. Follow this guide to learn how to create IAM policies, access keys and buckets in Minio console. 1. Enter the username that was set while configuring the cluster. 2. Enter the password that was set while configuring the cluster. 3. Click, login. 4. Object browser. Buckets which have been created and data uploaded will be shown here. Backslash dot 5. Access Keys.

Oz IAM Style Access Keys 1. Create Access Key. Click here to create access and secret separate from the one we used to launch the cluster. Backslash dot 6. Buckets. List all the buckets that are available. 1. Create Bucket. If there are no buckets, go ahead and create a new one. Backslash dot. 7. Policies. IAM policies. 8. Identity. Create and connect various IDPs such as OpenID and LDAP. Backslash dot. 9. Monitoring. Monitor all aspects of the cluster and even send

it to Prometheus. Once the source and target buckets are configured, let's set up the data movement policy. Data movement policy. A data movement policy is a definition in Dell ECS that can be set either via UIOR API that defines which objects in a Dell ECS source bucket should be copied to Minio target bucket. The data movement policy scan jobs are automatically triggered but can be paused or resumed at any time. This is very similar to Tomineo's batch replication process. By default, the data movement policy migrates the data to Mineo in order of last modified time. We'll show you two different data movement scenarios to give you an idea of how this

could work, but the sky's the limit when it comes to how you would want Toto the migration. Data movement to Mineo in this configuration will add the necessary Minio bits and bobs for Dell ECSTO communicate with. The first step, once MD search is enabled, is to enable data mobility to as shown below. Once the data mobility is set to, we can go ahead and configure the policy. Endpoint. Set this to the Minio endpoint. Access and secret key. This was created in the Minio console and saved in a previous step. Bucket name. Minio target bucket name. Logging bucket. This is the bucket in Dell ECS that logs any errors during the migration. There are other settings, I've not gone through all the settings,

just the most important ones. Please note that if data is deleted from the Dell ECS source bucket, it won't be deleted from Minio target bucket after the migration has been completed. Data movement with DREMIOTO Minio Now let's take a look at how the migration would look like using an application that uses Dremio. There are a few steps that take place during this migration. 1. A customer-facing application writes to a Dell ECS bucket. 2. ECS copies to a staging bucket within Minio configured using data movement policy. 3. Data is copied over to the staging bucket. 4. The Minio staging

bucket will use event notification to send a message to RabbitMQ, which Dremio will be subscribed to. 5. Dremio reads the message and ingests the data from the Minio staging bucket to the Dremio bucket in Minio. 6. Once the data is ingested, you can clean up the Minio staging bucket using a lifecycle policy. These are just two examples but you can use this methodology to migrate from any application using Dell ECS to Minio. Data movement monitoring and logging. During the data migration process, it's important to keep an eye on the overall migration process as it moves data to Minio. The Dell ECS GUI provides overview dashboards with advanced monitoring that shows total objects copied, total bytes copied,

watermark lag, total errors, objects copied, bytes copies, among others. You can further drill down to show source, target specific information such as the object count and bucket size over a selected period during the migration process. In the initial diagram at the start of this blog we've shown a log bucket we have on the ECS side, this is where all the operations from the data movement are logged. This is very helpful to debug any issues during the migration process especially when it takes a very long time due to hardware and physical constraints. Here is an example of how the logs look like, why migrate to Minio. Minio is a single go binary that

can be launched in many different types of cloud and on-prem environments. It's very lightweight but is also feature-packed with things like replication and encryption, and it provides integrations with various applications. We've benchmarked it at 325 GB per second, 349 GB per second on Gets and 165 GB per second, 177GB per second, on PUTs with just 32 nodes of off-the-shelf NVMe SSDs, and is used to build datalakes, lake houses and analytics and AI, ML workloads. Not only that but out-of-the-box Minio also includes encryption. Minio supports both encryption at rest and in transit. This ensures that data is encrypted in all facets of the transaction from the moment the call is made till the object is placed in the bucket. Backslash dot. Bitrot protection. There are several reasons data can be corrupted on physical disks.

It could be due to voltage spikes, bugs in firmware, misdirected reads and writes among other things. Minio ensures that these are captured and fixed on the fly to ensure data integrity. Backslash dot, erasure coding. Rather than ensure redundancy of data using RAID which adds additional overhead on performance, Minio uses this data redundancy and availability feature to reconstruct objects on the fly without any additional hardware or software. Backslash dot, secure access ACLs and PBAC. Supports IAMS3 style policies with built-in IDP. See Minio best practices, security and access control for more information. Backslash. Tiering. For data that doesn't get accessed as often you can siphon off

data to another cold storage running Minio so you can optimize the latest data on your best hardware without the unused data taking space. Backslash dot. Object locking and retention. Minio supports object locking, retention, which enforces write-once-and-ready many operations for duration-based and indefinite legal hold. This allows for key data retention compliance and meets SEC 17A4F, FINRA 4511C, and CFTC 1.31C, D requirements. Backslash dot. Not to mention, good software is nothing without good support. Minio provides one of the best support out there using our subnet portal. We have engineers who work on the MinioCore codebase answer questions directly in a Slack-style interactive and collaborative medium. When you speak with an engineer rather than endlessly escalate your issue to the next-level

engineer the folks who you speak with are capable of resolving any issue you come across. We've even had customers come back to us after going with a competitor storage platform because of the lack of proper support. No matter if you design your storage with all the features under the sun, if you do not promptly support your customer then it is of no use. For us supporting our customers and making them successful is our priority number one. If you have any questions on how to migrate your data from Dell ECS to MinIO Migration be sure to reach out to us on Slack. Thank you for listening to this HackerNoon story, read by Artificial Intelligence. Visit HackerNoon.com to read, write, learn and publish.