Using rclone to sync one DO space to another

If you need to either duplicate or sync one space on Digital Ocean with another, there's a great tool called rclone which makes this very straightforward.

To get started, you first need to install it:

• Mac: brew install rclone
• Linux: apt get rclone

This will give you access to a CLI tool named rclone.

The next step is to configure it to access your DO spaces. To do this, you'll need to create an API key for your DO account.

Once you did that, create a config directory and file:

mkdir -p ~/.config/rclone
nano ~/.config/rclone/rclone.conf

Once that is done, add the following configuration to the file:

1[my-spaces]
2type = s3
3provider = DigitalOcean
4env_auth = false
5access_key_id = <access_key_id>
6secret_access_key = <secret_access_key>
7endpoint = ams3.digitaloceanspaces.com
8acl = private

This basically allows us to connect to the storage engine and use it as a source.

Now, let's you have a bucket called my-source-bucket which you want to duplicate as my-target-bucket, you can run the subcommand called sync:

rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket

This will create a full copy of my-source-bucket in my-target-bucket if it doesn't exist yet. If it does, it will only copy the files that are changed. It's explained in the documentation as follows:

Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects, see below). If you don't want to delete files from destination, use the copy command instead.

If you want to see what it does before actually doing the copy / sync, you can use the --dry-run flag:

rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket --dry-run

If you want progress reports, you can use the --progress flag:

rclone sync my-spaces:my-source-bucket my-spaces:my-target-bucket --progress

Note that this also allows you to sync from one provider to another (e.g. from AWS S3 to DO Spaces or the other way around).

As rclone supports many different cloud providers, you need to check the documentation if you want to use another storage engine.

PS: in case you're interested, rclone is written in Go.