Cloudflare recently announced the open beta of Sippy, an incremental data migration service that copies data from Amazon S3 to Cloudflare R2 only the first time the data is requested. Sippy is designed to minimize migration-specific egress fees by leveraging requests within existing application flows while simultaneously copying objects to R2.
Earlier this year, Cloudflare announced Super Slurper, a data migration tool to copy large amounts of data from other object storage providers to Cloudflare R2. The new option targets scenarios where developers want to move only accessed data. Phillip Jones, senior product manager at Cloudflare, and Vlad Krasnov, principal system engineer at Cloudflare, explain the benefits of Sippy over the existing Super Slurper:
While Super Slurper is perfect for cases where you want to move all of your data to R2 at once, there are scenarios where you may want to migrate your data incrementally over time. Maybe you want to avoid the one-time upfront AWS data transfer bill? Or perhaps you have legacy data that may never be accessed, and you only want to migrate what’s required?
The new option acts as a middle layer between the client and the source object storage: when an object is requested from Workers, S3 API, or a public bucket, it is served from Cloudflare R2 if it is found. If not, the object will be returned from the source S3 bucket and copied simultaneously to R2. All other operations on R2, including put and delete, continue to work. Jones and Krasnov add:
On top of addressing vendor lock-in, Sippy makes stressful, time-consuming migrations a thing of the past. All you need to do is replace the S3 endpoint in your application or attach your domain to your new R2 bucket and data will start getting copied over.
The documentation shows how to enable Sippy for an existing R2 bucket with cURL:
curl -X PUT https://api.cloudflare.com/client/v4/accounts/{account_id}/r2/buckets/{bucket_name}/sippy \
--header "Authorization: Bearer <API_TOKEN>" \
--data '{"provider": "AWS", "bucket": "<AWS_BUCKET_NAME>", "zone": "<AWS_REGION>","key_id": "<AWS_ACCESS_KEY_ID>", "access_key":"<AWS_SECRET_ACCESS_KEY>", "r2_key_id": "<R2_ACCESS_KEY_ID>", "r2_access_key": "<R2_SECRET_ACCESS_KEY>"}'
Cloudflare plans on making incremental migration configurable from the dashboard and providing analytics showing the progress of the migration and how much has been saved on egress fees for objects that have been copied over.
Sippy is currently in open beta and supports only Amazon S3 as a migration source. During the Birthday Week, the connectivity company made other announcements that target migrations from what Cloudflare calls "legacy cloud providers": how to avoid paying Amazon’s IPv4 tax, the general availability of Turnstile, a reCAPTCHA alternative, and Hyperdrive, a service that accelerates SQL queries reducing the load and cost on cloud-managed databases.