Data Migrations During Scaling & Repairs

Last Updated October 2014

Whenever you scale data services on Pagoda Box, new instances are provisioned in parallel to your currently running instances and data is migrated to the new instances. To preserve data integrity and keep migration times to a minimum, data migrations are done offline for those services which do not natively support live migrations. This doc provides information about how these migrations happen and what types of migrations are supported per service. After reading it, you should be familiar with:

  • Why and how data migrations happen during scaling/repairing
  • What services can and can't use live migrations

How Scaling & Data Migrations Work

Scaling an apps infrastructure while it's handling traffic can be scary, especially when the service you're scaling houses important data. The scaling process for data services (databases, caches, network storage) on Pagoda Box is designed to minimize downtime while ensuring data integrity.

When scaling a service, new instances are provisioned in parallel to currently running instances. The live instances continue handling requests as the new instances are provisioned. Once fully provisioned, data is migrated from the "old" instances to the new.

To preserve data integrity, services are taken temporarily offline while completing this migration. There are a few service types that natively support live migrations, but the majority do not. The table below has more information.

Offline Migrations

Although offline migrations do cause your service to be temporarily unavailable during scaling transactions, they are designed to minimize downtime as much as possible while ensuring data integrity. Running instances are not taken offline until the new instances are fully provisioned and confirmed functional. Once offline, data is migrated to the new instances as quickly as possible, the new instances begin handling requests, and your service is back in business.

It should also be noted that offline migrations are incredibly fast. When the service is disabled, we simply copy the data from the old instances to the new. Obviously, the amount of time it takes to transfer data depends on the size of the dataset, but with our 1Gbit network backbone, file transfers don't take long.

Live Migration Compatibility

Click for details

  • MySQL
  • Live MySQL migrations are not supported on Pagoda Box. While live migrations are possible, the replication/sync process required is prone to data loss and we have opted out of providing this as an option. Offline migrations are much safer and much faster.
  • MariaDB
  • Live MariaDB (MySQL) migrations are not supported on Pagoda Box. While live migrations are possible, the replication/sync process required is prone to data loss and we have opted out of providing this as an option. Offline migrations are much safer and much faster.
  • Percona
  • Live Percona (MySQL) migrations are not supported on Pagoda Box. While live migrations are possible, the replication/sync process required is prone to data loss and we have opted out of providing this as an option. Offline migrations are much safer and much faster.
  • PostgreSQL
  • Live PostgreSQL migrations are not supported on Pagoda Box. While live migrations are possible, the replication/sync process required is prone to data loss and we have opted out of providing this as an option. Offline migrations are much safer and much faster.
  • MongoDB
  • MongoDB does not support live migrations.
  • Riak (Coming Soon)
  • Riak's distributed architecture natively supports live data migrations.
  • Redis
  • Live Redis migrations are not supported on Pagoda Box. Redis is primarily architected for a single instance without on-the-fly migrations. While live migrations are possible, the process required is prone to data loss and we have opted out of providing this as an option. Offline migrations are much safer and much faster.
  • Memcached
  • Memcached is an in-memory datastore. All data is flushed when scaling or rebuilding. Data is never migrated.
  • NFS (Network Storage)
  • NFS does not support live migrations.

If you have any questions, suggestions, or corrections, let us know.