Using Butler - Migrating from Pagoda Box v1 to v2

Last Updated December 2014

Pagoda Box V2 is here and ready for your apps. To help you easily migrate your apps from v1 to v2, we’ve built a migration tool, which we lovingly call "Butler", that will do most of the heavy lifting for you. This doc walks through the details of the migration process and things you should know and do when using Butler. After reading it, you should be familiar with:

  • The v2 migration process
  • How to use Butler, the v2 migration tool
  • Moving over to Pagoda Box v2 and never looking back

Butler Video Overview

Prepare Your App for the Migration

Before you start the migration process, there are some steps you can take to make sure the move from v1 to v2 goes as smoothly as possible. For more information, read through the following docs:

Things to Know When Moving to Pagoda Box v2
Prepping Your App for a Smooth v2 Migration

What Does Butler Do?

Pagoda Box v2 is a completely new infrastructure built from the ground up. It uses an entirely different operating system, new virtualization technology (both server and network), and a whole slough of other upgrades and features. Because of this, a migration process is required. Butler, the v2 migration tool, will do all of the following for you:

  • Analyze your app and provide a comparable v2 build

  • Migrate your SSH key(s) to your v2 user account

  • Create and deploy your app on v2

  • Migrate data & writable storage

  • Migrate all environment variables

  • Migrate all cron jobs

  • Migrate SMTP Settings

  • Migrate all DNS Aliases

  • Migrate all SSL certificates

  • Migrate New Relic ID

  • Minimize downtime

To complete the migration, all you’ll need to do is update your DNS A-record(s) to point to your app on v2. This isn’t something we can do for you.

Things to Know

Only App Owners Can Migrate Apps

In order to start the app migration process, you must be the owner of the app. This is referring to the collaborative role of “owner,” not the person who pays for the app (which in some cases is different). To check your v1 user role, go to the Admin tab of your app dashboard. The owner is the only user without an “x” to the right of their username and the “Admin” checkbox is grayed out.

Some Downtime is Required

To complete the migration process, some downtime is required. The length of the downtime depends on how quickly we’re able to migrate your data & writable storage as well as your domain’s TTL setting. More information below.

The Migration Tool Only Migrates to Production Apps

On Pagoda Box v2, there are two types of apps: Tinker & Production. Tinker apps are free apps with limited functionality. Production apps are paid apps with full access to all Pagoda Box functionality. When using Butler to migrate to v2, you can only migrate to a production app. If you want to migrate to a tinker app, you’ll have to do a manual migration. The Tinker & Production Apps doc provides more details about what functionality is available to each application type.

Butler Does Not Migrate Data in Cache Components

When using Butler, data housed in a Memcached or Redis component will not get migrated to v2.

Manually Migrating Your App(s)

If you’d rather migrate your app(s) manually, the Manually Migrating Apps from Pagoda v1 to v2 doc provides a step by step on how to do it.

Using Butler

Butler is available in your v1 app dashboard. It will appear on each of your apps' dashboard home pages.

Once started, it will walk you through the following steps:

1. Create / Log in to Your V2 User Account

If you haven’t already created an account on pagodabox.io, you’ll have the opportunity. If you already have an account, you’ll be able to sign in.

2. Verify You’re Human

In order to push custom code to an app on Pagoda Box v2, you must first verify you’re human. We will send you a verification code either by text or voice message. You’ll only have to do this once.

3. App Introspection

The app introspection process analyzes your app and outlines a compatible v2 architecture. This step is focused mainly on billable items and will provide a side-by-side comparison of your v1 and v2 line items. It checks the following:

  • What components exist in your app

  • The current scale of each component

  • How much writable storage your app is using

  • What, if any, SSL certificates are installed on your app

4. Compare v1 & v2 App Pricing

This is the side-by-side price comparison established in the app introspection process. If you have questions about why the line items and/or prices are different, the Why Are My v1 & V2 Apps Different? doc has more information.

There may be adjustments you can make to your app that will reduce the cost of your app on v2, most of which are covered in the Prepping Your App for a Smooth v2 Migration doc. For example, if you notice you’re using more shared writable storage than you expected and it’s increasing the size & cost of your v2 network storage, you can clear out any unnecessary writable storage. After clearing it out, reset the migration process and re-introspect your app.

What You See is What You Get

Whatever services, sizes, and other line items are shown in your v2 pricing list are what will be provisioned when your app is cloned to v2. If there are adjustments you want/need to make, make them before cloning your app to the new infrastructure.

5. Clone Your App to the New Infrastructure

Once you’re good with the v2 build/pricing and are ready to make the move, just click the “Clone App to New Infrastructure” button. This will start the first of two migrations to your v2 app. In this initial migration, the following actions are taken:

Begin 3 Free Days on v2

Once you clone your app to v2, a 3-day period begins where your entire v2 app is free. This is to keep you from having to pay for two infrastructures at the same time. Please note however, if the migration is not completed in the 3-day period, you will accrue charges on both your v1 and v2 apps.

Add/Select a Credit Card

A credit card is required to create a production app on v2. Even if your v2 app will be free, you’ll still need to provide a credit card. If you already have cards, associated with your v2 user account, you’ll be able to select which card to use.

SSH Key Migration

All SSH keys associated with your v1 user account are copied to your v2 user account. You will not have to add them manually. When you’re ready to push and pull code to and from your v2 app, you’ll be able to.

App Creation, Service Creation, & Code Deployment

Your app is created on the v2 infrastructure and is named using the following convention: appname-v2. "v2" is appended to the app name to prevent any naming conflicts. Resetting the process will remove an app that has already been created on v2.

Feel free to change the name of the app after it’s been created. The Changing Your App Name doc walks through how.

Once the app is successfully created on v2, all services detected in the app introspection process are created and scaled to their proper scale. Code is then deployed to all code services (webs & workers).

Your "Auto-Deploy" Branch Will Be Deployed to v2

Whatever branch is set as the “auto-deploy branch” on your v1 app is what will be pushed and deployed to your v2 app. Know that only the master branch is deployed on v2 apps. All other branches are rejected. It’s possible to push a local branch to a remote’s master branch. More information is available here.

Data Migration

Once all data services are provisioned in your v2 app, data is exported from your v1 data services and imported into your v2 data services.

If you have a MySQL database that is currently using most of it’s RAM, we recommend scaling that database up before running this initial migration. The export process does require memory, and if it runs out of memory, the migration process will get stuck and will not be able to proceed without manual intervention by an engineer.

Writable Storage Migration

All directories and files stored in your writable storage are copied into your v2 network storage. Depending on the size of your network storage, this process may take some time. If there are files stored in writable storage that you don’t need, it’s recommended you remove them before starting the migration process.

App Details Migration

All of the following application details, if present, are added to your v2 app:

  • SMTP Credentials

  • Environment Variables

  • Cron Jobs

  • DNS Aliases

  • New Relic ID

SSL Migration

If you have any SSL certificates installed on your v1 app, the Private Key, Certificate, Certificate Authority, and any Intermediated Certificates are copied to your v2 app. For each SSL certificate, a new dedicated IP is allocated and bound to the cert. You do not need to rekey the certificate when migrated through the Butler.

The SSL Migration Will Not Affect Uptime

The copying existing SSL certificates to your v2 app will not affect the uptime of your v1 app. It’s possible for the same certificate to exist on multiple servers. The v2 SSL certificate will not be used until you point your domain at the IP to which the certificate is bound.

V2 App is Up & Running

Once complete, your app will be up and running on v2, accessible at app-name.gopagoda.io. There may be some necessary code adjustments to get things fully functionally on v2. These adjustments should be made before finalizing your migration.

In order to push updates to your v2 app, you will need to change the clone url associated with your pagoda remote. Your app’s Git Clone URL is available under the “Admin” tab of your app dashboard. The command to change a remote’s clone url is shown below.

If you need to continue to push changes to both your v1 and v2 apps, the workflow for doing so is described here.

6. Finalize Migration

Once you’ve confirmed your v2 app is running. You’re ready to finalize your app’s migration. Before doing so, there’s a few things you should know and do.

Downtime is Required

During the finalization process, one last data migration is made to ensure all data in your v2 app is up to date. To prevent additional data from being added after the migration begins, both your v1 and v2 app are taken offline. Once offline, your v1 app will never come back online. Our routers will serve a maintenance page on both apps. Once we detect the data migration is complete, your v2 app will come back online and the maintenance page will be removed.

Plan a Time & Let Your Users Know

When the final migration of your app takes place is completely up to you. Because downtime is required, we recommend planning it when you know traffic to your app will be low as well as notifying your users of system maintenance.

Lower Your TTL 48 Hours Ahead of Time

Your domain’s Time to Live (TTL) defines how long your A-record is cached in nameservers. Turning your TTL down 48 hours ahead of time will allow any cached routes to expire and updates to your A-record will propagate quickly.

The 48 hour window is somewhat arbitrary. It all depends on what your TTL is currently set at. If it’s only set to an hour (3600), you’re probably fine to leave it as is and move ahead with the migration. Most DNS providers don’t allow TTLs to be set any lower than an hour. The time between turning your TTL down and beginning the final migration should be no less than your current TTL.

This is the Point of No Return

Once you finalize your migration, you will no longer have any access to your v1 app. All data will be moved to v2 and your v1 app will be decommissioned.

Start the Final Migration

Once you’ve notified your users of downtime, turned your TTL down and waited for cached routes to expire, go ahead and click “Finalize & Migrate”.

Point Your A-Record to a v2 IP

Once you’ve started the final migration (not before), update your domain’s A-record to point to an IP on v2. Available IPs can be found under Network > Inbound IPs in your v2 app dashboard. If your domain needs https, you need to point your A-record to the IP to which your SSL certificate is bound. This can be found under Network > SSL Certificates in your v2 app dashboard.

Once the change to your A-record has propagated, and the final migration is finished, your app will come back online and the migration will be complete.

Your App Is Migrated! What Next?

Congratulations, your app is now on Pagoda Box v2. If you haven’t already, you need to delete your v1 app and update a few local git settings to be able to push to your v2 app.

Delete Your v1 App

As a safety precaution, your v1 app persists on the v1 infrastructure after the v2 migration is finalized. If anything in the final migration goes wrong, we still have the data & code on hand and will keep it on hand as long as you'd like us to (or until v1's end of life). Know however, that because data is persisted and resources are still being used, billing on your v1 app will continue.

As soon as you're certain your v2 app is fully fucntional, delete your v1 app to stop billing. If you have any issues in the final migration, submit a ticket and we'll help in any way we can.

Update Your Git Remote URL

All apps on Pagoda Box v2 have a new git clone url. This needs to be set as a git remote in order push changes to your v2 app. Simply cd into your apps repo and change the git clone url tied to your pagoda remote.

Changing the Clone URL of Your Pagoda Remote Terminal

  $ git remote set-url pagoda git@git.pagodabox.io:apps/app-name.git

Once you’ve updated your pagoda remote’s git url, you’ll be able to deploy changes your app just as you did on v1. If you haven’t already read through it, now would be a good time to read through the Things to Know When Moving to Pagoda Box v2 doc.

For Those Who Were Using SFTP Mode on V1

If you were using the now removed SFTP/Vintage mode, you’ll need to familiarize yourself with the Git workflow. While it can be intimidating at first, once you get the hang of it, it’ll be hard to go back. Our Basics of Using Git with Pagoda Box doc provides a quick introduction to the git workflow and walks through what you need to know to manage and deploy code using git. There are also many Git GUIs out there to will help anybody who isn’t comfortable working from the command line.

If you have any questions, suggestions, or corrections, let us know.