On Aptible Deploy, all changes to your resources are performed through Operations.
Reliability is a top priority at Aptible in general and for Aptible Deploy in particular. That said, occasional failures during Operations are inevitable, and may be caused by:
- Failing third-party services: we strive to minimize dependencies on the critical path to deploying an App or restarting a Database, but Aptible Deploy nonetheless depends on an number of third party services. Notably, Aptible Deploy depends on AWS EC2, AWS S3, AWS ELB, the Docker Hub (with a failover or Quay.io and vice-versa). These can occasionally fail, and when they do, they may cause Aptible Deploy operations to fail.
- Crashing instances: ultimately, Aptible Deploy is built on a fleet of Linux instances running Docker. Like any other software, Linux and Docker have bugs, and may occasionally crash. Here again, when they do, Aptible Deploy operations may fail.
With this in mind, Aptible Deploy was designed to work around failures and mitigate their impact. With exception of a handful of trivial operations with no side-effects (such as launching Ephemeral SSH Sessions), all Aptible Deploy Operations are designed to support rollbacks in the event of a failure.
This means that if an Operation fails, Aptible Deploy will proceed to undo anything that can be undone (ideally and usually rolling back all changes that were performed).
If a rollback happens, you'll be notified in the logs streamed to your terminal while the Operation is running. Ultimately, Aptible Deploy will indicate whether the rollback succeeded (i.e. everything was restored back to the way it was before the Operation) or failed (some changes could not be undone).
Some side-effects of deployments cannot be rolled back by Aptible Deploy. In particular, database migrations performed in
before_releasecommands cannot be rolled back (unless you design your migrations to roll back on failure, of course!).
We strongly recommend designing your database migrations so that they are backwards compatible across at least one release. This is a very good idea in general (not just on Aptible Deploy), and a best practice for zero-downtime deployments (see Concurrent Releases for more information).
You can initiate a rollback manually using the
To further mitigate the impact of failures Aptible Deploy Operations are designed to be interruptible at any stage whenever possible.
In particular, when deploying a web application, Aptible Deploy performs Zero-Downtime Deployment. This ensures that if the Operation is interrupted at any time and for any reason, it still won't take your application down.
When downtime is inevitable (such as when resizing a Database volume or redeploying a Database to a bigger instance), Aptible Deploy optimizes for minimal downtime.
For example, when redeploying a Database to another instance, Aptible Deploy must perform the following steps:
- Shut down the old Database Container.
- Unmount then detach the Database volume from the instance the Database was originally scheduled on.
- Attach then remount the Database volume on the instance the Database is being re-scheduled on.
- Start the new Database Container.
When performing this Operation, Aptible Deploy will minimize downtime by ensuring that all preconditions are in place to start the new Database Container on the new instance before shutting down the old Database Container. In particular, Aptible Deploy will ensure the new instance is available and has pre-pulled the Docker image for your Database.
Updated 6 months ago