On Aptible, all changes to your resources are performed through Operations.
You can find a list of all the Operations in a given Environment in Activity Reports.
What happens when things go wrong?
Reliability is a top priority at Aptible in general and for Aptible in particular. That said, occasional failures during Operations are inevitable, and may be caused by:
- Failing third-party services: we strive to minimize dependencies on the critical path to deploying an App or restarting a Database, but Aptible nonetheless depends on an number of third party services. Notably, Aptible depends on AWS EC2, AWS S3, AWS ELB, the Docker Hub (with a failover or Quay.io and vice-versa). These can occasionally fail, and when they do, they may cause Aptible operations to fail.
- Crashing instances: ultimately, Aptible is built on a fleet of Linux instances running Docker. Like any other software, Linux and Docker have bugs, and may occasionally crash. Here again, when they do, Aptible operations may fail.
With this in mind, Aptible was designed to work around failures and mitigate their impact. With exception of a handful of trivial operations with no side-effects (such as launching Ephemeral SSH Sessions), all Aptible Operations are designed to support rollbacks in the event of a failure.
This means that if an Operation fails, Aptible will proceed to undo anything that can be undone (ideally and usually rolling back all changes that were performed).
If a rollback happens, you'll be notified in the logs streamed to your terminal while the Operation is running. Ultimately, Aptible will indicate whether the rollback succeeded (i.e. everything was restored back to the way it was before the Operation) or failed (some changes could not be undone).
Some side-effects of deployments cannot be rolled back by Aptible. In particular, database migrations performed in
before_releasecommands cannot be rolled back (unless you design your migrations to roll back on failure, of course!).
We strongly recommend designing your database migrations so that they are backwards compatible across at least one release. This is a very good idea in general (not just on Aptible), and a best practice for zero-downtime deployments (see Concurrent Releases for more information).
You can initiate a rollback manually using the
To further mitigate the impact of failures Aptible Operations are designed to be interruptible at any stage whenever possible.
In particular, when deploying a web application, Aptible performs Zero-Downtime Deployment. This ensures that if the Operation is interrupted at any time and for any reason, it still won't take your application down.
When downtime is inevitable (such as when resizing a Database volume or redeploying a Database to a bigger instance), Aptible optimizes for minimal downtime.
For example, when redeploying a Database to another instance, Aptible must perform the following steps:
- Shut down the old Database Container.
- Unmount then detach the Database volume from the instance the Database was originally scheduled on.
- Attach then remount the Database volume on the instance the Database is being re-scheduled on.
- Start the new Database Container.
When performing this Operation, Aptible will minimize downtime by ensuring that all preconditions are in place to start the new Database Container on the new instance before shutting down the old Database Container. In particular, Aptible will ensure the new instance is available and has pre-pulled the Docker image for your Database.
Updated 10 months ago