Recently, we hit a problem with Ruby’s “exit” command. If something went horribly wrong and it made no sense for our application to continue in its current state then we would abort with “exit 1″. We use supervisord to manage processes, so in this case when we exited with exit status of 1, supervisord would assume something went wrong and restart the process for us. Or at least that is what we thought…
begin exit puts "never get here" rescue SystemExit puts "rescued a SystemExit exception" end puts "after begin block"
In our case, nothing was catching the exception. So what was it?
The other way to handle exits is to run an “at_exit” block. Ruby runs any “at_exit” block when it gets an “exit” call, but does not run these with a “exit!” call.
Here is what our rather naive at_exit block looked like…
at_exit do # do cleanup # now exit for real exit end
This harmless looking piece of code was turning our “exit 1″ into an “exit 0″. When supervisord sees one of its processes exit with a status of zero, it assumes all is good and does not try to restart the process. This is big problem for uptime and a major reason for using a tool like supervisord.
Instead we should be using “exit!” if we want to die hard [with a vengeance] and be sure that the process exited with a status of 1 and is restarted by supervisord. This would bypass all SystemExit rescue blocks and at_exit blocks.
Alternatively, and a better solution, is that we never call “exit” inside an “at_exit” block and we make sure that all SystemExit rescue blocks and at_exit blocks are used with great caution and echo the original exit status when necessary.