Ruby exit, exit!, SystemExit and at_exit blunder

Recently, we hit a problem with Ruby’s “exit” command. If something went horribly wrong and it made no sense for our application to continue in its current state then we would abort with “exit 1″. We use supervisord to manage processes, so in this case when we exited with exit status of 1, supervisord would assume something went wrong and restart the process for us. Or at least that is what we thought…

“exit 1″ does not actually cause the process to exit, it just raises a SystemExit exception. This is very clearly explained in the documentation.

begin
  exit
  puts "never get here"
rescue SystemExit
  puts "rescued a SystemExit exception"
end
puts "after begin block"

In our case, nothing was catching the exception. So what was it?

The other way to handle exits is to run an “at_exit” block. Ruby runs any “at_exit” block when it gets an “exit” call, but does not run these with a “exit!” call.

Here is what our rather naive at_exit block looked like…

at_exit do

  # do cleanup

  # now exit for real
  exit

end

This harmless looking piece of code was turning our “exit 1″ into an “exit 0″. When supervisord sees one of its processes exit with a status of zero, it assumes all is good and does not try to restart the process. This is big problem for uptime and a major reason for using a tool like supervisord.

Instead we should be using “exit!” if we want to die hard [with a vengeance] and be sure that the process exited with a status of 1 and is restarted by supervisord. This would bypass all SystemExit rescue blocks and at_exit blocks.

Alternatively, and a better solution, is that we never call “exit” inside an “at_exit” block and we make sure that all SystemExit rescue blocks and at_exit blocks are used with great caution and echo the original exit status when necessary.

Update

Jonathan Rochkind made a really great clarification of the exit / at_exit situation in a comment below. I think it is important to read it.

Comments