I think my problem is when I realize that I had unsaved changes open in a different neovim instance. If the file was not dirty in any other open neovim instances then I don't have the same problem.