I think that this is a Mac-specific problem as Win and Linux guests have guest additions. I'm inclined to report this as a VB bug, but first I need to replicate this on a Mac host rather than a Win host, or they'll ignore me. It's as if running the machine in a recent version of VB somehow does something to the install of Mac OS X in the virtual machine that can't be undone. But the fact that a downgrade of VB doesn't fix my problems suggests that it's more complex. The question is - is this a Mac OS X os issue or a VB one? The fact that you reported VMWare as having the same problems suggests a Mac OS X one.

Moreover, if I downgrade to VB 4.3 and open that ML machine, the custom resolution doesn't work anymore - ML now has the same screen resolution problems. What is particularly frustrating is that I was running Mountain Lion in VB 4.3.30 and the custom resolution trick used to work. I've done a lot of searching and tried as many things as I could think of without success. EfiGopMode works but not a custom resolution. I've just used this guide to successfully install Sierra in VB 5.1.6 and have exactly the same problem.

For me VBoxManage setextradata 'OS X VM' 'VBoxInternal2/EfiGopMode' N works fine, but custom resolution (for widescreen) is not working, However, it seems there is a problem with ElCapitan, because VmWare has similar problems.

Install macos high sierra virtualbox