I can boot with that same ISO in VM workstation by creating a new vm in workstation and it works fine.. It's only from ESXi it doesn't work.. Even when with all the boot options, as soon as I boot it goes right to pxe.
a .dmg iso is an encrypted image, so the log file seems to be verifying it is not encrypted which is true.. I have several vm machines in vm workstation all running fine with various ISO images, yet none work in ESXi