you can only install custom recoveries with an unlocked bootloader.
you can access recovery with keys, like you tried, or adb commands, or if you have root download a reboot manager app (easiest solution)
you can restore an device by downloading the firmware you had.
format your sd, place your firmware like this on your sd:
SD:/dload/update.zip or SD:/HwOUC/update.zip
(update.zip can also be update.app if you have android 4.3)
then just reboot, if nothing happens reboot to recovery and select "install zip from sd"