Many people have complained about the Wave bootloader being closed and that being major problem for the development of alternative OS.
I had a closer look at the booting process and would like to contribute my observations to the community. I shall have little time (next to none) to work on it further, so I'd like someone to take it from this point.
OK, that said I can introduce you to what I found:
The booting process starts with initialization of the hardware, interrupts, etc. and gets to the selection of the booting mode. This is the place that checks the key combination, JIG and possible problems. Basing on this the bootloader will run the phone in either normal boot mode, go to dowload or upload mode.
Normal boot shall start with checking the FOTA module. If you already tried flashing your phone you probably noticed that some versions of the FW include a file with *.fota extension. The file is unencrypted and not signed. It's about 2MB, but the bootloader reserves exactly 3MB for it. FOTA is intended to be used for firmware update over the air, but I know nothing about it being used for Wave. You may read something about the design and get a concept of that process here:
Basically, it is possible that boot would need to perform some actions that are a result of FOTA. Therefore, during the normal boot it reads the FOTA module from the NAND (0xC600000) and checks whether the module exists and is in the right version. That is done by checking a magic (text "FOTA_ENGINE_VER_INFO_2.0") under the 0xC600100. If it is found missing or incorrect you will end up with the message "FOTA Engine is not intalled" or "FOTA Engine version mismatch" on the screen and you will need to restart your phone in the download mode to load it.
After that, the code checks for additional magic values at 0xC880000. In case it is "BPDZ" it jumps to the code in the FOTA file. The contents of the file is loaded to RAM location 0x43800000 and executed from there.
I've made an experiment as a proof-of-concept and have a confirmed that the above is true and valid information. I crafted a FOTA file longer that the usual attached (to be bigger than 2,5 MB). In case you want to repeat that, remember that last 1024 bytes are not loaded and insert additional data before that. My file had two magic values:
"FOTA_ENGINE_VER_INFO_2.0" at 0x100 offset and "BPDZ" at 0x280000. At offset 0 I've placed my code that started with several NOPs (just in case) and code that called original bootloader functions to display text on the screen.
After loading the file with Multiloader, the message appeared on the screen as expected. Reloading of the original FOTA file made the phone boot normally.
The discovery opens wide area of possibilities starting with replacing bootloader without signing it or using JTAG, multiboot, etc.
As the original bootloader is in the memory as well, we can use it, but I would not recommend that approach as we would need additional version control and changing original routines and data addresses for each version.
OK. I hope I made it clear enough to understand, but I can clarify what I might have omitted in the description. The idea is that someone here would pick that up from where I finished and develop a decent loader leaving the original files (apart from FOTA) untouched.
Edit: Added proof-of-concept FOTA file (based on XXJL2 FOTA). Use wisely - remember you take full responsibility for what you load on your phone. Works ONLY with XXJL2 bootloader.