As you know, Archos bootloaders check digital signatures of init and recovery kernels, so you need to install SDE to use custom kernels, and it somehow "watermarks" the device.
Good news everyone! I've disassembled both bootloaders, found the code which checks signature, and replaced it (first instructions of verify_hash function) with "return 0" which is "mov r0, #0; bx lr" in ARM assembly. It's much the same hack as on Archos 5, thanks EiNSTeiN from archos.g3nius.org for reverse engineering previous generation.
Archos gen8 boots using OMAP boot ROM from internal eMMC card. Primary bootloader ("boot0") is in 0x20000 bytes after the first sector of internal flash (i.e. at 0x200) and secondary bootloader is written into rawfs, /mnt/rawfs/avboot. boot0 contains image size and loading address in first 8 bytes.
So, here is the patch:
1) boot0: replace 8 bytes at 0x7520 from the beginning of mmcblk0 from 7F402DE9003091E5 to 0000A0E31EFF2FE1.
2) avboot: replace 8 bytes at 0x14424 in avboot from 7F402DE9003091E5 to 0000A0E31EFF2FE1 (same patch). 0x14424 from avboot beginning is usually 0x14824 from the beginning of mmcblk0p1 (avboot comes first in rawfs, just after 2 blocks of header).
Of course you need root to do it. I've done it on my Archos 101, then changed 1 byte in recovery image - it boots into recovery without problem (before the hack it didn't boot into this 1-byte changed recovery).
And of course do it with caution and at your own risk DO NOT replace the bytes if you find other original data at these offsets! Bad boot0 or avboot means bricked Archos. There must be some sort of test point (something connected to OMAP SYS_BOOT5 pin) to boot from USB, or a boot UART interface, so debricking the device must be possible, but it would require some effort to find it, find a proper bootloader and use it.
If someone wants to see IDA database, I'll send my.
P.S: I do not have enough messages to post inside Development subforum, so I'm posting here.