That RUU file is for Europe.... Are you in Europe? Is that a GSM phone? If so I'm afraid I can't advise about radio firmware versions. Firmware for radio is probably different for European users and the guide which I was referencing is Ramjet's guide found here:
http://xdaforums.com/showthread.php?t=1889438 but that guide is for Sprint or Virgin Mobile users (CDMA phones). Basically the guide is accurate I think for European user too but the RUU file should be different. Well, you should consult Ramjet on that as he wrote the guide and may know more. Also, you are correct that running the RUU will return the phone almost completely to stock. It replaces radio firmwares and HBOOT and will put you back to non-root. I just suggested it as a way to make sure you go completely back to stock and start from scratch so I can be sure you were on same versions of everything as me but if you are in Europe you cannot exactly follow it maybe. Anyway, I think you have explained all your issues though. By the sound of it you probably flashed USA (Sprint) radio firmware by accident when you downgraded your HBOOT. Next time you downgrade HBOOT you should make sure the file you are using is HBOOT only with NO radio firmwares in it.
So I think you have two options....
1.) To go back to stock using latest Europe RUU, root/S-OFF phone again (S-OFF might stay, I forget), put 4EXT back and then try to downgrade HBOOT to a compatible version again. This time you must downgrade without changing radios or if you can find a flashable zip with just your proper European 11.77 radios then you can flash just the radios back to 11.77 after downgrade. I don't know the version numbers for GSM phones for HBOOT or radios so I apologize if I am getting this wrong.
2.) Assuming your only problem is the 3g connection currently then maybe you can just find a flashable zip which will update only your radio firmware back to 11.77 without having to go completely back to stock. If everything else about the phone seems OK then this is definitely faster assuming you can find the right file.
Again, I know little about GSM phones and European radios so you may want to find a more knowledgeable person to help you. Definitely though if you flashed the wrong radios then you will lose your 3G connection! I have seen HBOOT's posted both with and without radio firmwares included so I think from now on you must be extra careful when flashing HBOOT to make sure you are not changing radios to USA radios and you should keep around a flashable zip with only your 11.77 radios in it so you can fix this as it may happen again.
Good luck!