If you have no recovery or a factory reset doesn't work, you only have one option - back to stock via Odin to get a clean slate (How ? Thread stickied near top of Q&Q has instructions). If you have problems with the flash, Hopper8's 'Odin troubleshooting' thread stickied in General has all the info you need to get a successful flash.
Once you've done that, have another go at what you were trying to do. Nobody here can tell you exactly what went wrong because we're weren't there looking over your shoulder at what you were doing; but it's probably something small/obvious you did/didn't do - go back over the process carefully next time you try it.