Either you go back to STOCK using Odin/Hemidall , Root it and Flash a Custom Kernel like Speedmod/Corn
Flash a Custom Kernel through Odin;
Download a custom kernel like Semahore/Corn/Speedmod (.tar version) depending upon the android version you're using(GB/ICS/JB/KK). Then place the downloaded tar file in PDA and Flash it (dont make any changes in the other fields). After flashing, Try going to recovery mode( by Volume UP+DOWN )..It should open recovery now.
PS: If you're man who likes to play with custom ROMs on Captivate(with broken power button), Its High likely that sometimes you'll get stuck with NO Recovery+NO Download mode, for which you'll need USB JIG to fix your cappy