I've had problems with this as well.
I've tried it without any extra app, just the btooth on the phone and it won't work. I have the FTP Server checked to be on and after I've paired with a mate's phone, I try to send a file and it complains that the OBS protocol or whatever is not enabled on the receiver's phone.
I have had luck using the app from the market, "Bluetooth File Transfer" (green/orange icon) but again, I often get the same OBS or OBT protocol error.
very annoying - I just want to grab a file and btooth it to a friend's phone. It really shouldn't be so difficult.