In my case, that depends: Versions 0.8 and below used RFS. 0.9 switches to ext4.
To know which file system is used, open the terminal (or adb shell) and type the "mount" command. It will list all mounted partitions and their file system, as well as mounting options in use.
To support ext4, you need a compatible kernel (the stock Samsung kernels aren't.) plus the proper ramdisk scripts in your boot.img.
My phones: Galaxy Note II SGH-T889V (with LTE enabled
/ ext4 SD card support on TouchWiz ROMs
Previous phones: Nexus S GT-I9020A, Galaxy Gio GT-S5660M (Arpeggiomod ROM
/ Discussion, stock rooted ROM
Phones for which I play "tech support" for family and friends: Galaxy Nexus GT-I9250, Galaxy Q SGH-T589R, Galaxy 551 GT-I5510M, LG C-800G Eclypse
- Level 8