Seems like a Samsung issue....
Looks like all along the Galaxy lineup there are wifi issues.
I tried everything, preventing wifi from sleeping, using wifi fixer, nothing seems to work.
Sometimes my wifi will stay connected, the upload will work, but I get no download.
Rebooting wifi on quick toggle seems to only do the trick.