File transfer over TCP sockets

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
I've been trying to get a simple file transfer between desktop app and phone app. The transfer works up to a point, when it simply ...stops dead.
The server(aka desktop client) enters the listening state, and the phone goes idle.

Anyone has any samples on transfers of large file (bigger than 1 MB)?
 

snickler

Retired Forum Moderator / Inactive Recognized Deve
Aug 17, 2010
1,320
1,130
0
Dub V
www.sinclairinat0r.com
I've been trying to get a simple file transfer between desktop app and phone app. The transfer works up to a point, when it simply ...stops dead.
The server(aka desktop client) enters the listening state, and the phone goes idle.

Anyone has any samples on transfers of large file (bigger than 1 MB)?
Have you looked through GoodDayToDie's source code for the File Server? I wonder if he has anything in there that could make that work.
 
  • Like
Reactions: mcosmin222

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
Have you looked through GoodDayToDie's source code for the File Server? I wonder if he has anything in there that could make that work.
lalz.

Completely forgot about that one xD
Meh he has it written in C++

Apparently, he didn't do anything that I didn't.
 
Last edited:

ScRePt

Member
Dec 24, 2009
36
3
0
Athens
Did you double-check your socket multithreading code?
I recently had problems with sockets and it turned out that I had the muti-threading thing wrong.
I think you shouldn't use only one connection and fail if it drops ...
 

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
Did you double-check your socket multithreading code?
I recently had problems with sockets and it turned out that I had the muti-threading thing wrong.
I think you shouldn't use only one connection and fail if it drops ...
What do you mean by socket multthreading code? You mean the use of async methods? or having the thread work on background, using the socket?
 

GoodDayToDie

Inactive Recognized Developer
Jan 20, 2011
6,066
2,930
0
Seattle
@mcosmin222: The most common reason I saw for why that happened was the thread doing the transfer would crash. There's a lot of things that could cause such a crash, but because it's not the main thread or a UI thread, you don't see it. It just stops. In fact, even the debugger usually doesn't catch it (annoying as hell...)

There are a few common things that non-UI threads aren't allowed to do which you might be trying. For example, attempting to show a MessageBox on a non-UI thread will crash the thread (you can do it by adding a lambda or function to the dispatcher for the UI). In any case, feel free to use or adapt my code, or share yours here and if there's an obvious issue I'll point it out. Incidentally, you can set a larger buffer on the socket if you want the operation to complete without looping.

By the way, the only portion of my webserver that's written in C++ is the file I/O code, which I chose to do in C++ rather than .NET because the phone's stunted .NET framework makes it more difficult than I like to access arbitrary file paths. That code is all fairly clean wrappers around the Win32 calls; I suppose I could comment it more but it's very straightforward to read even if you aren't familiar with managed C++. The actual network code is entirely written in C# 4.5. You could actually simplify it a bit for a direct transfer app, too; I wrote it with a lot of multithreading in case I wanted to re-use the code somewhere that might be expected to have more than one client connecting at a time.
 
  • Like
Reactions: mcosmin222

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
@mcosmin222: The most common reason I saw for why that happened was the thread doing the transfer would crash. There's a lot of things that could cause such a crash, but because it's not the main thread or a UI thread, you don't see it. It just stops. In fact, even the debugger usually doesn't catch it (annoying as hell...)

There are a few common things that non-UI threads aren't allowed to do which you might be trying. For example, attempting to show a MessageBox on a non-UI thread will crash the thread (you can do it by adding a lambda or function to the dispatcher for the UI). In any case, feel free to use or adapt my code, or share yours here and if there's an obvious issue I'll point it out. Incidentally, you can set a larger buffer on the socket if you want the operation to complete without looping.

By the way, the only portion of my webserver that's written in C++ is the file I/O code, which I chose to do in C++ rather than .NET because the phone's stunted .NET framework makes it more difficult than I like to access arbitrary file paths. That code is all fairly clean wrappers around the Win32 calls; I suppose I could comment it more but it's very straightforward to read even if you aren't familiar with managed C++. The actual network code is entirely written in C# 4.5. You could actually simplify it a bit for a direct transfer app, too; I wrote it with a lot of multithreading in case I wanted to re-use the code somewhere that might be expected to have more than one client connecting at a time.
I am aware that some calls from background threads are not allowed, especially those that have to do with the UI thread.

This is the code for the server. It would seem this one is the problem, somewhere...I just can't see where...
I tried limiting the number of packages sent (that's what the timer is all about).

Code:
       public class StateObject
    {
        // Client  socket.
        public Socket workSocket = null;
        // Size of receive buffer.
        public const int BufferSize = 1024;
        // Receive buffer.
        public byte[] buffer = new byte[BufferSize];
        // Received data string.
        public StringBuilder sb = new StringBuilder();
    }

   public class AsynchronousSocketListener
   {
       // Thread signal.
       public static ManualResetEvent allDone = new ManualResetEvent(false);
       public static string[] TransferStages = new string[] { "sendmetadataz-length", "sendmetadataz", "file-length", "file" };
       public static int Index = -1;
       public static List<string> FilePaths = new List<string>();
       public static long CurrentStreamPosition = 0;
       public static FileStream ifs;
       static int pocketspersecond = 0;
       static bool LimitExceded = false;
       DispatcherTimer timer = new DispatcherTimer();
       public static int CurrentArraySize = 0;
       public static int FileIndex = 0;
       public AsynchronousSocketListener()
       {
           timer.Interval = TimeSpan.FromSeconds(1);
           timer.Tick += timer_Tick;

       }

       void timer_Tick(object sender, EventArgs e)
       {
           LimitExceded = false;
       }

       public static void StartListening()
       {
           // Data buffer for incoming data.
           byte[] bytes = new Byte[StateObject.BufferSize];

           // Establish the local endpoint for the socket.
           // Note: remember to keep the portnumber updated if you change
           // it on here, or on the client
           IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, 13001);

           // Create a TCP/IP socket.
           Socket listener = new Socket(AddressFamily.InterNetwork,
            SocketType.Stream, ProtocolType.Tcp);

           // Bind the socket to the local endpoint and listen for incoming connections.
           try
           {
               listener.Bind(localEndPoint);
               listener.Listen(10);

               while (true)
               {
                   // Set the event to nonsignaled state.
                   allDone.Reset();

                   // Start an asynchronous socket to listen for connections.
                   Console.WriteLine("Waiting for a connection...");
                   listener.BeginAccept(
                       new AsyncCallback(AcceptCallback),
                       listener);

                   // Wait until a connection is made before continuing.
                   allDone.WaitOne();
               }

           }
           catch (Exception e)
           {
               Console.WriteLine(e.ToString());
           }

           Console.WriteLine("\nPress ENTER to continue...");
           Console.Read();

       }

       public static void AcceptCallback(IAsyncResult ar)
       {
           // Signal the main thread to continue.
           allDone.Set();

           // Get the socket that handles the client request.
           Socket listener = (Socket)ar.AsyncState;
           Socket handler = listener.EndAccept(ar);

           // Create the state object.
           StateObject state = new StateObject();
           state.workSocket = handler;
           handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
               new AsyncCallback(ReadCallback), state);
       }

       public static void ReadCallback(IAsyncResult ar)
       {
           String content = String.Empty;

           // Retrieve the state object and the handler socket
           // from the asynchronous state object.
           StateObject state = (StateObject)ar.AsyncState;
           Socket handler = state.workSocket;

           // Read data from the client socket. 
           int bytesRead = handler.EndReceive(ar);

           if (bytesRead > 0)
           {
               // There  might be more data, so store the data received so far.
               state.sb.Append(Encoding.UTF8.GetString(
                   state.buffer, 0, bytesRead));

               // Check for end-of-file tag. If it is not there, read 
               // more data.
               content = state.sb.ToString();
               if (content.IndexOf("<EOF>") > -1)
               {
                   // All the data has been read from the 
                   // client. Display it on the console.
                   Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
                       content.Length, content);

                   // Respond to the client
                   Send(handler, content);
               }
               else
               {
                   // Not all data received. Get more.
                   handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                   new AsyncCallback(ReadCallback), state);
               }
           }
       }

       public static void Send(Socket handler, String data)
       {
           //handler.SendBufferSize = File.ReadAllBytes(@"D:\MUZICA\Activ - Visez.mp3").Length;
          // handler.BeginSendFile(@"D:\MUZICA\Activ - Visez.mp3",  new AsyncCallback(SendCallback), handler);
          
           #region cotobatura 
           data = data.Replace("<EOF>", "");
           if (data.Contains("sendmetadataz") && data.Contains("length")==false)
           {
               
              
                 data = MainWindow.DataContextModel.Files.ElementAt(FileIndex).ToString()+"<EOF>";
            

                   byte[] byteData = Encoding.UTF8.GetBytes(data);
             
               // Begin sending the data to the remote device.
                 handler.BeginSend(byteData, 0, byteData.Length, 0,
                  new AsyncCallback(SendCallback), handler);
               
           }
           else if (data.Contains("sendmetadataz-length"))
           {
               Index++;
               if (Index >= MainWindow.DataContextModel.Files.Count)
               {
                   //FileIndex++;
                   data = "TransfersComplete<EOF>";
               }

               data = Encoding.UTF8.GetByteCount((MainWindow.DataContextModel.Files.ElementAt(FileIndex).ToString() + "<EOF>").ToString()).ToString();
               byte[] MetaDataLength = Encoding.UTF8.GetBytes(data);
               handler.SendBufferSize = MetaDataLength.Length;
               handler.BeginSend(MetaDataLength, 0, MetaDataLength.Length, 0, new AsyncCallback(SendCallback), handler);


           }
           else if (data.Contains("file-length"))
           {
               ifs = File.Open(MainWindow.DataContextModel.Files.ElementAt(FileIndex).Location, FileMode.Open);
               byte[] gugu = Encoding.UTF8.GetBytes(ifs.Length.ToString());

               handler.SendBufferSize = gugu.Length;
               handler.BeginSend(gugu, 0, gugu.Length, 0, new AsyncCallback(SendCallback), handler);
              
               
            

           }
           else if (data.Contains("file") && data.Contains("length") == false)
           {
               
                   //byte[] filedata = File.ReadAllBytes(MainWindow.DataContextModel.Files.ElementAt(FileIndex).Location);
                   //handler.BeginSend(filedata, 0, filedata.Length, 0,
                   //new AsyncCallback(SendCallback), handler);
                   byte[] filedata = new byte[150];

                   for (int i = 0; i < 150; i++)
                   {

                       if (CurrentStreamPosition < ifs.Length)
                       {
                           filedata[i] = (byte)ifs.ReadByte();
                           CurrentStreamPosition++;
                           CurrentArraySize++;
                       }
                       else
                       {

                           Array.Resize(ref filedata, CurrentArraySize);
                           break;
                       }
                       CurrentArraySize = 0;
                   }
                   
                  // if (pocketspersecond == 25) LimitExceded = true;
                   //Thread.Sleep(1000);
               
                   handler.BeginSend(filedata, 0, filedata.Length, 0, new AsyncCallback(SendCallback), handler);
               }
               //handler.BeginSendFile(MainWindow.DataContextModel.Files.ElementAt(FileIndex).Location, filedata, null, TransmitFileOptions.ReuseSocket, new AsyncCallback(SendCallback), handler );

    
         
           // What we want to send back in this application is a game move based on what
           // has been received. So we call Play on the GameLogic to give us a move to send back
          // data = GameLogic.Play(data);

           // Convert the string data to byte data using ASCII encoding.
           //byte[] byteData = Encoding.UTF8.GetBytes(data);

           // Begin sending the data to the remote device.
           //handler.BeginSend(byteData, 0, byteData.Length, 0,
           //    new AsyncCallback(SendCallback), handler);
         
           #endregion
       }

       public static void SendCallback(IAsyncResult ar)
       {
           try
           {
               // Retrieve the socket from the state object.
               Socket handler = (Socket)ar.AsyncState;

               // Complete sending the data to the remote device.
               int bytesSent = handler.EndSend(ar);
               Console.WriteLine("Sent {0} bytes to client.", bytesSent);

               handler.Shutdown(SocketShutdown.Both);
               handler.Close();

           }
           catch (Exception e)
           {
               Console.WriteLine(e.ToString());
           }
       }
   }
This is basically modified from the tick tak toe over sockets sample from MSDN.
The only possible call that would affect the UI is the call to the Console, but the code works fine for a while, after it just crashes.

I tried running the whole thing synchronously on the UI thread, the result appears to be the same.

In the Send method, the first 3 stages work (file-legth, metadata, metadata-length) and a few steps in the file stage (which actually sends the file).

AT some point, I assumed the thread was guilty somehow, but I just can't prove it. Running the thing directly on UI thread does not seem to change anything.
If the async method finishes and the socket gets disposed, the thread would "die".

PS: the entire thing is hosted by a WPF application.
 
Last edited:

GoodDayToDie

Inactive Recognized Developer
Jan 20, 2011
6,066
2,930
0
Seattle
Hmm... OK, there are several bugs here. I'm not sure which, if any, are responsible for the problem. I'd be tempted to overuse try-catch-log (for example, on the Send() function) and debug-print statements, but here are some things I can see that could cause a crash or other unexpected failure:

There is no guarantee that "ifs" is instantiated before use. If you for some reason skip the file-length step, the file step will crash with a null pointer exception.
The entire send function is hugely thread-unsafe. For example, if a second request arrives before you're done servicing the first one (which is entirely possible due to where the event gets signaled) then the values of "ifs" and "CurrentStreamPosition" and so on will be unpredictable at any given time. Since CurrentStreamPosition seems to be monotonically increasing, that's probably not going to cause an out-of-bounds exception, but it could cause you to enter a state where the test "if (CurrentStreamPosition < ifs.Length)" always fails.
The line "data = "TransfersComplete<EOF>";" never does anything; the next line (immediately following it) overwrites that variable. If you cared about that string, too bad.
FileIndex never changes; I hope you're only ever sending one file here...
You don't actually check that the number of bytes sent is the number you meant to send (admittedly, it *should* be, but there are cases where it won't be).
The last 150-byte chunk of every file transfer is truncated to the first byte. This is because "CurrentArraySize" is reset to 0 on every iteration of the byte-read loop (why use a byte-read loop?) so whenever "CurrentStreamPosition < ifs.Length" tests false, the "filedata" array will be resized to one byte (or zero if the file is an exact multiple of 150 bytes, which presumeably be correct).

There are probably more, but that's what jumped out at me (well, and some technically correct stylistic issues, like the "... == false" test). Given that your protocol seems to rely on end-of-message flags, I'm guessing that your problem is that since the last part of the file is almost always truncated, that marker is never getting sent. This probably leads to the client concluding that the server will be sending it more data, which it does by sending another "file" request. The server attempts to respond and immedately hits the CurrentStreamPosition < ifs.Length check, fails, goes to the else case, and tries to send a 1-byte packet containing a NULL byte.

Incidentally, does your file transfer protocol really require that the client request each 150-byte chunk one at a time, using a new TCP connection each time? That's... awfully inefficient.
 

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
Hmm... OK, there are several bugs here. I'm not sure which, if any, are responsible for the problem. I'd be tempted to overuse try-catch-log (for example, on the Send() function) and debug-print statements, but here are some things I can see that could cause a crash or other unexpected failure:

There is no guarantee that "ifs" is instantiated before use. If you for some reason skip the file-length step, the file step will crash with a null pointer exception.
The entire send function is hugely thread-unsafe. For example, if a second request arrives before you're done servicing the first one (which is entirely possible due to where the event gets signaled) then the values of "ifs" and "CurrentStreamPosition" and so on will be unpredictable at any given time. Since CurrentStreamPosition seems to be monotonically increasing, that's probably not going to cause an out-of-bounds exception, but it could cause you to enter a state where the test "if (CurrentStreamPosition < ifs.Length)" always fails.
The line "data = "TransfersComplete<EOF>";" never does anything; the next line (immediately following it) overwrites that variable. If you cared about that string, too bad.
FileIndex never changes; I hope you're only ever sending one file here...
You don't actually check that the number of bytes sent is the number you meant to send (admittedly, it *should* be, but there are cases where it won't be).
The last 150-byte chunk of every file transfer is truncated to the first byte. This is because "CurrentArraySize" is reset to 0 on every iteration of the byte-read loop (why use a byte-read loop?) so whenever "CurrentStreamPosition < ifs.Length" tests false, the "filedata" array will be resized to one byte (or zero if the file is an exact multiple of 150 bytes, which presumeably be correct).

There are probably more, but that's what jumped out at me (well, and some technically correct stylistic issues, like the "... == false" test). Given that your protocol seems to rely on end-of-message flags, I'm guessing that your problem is that since the last part of the file is almost always truncated, that marker is never getting sent. This probably leads to the client concluding that the server will be sending it more data, which it does by sending another "file" request. The server attempts to respond and immedately hits the CurrentStreamPosition < ifs.Length check, fails, goes to the else case, and tries to send a 1-byte packet containing a NULL byte.

Incidentally, does your file transfer protocol really require that the client request each 150-byte chunk one at a time, using a new TCP connection each time? That's... awfully inefficient.
I know it is inefficient, but I'm rather new to sockets. I just want it to get working in a "beta stage" then ill optimize it (hance the FileIndex never increasing, the blatant lack of try-catch blocks).

On the client side, once the bytes in the buffer are processed, the server gets another send that to send the following 150 bytes (i use 150 just for the lulz).

So basically, the workfow is as follows:

ask metadata length >server gives the length >client adjusts buffer>ask metadata
ask metdata >server gives metdata>client processes the data>asks file length
ask file length>server gives file length>client adjusts a huge array of bytes in which the file will reside (i know this is horribly inefficient, but at some point i will write directly to a file stream)>asks for the first 150 bytes in the file.
server gets the request, sends 150 bytes to client>client copies the 150 bytes in the array created earlier, the asks for the next 150.

I am using 150 just to make sure the data never splits in more than one buffer.
When the file transfer occurs, a different message is used to signal the end of transfer. Client side counts the bytes it gets, and when it is equal to the file length, it no longer asks 150 bytes.

The whole thing seems to be safe from crashing until it gets to the part where it sends the file. I am aware that in the code i gave you there's some file streams not getting closed, but i've fixed that and the problem still occurs.

Since The debugger won't help me at all, I decided to use a WCF service instead.
 
Last edited:

ScRePt

Member
Dec 24, 2009
36
3
0
Athens
What do you mean by socket multthreading code? You mean the use of async methods? or having the thread work on background, using the socket?
I mean worker threads not async.You will always have to have a thread in the background to "accept". once you accept you "read" in a new thread and the parent thread "accepts" again. Accept will freeze the thread.
On the other side, you simply "connect" and "write" in the same thread.
Read and Write is done in a loop via pre-defined buffers syncronously.

But if you want the server to give a response, the above flow is the other way around, and it is then when things get complicated. (server needs to "connect" and client needs to "accept" over a different set of ports and different threads)
Probably if you want to have reliable connection you will need the server to come back with a response "give me more" or sth.

So, trying to assist, it was my guess that drops or stalls could be because the above flow is not implemented properly.

Edit Oh ho, missed a whole new page so I am sorry if the reply is irrelevant now.
I would suggest you use the sync methods of sockets and not callbacks because is super easier to debug. ThreadPool.QueueSth (ctr + space I dont remember how it's called :) is your friend to handle threads yourself.

And try to separate pure socket handling from domain handling (lengths, metadata, etc). Send some bytes back and forth, clean-up and then move to domain specific things!
 
Last edited:

GoodDayToDie

Inactive Recognized Developer
Jan 20, 2011
6,066
2,930
0
Seattle
Moving the line that resets CurrentArraySize to outside of the for loop may well sove your problem. I'd try that first.

Optimization involves, among other things, removing try blocks. Unoptimized code, when you're trying to just make thigns work, ought to be full of them.

Don't forget that exceptions will not bubble up the call stack across threads. In addition to threads you create yourself, any async callback will happen on a different thread than the one that called the async function. If an uncaught exception occurs, the thread will die. If enough threads die, the program may crash (or at least hang) due to threadpool exhaustion.
 

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
Moving the line that resets CurrentArraySize to outside of the for loop may well sove your problem. I'd try that first.

Optimization involves, among other things, removing try blocks. Unoptimized code, when you're trying to just make thigns work, ought to be full of them.

Don't forget that exceptions will not bubble up the call stack across threads. In addition to threads you create yourself, any async callback will happen on a different thread than the one that called the async function. If an uncaught exception occurs, the thread will die. If enough threads die, the program may crash (or at least hang) due to threadpool exhaustion.
I avoid try-catch in unoptimized code to see where actual exceptions occur (sometime the debugger doesn't break on exception).

Nop still not working.

The thread still appears to be crashing, even wrapped with try catch.
 
Last edited:

GoodDayToDie

Inactive Recognized Developer
Jan 20, 2011
6,066
2,930
0
Seattle
Do you at least know *which* function it's crashing in? Try putting a breakpoint on each function header and then, when one of them is hit, step through the execution until the thread dies. Work backward from there to find the problem.
 

mcosmin222

Senior Member
May 1, 2012
1,130
287
0
Do you at least know *which* function it's crashing in? Try putting a breakpoint on each function header and then, when one of them is hit, step through the execution until the thread dies. Work backward from there to find the problem.
The send function (the one with the case switch) appears to be crashing.

It executes the function then enters the listening stage (does not execute the callback).
 
Our Apps
Get our official app!
The best way to access XDA on your phone
Nav Gestures
Add swipe gestures to any Android
One Handed Mode
Eases uses one hand with your phone