INDY. Data/File transfer using UDP

This is the forum for miscellaneous technical/programming questions.

Moderator: 2ffat

INDY. Data/File transfer using UDP

Postby SolodovRD » Mon Aug 10, 2015 5:41 pm

Hi,

I try to make file transfer over UDP. I use Indy UDP Server (TIdUDPServer) as sender and Indy UDP Client (TIdUDPClient) as receiver. Or UDP Server (TIdUDPServer) as sender and as receiver on "client part". First at all receiver sends request for some file, and if it exist on sender's part, the sender send response that the file exist.
Then receiver request some file part from sender, and after receiving sends receiving confirmation. Sender wait for the confirmation and another request for other part.
I need to keep possibility to receive data/file from different sources.
So, after receiving request sender open required file

if (OpenDialog1->Execute)
{
String FName=OpenDialog1->Filename;
etc...
}

After receiving all the parts receiver seve it using SaveDialog.

Please, could you show me how to make sending file part by part, each part upon a request. As an example each part = 256 kb. How to send requests to different transmitter(s), wait for the required part(s), make confirmation sending and save file after receiving.

I think after receiving file request from receiver each sender sends response file<ID> exist, <FileSize>, <Parts count>, <Something else... probably>. - it just a String. In case if file wasn't found - another response... Receiver count all received parts to make sure if all of them was received.

Please show me some samples or let me know how to use this INDY components. As Software development tool I use Embarcadero XE7. C++

Thank you.
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby rlebeau » Mon Aug 10, 2015 8:03 pm

SolodovRD wrote:I try to make file transfer over UDP.


You do know that UDP is not a reliable transport for file transfers, right? You really should use TCP for that. But, if you must use UDP, you should at least use an existing UDP-based file transfer protocol, such as TFTP (see RFC 1350 and related RFCs for TFTP extensions). Indy has TIdTrivialFTP and TIdTrivialFTPServer components.

SolodovRD wrote:I use Indy UDP Server (TIdUDPServer) as sender and Indy UDP Client (TIdUDPClient) as receiver. Or UDP Server (TIdUDPServer) as sender and as receiver on "client part". First at all receiver sends request for some file, and if it exist on sender's part, the sender send response that the file exist.
Then receiver request some file part from sender, and after receiving sends receiving confirmation. Sender wait for the confirmation and another request for other part.


That is exactly the model that the TFTP protocol uses. Every transmitted part of the file has to be acknowledged to ensure the file's delivery and integrity. Unacknowledged parts are retransmitted, and the receiver has to ignore any packets it already received.

SolodovRD wrote:I need to keep possibility to receive data/file from different sources.
...
Please, could you show me how to make sending file part by part, each part upon a request. As an example each part = 256 kb. How to send requests to different transmitter(s), wait for the required part(s), make confirmation sending and save file after receiving.


Your are asking us to essentially write an entire working file transfer system for you. That is not really what you should be asking here. You should try to write this yourself first, and then ask questions about any problems you run into.

In any case, you are trying to perform multiple file transfers at the same time? If you want to use TIdTrivialFTP/TIdFTPTrivialFTPServer for that, you would have to run them in separate threads. You probably should do that anyway, to keep each transfer separate and manageable. Otherwise, you have to implement multiplexing logic in your main thread, and that can get complicated.

SolodovRD wrote:I think after receiving file request from receiver each sender sends response file<ID> exist, <FileSize>, <Parts count>, <Something else... probably>. - it just a String. In case if file wasn't found - another response... Receiver count all received parts to make sure if all of them was received.


What you described is what TFTP already does.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1532
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 11, 2015 11:10 am

Thank you very much for this explanation. Actually I can use any of INDY components I want. All I really need - is multiple different data/file sources, part receiving upon a request, waiting confirmation for receiving of each of the part. Algorithm:
1. Receiver sends request
2. Transmitter(s) receive data request (as string) and sends response if the file exist or not.
3. Receiver sends request(s) for some part(s) to transmitter(s) and wait for parts receiving.
4. Upon part request transmitter sends required part to receiver with some header (string). Each part transmitted have some header (I need it).
5. Transmitter wait for confirmation of receiving part. After confirmation received - waiting for another request.
6. when all the parts received - receiver disconnect.

I need you professional advise - what component better for such task? Please show me the methods how to use it? How to make a stream and requests/confirmations.

Thank you again. You really helps me.
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 11, 2015 2:14 pm

I see two other components available for this problem solving one of them is IdTCPServer and other IdSimpleServer. May be available other components and better to use something else. In case of this components which of them is better? Before I used TserverSocket / TClientSockets for message sending and file sending. It isn't INDY components.

How I used it? Sample (not checked just for understanding) SEND button
Code: Select all
void __fastcall TForm1::Button3Click(TObject *Sender)
{
  TMemoryStream *MS = new TMemoryStream;
  void *P;   // Pointer to file
  int Size; // File Size
  if( OpenDialog1->Execute() )
  {
   MS->LoadFromFile( OpenDialog1->FileName ); // File Select
   Memo1->Lines->Add( "File was added to flow" ); // LOG
  }
  ServerSocket1->Socket->Connections[0]->SendText( "file" + OpenDialog1->FileName + " " + IntToStr( MS->Size ) + " " );  // Send Header


Memo1->Lines->Add ( "Header sent" );
MS->Position = 0 ;      // Stard flow ;
P    = MS->Memory ;
Size = ServerSocket1->Socket->Connections[0]->SendBuf( P , MS->Size );               // buffer send toclient; Size
Memo1->Lines->Add( "Sent: " + IntToStr( Size ) + " from " + IntToStr( MS->Size ) ); // LOG
}


After receiving file by client:
In OnClientRead event:
Code: Select all
if(ServerSocket1->Socket->Connections[0]->ReceiveText()=="end") // If cliend send END command
{
   Memo1->Lines->Add("File Received"); // LOG
   MS->Clear() ;                            // Clear flow
}


Here no anything about "part request" or "receiving part confirmation" or "waiting for another request" or INDY. No anything about multi sources... Please let me know how to make it according my requirements. Which of component is better to use and which metods/events to use
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby rlebeau » Tue Aug 11, 2015 4:20 pm

SolodovRD wrote:Actually I can use any of INDY components I want.


Then you should have a look at TIdTrivialFTP and TIdTrivialFTPServer, if you must use UDP.

TIdTrivialFTP is very simply. It has 4 methods available - 2 for downloads and 2 for uploads:

Code: Select all
procedure Get(const ServerFile: String; DestinationStream: TStream); overload;
procedure Get(const ServerFile, LocalFile: String); overload;
procedure Put(SourceStream: TStream; const ServerFile: String); overload;
procedure Put(const LocalFile, ServerFile: String); overload;


Set the TIdTrivialFTP.Host and TIdTrivialFTP.Port properties to the server host/port, and then call Get() and Put() as needed.

TIdTrivialFTPServer is similarly very simple. It has 3 events available:

Code: Select all
type
  TAccessFileEvent = procedure (Sender: TObject; var FileName: String; const PeerInfo: TPeerInfo;
    var GrantAccess: Boolean; var AStream: TStream; var FreeStreamOnComplete: Boolean) of object;
  TTransferCompleteEvent = procedure (Sender: TObject; const Success: Boolean;
    const PeerInfo: TPeerInfo; var AStream: TStream; const WriteOperation: Boolean) of object;
...
property OnReadFile: TAccessFileEvent read FOnReadFile write FOnReadFile;
property OnWriteFile: TAccessFileEvent read FOnWriteFile write FOnWriteFile;
property OnTransferComplete: TTransferCompleteEvent read FOnTransferComplete write FOnTransferComplete;


Provide an OnReadFile handler for sending file, an OnWriteFile hanlder to receive a file, and an OnTransferComplete handler for any cleanup needed.

Setup the TIdTrivialServer.Bindings collection for the desired listening port(s) as needed, and set TIdTrivialFTPServer.Active to true to start accepting clients.

When a client requests to download a file, OnReadFile is triggered. You are provided with the requested filename and information about the client. You would set GrantAccess as needed, and if true than optionally also provide a TStream to read the file data from. If you do not provide a TStream, TIdTrivialFTPServer creates a TIdReadFileExclusiveStream internally.

When a client requests to upload a file, OnWriteFile is triggered. You are provided with the requested filename and information about the client. You would set GrantAccess as needed, and if true than optionally also provide a TStream to write the file data to. If you do not provide a TStream, TIdTrivialFTPServer creates a TIdFileCreateStream internally.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1532
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: INDY. Data/File transfer using UDP

Postby rlebeau » Tue Aug 11, 2015 4:37 pm

SolodovRD wrote:I see two other components available for this problem solving one of them is IdTCPServer and other IdSimpleServer.[


Yes, you can use those for file transfers as well. However, do note that those components use TCP, but you asked for UDP instead.

SolodovRD wrote:May be available other components and better to use something else.


It really depends on your networking needs. For instance, you could use TIdFTP and TIdFTPServer instead (which are TCP-based).

SolodovRD wrote:In case of this components which of them is better?


That is a very subjective question. Again, it really depends on your particular needs. Different components do different things, and offer different functionalities.

SolodovRD wrote:Before I used TserverSocket / TClientSockets for message sending and file sending. It isn't INDY components.


No, they are not. But they work just fine, when used properly. The Indy equivalents are TIdTCPServer and TIdTCPClient.

SolodovRD wrote:How I used it? Sample (not checked just for understanding) SEND button


That is definitely NOT the proper way to use TServerSocket, for a transfer or otherwise. But whatever. In Indy, the equivalent would look roughly something like this:

Code: Select all
void __fastcall TForm1::Button3Click(TObject *Sender)
{
    if( !OpenDialog1->Execute() )
        return;

    TMemoryStream *MS = new TMemoryStream;
    try
    {
        MS->LoadFromFile( OpenDialog1->FileName ); // File Select
        Memo1->Lines->Add( "File was added to flow" ); // LOG

        TList *list = IdTCPServer1->Contexts->LockList();
        try
        {
            TIdContext *ctx = (TIdContext*) list->Items[0];

            ctx->Connection->IOHandler->Write( "file " + OpenDialog1->FileName + " " + IntToStr( MS->Size ) + " " );  // Send Header
            Memo1->Lines->Add ( "Header sent" );
            ctx->Connection->IOHandler->Write(MS);
            Memo1->Lines->Add( "Sent: " + IntToStr( MS->Size ) ); // LOG
            if (ctx->Connection->IOHandler->ReadLn() == "end")
                Memo1->Lines->Add("File Received"); // LOG
        }
        __finally
        {
            IdTCPServer1->Contexts->UnlockList();
        }
    }
    __finally
    {
        delete MS;
    }
}


SolodovRD wrote:Here no anything about "part request"


That is simply a matter of the parameters you include in the "file" request. You can certainly add additional parameters to specify which particular bytes you are interested in.

SolodovRD wrote:or "receiving part confirmation"


You don't really need that in TCP. All you need is a final confirmation that the end of the requested data was reached. You don't need to confirm each individual data packet, like you do in UDP.

SolodovRD wrote:or "waiting for another request"


The code you showed is sending a request, not receiving a request.

SolodovRD wrote:No anything about multi sources...


I already told you earlier how to handle that - you can use a different thread for each source, and let each thread manage its individual transfer(s) as needed.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1532
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 11, 2015 4:49 pm

Thanks, but how to make requests part by part with each part receiving confirmation using this components?
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 11, 2015 5:17 pm

Sorry, not seen your answer. Now I will try. I know I not need any confirmation in case of TCP using. I need it because of the task. The task is receiving file part by part from different sources with confirmation after each part receiving. So, I will try to do that. And one more thing before I start. In case of big file, how to prevent "Out of memory" when file added to stream?
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Wed Aug 12, 2015 12:56 pm

So, thanks to your advice, I decided to use UDP indy components for my application. I still need to send all data by parts no more than 256 kb, and each of this part by request. I know, UDP not guarantee the data delivery, so, because of my requirements i need to receive some confirmation as string after EACH of the part, and only then wait for another part request. What components I can use for UDP - part by part file/data transfer, how the component support each part request. Or how to make it using the component? How i can add some header for each of the part for sending? All requests and confirmations i will send and receive using TCP, as simple chat between receiver and transmitter.How to make TCP chat I know.
Something like

Form1->IdSimpleServer1->Socket->WriteLn("Request or confirmation row that I need");
Form1->IdSimpleServer1->Socket->ReadLn("The same");

Of course after connection is set.

Thank you so much.
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby rlebeau » Wed Aug 12, 2015 5:57 pm

SolodovRD wrote:Sorry, not seen your answer. Now I will try. I know I not need any confirmation in case of TCP using. I need it because of the task. The task is receiving file part by part from different sources with confirmation after each part receiving.


So, you are creating a distributed system, where different pieces of the same file come from different sources, like how BitTorrent works? That complicates things quite a bit.

SolodovRD wrote:And one more thing before I start. In case of big file, how to prevent "Out of memory" when file added to stream?


By not storing the complete data in memory before writing it to the intended file. Write it to the local file as it is being received from the socket. Nothing says you have to use a memory stream, you can use a file stream instead.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1532
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: INDY. Data/File transfer using UDP

Postby rlebeau » Wed Aug 12, 2015 6:04 pm

SolodovRD wrote:What components I can use for UDP - part by part file/data transfer, how the component support each part request.


You are going to have to resort to the lowest denominator - TIdUDPClient/TIdUDPServer - to give you the most control over the packets. You are implementing a custom protocol that no other Indy UDP components support.

SolodovRD wrote:Or how to make it using the component? How i can add some header for each of the part for sending?


You cannot add custom headers to the UDP packet itself, you have to put your custom header in the data payload that each packet contains.

SolodovRD wrote:All requests and confirmations i will send and receive using TCP, as simple chat between receiver and transmitter.


Why would you mix UDP and TCP in this manner? It does not make sense to send a request on TCP, transfer the requested data on UDP, then receive the confirmation on TCP. If you are already using TCP anyway, you may as well use TCP for the data transfers as well. That makes the coding easier, that makes the network management easier.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1532
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 18, 2015 5:51 pm

Thank you so much. I checked all the possibilities and carefully learned all of you responses.
I decided to use UDP for data transfer because of safety - it seems for "man in a middle" more easy to make an intrusion in case of TCP connection. If I not right - let me know.

So, all the algorithm as I can see, looks like:
1. Client (receiver) makes a request for some data using UDP. Reason - in not require any existing connection and also request can be received by all machines and devices in a network. In this stage most important is ask every machine in a network. All the machines can be a Client (receiver). I mean all of them absolutelly the same with the same possibilities.
2. If the requested information exist, Transmitter open TCP connection and sends response for the Client (receiver) that the file (data) exist. Using TCP. Reason - such response will be delivered and it will be sent only for the client.
3. Client makes TCP requests for different file parts to different transmitters (which sends response that the requested file exist). Reason - TCP request will be delivered, nothing to keep or hide from intrusion.
4. After transmitter receive some part request it add some header into the data package (It will my header that I need) and sends it. BY TCP OR UDP?? Safety - priority no 1. After sending it wait for confirmation as string in some format <...><...>...<...> by TCP. Only after receiving of such string it can receive another request. No confirmation during some time (preset) - some actions, notifications, etc.
4.1 Receiver, after the part receiving sends such confirmation.
5. After all the parts was received, received close all TCP connections. But it still can be Transmitter for other machine, and any time it wait for some UDP response for some data....

Probably my algorithm have disadvantages, but I still try to make it better...
Thank you for participation...
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Tue Aug 18, 2015 5:53 pm

English is not my native language. Sorry for the mistakes...
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Fri Aug 21, 2015 11:56 am

While I make some request/response chat using TCP IdTCPClient / IdTCPServer I found a problem with this components.
As an example:
I need to send some message
Code: Select all
IdTCPServer1->Connection->IOHandler->Write(Message.Length());
        IdTCPServer1->Connection->IOHandler->Write(Message);


So, Anyway IdTCPServer not include such method as Write or Writeln that I seen in several samples in internet. Also it Have no "Connection" method, I found IOHandler, but directly in a IdTCPServer

Like this
Code: Select all
Form1->IdTCPServer1->IOHandler->



Also Server and Client have no any method for receiving strings I seen in a samples

Code: Select all
void __fastcall TForm1::IdTCPServer1Execute(TIdContext *AContext)
{
  Form1->Memo1->Lines->Add(Form1->IdTCPServer1->IOHandler->); //Nothing like "ReceiveString"
}


How I can Solve it? I use embarcadero EX8 may be some version....
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Re: INDY. Data/File transfer using UDP

Postby SolodovRD » Fri Aug 21, 2015 2:06 pm

Indy Version 10.6.2.5263.
SolodovRD
Active Poster
Active Poster
 
Posts: 12
Joined: Mon Aug 10, 2015 10:45 am

Next

Return to Technical

Who is online

Users browsing this forum: No registered users and 8 guests

cron