I have been in the managed file transfer space now for almost 20 years. One of the most common and recurring misconceptions I see among file transfer users is that file transfer is file transfer. This is not actually the case.
Standards based file transfer protocols such as FTP and FTPS are good for certain types of transactions but miss a key capability required for mission critical business functions. That capability is guaranteed delivery.
Guaranteed delivery is a critical requirement for businesses today. One of the biggest problems companies have when using FTP – or other, older file transfer protocols – is that there is no way to truly guarantee that the data is delivered to the remote system. These protocols lack the necessary capabilities to effectively identify if a transfer is interrupted during transmission.
Most managed file transfer (MFT) solutions have the option to leverage a proprietary protocol between their components. Among other things, this adds capabilities such as built in guaranteed delivery and automated exception management. These capabilities are supported by standard MFT features which include:
Data Validation: Proprietary protocols are designed to ensure reliable file transfer. Typically, the protocol has an initial handshake which includes information about what file is to be transferred and how it is transferred. Once the transfer initiates, the standard CRC (cyclic redundancy check) is performed on a packet-by-packet basis to ensure that there is no data corruption during transmission. At the end of the transfer, a final handshake is performed where details of the transmission (number of packets, etc.) are exchanged. This gives both sides an opportunity to notify the other if any problems have been encountered during the transfer (the protocol also allows for either side to notify of issues encountered during the transfer as well). By taking this approach, MFT solutions are able to ensure that files are delivered in their entirety with no data corruption. In the event that there is an error, both sides in the transmission are immediately made aware of the error.
Checkpoint Restart: Checkpoint Restart, when enabled, periodically sends control packets within the file transfer. This informs the receiver of the current point within the file. The receiver commits the latest data received to the file system and records the sender’s checkpoint and its own checkpoint in persistent storage. In the event that a link connecting two machines goes down during a transfer, or the transfer fails due to a recoverable error, the system will automatically restart the transfer from the point of failure; transfers are not required to start from the beginning. This is critical when transferring very large files (multi-gigabytes or even terabytes), and can save a tremendous amount of time and network bandwidth.
Recoverable Work Queue: Recoverable work queues ensure that all queued transactions are retained in the event that the system comes down for any reason. If the system comes down (e.g., a machine loses power), all transactions will still be in the work queue when it is brought back up.
Automatic Retries: If a transmission is not successful, it is placed back in the queue and automatically retried at the pre-configured interval (set by the administrator). No human intervention is required for the retransmission. If the error condition is determined to be a permanent error, the system will send notification to an administrator. This enables the administrator to correct the error in order that the transmission can complete successfully.
Businesses Need Something They Can Depend On
Solutions that rely solely on open protocols like FTP simply can’t match the levels of reliability and automation available from an MFT solution that leverages a proprietary protocol. To compensate for this, some organizations will have their developers write scripts and applications around the open protocol to help overcome its shortcomings. This is an unrealistic approach in the medium- to long-term. The organization is forced to waste valuable development resources coming up with a solution that is neither flexible nor reliable. Added to that, if your developers leave the organization, you are left with a highly-customized application that is difficult to maintain.
My position is that FTP can’t cut it at the corporate level. Companies that can control both sides of a connection are better off leveraging an MFT solution that offers guaranteed delivery right out-of-the-box.
-Kevin
About:
Kevin Bohan is Chief Information Officer at Proginet and has more than 20 years experience in international MFT best practices.
Filed under: Enterprise File Transfer | Leave a Comment
Tags: checkpoint restart, data validation, electronic data transfer, enterprise software, guaranteed delivery, managed file transfer, recoverable work queue
No Responses Yet to “All File Transfer is Not Created Equal”