From FTP
The From FTP component connects with a FTP, FTPS, or SFTP server and retrieves files from a target directory for processing. These files are downloaded to a local folder and then either deleted from the server or moved to a designated folder on the server. The downloaded files are then read and pushed downstream for further processing in the same way the From File component processes files.
By default, this component polls every 10 seconds.
Running the Component
The below steps assume an existing FTP/FTPS/SFTP server is available to connect to and the necessary user permissions and folder structure are in place on the FTP server and locally.
STEP 1: Import the From FTP component
Using +COMPONENT, import the From FTP component.
STEP 2: Set up the component configurations
Bolded fields are required, otherwise default values defined in config.json are used:
FTP Configurations:
Field | Description | Default Value |
|---|---|---|
FTP Protocol | Specifies the protocol you'll be using. Options: |
|
FTP Server | The server you will be connecting with. | |
FTP Port | The port to use. | |
FTP Username | The remote user to login as. This account will need permissions to read, delete, and/or move files to the specified Remote Destination Directory | |
FTP Password | The password to authenticate with. | |
Private Key File | The name of your private key file, if in use. By default, this should be a PEM certificate and not be password-locked. The default configurations can be changed by modifying the net.[protocol].init function directly in the script. See Adapting the Component. This configuration is only applicable for SFTP and FTPS. | |
Certificate File | The name of your certificate file, if in use. By default, this should be a PEM certificate. The default configurations can be changed by modifying the net.[protocol].init function directly in the script. See Adapting the Component. This configuration is only applicable for FTPS. | |
Public Key File | The name of your public key file, if in use. By default, this should be a PEM certificate and not be password-locked. The default configurations can be changed by modifying the net.[protocol].init function directly in the script. See Adapting the Component. This configuration is only applicable for SFTP. | |
Remote Destination Directory | The target directory on the FTP server you'll be picking up files from. | |
Connection Retry Count | The number of times to retry the connection if it drops. Default is set to 60 attempts. |
|
Retry Pause (s) | The interval of time in seconds the component will wait between retries. Default is set to 10 times. |
|
After Download | Keep remote files or delete them once the files have been downloaded to the Source Directory on the local machine. Options: |
|
Processed Remote Directory* | Specifies the remote directory that the downloaded/processed files will be moved to. * Only required if After Download configuration is set to Move remote files. |
Local Workflow Configurations:
Field | Description | Default Value |
|---|---|---|
Source Directory | The local directory where the FTP files will be downloaded to, and picked up and processed from. | |
Extension of Files | Extension of files, comma separated e.g., |
|
Minimum File Age (s) | Time (in seconds) to wait after last modified date before processing. |
|
After Processing | Local file handling procedure after reading Options: |
|
Processed Files Directory* | The directory to move processed files to. * Only required if the After Processing field is set to Move processed files | |
Extract Messages | Controls whether to try extracting messages from the file based on the Input File Type field. The component will error out if the Input File Type field is set to Arbitrary text and this field is set to true. |
|
Input File Type | Specifies the type of message to be extracted from the file. Note: Has no effect if the Extract Messages field is set to false. Note: Custom message must be defined inside DATAformats.lua and be similar in structure to HL7 and X12 (pre-set header and delimiter). Options: |
|
Ignored Segments List | Segments that will be removed from extracted messages. Works with any type of message. Note: Has no effect if the Extract Messages field is set to false. | |
(HL7 Only) Replace Segment Delimiter With | Controls the segment delimiter in the message. Hint: The HL7 standard for the segment delimiter is carriage return (\r). Note: Has no effect if the Extract Messages field is set to false. Options: |
|
Input File Encoding | The encoding used in the input data. List of available encodings is available when you click to change the encoding. |
|
Component-wide Configurations:
Field | Description | Default Value |
|---|---|---|
Poll Time (s) | The frequency (in seconds) the component polls the Source Directory for local files and remote directory for remote files. |
|
Print Debug Logs | Enable/disable logging of additional debug information. |
|
STEP 3: Start the component and view the processed files' contents in the logs
Once the component has been started, it will download files that match the provided parameters to the specified local directory. It will then process the downloaded files and push their contents to the component's queue for further processing. Open the component's logs to see the queued files and verify that the contents are being read as expected.
Adapting the Component
There are a few ways you can adapt and expand this component for more complex workflows:
Optional configuration: Custom message parsing and extracting
In the DATAformats.lua file, you can also modify the message delimiter used for each data type, as well as its header. You can also define your own custom message type's header and delimiter.
-
Keep in-mind that the length of the header is used in extracting the message properly.
-
The built-in function for extracting messages expects the segment headers to be alphanumeric (numbers and letters - no special characters).
Customize the FTP/FTPS/SFTP connection
By default, the FTP/FTPS/SFTP connection is authenticated using username and password authentication or via keys and uses the default configurations. In some cases, additional configurations may be desired, such as:
-
Specifying a FTP(S) mode
-
Modifying public and private key authentication
-
Disabling verify_peer or verify_host
-
Increasing the timeout
-
and more. Full details can be found by accessing the net.ftp.init, net.ftps.init, or net.sftp.init function help in the Translator
These configurations can be customized by modifying the desired ConnectionParameters table in the FTPconnector folder, each protocol has its own parameter table. For example, you can modify FTP's parameters without affecting the others:
