Transferring files from one machine to another is a crucial part of the ETL (extract, transform, load) process. But how can you be sure that these files can be sent and received without falling into the hands of malicious third parties?

FTP (File Transfer Protocol) has been used for decades to transfer files between a client and server on a network. For more sensitive data, however, SFTP has become the standard way to perform secure file transfers. So what is SFTP exactly, and how can you use SFTP to securely transfer files?

Table of Contents

  1. What is SFTP and What Does It Stand For?
  2. How Does SFTP Work?
  3. How to Set Up an SFTP Server
  4. How to Connect with SFTP
  5. How to Transfer Files with SFTP
  6. List of Useful SFTP Commands

What is SFTP and What Does It Stand For?

SFTP is a network protocol for securely transferring, accessing, and managing files on a remote computer. The SFTP protocol is intended as a more secure alternative to the traditional FTP protocol.

The term SFTP stands for SSH File Transfer Protocol, where SSH is a cryptographic protocol that allows clients and servers to connect remotely. The files that you send or receive using SFTP are protected by SSH encryption in transit. This added layer of security means that SFTP is preferable to FTP in the vast majority of cases.

For data-driven businesses that depend on the insights from their business intelligence and analytics workloads, SFTP has the following benefits:

  • Protecting sensitive information through end-to-end encryption and user authentication.
  • Achieving compliance with industry regulations such as HIPAA, Sarbanes-Oxley, PCI DSS, and GDPR.
  • Making data more accessible to people throughout the organization.
  • Improving speed and efficiency by supporting data transfers in bulk.

For more information about Integrate.io's native SFTP connector, visit our Integration page.

How Does SFTP Work?

SFTP builds on the existing FTP protocol, which is not safe enough to use for many purposes. FTP transfers files using plaintext, which potentially allows bad actors to "eavesdrop" on your communications.

As mentioned above, SFTP depends on the SSH protocol to securely encrypt the files it transfers. Thanks to this protection, anyone trying to spy on an SFTP connection will see nothing but unintelligible bits and bytes, rather than meaningful data.

Part of this increased security includes multiple ways to authenticate users. As we'll discuss in the next section, you can log into an SFTP connection using a username and password, or a username and SSH key.

How to Set Up an SFTP Server

There are two main ways to set up an SFTP server: 

  1. Build your own.
  2. Use a managed cloud service.

If you’d like to build your own, any Linux machine can act as an SFTP server as well. You would have to create users, set their passwords or add public keys that can be used for authentication, and assign them home directories and permissions. You may also want to enforce inbound network rules for enhanced security, and depending on how critical this service is to you,  configure backups, monitoring, and alerting. On Windows servers, you will need to use 3rd party software like WinSCP or Filezilla SFTP Server, as SFTP is not native to Windows.

If instead, you’d like to have an SFTP server without the hassle, there are many cloud-based offerings out there with different features and pricing plans. The most popular and trusted one is AWS Transfer for SFTP, which uses Amazon S3’s durable, scalable, and highly available storage to store files, and allows you to access files via multiple protocols (including Amazon S3). 

SFTP To Go is based on AWS Transfer for SFTP and delivers the same level of service with added simplicity and cost savings.

How to Connect with SFTP

Before securely transferring files with SFTP, you need to establish a connection to the remote server. In this tutorial, we’ll assume you’re using a Unix-based operating system such as Linux or macOS. 

The sftp command connects to a remote server using the SFTP protocol. To get started, you will need the server’s domain name or IP address, as well as your username on the server. For example:

> sftp username@example.com > sftp username@192.168.1.1

By default, SFTP uses TCP port 22 for communications. If you want to use a different port, you’ll have to use the -P flag to indicate the alternate port number:

> sftp -P 2222 username@example.com

Enter the appropriate sftp command into the terminal. The connection will be initialized and the server will ask you to enter your password. After entering your password correctly, you’ll be greeted by a welcome message and see an SFTP command prompt:

Connected to example.com. sftp>

If you would prefer not to enter your password every time you use SFTP, you can set up an SSH key and transfer your public key to the remote server. Once the server has your public key, it can authenticate you automatically.

How to Transfer Files Using SFTP

Now that you’ve connected to a remote server, how do you transfer files using SFTP? The steps to follow will depend on whether you’re transferring files with SFTP from a remote server to your machine, or vice versa.

Regardless of which direction you want to transfer files, it’s a good idea to first check your current directories using the lpwd and pwd commands. The lpwd command tells you the current working directory on your local machine, and the pwd command tells you the current working directory on the remote server:

sftp> lpwd Local working directory: /home/user sftp> pwd Remote working directory: /

You can use the lcd command to change the local working directory, and the cd command to change the remote working directory.

Transferring Files with SFTP From a Remote Server to the Local Machine

The get command is used to transfer files from a remote server to your local machine. You need to specify the relative path to the file you want to download, based on your current working directory on the server:

sftp> get Winter.jpg

The above command will download the file “Winter.jpg” from the current remote working directory.

sftp> get Downloads/manual.pdf

The above command will download the file “manual.pdf” from the directory “Downloads,” which is located in the current remote directory.

Once you execute a get command, you’ll see a progress bar in the terminal window that indicates the status of the download, e.g.:

Fetching /Downloads/manual.pdf to manual.pdf /Downloads/manual.pdf                                                   100%   12MB   2.9MB/s   00:04

When the download is complete, you can find the file in your current local working directory.

To download an entire directory at once, use the -r flag:

get -r Downloads

Transferring Files with SFTP From the Local Machine to a Remote Server

Transferring files from your machine to a remote server is very similar to the reverse operation. The major difference is the command that you use: put instead of get. For example:

sftp> put Winter.jpg sftp> put Downloads/manual.pdf

The first command above transfers the file "Winter.jpg," which is located in your current local working directory, to the current working directory on the server. The second command transfers the file "manual.pdf," which is located in the directory "Downloads" in the current local working directory, to the server.

As before, you should see a progress bar in the terminal window during the upload:

Uploading Winter.jpg to /Winter.jpg Winter.jpg                                100% 847KB 488.5KB/s  00:01

List of Useful SFTP Commands

  • bye: This command closes the SFTP connection.
  • cd: This command changes the current remote working directory.
  • chmod: This command changes the permissions of a given file or directory. You can grant or revoke the permissions to read, write, and/or execute the given file or directory.
  • df: This command displays the available disk space on the server, which can help you troubleshoot the reason for a failed transfer.
  • exit: This command closes the SFTP connection.
  • get: This command downloads the given file or directory.
  • lcd: This command changes the current local working directory.
  • lls: This command displays the contents of the current local working directory.
  • ls: This command displays the contents of the current remote working directory.
  • lpwd: This command displays the current local working directory.
  • put: This command uploads the given file or directory.
  • pwd: This command displays the current remote working directory.
  • quit: This command closes the SFTP connection.
  • sftp: This command initiates an SFTP connection.

Transferring Files with SFTP and Integrate.io

 

Integrate.io includes full support for the SFTP protocol in ETL workflows. Using Integrate.io and SFTP, you can send data to, and receive data from, dozens of cloud data warehouses and analytics platforms.

To start using SFTP with Integrate.io, follow the steps below:

  • Click on your avatar, and select Account Settings from the dropdown menu.
  • Click on Connections in the left menu, and select "New Connection"

thumbnail image

  • Click on "Secure File Transfer Protocol (SFTP)", and enter the relevant connection information.

thumbnail image

  • Click on "Test Connection" to test the connection. If successful, click on "Create SFTP Connection."

For more details, check out our full article on creating an SFTP connection in Integrate.io.

Want to learn more about how the Integrate.io platform can help you build powerful data integration pipelines to the cloud, including with the SFTP protocol? Schedule a call with our team for a chat about your business needs and a trial of the Integrate.io platform.