Access files from sftp server

In one of the feature, we had to import xlsx which was exported from sqlite database dump. The rows had few image names which was uploaded to sftp server. We had to upload those to images on S3.

Importing xlsx row data was simple but the challenge was how to upload images from sftp server to S3. This is where net-sftp gem came to our rescue (though its not managed anymore).

To get this working, we need 4 things: host, username and password using which we are going to login to sftp server and path where images are uploaded.

We can login to sftp server using Net::SFTP.start(host, username, password: password){ |sftp| ... } and to download files/images, we can use sftp.download!(uploaded_path)

(This is nicely explained in gem documentation)

We wrote following code to upload images from sftp to S3:

Net::SFTP.start(host, username, password: password) do |sftp|
  # documents\_array is the array of file/images names
  documents\_array.each do |document|
    uploaded\_path = "/db/uploaded_files/#{document}"

    begin
      S3QueuedUpload.where(path: path\_to\_upload\_on\_s3).
        first\_or\_create!.update_attribute(:data, sftp.download!(uploaded_path))
    rescue
      puts %(Reconnect)
    end
  end
end

NOTE: S3QueuedUpload is the model which saves image/file data in binary format which in turn is uploaded to S3 via background job. You can also upload files directly to S3 instead of saving file to a model and then uploading it via background.

Hope this helps.