Upload files on S3 with Elixir/Phoenix using ex_aws

In one of our Rails 4 app, we decided to move file and image uploads to another microservice so that the load on server is reduced when a big file is uploaded. We decided to do this in Phoenix.

In Phoenix, we have ex_aws package which makes file uploads to S3 very simple just like Rails. So lets get started.

Add ex_aws

Update mix.exs to include following dependencies.

defp deps do
  [
    ...,
    {:ex_aws, "~> 1.0"},
    {:poison, "~> 2.0"},
    {:hackney, "~> 1.6"},
    {:sweet_xml, "~> 0.6"},
  ]
end

We need :sweet_xml so that we can parse XML response from S3.

Don't forget to update the applications list.

 def application do
   [
     mod: {ApplicationName, []},
     extra_applications: [..., :ex_aws, :hackney, :poison]
   ]
 end

Finally run mix deps.get to install above dependencies.

Set AWS credentials

I prefer to add all important keys specific to a project as environment variables. So we will create .env file in root directory of the project and add following code to it.

export AWS_ACCESS_KEY_ID=<......>
export AWS_SECRET_ACCESS_KEY=<.....>
export BUCKET_NAME=<.....>

Don't forget to run source .env on console to make sure all environment variables are loaded before you compile the project.

Next, update config.exs to include the AWS credentials.

# Configure :ex_aws
config :ex_aws,
  access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
  secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
  region: "us-east-1"

System.get_env() is used to read environment variables.

Write code

When Phoenix receives a file to upload, a Plug.Upload struct is received in the params. Here's an example of what the params would look like for an image upload:

%{
  "upload" => %Plug.Upload{
       content_type: "image/png",
       filename: "some_image.png",
       path: "/var/folders/nd/snztgzpd6t92bdkfm6lnm9l00000gn/T//plug-1484/multipart-890782-844137-2"
  }
}

Phoenix will put the image in a temporary location. The image is saved till the request is completed i.e. once the conn is returned, the image will be deleted.

Now we create upload service (UploadService) under /lib directory.

defmodule S3Upload.UploadService do
  def upload_to_s3(upload_params) do
    file = upload_params.path
    bucket_name = System.get_env("BUCKET_NAME")
    s3_path = "path/on/s3"
    file
      |> ExAws.S3.Upload.stream_file
      |> ExAws.S3.upload(bucket_name, s3_path)
      |> ExAws.request!

    s3_url = "http://#{bucket_name}.s3.amazonaws.com/#{s3_path}"
    %{
      s3_url: s3_url
    }
  end
end

Next we will add a controller UploadController -

defmodule S3UploadWeb.UploadController do
  use S3UploadWeb, :controller
  alias S3Upload.UploadService

  def upload(conn, %{"upload" => upload_params}) do
    resp = UploadService.upload_to_s3(upload_params)
    render(conn, "s3_response.json", resp: resp)
  end
end

And a view s3_response.ex -

defmodule S3UploadWeb.UploadView do
  use S3UploadWeb, :view

  def render("s3_response.json", %{resp: resp}) do
    resp
  end
end

Lastly, we add route in router.ex to accept files

scope "/", S3UploadWeb do
  ...
  post "/upload", UploadController, :upload
end

Now you're set to start uploading files.

Be sure to checkout the documentation on Plug.Upload and ex_aws.