HTTPoison + SSL: Connection Closed

OK, so I just had my first serious "WTF" moment in the Erlang/Elixir ecosystem. It took me a few hours to understand the problem and fix it, so I decided to write a blog post about it in case someone else stumbles upon the same issue.

The Problem

Our app uses the HTTPoison package to communicate with a third-party API over HTTPS. In all our tests performed from a local environment, everything worked perfectly - however, once we generated an EXRM release, put it in a shiny Docker image and deployed to Amazon ECS, we kept getting a mysterious "connection closed" error with no further information.

iex> HTTPoison.get("https://oh-noes.com")
{:error, %HTTPoison.Error{id: nil, reason: :closed}}

We could fetch the URL successfully from the command line using curl, so we knew it wasn't anything related to firewalls, DNS, etc. We could also fetch other SSL-enabled websites without a problem with HTTPoison, so we knew it had to be a combination of things here.

The Solution

After some investigation, we noticed the root of the problem was in the SSL handshake part of the HTTP connection. Going down the rabbit hole, we traced the problem all the way back to the core Erlang ssl module.

Some Google searches later, we found this issue in GitHub - at first sight, this was not directly related to our problem but down there in the comments, the magic configuration that helped us crack the problem:

config :ssl, protocol_version: :"tlsv1.2"

It turns out some versions of Erlang (it's unclear which ones, unfortunately) try using deprecated protocols on SSL handshakes by default, like SSL v3. Some servers are strict about which protocols have to be used, so requests will get closed prematurely. The version of Erlang we use in our development machines (18.3 for Mac OSX) doesn't have the problem, but the one we use in our Docker container (18.3.2 for Alpine Linux) does. Boom! There's the problem.

If you don't want to override the global ssl configuration by using the solution above, you can also achieve the same result on a request-by-request basis on HTTPoison:

HTTPoison.get("https://oh-noes.com", [], ssl: [versions: [:"tlsv1.2"]])

I haven't tried it, but it's possible to pass a list of protocols there - I'm assuming Erlang will try all of them in order until one succeeds.

Conclusion

Crisis averted! :)