0.8.0 The Great Refactor

.github/workflows/elixir.yml

@@ -10,24 +10,28 @@ permissions:
   contents: read
 
 jobs:
-  build:
-
-    name: Build and test
-    runs-on: ubuntu-20
-
+  test:
+    name: Build and Test
+    runs-on: ubuntu-20.04
+    strategy:
+      # Don't crash if the other workers fail (i.e. 27+1.17 currently)
+      fail-fast: false
+      matrix:
+        otp: ['25', '26', '27']
+        elixir: ['1.15', '1.16', '1.17']
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Elixir
-      uses: erlef/setup-beam@988e02bfe678367a02564f65ca2e37726dc0268f
+      uses: erlef/setup-beam@v1
       with:
-        elixir-version: '1.13.3' # Define the elixir version [required]
-        otp-version: '24.1' # Define the OTP version [required]
+        otp-version: ${{matrix.otp}} # Define the OTP version [required]
+        elixir-version: ${{matrix.elixir}} # Define the elixir version [required]
     - name: Restore dependencies cache
       uses: actions/cache@v3
       with:
         path: deps
-        key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
-        restore-keys: ${{ runner.os }}-mix-
+        key: otp-${{matrix.otp}}-ex-${{matrix.elixir}}-${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
+        restore-keys: otp-${{matrix.otp}}-ex-${{matrix.elixir}}-${{ runner.os }}-mix-
     - name: Install dependencies
       run: mix deps.get
     - name: Run tests

CHANGELOG.md

@@ -1,3 +1,20 @@
+# 0.8.0
+
+## Breaking Changes
+
+* Changed underlying XML encoding and decoder library to `saxy`, while encoding should remain the same, those handling XML responses must change their code to accept Saxy's "simple" format.
+  * If you use the `normalize_xml` option, keys are no longer atoms but strings.
+* Removed `:jsonapi` response body type, it will just report as `:json` instead
+* Remove `mode` option, do not use it the respective connection manager will set the mode for itself
+* ContentClient will now return `:unaccepted` and `{:malformed, term}`` in addition to :unk to differentiate response bodies.
+  * `:unaccepted` will be returned when the response `content-type` was not acceptable from the request's accept header
+  * `:malformed` will be returned whenever the response body could not be parsed (or the `content-type` header was malformed)
+  * `:unk` will be returned for all other cases
+
+## Changes
+
+* Pepper.HTTP.ConnectionManager.PooledConnection is always in active mode
+
 # 0.7.0
 
 * `Pepper.HTTP.Client` and `Pepper.HTTP.ContentClient` can now accept a URI struct in place of a URL string

README.md

@@ -19,3 +19,6 @@ resp.body # => "Hello, World"
 * [Clients](docs/clients.md)
 * [Connection Managers - Pools](docs/connection_managers.md)
 * [Response Body Handlers](docs/response_body_handlers.md)
+* [Body Decoders](docs/body_decoders.md)
+* [Body Encoders](docs/body_encoders.md)
+* [Additional Headers](docs/additional_headers.md)

docs/additional_headers.md

@@ -0,0 +1,37 @@
+## Additional Headers
+
+Sometimes it's nice to have a common module that decorates the request with some additional headers, for example adding authorization based on the request options.
+
+By default Pepper provides an Authorization module which
+
+Starting at `pepper_http >= 0.8.0`, ContentClient additional header modules can be added:
+
+```elixir
+config :pepper_http,
+  additional_header_modules: [
+    MyAdditionalHeaderModule
+  ]
+```
+
+Additional header modules are expected to define a `call/3` function which returns a tuple with the headers and options respectively:
+
+```elixir
+defmodule MyAdditionalHeaderModule do
+  def call(_body, headers, options) do
+    # Note the body is also provided for reference, in case the header module needs to know
+    # the body ahead of time, the body is expected to be in its encoded form but may not be
+    # a valid binary
+    {headers, options}
+  end
+end
+```
+
+As with any configurable property pepper also provides a `base_additional_headers_modules`:
+
+```elixir
+config :pepper_http,
+  # by default the authorization module is provided
+  base_additional_header_modules: [
+    Pepper.HTTP.ContentClient.Headers.Authorization,
+  ]
+```

docs/body_decoders.md

@@ -0,0 +1,50 @@
+## Body Decoders
+
+__Accept__ `Accept`
+
+__Content__ `Content-Type`
+
+Starting at `pepper_http >= 0.8.0`, ContentClient decoders can be set using the config:
+
+```elixir
+config :pepper_http,
+  # Before you can use your decoder, pepper needs to know how to translate content-type headers into internal type names.
+  # The decoder_content_types (and base_decoder_content_types) config provides that information.
+  # Note the the decoder is expected to return the same type name or similar.
+  decoder_content_types: [
+    {{"application", "x-my-type"}, :my_type}
+  ],
+  # Once you have your type name, you can finally decode it using the specified decoder module
+  # The module is expected to return {type_name::atom(), any()}
+  # This is the third element returned in the response tuple:
+  #   {:ok, Pepper.HTTP.Response.t(), {type_name::atom(), any()}}
+  #   Example:
+  #     {:ok, _resp, {:json, doc}}
+  decoders: [
+    my_type: MyType.Decoder
+  ]
+```
+
+As one may have noticed, `base_decoder_content_types` was mentioned, this config is the _default_ for pepper, in addition to `base_decoders`:
+
+```elixir
+config :pepper_http
+  base_decoder_content_types: [
+    {{"application", "json"}, :json},
+    {{"application", "vnd.api+json"}, :json},
+    {{"application", "xml"}, :xml},
+    {{"application", "vnd.api+xml"}, :xml},
+    {{"text", "xml"}, :xml},
+    {{"text", "plain"}, :text},
+    {{"application", "csv"}, :csv},
+    {{"text", "csv"}, :csv},
+  ],
+  base_decoders: [
+    csv: Pepper.HTTP.BodyDecoder.CSV,
+    json: Pepper.HTTP.BodyDecoder.JSON,
+    xml: Pepper.HTTP.BodyDecoder.XML,
+    text: Pepper.HTTP.BodyDecoder.Text,
+  ]
+```
+
+Normally there is no need to overwrite that config, but it is provided just in case, otherwise you are expected to use `:decoder_content_types` and `:decoders` which will be added to the base.

docs/body_decompressors.md

@@ -0,0 +1,37 @@
+## Body Decompressors
+
+__Accept__ `Accept-Encoding`
+
+__Content__ `Content-Encoding`
+
+Starting at `pepper_http >= 0.8.0`, ContentClient decompressors can be set using the config:
+
+```elixir
+config :pepper_http,
+  # As with most custom modules, content-encoding types must be translated to an internal name
+  decompressor_types: [
+    {"br", :br},
+  ],
+  # Once the name is available, the decompressor module can be specified
+  decompressors: [
+    br: MyBrDecompressor
+  ]
+```
+
+Pepper provides modules for Identity, Gzip and Deflate out of the box
+
+```elixir
+config :pepper_http
+  base_decompressor_types: [
+    {"identity", :identity},
+    {"deflate", :deflate},
+    {"gzip", :gzip},
+  ],
+  base_decompressors: [
+    identity: Pepper.HTTP.BodyDecompressor.Identity,
+    deflate: Pepper.HTTP.BodyDecompressor.Deflate,
+    gzip: Pepper.HTTP.BodyDecompressor.Gzip,
+  ]
+```
+
+Normally there is no need to overwrite that config, but it is provided just in case, otherwise you are expected to use `:decompressor_types` and `:decompressors` which will be added to the base.

docs/body_encoders.md

@@ -0,0 +1,30 @@
+## Body Encoders
+
+Starting at `pepper_http >= 0.8.0`, ContentClient encoders can be set using the config:
+
+```elixir
+config :pepper_http,
+  # Unlike the decodes, the encoders use the type name from the body parameter during the request
+  # so there is no need to transform content types
+  encoders: [
+    my_type: MyType.Encoder,
+  ]
+```
+
+Same as with the body decoders, encoders also have a `base_encoders` config:
+
+```elixir
+config :pepper_http
+  base_encoders: [
+    csv: Pepper.HTTP.BodyEncoder.CSV,
+    form: Pepper.HTTP.BodyEncoder.Form,
+    form_stream: Pepper.HTTP.BodyEncoder.FormStream,
+    form_urlencoded: Pepper.HTTP.BodyEncoder.FormUrlencoded,
+    json: Pepper.HTTP.BodyEncoder.JSON,
+    stream: Pepper.HTTP.BodyEncoder.Stream,
+    text: Pepper.HTTP.BodyEncoder.Text,
+    xml: Pepper.HTTP.BodyEncoder.XML,
+  ]
+```
+
+Normally there is no need to overwrite that config, but it is provided just in case, otherwise you are expected to use `:encoders` which will be added to the base.

docs/clients.md

@@ -85,7 +85,7 @@ case result do
       # Content-Type: application/xml
       # Content-Type: text/xml
       {:xml, doc} ->
-        # SweetXml will be used to parse the blob and returns the document
+        # `Saxy.SimpleForm.parse_string/1` will be used to parse the blob and returns the document
         :ok
 
       {:xmldoc, doc} ->

lib/pepper/http/body_decoder.ex

@@ -0,0 +1,122 @@
+defmodule Pepper.HTTP.BodyDecoder do
+  alias Pepper.HTTP.Response
+  alias Pepper.HTTP.Proplist
+
+  def decode_body(%Response{body: body} = response, options) do
+    accepted_content_type = determine_accepted_content_type(response, options)
+
+    type =
+      case accepted_content_type do
+        nil ->
+          # no content-type
+          :unk
+
+        :unaccepted ->
+          # mismatched content-type and accept, return unk(nown)
+          :unaccepted
+
+        accepted_content_type ->
+          case Plug.Conn.Utils.content_type(accepted_content_type) do
+            {:ok, type, subtype, _params} ->
+              content_type_to_type(type, subtype)
+
+            :error ->
+              {:malformed, :content_type}
+          end
+      end
+
+    decode_body_by_type(type, body, options)
+  end
+
+  defp determine_accepted_content_type(%Response{
+    request: %{
+      headers: req_headers,
+    },
+    headers: res_headers,
+  }, _options) do
+    # retrieve the original request accept header, this will be used to "allow" the content-type
+    # to be parsed
+    accept = Proplist.get(req_headers, "accept")
+    # retrieve the response content-type
+    content_type = Proplist.get(res_headers, "content-type")
+
+    # ensure that we only parse content for the given accept header to avoid parsing bodies we
+    # didn't want or even expect
+    case accept do
+      nil ->
+        # no accept header was given, expect to parse anything, this is dangerous
+        # but allows the default behaviour to continue
+        # you should ALWAYS specify an accept header
+        content_type
+
+      "*/*" ->
+        content_type
+
+      _ ->
+        if content_type do
+          # a content-type was returned, try negotiate with the accept header and content-type
+          case :accept_header.negotiate(accept, [content_type]) do
+            :undefined ->
+              # mismatch accept and content-type, refuse to parse the content and return
+              # nil for the accepted_content_type
+              :unaccepted
+
+            name when is_binary(name) ->
+              # return the matched content_type
+              name
+          end
+        else
+          # there was no content-type, return nil
+          nil
+        end
+    end
+  end
+
+  decoder_content_types =
+    Application.compile_env(:pepper_http, :base_decoder_content_types, [
+      {{"application", "json"}, :json},
+      {{"application", "vnd.api+json"}, :json},
+      {{"application", "xml"}, :xml},
+      {{"application", "vnd.api+xml"}, :xml},
+      {{"text", "xml"}, :xml},
+      {{"text", "plain"}, :text},
+      {{"application", "csv"}, :csv},
+      {{"text", "csv"}, :csv},
+    ]) ++ Application.compile_env(:pepper_http, :decoder_content_types, [])
+
+  Enum.each(decoder_content_types, fn {{type, subtype}, value} ->
+    def content_type_to_type(unquote(type), unquote(subtype)) do
+      unquote(value)
+    end
+  end)
+
+  def content_type_to_type(_type, _subtype) do
+    :unk
+  end
+
+  decoders =
+    Application.compile_env(:pepper_http, :base_decoders, [
+      csv: Pepper.HTTP.BodyDecoder.CSV,
+      json: Pepper.HTTP.BodyDecoder.JSON,
+      xml: Pepper.HTTP.BodyDecoder.XML,
+      text: Pepper.HTTP.BodyDecoder.Text,
+    ]) ++ Application.compile_env(:pepper_http, :decoders, [])
+
+  Enum.each(decoders, fn {type, module} ->
+    def decode_body_by_type(unquote(type), body, options) do
+      unquote(module).decode_body(body, options)
+    end
+  end)
+
+  def decode_body_by_type(:unaccepted, body, _options) do
+    {:unaccepted, body}
+  end
+
+  def decode_body_by_type(:unk, body, _options) do
+    {:unk, body}
+  end
+
+  def decode_body_by_type({:malformed, _} = res, body, _options) do
+    {res, body}
+  end
+end

lib/pepper/http/body_decoder/csv.ex

@@ -0,0 +1,5 @@
+defmodule Pepper.HTTP.BodyDecoder.CSV do
+  def decode_body(body, _options) do
+    {:csv, body}
+  end
+end

lib/pepper/http/body_decoder/json.ex

@@ -0,0 +1,11 @@
+defmodule Pepper.HTTP.BodyDecoder.JSON do
+  def decode_body(body, _options) do
+    case Jason.decode(body) do
+      {:ok, doc} ->
+        {:json, doc}
+
+      {:error, _} ->
+        {{:malformed, :json}, body}
+    end
+  end
+end

lib/pepper/http/body_decoder/text.ex

@@ -0,0 +1,5 @@
+defmodule Pepper.HTTP.BodyDecoder.Text do
+  def decode_body(body, _options) do
+    {:text, body}
+  end
+end

lib/pepper/http/body_decoder/xml.ex

@@ -0,0 +1,13 @@
+defmodule Pepper.HTTP.BodyDecoder.XML do
+  import Pepper.HTTP.Utils
+
+  def decode_body(body, options) do
+    # Parse XML
+    {:ok, doc} = Saxy.SimpleForm.parse_string(body)
+    if options[:normalize_xml] do
+      {:xmldoc, handle_xml_body(doc)}
+    else
+      {:xml, doc}
+    end
+  end
+end