Skip to content

Commit

Permalink
Feature complete for mvp version 1 (#1)
Browse files Browse the repository at this point in the history
  • Loading branch information
wshaddix authored Feb 6, 2018
1 parent 257183d commit da7e471
Show file tree
Hide file tree
Showing 28 changed files with 958 additions and 231 deletions.
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,8 @@

* if the `NatsMessage.errorMessage` property is set then the http-nats-proxy will return a status code 500 with a formatted error message to the api client.

* now returning the `NatsMessage.response` as the http response.
* now returning the `NatsMessage.response` as the http response.

**1.0.0**

* added pipeline feature and refactored the solution to have working examples of logging, metrics, authentiation and trace header injection.
69 changes: 48 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,36 +38,63 @@ When an http request is received, the http-nats-proxy will extract the headers,

Each of the http request headers, cookies and query parameters will be represented as a key/value pair. The body will be represented as a string.

### Metrics
### Pipelines

The http-nats-proxy can be configured to collect metrics. The information collected is the name of the subject, the utc date/time of the api call as well as how long the system took to provide a response. From that information you can determine response times and call volume per api request. See the Configuration section for details on how to enable metrics.
Every http(s) request that comes into the http-nats-proxy can go through a series of steps before being delivered to the final destination (your microservice). Those steps are defined in a yaml based configuration file and the location of the file is passed to the https-nats-proxy via the `HTTP_NATS_PROXY_REQUEST_PIPELINE_CONFIG_FILE` environment variable. Each step defined in the pipeline configuration file contains the following properties:

### Logging
**subject:** The NATS subject name where the request will be delivered for this particular step in the pipeline. In order to notify the http-nats-proxy to send the message to your microservice use `*` for the subject name.

The http-nats-proxy can be configured to log http request/response pairs. In addition to the request/response it will also include metadata such as the subject, response code, execution time (if metrics are enabled), etc. See the Configuration section for details on how to enable logging.
**pattern:** Either `publish` or `request`, this tells the http-nats-proxy whether it should do a fire and forget message exchange pattern or if it should use the request/reply pattern. If you are not going to change or cancel the http request then you should use `publish`. If your pipeline step will potentially modify the request or cancel the request you should use `request`

### Tracing
**direction:** Either `incoming, outgoing` or `both`. This tells the http-nats-proxy when to call your pipeline step. `incoming` is when the message is inbound, `outgoing` is after the message has reached the end of the pipeline and the response is being returned. `both` will call you pipeline step twice, once when the request is inbound and a second time when the response is outbound.

The http-nats-proxy can be configured to inject a trace header into the http request before sending it to the NATS messaging system. This can be used to track which microservices worked on an api call as well as to correlate logs with metrics (both will contain the trace id). See the Configuration section for details on how to enable tracing.
**order:** A numeric value that is the order that your step should be called in relation to other steps in the pipeline

#### Example pipeline-config.yaml file
```
steps:
- subject: pipeline.metrics
pattern: publish
direction: outgoing
order: 1
- subject: trace.header
pattern: request
direction: incoming
order: 2
- subject: pipeline.logging
pattern: publish
direction: outgoing
order: 3
- subject: authentication
pattern: request
direction: incoming
order: 4
- subject: '*'
pattern: request
direction: incoming
order: 5
```

## Configuration
All configuration of the http-nats-proxy is done via environment variables.

| Environment Variable | Default Value | Description |
|--------------------------------------|---------------------------------|------------------------------------------|
| HTTP_NATS_PROXY_HOST_PORT | 5000 | The port that the http-nats-proxy will listen for incoming http requests on |
| HTTP_NATS_PROXY_NAT_URL | nats://localhost:4222 | The NATS url where the http-nats-proxy will send the NATS message to |
| HTTP_NATS_PROXY_WAIT_TIMEOUT_SECONDS | 10 | The number of seconds that the http-nats-proxy will wait for a response from the microservice backend before it returns a Timeout Error to the http client |
| HTTP_NATS_PROXY_HEAD_STATUS_CODE | 200 | The http status code that will be used for a successful HEAD request |
| HTTP_NATS_PROXY_PUT_STATUS_CODE | 201 | The http status code that will be used for a successful PUT request |
| HTTP_NATS_PROXY_GET_STATUS_CODE | 200 | The http status code that will be used for a successful GET request |
| HTTP_NATS_PROXY_PATCH_STATUS_CODE | 201 | The http status code that will be used for a successful PATCH request |
| HTTP_NATS_PROXY_POST_STATUS_CODE | 201 | The http status code that will be used for a successful POST request |
| HTTP_NATS_PROXY_DELETE_STATUS_CODE | 204 | The http status code that will be used for a successful DELETE request |
| HTTP_NATS_PROXY_CONTENT_TYPE | application/json; charset=utf-8 | The http response Content-Type header value. This should be set to whatever messaging format your microservice api supports (xml, json, etc) |
| HTTP_NATS_PROXY_METRICS_SUBJECT | | If set, this is the NATS subject that metrics will be published to |
| HTTP_NATS_PROXY_LOGS_SUBJECT | | If set, this is the NATS subject that logs will be published to |
| HTTP_NATS_PROXY_TRACE_HEADER | | If set, this is the http request header name that will be injected into each http request with a globally unique value for a trace id. If the http request header already exists then it will not be overwritten |
| Environment Variable | Default Value | Description |
|------------------------------------------|---------------------------------|------------------------------------------|
| HTTP_NATS_PROXY_HOST_PORT | 5000 | The port that the http-nats-proxy will listen for incoming http requests on |
| HTTP_NATS_PROXY_NAT_URL | nats://localhost:4222 | The NATS url where the http-nats-proxy will send the NATS message to |
| HTTP_NATS_PROXY_WAIT_TIMEOUT_SECONDS | 10 | The number of seconds that the http-nats-proxy will wait for a response from the microservice backend before it returns a Timeout Error to the http client |
| HTTP_NATS_PROXY_HEAD_STATUS_CODE | 200 | The http status code that will be used for a successful HEAD request |
| HTTP_NATS_PROXY_PUT_STATUS_CODE | 201 | The http status code that will be used for a successful PUT request |
| HTTP_NATS_PROXY_GET_STATUS_CODE | 200 | The http status code that will be used for a successful GET request |
| HTTP_NATS_PROXY_PATCH_STATUS_CODE | 201 | The http status code that will be used for a successful PATCH request |
| HTTP_NATS_PROXY_POST_STATUS_CODE | 201 | The http status code that will be used for a successful POST request |
| HTTP_NATS_PROXY_DELETE_STATUS_CODE | 204 | The http status code that will be used for a successful DELETE request |
| HTTP_NATS_PROXY_CONTENT_TYPE | application/json; charset=utf-8 | The http response Content-Type header value. This should be set to whatever messaging format your microservice api supports (xml, json, etc) |
| HTTP_NATS_PROXY_REQUEST_PIPELINE_CONFIG_FILE | | The full file path and name of the configuration file that specifies your request pipeline |



Expand Down
13 changes: 13 additions & 0 deletions src/AuthenticationPipelineStep/AuthenticationPipelineStep.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="NATS.Client" Version="0.8.0" />
<PackageReference Include="Newtonsoft.Json" Version="10.0.3" />
</ItemGroup>

</Project>
45 changes: 45 additions & 0 deletions src/AuthenticationPipelineStep/MessageHelper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
using NATS.Client;
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace AuthenticationPipelineStep
{
public class MessageHelper
{
private static readonly JsonSerializerSettings SerializerSettings = new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
};

public static T Deserialize<T>(string data)
{
return JsonConvert.DeserializeObject<T>(data);
}

public static NatsMessage GetNatsMessage(Msg msg)
{
return JsonConvert.DeserializeObject<NatsMessage>(Encoding.UTF8.GetString(msg.Data));
}

public static string GetValue(string key, IEnumerable<KeyValuePair<string, string>> parameters)
{
// force the enumeration of queryParms
var parameterList = parameters.ToList();

// try to find the matching key
var match = parameterList.FirstOrDefault(kv => kv.Key.Equals(key, StringComparison.OrdinalIgnoreCase));

// if we didn't find a match then just return an empty string
return string.IsNullOrWhiteSpace(match.Key) ? string.Empty : match.Value;
}

public static byte[] PackageResponse(object data)
{
return Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(data, SerializerSettings));
}
}
}
63 changes: 63 additions & 0 deletions src/AuthenticationPipelineStep/NatsMessage.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Text;

namespace AuthenticationPipelineStep
{
public sealed class NatsMessage
{
public string Body { get; set; }
public long CompletedOnUtc { get; set; }
public List<KeyValuePair<string, string>> Cookies { get; set; }
public string ErrorMessage { get; set; }
public long ExecutionTimeMs => CompletedOnUtc - StartedOnUtc;
public Dictionary<string, string> ExtendedProperties { get; set; }
public string Host { get; set; }
public List<KeyValuePair<string, string>> QueryParams { get; set; }
public List<KeyValuePair<string, string>> RequestHeaders { get; set; }
public string Response { get; set; }
public string ResponseContentType { get; set; }
public List<KeyValuePair<string, string>> ResponseHeaders { get; set; }
public int ResponseStatusCode { get; set; }
public bool ShouldTerminateRequest { get; set; }
public long StartedOnUtc { get; set; }
public string Subject { get; set; }

public NatsMessage(string host, string contentType)
{
// capture the time in epoch utc that this message was started
StartedOnUtc = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();

// default the response status code to an invalid value for comparison later on when the response is being processed by the RequestHandler
ResponseStatusCode = -1;

// capture the host machine that we're executing on
Host = host;

// capture the content type for the http response that we're configured for
ResponseContentType = contentType;

// initialize the default properties
Cookies = new List<KeyValuePair<string, string>>();
ExtendedProperties = new Dictionary<string, string>();
QueryParams = new List<KeyValuePair<string, string>>();
RequestHeaders = new List<KeyValuePair<string, string>>();
ResponseHeaders = new List<KeyValuePair<string, string>>();
}

public void MarkComplete()
{
CompletedOnUtc = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
}
}

internal static class NatsMessageExtensions
{
internal static byte[] ToBytes(this NatsMessage msg, JsonSerializerSettings serializerSettings)
{
var serializedMessage = JsonConvert.SerializeObject(msg, serializerSettings);
return Encoding.UTF8.GetBytes(serializedMessage);
}
}
}
63 changes: 63 additions & 0 deletions src/AuthenticationPipelineStep/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
using NATS.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;

namespace AuthenticationPipelineStep
{
internal class Program
{
// we use a mre to keep the console application running while it's waiting on messages from NATS in the background
private static readonly ManualResetEvent ManualResetEvent = new ManualResetEvent(false);

private static IConnection _connection;

private static void EnsureAuthHeaderPresent(object sender, MsgHandlerEventArgs e)
{
// deserialize the NATS message
var msg = MessageHelper.GetNatsMessage(e.Message);

// if the msg doesn't include an authentication header we need to return a redirect to the auth url
var authHeader = msg.RequestHeaders.FirstOrDefault(h => h.Key.Equals("Authorization"));

if (null == authHeader.Value)
{
Console.WriteLine($"No Authorization header found. Returning a 301 redirect to http://google.com");
msg.ResponseStatusCode = 301;
msg.ResponseHeaders.Add(new KeyValuePair<string, string>("Location", "https://google.com"));
msg.ShouldTerminateRequest = true;
}
else
{
Console.WriteLine($"Authorization header was already on the message so nothing to do.");
}

// send the NATS message back to the caller
_connection.Publish(e.Message.Reply, MessageHelper.PackageResponse(msg));
}

private static void Main(string[] args)
{
// configure the url to the NATS server
var natsUrl = Environment.GetEnvironmentVariable("HTTP_NATS_PROXY_NAT_URL") ?? "nats://localhost:4222";

// create a connection to the NATS server
var connectionFactory = new ConnectionFactory();
_connection = connectionFactory.CreateConnection(natsUrl);

// setup a subscription to the "authentication" queue using a queue group for this microservice
var subscriptions = new List<IAsyncSubscription>
{
_connection.SubscribeAsync("authentication", "authentication-microservice-group", EnsureAuthHeaderPresent),
};

// start the subscriptions
subscriptions.ForEach(s => s.Start());

// keep this console app running
Console.WriteLine($"Authentication Pipeline Step Connected to NATS at: {natsUrl}\r\nWaiting for messages...");
ManualResetEvent.WaitOne();
}
}
}
14 changes: 13 additions & 1 deletion src/HttpNatsProxy.sln
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 15
VisualStudioVersion = 15.0.27130.2024
VisualStudioVersion = 15.0.27130.2026
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Proxy", "Proxy\Proxy.csproj", "{B2256168-3B5A-4F93-8A75-8486A9F6C282}"
EndProject
Expand All @@ -11,6 +11,10 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "MetricsMicroservice", "Metr
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LoggingMicroservice", "LoggingMicroservice\LoggingMicroservice.csproj", "{9B92B68C-D617-4D8E-8B92-EC321E05BE1A}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TraceHeaderPipelineStep", "TraceHeaderPipelineStep\TraceHeaderPipelineStep.csproj", "{820BE8BB-D98D-4AAA-A15D-E88763CB39DE}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AuthenticationPipelineStep", "AuthenticationPipelineStep\AuthenticationPipelineStep.csproj", "{87124511-935E-483D-A170-D4D9806F7361}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand All @@ -33,6 +37,14 @@ Global
{9B92B68C-D617-4D8E-8B92-EC321E05BE1A}.Debug|Any CPU.Build.0 = Debug|Any CPU
{9B92B68C-D617-4D8E-8B92-EC321E05BE1A}.Release|Any CPU.ActiveCfg = Release|Any CPU
{9B92B68C-D617-4D8E-8B92-EC321E05BE1A}.Release|Any CPU.Build.0 = Release|Any CPU
{820BE8BB-D98D-4AAA-A15D-E88763CB39DE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{820BE8BB-D98D-4AAA-A15D-E88763CB39DE}.Debug|Any CPU.Build.0 = Debug|Any CPU
{820BE8BB-D98D-4AAA-A15D-E88763CB39DE}.Release|Any CPU.ActiveCfg = Release|Any CPU
{820BE8BB-D98D-4AAA-A15D-E88763CB39DE}.Release|Any CPU.Build.0 = Release|Any CPU
{87124511-935E-483D-A170-D4D9806F7361}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{87124511-935E-483D-A170-D4D9806F7361}.Debug|Any CPU.Build.0 = Debug|Any CPU
{87124511-935E-483D-A170-D4D9806F7361}.Release|Any CPU.ActiveCfg = Release|Any CPU
{87124511-935E-483D-A170-D4D9806F7361}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down
Loading

0 comments on commit da7e471

Please sign in to comment.