Feature: Make Request and Response types natively JSON Serializable

[will-jac]

Is your feature request related to a problem? Please describe.
In my typical use of Elasticsearch, I have an API Gateway sitting in front of Elasticsearch, ie:
[Service]-->[API Gateway]-->[Elasticsearch]

This gateway does some small amounts of business logic, routing requests to the right index, and custom authentication, but mostly is in the business of passing through requests/responses to/from Elasticsearch. It, and all consuming services, are C# ASP.Net services.

The challenge we run into is exposing strongly-typed request/response objects to the consuming services, because Elasticsearch classes are not natively JSON-serializable, meaning that the following example (building on the CRUD usage example) will raise an exception:

// Does not work because .Net will try to call JsonSerializer.Serialize<GetResponse<Tweet>>(response), which fails
[HttpGet("get")]
public async Task<GetResponse<Person>> GetExample()
{
	return await client.GetAsync<Tweet>(1, x => x.Index("my-tweet-index"));
}

On the side of the consuming service, we're also unable to use the Elasticsearch client, because our API Gateway isn't Elasticsearch and so the client will fail to connect and refuse to make requests (even if they would work if we pretended to be Elasticsearch). Since we can't easily JSON deserialize Elasticsearch response classes, this means consuming services don't get a strongly-typed response API. Similarly, because we can't easily JSON serialize request classes, consuming services don't get a strongly-typed request API.

Describe the solution you'd like
The Request and Response classes for Elasticsearch should be natively JSONSerializable so that an API Gateway can accept/emit them and consuming services can send them / build them from the response.

Decoupling the data classes from the logic of sending requests is generally best practice, and is the pattern most commonly seen in other C# client libraries we work with since it allows this re-usability.

Describe alternatives you've considered
We currently hack around this by getting the underlying Elastic serializer, eg:

var nodePool = new Elastic.Transport.SingleNodePool(new Uri("http://example.com"));
var settings = new ElasticsearchClientSettings(
	nodePool,
	sourceSerializer: (defaultSerializer, settings) =>
		new DefaultSourceSerializer(settings, configureOptions)
);
// This is not used, but required by the client. yes it's annoying.
// Ideally we could strip it from the request, but the serializer complains if we don't set it.
settings.DefaultIndex("UNUSED");
var e = new ElasticsearchClient(settings);
_config = e.Transport.Configuration;
_serializer = e.RequestResponseSerializer;
...
public async Task<CreateResponse> CreateAsync<TDocument>(string index, string id, TDocument body)
{
	string url = "elasticsearch_url/v1/_doc/id";
	HttpResponseMessage response = await _httpClient.PostAsJsonAsync(url, body!);
	response.EnsureSuccessStatusCode();
	using Stream streamResponse = await response.Content.ReadAsStreamAsync();
	return await _serializer.DeserializeAsync<CreateResponse>(streamResponse) ?? throw new InvalidOperationException("Null response");
}

This isn't great--we can't set the transport information on the CreateResponse (it's internal set), so all validation has to be done in the CreateAsync function. It also means every request has an extra "indexes" parameter that we have to strip out our gateway, because despite the property being optional, the serializer will throw an exception if it's unset.

Additional context
This request corresponds to support case 01933280.

[flobernd]

Hi @will-jac ,

I understand your demand for this feature, but your usecase is very specific. The .NET client was never designed to be used in a gateway/proxy.

Please also have a look at the corresponding documentation:
https://www.elastic.co/docs/reference/elasticsearch/clients/dotnet/request-response-serialization

We strongly advise against persisting serialized Elasticsearch types, as their structure may vary between different versions of the client.

Instead, you should create your own POCO type and map/assign the desired properties to it.

In your case I would probably create POCO/DTO types for all the requests/responses you actually need and (manually) map all the properties that you use. I understand that this is a good amount of work, but you'll end up with a reliable solution.

This might be helpful as well:

You don't have to manually send requests using the HttpClient. Instead, you could use the low level transport as documented here:
https://www.elastic.co/docs/reference/elasticsearch/clients/dotnet/transport