TRIAS endpoint is blocking certain User-Agents

[1Maxnet1]

While working with the endpoint I discovered that it blocks certain user-agent headers, as described here: https://python.tutorialink.com/post-request-to-trias-api-does-not-work-with-requests/

The solution for me was just to set the user-agent manually to a working string, but it was hard to find out about this issue in the first place.

This seems to be a very weird issue and is rarely documented anywhere. However it would might make sense to document it, as else the requests will fail with 502 Error code, with no obvious reason. I contacted a contact at nvbw regarding this issue, but I'm lacking an answer for now, so this discussion forum seemed to be a good place to me.

If anyone has any thoughts on that, or did encounter this issue as well, I would appreciate some opinions :)

[derhuerst]

While working with the endpoint I discovered that it blocks certain user-agent headers […].

Public APIs commonly block user agents like curl's or Python's request's, in order to prevent poorly/quickly written scripts from causing excessive amounts of traffic. I think there's a point in doing that, but there are much more elegant and effective means to a) prevent "rude" clients from causing too much load and b) not bother legitimate clients too much:

Enforce a rather strict rate-limit, unless the client sends a User-Agent generated by the API provider (e.g. an individual API token, or generate one from a secret); This allows a) identifying individual clients and b) only allowing clients of devs who have actually visited the documentation page once. BART has been doing this with their (now-legacy) API for several years.

The solution for me was just to set the user-agent manually to a working string, but it was hard to find out about this issue in the first place.

I agree, this behaviour should be documented!

[…] as else the requests will fail with 502 Error code, with no obvious reason.

A more appropriate and intuitive status code would be 400 "bad request" (with a more detailed error message in the body), and 429 "too many requests" if the client's limit has been reached.

[derhuerst]

I contacted a contact at nvbw regarding this issue, but I'm lacking an answer for now […].

Maybe @NVBWSeifert can take care of this?

[NVBWSeifert]

I can confirm, that we have a problem, when the user agent is "python- requests/2.27.1". In this case, we found out, that somethimes requests will not served correctly.
Our earlier invesigation was not successfully, where the problem occured. Probably, our load balancer could make the problem, but we didn't get a good answer about it.

We make another try to find out, where the problem is and hopefully we can solve the problem.

Should we open an Issue for this?
Is this discussion area the right position, where other consumers look for such problems or should we active the wiki and and add a page for this?

Greetings,
Dietmar Seifert, NVBW

[derhuerst]

Should we open an Issue for this? Is this discussion area the right position, where other consumers look for such problems or should we active the wiki and and add a page for this?

My personal opinion: I think it depends on how much time is needed to resolve this: If it will take a long time, consider creating an Issue (that links to this discussion) in order to make the problem more visible; If resolving this won't take that long, I guess this discussion is google-able enough.

[NVBWSeifert]

we have two external systems, it will take some weeks, i think so.

[jendrikw]

any updates?