Supporting subdirectory path

[mikiher]

During a recent discussion on the reasons for moving to SSR, issue #385 came as one of the reasons.

Background

In a nutshell, issue #385 requests an ability to have a server that supports reverse-proxied subdirectory requests, i.e. supporting external requests like https://my.domain.com/audiobokshelf instead of https://audiobookshelf.my.domain.com. The latter (aka subdomain proxy) is the standard and recommended reverse-proxy setup for Audiobookshelf, but the former (aka as subdirectory proxy) is not supported.

The main difference between the two, which currently makes supporting subdirectory proxy harder, is that when you use a subdomain proxy, it doesn't affect server routes, but with a subdirectory proxy, the server needs to account for the subdirectory when generating its responses - all the paths in URLs that are referenced in the HTML and Javscript rendered by the server need to account for that subdirectory. In Audiobookshelf, the problem is compounded by the fact that we statically serve the web client (aka SSG) - in an SSR framework, all the paths can be generated dynamically at runtime, but we don't currently support SSR.

In a discussion on Discord, I suggested that since the server and the web client already seemed to support subdirectory serving through the ROUTER_BASE_PATH environment variable, then a workaround for those who need a subdirectory proxy would be:

• Build a server and a nuxt client that have some non-empty static ROUTER_BASE_PATH (e.g /audiobookshelf)
• Configure their reverse proxy to proxy https://my.external.domain.com/audiobookshelf to http://<server-address>:<server-port>/audiobookshelf

And with that, they'd have a working workaround solution.

I also noted that we could build and provide a separate docker image with ROUTER_BASE_PATH=/audiobookshelf for those who needed it.

@advplyr said he was OK with implementing the workaround, but he asked that we try to come up with a solution that does not require building and maintaining an additional docker (and debian) image.

Objectives

So based on these, here's a summary of the objectives:

• Provide a solution that allows Audiobookshelf to be served from a subdirectory
• Continue maintaining just a single docker/debian image
• Serve existing mobile and third-party clients seamlessly (i.e. without requiring them to change their code or server URLs)
• The solution should not require SSR

Suggested solution

The suggestion, in a nutshell, is to switch the server to always use a static non-empty ROUTER_BASE_PATH /audiobookshelf.

1. To support subdirectory access (https://my.domain.com/audiobookshelf):

   • We need to make sure that both the server and web client fully support the ROUTER_BASE_PATH environment variable.
     • This was already mostly the case, with a few additional fixes applied in PRs Fixes and cleanup for subdirectory serving support #3521 and Add server proxies for all server paths #3530.
   • We need the mobile app to support accessing the server with a subdirectory path (e.g. https://my.external.domain.com/audiobookshelf).
     • Currently working on that. This mostly already works, except for a few required fixes, including:
       • Server address validation
       • Modifications for cover images URLs and audio file URLs
       • Modification of the web socket connection request path

2. To support non-subdirectory access (https://audiobookshelf.my.domain.com, for existing mobile and third-party clients):

   • We need to make sure that the server can still be accessed without a subdirectory path (e.g. https://audiobookshelf.my.domain.com).
     • All request paths not starting with /audiobookshelf should be rewritten to start with /audiobookshelf
       • This is performed by a small Express middleware, implemented in PR Fixes and cleanup for subdirectory serving support #3521.
     • In addition to supporting a socket.io server at /audiobookshelf/socket.io (already implemented in Fixes and cleanup for subdirectory serving support #3521), we need to support it at /socket.io as well (for existing clients).
       • Currently working on that.

3. The docker image and the debian package will be updated to always set ROUTER_BASE_PATH to /audiobookshelf.

Implications:

• The web client will always be served from the subdirectory path. When accessing through a non-subdirectory path, Nuxt will redirect to the subdirectory path.
• The server will always internally be accessed through the subdirectory path. When accessing through a non-subdirectory path, the server will rewrite the request path to start with the subdirectory path.
• One hidden implication is that there can never be an /audiobookshelf entrypoint on the server, as it would conflict with the subdirectory path. This doesn't seem to be a problem, as we don't currently have any such entrypoint.

[advplyr]

Just to clarify, if this is implemented and I'm using a subdomain say https://audiobookshelf.mydomain.com

and I navigate to the https://audiobookshelf.mydomain.com/login.

Nuxt client will redirect me to https://audiobookshelf.mydomain.com/audiobookshelf/login?

[mikiher]

I want to be very precise about my answer:

Case 1:
If by "I navigate to" you mean "I go to the address bar and type" https://audiobookshelf.mydomain.com/login then what happens is:

• The small app middleware I added (see below), modifes the request path from /login to /audiobookshelf/login

   if (global.RouterBasePath) {
     app.use((req, res, next) => {
       if (!req.url.startsWith(global.RouterBasePath)) {
         req.url = `${global.RouterBasePath}${req.url}`
       }
       next()
     })
   }
   app.use(global.RouterBasePath, router)

• The last line above will strip /audiobookshelf from the path and pass on /login to the main router.
• In the main router, this is speciifically handled by the express.static() middleware, which will (in this very specific case) generate a 301 redirect to /login/
• The /login/ request will be handled in the same manner as above and will serve client/dist/login/index.html from express.static()

So unless express.static() logic decides to redirect (as in this case), there will be no actual http redirect involved, just path rewrites.

Case 2:
If by "I navigate to" you mean "I click on some link within the nuxt app", then of course there's no redirect involved, since Nuxt is a PWA and handles all routing internally.

So the bottom line answer is that either way there's usually no actual HTTP redirect involved. just path rewrites. So when I said "Nuxt will redirect to the subdirectory path." it was mostly a figure of speech, not a precise statement.

[advplyr]

I think this is a good solution.

One of the unknowns I have so far is what will need to be updated for the OIDC authentication. @Sapd put all that logic together but I can see the redirect URIs would need to be updated

audiobookshelf/server/Auth.js

Lines 714 to 716 in d80752c

	// TODO: ABS does currently not support subfolders for installation
	// If we want to support it we need to include a config for the serverurl
	postLogoutRedirectUri = `${protocol}://${host}/login`

Also, when creating RSS feeds we store the full server URL. In the feeds and feedEpisodes table. In hindsight I think that could have been built without storing full URLs. The recent media item shares feature was built without storing URLs.
If a user chooses to migrate from using a subdomain to using a subdirectory these URLs would need to be updated.

[Sapd]

I need to think if there's a way of not breaking it when the switch happens. I don't immediately see one...

Why not just add the env variable to the code listed above? Or is it not empty when the user does not use a subdirectory?

When the user switches his app to a subpath it is clear anyway that is has to adapt his identity provider settings (like googles)

[mikiher]

Why not just add the env variable to the code listed above? Or is it not empty when the user does not use a subdirectory?

Yes, the intention is to add the env variable, but the env variable is not set based on the requests - it is static throughout the server's life.

The objective in this design is to switch everybody to using the subpath without requiring existing clients to make any changes.

Although the OIDC service isn't a client per-se, I'd prefer it if I can find a way to make it work without telling OIDC admins to change their existing Audiobookshelf configuration when I release this.

Perhaps I have a solution for this, I need to think about it a little bit more.

[Sapd]

The objective in this design is to switch everybody to using the subpath without requiring existing clients to make any changes.

Ah Ive re-read your proposal. I just skimmed through it at first (focusing on the oidc question) and did not realise you want to force the subdirectory.
Thats something some users probably will not like (including myself tbh). Even for Jellyfin there exists some reverse proxy rules which try to get rid of forced sub-paths (index.html#path). I think something like that should be more handled on the reverse proxy side (which both nginx and traefik can do well, and is a standard for most apps). Basically the apps should make sure that links are prefixed correctly, and the reverse-proxy that the correct routes are called.

In any case if you force a subdirectory, the users will have to reconfigure on the identity provider side. The only exception being when you still provide an alias route domain/openid/callback of domain/audiobookshelf/openid/callback. But you also need to adapt then redirect_uri in this line based on the request:

audiobookshelf/server/Auth.js

Line 472 in d80752c

oidcStrategy._params.redirect_uri = redirectUri

As the redirect_uri is also validated on audiobookshelf-side by the openid library.
But providing that work-around also is somehow weird and complicated the logic.

[mikiher]

Thats something some users probably will not like (including myself tbh).

Why would it matter to you if you don't need to change anything for it to work? Is it just a matter of style?

Again, one of the main objectives in the design is that existing mobile and 3rd-party clients don't have to change at all, so from the client's perspective, the forced subpath isn't even visible. It's only visible on the web client, and then only if you watch at the URL line.

Edit: It might be even possible to change the implementation so that even the web client address bar URLs do not change, assuming you didn't use the subpath to navigate to it. If that's very important, I can give it some more thought

And this would allow us to resolve #385, which has been voted for by many users - and it is not possible to resolve it without supporting a subpath on the server.

But you also need to adapt then redirect_uri in this line based on the request

That was the general direction of the solution I was thinking about.

[mikiher]

Thanks for having this disucussion. It gave me food for thought.

[mikiher]

Thank you both for you comments. I'll address them in upcoming PRs.

[devnoname120]

@mikiher continuing our discussion from #3958

than setting the standard <base> HTML element at runtime?

I haven't considered using <base>, but note that our code doesn't currently support server side rendering - I can only serve static html and Javascript. Also, there are other things beside html links that need to be considered for subdirectory support (Javascript fetch requests etc.).

It's not pretty but a workaround for Nuxt 2 is detailed here: nuxt/nuxt#7287

Alternatively, this approach could work but it's kinda annoying: https://github.com/undone-labs/nuxt-module-ipfs/blob/master/plugin.js

Note that Nuxt 3 supports it natively: nuxt/nuxt#9170 (comment)
Audiobookshelf currently uses Nuxt 2 though. I'm not sure how much effort it would require to migrate to Nuxt 3.

[mikelococo]

There has been a lot of confusion in Discord and users thinking it is a breaking change when it isn't for the vast majority of users.
The only breaking change that we didn't account for was users running custom middleware and that was only 2 users so far that we know of.

The real cost of this change isn't the 2 users that were confirmed to be broken by the change, it's the increased debugging complexity in all the cases like #3913 and all the people that give up on debugging solvable problems because they can't understand how requests are supposed to be routed. People are confused because the request routing logic has become confusing, and debugging every proxy and request-routing problem must now engage with that complexity even in (especially in, given that the subpath urls are now canonical) cases where people aren't trying to use subpaths.

I don't think it is worth anymore time investment into this.

The investment will come in the form of support requests if it doesn't come in the form of simplifying the default routing architecture.

If the primary blocker preventing proper configurable subpaths (defaulting to off) is that Nuxt2 needs a client rebuild, has consideration been given to having the Docker entrypoint do the client build on container startup? Doing so is a bit of a hack and might slow container startup, but a hack like that has a much smaller blast radius than these routing changes and is trivial to unwind once Nuxt2 is gone.

[devnoname120]

If the primary blocker preventing proper configurable subpaths (defaulting to off) is that Nuxt2 needs a client rebuild, has consideration been given to having the Docker entrypoint do the client build on container startup?

Hmmm this is a pretty good point. What would be the warmup perf impact exactly? Could you measure it?

[advplyr]

Other than #3913 lining up with the change I don't see any evidence that the user was impacted by the change. It is almost a month now with at least 100k active users and so far 2 users have shared specific examples of custom middleware they had to modify because of it.
So if you can share a specific example that would be helpful.

[mikelococo]

I don't understand why unfixably broken proxy configs are the only tradeoff worth paying attention to. Changing the UX around URLs itself has a cost in user confusion and irritation, as evidenced by the bug reports and discord dialogue. Complex routing schemes have a debugging cost, even when they aren't the root cause of problems they make it harder to find the root cause. Shipping redirect bugs to users that never wanted subpaths in the first place has a cost. That the routing scheme nominally works does not mean that it's without tradeoffs.

But even granting that broken proxy configs are the only tradeoff that matters, there are other solutions with that attribute:

1. Deferring shipping this feature until Nuxt2 was retired also wouldn't have broken proxy configs.
2. Shipping docker container flavors for subpathless and subpathed installs won't break proxy configs.
3. Rebuilding the client in the Docker container entrypoint won't break proxy configs.
4. Documenting the process of modifying and removing the basepath by rebuilding the client won't break proxy configs.

This discussion is titled "Supporting subdirectory path", but the feature that shipped was "Remove support for root paths". There's mounting evidence that people value the feature that was removed, irrespective of the number of broken proxy configs. Given the angst and confusion its generated in the community, I struggle to imagine what benefits this approach offers over alternatives that preserve proper root paths... and this feels doubly true if the frontend framework roadmap provides a way to eventually unwind any hacks required to maintain root-path support in the interim.

[nichwall]

I don't understand why unfixably broken proxy configs are the only tradeoff worth paying attention to. Changing the UX around URLs itself has a cost in user confusion and irritation, as evidenced by the bug reports and discord dialogue. Complex routing schemes have a debugging cost, even when they aren't the root cause of problems they make it harder to find the root cause. Shipping redirect bugs to users that never wanted subpaths in the first place has a cost. That the routing scheme nominally works does not mean that it's without tradeoffs.

This discussion is titled "Supporting subdirectory path", but the feature that shipped was "Remove support for root paths". There's mounting evidence that people value the feature that was removed, irrespective of the number of broken proxy configs. Given the angst and confusion its generated in the community, I struggle to imagine what benefits this approach offers over alternatives that preserve proper root paths... and this feels doubly true if the frontend framework roadmap provides a way to eventually unwind any hacks required to maintain root-path support in the interim.

As has been mentioned already, yes we agree that this could have been communicated better. We also recognize that it "feels weird".

Personally, this issue feels like it has been blown out of proportion by a very vocal minority of people thinking it affects them (in a way that requires changes), but so far none of them have been running custom middleware or are otherwise affected by it beyond incorrectly perceiving required changes on their end (other than the instances already mentioned). Any bugs found can have an issue opened the same way other bugs are reported.

You keep saying how it breaks existing reverse proxy configurations for everyone, but we still don't have specific examples of this other than custom middleware implementations breaking (as advplyr mentioned in those 2 cases). Intercepting web client traffic also isn't really a supported use case for general users because everything should be accessible through the API. If something is not accessible through the API, then that is an issue that should be resolved so the API provides that information and is separate from having a forced subdirectory.

With that said, we are looking at removing the forced subdirectory, potentially by bundling multiple pre-generated clients at different subpaths (and a root path). However, it will probably still be a bit due to needing to verify it works across install methods (docker, package repo, Windows, and running from source)