Make allow-directory-listing and provide-index-page compatible

[johanvdw]

I wonder if it is somehow possible to have directory listings in case the index.html page does not exist. I realise that in that case two requests to s3 have to be made, either first the page and then the listing if that fails, or first the listing and then the page if it is in the list.

[dekobon]

Hi @johanvdw, even if we wanted to do two requests to s3, I don't think we can do it with the current implementation of nginx and njs. If we make a s3 gateway binary module, then it may be possible.

[ajschmidt8]

@dekobon, I also have a use case where I need both ALLOW_DIRECTORY_LIST and PROVIDE_INDEX_PAGE to be enabled simultaneously.

Is there any way we could make this work?

I'm happy to open a PR if we can agree on an approach.

[dekobon]

There may be a way now. Njs now supports XML

So, you could parse the output of the directory list results and determine if there are no results. Basically, you would rewrite the logic here.

Then, one may could use fetch to get the index page. When doing that, you may actually want to fetch the index page by querying the S3 gateway itself over localhost so that the aws auth creds are embedded.

[ajschmidt8]

Ok. I will try to take a look at this when time permits.

Please feel free to leave additional feedback on the approach if anything comes to mind.

In the meantime, can you re-open this issue, @dekobon?

[AyodeAwe]

Hi @dekobon, there might be an issue with the mentioned approach. I found the following in the docs:

As the js_body_filter handler returns its result immediately, it supports only synchronous operations. Thus, asynchronous operations such as r.subrequest() or setTimeout() are not supported.

It is not immediately clear how the fetch (being an async function) should be implemented within the logic here given this limitation.

Could you provide additional feedback?

[dekobon]

I wonder if you could modify it to a redirect of the location of the index page?

[AyodeAwe]

Hmm, seems like r.internalRedirect simply gets ignored when called in js_body_filter. I created an issue in the njs repo describing this.

nginx/njs#645

[ajschmidt8]

@dekobon, it looks like @AyodeAwe has opened a PR for this #139.

Can you take a look when you have a moment?

He uses a combination of js_content and ngx.fetch to see if an index.html page exists for a specific directory. If it does, he loads it. Otherwise, he loads the directory listing.

[dekobon]

Please see my comments on the PR.

[ajschmidt8]

This issue can be closed now that #139 is merged.

Though we will likely open a subsequent PR to switch the following lines to use a HEAD request once this njs issue is resolved:

nginx-s3-gateway/common/etc/nginx/include/s3gateway.js

Lines 357 to 359 in 309053c

	let reply = await ngx.fetch(
	`http://127.0.0.1:80${url}`
	);

[dekobon]

Thank you @ajschmidt8 !

Resolved in PR #139