Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: handle aux file keys in imports #8202

Closed
skyzh opened this issue Jun 28, 2024 · 3 comments · Fixed by #8316
Closed

pageserver: handle aux file keys in imports #8202

skyzh opened this issue Jun 28, 2024 · 3 comments · Fixed by #8316
Assignees
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged

Comments

@skyzh
Copy link
Member

skyzh commented Jun 28, 2024

currently it does not work with the page server import functionality https://neondb.slack.com/archives/C033RQ5SPDH/p1719553893686989?thread_ts=1719403793.586119&cid=C033RQ5SPDH

We need some logic to auto-detect if there exists v2 aux file keys during the import. If yes, we should put last_aux_file_policy = v2 in the index_part file; or forcing v2 for all imported tenants.

@skyzh skyzh added t/bug Issue Type: Bug c/storage/pageserver Component: storage: pageserver labels Jun 28, 2024
@skyzh skyzh self-assigned this Jun 28, 2024
@jcsp
Copy link
Collaborator

jcsp commented Jul 3, 2024

or forcing v2 for all imported tenants.

That sounds fine to me. The import API isn't used in prod under normal circumstances, and if it doesn't work for people with weird beta-feature stuff in their databases that's fine.

@jcsp jcsp changed the title handle aux file imports pageserver: handle aux file keys in imports Jul 3, 2024
@knizhnik
Copy link
Contributor

knizhnik commented Jul 4, 2024

Please notice that right now import_file just ignores all AUX files (replication slots,...).
So if we are importing data from tar, then this files are missed.
It was fixed in #6560

@jcsp
Copy link
Collaborator

jcsp commented Jul 4, 2024

We can also consider removing this API, as it is not used in production. Last time we discussed removing it there was concern that it was used in some support cases by the compute team.

@jcsp jcsp added the triaged bugs that were already triaged label Jul 4, 2024
@skyzh skyzh closed this as completed in df3dc6e Jul 8, 2024
skyzh added a commit that referenced this issue Jul 15, 2024
close #8202 ref
#6560

For tenant imports, we now write the aux files into both v1+v2 storage,
so that the test case can pick either one for testing. Given the API is
only used for testing, this looks like a safe change.

Signed-off-by: Alex Chi Z <chi@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug triaged bugs that were already triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants