Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor compute dockerfile #10371

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

hlinnaka
Copy link
Contributor

Refactor how extensions are built in compute Dockerfile

  1. Rename some of the extension layers, so that names correspond more
    precisely to the upstream repository name and the source directory
    name. For example, instead of "pg-jsonschema-pg-build", spell it
    "pg_jsonschema-build". Some of the layer names had the extra "pg-"
    part, and some didn't; harmonize on not having it. And use an
    underscore if the upstream project name uses an underscore.

  2. Each extension now consists of two dockerfile targets:
    [extension]-src and [extension]-build. By convention, the -src
    target downloads the sources and applies any neon-specific patches
    if necessary. The source tarball is downloaded and extracted under
    /ext-src. For example, the 'pgvector' extension creates the
    following files and directory:

     /ext-src/pgvector.tar.gz  # original tarball
     /ext-src/pgvector.patch   # neon-specific patch, copied from patches/ dir
     /ext-src/pgvector-src/    # extracted tarball, with patch applied
    

    This separation avoids re-downloading the sources every time the
    extension is recompiled. The 'extension-tests' target also uses the
    [extension]-src layers, by copying the /ext-src/ dirs from all
    the extensions together into one image.

This refactoring came about when I was experimenting with different
ways of splitting up the Dockerfile so that each extension would be in
a separate file. That's not part of this PR yet, but this is a good
step in modularizing the extensions.

@hlinnaka hlinnaka changed the base branch from main to compute-image-set-env January 13, 2025 15:09
Copy link

github-actions bot commented Jan 13, 2025

7370 tests run: 6984 passed, 0 failed, 386 skipped (full report)


Flaky tests (7)

Postgres 17

Postgres 16

Postgres 14

Code coverage* (full report)

  • functions: 33.5% (8495 of 25334 functions)
  • lines: 49.4% (71405 of 144677 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
a020d37 at 2025-01-22T17:23:04.304Z :recycle:

Base automatically changed from compute-image-set-env to main January 14, 2025 17:03
@hlinnaka hlinnaka force-pushed the refactor-compute-dockerfile branch 8 times, most recently from fbe2a18 to ef9d7d3 Compare January 21, 2025 03:56
@hlinnaka hlinnaka marked this pull request as ready for review January 21, 2025 03:56
@hlinnaka hlinnaka requested review from a team as code owners January 21, 2025 03:56
@hlinnaka hlinnaka requested review from knizhnik, lubennikovaav and tristan957 and removed request for knizhnik January 21, 2025 03:56
compute/compute-node.Dockerfile Outdated Show resolved Hide resolved
compute/compute-node.Dockerfile Show resolved Hide resolved
compute/compute-node.Dockerfile Outdated Show resolved Hide resolved
@hlinnaka hlinnaka force-pushed the refactor-compute-dockerfile branch from ef9d7d3 to 3831a80 Compare January 21, 2025 22:18
1. Rename some of the extension layers, so that names correspond more
   precisely to the upstream repository name and the source directory
   name. For example, instead of "pg-jsonschema-pg-build", spell it
   "pg_jsonschema-build". Some of the layer names had the extra "pg-"
   part, and some didn't; harmonize on not having it. And use an
   underscore if the upstream project name uses an underscore.

2. Each extension now consists of two dockerfile targets:
   [extension]-src and [extension]-build. By convention, the -src
   target downloads the sources and applies any neon-specific patches
   if necessary. The source tarball is downloaded and extracted under
   /ext-src. For example, the 'pgvector' extension creates the
   following files and directory:

    /ext-src/pgvector.tar.gz  # original tarball
    /ext-src/pgvector.patch   # neon-specific patch, copied from patches/ dir
    /ext-src/pgvector-src/    # extracted tarball, with patch applied

    This separation avoids re-downloading the sources every time the
    extension is recompiled. The 'extension-tests' target also uses the
    [extension]-src layers, by copying the /ext-src/ dirs from all
    the extensions together into one image.

This refactoring came about when I was experimenting with different
ways of splitting up the Dockerfile so that each extension would be in
a separate file. That's not part of this PR yet, but this is a good
step in modularizing the extensions.
That was removed in 'main' already, but got reintroduced when I
resolve merge conflicts.
@hlinnaka hlinnaka force-pushed the refactor-compute-dockerfile branch from 8a5953c to a020d37 Compare January 22, 2025 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants