Yarn's bundled node-gyp broken

[wyattjoh]

The current version of yarn still has a pre-existing issue with the pre-bundled version of node-gyp related to yarnpkg/yarn#2266

A temporary fix relates to adding:

RUN yarn global add node-gyp

To our Dockerfile.

[wyattjoh]

You can refer to: https://circleci.com/gh/coralproject/talk/1921 with details of why the build failed.

[LaurentGoderre]

To me it failed only because the build was trying to override the included yarn. I don't see any node-gyp error.

[LaurentGoderre]

As you can see, several other build because of trying to install yarn via npm: #337 (comment)

[wyattjoh]

I linked the wrong build logs :( below is the relevant error:

Step 9/11 : RUN yarn install --production
 ---> Running in c936d65bcc45
yarn install v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
warning fsevents@1.0.17: The platform "linux" is incompatible with this module.
info "fsevents@1.0.17" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /usr/src/app/node_modules/bcrypt: Command failed.
Exit code: 127
Command: sh
Arguments: -c node-gyp rebuild
Directory: /usr/src/app/node_modules/bcrypt
Output:
sh: 1: node-gyp: not found
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
The command '/bin/sh -c yarn install --production' returned a non-zero code: 1

Which was ran with the repo here https://github.com/coralproject/talk/tree/2c98aa20977155d09b08e927c574be1221d2d707 where I had removed my patch from the Dockerfile to make it the following:

FROM node:7.6

# Add node-gyp for bcrypt build support
# RUN yarn global add node-gyp

# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Setup the environment
ENV NODE_ENV production
ENV PATH /usr/src/app/bin:$PATH
ENV TALK_PORT 5000
EXPOSE 5000

# Install app dependencies
COPY package.json yarn.lock /usr/src/app/
RUN yarn install --production

# Bundle app source
COPY . /usr/src/app

CMD [ "yarn", "start" ]

[LaurentGoderre]

Node-gyp is not included in node though so you would always need to install it no?

[wyattjoh]

Previously, using npm install it successfully installs.

[LaurentGoderre]

It doesn't install with yarn?

[wyattjoh]

That's correct. With the following Dockerfile:

FROM node:7.6

# Add node-gyp for bcrypt build support
# RUN yarn global add node-gyp

# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Setup the environment
ENV NODE_ENV production
ENV PATH /usr/src/app/bin:$PATH
ENV TALK_PORT 5000
EXPOSE 5000

# Install app dependencies
COPY package.json yarn.lock /usr/src/app/
# RUN yarn install --production
RUN npm install --production

# Bundle app source
COPY . /usr/src/app

CMD [ "npm", "start" ]

I got no errors during install.

[LaurentGoderre]

That's weird...I ran the same thing and it worked:

Sending build context to Docker daemon 2.048 kB
Step 1/2 : FROM node:7.6
7.6: Pulling from library/node
693502eb7dfb: Pull complete 
081cd4bfd521: Pull complete 
5d2dc01312f3: Pull complete 
54a5f7da9a4f: Pull complete 
f6ebc0704397: Pull complete 
cd487f72ac9f: Pull complete 
d3e88bf53e70: Pull complete 
88dbf525e469: Pull complete 
Digest: sha256:34d21bab49105c1ab384e50432aa6ebd6dac69c2637378e7ecb6f40cda97118b
Status: Downloaded newer image for node:7.6
 ---> 7b74673e3e78
Step 2/2 : RUN yarn global add node-gyp
 ---> Running in 58721d12a3bd
yarn global v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "node-gyp@3.5.0" with binaries:
      - node-gyp
Done in 5.15s.
 ---> d383b79b9c88
Removing intermediate container 58721d12a3bd
Successfully built d383b79b9c88

[LaurentGoderre]

Is it possible that node-gyp installs but isn't working?

[wyattjoh]

This is my issue:

When installing via yarn, I need to run:

yarn global add node-gyp
yarn install --production

While with npm I can run:

npm install --production

I don't need to explicitly install node-gyp, hence why it seems that the upstream issue with yarn (yarnpkg/yarn#2266) might be causing the issue now where when installed via yarn (specifically, the bundled version), that node-gyp fails.

[wyattjoh]

Notice that in your example, you explicitly ran

yarn global add node-gyp

As the second step.

[LaurentGoderre]

This seems to be similar to your issue: yarnpkg/yarn#1848

[wyattjoh]

I can confirm that after running:

yarn upgrade bcrypt

Which resulted in the following changes:

diff --git a/package.json b/package.json
index 740063dd..eaa15303 100644
--- a/package.json
+++ b/package.json
@@ -49,7 +49,7 @@
   },
   "homepage": "https://github.com/coralproject/talk#readme",
   "dependencies": {
-    "bcrypt": "^0.8.7",
+    "bcrypt": "^1.0.2",
     "body-parser": "^1.15.2",
     "cli-table": "^0.3.1",
     "commander": "^2.9.0",
diff --git a/yarn.lock b/yarn.lock
index 0331202e..f8a49205 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -1089,12 +1089,13 @@ bcrypt-pbkdf@^1.0.0:
   dependencies:
     tweetnacl "^0.14.3"
 
-bcrypt@^0.8.7:
-  version "0.8.7"
-  resolved "https://registry.yarnpkg.com/bcrypt/-/bcrypt-0.8.7.tgz#bc3875a9afd0a7b2cd231a6a7f218a5ce156b093"
+bcrypt@^1.0.2:
+  version "1.0.2"
+  resolved "https://registry.yarnpkg.com/bcrypt/-/bcrypt-1.0.2.tgz#d05fc5d223173e0e28ec381c0f00cc25ffaf2736"
   dependencies:
     bindings "1.2.1"
-    nan "2.3.5"
+    nan "2.5.0"
+    node-pre-gyp "0.6.32"
 
 big.js@^3.1.3:
   version "3.1.3"
@@ -5006,7 +5007,11 @@ mute-stream@0.0.7:
   version "0.0.7"
   resolved "https://registry.yarnpkg.com/mute-stream/-/mute-stream-0.0.7.tgz#3075ce93bc21b8fab43e1bc4da7e8115ed1e7bab"
 
-nan@2.3.5, nan@^2.3.0:
+nan@2.5.0:
+  version "2.5.0"
+  resolved "https://registry.yarnpkg.com/nan/-/nan-2.5.0.tgz#aa8f1e34531d807e9e27755b234b4a6ec0c152a8"
+
+nan@^2.3.0:
   version "2.3.5"
   resolved "https://registry.yarnpkg.com/nan/-/nan-2.3.5.tgz#822a0dc266290ce4cd3a12282ca3e7e364668a08"
 
@@ -5111,7 +5116,7 @@ node-libs-browser@^2.0.0:
     util "^0.10.3"
     vm-browserify "0.0.4"
 
-node-pre-gyp@^0.6.29:
+node-pre-gyp@0.6.32, node-pre-gyp@^0.6.29:
   version "0.6.32"
   resolved "https://registry.yarnpkg.com/node-pre-gyp/-/node-pre-gyp-0.6.32.tgz#fc452b376e7319b3d255f5f34853ef6fd8fe1fd5"
   dependencies:

It was successfully able to build! I'm still hesitant as the fact that the same package.json + yarn.lock file can build the project successfully without explicitly installing a package (node-gyp) yet yarn required some convincing to get it to run.

To me it still feels like there is an issue that yarn should sort out upstream to match the ability that node currently has to build these dependancies properly.

I'll close this for now, but feel free to ask any questions if anyone wants to explore this topic more.

[LaurentGoderre]

I just tried switching to bcrypt to bcryptjs and it worked, there still is the warning about fsevents though.

[LaurentGoderre]

It is indeed weird. I isolated the problem

This works:

FROM node:7.6

RUN npm install bcrypt@0.8.7

This doesn't:

FROM node:7.6

RUN yarn add bcrypt@0.8.7

[wyattjoh]

That would be my conclusion as well. The fact that 0.8.7 did not include "node-pre-gyp": "0.6.32" as a dep is why 1.0.2 worked. This issue is still persisting then.

[terribleplan]

So when building a docker image descended from node:6 everything is really messed up:

Step 6/7 : RUN yarn
 ---> Running in f1c656e5fc13
yarn install v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[4/4] Building fresh packages...
error /usr/src/app/node_modules/scrypt: Command failed.
Exit code: 127
Command: sh
Arguments: -c node-gyp rebuild
Directory: /usr/src/app/node_modules/scrypt
Output:
sh: 1: node-gyp: not found
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
ERROR: Service 'web' failed to build: The command '/bin/sh -c yarn' returned a non-zero code: 1

I'd be curious if @Daniel15 has any ideas since he is a member of yarnpkg and was involved with the PR that included yarn with this image. yarn/2266 seems to be the cause.

All of our builds/containers have been broken for 2 days now. I'd love to see yarn get removed until node:7, since this is basically a breaking change.

[juliocanares]

Still having the issue. Currently using @wyattjoh fix.

Adding RUN yarn global add node-gyp before yarn install works

[Daniel15]

This is fixed in the next version of Yarn (currently only available as an RC, but there should be a stable release soon)

[juliocanares]

@Daniel15 cool to know it 👍 I will try

[spinlock99]

I just wanted to note my experience here. I was using nvm on a debian box and getting what looked like this node-gyp issue. I had to uninstall nvm and use the system node to actually get everything working. But, I did not need to install node-gyp