@newrelic/native-metrics not working since v12.0.0

[jeremiewtd]

Since upgrading to v12.0.0 we have an error log running newrelic on our nodejs environment (node 20 or 22)

Description

When loading newrelic we have the error below :

Cannot find module '@newrelic/native-metrics'
Require stack:
- /app/node_modules/newrelic/lib/sampler.js
- /app/node_modules/newrelic/lib/agent.js
- /app/node_modules/newrelic/index.js
- /app/dist/src/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1248:15)
    at Module._load (node:internal/modules/cjs/loader:1074:27)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
    at Module.require (node:internal/modules/cjs/loader:1339:12)
    at Hook._require.Module.require (/app/node_modules/[...]/node_modules/require-in-the-middle/index.js:167:34)
    at Hook._require.Module.require (/app/node_modules/@newrelic/ritm/index.js:167:34)
    at require (node:internal/modules/helpers:125:16)
    at Object.start (/app/node_modules/newrelic/lib/sampler.js:157:30)
    at Agent.start (/app/node_modules/newrelic/lib/agent.js:246:11)

It seems that the prebuilt version of this module is not installed in our Docker container, it worked perfectly before the version upgrade.

Expected Behavior

The module is well loaded and we can view metrics in the newrelic interface.

Steps to Reproduce

Use newrelic in a nodejs (express) application that is packaged in a Docker container using FROM node:20-alpine or FROM node:22-alpine and running npm install in the Dockerfile

When starting the application (docker run ...) via the container an error is logged in the newrelic_agent.log (the error above).

[workato-integration]

https://new-relic.atlassian.net/browse/NR-301211

[jsumners-nr]

packaged in a Docker container using FROM node:20-alpine or FROM node:22-alpine

This is the issue. We do not provide pre-built binaries for musl based systems. As it stands today, the only way to get a musl based Node.js is to use an experimental unofficial build (https://github.com/nodejs/unofficial-builds).

Note that the Docker images Node.js provides are shipping this unofficial build (unless something goes wrong): https://github.com/nodejs/docker-node/blob/766b2dec6f59b6c98bf190e818edb1b0c7e532c5/20/alpine3.20/Dockerfile#L21-L24

[jeremiewtd]

Ok so why did it work before v12.0.0 of newrelic ?

[mrickard]

@jeremiewtd Which version of newrelic were you upgrading from?

[jeremiewtd]

@mrickard we upgraded from v11.23.2

[mrickard]

v11.23.2 would have had the same condition.

v11.x of the agent would install v10 of the native metrics package, and v12 of the agent installs v11 of the native metrics package. Between those releases of native metrics, the primary changes are the removal of support for Node 16, and a change to the CI runner used. The prebuilt binaries generated during release are

• linux-arm64
• macos-14-arm64
• ubuntu-latest-x64
• windows-latest-x64
• windows-latest-x86
  and the NPM module.

That's the valid list of binaries for v10.x and v11.x of the native metrics module. (v10.2.0 introduced support for Node 22, and v11.0.0 dropped support for Node 16.)

Are you able to install the native metrics module from NPM?

[jeremiewtd]

Yes I can but in the Dockerfile I need to install the following tools for the module to install correctly (for building I guess) otherwise I get an error "cannot find Python".

RUN apk add python3 make g++

[mrickard]

Ok, that would be missing from an alpine image. Would you be able to try a slim image?

[jeremiewtd]

That doesn't work this the slim image either.

Could this be a problem with changing your CI architecture? We build the image for amd64 and not arm64.

[jeremiewtd]

After looking at our previous package-lock.json content I see that we were using @newrelic/native-metrics@10.0.1 so the issue may be introduced after this version.

I made a repository to reproduce the issue.

[jsumners-nr]

That doesn't work this the slim image either.

Could this be a problem with changing your CI architecture? We build the image for amd64 and not arm64.

No, that is not the source of your issue. Release 10.1.0 (https://github.com/newrelic/node-native-metrics/releases/tag/v10.1.0) changed the way pre-builts are included. Prior to 10.1.0, they were published to a download site. On install of the module, the module would attempt to build itself locally. If that failed, it would attempt to find one via the download site:

https://github.com/newrelic/node-native-metrics/blob/1fcd2f354cd6cdd048c57c80fc968665c9d41239/README.md?plain=1#L24-L40

The @newrelic/native-metrics module has never provided a pre-built binary that works on musl based systems. In particular, the available pre-built binaries for 10.0.1 are:

• https://download.newrelic.com/nodejs_agent/builds/_newrelic_native_metrics-10_0_1-native_metrics-108-linux-x64.gz
• https://download.newrelic.com/nodejs_agent/builds/_newrelic_native_metrics-10_0_1-native_metrics-115-linux-x64.gz
• https://download.newrelic.com/nodejs_agent/builds/_newrelic_native_metrics-10_0_1-native_metrics-93-linux-x64.gz

Notice that the -108, -115, and -93 are references to the Node.js ABI and that all of them target baseline amd64 (i.e. glibc).

So, in your setup with 10.0.1, the module was performing a custom build upon installation according to the script: https://github.com/newrelic/node-native-metrics/blob/v10.0.1/lib/pre-build.js

With 10.1.0 we utilize the standard node-gyp build system for building the native module when a pre-built is not found. Therefore, non-standard systems, e.g. ones using musl instead of glibc, need a node-gyp compatible build system in order to successfully install the module. This is highlighted in the readme: https://github.com/newrelic/node-native-metrics/blob/d7c339a978e42c04c9f1b1be92054b653b3ebaa1/README.md?plain=1#L42-L48

[jeremiewtd]

Ok thank you for explanation, so the solution for us should be :

• to use the "full node image" => FROM node:20 / FROM node:22
  OR
• continue using FROM node:20-alpine but install required modules to build the the binary like this
  RUN apk add python3 make g++

Is that correct?

Isn't this change breaking the functionality of newrelic? Because by updating the module we lost these “native metrics”

[jsumners-nr]

Yes, using node:20 should solve the issue as a pre-built for it is included in the @newrelic/native-metrics package. If the desire is to continue using Alpine, I recommend using a multi-stage build to construct your container:

package.json

{
  "dependencies": {
    "@newrelic/native-metrics": "^11.0.0"
  }
}

index.js

'use strict'

const metrics = require('@newrelic/native-metrics')
console.log("gcEnabled:", metrics().gcEnabled)

Dockerfile

FROM node:20-alpine AS builder

WORKDIR /app
COPY index.js package.json .

RUN apk add g++ make py3-pip
RUN npm install --production

FROM node:20-alpine
COPY --from=builder app/ /app/
WORKDIR /app
CMD node index.js

$ docker build --tag demo .
$ docker run --rm -it demo
gcEnabled: true

Isn't this change breaking the functionality of newrelic?

I don't think so. We now ship a much more comprehensive matrix of pre-built binaries that do not require calls to an outside network to install. And we are now utilizing the standard build process in the absence of a pre-built for the target environment. So, in my view, we are more compatible, not less.

Also, the @newrelic/native-metrics module failing to load is accounted for:

node-newrelic/lib/sampler.js

Lines 154 to 169 in 32abe5f

	// This requires a native module which may have failed to build.
	if (agent.config.plugins.native_metrics.enabled && !this.nativeMetrics) {
	try {
	this.nativeMetrics = require('@newrelic/native-metrics')({
	timeout: SAMPLE_INTERVAL
	})
	} catch (err) {
	logger.info(
	{ error: { message: err.message, stack: err.stack } },
	'Not adding native metric sampler.'
	)
	agent.metrics
	.getOrCreateMetric(NAMES.SUPPORTABILITY.DEPENDENCIES + '/NoNativeMetricsModule')
	.incrementCallCount()
	}
	}

[jeremiewtd]

We fixed the problem by installing the modules in our alpine container, thank you.

RUN apk add g++ make py3-pip

[jsumners-nr]

Glad to hear you have got it resolved.