JSON files included with package making it hard to bundle

[Goolyio]

Hey, I've been trying to use this package within a lambda that is bundled using esbuild.

Unfortunatley the way the JSON files are included means that they are often not included in final builds when bundled and minified.

this.headersOrder = JSON.parse(readFileSync(`${__dirname}/data_files/headers-order.json`).toString());

Is there a reason these are JSON files? Could they not just be constant values within the javascript and you can optionally override them.

Currently I will need to inject these files manually into build outputs but try to make sure they are only included in bundles that use your package which is pretty janky.

[barjin]

Hello and thank you for your interest in this package!

For the time being, I converted this issue into a discussion, as I don't have a clear idea about the fix for this right now (and it's also not breaking the library... too much).

This sounds quite similar to an issue we encountered quite a while ago (see #135). Back then, it was caused by the dynamic paths to the JSON files - the Vercel bundler couldn't follow those and therefore dropped those files. I'm assuming something similar is happening here - esbuild's tree-shaking might skip readFileSync calls (not consider them actual references to the files?). I'm thinking that converting those to require / import calls might fix this, but that comes with its own issues we would need to solve before that.

As to your question about why are those files JSON - well, they are data files, and they are automatically generated. While it's possible to generate .js files (that are exporting the data), it seems to me like a very bad code smell.

In the meantime, would you mind sharing details about your esbuild setup? I have only used it before as a part of larger, preconfigured projects, so I never really got to learn much about it. Do you have a minimal reproducible example?

[aecorredor]

+1 on this. I literally ran into this just today while deploying a scraper in AWS Lambda.

[aecorredor]

For the time being, I'm using patch-packageto just inline the headers as JavaScript. Just curious, why do you think it'd be a code smell to generate the data files JavaScript files that just export a constant object? It could even use something like Object.freeze to make it immutable.

[aecorredor]

I actually just realized that the code is also depending on other more complicated data, like zip files, so nvm on the patching.

[aecorredor]

I'm actually going to try to include the files during the build step.

[aecorredor]

Yeap, that did it. You can just bundle the data files in your lambda code. I'm using pulumi to deploy, and here's an example of how you'd do it:

// Lives in deploy/handler.ts
import { createHandlerAssetArchive } from './utils.ts';

export const handler = new aws.lambda.Function('functionName', {
  role: role.arn,
  runtime: 'nodejs18.x',
  memorySize: 1024,
  code: createHandlerAssetArchive('../dist/handler.js', {
    extraContent: {
      data_files: new pulumi.asset.FileArchive(
        '../node_modules/header-generator/data_files'
      ),
    },
  }),
  timeout: 900,
  handler: 'index.handler',
});

// Helper in another file that I always use to deploy lambdas with pulumi
// Lives in deploy/utils.ts
import * as pulumi from '@pulumi/pulumi';

export const createHandlerAssetArchive = (
  handlerPath: string,
  opts?: {
    extraContent?: Record<
      string,
      pulumi.asset.FileAsset | pulumi.asset.FileArchive
    >;
  }
): pulumi.asset.AssetArchive => {
  return new pulumi.asset.AssetArchive({
    'index.js': new pulumi.asset.FileAsset(handlerPath),
    ...opts?.extraContent,
  });
};

[MattGould1]

Hi for anyone encountering this but using Serverless Framework (or CloudFormation) the way I solved this was by creating a Lambda Layer. I added the entire fingerprint-injector module as a layer rather than just the header-generator so modify to your requirements.

Create a directory for the layer, mine is in layers/fingerprintInjector. In that directory create a package.json with:

{
  "name": "fingerprint-injector",
  "description": "fingerprint injector",
  "dependencies": {
    "fingerprint-injector": "2.1.49"
  }
}

Run npm install in the directory with your newly created package.json.

Add your layer to your serverless.yml file.

# The rest of your Serverless.yml file
layers:
  fingerprintInjector:
    path: 'layers/fingerprintInjector/'
  
functions:
  yourLambdaName:
    handler: ../index.handler
    layers:
      - !Ref FingerprintInjectorLambdaLayer

Finally mark fingerprint-injector as an external package in esbuild/webpack/other so that it's not included in your lambda. For me, I'm using serverless-esbuild so that looks like:

{
      external: [
        'fingerprint-injector'
      ]
 }

[ollebergkvist]

How could one solve this with Next.js?

[maxbeatty]

this solved it for us https://nextjs.org/docs/app/api-reference/next-config-js/serverComponentsExternalPackages

// See https://nextjs.org/docs/app/api-reference/next-config-js.
/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    serverComponentsExternalPackages: [
      "fingerprint-generator",
    ],
  },
};