How do I work with "internal packages" in a node server application?

[piedrahitapablo]

The docs introduce the internal packages concept using examples for Next.js and Vite but as far as I know, that doesn't cover the case for server apps.

I know that for development nodemon/ts-node can be used to run the code that imports untranspiled packages and it just works, but how do I build my server app for production? Is there a way to build/transpile the app with its dependencies? Using tsc and the turbo pipelines could be an option, that way when I build my server app it will build the internal packages, but then I will have to modify the main and types values for the internal packages which will defeat the whole "internal package" thing.

I know there are things like ncc, but I'm not sure if that will be the best approach. Thanks!

[BenJackGill]

What did you end up doing?

I'm having difficulty using Firebase Functions as one of my monorepo apps because it uploads and builds everything in the cloud. The cloud build process tries to download the internal monorepo packages from the npm registry but since they are internal packages it can't find them and the cloud build fails.

There are various solutions for bundling the internal packages before uploading the Firebase Functions code to Firebase, but most of them are messy or have drawbacks.

Hoping you found something else! Am checking out ncc now because I never knew about that one.

[predrag-codetribe]

"internal packages" are a great way to reuse code
it is pretty straight forward to setup and use with Vite.

Today I lost an entire day looking at ways for "internal packages" to just work in a compiled TypeScript NodeJS app.
Here is what I have concluded.

I ended up installing tsup
and using tsup instead of tsc to compile my project. (I left tsc --noEmit to typecheck before I compile)
but I still haven't been able to make it work for my setup the way tsc works...

Here are the changes I made to my package.json:

{
  "name": "server",
  "version": "1.0.0",
  "scripts": {
    "dev": "tsx watch ./src/index.ts",
-   "build": "tsc",
+   "build": "tsc --noEmit && tsup ./src/index.ts",
    "start": "node ./dist/index.js",
  },
  "dependencies": {
    "express": "4.18.2",
    "@shared/utils": "*", // internal package
    //...
  },
  "devDependencies": {
    "typescript": "5.2.2",
+   "tsup": "8.0.1",
    // ...
  }
}

[predrag-codetribe]

but I'm not sure if that will be the best approach.

I think the same.
I would really appreciate someone with more knowledge about this topic to give some advice.

[BenJackGill]

@predrag-codetribe I ended up with this solution for Firebase. Maybe it will help you also?

But I think maybe I missed the mark and we could be talking about seperate issues here.

[predrag-codetribe]

Thanks I tried that, didn't work for me.

tsup also didn't work:

• because my server uses TypeOrm, and my migrations are loaded like this entities: [__dirname + '/app/model/*.entity{.js,.ts}'], once compiled typeorm cannot find the entities...
• it is a pain to set tsup to mimic tsc, and I don't expect it to make it work over night, but I quit from that approach as it seems complicated.

I think I am gonna quit from the "internal packages" for the Node side,
I will continue to use "internal packages" for the Vite projects in my monorepo.

Here below is what I do on the Node side that works for me:

tsconfig.json

{
    "compilerOptions": {
        "module": "CommonJS",
        "target": "ES2021",
        "esModuleInterop": true,
        "outDir": "./dist",
        "noEmit": false,
        "noFallthroughCasesInSwitch": true,
        "noImplicitReturns": true,
        "resolveJsonModule": true,
        "emitDecoratorMetadata": true,
        "experimentalDecorators": true,
+        "baseUrl": ".",
+        "paths": {
+            "@shared/utils": [ // required for code sharing
+                "../shared/utils/src/*"
+            ]
        }
    },
    "include": [
        "./*/**/*",
+        "../shared/utils" // required for code sharing
    ]

package.json

{
  "name": "server",
  "version": "1.0.0",
  "scripts": {
+    "dev": "tsx watch ./src/index.ts",
-   "build": "tsc",
+   "build": "tsc && tsc-alias",
-    "start": "node ./dist/src/index.js"
+   "start": "node ./dist/server/src/index.js"
  },
  "dependencies": {
    // ...
-   "@shared/utils": "*"
  },
  "devDependencies": {
+    "tsc-alias": "^1.8.8",
    "tsx": "4.1.4",
    "typescript": "5.2.2"
  }
}

I will still continue looking at this topic.

[gtjamesa]

I spent a couple of days trying out different solutions, including those mentioned here, but I wasn't 100% happy with them. Some build tools required more changes than others, and I found the added complexity and overhead quite annoying. I even considered just running ts-node and being done with it 😂

I've finally found a solution that will work well for everyone here and 100% maintains the "internal package" perks and local development experience. You need to define a publishConfig, and this will be applied to the dependency's package.json post-build. Surprisingly, I couldn't find an existing package to do this, so I made one: apply-publishconfig. I've just hoisted this from my monorepo in the hope that it will help someone 🙏

The package will publish an apply-publishconfig command that applies any config set in publishConfig, and I have configured this as a Turbo pipeline (publish:apply). During my Docker build, I execute the pipeline after building all dependent packages:

RUN --mount=type=bind,source=.git,target=.git \
    turbo run build -F "${WORKSPACE}"... \
    && turbo run publish:apply -F "${WORKSPACE}"...

The package.json's fields (i.e. main or exports) are swapped out for their desired compiled counterparts. This change then lives solely inside the image (or wherever) you're packaging your application and will not affect your local environment.

   "version": "1.0.0",
-  "main": "./src/index.ts",
+  "main": "./dist/index.js",
   "types": "./src/index.ts",
   "scripts": {
     "build": "tsup src/index.ts",
@@ -16,9 +16,6 @@
-  "publishConfig": {
-    "main": "./dist/index.js"
-  },

[tannerhladek]

@gtjamesa , if utilizing this method, does the internal package itself need to be built? I assume yes, but the "internal" package method implies that said internal package can skip the build step altogether.

[tannerhladek]

To address this challenge, I implemented a solution using tsup. Unlike the default behavior, which compiles code into a single file while excluding dependencies and peer dependencies, tsup's configuration can be adjusted to include specific dependencies, like internal packages.

"scripts": {
  "build": "tsc --noEmit && tsup",
  "start": "node dist/server.js"
},
"tsup": {
  "entry": ["src/server.ts"],
  "noExternal": ["@internal-package-to-include"]
},

This setup ensures that tsup compiles the code, including the specified internal package, into the build output, effectively addressing the issue with internal packages in a Node server application.

[yamcodes]

Edit: The issue below is mitigated by referring the internal package in the external package's package.jsons devDependencies, rather than dependencies.

While this works great to eliminate the imports and grab the source code from the internal package to the external one, the external package still has the internal one in its package.jsons dependencies, which prevents the external package's installation in projects outside the monorepo.

Any practical solution will also need to modify the package's package.json to avoid references to the internal package, which is not actually published to NPM.

At least, this happens when using npm. Not sure if the same happens with the workspace:* directive of pnpm and Yarn.

[BenJackGill]

@tannerhladek thanks for pointing me in the direction of tsup. I think it could be a good solution. But I'm confused why you're running tsc --noEmit before tsup. Isn't that pointless since tsup is doing the compiling?

[tannerhladek]

@BenJackGill - glad this helped! The reason for tsc --noEmit before tsup is to ensure type safety. While tsup compiles and bundles the code, it doesn’t check types. tsc --noEmit does the type checking without producing output files. This step catches type errors early, making sure the code is not only compiled but also type-safe before bundling with tsup.

[tannerhladek]

@yamcodes makes sense, but in my case, the imported internal package needs to be bundled as a non-dev dependency. Also, neither package is being published to npm, so for my use case, tsup solved the problem perfectly 😎

I will keep this in mind for future published packages though!

[yamcodes]

@yamcodes makes sense, but in my case, the imported internal package needs to be bundled as a non-dev dependency. Also, neither package is being published to npm, so for my use case, tsup solved the problem perfectly 😎

I will keep this in mind for future published packages though!

Ohhh! I realize I am off topic, this topic is about using internal packages in Node.js apps. In that case, there is no "external package" involved, so the resulted package.json is irrelevant. What matters is the bundle. Your solution should work perfectly 😎

Anyway, just to update, we still use my solution (installing as a devDependency) in to use internal packages within external packages and it works great. Sorry for being off-topic, but I hope it's useful!

[anthonyshew]

We've shipped some documentation updates that describe this in detail: https://turbo.build/repo/docs/core-concepts/internal-packages

[j4tmr]

my solution.
The main thing is to configure external and noExternal of tsup config by yourself, because there may be many jit packages written by yourself, you need to get them in depth.
what need to do next is to create a new package.json for this app and maybe add to dist

tsup.config.ts

import { defineConfig } from 'tsup';
import { getConfigs } from './extract';

export default defineConfig({
	entry: ['./src/index.ts'],
	format: ['cjs', 'esm'],
	sourcemap: true,
	splitting: false,
	clean: true,
	dts: true,
	...getConfigs({ packageUrl: './', packagesUrl: '../../packages' })
});

package.json

{
  "name": "fastify",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "main": "index.js",
  "author": "",
  "license": "ISC",
  "dependencies": {
    "fastify": "^4.28.1"
  },
  "devDependencies": {
    "@repo/ui": "*"
  },
  "scripts": {
    "build": "tsc --noEmit && tsup",
    "start": "node ./dist/index.js"
  }
}

extract.ts

import * as fs from 'fs';
import * as path from 'path';

interface PackageDependencies {
	name: string;
	dependencies?: Record<string, string>;
	devDependencies?: Record<string, string>;
}

const getPackage = ({ url }: { url: string }) => {
	const packageJsonPath = path.join(url, 'package.json');
	if (fs.existsSync(packageJsonPath)) {
		return JSON.parse(fs.readFileSync(packageJsonPath, 'utf-8')) as PackageDependencies;
	}
	return undefined;
};

function getPackages(dir: string) {
	const packagesDirs = fs.readdirSync(dir).filter((file) => {
		return fs.statSync(path.join(dir, file)).isDirectory();
	});
	let packages: PackageDependencies[] = [];
	for (const subDir of packagesDirs) {
		const packageJsonPath = path.join(dir, subDir);
		const p = getPackage({ url: packageJsonPath });
		if (p) {
			packages.push(p);
		}
	}
	return packages;
}

export const getConfigs = ({ packageUrl, packagesUrl }: { packageUrl: string; packagesUrl: string }) => {
	const p = getPackage({ url: packageUrl });
	if (p) {
		const packages = getPackages(packagesUrl);
		const packageNames = packages.map((p) => p.name);

		const findexternal = (currentPackage: PackageDependencies) => {
			let externals: string[] = [];
			if (currentPackage.dependencies) {
				const currentPackageNames = [
					...Object.keys(currentPackage.dependencies || {}),
					...Object.keys(currentPackage.devDependencies || {})
				];
				for (const name of currentPackageNames) {
					if (packageNames.includes(name)) {
						const p = packages.find((p) => p.name === name);
						console.log(name);
						externals = [...externals, ...findexternal(p!)];
					} else {
						externals.push(name);
					}
				}
			}
			return externals;
		};

		const findNoExternal = (currentPackage: PackageDependencies) => {
			let noExternals: string[] = [];
			if (currentPackage.dependencies) {
				const currentPackageNames = [
					...Object.keys(currentPackage.dependencies || {}),
					...Object.keys(currentPackage.devDependencies || {})
				];
				for (const name of currentPackageNames) {
					if (packageNames.includes(name)) {
						const p = packages.find((p) => p.name === name);
						noExternals.push(name);
						noExternals = [...noExternals, ...findNoExternal(p!)];
					}
				}
			}
			return noExternals;
		};

		return {
			external: findexternal(p),
			noExternal: findNoExternal(p)
		};
	}
};

[y-nk]

one alternative is to add pnpm -filter="packageName^..." build as a prebuild step.

not ideal but this would avoid the mess.

https://pnpm.io/filtering#--filter-package_name-2