Migrate TypeScript Output from CommonJS (CJS) to ES Modules (ESM) - Epic

[gerteck]

Context

Helpful understanding:

• CJS vs. ESM
• Migrate TS repo to ESM blog

TLDR

• ESM modules can import CJS modules.
• CJS modules cannot import ESM modules.
• MarkBind is built as a CJS module.
• We cannot use modern ESM-only dependencies. If existing packages migrate to esm, we cannot update dependencies. If we want to integrate new esm-only packages, we also can't do so.
  • Many popular packages (like node-fetch v3, chalk v5, unified, remark) have already dropped CJS support. If you stay on CJS, you are locked into old versions forever.
• Only by migrating, will allow usage of both old CJS and new ESM packages. Hence, for future compatibility + usage of modern JavaScript features, it is a necessary infrastructure change.

Implementation Plan

• Migrate both core and clito Typescript. Useimport/export` syntax.
• Continue emitting CJS initially for both packages.
• Verify that it is working (imports, dynamic loading)
• Migrate core and cli to ESM together.

Describe the Enhancement

MarkBind's TypeScript configuration currently outputs CommonJS ("module": "commonjs"), a legacy module system. At some point, we should do a rewrite to migrate to a modern ESM-based output ("module": "NodeNext") to align with the evolving JavaScript ecosystem, unlock performance benefits, and ensure future compatibility.

• Ecosystem Shift: The JavaScript/Node.js ecosystem is standardizing on ES Modules (ESM).
• Performance & Features: ESM enables advanced optimizations like superior tree-shaking and supports modern language features like Top-Level await, which are impossible with CommonJS output.
• Forward Compatibility: Using "NodeNext" ensures the project correctly interoperates with both ESM and CJS packages according to Node.js's own modern rules, making it resilient to future ecosystem changes.
  • Newer versions of critical dependencies (e.g., markdown-it@14.0.0 (?)) may become pure ESM, causing integration issues to only become more frequent, especially as conduct maintainence and pdate dependencies.
  • Newer packages that we might want to integrate in the future may not support CJS require, and only support being used as a ESM module via import. (Case in point: pagefind). As future batches work to enhance MarkBind, this will be a significant roadblock if not resolved, and workarounds will only add to tech debt.
• Official Guidance: The TypeScript documentation now explicitly discourages "commonjs" for new projects, recommending "node16"/"nodenext" for Node.js environments.

Proposed Changes

1. Update tsconfig.json:

• Change "module": "commonjs" to "module": "NodeNext" or as appropriate.
• Change "moduleResolution": "node" to "moduleResolution": "NodeNext".

2. Migrate Legacy JavaScript Files: Convert remaining .js files using require/module.exports (e.g., in src/patches/) to use ESM import/export syntax.
3. Update package.json: Ensure "type": "module" is set appropriately for the project to be recognized as ESM by Node.js.

Others

• Searched existing issues.
• Possibly related Issues / PRs: Adopt TypeScript for core package #1877 , Package.json / Package-lock.json dependency Issue #2613
• Environment: Win 11
• MarkBind version: 6.0.2
• It is a major breaking change

Also, don't get confused:

• Even tho right now in the ts files, we are using import etc. (which is esm syntax), because of the ultimate output as cjs that in the end converts everything to require, we can't use esm only modules.

[gerteck]

Moving this to comments to keep the issue description cleaner:

Investigation

Current typescript config

The project's current typescript configuration settings are:

markbind/tsconfig_base.json

Lines 1 to 14 in 2cadb78

	{
	"compilerOptions": {
	"target": "es6",
	"module": "commonjs",
	"moduleResolution": "node",
	"esModuleInterop": true,
	"strict": true,
	"sourceMap": true,
	"declaration": true,
	"declarationMap": true,
	"forceConsistentCasingInFileNames": true,
	"newLine": "lf"
	}
	}

While reading typescript documentation, I noticed that the module option should be updated moving forward (particularly, set to node16 / nodenext), moduleResolution also needs to be updated correspondingly, especially as we continue to update Node.js versions (See tip-box in picture):

• Image-Source, ESM support in Node.js introduced in TypeScript 4.7, Module Option, Module Resoultion Option

Why is it important to migrate?

Many project dependencies have also rewritten to ESM, while CJS fallback is still available for some, it is likely some packages in the future may not have so.

• E.g. markdown-it changelog here Also, I think because of their rewrite to ESM, cannot import html_blocks.mjs from updated version of markdown-it.

I haven't fully processed what this means and there is any possiblity for regressions etc. However, I note that previous efforts in integrating new node packages was impeded by this issue, as in #2477 , which was resolved by updating this module option.

Any further investigation would be helpful.

Additionally, some js files need to be migrated to typescript as well, which are still using old require and module.exports (CJS) -> should use ESM:

• e.g. markbind\packages\core\src\patches\index.js,
• markbind\packages\core\src\patches\htmlparser2.js