Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parcel 2 doesn't detect changes in linked dependencies. #4332

Open
samuelgozi opened this issue Mar 15, 2020 · 90 comments
Open

Parcel 2 doesn't detect changes in linked dependencies. #4332

samuelgozi opened this issue Mar 15, 2020 · 90 comments

Comments

@samuelgozi
Copy link

🐛 bug report

When working on a dependency locally by linking it, Parcel 2 doesn't detect changes and therefore doesn't rebuild. You also need to remove the cache manually in order for it to work on re-runs.

🎛 Configuration (.babelrc, package.json, cli command)

Package.json

{
	"name": "parcel-tamplate",
	"version": "1.1.0",
	"description": "My personal Parcel Template",
	"main": "public/index.html",
	"source": "src/index.html",
	"author": "Samuel Elgozi <samu.elgozi@gmail.com>",
	"license": "none",
	"private": true,
	"scripts": {
		"dev": "parcel src/index.html",
		"build": "parcel build src/index.html"
	},
	"targets": {
		"main": {
			"includeNodeModules": true
		}
	},
	"devDependencies": {
		"@babel/plugin-proposal-class-properties": "^7.8.3",
		"babel-eslint": "^10.0.2",
		"eslint": "^6.0.1",
		"less": "^3.9.0",
		"parcel": "^2.0.0-alpha.3.2"
	},
	"dependencies": {
		"firebase-auth-lite": "^0.4.1"
	}
}

The commands I use:

parcel src/index.html

🤔 Expected Behavior

Like in version 1, Parcel should detect changes in linked dependencies, invalidate the cache and rebuild.

😯 Current Behavior

No changes are made, which require manual removal of .parcel-cache and then building again.

🔦 Context

I use Parcel to work on npm modules often. Without detecting the changes in linked packages there is no reason for me to use parcel. In fact, it becomes painful to do so...

🌍 Your Environment

Software Version(s)
Parcel 2.0.0-alpha.3.2
Node v13.10.1
Yarn 1.22.1
Operating System MacOs Catalina 10.15.3
@DeMoorJasper
Copy link
Member

Could you create a repo with some instructions on how I'd reproduce this? Would help a lot in fixing this

@samuelgozi
Copy link
Author

Ok, here you go: https://github.com/samuelgozi/Parcel2-issue-4332
Further instructions can be found in the repo's README file.

@DeMoorJasper
Copy link
Member

@samuelgozi thanks for creating the repo, I can reproduce it pretty consistently, will try and figure out why it happens and hopefully fix it

@samuelgozi
Copy link
Author

@DeMoorJasper Thank you!

@kalaschnik
Copy link

I develop a local npm package and experience the same issue using the npm link functionality. Is there any news on this?

@devongovett
Copy link
Member

devongovett commented Oct 12, 2020

This is because the file watcher only watches for changes in the project root. It would be somewhat hard to implement support for symlinks outside the project I think. If we implemented this in the watcher we'd need to somehow crawl the whole file system to detect symlinks in the first place which would be very slow. We could potentially do it by detecting files that resolve to symlinks as we build the asset graph and keep nodes in the request graph to track watch roots. Then we'd have to re-watch all of those on startup and invalidate each of them for changes as opposed to just the project root.

(parcel-bundler/watcher#32)

@ramirofages
Copy link

This is because the file watcher only watches for changes in the project root. It would be somewhat hard to implement support for symlinks outside the project I think. If we implemented this in the watcher we'd need to somehow crawl the whole file system to detect symlinks in the first place which would be very slow. We could potentially do it by detecting files that resolve to symlinks as we build the asset graph and keep nodes in the request graph to track watch roots. Then we'd have to re-watch all of those on startup and invalidate each of them for changes as opposed to just the project root.

(parcel-bundler/watcher#32)

This was actually working in parcel 1. Currently I had to revert back to parcel 1 while keeping its dependencies in parcel 2 to correctly build as module.

@krnlde
Copy link

krnlde commented Mar 18, 2021

This issue keeps me from migrating to v2, too. I wonder how other developers handle this workspaces issue - also considering monorepos are now en vogue. We can't possibly be alone here @ramirofages

@krnlde
Copy link

krnlde commented Apr 9, 2021

You guys should have a look at #6039 regarding the yarn.lock / package-lock.json files. This probably solves the issue for you too.

@kristojorg
Copy link

That linked issue doesn't resolve this for me. I don't have a dangling yarn.lock file, just the one in my primary repository I'm working on. Linked dependency changes don't trigger a recompile, and it seems even restarting parcel and running it with --no-cache don't trigger a full rebuild. I have the delete .parcel-cache in order to have the changes picked up : /

@krnlde
Copy link

krnlde commented Jun 14, 2021

Afaik parcel can't look outside its start context so if you want to watch all workspaces in a workspace root you'll need to start parcel in that root. Did you do that? Be extra sure to not have any lock files inside any of the workspaces. Parcel unfortunately has no warning for that and will just silently fall back to the workspace scope instead of root scope, which took a lot of lifetime from me in the past the point where I wanted to quit my job and start a lumber business :D

@kristojorg
Copy link

I am trying to link a dependency that is located in a separate folder and repository on my machine, but installed via npm. The node_modules/my-dep folder does change when I update the linked dependency, but parcel doesn't seem to pick up the changes there. Do you know if there is a way to have it watch and rebuild node_modules as well?

@krnlde
Copy link

krnlde commented Jun 14, 2021

As long as the node_modules is inside the parcel start context it works. Again parcel picks its start context by probing all folders for .lock files starting with the most-inner one from your start path: parcel serve workspaces/workspace-a/index.html probes:

  1. workspaces/workspace-a/*.lock
  2. workspaces/*.lock
  3. *.lock

wherever a lock file is found it will not recognize changes beyond that folder. Ever. And they are not planning to change that.

@kristojorg
Copy link

Yep I hear you, but let me be more explicit about my setup. I have:

project-a/
    src/                      # code lives here
    example/
        index.html       # no lockfile here, no package.json, no node_modules
    package.json      # here is my script "npm run example": "parcel example/index.html"
    package-lock.json
    node_modules/
        project-b/
project-b/                # linked to ^
    src/
    dist/

It is the linked project-b dependency which is changing and not triggering a rebuild inside project-a when I run npm run example. So what I'm trying to say is that this isn't an issue of a misplaced or extra *.lock file it seems.

@krnlde
Copy link

krnlde commented Jun 15, 2021

So its linked by npm link and not by workspace logic? I couldn't repeat myself more but:

wherever a lock file is found it will not recognize changes beyond that folder

Edit: also to clarify, parcel doesn't watch the node_modules (since its essentially a black hole), it watches source files and triggers a reload when they change.

@mischnic
Copy link
Member

mischnic commented Jun 15, 2021

If project-a contains a lockfile or a .git directory, it will be the project root and only files inside of that are watched.

Edit: also to clarify, parcel doesn't watch the node_modules (since its essentially a black hole), it watches source files and triggers a reload when they change.

No, it watches everything except for VCS directories and .parcel-cache:

let vcsDirs = ['.git', '.hg'].map(dir => path.join(options.projectRoot, dir));
let ignore = [options.cacheDir, ...vcsDirs];
return {ignore};

@kristojorg
Copy link

Okay yes it is the "Edit" there that is my problem then. The confusing thing is that this did work with parcel v1. When I would npm link package-b, parcel would rebuild package-a whenever it changed.

As mentioned, project-a does contain a lockfile and a .git directory, but node_modules are within that folder so I expected them to be watched, or at least not permanently cached when I restart with npm run example. I currently have to delete .parcel-cache in order for it to include the updated node_modules.

@krnlde
Copy link

krnlde commented Jun 15, 2021

Ah cool! Didn't know that. How about symlinks then?

@kristojorg
Copy link

@mischnic so it does appear that this is a legitimate bug and not intended behavior, then. Should node_modules possibly be excluded from being written to the .parcel-cache?

@mischnic
Copy link
Member

mischnic commented Jun 15, 2021

How about symlinks then?

"only files inside of that are watched" essentially ignores symlinks. This is mostly because the file watching APIs from the various OSs only fire change events for the actual parent directories and don't traverse through every symlink

So node_modules is watched, but node_modules/symlinked-package/* isn't

So this is more of a technical limitation at the moment, as explained in #4332 (comment) and not a design decision

a legitimate bug and not intended behavior, then. Should node_modules possibly be excluded

What do you mean? It intentionally watches node_modules so that updating dependencies and rerunning Yarn triggers a Parcel rebuild for changed files.

@kristojorg
Copy link

I see, thanks for the explanation. I wonder if it would be possible to make an option to explicitly tell parcel about a symlinked dependency you would like to be watched, as this does seem like it would be a common use case.

What do you mean? It intentionally watches node_modules so that updating dependencies and rerunning Yarn triggers a Parcel rebuild for changed files.

I meant should node_modules be watched but not cached, such that when you restart parcel it is reading node_modules fresh and not using a cached version. I don't think I totally understand the value of including node_modules files in the cache if they are not bundled. Seems like I'm missing something tho : )

@devongovett
Copy link
Member

That's not the issue. Symlinks can still resolve to locations outside Parcel's project root, which is defined to be the nearest directory with a yarn.lock or package-lock.json file. So if those are not at your repo root, but you reference a file outside of this directory, it won't be watched.

Is there a reason not to put your lock file at the root of your repo?

@bminer
Copy link

bminer commented Feb 8, 2023

@Systemcluster - As discussed above, allowing Parcel to watch symlinks outside the project root is trickier to implement. Allowing users to specify additional paths to watch outside of Parcel's project root seems like a fair compromise - and it sounds rather easy to do. As many have mentioned, the "project root" means the folder containing the package lock file.

@rodrigo-nexudus
Copy link

This used to work without issue in V1 (parcel-bundler) just by adding a link in package.json of the main project. Any changes on the linked package caused a refresh. This has now been open for 3 years and it was something that worked flawlessly on the first version of Parcel. It's an incredibly useful feature when you're working on a library.

@bminer
Copy link

bminer commented Mar 16, 2023

Wow it really has been 3 years! Like I've said, I'd be happy to open a PR to resolve this one if I can get some feedback from the devs that it's likely to be merged. If the devs explicitly don't want this feature, I don't want to waste my time.

Thoughts? @devongovett

@maxx0r
Copy link

maxx0r commented Mar 16, 2023

I for one would be really happy with a fix. I've built a script in my webapp that copies on change, not really the solutions you want

@rodrigo-nexudus
Copy link

@bminer I don't see why at least giving developers an option through a flag as you suggested would be an issue, if respecting symlinks outside the project root as before is out of the question. In our case, the package I'm working on is a dependency on more than one in-house project, and is not an integral part of any of the parent projects. It doesn't make sense to build all as "workspaces" as the parent projects have no relation between them.

@tpompon
Copy link

tpompon commented Apr 6, 2023

I'm getting the same issue. Building a package and I have a demo app using Parcel 2 in an folder example, I don't want to mix my .lock files and I would like to be able to get the app refreshing when I apply changes on the package components.

@AmrAhmedA
Copy link

I run into this issue recently, the fact that it is standing for 3 years is amazing. unfortuntly, such limitition in parcel 2 makes it less productive when working with linked dependencies.
Possible workaround I found is to refresh package.json for triggering new build.
any convenient solution proposed so far?

@ThomasZeman
Copy link

ThomasZeman commented Apr 23, 2023

I agree it would be great to have this fixed. My scenario seems even simpler:

some_root_dir/

├─ client/
│ ├─ index.html
│ ├─ package.json
│ ├─ package-lock.json
│ └─ client.js

└─ shared/shared.js

where client.js imports shared.js -- When I change client.js Parcel does its thing, updates the browser and life is good. When I updated shared.js nothing happens. So to me, the file-watching issue comes down to:
Parcel should watch all files referenced starting from index.html as entry point no matter which directories these are in.

@rowthan
Copy link

rowthan commented Jun 7, 2023

I agree it would be great to have this fixed. My scenario seems even simpler:

some_root_dir/ │ ├─ client/ │ ├─ index.html │ ├─ package.json │ ├─ package-lock.json │ └─ client.js │ └─ shared/shared.js

where client.js imports shared.js -- When I change client.js Parcel does its thing, updates the browser and life is good. When I updated shared.js nothing happens. So to me, the file-watching issue comes down to: Parcel should watch all files referenced starting from index.html as entry point no matter which directories these are in.

some issue, any way to solve this problem?

@rowthan
Copy link

rowthan commented Jun 7, 2023

I agree it would be great to have this fixed. My scenario seems even simpler:

some_root_dir/ │ ├─ client/ │ ├─ index.html │ ├─ package.json │ ├─ package-lock.json │ └─ client.js │ └─ shared/shared.js

where client.js imports shared.js -- When I change client.js Parcel does its thing, updates the browser and life is good. When I updated shared.js nothing happens. So to me, the file-watching issue comes down to: Parcel should watch all files referenced starting from index.html as entry point no matter which directories these are in.

@rodrigo-nexudus there is the case what we want. example serve by parcel is just a side part of parent lib dir. we just want limit all test or dev code in example dir. But it should be rebuild after file changed in src.
So it's a very important for us to listen the change from other dir.

@nullablemind
Copy link

@jteppinette Thanks for your message. #4332 (comment)
Thanks to you, at least I was able to fasten the crutch so that it all works fine.
I have added a script to watch files using chokidar and simply call a touch package.json when have a change.

The best I could do at the moment. Since inside the folder, there are still their dependencies with package-lock.json.

@jfcere
Copy link

jfcere commented Dec 2, 2023

I personally worked around this using nodemon and concurrently with npm scripts, this solution is not perfect but works for me so I thought it worth sharing.

npm i concurrently nodemon --save-dev

See start script below which will concurrently...

  • watch my lib src folder and rebuild it when it changes (I use rollup to bundle my lib)
  • link my local library to node_module (I use the old linklocal package but you can use npm-link for that step)
  • and finally start the web page using parcel
  "scripts": {
    "prestart": "rimraf .parcel-cache",
    "start": "concurrently \"npm run watch:lib\" \"npm run link:lib && npm run start:demo\"",
    "start:demo": "parcel demo/index.html --open",
    "build:lib": "rollup -c rollup.config.ts --configPlugin @rollup/plugin-typescript",
    "link:lib": "rimraf node_modules/my-lib && linklocal",
    "watch:lib": "nodemon -e ts --watch src --exec npm run build:lib",
  },

Hope that helps anyone!

@jondlm jondlm mentioned this issue Dec 7, 2023
3 tasks
@devongovett
Copy link
Member

FYI, thanks for @jondlm, there is a new --watch-root CLI option to configure the root directory where the watcher listens which will be available in the next release. See #9424 for details.

@remifroger
Copy link

remifroger commented Jun 18, 2024

Thanks, it works for me with --watch-dir

I've the app and, outside, I've shared js files (assets dir).

app/
├─ src/client/js/index.js
├─ package.json
├─ package-lock.json
assets/js

The dir assets is installed with npm i ../assets in the app
Then I import my modules in the app/src/client/js/index.js, for example : import { myFunc1, myFunc2 } from 'assets/js/map.js'

And now with this config in the app/package.json and the new --watch-dir :

"scripts": {
  "build-watch": "parcel watch ./src/client/js/index.js --watch-dir ../assets --host localhost --no-cache --public-url ./ --dist-dir ./dist && npm cache clean --force",
  "start-watch": "nodemon ./src/server/app.js",
   "dev": "concurrently --kill-others \"npm run start-watch\" \"npm run build-watch\""
}

The HMR is working again, like in parcel1. I mean, if I make a change in my assets js files, the app rebuild automatically.

@everdimension
Copy link

For those having issues, I have created a CLI-tool that can help deal with local npm packages: physical-link

It's not parcel-specific, but it solves the problem of aliased packages not being watched. What it does is basically watch your package files and copy changes to the node_modules directory of the project where you're using that package.

It's a handy tool when you can't have a monorepo and develop your packages as separate projects, and use them in your other projects.

@zeel01
Copy link

zeel01 commented Sep 7, 2024

I really hoped that when I Googled this issue I would find a solution. I went off on a tangent one night writing a bunch of utilities to simplify a project, then realized it made no sense for them to belong to that project instead of being somewhere I could share with other projects. So I pulled it out into its own package... and now Parcel ignores any changes I make to it because the linked package doesn't trigger the watcher.

@bminer
Copy link

bminer commented Sep 13, 2024

Since --watch-dir is a good workaround, is it safe to close this issue?

@bradleat
Copy link

This still does not work for me when I do a parcel build in a monorepo using the latest version (2.12.0)

I am using a lerna monorepo and this is my full parcel command:

NODE_ENV=development npx parcel build --no-autoinstall --no-optimize --target frontends --cache-dir ./.parcel-cache --watch-dir ../../,

I didn't really want to setup a dev / watch command although I might now...

@bminer
Copy link

bminer commented Sep 19, 2024

Interesting. I am doing something quite similar, and it seems to work fine: parcel --watch-dir ../ index.html

@zeel01
Copy link

zeel01 commented Sep 19, 2024

I don't think --watch-dir even comes close to solving the original issue. If you use npm link to symlink a package into your project, Parcel doesn't follow the link. Adding --watch-dir to watch the linked package that's somewhere else on your machine (possibly not in the same parent directory as your project even) is at best a hacky work-around since you might need to resort to an absolute file path. But it also doesn't seem to work reliably anyway.

The entire purpose if npm link is so you can include a local version of a dependency that might be unrelated to the current project without replicating a specific directory structure. As such, the bundler should support the feature as it is intended to be used without need for a work-around. Even if --watch-dir is a workable work-around for some users, the core issue remains.

Personally, I think the original labeling of this as a bug was correct since it's a regression from Parcel 1. But even as a feature request, it's still more than valid and is not adequately satisfied by --watch-dir.

@bradleat
Copy link

I agree. However, this especially noticeable when using parcel build over parcel watch.

@bminer
Copy link

bminer commented Sep 20, 2024

I agree. However, this especially noticeable when using parcel build over parcel watch.

@bradleat - Are you saying that parcel build does not detect changes to paths outside of the Parcel "project root?" Are you absolutely sure about this? I was under the impression that this issue is specific to parcel watch only.

@bradleat
Copy link

@bminer Yes. When I parcel build with the command above and have updated linked build files, they are sometimes not included in the build. I really can't figure out why it is sometimes.

When watching, it will work.

@zeel01
Copy link

zeel01 commented Sep 20, 2024

It seems like it just uses it's cached version since it doesn't detect that anything is changed. I'm constantly needing to use rimraf to clear the cache out and get it to rebuild.

@brianhelba
Copy link

@bminer In my experience with this bug, the problem was that Parcel watching didn't follow symlinks. If watching .. coincidentally also included the real destination of the symlink, then everything might work, but other project layouts could still cause problems.

In my case, I wasn't using npm link, but I had some application code which depended on an associated library via a local path. On Linux, npm or Yarn Classic (I can't remember which 😕) would create a symlink within node_modules, which Parcel wouldn't follow.

I recall that I was able to bisect the origin of this bug to some refactor of how symlinks were resolved when traversing node_modules. At that point, I gave up and just used --no-cache. Sorry that I can't be more precise or certain, since this occurred for me several years ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests