18 Tests didn't pass!

[Student007]

Anybody out there can reproduce this ?
Maybe it is a problem by a fresh installed Upuntu 14.04 within a virutal machine (for some unknown reason).

After I got Tessel 1 CLI running on Ubuntu 10.04:
https://forums.tessel.io/t/npm-install-g-tessel-failed-in-ubuntu-14-0-4/1548/12
I did:

git clone https://github.com/tessel/runtime.git
cd runtime
make update
make colony
make test

what finally gives me
make: *** [test-node] Fehler 18

here you can find the hole compiler output:

Test.log: https://tessel-discourse.s3.amazonaws.com/943a5e783c5ec425f3de456cfea418c639d40cfeeaf.log

[Student007]

Today I got the idea it is possible the 18 failed tests causes from the 0.12 version of nodejs is istalled on Ubuntu. Today I will not have much time to setup a clean Ubuntu 14.04 with node 0.10.13 or what tessel is running. But this should give more information.
If this would show a significant difference, I would assume the failing tests don't matter because Tessel is running 0.10.13 ... hole failing tests are low level if I had seen it right.

[Student007]

EDIT: Updated on 4th of Jun 2015
When doing

nvm use 0.10.13 ## or different version
make clean
make update
make colony
make test

I will get the following results (see table)
Before links to former successful build results. Something runs totally wrong at the moment.

---

OK I found the time to analyze the reasons and here are my results:

Different numbers of failing tests causes from different versions of Node.js!

Node.js version	Number of failing tests	log
0.10.13	106 ⚡	make test log
0.10.25	106 ⚡	make test log
0.10.32	106 ⚡	make test log
0.10.36	106 ⚡	make test log
0.11.16	💥 (before: 19)	make colony log
0.12.0	💥	make colony log
0.12.1	💥 (before 17)	make colony log
0.12.2	💥 (before: 20)	make colony log
0.12.3	💥 (before: 20)	make colony log
0.12.4	💥 (before: 20)	make colony log

⚡ stack smashing detected
💥 building colony failed

I will do a pull request for updating the Readme.md to warn other contributing nice people to warn about this. Of cause you can rework the wording but please accept that pull request.

I think it doesn't help much to know the change log of Node.js.

Conclusion:

It looks like there wasn't any early sensible mind about Node.js will change fast and much. For that reason you didn't decide early enough to organize the architecture to be forward compatible. For that you later run into the trap of being forced to fix from issue to issue. Also I would expect to see no clean version dependency based on break.feature.bug for Colony what also forced you to write and change additional tests. But this is only an assumption.

[Student007]

Maybe it is a good idea to use https://travis-ci.org/ because you can test against different Node.js versions

[kevinmehall]

Colony is intended to match a specific Node version, currently 0.10.32. Since as you've discovered, the node versions are mutually incompatible, we had to choose one and match it. Our test infrastructure runs the tests on every pull request before merging it to master, and these tests are passing there. That server has Node v0.10.25 from apt-get Ubuntu 14.04.

Now, it's very possible that the tests depend too tightly on certain aspects of the system environment, and that would be considered a bug in the test suite.

[Student007]

@kevinmehall: thanks for clarification, thought it is 0.10.32 - I try to run make test of Colony against that Node.js version just for being snoopy:

*** stack smashing detected ***: ./out/Release/colony terminated
/home/daniel/runtime/tools/tap-colony.sh: Zeile 10: 16625 Abgebrochen            
(Speicherabzug geschrieben) ./out/Release/colony "$RELATIVE/$1"

*** stack smashing detected ***: ./out/Release/colony terminated

Does this cause from collisions with a tmp path while building tap-colony.sh? (same happens to 0.10.25)

I will update the above table for the log results.

Is it enough to change NODE_VERSION ?= 0.10.32 within the makefile to use newer versions of the node-libs ? It looks to me the makefile only points on the libs what will be part of Colony in some way ?!?
Am I wrong with my assumption that colony index.js is using the current local node version instead of a build node libs ?

[kevinmehall]

@tcr Any ideas about that crash?

The test suite runs the tests with both your local node installation (to verify the tests against a known-good JS interpreter) and the compiled copy of Colony. Colony uses some libraries from node, which are included in the repository in deps/node-libs. The NODE_VERSION variable only changes what version of Node that Colony pretends to be in process.versions, and the tarball URL for the update-node-libs makefile target. It can't simply be changed to an 0.12 release because Colony doesn't support JS features used by the Node 0.12 libraries.

[Student007]

@kevinmehall ... and what about to use a different version for downloading node-libs and remove or ignore those what are incompatible. Based on this it could become a more compatible framework. Maybe a blacklist would be a better solution here ?

[Student007]

updated links