Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to rebar3 or mix #1428

Open
wohali opened this issue Jul 9, 2018 · 13 comments
Open

Migrate to rebar3 or mix #1428

wohali opened this issue Jul 9, 2018 · 13 comments

Comments

@wohali
Copy link
Member

wohali commented Jul 9, 2018

We're long past due upgrading to rebar 3 (latest).

This requires using @davisp's erlang-native-compiler to continue to build our NIFs.

@tsloughter
Copy link

You might find it easier to use https://github.com/blt/port_compiler/ instead of having to call out to the external enc escript just to build NIFs.

@davisp
Copy link
Member

davisp commented Mar 18, 2019

The issues I had trying to use port_compiler is precisely what motivated me to write enc in the first place. I haven't paid any attention at all to changes to port_compiler in the meantime but that's precisely because I've not had any issues reported with enc in the nearly two years since I last touched it.

I'm not against changing back to port_compiler if someone wants to take a stab at it.

@nickva
Copy link
Contributor

nickva commented Mar 18, 2019

I started on this a while back:

cloudant@b0f3fee

Besides the port compiler issue the other one was using raw dependencies (Fauxton and docs). rebar3 doesn't do those by default (or didn't do when I checked): erlang/rebar3#110 There might be a plugin to handle those. The other option could be to make a "fake" Erlang OTP directory structure in those dependencies.

@tsloughter
Copy link

Yea, there is a raw resource plugin that can be used.

@iilyak
Copy link
Contributor

iilyak commented Mar 18, 2019

I think we should consider trying out mix (Elixir build tool) instead of rebar3. Although we would have same compatibility issues as with rebar3 (NIFs and raw deps). Nevertheless I find mix implementation faster and easier to hack on. I am happy to go with any direction the community would decide to pursue.

@garrensmith
Copy link
Member

garrensmith commented Mar 18, 2019 via email

@eiri
Copy link
Member

eiri commented Mar 18, 2019

I'd +1 for trying mix too. I don't have an opinion on it itself, but I personally don't find rebar3 a significant improvement over "classical" rebar.

@tsloughter
Copy link

:(

I don't work on couchdb so my opinion doesn't matter here, but I am always interested in ways to improve rebar3, so please pass along (probably not on this issue unless it somehow directly relates to the question of couchdb migration) any suggestions or complaints (particularly how it doesn't do better than rebar2 which was one of the main goals and why it was essentially a rewrite).

rebar3 is fairly different from mix, so personal preference for one or the other over particular decisions we made differently is an understandable separation that can't necessarily be bridged, but always interested in hearing about them.

@wohali wohali changed the title Migrate to rebar3 Migrate to rebar3 or mix Mar 18, 2019
@wohali
Copy link
Member Author

wohali commented Mar 18, 2019

@tsloughter Thanks so much for stopping by!

The big issues for us when rebar3 dropped were problems with NIFs (why @davisp had to build enc.), coupled with the problems Nick mentioned on raw dependencies, and some Windows compile issue (probably NIF related) I ran into that I can't remember. Because that was early and we were waiting, you have probably fixed some or all of these by now, but this hasn't been a high priority ticket. It's becoming more important as we look at architectural changes to CouchDB and newer versions of Erlang.

While mix is interesting, it matters more in my opinion if we're going to start accepting Elixir as a core language, not just for our test suite. Either way, I don't have the cycles to drive the work on either, so I hope someone picks up either and runs with it.

@iilyak
Copy link
Contributor

iilyak commented Aug 15, 2019

@davisp
Copy link
Member

davisp commented Aug 15, 2019

Looks like I never actually described my original issues with the port compiler very well. But the gist of it was that projects that had a dependency that depended on Jiffy would fail to get that NIF built correctly. There is/was also a confounding issue of making it work with rebar2 since CouchDB hadn't moved off it. Creating the standalone enc script ended up being the easiest solution at the time. Its been years so its quite possible that's all fixed now.

As to mix vs rebar3 in particular, the only worry I'd have is that Elixir is going to tighten our acceptable versions of Erlang. And in my completely limited experience it seems as though they tend to follow the newest releases quite closely. But that's super anecdata so I could be way off there.

@iilyak
Copy link
Contributor

iilyak commented Aug 16, 2019

As to mix vs rebar3 in particular, the only worry I'd have is that Elixir is going to tighten our acceptable versions of Erlang.

I did use erlang 20.3.8.14 and elixir 1.6.6. However 1.6.6. is supported on erlang 19 which is AFAIK a minimal erlang version CouchDB supports.

@iilyak
Copy link
Contributor

iilyak commented Jul 29, 2022

Just wanted to provide update on this one

@jiahuili430 tried to switch to rebar3 here #4120 and here #4089

I took @jiahuili430 work and tried to DRY the rebar.config.script and implement release generation here. The problem I faced is that rebar3 assumes every application either in the same repository (monorepo) or it is a dependency. In case when it is a dependency rebar3 fetches it into the _build/<profile>/lib/<app>. This means that if we create a source tarball and copy sources of dependencies into src we would need to update rebar.config. Which means two separate rebar configurations for development and building from a source archive.

In order to solve this issue I implemented a plugin which allows rebar3 to copy source files to _build/<profile>/lib/<app> from either src/<app> if it is present or fetch from git otherwise. The plugin is based on https://github.com/benoitc/rebar3_path_deps.

Another issue was the boot_script generation. Currently we use reltool here. Rebar3 don't use reltool and uses relx for release generation. To avoid maintaining two configurations I wrote a plugin to generate boot file.

dependency definition problem: However the work not yet complete. Currently it complains about cyclic dependencies.

===> Verifying dependencies...
===> Dependency cycle(s) detected:
applications: b64url bear config ets_lru folsom hyper ibrowse jiffy khash meck mochiweb recon snappy depend on each other

This is because the same dependencies defined twice. Once in applications' rebar.config files and once in the main rebar.config.script. I see two options to resolve it.

  1. define dependencies in each app directly. This would mean that dependencies would be in multiple places. Also I expect problems when we would want to build from released source archive.
  2. implement plugin which would take the dependencies from main repository and inject overrides into the applications which refer to them in their <app>.app.src files. Effectively, it would be the same as 1) but configuration would be maintained in one place.

I also tried experimental vendoring mechanism in recent rebar3 versions. Unfortunately it doesn't do what we need. More experimentation is needed after the dependencies definition problem is sorted out.

Unfortunately, I cannot allocate more time into it at the moment. So I decided to document my current results in a hope that it would be helpful to someone who would try to do this transition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants