Which native extensions should we package?

[FooBarWidget]

In issue #3 I have described how I envision support for native extensions to work. Since requiring developers to build native extensions themselves is too much hassle, I proposed providing precompiled versions of the most popular native extensions, that developers can just easily drop into their package.

But this raises the question: which native extensions? We can't automatically package them all. For reasons explained in #3, packaging native extensions requires a lot of manual work. So we have to choose.

I plan on packaging these gems:

Because they're popular:

• nokogori
• json

Because Rails apps likely need these:

• sqlite3
• mysql2
• pg
• bcrypt

I've opened a poll for more native extensions. Please add your input here: https://docs.google.com/spreadsheets/d/1oNgrwHyV9UaMGFTttsfA9QPvL461NWLfHeeAap3Az_A/edit?usp=sharing

[antoinelyset]

This gist seems interesting to list all your gems with native extensions :

require 'rubygems'

specs_with_extensions = Gem::Specification.each.select { |spec| spec.extensions.any?  }
specs_with_extensions.each do |spec|
  puts "#{spec.name} (extensions: #{spec.extensions.inspect})"
end

(source https://gist.github.com/aelesbao/1414b169a79162b1d795)

P.S: I don't know if this is wanted, but your Google Doc is restricted.

[FooBarWidget]

The doc is supposed to be public. I've fixed that and updated the link.

I know I can lookup all gems with native extensions, but I cannot package all gems because it's too much work. The best thing I can do is package the most popular ones, hence the poll.

[sheerun]

@FooBarWidget Can't packaging be automated for any gem? Do you need to choose?

[FooBarWidget]

No it cannot be automated. See #3 (comment) where I said:

I think it is impossible to generalize and to automate this. The current Traveling Ruby binaries are hand-tweaked to ensure that they are portable and have no dependencies on non-system libraries. I think it is impossible to guarantee this for gems: to make the resulting binary portable would probably require a lot of manual work.

Packaging nokogiri and sqlite already took me a lot of manual work. I had to install libxml, libxslt and libsqlite3 from source, compile them with very specific settings, edit Makefiles everywhere to ensure correct linking flags, and do this twice (once for Linux, once for OS X).

So yes, I have to choose.

[drnic]

The set above includes my requirements for BOSH CLI https://github.com/cloudfoundry/bosh/blob/master/Gemfile.lock thanks!

[FooBarWidget]

@drnic Thanks, but can you help me by filtering out the gem names that contain native extensions, and modifying the spreadsheet? :)

[sheerun]

I'd vote for porting https://github.com/ffi/ffi

[FooBarWidget]

@sheerun Ruby 2.1 has Fiddle: http://ruby-doc.org/stdlib-2.1.0/libdoc/fiddle/rdoc/Fiddle.html
Doesn't that make ffi obsolete?

[sheerun]

There are many many existing gems that use ffi/ffi

[drnic]

I was linking for completeness. I think nokogiri, mysql2, sqlite, pg are the native gems

On Wed, Dec 10, 2014 at 8:48 AM, Hongli Lai notifications@github.com
wrote:

@drnic Thanks, but can you help me by filtering out the gem names that contain native extensions, and modifying the spreadsheet? :)

Reply to this email directly or view it on GitHub:
#6 (comment)

[noahgibbs]

Also, gems that want to be compatible with Ruby versions before 2.1 still have to use FFI for awhile. I also have no idea where JRuby is on implementing Fiddle, which may matter for compatibility as well.

[FooBarWidget]

ffi, mysql2, pg and bcrypt have now been packaged.

[youngbrioche]

Nokogiri should definitely be on the list.

[FooBarWidget]

Nokogiri is already packaged. See spreadsheet.

[drnic]

And sqlite3 please?

[FooBarWidget]

Already packaged. See spreadsheet.

[Nowaker]

Great work guys. Traveling Ruby saves us working on VirtKick a lot of work. :-) Thanks for this!

[FooBarWidget]

@Nowaker Glad you like it. :) I'm compiling a list of Traveling Ruby users. Could you write a short ~50 words introduction about Virtkick, and could you write another ~50 words paragraph in which you describe how Traveling Ruby has helped you?

[Nowaker]

@FooBarWidget Sure, no problem!

VirtKick in 50 words: VirtKick is a self-hosted cloud panel, similar to DigitalOcean. With privacy in mind we simplify creating, managing, hosting and providing virtual servers. Forget about installing packages or editing configuration files - all is set up automatically and ready to use. 100% open source.

VirtKick + Traveling Ruby: @Rush will provide a nice text, he did the hacking. :)

[drnic]

@FooBarWidget https://github.com/cloudfoundry-community/traveling-bosh is using it too

BOSH is a Cloud Foundry project for release engineering, deployment, and lifecycle management of large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs and performs monitoring, failure recovery and software updates with zero-to-minimal downtime.

[Rush]

@FooBarWidget: Here is how traveling-ruby helps VirtKick:

VirtKick's goal is a one click installation via standalone installer and/or a distro package. We use rails and other dependencies, many of which we have at bleeding-edge versions from git. traveling-ruby bundles it all. Without it, the installation process was long, required a network connection and lots of hard dependencies.

[FooBarWidget]

@Rush @drnic Thanks. :) Could you also provide photos, icons or pictures that I can use? Should be at least 256x256 pixels.

[Nowaker]

@FooBarWidget Some logos are here: https://press.virtkick.io/presskit. Let me know if it's enough, or you need something more.

[FooBarWidget]

@Nowaker Looks good. Should I quote you or @Rush? What are your positions within your organization?

[FooBarWidget]

I've added your testimonials to the website, but I've edited them a little bit: http://phusion.github.io/traveling-ruby/
Is this ok?

[Nowaker]

@FooBarWidget Looks good, thanks. :-)

[drnic]

Is it possible to include https://github.com/brianmario/yajl-ruby?

Sorry I didn't spot it in https://github.com/cloudfoundry/bosh/blob/master/Gemfile.lock#L31 before as a native extension.

[FooBarWidget]

RedCloth, escape_utils, posix-spawn, nokogumbo and github-markdown have been added because they're needed by Octodown.

[FooBarWidget]

rugged has been added.

[FooBarWidget]

charlock_holmes has been added.

[etdsoft]

Hey Hongli, do you plan to push a new release now that all the Octodown gems have been added?
We're planing to use traveling-ruby for the next release of Dradis Framework too. This is where all the RedCloth stuff was coming from.

[FooBarWidget]

Soon. I work on Traveling Ruby occasionally during the weekend, but during the week most of my time is occupied by Passenger maintenance/development as well as business operations.

[etdsoft]

Of course, makes sense, no rush.

[FooBarWidget]

The unf_ext gem has been added. It's used by Elasticrawl.

[FooBarWidget]

Traveling Ruby 20150130 is out. Next version is planned to use Ruby 2.2, as per GH-28.

[etdsoft]

excellent!

[ghost]

what about Puma?

[FooBarWidget]

Added puma, unicorn, kgio, raindrops, fast-stemmer, hitimes, redcarpet.

[ghost]

awesome!
thanks.

[ghost]

ok, i see puma, unicorn etc. in shared/gemfiles/20150210-next of next branch.
would they be available on s3 any soon?
thanks for your efforts.

[Nowaker]

I added nio4r to the list. This is a dependency of Celluloid::IO which is great for making concurrent HTTP requests.

[Nowaker]

@FooBarWidget The same as @sleewoo - when can you release binaries for puma? The spreadsheet says "Already packaged" but Puma is not available at http://traveling-ruby.s3-us-west-2.amazonaws.com/list.html.

[FooBarWidget]

It's packaged in the development branch. I'm sick right now and there are lots of high-priority non-Travling-Ruby things on my todo list, so I'm afraid I can't give you a time indication right now.

[Nowaker]

Are there any binaries for the development branch?

[FooBarWidget]

No. I build it from my laptop on every release. If you need binaries now, you should run the build scripts yourself.

[ghost]

i was able to build everything in next branch.
could provide output if interested.
extensions are full of backdoors of course :)

[michaelbarton]

Could it be possible to provide a native extension of gherkin, the gem that is used for cucumber? We would like to share our test suite for bioinformatics tools powered by cucumber. I gave this a try today but I saw that gherkin is a native extension.

[FooBarWidget]

I'll look into it. I've scheduled some time for Traveling Ruby maintenance work this week. Right now I'm blocked by https://bitbucket.org/ged/ruby-pg/issue/219/please-release-a-new-gem.

[michaelbarton]

Thanks. I took a look at the code for building native extensions. I think I almost got it to work, I was able to get everything to appear to compile but I wasn't sure about where the final output was produced, or if it was produced. I think with a little more documentation I think I might have been able to do this myself.

[FooBarWidget]

@michaelbarton The build output is in the 'output' directory. The 'package' script packages the output into tarballs.

Are you talking about gherkin or gherkin3?

[michaelbarton]

I think gherkin. From what I understand gherkin3 is still in development.

[FooBarWidget]

Ok, version 20150517 is released now, with support for many extensions requested in the past 2 months.

[dapicester]

I added websocket-driver. It was extracted from Faye to provide WebSocket support for Ruby.

[ashes999]

Hi,

I added gosu. Like most remaining gems on the list, it only has one vote (mine). If you can get back to me about whether this is feasible or not (not actually building the binaries, but assessing it), that would be great @FooBarWidget

[Nowaker]

I've added concurrent-ruby to the list. It's become a very important library now that Sprockets and Sidekiq depend on it.