Binary packages

[sstephenson]

I'd like to add binary package support to ruby-build. I'd guess that most ruby-build installations are on one of two platforms—Mac OS X x86_64 or Linux x86_64—and most people really don't need to compile their own Ruby runtimes. Binary packages would make it much quicker to get up and running.

In order to do this, we'll need a command to build a definition into a temporary directory and make a tarball. We'll also need a way to download and install package tarballs from the internet. This implies having package repository sources, an architecture naming system, and an index of packages with checksums.

[sstephenson]

Looks like --prefix is used to define the initial load path. It's compiled into the ruby binary itself:

$ strings bin/ruby | grep rbenv
/Users/sam/.rbenv/versions/1.9.3-dev
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1/x86_64-darwin11.1.0

Obviously these paths aren't going to be the same for everyone. Can we work around this?

[sstephenson]

Crazy idea: build Ruby with an unreasonably long --prefix, like

/ruby-build/placeholder----------------------------------------------------------------/1.9.3-rc1

...and search-and-replace that with the actual installation location after extracting the package.

We'd need to do this for the shebangs for commands like irb too.

[josh]

Skeptical about that. Compiled constants are going to be offset by their length. So replacing the placeholder with a shorter string is probably going to fuck up the memory addresses off all the other constants after it in the data segment.

[sstephenson]

We'd keep the strings the same length, obviously, and just replace the rest of the space with null bytes.

Proof of concept: https://gist.github.com/a95fd5e818f9237a1c9f

Save as rewrite.sh and run like so:

$ ./rewrite.sh < bin/ruby > bin/ruby2

[clowder]

I was digging round the MacRuby source today and came across install_name_tool and instantly thought of this ticket.

On MacRuby it is being used to replace linking to the system MacRuby with one packaged inside the executable (see MacRuby/MacRuby@master).

EDIT: Also useful headerpad_max_install_names

[fschwahn]

so... what happened to this? isn't it ready to go, I remember you tweeting about this a few month ago? if not, is there anything that needs to be done?

I really like the prospect to install ruby in a few seconds instead of a few minutes :)

[john-griffin]

Is there anything we can do to help out with this and get it finished?

[haberbyte]

Here is an interesting blog post: http://yehudakatz.com/2012/06/

I think Yehuda has solved that problem already!

[petejkim]

is it possible by any chance to use rvm's precompiled packages?

[rosenfeld]

When using precompiled packages ruby-build should probably build them with optimization flags (-O3 or -O2) turned on.

Also, they should be also the default when compiling from source. In that case, it would be a good idea to also add -march=native.

[pranas]

Any progress? It's highly desirable feature.

[ptrf]

👍 Any progress? Post a gist maybe?

[barttenbrinke]

+1 Rubber (Ec2 automation) recently switched to Ruby-build (which is great!) but compiling Ruby 2.0 for each Amazon instance you spin up is something the world can really do without.

[loopj]

👍 would love to see support for this

[mattes]

+1
see here also:
tokaido/tokaido-build#6 (comment)

[md5]

@mconigliaro If you want to do that, you can use rbenv by itself and not use ruby-build since its purpose is to build Ruby for rbenv.

From the rbenv docs:

Alternatively to the install command, you can download and compile Ruby manually as a subdirectory of ~/.rbenv/versions/. An entry in that directory can also be a symlink to a Ruby version installed elsewhere on the filesystem. rbenv doesn't care; it will simply treat any entry in the versions/ directory as a separate Ruby version.

[mconigliaro]

@md5 oh yeah, that's a good point. I could always just have my config management system download and unpack a set of precompiled versions to ~/.rbenv/versions/. Thanks!

[jasonkarns]

Super old issue, but I just wanted to pop in and say that node-build (direct fork, minimally modified) has added support for installing precompiled binaries. nodenv/node-build@474e872 (source has been modified slightly since that PR was merged, though)

We grab the system info via:

platform(){
  local arch="$(uname -m)"
  local os="$(uname -s | tr '[:upper:]' '[:lower:]')"

  case "$arch" in
    x86_64 | amd64 | i686-64 ) arch=x64 ;;
    i[36]86* | [ix]86pc )      arch=x86 ;;
  esac

  echo "${os}-${arch}"
}

And a build definition looks like:

binary darwin-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-darwin-x64.tar.gz#658686795fac9669d411ac5c5be2da8159058e386767322d8d8151dcdb4810b9"
binary linux-arm64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-arm64.tar.gz#05df4aeb8a53798f8b10074600518040fc317f2919f9755aeab57b0aaf7227b0"
binary linux-armv6l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv6l.tar.gz#6f29286464efda4a362d9464d82733398b98882051c20c0f190004480b6e506a"
binary linux-armv7l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv7l.tar.gz#fb4fbef9306962e800804ab5ba615c06bba28deb4e7a3e945a291dba986ef816"
binary linux-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x64.tar.gz#e766e387934e17daaad92d0460ed76f756655da62b627a5c9cc07faea4a0b824"
binary linux-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x86.tar.gz#97b5ccea7044073c87a21bcc4b0762f4a6bd77db9cc958206f684ecdfeb89b1f"
binary sunos-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x64.tar.gz#acec44790ff0069620c0fd03945d14b9f97c7ccb0a0450f766a5cbe4a906510b"
binary sunos-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x86.tar.gz#d7bf8dcf353115b5e55bf64d25c2c34fa2d237e201e46dab97ae3d8e3f051583"

install_package "node-v4.2.1" "https://nodejs.org/dist/v4.2.1/node-v4.2.1.tar.gz#8861b9f4c3b4db380fcda19a710c0430c3d62d03ee176c64db63eef95a672663"

How it works:

The binary function is invoked multiple times from the build definition; given the platform and package url. If the binary's platform matches the current system, the package url is saved to a global variable.

Then when install_package is invoked, if a global BINARY_URL has been set (and the user hasn't forced a compile with --compile), it does an install_package ... copy.

It's a pretty hacky workflow, and I would love to find a more elegant solution. But it works and, I think, could be adopted for ruby-build.

[mislav]

@jasonkarns Thanks for sharing! That's an interesting approach to fetching precompiled binaries.

In my opinion, it's not such a problem to make ruby-build fetch precompiled binaries as it is to make a system to package and publish precompiled binaries in the first place.

[koenpunt]

Travis actually has precompiled rubies: http://rubies.travis-ci.org.

Edit; I believe these are build against local libraries, so are not universally usable;

dyld: Library not loaded: /Users/travis/.rvm/rubies/ruby-2.3.3/lib/libruby.2.3.0.dylib
  Referenced from: /Users/xcode7/.rbenv/versions/2.3.3/bin/ruby
  Reason: image not found

[AlexWayfer]

No progress here? Or any new ideas?

[wlnirvana]

any update?

[rafaelsales]

Any progress? RVM makes ruby binaries for ages and it's just fine

[mislav]

No progress yet. Anyone is welcome to pick this up!

Some notes:

• how rvm builds "movable" ruby
• how Travis builds them (uses rvm)
• how we would build them, in theory:
  1. In our CI, use build flags to make rubies "movable",
  2. (Non-critical) strip unnecessary files from the resulting build,
  3. Upload packaged binaries up to S3,
  4. On the user end, allow an opt-in flag ruby-build --binary 2.6.5 /path/to/destination that first tries to download and extract pre-build binaries,
  5. Eventually, if all this process yields stable result, enable --binary flag by default so everyone can benefit from fast Ruby installs on supported platforms (probably macOS plus several propular Linux distros).

[mattes]

Upload packaged binaries up to S3

Can we re-use http://rvm.io/binaries (cached by fastly)? Unless someone else wants to pay for the S3 traffic :-)

[technicalpickles]

Posted this on a duplicate #1054 :

GitHub packages is an option now too. Homebrew recently moved to it from bintray recently.

That doesn't address building the packages though.

Some notes from reading through about how RVM does it:

• rvm has a --movable flag
• it's documented at https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/help/install.md#movable-rubies
• there is logic for checking which environments and ruby versions can be built movable: https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/scripts/functions/build_config#L32-L77
• when it comes to building:
  • on macos, there's a patch applied for static building, and a set of configure flags
  • otherwise there's a configure flag (--enable-load-relative)

Edit: correction, the osx patch is empty:
https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/patchsets/ruby/osx_static

[technicalpickles]

Some more thoughts after the initial work and some time to think...

I'm considering if there should be a separate command for the fetch binary ruby from source or not. I'm also thinking through how we'd do checksums on the tarballs.

For hosting of them, homebrew recently moved from bintray to GitHub Packages. It uses skopeo to work with remote images registries (which GitHub Packages is), like retrieving information, images, signing content. I'm not sure how they are building the tarballs to publish yet though.

The big question there for me is how to organize things, and how to automate building of the various versions. My initial thought is like:

• add a 'package' for each ruby+version combo on the rbenv org, ie ruby/3.0.1
• use a timestamp and/or sha of ruby-build for to allow rebuilding when there's an update
  • I don't think they always need to be rebuilt, but I suspect it will be necessary to push an update if there's a bug in ruby-build. I don't know how to manage that without requiring always building (and thus always having updates) or manually doing it

Something I'd like to enable as well is to allow organizations to be publishing their own builds, like if they have custom patches or configuration flags they use. This would also give the option to experiment with this before having a working solution for the hosting side of things.

[wlnirvana]

Any update on this? Ten years have passed since this issues is raised🤦

[eregon]

It's pretty hard and fragile to create movable CRuby builds, see #1761 (comment)
So basically binary builds of CRuby are a problem, they need to be at least (OS, OS version, C compiler path, system libssl version and path, architecture, destination path and probably more)-specific.
And since ruby-build allows installing to any path it could only use those if it's all matching.

IMHO it's fast enough and more reliable to build from source.
If the user's setup is messed up for compilation it won't work when installing C extensions anyway.

[aleksandrs-ledovskis]

It would definitely be a bonus if "ruby-build" provides pre-built packages for multitude of aforementioned OS runtime combinations. However, the situation 10 years ago vs. today is quite different with widespread proliferation and usage of containers which subsequently reduces a need to have precompiled interpeters for OP's stated use case "[b]inary packages would make it much quicker to get up and running".