Guix, grafts, gee… #96
Comments
|
I've only followed this repo on the side and I don't really understand much about it. I'm only interested in building nice GTK apps using GNU Guile at some point. That said, since you already started explaining, let me ask some uninformed, possibly dumb questions : )
|
|
|
Wait, shouldn't Guix grafting also graft guile-gi (it should actually edit the shared objects in the derivation, too)? Then everything would be fine. Does this problem actually happen when using an installed/dependent guix guile-gi package? Right now I'm always using a manual git checkout of guile-gi, so that can't be grafted of course (because guix doesn't know that that checkout exists). Also, I recompile it very rarely, even as I update guix (read: it's stale). I don't want to be the one responsible for a lot of unnecessary rework. Please make sure it's actually required and I've not been making the problem seem worse than it is. |
|
You are not the only one responsible for this. I use Guix myself as a basis for developing this library and am very weirded out by having to resort to such hacks. There is so far no precedence for Guile-GI packages in Guix, which might also have to do with the fact, that the guile-gi recipe on Guix was rather broken for a long time (is it fixed now? I don't remember). I would assume some weird workaround would be required to get them to run as with all the PyGI and GJS packages. |
In order to debug this, I'd |
|
I'm sorry to say that, but I don't get any meaningful results (or even results at all) from adding this to LD_PRELOAD. It doesn't even appear to execute at all. |
|
I do. Result: dlopen is sometimes called without full path (for example: New version (inside Create Then compile it via Then create I also edited Then Then I got this: |
|
Reading the source code of gobject-introspection, they do Aaaand that was patched by Guix. I suspect it gets into the A kingdom for a |
|
I think I have something simpler: #define _GNU_SOURCE
#include <dlfcn.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
typedef void *(*dlopen_t) (const char *filename, int flags);
#ifndef DLOPEN_BREAK_KEY
#define DLOPEN_BREAK_KEY "gobject"
#endif
void *dlopen(const char *filename, int flags) {
fprintf (stderr, "dlopen %s", filename);
dlopen_t dlopen = dlsym (RTLD_NEXT, "dlopen");
if (strstr (filename, DLOPEN_BREAK_KEY)) asm volatile ("int $03");
return dlopen (filename, args);
}That allows you to GDB into the dlopens of any particular shared library simply by defining |
|
Good idea! I've read through gobject-introspection source code by now and it seems that gobject-introspection upstream take it upon themselves to provide gir files for a few other libraries (like cairo--see But if I understand the patch that Guix did to gobject-introspection's build system (not pasted above) correctly, then they only pick up libraries in the output of the package currently built. Well, currently we are building The reason why it then doesn't fail nicely at startup of guile-gi as it should is because I have cairo in my profile (it was probably propagated by something--I can't remove it). Other gir files provided in gobject-introspection are: It's possible to manually specify In any case, I think this is a Guix bug (at least additionally). |
|
This is somewhat off-topic, but I think the issue becomes clear if you do the following: Perhaps Guix fails to graft typelibs at all? In that case, we might as well file an upstream bug, but the fundamental issue remains, that we can't load any GObject other than the one we're linking against. Interestingly |
|
What does
say? FWIW, for me, there seem to be references to absolute paths of libgobject in the I'm 100% sure I found how the weird reference got in. The problem is making Guix find the problem automatically. It's because: (1) I don't update my user profile often Next I'm trying to fix Guix's gobject-introspection to FAIL instead of embedding relative references if built inside a guix build container. |
|
Hmm, it appears the gir inside Your venture into the depths of cairo is nice and all, but keep things simple and use |
|
The point is that the "depth of cairo" loaded another version of glib. So that's where another version of glib comes from. I'm trying |
|
The thing is, you don't need cairo to load a different version of GLib. It already happens with GObject alone. |
|
I agree. But what guix's gobject-introspection does is a wrong thing in general, and libcairo-gobject is being loaded by guile-gi right now (note: I did not directly use cairo anywhere). That is not going to end well. Fixing it might fix this problem and other potential problems, too. (I tried making gobject-introspection depend on cairo by now. That causes a circular reference. Sigh.) In any case, trying insanity.scm now. |
|
Ohhh, libguile-gi links to libgobject at compile time, too. Well, that's gonna be a problem if that library is different to the one dlopen'ed... |
|
You'll get cairo through GTK, Pango, and many other graphical stuff and the recipe seems to contain gtk+ even though it is afaik not strictly necessary. I highly doubt you'll find a fix to this in Guix. It is likelier that whatever change you make breaks something in PyGI or GJS instead, so please be cautious. A patch to dynamically link libguile-gi against GObject etc. would be very welcome on the other hand. |
Right now I'll settle for a procedure to flag this problem in Guix--nevermind fixing it.
I don't know at all how something like that would look. gobject-introspection seems to require glib (see below). Things it loads then also depend on glib, but not necessarily on the same version. That is not going to end well. I don't think that this can be fixed in guile-gi. It can be fixed in gobject-introspection (by changing its architecture), I guess. |
|
|
|
ldd shows what would be loaded when loading this
So does gobject-introspection, and that's just as wrong. |
|
In the end this is the usual recursive definition problem. I don't get why people always do that--that has to lead to problems sooner or later. For example having a compiler written in the language of that compiler, or xslt (which is a language for transforming xml) specs itself written in XML etc. In this case, gobject-introspection is supposed to make glib usable in target languages other than C. In a sane world that would mean that a binding generator doesn't use glib as a non-native input. Or if it absolutely has to (it really shouldn't [1]), then at least it wouldn't expose that glib or any of its contents to target language users (because philosophically, that's just wrong--even if it happens to work sometimes), i.e. it should be a native-input. But no, gobject-introspection has glib as a regular input. (Trying to move gobject-introspection to native-inputs, I get If it has to do stuff like that, you'd think at least it would have two levels: a meta-level where it mangles definitions of glib, and that just happens to use glib for the mangling internally (but not ever expose that glib or any of its objects directly to the user), and another normal level where it actually provides glib to the target language. But no :( Personally I'd write a replacement for libgirepository that just parses the girs or typelibs on its own for guile-gi (not using Because what gir files do is describe glib at a C level. The interface of the glib is gobjects, but the implementation of glib is C (Pascal got this right in 1970, only to be ignored by almost everyone). So there's no reason for a binding generator to depend on glib at all--it would have to use the gobject interface if it did. Of course there'll be some "syntactic sugar" provided for the target language depending on the library when it is loaded--but that's pretty much it. I remember pygtk doing this right ("override" files) a long long time ago. An easy way to catch those problems early-on is to try to cross-compile your program. If it tries the self-reference outlined above, that would cause a failure (which is what you want) because the architectures of the parts don't match. [1] because eventually it will become part of glib, and what then? |
|
I'm not sure if this adds any useful information to this discussion, but, on Linux, you can pull the results of dlopening in the following fashion. Compile this with |
To guess the level of effort, I built My rough estimate was done this way, after removing glib from meson.build |
|
Not as far as I know, but I'm personally not convinced that this is the right move here. Philosophically speaking, it wouldn't be much of an introspection if one was doing it from the outside, would it? I've had a short look at G-Golf and they seem to be doing stuff similar to us in that they partly export the base GLib that they get, but I assume this is fine for them, since they use dynamic links for GObject. That being said, I still don't have any experience with using G-Golf as a library, but there seem to be projects built on it in Guix, so it appears likely to work out that way. TL;DR: Dynamic linking would probably be at least a short-term solution. For the long term we should think about what "introspecting your twin" really means. @daym Do you count |
It's doing it from the outside anyway because C has no introspection (and neither does glib, generally--except for small islands). They can be faking it, but that doesn't change this fundamental fact. But I know what you mean: gobject-introspection itself wants to be a gobject. But if gobject-introspection itself wants to be a gobject, then it should be made impossible to load another glib using gobject-introspection (making the rest of glib similar to what compiler or shell builtins would be).
Yeah. In the end I don't see how gobject-introspection can work reliably on Guix like that--nevermind guile-gi for the time being. The easiest way to find out in detail what is what would be to remove glib from the dependencies of gobject-introspection (and their header files) entirely and stub the things below like written below (also remove If not, add glib back to the package dependencies but remove Object-like glib interface types that even gitypelib-internal.h uses: And the public interface of gobject-introspection has: Primitive glib interface types which are used and fine to use since they have obvious definitions and shouldn't change (and could be just be defined manually): I'm all for dynamically loading stuff from glib in guile-gi, but I just want to make sure first that this actually fixes the entire problem. Otherwise the problem will be back, one library over there. (Also, the "cairo" problem won't vanish: gobject-introspection totally can load yet another glib version when traversing through to cairo, even after all those fixes. And it does traverse there. That could also happen with other libraries like gnome highlevel libs etc. I don't want to single glib and cairo out--it's just an example) All in all, I wonder if GNOME can be made to work reliably like Guix wants it to at all. After all, gobject.so has a central type registry and thus having two gobject.so loaded (even indirectly) in the same executable is going to make things weird... |
|
Well, this has been sitting here for a while. As someone who rarely uses Guix, I have a simplistic understanding, but, I gather that a step toward improvement could be something like this:
|
|
That would be a big step towards improvement, as far as I can see, yes. The open question (whether libgirepository allows us to load a GObject typelib other than the one it was linked against) does remain, but there's no way of answering that other than trying to implement such a thing. |
The only functions that need default linkage are those called with load-extension. The new macro GIG_API explicitly marks a function as visibile. * src/gig_visibility.h (GIG_API, GIG_LOCAL): new macros * src/gig.h: new file, make gig_init_visible * src/gig_document.h: make gig_init_document visible * src/gig_callback.h: make gig_init_callback visible * src/gig_closure.h: make gig_init_closure visible * src/gig_logging.h: make gig_init_logging visible * src/gig_object.h: make gig_init_object visible * src/gig_repository.h: make gig_init_repository visible * src/gig_type.h: make gig_init_types visible * src/gig_value.h: make git_init_value visible * src/gig_util.h: updated * src/gig.c: use new header * src/gig_logging.c: use new header * src/gig_document.c: use new header * Makefile.am: add new header files, don't export gig_* functions by default (libguile_gi_la_CPPFLAGS): new macro BUILDING_GIG (CFLAG_VISIBILITY): new macro
* src/gig_logging.c (gig_log_custom_helper): use strcmp instead
* src/gig_util.h: declare xstrdup * src/gig_util.c (xstrdup): new function (g_registered_type_info_get_qualified_name): use xstrdup * src/gig_function.c (create_gsubr): use xstrdup * src/gig_callback.c (callback_binding_inner): use xstrdup * src/gig_arg_map.c (arg_map_entry_init): use xstrdup
* src/gig_argument.c (scm_to_c_string): use xstrndup * src/gig_util.c (xstrndup): new function * src/gig_util.h: declare xstrndup
* src/gig_logging.c (gig_log_writer): use getenv
|
Well, I did do some hacking in a branch called type_init_noodle2 It doesn't solve the problem but it does take a couple of small steps in that direction
It does appear that to solve the problem, the right approach was as previously suggested: reimplementing GIRepository itself. |
|
As a stylistic choice, I think we ought to avoid "_pub.h" and instead put private implementation details into a _private.h, as is the normal GNOME convention. I haven't had too much of a look at the internals, but structurally speaking, this might be a improvement. Yet, as far as linking is concerned |
|
OK. From here, split the libguile-gi.so into two separate shared libraries: a preprocessor of typelib, and a runtime. The preprocessor will link to The runtime will only depend on
The number of GLib functions required for SCM/C arg conversion stands at 140 in my hacking branch, down from 179 in version 0.3.2. This would require replacement of libgirepository functions
|
|
I'm not sure whether we can replace the typelibs with their XML counterparts. At the very least, that'd be difficult w.r.t. environment variables. What we could do OTOH is moving from GIRepository's internal representation to the one we actually require (which we could describe in XML, JSON, what have you) by first having a GI-based library write that internally with the rest of our libraries consuming it and then have it generated by a pure C/Scheme implementation of said library. WDYT? |
|
Well this branch I've been playing with https://github.com/spk121/guile-gi/tree/split-parse-runtime is trying to create an intermediate representation in the hopes of splitting the C library in twain. The branch is totally broken at the moment, but, runs enough to generate, and then parse intermediate code like the following |
|
OK, at this point, the split-parse-runtime branch has split libguile-gi in twain: a libguile-giparse and a libguile-gi. Anything having to do with gobject-introspection of libgirepository is in the former, removing the dependency on libgirepository on the latter. At the moment, the parser calls the runtime directly, but, by calling From here, there are just ~140 GObject/GLib calls remaining on the runtime side. These are all present for SCM-to-C conversion for function arguments, or to do GType-to-SCM class conversion. It should be a rote task to dlopen/dlym those at runtime after loading the user's chosen version of GObject/GLib. |
|
OK. The latest commit at https://github.com/spk121/guile-gi/tree/split-parse-runtime sketches out a solution to this bug. It is very rough, but the outline is all there. It passes most of
You can still use But that tree is a huge mess. It has some ideas I started and later abandoned. I'm going to rewind, rework, and make a sensible patchset in a new branch. From there, a problems remain
|
|
I think we might still be duplicating some work here in that we need to actually read and write data to disk a few times rather than hadling things in memory. The guile You are right in that Long term however, it would be better to bring everything back into one compilation tower, with the |
When I experimented, I found that
This makes sense. I wonder if there are 32-bit/64-bit differences in typelib files. I don't know.
One could be quite meta, and use the current |
Note that comile-file uses the language printer of the target file and passes
They do actually describe their file format [1,2]. Only the endianness appears to make a difference, and in a cross-compiling architecture that ought to be the target endianness.
I don't quite know how to interpret this. Do you mean we'd only implement enough GI parsing to load GIRepository and then hand things off from there (similar to But before we're tacking on features upon features, I think it is time to refactor and make what we have currently work in the way we want. This would at the very least also include a lot of (shell) tests for the gi-parse part. Integration tests would also be nice, but I don't think we could put those into CI, can we? [1] https://developer-old.gnome.org/gi/unstable/gi-GITypelib-Internals.html |
Part of our design goals seems to be building against one version of GLib/GObject/GI, while allowing users to load any version. Today I am here to tell you, that this is extremely broken.
Why Guix?
Guix is just the messenger, but there is a stronger reason behind why GI appears rather broken in any Guix package, and that is grafts.
What are grafts?
Grafts are Guix' way of not rebuilding the world when an important security is rolled out. Basically, they allow you to build and link against old versions of a library while running the program against a new one. Traditional distros do that all the time and you don't even notice, but on Guix you actually have two versions of that library still lying around. The ungrafted one and the grafted one.
Why is this an issue?
Because it is possible to get those two mixed up, e.g. in
guix environment. I am not sure, which use cases are affected, but that one surely is. To see the difference, runonce inside a guix environment with grafts, and once in one without them. If you want to use Guix environments to prototype your applications, that means you'll have to use
--no-graftsto work around these types of issues for now.What to do from now on?
It is pretty clear to me, that the main culprit here is a different version of GLib being linked to Guile-GI than the one that should be loaded through Guile-GI. To fix that, we'll probably have to overhaul our entire bootstrapping procedure starting at GTypes. And we'll likely have to preload some version of GObject before defining them. Much fun.
Workaround
If you are working on Guile-GI code inside
guix environmentand do not wish to be haunted by this issue and how to perhaps resolve it, for the time being add--with-gir-hacksto your invocation of./configure. If you are experiencing similar issues in your own GI-based projects, consider patching your GIRs in a manner similar to what we do.The text was updated successfully, but these errors were encountered: