How to build Guix package for nchat on low-memory systems? #151

Kabouik · 2023-11-12T14:47:43Z

Kabouik
Nov 12, 2023

Here is a first attempt at a package definition for Guix:

(define-module (nchat)
  #:use-module (guix build-system cmake)
  #:use-module (gnu packages ccache)
  #:use-module (gnu packages cmake)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages databases)
  #:use-module (gnu packages gcc)
  #:use-module (gnu packages golang)
  #:use-module (gnu packages gperf)
  #:use-module (gnu packages llvm)
  #:use-module (gnu packages man)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages sqlite)
  #:use-module (gnu packages compression)
  #:use-module ((guix licenses)
                #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix git) 
  #:use-module (guix git-download)
  #:use-module (guix build-system gnu)
  #:use-module (guix deprecation)
  #:use-module (guix download)
  #:use-module (guix gexp)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-1))

(define-public nchat
  (package
    (name "nchat")
    (version "3.67")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/d99kris/nchat")
             (commit (string-append "v" version))))
       (file-name (git-file-name name version))
       (sha256
        (base32 "1hi66yda2vkzc1p7ahfhysricgg93vifc3w3rrnkm82j7dxdj6ry"))))
    (build-system cmake-build-system)
    (arguments
     (list #:tests? #f ;no tests
           #:phases #~(modify-phases %standard-phases
                        (add-before 'configure 'set-compilers-and-home
                          (lambda* (#:key inputs #:allow-other-keys)
                            ;; (let ((clang (string-append (assoc-ref inputs "clang") "/bin/clang")) ;;Those two lines are to set the full path to clang binaries, but the two `setenv` below are probably enough
                            ;;       (clang++ (string-append (assoc-ref inputs "clang") "/bin/clang++")))
                            (setenv "CC" "clang")
                            (setenv "CXX" "clang++")
                            (setenv "HOME" "/tmp")
                          #t)))))
    (native-inputs (list ccache gperf cmake help2man go))
    (inputs (list ncurses openssl readline sqlite zlib clang))
    (synopsis "Terminal-based Telegram/WhatsApp client")
    (description
     "@command{nchat} is a terminal-based Telegram and WhatsApp client.")
    (home-page "https://github.com/d99kris/nchat")
    (license license:expat)))
nchat ;;This line is only required for local builds with `guix build -f nchat.scm` will have to be removed when submitting the patch.

It builds nicely with guix build -f /path/to/nchat.scm and could be submitted as a patch already, but given the RAM use, I don't think it's taking into account the CC and CXX variables to build with the low-memory procedure described in LOWMEMORY.md. I am not trying to do the source split with php-cli, but just build with clang and clang++ would allow the package to work even on old or cheap hardware, or servers.

In case you want to have a go at it, the Guix package manager is available to any distribution, so there is no need to create a Guix System VM, you could just add Guix package manager to a testing VM you may already have.

Guix packages are typically release-based, at least for the main and official package channel.

d99kris · 2023-11-13T03:40:35Z

d99kris
Nov 13, 2023
Maintainer

Hi @Kabouik,

Thanks for providing a very detailed background. I don't know the answer, but I'll try to look into it.

Meanwhile if anyone else is familiar with Guix, feel free to chip in.

0 replies

Kabouik · 2023-11-15T13:30:13Z

Kabouik
Nov 15, 2023
Author

Below is a variation that is not meant to replace the previous one, but that allows installing nchat master instead of fixed releases, using guix install nchat@master. This may be useful when significant features have been implemented and being patient is too much of a challenge:

(define-module (nchat-master)
  #:use-module (guix build-system cmake)
  #:use-module (gnu packages ccache)
  #:use-module (gnu packages cmake)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages databases)
  #:use-module (gnu packages gcc)
  #:use-module (gnu packages golang)
  #:use-module (gnu packages gperf)
  #:use-module (gnu packages llvm)
  #:use-module (gnu packages man)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages sqlite)
  #:use-module (gnu packages compression)
  #:use-module ((guix licenses)
                #:prefix license:)
  #:use-module (guix packages)
  #:use-module (guix git) 
  #:use-module (guix git-download)
  #:use-module (guix build-system gnu)
  #:use-module (guix deprecation)
  #:use-module (guix download)
  #:use-module (guix gexp)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-1))

(define-public nchat
  (package
    (name "nchat")
    (version "master")
    (source
     (git-checkout
      (url "https://github.com/d99kris/nchat")
      (branch "master")))
    (build-system cmake-build-system)
    (arguments
     (list #:tests? #f ; no tests
           #:phases #~(modify-phases %standard-phases
                        (add-before 'configure 'set-compilers-and-home
                          (lambda* (#:key inputs #:allow-other-keys)
                            (setenv "CC" "clang")
                            (setenv "CXX" "clang++")
                            (setenv "HOME" "/tmp")
                            #t)))))
    (native-inputs (list ccache gperf cmake help2man go))
    (inputs (list ncurses openssl readline sqlite zlib clang))
    (synopsis "Terminal-based Telegram/WhatsApp client")
    (description
     "@command{nchat} is a terminal-based Telegram and WhatsApp client.
  This version is based on the master branch and may be unstable.")
    (home-page "https://github.com/d99kris/nchat")
    (license license:expat)))

I have both packages in my Guix private channel right now (kind of like a PPA repository) and they both work. I don't think the master version can make it to the official Guix channel because this contradicts the usual paradigm of Guix packages being based on releases, but if there is interest for it, I can make my private channel usable by others (they just need to add it to their channels.scm). First things first though: finding a way to build with the low memory procedure.

3 replies

d99kris Nov 15, 2023
Maintainer

Thanks for sharing 👍

Btw, just in case you know, is there some guix repository with "all" official packages?

I'm hoping there might be some other package passing special build instructions that I could use as sone hint.

Kabouik Nov 16, 2023
Author

Yes, you can find the repository with the official channel at https://git.savannah.gnu.org/git/guix.git/, then after cloning it, look at gnu/packages/*.scm.

There most certainly are some packages with custom build instructions. Here, I'm wondering whether (build-system cmake-build-system) may conflict with the environment variables that we set in (arguments …). We are also importing the module gcc, I'm not sure that this is necessary.

This is not a syntax for a package definition, but the --with-c-toolchain=package=toolchain (https://guix.gnu.org/manual/en/html_node/Package-Transformation-Options.html) flag to guix build could be an inspiration too. Overall, I think the #guix channel on irc.libera.chat would be of great help, but I bothered them a lot lately so I'm waiting a little bit before asking for help again!

d99kris Nov 20, 2023
Maintainer

I haven't gotten a chance to look into this properly yet, but just wanted to share that I found https://guix.gnu.org/manual/en/html_node/Build-Phases.html which states

build

Run make with the list of flags specified with #:make-flags. If the #:parallel-build? argument is true (the default), build with make -j.

make -j means using any number of concurrent jobs/processes, so it could result in a very high memory usage. I would suggest trying to set this parallel-build argument to false (I don't know how to do it in practise though, but maybe you're familiar).
It will be slow though. Tdlib (official Telegram client library) is a quite large library.

Kabouik · 2023-11-20T15:02:45Z

Kabouik
Nov 20, 2023
Author

Hey, Yes, this is already what I did but I haven't reported it in this issue yet, only in the Telegram group. This builds fine (even on the phone, and at some point it takes more than 1.5GB, which is surprising), it just takes forever to build, so I would like to investigate ways to build using `make.sh` to optimize the number of cores being used.

…

On 2023-11-20 15:29 Kristofer Berggren ***@***.***> wrote: I haven't gotten a chance to look into this properly yet, but just wanted to share that I found https://guix.gnu.org/manual/en/html_node/Build-Phases.html which states ``` build Run make with the list of flags specified with #:make-flags. If the #:parallel-build? argument is true (the default), build with make -j. ``` `make -j` means using any number of concurrent jobs/processes, so it could result in a very high memory usage. I would suggest trying to set this `parallel-build` argument to false (I don't have to do it in practise though, but maybe you're familiar). It will be slow though. Tdlib (official Telegram client library) is a quite large library. -- Reply to this email directly or view it on GitHub: #151 (comment) 0165 You are receiving this because you were mentioned. Message ID: ***@***.***>

0 replies

Kabouik · 2023-11-27T20:35:56Z

Kabouik
Nov 27, 2023
Author

I forgot to post the WIP packages, here they are:

master (will always fetch the latest commit):

(define-module (nchat-master)
  #:use-module (gnu packages ccache)
  #:use-module (gnu packages cmake)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages databases)
  #:use-module (gnu packages golang)
  #:use-module (gnu packages gperf)
  #:use-module (gnu packages llvm)
  #:use-module (gnu packages man)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages sqlite)
  #:use-module (guix build-system cmake)
  #:use-module (guix deprecation)
  #:use-module (guix download)
  #:use-module (guix gexp)
  #:use-module (guix git) 
  #:use-module (guix git-download)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-1))

(define-public nchat
  (package
    (name "nchat")
    (version "master")
    (source
     (git-checkout
      (url "https://github.com/d99kris/nchat")
      (branch "master")))
    (build-system cmake-build-system)
    (arguments
     (list #:tests? #f ; no tests
	   #:parallel-build? #f
           #:phases #~(modify-phases %standard-phases
                        (add-before 'configure 'set-compilers-and-home
                          (lambda* (#:key inputs #:allow-other-keys)
			    (setenv "CC" "clang")
                            (setenv "CXX" "clang++")
                            (setenv "HOME" "/tmp")
                            #t)))))
    (native-inputs (list ccache gperf cmake help2man go))
    (inputs (list ncurses openssl readline sqlite zlib clang))
    (synopsis "Terminal-based Telegram/WhatsApp client")
    (description
     "@command{nchat} is a terminal-based Telegram and WhatsApp client.
  This version is based on the master branch and may be unstable.")
    (home-page "https://github.com/d99kris/nchat")
    (license license:expat)))

v3.67:

(define-module (nchat)
  #:use-module (gnu packages ccache)
  #:use-module (gnu packages cmake)
  #:use-module (gnu packages compression)
  #:use-module (gnu packages databases)
  #:use-module (gnu packages golang)
  #:use-module (gnu packages gperf)
  #:use-module (gnu packages llvm)
  #:use-module (gnu packages man)
  #:use-module (gnu packages ncurses)
  #:use-module (gnu packages tls)
  #:use-module (gnu packages readline)
  #:use-module (gnu packages sqlite)
  #:use-module (guix build-system cmake)
  #:use-module (guix deprecation)
  #:use-module (guix download)
  #:use-module (guix gexp)
  #:use-module (guix git) 
  #:use-module (guix git-download)
  #:use-module ((guix licenses) #:prefix license:)
  #:use-module (guix packages)
  #:use-module (ice-9 match)
  #:use-module (srfi srfi-1))

(define-public nchat
  (package
    (name "nchat")
    (version "3.67")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/d99kris/nchat")
             (commit (string-append "v" version))))
       (file-name (git-file-name name version))
       (sha256
        (base32 "1hi66yda2vkzc1p7ahfhysricgg93vifc3w3rrnkm82j7dxdj6ry"))))
    (build-system cmake-build-system)
    (arguments
     (list #:tests? #f ; no tests
	   #:parallel-build? #f
           #:phases #~(modify-phases %standard-phases
                        (add-before 'configure 'set-compilers-and-home
                          (lambda* (#:key inputs #:allow-other-keys)
                            (setenv "CC" "clang")
                            (setenv "CXX" "clang++")
                            (setenv "HOME" "/tmp")
                            #t)))))
    (native-inputs (list ccache gperf cmake help2man go clang))
    (inputs (list ncurses openssl readline sqlite zlib))
    (synopsis "Terminal-based Telegram/WhatsApp client")
    (description
     "@command{nchat} is a terminal-based Telegram and WhatsApp client.")
    (home-page "https://github.com/d99kris/nchat")
    (license license:expat)))

0 replies

Kabouik · 2024-01-02T14:41:27Z

Kabouik
Jan 2, 2024
Author

So after some experiments and discussion with @d99kris over the week-end, it seems a major difficulty will be Whatsapp support.

Packaging nchat without Whatsapp support should be straightforward from the above definitions, but Whatsapp support depends on Go whatsmeow, and Go packages typically fetch their dependencies online, which is an issue for Guix packaging as the build environment has no network access for security and reproducibility reasons, and should just fetch the sources online, then check if the hash matches the expectatoin before compiling using dependencies already packaged the same way.

We do have many Go packages in Guix so I believe the package manager makes it so dependencies are picked for the build environment instead of the Internet if they are packaged for Guix too and added as inputs for the final package, so it should be possible to package all dependencies. I made a (draft) attempt at nchat-deps.scm which should cover dependencies for whatsmeow, but I'm not 100% sure of what I did. Also, go.mau.fi/util claims to include whatsmeow, so I wonder if we can package that instead and be done with it. I really am lacking Guix knowledge here to properly test things and conclude.

As far as I understad, @d99kris already bundles whatsmeow in the nchat sources so all this may not even be necessary.

Once this is solved, another issue is lib/wmchat/go/go.mod makes it so Golang will fetch online other dependencies that are not bundled into nchat. I don't know the syntax, but I suppose we could package those dependencies, then patch go.mod so that they are picked from the local build environment instead of the Internet, maybe using replace inside the require block.

Now I am not sure what path the dependencies would be built in if they are packaged, and I haven't found a proper way to test things quickly: typically, at the moment, I need to commit changes to the packages so that they are in the git private channel, guix pull, then guix install nchat@master, and wait a long time before the Go part kicks in because it occurs at 98% of the process, after the long tdlib compilation (especially without parallel building). There sure is a way to do all that from a local directory without pushing to a private channel, but I don't know how to make Guix find dependency packages in that case.

Finally, patching go.mod could be easily done in a Guix package with something like:

    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/d99kris/nchat")
             (commit (string-append "v" version))))
       (sha256
        (base32 "1hi66yda2vkzc1p7ahfhysricgg93vifc3w3rrnkm82j7dxdj6ry"))
      (patches (search-patches "nchat-go-deps.patch"))
      (file-name (git-file-name name version))))

However, this works for (origin (method git-fetch) …) which is what is used to package releases (the only accepted packages for the main Guix channel), but I don't know if somethign similar can be used with the (git-checkout) method, currently used to keep a version of the package always updated to the latest commit, which is convenient at least for development even though it cannot be accepted in the Guix channel as is (the hash is not checked with this method). An alternative to this master package could be to use command line package transformations on the release package to point to a particular commit instead of the release it is based on.

Not asking anything to @d99kris with this long message, just trying to write down the observations before I forget them.

0 replies

d99kris · 2024-09-15T06:36:34Z

d99kris
Sep 15, 2024
Maintainer

Will proceed to close this discussion for now, feel free to re-open it for any follow-up discussions.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to build Guix package for nchat on low-memory systems? #151

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

How to build Guix package for nchat on low-memory systems? #151

Kabouik Nov 12, 2023

Replies: 6 comments · 3 replies

d99kris Nov 13, 2023 Maintainer

Kabouik Nov 15, 2023 Author

d99kris Nov 15, 2023 Maintainer

Kabouik Nov 16, 2023 Author

d99kris Nov 20, 2023 Maintainer

Kabouik Nov 20, 2023 Author

Kabouik Nov 27, 2023 Author

Kabouik Jan 2, 2024 Author

d99kris Sep 15, 2024 Maintainer

Kabouik
Nov 12, 2023

Replies: 6 comments 3 replies

d99kris
Nov 13, 2023
Maintainer

Kabouik
Nov 15, 2023
Author

d99kris Nov 15, 2023
Maintainer

Kabouik Nov 16, 2023
Author

d99kris Nov 20, 2023
Maintainer

Kabouik
Nov 20, 2023
Author

Kabouik
Nov 27, 2023
Author

Kabouik
Jan 2, 2024
Author

d99kris
Sep 15, 2024
Maintainer