Guix: Quasiquote and G-expression
Preparing some materials for the First Workshop on Reproducible Software Environments for Research and High-Performance Computing, I am re-reading the G-Expressions section from the Guix manual. A must to read! Some times ago, I have read a blog post by Marius Bakke, it inspired me this blog post, thanks Marius.
Well, the best explanation is show me the code, isn't it? So let start guix
repl
and explore…
Before introducing G-expressions we need to have a clear idea about classic
Scheme concepts quasiquote
( `
) and unquote
( ,
); as well as quote
( '
). In Lisp-family language, “quoted” data remains unevaluated.
scheme@(guix-user)> (define x 42) scheme@(guix-user)> x $1 = 42 scheme@(guix-user)> (quote x) $2 = x
This quote
is so frequent that it has syntactic sugar ( '
). Somehow,
'(x y z)
is short1 for (list 'x 'y 'z)
.
scheme@(guix-user)> '(x y z) $3 = (x y z) scheme@(guix-user)> (list? '(x y z)) $4 = #t scheme@(guix-user)> (list 'x 'y 'z) $5 = (x y z) scheme@(guix-user)> (equal? '(x y z) (list 'x 'y 'z)) $6 = #t
Everything on the list is a value – e.g., 'x
or 'y
or 'z
; namely there
are symbols. In short, a symbol looks like a variable name except that it
starts with quote
( '
) and it plays a role similar as strings; somehow
symbols are a great way to represent “symbolic” information as data.
That’s said, we would like to be able to escape back inside of a quoted list
and evaluate something. Thanks to quasiquote
( `
) and unquote
( ,
),
it is possible.
scheme@(guix-user)> (define y 24) scheme@(guix-user)> `(,x y z) $7 = (42 y z)
Here, the unquoted
expression is evaluated during the construction of the
list, while the other remaining unevaluated. Another example:
scheme@(guix-user)> (define something 'cool) scheme@(guix-user)> (define (tell-me) `(Guix is ,something)) scheme@(guix-user)> (tell-me) $8 = (Guix is cool) scheme@(guix-user)> (define something 'awesome) scheme@(guix-user)> (tell-me) $9 = (Guix is awesome)
And unquote
evaluates everything, including procedures if any.
scheme@(guix-user)> `(Again ,(tell-me)) $10 = (Again (Guix is awesome)) scheme@(guix-user)> `(1 ,(+ 2 3) 4) $11 = (1 5 4)
For instance, let build2 the Fibonacci sequence.
scheme@(guix-user)> (define (fibo n) (if (or (= 0 n) (= 1 n)) `(begin ,n) (let ((f_1 (fibo (- n 1))) (f_2 (fibo (- n 2)))) `(begin (+ ,f_1 ,f_2))))) scheme@(guix-user)> ,pp (fibo 7) $12 = (begin (+ (begin (+ (begin (+ (begin (+ (begin (+ (begin (+ (begin 1) (begin 0))) (begin 1))) (begin (+ (begin 1) (begin 0))))) (begin (+ (begin (+ (begin 1) (begin 0))) (begin 1))))) (begin (+ (begin (+ (begin (+ (begin 1) (begin 0))) (begin 1))) (begin (+ (begin 1) (begin 0))))))) (begin (+ (begin (+ (begin (+ (begin (+ (begin 1) (begin 0))) (begin 1))) (begin (+ (begin 1) (begin 0))))) (begin (+ (begin (+ (begin 1) (begin 0))) (begin 1)))))))
Here, nothing is evaluated. All is data and symbolically manipulated. The
evaluation (computation) is then done with eval
,
scheme@(guix-user)> (eval (fibo 7) (interaction-environment)) $13 = 13
So far, so good. What about G-expressions? Quoting dedicated section of Guix manual:
To describe a derivation and its build actions, one typically needs to embed build code inside host code. It boils down to manipulating build code as data, and the homoiconicity of Scheme — code has a direct representation as data — comes in handy for that. But we need more than the normal
quasiquote
mechanism in Scheme to construct build expressions.
Somehow, G-expressions simplifies the machinery for staging code. Compared to
classic expression, it introduces both: context – e.g., the set of inputs
associated with the expression – and the ability to serialize high-level
objects – i.e., to replace a reference to a package object with its
/gnu/store/
file name.
G-expressions consist of syntactic forms: gexp
, ungexp
– or simply: #~
and #$
– which are comparable to quasiquote
and unquote
, respectively.
Other said, it allows to control the context of the evaluation.
Let make it concrete with the Fibonacci example. We need the modules (guix
gexp)
, (guix derivations)
and (guix store)
.
scheme@(guix-user)> (use-modules (guix gexp) (guix derivations) (guix store))
Now, instead of manipulating quasiquote
( `
) and unquote
( ,
), we
are going to manipulate gexp
( #~
) and ungexp
( #$
). Let start with
three helpers:
(define (number->name n) (string-append "Fibonacci-of-" (number->string n))) (define (number->gexp n) #~(begin (use-modules (ice-9 format)) (call-with-output-file #$output (lambda (port) (format port "~d" #$n))))) (define (store-item->number path) #~(begin (use-modules ((ice-9 textual-ports) #:select (get-string-all))) (string->number (call-with-input-file #$path get-string-all))))
Nothing special to say about number->name
. What do the others do?
number->gexp
takes an interger number and returns a G-expression, such that,
after evaluation, it will write this number to some file. What makes the
machinery convenient is that #$output
will be replaced – evaluated with
adequate context – by a string containing the output reference to its
/gnu/store/
file name. Can you guess what store-item->number
does?
The core: computing the Fibonacci sequence using G-expressions,
(define (fibonacci n) (if (or (= 0 n) (= 1 n)) (gexp->derivation (number->name n) (number->gexp n)) (let* ((store (open-connection)) (drv-1 (run-with-store store (fibonacci (- n 1)))) (drv-2 (run-with-store store (fibonacci (- n 2)))) (f_1 (store-item->number drv-1)) (f_2 (store-item->number drv-2))) (gexp->derivation (number->name n) (number->gexp #~(+ #$f_1 #$f_2))))))
First, gexp->derivation
returns a derivation with the name ( number->name
) that runs the gexp
( number->gexp
); somehow this gexp
stores
something to compute. Second, run-with-store
runs a gexp
in the context
of the store
– remember monad.
The procedure fibonacci
takes an integer number and constructs some
G-expressions controlling the context of evaluation. Let run it!
scheme@(guix-user)> ,run-in-store (fibonacci 7) $14 = #<derivation /gnu/store/db97xy9d5icaa64n2n9l7q2v66npmm6c-Fibonacci-of-7.drv => /gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7 7f7ca8ed3640>
Awesome! We get back a derivation. Note the handy ,run-in-store
command –
Guix specific REPL command – which hides the plumbing of run-with-store
. From
the REPL, it is possible to explore this derivation, although the
pretty-printer is not handy. Well, this derivation reads:
Derive ([("out","/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7","","")] ,[("/gnu/store/0wkxvd2ll0gff37wghamb12dz4x50n14-Fibonacci-of-6.drv",["out"]) ,("/gnu/store/9r95y1j1rg4q7vb528lh51w0cz3c5hvi-Fibonacci-of-5.drv",["out"]) ,("/gnu/store/zraigp7miin3vzr5dcbr4i9rvds0i07r-guile-3.0.9.drv",["out"])] ,["/gnu/store/z0jspla9advx77ihbc7nfjvnky2gfvjz-Fibonacci-of-7-builder"] ,"x86_64-linux","/gnu/store/g8p09w6r78hhkl2rv1747pcp9zbk6fxv-guile-3.0.9/bin/guile",["--no-auto-compile","/gnu/store/z0jspla9advx77ihbc7nfjvnky2gfvjz-Fibonacci-of-7-builder"] ,[("out","/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7")])
And the 7th term depends on the 6th and 5th (\(F_{7} = F_{6} + F_{5}\)); the expected recursive sequence. The interesting part is the builder,
(begin (use-modules (ice-9 format)) (call-with-output-file ((@ (guile) getenv) "out") (lambda (port) (format port "~d" (+ (begin (use-modules ((ice-9 textual-ports) #:select (get-string-all))) (string->number (call-with-input-file "/gnu/store/rhjmlgaz4f1niwhrnm2nsfdj2g6dya6h-Fibonacci-of-6" get-string-all))) (begin (use-modules ((ice-9 textual-ports) #:select (get-string-all))) (string->number (call-with-input-file "/gnu/store/xcp9b4j0nskk6lk5jxlpv3926j19vpw0-Fibonacci-of-5" get-string-all))))))))
All had been correctly replaced. Somehow, that builder script is similar as
the output of the previous fibo
procedure, and instead of eval
, now the
computation will be done by Guix daemon evaluating this builder script.
Let make that computation!
scheme@(guix-user)> ,build $14
$15 = "/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7"
And guess what? This file contains the value 13. 👍
As you can see, all the previous values of the Fibonacci sequence are also
computed via derivations and the result stored as files. Guix daemon starts
to construct the derivation Fibonacci-of-0.drv
, then Fibonacci-of-1.drv
,
and compute them by writing 0 then 1 inside the output store item files
Fibonacci-of-0
and Fibonacci-of-1
. Then Guix daemon evaluates the builder
script of the derivation Fibonacci-of-2.drv
, i.e., it reads the values from
two previous store item files, adds them and writes the result inside the
store item file Fibonacci-of-2
. The Guix daemon repeats until 7. Other
said, if we want to compute the value for the 8th Fibonacci number, all the
previous computations are cached in the Guix store; the Guix store acts as a
good memoization mean. Check it with:
$ guix gc --list-dead | grep Fibonacci-of
Willing to go further, give a look to the Code Staging in GNU Guix paper.
Join the fun, join Guix!
Footnotes:
Update 2023-11-06 (see this question/answer). To be precise, '(x y z)
is a sugar for (quote x y z)
. And the subtlety is that list
returns a new
list at each call, i.e., (eq? (list 'x) (list 'x)) => #false
. While it would
be expected that (eq? '(x) '(x))
return #true
. Note that eq?
is essentially a
pointer comparison, quoting Equality section from the Guile manual and equal?
looks (recursively) into the contents of lists.
It would be possible to make the code more symmetric and implement the
Fibonacci sequence using quasiquote
/ unquote
and files. However, it makes
everything far more complicated. Somehow, that’s why G-expressions had been
introduced, after all! ;-)