Redoing our paper in Nature Scientific Data (Oct. 2022)

transparent, verifiable and long-term… one year later

Some days ago, I prepared a demo for my talk at Pasteur Institute. Why not try out one example from the paper Toward practical transparent verifiable and long-term reproducible research using Guix, Nature Scientific Data, vol. 9, num. 597. Well, I spent the morning just before the talk checking out that example for demoing… And the computational environment reproduction failed. This post investigates and draws counter-measures. The aim here is twofold:

  1. Expose the roadblocks about reproducing a computational environment; here the reproduction of one computational environment distant of three years.
  2. Show how Guix project helps; from technical workarounds to work-in-progress features.

Our paper had been published on October 2022 and we demoed using a published paper from 2020. The Guix revision (channel file) – the one we published in 2022 – had been thus picked from an older Guix revision – mimicking a Guix channel file as it would have been published by the paper in 2020. To be precise, the selected Guix revision we published1 was 1971d11db9 (April, 14th 2020). And as explained in our paper, we chose this revision in order to get Bioconductor 3.10, as we extrapolated from the published material.

Other said, the command-line,

guix time-machine -C channels.scm \
     -- environment -C -L my-pkgs -m manifest.scm

builds the exact same computational environment as it was in April 2020. This command-line was just working out-of-the box in October 2022. Now, more than one year later, this very exact same command-line fails.

Long story short: today, three years later after one publication, the link-rot defeats the ability of checking this very published result. The failure starts very early in the attempt of reproducing: it starts at getting all the source code for composing the computational environment.

Evil is about details. Let review them!

Disclaim. Re-run and compare would be another goal. I think “re-run and compare” is a tangential corollary of the main objective: reproduce the exact same computational environment. Somehow, if we run inside the exact same computational environment, then the experimental conditions of the computations are (almost) bounded, and so “re-run and compare” is just repeat yet another same experiment so re-generating equivalent data. In addition, for many results, it is not affordable to “re-run and compare” because the cost (energy consumption, computing resource availability, CPU time, memory, etc.) is too high or does not worth it. To me, the scientific method applied to the computational environment means its full source-to-binary transparency, other said, the ability to deeply audit and verify all, in the long term. That’s the first main objective, then, if completed and if required, secondly we could apply variations about the computational environment in order to challenge the conclusion under study.

How to rebuild the past?

Before jumping into details, what do we mean by “computational environment” here? This computational environment is composed by the direct packages required by the analysis and also by all the indirect packages not explicitly listed. How many packages do we speak about? More than 720! Dependencies of dependencies matter, perhaps.

Reproducing the exact same computational environment means that we must compose again from more than 720 source code, and today on 2023 as it was in 2020. This asks two important questions:

  1. How to identify a source code?
  2. How to build the identical computational environment composed by more than 720 components?

My opinionated answer is: label version as with “I used flowCore at version 1.52.1” is not enough because,

  1. it identifies poorly the source code,
  2. it does not scale; more than 720 label versions should be provided.

Ready for the arcane of rebuilding the past? Let’s go!

First thing first

The computational environment is described by the manifest file,

  ;; Packages from Guix collection
  "r-rmarkdown"                         ;render .Rmd files

  ;; Extend collection by defining in folder my-pkgs/my-pkgs.scm

and all the versions of all these packages are implicitly defined by the channel file, which reads in this case,

(list (channel
        (name 'guix)
        (url "")

Here, only the “Guix channel” is required. More channels could also be added for extending the package collection. In our case, the package collection had been locally extended with one package defined in the directory my-pkgs.

Complete transparency

The commitment that the Guix project tries to demonstrate is, from my point of view, unique and a real challenge: provide tools that produce the same computational environment at two distant points, in time and space. Concretely running this command-line,

$ guix time-machine -C channels.scm \
       -- environment -C -L my-pkgs -m manifest.scm

outputs this long list:

guile: warning: failed to install locale
substitute: updating substitutes from ''... 100.0%
substitute: updating substitutes from ''... 100.0%
The following derivations will be built:

building /gnu/store/y1bnskpk88qh1adw7hpvds125m35p8xp-r-minimal-3.6.3.drv...
- 'build' phase

What does it mean? It means that the computational environment is completely transparent and verifiable. Namely, these files ending with .drv. describe how to build or where to fetch source code. We have access to all the details for producing all the binary artifacts from the source code.

First, note that the Guix project provides pre-built substitutes and these binary artifacts from 2020 are gone. Other said, we need to download the source code and locally build them. For instance, we see that we are building the package named r-minimal and then we will download the source code for the R library MASS in order to build the corresponding Guix package r-mass. And so on.

Second, have you noticed these ghc items? They are Haskell libraries and required for Pandoc – an universal document converter. The Guix package ghc-pandoc is indirectly required by R libraries as ncdfFlow. In addition, we can also see clisp or sbcl which are from Common Lisp ecosystem. Have you guessed them beforehand? Me, not! But told you – more than 700 dependencies of dependencies.

Note. One might ask if one of these dependencies of dependencies matters for the ending result of the analysis and it is legitimate. Without using Guix, I would not be able to start an answer – confirm or contradict my intuition. Other said, the scientific method implies that we need to challenge the hypothesis – say it has no impact – by testing some variations. Therefore, we need two things: on one hand, a reference point and on the other hand the capacity to generate fully controlled variations of that reference point. As experimenters would do, they would run experiments varying parameters carefully selected under controlled conditions and the output of these experiments checked against one control output.

The unexpected failure?

Ah, the previous command-line quickly fails. The error is about the R library BiocNeighbors. It is important to notice that this R library is not explicitly required by the analysis but appears to be an indirect dependency – required by the package r-cydar.

It fails because Guix tries to fetch the source code from the location encoded in Guix package definition from 2020 and the content at that location is gone today in 2023. Yet another observation of the well-known link-rot phenomenon.

building /gnu/store/b7x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv...
-builder for `/gnu/store/b7x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv' failed to produce output path `/gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz'
build of /gnu/store/b7x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv failed
View build log at '/var/log/guix/drvs/b7/x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv.gz'.
building /gnu/store/smky0svf3gw5dz8ajj0im7kr7mzv12lr-BiocParallel_1.20.1.tar.gz.drv...
cannot build derivation `/gnu/store/xnmx8c5jgksv56g4qhsr17fsm62qclni-r-biocneighbors-1.4.2.drv': 1 dependencies couldn't be built
guix environment: error: build of `/gnu/store/xnmx8c5jgksv56g4qhsr17fsm62qclni-r-biocneighbors-1.4.2.drv' failed

What could be done? Let ask first: what is already done? What are the attempts behind this error message? Let open the build log file as reported, it reads:

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
download failed "" 404 "Not Found"

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
following redirection to `'...
download failed "" 404 "Not Found"

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
download failed "" 404 "Not Found"

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
download failed "" 404 "Not Found"

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
download failed "" 404 "Not Found"

Starting download of /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz
download failed "" 404 "NOT FOUND"
Trying to use Disarchive to assemble /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz...
could not find its Disarchive specification
failed to download "/gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz" from ("" "")
builder for `/gnu/store/b7x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv' failed to produce output path `/gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz'
build of /gnu/store/b7x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv failed
View build log at '/var/log/guix/drvs/b7/x2m61j3979mfysb9vgqih1c2qqr3nf-BiocNeighbors_1.4.2.tar.gz.drv.gz'.

Before error out, Guix tries from 5 different locations: 2 official Bioconductor locations, 1 location maintained by the Guix project, 1 another location maintained by the NixOS project, and last 1 location from Software Heritage – the universal software archive.

Sadly, if only one item misses, all fails down. For this specific case where Bioconductor is involved, the Guix project considers it is a bug and tracks it with bug #39885.

Rewriting past origin fields

Part of this Guix bug #39885 had been fixed with b032d14ebd. Sadly, this fix is from late June 2020, hence not available for the Guix revision (April 2020) we are considering.

What could be done? We provide a channel file that totally freezes one specific state. All the packages described by that state are immutable. We are doomed, isn’t it?

Maybe not yet… Guix is flexible enough that it allows to rewrite the complete graph of dependencies. Indeed, but one should object that it will not be the exact same computational environment. And why would it not be? We need to speak about the identification of source code.

How does Guix fetch the source code? The answer is fixed-output derivation. A package as r-biocneighbors defines, among many other fields, the source field which describes how to download (url-fetch) and from where (see uri).

  (define-public r-biocneighbors
      (name "r-biocneighbors")
      (version "1.4.2")
         (method url-fetch)
         (uri (bioconductor-uri "BiocNeighbors" version))
      (properties `((upstream-name . "BiocNeighbors")))
      (build-system r-build-system)
       `(("r-biocparallel" ,r-biocparallel)

In addition, we see a checksum (sha256). This field allows to verify that Guix downloads the expected content – the one that had been packaged. We could provide any other URL location (uri) while the content checksum matches.

Note. If I am extreme, the field version is a label but does not – at all – describe the source code version. Only an identifier depending on the content itself allows to exactly know which source code version we are really using.

What revision b032d14ebd from Guix bug #39885 does it fix? One of the Bioconductor URL location. Look the content is still available at another location,

$ guix time-machine -C channels.scm \
       -- download
guile: warning: failed to install locale

Starting download of /tmp/guix-file.kQt81F
following redirection to `'...
 …_1.4.2.tar.gz  882KiB                                                     348KiB/s 00:03 [##################] 100.0%

The checksum matches. Now, we have the source code in our local store. Problem solved for this missing source code of BiocNeighbors. Then, let re-run the command-line above for creating the computational environment. Ah, it fails again because the source code of DelayedArray is missing. Then, again for another (GenomeInfoDb), and again.

Could we programmatically extend the location where Guix downloads source code from Bioconductor? Let rewrite the origin field of the packages coming from the Bioconductor project. First, we identify these packages because the URL of their source code contains the string "". Let define a predicate procedure that returns #true or #false if the uri field matches or not this string "". For details about pattern-match-ing, give a look to this post or this documentation.

(define (bioconductor? p)
  (match (package-source p)
    ((? origin? o)
     (match (origin-uri o)
       ((url rest ...)
        (string-contains url ""))
       (_ #false)))
    (_ #false)))

Second, Guix provides the package-mapping procedure that, given a package, applies an user-defined procedure to all the packages depended on and returns the resulting package. Other said, this package-mapping allows to customize the whole dependencies of dependencies (graph). Therefore, we need to define a procedure that takes a package and creates a new package with an extended list of URL if it is a package from Bioconductor else does nothing.

Wait, for instance the packages r-flowcore and r-ncdfflow both depends on r-bh, and even r-ncdfflow depends on r-flowcore. Therefore, we need to rewrite r-flowcore or r-bh only once and not traverse again all the dependencies. We need a predicate procedure package-seen? that returns true when the origin field of the package coming from Bioconductor had already been extended. A Guix package may own a properties field that can store any customized/user-defined pair values; from an association list. Each time we extend the list of URL, we also add an element to the properties field. Then we use it for detecting if the package had already been seen.

All in all, it reads,

(define (package-seen? pkg)
  (assq-ref (package-properties pkg) 'bioconductor))

(define (extend-url pkg)
   ((package-seen? pkg) pkg)          ; Already processed
   ((bioconductor? pkg)               ; Process if it comes from Bioconductor
    (let ((src (package-source pkg)))
        (inherit pkg)
           (inherit src)
           (uri (append
                 (origin-uri src)
                 (list (string-append
                        "https://" bioconductor-url "/packages/"
                         (assq-ref (package-properties pkg) 'upstream-name)
                         (package-name pkg))
                        (package-version pkg) ".tar.gz"))))))
        (properties `((bioconductor . #true)
                      ,@(package-properties pkg))))))
   (else pkg)))                        ; Do nothing for all the other packages

When pkg comes from "" servers, we create a new package (package) where all the fields are copied (inherit) except the source and properties fields. Since that’s a Bioconductor package, it is marked as such using the properties field. All the elements of the origin field are also copied from the original package except the uri field where the new URL is appended.

Here, the checksum field is not modified at all, therefore we are building the exact same computational environment. We are only implementing an ad-hoc workaround for fetching from more locations.

Last, in order to avoid extra work and traverse the deep dependencies of dependencies, we assume that if, for one package, the dependency is not from Bioconductor or already seen, it is not worth to process the dependencies of this dependency. This strategy is what the procedure cut? implements; see here for the details.

Launching the command-line above for creating the exact same computational environment, now we get,

building /gnu/store/0cs04zymfxpwh49z5da2ps2d4vinakhi-GenomeInfoDb_1.22.1.tar.gz.drv...
downloading from
building /gnu/store/0k520rcg3qa4bamkgrn1x8nd1nvxbbs2-Diff-0.3.4.tar.xz.drv...
building /gnu/store/8diprmghchxf62svbapmjd2nq4g3yhhn-GenomicRanges_1.38.0.tar.gz.drv...
downloading from
building /gnu/store/9i3c5vv9lmlkd0dr91252fmhzv7sa5vm-IRanges_2.20.2.tar.gz.drv...
downloading from

Awesome! Guix rocks…

Another failure

…Guix rocks but Guix cannot fix all the issues of the world. The failure now reads,

building /gnu/store/5257ay3c3qkiy51yv4frsvmjmh3hxk08-codetools_0.2-16.tar.gz.drv...
downloading from
\sha256 hash mismatch for /gnu/store/9kd2dj46zy0m8ciz2m57f0rij9m3lj5c-codetools_0.2-16.tar.gz:
  expected hash: 00bmhzqprqfn3w6ghx7sakai6s7il8gbksfiawj8in5mbhbncypn
  actual hash:   1dklibnp747a0p41ggcf8fyw36xhj9c869gay80ggfns79y7axn2
hash mismatch for store item '/gnu/store/9kd2dj46zy0m8ciz2m57f0rij9m3lj5c-codetools_0.2-16.tar.gz'

We are doomed! Game over for reproducing the exact same computational environment as the one from our paper. The CRAN project did an in-place replacement. Again, label version is not enough when we speak about identification of source code. At package time back in 2020, codetools labelled 0.2-16 had a checksum and now this very same labelled 0.2-16 had another checksum. Sadly, because we do not have access to the past source code, now we have no mean to know the difference and/or if the difference matters. We have lost the ability to verify and audit how the computations had been done.

Assume that I just discover the result – say one, two or three years later after the publication. How can I trust the result if I cannot audit? If I stretch, what are the guarantees for trusting the result when the collective scientific method principles do not have the time to be applied? One, two or three years is not enough time for challenging a result, in my humble opinion.

That’s said, what are the options at hand? For one, we need to create the best computational environment approximation, for example by using this other version labelled 0.2-16. For two, the skeptical of the result should use this computational environment approximation to re-run and compare. Let focus on one since I am not competent at all for two.

Again, we rely on package transformation as package-input-rewriting which allows to replace dependencies. First, we need to locally define a new package, named r-codetools-bis. And second, we need to rewrite the dependencies of dependencies for replacing the old r-codetools by the new r-codetools-bis. The definition of r-codetools-bis is straightforward,

(define-module (my-pkgs-fix)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module ((gnu packages statistics) #:select (r-codetools)))

(define-public r-codetools-bis
    (inherit r-codetools)
    (name "r-codetools-bis")
         (method url-fetch)
          (base32 "1dklibnp747a0p41ggcf8fyw36xhj9c869gay80ggfns79y7axn2"))))))

From the origin definition of r-codetools, we just copy (inherit) all the fields except the source field, the one we replace with the new source code version matching the new checksum. And the manifest file is updated with,

(define with-r-codetools-bis
   `((,(specification->package "r-codetools")
      ,(specification->package "r-codetools-bis")))))

Nothing more.

Go for it

Let recap how all the pieces fit altogether. First, we have a list of package names (specification) that we transform to internal representation (package). Second, for each we rewrite the dependencies of dependencies to replace r-codetools with r-codetools-bis. Third, for each we write the dependencies of dependencies to extend the Bioconductor servers. Somehow, it reads,

   ;; Packages from Guix collection
   "r-rmarkdown"                         ;render .Rmd files

   ;; Extend collection by defining in folder my-pkgs/my-pkgs.scm

In summary, we compose the three transformations and apply this composition for each (map) listed package name. See the companion Git repository for all the details.

Last, be patient… then be more patient. It needs to build many packages from source since most of the pre-built binary artifacts (substitutes) are gone. It takes many hours depending on your hardware. And the end, we do not have the exact same computational environment as the one we described in our paper. Instead, we have an approximated one where the R library codetools has been replaced. For sure, Guix provides all the tools to manipulate the computational environment controlling with very fine-grain all the details.

Next steps, work in progress

Still reading? Such a journey, isn’t it? My first opinionated and main conclusion is that Guix is very flexible and strict in the same time. The computational environment had been frozen – all is immutable – and although the reproduction is impacted by the link-rot phenomenon, still, Guix helps. Maybe it does not appear to you “easy” but I do not have in mind any other tools able of such features: time-travel, rewrite components on the fly, etc. If you are an expert of some other tools, let me know how you would do. Any feedback is very welcome!

How this reproduction could be even easier? Yes, I, among many others, are not satisfied by the current situation. We can collectively do better!

  1. If you are paper’s author, please cite without any ambiguity your scripts or more required by your work. Do not think that publishing your Git repository containing your script is enough for inspecting, verifying and auditing what you did.
    • You need to pin a specific revision when publishing.
    • You need an identifier that depends only on the content itself. And not some label version as “version 1.2.3”.
  2. If you are paper’s author, please think how to cite all the software required by your work. Do not take me wrong, publishing your scripts or more is a very good practise toward a better scientific production. In addition, please also capture the information from your computational environment as guix describe does, and publish it.

In summary,

  • The way the source code is uniquely identified matters.
  • The way how source code is transformed to binary also matters.

How to lookup inside an archive?

As we have seen above, when the content is missing at all the expected locations, then Guix tries to fetch such content from the Software Heritage archive. Currently 75% of source code that the Guix collection provides is archived. To put it in precise terms: for 75% of source code that Guix collection provides, Guix is able to automatically fetch back the content from Software Heritage. Other said, the remaining 25% may be already archived and Guix does not implement the capacity to download them from Software Heritage.

One example is about Subversion (svn) version control system. Most of CTAN TeX packages is versioned using Subversion and may be archived in Software Heritage. However, in these cases the mechanism to lookup is not implemented by Guix (see bug#43442). Help is very welcome!

Another example is about “compressed tarballs”. In short, a “compressed tarball” could be split into two parts: the content itself and metadata around. For instance, metadata is compression level, compression algorithm parameter, or related to specific file or directory structures. Software Heritage archive only the content itself – and that’s already a lot! – but drop the metadata around. Then, without these metadata, it is impossible to verify (checksum) that it exactly matches the version at package time.

That’s the purpose of Disarchive database. For all the “compressed tarballs” that the Guix collection relies on, on one hand the metadata is automatically extracted and stored in a database and on the other hand the content is archived by Software Heritage. Awesome, isn’t it?

However, as we have seen previously,

Trying to use Disarchive to assemble /gnu/store/3zifa3x7yvmznic69j00q8qad4f588ah-BiocNeighbors_1.4.2.tar.gz...
could not find its Disarchive specification

this Disarchive database has holes. For instance, it does not contain the package r-biocneighbors for the Guix revision 1971d11db9. Help is very welcome. From discussion to example of hole or infrastructure improvement.

Last but not least, the choice of the unique identifier. It is not a detail, it is the masterpiece. And the situation seems as the well-known xkcd strip about standards. Guix had been initiated in 2012 and reuses principles pioneered by Eelco Dolstra’s PhD thesis from 2006. One is normalized archive or nar format – comparable in spirit to tarball. Then, Software Heritage designed later SoftWare Heritage persistent IDentifiers (SWHIDs) adapted for their archiving purpose. For the interested reader, an entry-point for the difference reads,

guix hash --serializer=nar --format=nix-base32 --hash=sha256
guix hash --serializer=git --format=hex        --hash=sha1

The first one matches the sha256 field in a Guix package definition. The second matches swhid:1:dir: identifier. Compare for instance,

$ guix time-machine -C channels.scm -- edit r-catterplots

$ guix hash -S nar -f nix-base32 -H sha256 \
     $(guix time-machine -C channels.scm -- build r-catterplots --source)

$ guix hash -S git -f hex -H sha1 \
     $(guix time-machine -C channels.scm -- build r-catterplots --source)

And visit

Somehow, Disarchive database provides such mapping from NAR identifier to SWHID identifier. It could be nice if Software Heritage could directly integrate such map. Guess what? The work in progress is to bridge both and thus ease the lookup. For instance, SWH ticket#4979 introduces some mapping components. Stay tuned for more…

Rendez-vous next year! I cannot wait. Let see if we will collectively make progress for reproducing what we just did in this post.

Join the fun, join the initiative Guix for Science!



The paper itself incorrectly refers to Guix revision 0105f33a4d which dates from October 2021, the time of the first reproduction attempt. We then replaced by a Guix revision from 2020. The companion Git repository had then been updated but not correctly reported in the paper.

© 2014-2024 Simon Tournier <simon (at) >

(last update: 2024-06-05 Wed 09:47)