Discussion : Reproducible Typst compilation

[Paul]

Introduction

A good way to ensure reproducability of Typst source code would make it easier for publishers in academia to adopt Typst. That’s why this discussion began in the thread : “Discussion: Lobbying for Typst adoption by publishers”,

with this remark :

Regarding the question of having a Typst distribution: an alternative would be to have a small script to make a Typst document self-contained. The typst would fetch all external packages that the document relies on, possibly font files as well, and return a version of the document that builds from these local assets in a self-contained way, along with a metadata file describing where the packages are coming from and which Typst version is known to work for this self-contained document. The result would be highly reproducible.

from this thread :

Discussion: Lobbying for Typst adoption by publishers

Two minor points: I wrote a Typst template for the [acmsmall]{acmart} LaTeX configuration expected by some ACM conferences and journals, approximate-acmsmall. I encourage people to try to use it, and tell conference/journal editors about them. Currently it is possible to submit a Typst paper (with this template or another) to arXiv, asking for a PDF-only submission, but not possible to actually submit such a document to the ACM for publication, as the conference/journals I am familiar with (S…

And this one from the same thread :

Discussion: Lobbying for Typst adoption by publishers

mmh, how about expanding the --deps argument of the typst compile command? As in add an additional format called standalone, that creates a zip with all the used elements?

Reproducability

The goal is to be as confident as posssible that Typst source can be shared and recompiled when having access to the right version of the Typst compiler. Different problem comes into mind :

• system fonts that have to be installed to be used
• online packages with the right version
• common build system to build the same way every time

I will now discuss counter measure that permits to be confident that you can share Typst source code between people on the same project or to publishers.

Fonts

This is solved by using the --ignore-system-fonts and --font-path parameter of the typst compiler command. Other option is to only used embed fonts of the Typst compiler but this is not ideal.

In a given project I usually put fonts in a fonts/ directory (I am not very knowledgable in potential Licensing issue with dowloading random fonts on the internet).
This command to compile file is good to ensure that multiple authors have no problem to compile on their system :

typst compile main.typ --ignore-system-fonts --font-path fonts/

Online Packages

Packages are right now imported this way : #import "@preview/package_name:version".This is pretty flexible however this forces internet access multiple times. For this to be reproducible we have to assume multiple things :

• version has to be specified : this is already the case and mandatory so this is good
• the github typst packages repository has to be always available : in my experience this has been the case but cannot be garanteed in long term (especially for publishers)

Downloading once and for all the package you use is one of the solution to allow to distribute things in a reproductible way. You can this in multiple ways :

• downloading all the packages repository (maybe a bit too much however this is the Latex approach)
• use the --deps arguments of the typst compile command to get all dependancy however I found this to be too precise as it gets you files that you have used however this may change if you modify a bit your source file. For this to be effective packages should have in their typst.toml their exact depedancy written somewhere for easier import of indirect depedancy.

Build tool

I usually use a Makefile even if I prefer justfile. But I don’t want to make people install something they don’t have on their system. As with Rust or other language, it is very appreciated to not have to use an external tool to build something. Or at least to use a well polished tool maintained by the community/fondations.

Proposal

Use typst.toml the optional entry point for typst compilation.

I wish we could do :

typst compile

this would look at a typst.toml (if present) or a config file that describe the command line arguments of the command. typst.toml already exists for packages and should be configures a bit differently for document project. In Rust there is the same dichotomy between binary project or library projects that bot use the Cargo.toml file. Here a “binary” project would produce a pdf (or other format) while a “library” project would give access to library function just as packages do today in Typst.

• Package should indicate their dependancy before compiling to be able to downloading all the code needed when using a given package. To do that we could extend the typst.tomlwith a “dependancies” field.

[dependancies]
cetz = "2.3.4"
fletcher = "5.2.3"

• we should be able to define a “binary” compilation to pdf or other format in the typst.toml with “bin” field. Every single argument of the Typst CLI would be accessible in the typst.toml with the exact same name and parameter.

Here is a typst.toml that would be the result of everything I just said :

[dependencies]
cetz = "2.3.4"
fletcher = "5.2.3"

[[bin]]
entry-point = "main.typ"
output = "main.pdf"

#   Reproducible fonts
ignore-system-fonts = true
font-path = "fonts/"

#   Reproducible packages
package-path = "packages/"
ignore-online-package = true

The ignore-online-package does not exists yet but could be useful in the same vain as ignore-system-fonts.

Of course I don’t want to reinvent the wheel and this is proposal is of course inspired by the way Rust do it with Cargo.toml.

One major thing that the compiler should also make possible it to not have to patch existing Typst code. For example #import "@preview/cetz:2.3.4 should be able to point to a local package without changing the source code.

Conclusion

This is one way to do it. Please share your setup for making it easier to work in group on Typst project. The web app also have been a solution for me because a lot is being taken care of for you. Especially Typst version is very well handled on the web app. However the web app does not solve the publisher problem that won’t want to use the web app to compile a document.

Another possibility would be to define all of this in the Typst language directly. Maybe this is the direction that would be choosen especially because the document function has already been enriched in previous version of Typst to be able to write a “build” file that compiles multiple document for a web page for example. This is handy but it might be too heavy to put allll the CLI arguments inside the document function API.

[gasche]

In the approach that I proposed, Typst proposes a tool to “extract” a reproducible build out of an arbitrary typst project, by collecting the dependencies. This works whether or not the user plays nice, lists their dependencies correctly, etc.: even for badly-written Typst projects, you get a reproducible version out. In particulars, publishers could accept arbitrary project sources as input, and run this tool on their side and they get something they can trust. (To be precise, one also wants information on which Typst version precisely was used / will work in the future to do the build. Having tracability on where the various parts were pulled from would also be a plus.)

In the approach that you propose, there exists some dependency configuration file that, if used correctly, can provide reproducibility. This is useful for users who want to ensure that their build is reproducible, but in practice most users don’t care about this. This is not so useful for publishers, because the metadata format does not ensure reproducibility. At minimum you would then need an additional tool, which is a sort of linter that checks the .toml file to verify that it actually ensures reproductibility – then publishers can run this linter in their automated submission-check pipelines and complain to users, who may in turn complain that they are asked to provide stuff that they do not need in normal Typst usage.

Note: the project-description metadata that you propose could be produced by a project-snapshotting tool such as typst compile --deps-format reproducible-archive --deps foo.zip as part of its output, to describe how to rebuild the project. (And it could serve other purposes.)

[Paul]

Yes completely agree with what you said. I answered more about my personal problem which is : having co-authors compile easily an article we are writting together. Rather than for publishers. In this use case it is a bit more loose and I accept to rely on typst/packages repo however it is important to specify fonts locally and package version.

[Paul]

OKK I tweaked with all parameters I could find on typst compile . I think that what we want to use is the--package-cache-path. At first compilation it populates the directory but downloading the dependency and then you can recompile with the same command and get the same result without any internet access because you only look at the generated directory and Typst compilation are deterministic (when fixing the current time of day).

The publisher get a zip folder named typst_project.zip. Things you still have to ask to authors :

• entry point of the project (would be great to have a fixed convention like main.typ or anything)
• to put all the fonts in the typst_project/fonts/ folder (I wish we could have the argument --font-cache-path for the fonts used to be put in the same directory !)

We now assume that typst_project/main.typ exists and is the entry point. We now assume that typst_project/fonts/ contains all necessary fonts for the project to compile (fonts can also be the one embedded with the Typst binary.

I suppose the directory structure is this one with nothing in the directory except the typst_project directory and a Makefile used by the publisher :

/typst_project/main.typ
/typst_project/other_files_of_authors.typ
/typst_project/fonts/bunch_of_fonts...
/typst_project/*
/Makefile

This Makefile is the one a publisher could use to produce a reproducible build for long term archiving :

create_archive_and_compile:
   typst compile typst_project/main.typ main.pdf
       # even if the source use today time information the build in deterministic
       --creation-timestamp 0
       # make sure fonts are all embedded
       --ignore-system-fonts 
       # use fonts provided by the author
       --font-path typst_project/fonts/
       # make sure you don't use files outside the current directory
       --root .
       # make sure to download once all packages
       --package-cache-path packages/

First run of make create_archive downloads all dependancy needed and you get this directory structure :

/typst_project/main.typ
/typst_project/other_files_of_authors.typ
/typst_project/fonts/bunch_of_fonts...
/typst_project/*
/Makefile

# This was generated :
/packages/one_directory_for_each_dependency
/main.pdf

Second run of make create_archive_and_compile you can turn off internet and it will recompile main.pdf as long as you want.

To be clear this has not been battle tested (I tested 10minutes and seems to work) but in theory this seems to be OK !

[Paul]

In conclusion there are still 2 non perfect element :

1. fonts have to be provided by the user in a specific directory known by the publishers
2. you need to specify an entrypoint
3. we don’t specify the version of the typst compiler

In conclusion I reevaluate my typst.toml proposition as being too complicated for something we can already “almost” achieved in an easy way without forcing every package author to add more structured information.

Answer to the problems :

1. The publisher will have warning regarding font not existing. They can then ask authors to send document that compiles with the --ignore-system-font argument. In the author side their should be a tool that automatically bundles fonts used in a working compilation and put them in the right directory. The publisher cannot make this since it might receive a document not compiling.
2. Very easy publisher just have to say that main.typ is the entrypoint and everybody should respect that.
3. Add a random file that indicates which typst version is known to work