Discussion: Lobbying for Typst adoption by publishers

[Kevin_McCurley]

I also don’t see anything “wrong” with typst for publishing - in fact I came here to figure out how to make it work. One possible conflict that could arise in an offline compilation environment is if an author specifically has a dependency on package_a version 1.1.2 but we only have 1.1.1 installed in our system at the time that our Docker image was built. Presumably in this case it’s not going to compile. This is not unlike what happens when people upload their LaTeX files, though in that case it’s usually because the author is using an older distribution of TeX rather than a newer one. If they are truly depending on features of a new version, they can always request that the publisher add this, but it’s more likely that the author would just revert to an older version so they could get their article to compile.

It’s also possible that an author would include a dependency that we don’t support. Our LaTeX docker image was built mostly to include everything unless it had a specific feature we would not support. Examples include anything that requires calling out to a shell, or tcolorbox (because it doesn’t coexist with the lineno package), or savetrees. ACM is far more restrictive in what LaTeX packages they support. They only support things in their official list. I don’t have any way to evaluate whether a typst package should be supported, unless it made modifications to layout that were inconsistent with our journal style. The first thing I found in this category is hydra, which we would not support. It would be painful to go through the entire list of typst packages.

In our LaTeX document class, we have a few \@ifpackageloaded macros to flag packages that we definitely do not support (like natbib). That way we warn authors early in their authoring process not to start using something that they can’t submit. I don’t know how to accomplish that in typst, but it would be very annoying to authors if they invested a lot of time in using a package that turned out to be unsupported. It’s like what authors face now if they invest a lot of effort in writing their document in typst, only to discover that they have to convert it to LaTeX for the publisher. We should be more respectful of authors. There have been a lot of complaints about the ACM system when an author uses a package that they later discover is not supported. We only want to block packages if they are inconsistent with the style of the journal, but we’re not going to spend the time checking over all of the packages (or checking for things in the document that would violate the style). We literally only have to spend 5-10 minutes per article doing copy editing because LaTeX does a lot of the work for us.

The problem of fonts might come up. Luckily we have not seen authors try to typeset their entire article in comic sans. On the other hand, they might use variations on a greek font to achieve some specific mathematical notation. We would only use FOSS fonts and we would distribute those with our template. One question is whether there is any way to warn the author that they are using an unapproved font in their document at runtime. These runtime checks are quite important to help the author comply with journal style. If we wait until they upload their article then it’s too late and we’ve just annoyed the author. I asked gemini and it suggested the following (ignore their actual font selections):

#let approved-fonts = (
  serif: "Libertinus Serif",
  sans: "Fira Sans",
  mono: "Fira Code",
  math: "Libertinus Math"
)

#let project(body) = {
  // Enforce the approved fonts globally
  set text(font: (approved-fonts.serif, "linux libertine"), fallback: false)
  set math.equation(font: approved-fonts.math)
  
  // You can also restrict show rules to prevent unauthorized overrides
  show text: it => {
    // Logic to verify font usage could go here
    it
  }
  
  body
}

It argued that having fallback: false would be enough to enforce font restrictions. Since I’m not yet a typst programmer I will have to rely heavily on these kind of suggestions. Unfortunately gemini asserted that there is no way to control whether a document loads a package like hydra unless we control the compilation environment.

[Kevin_McCurley]

I should perhaps mention something else about the adversarial relationship between publishers and authors. Without getting into details, we have enforced some things in our document style that authors have objected to. In the end the decision comes down to what the editor-in-chief says, and there are examples where the article went fully through peer review, was accepted, but the author refused to cooperate on complying with the style. In the end, the article was blocked from publication. Ideally these things should not arise, and the author should be aware from the beginning that there was a conflict. That’s another reason why we require compile time checks on things. Perhaps the author can code around them (they certainly can in LaTeX) but then they have to know that they are willfully violating what the journal implements. If it gets to the point where the author is submitting their final version, it’s too late for our satisfaction.

[ensko]

Thanks for the detailed feedback! I think that does identify some issues to work on.

\@ifpackageloaded does not exist in Typst; the simple but crude way to find offending packages would be something grep style; I think the way to go would be a compiler feature for black- or whitelisting certain packages/versions. That would allow authors to receive errors early when they use too recent package versions, when they use e.g. hydra, etc.

I don’t think letting a template decide that would fit Typst well, but one way or another, if it’s necessary for publishing, it will be doable. I will set a reminder to open an issue about this later. Support allow/denylists for packages/versions as dependencies · Issue #8545 · typst/typst · GitHub

@Kevin_McCurley

tcolorbox (because it doesn’t coexist with the lineno package)

Does that mean that you use the lineno package in your document class and therefore tcolorbox is not possible, or that since an author can’t use both you decided that you’d rather include lineno? Just asking because that is one thing I wondered about before when the topic was on incompatible packages in a distribution.

@Kevin_McCurley

One question is whether there is any way to warn the author that they are using an unapproved font in their document at runtime.

I don’t think there’s a good way, and I’m fairly certain what Gemini has produced doesn’t work. That will make the default something that can’t accidentally fall back to a different font (for unsupported glyphs), but it will not prevent authors from setting a new font. That would have to happen through --ignore-system-fonts --font-path ... making other fonts inaccessible, I think.

[Kevin_McCurley]

The article goes through three phases and gets compiled each time. When the author compiles during development, they can see what it will look like. When they upload it, the document gets compiled the same way so we expect it to look exactly like what the author was working with. When they are satisfied that it looks right and they no longer have fatal errors, they click a button to submit it for copy editing and it gets compiled with the lineno package (the document class reads a config file when it runs). This is what the copy editor sees, and the only difference should be that it includes line numbers. When the author responds to all of the items from the copy editor, they upload their final version, which gets compiled without line numbers. After looking back at this, I think we can make it work with tcolorbox but it requires a change to the document class. I think there are about 90 other packages that are explicitly forbidden (among the 5000 packages in texlive).

The problem with the command line option is that we would have to tell authors to use it, and we have no control over the environment they use - we only have control over the style we give them in the form of a typ file. It sounds like it would be hard for us to provide a good environment for authors just using a typ file. Other publishers (ones that charge an arm and a leg) might be willing to devote the human effort to detect violations by the author, but we want an automated system. We could make something work, but it would probably be annoying to authors when our compiler system works differently than the one they use.

I should say that I don’t think our system is in any way similar to what other publishers use. Ours was specifically designed to eliminate human labor by the journal.

I’ve spoken to the PreTeXt team and I concluded that they are even further from being able to support journal articles. They are focused on books, or rather the online interactive form of books.

[aarnent]

@ensko

I don’t think there’s a good way, and I’m fairly certain what Gemini has produced doesn’t work.

What gemini has produced didn’t work, but there’s a way to make it work:

show text: it => {
  let fonts = text.font
  if type(fonts) == str {
    fonts = (fonts,)
  }
  if type(fonts) == dictionary {
    fonts = (fonts.name,)
  }
  fonts = fonts.map(it => if type(it) == dictionary {
    it.name
  } else {
    it
  }).map(lower)

  for font in fonts {
    assert(
      font in approved-fonts.values().map(lower)
      or font == "linux libertine", // manually set fallback font, remove if needed

      message: "The font " + font + " is not approved"
    )
  }
  it
}

Similarly you could use a show rule to check that e.g. citations use the correct format or that figures have alt text.

@Kevin_McCurley

It sounds like it would be hard for us to provide a good environment for authors just using a typ file

Besides fonts and packages discussed above, can you think of other things you would need to be checking automatically? I think a similar show rule approach as above could automatically enforce many things needed for a good submission pipeline, at least it should be possible to enforce many styling requirements.

[Kevin_McCurley]

I don’t want to distract too much, because in order for typst to become accepted to publishers, what really matters is if the big publishers like ACM, IEEE, Springer, and Elsevier accept it. Also maybe OJS, but they publish almost no mathematics where typst is most useful. I thought about writing a LaTex plugin for OJS but I gave up. It might be useful for someone to write an OJS plugin for typst.

Our workflow uses a parser for the bibtex and latex output logs that looks for things like an overfull hbox that is too large, or any references that don’t resolve, font substitutions that could go wrong, unicode characters that don’t exist in the font, bad crossreferences in bibtex, missing mandantory fields in a bibtex entry, and some other things. Some of these things (like overflowing content) might be difficult to detect from the output of the typst compiler. How would it detect when content overflows outside the margin or overlaps other elements? We want to do this programatically - not just visually. Sometimes an overfull hbox is not important if they are inside another box, but we need to have them identified for inspection.

Our compilation environment also uses bibexport to extract the bibtex entries that are actually used in the uploaded bibtex file, and then we check those for required fields. We also check for a missing DOI on an @article entry. We would need something like bibexport to extract the relevant bibtex entries, because authors often upload a giant bibtex file that contains articles they don’t actually cite. We only want the references that are actually used in the article.

Our workflow is unusual because our LaTeX document class produces an external file when you run latex. This file contains all of the metadata that the author used in their document (e.g, author names, surname, orcid, email, affiliations with country, ROR, and some other fields). Some of this data is validated when the document is compiled, but we also use a parser for the external metadata file to check for required fields. As an example that we do in python, we validate the checksum on an ORCID to make sure it looks legit. In typst I gather that is impossible to write an external file, so we would have to use so we’d have to use query on the document itself and then validate things in typst. Unfortunately, I found the documentation for query to be difficult to understand, but I assume we could make it work. Examples of things we check are:

• math characters in an author name
• one journal requires a country on every affiliation
• a journal may require an ORCID for every author or at least the submitting author.
• a journal may require at least one affiliation for each author (“Self” or “Unaffiliated” is accepted, but something has to be declared).
• We validate the checksum on the ORCID.
• We check that the name on the ORCID is a close match to the name of the author (authors sometimes cut and paste the wrong value).
• we are collecting ROR IDs for affiliations (much like ORCID). We validate those.
• We check that the abstract can be converted to HTML.

On the subject of HTML, getting HTML output is becoming more and more important for publishers. At present we are experimenting with latexml (like arxiv) but we do not have a complete solution yet. At the moment we convert metadata, abstract, and biliographic references to HTML. It will be very important for typst to implement HTML conversion without loss.

[sijo]

I think we’re all more than happy to get into the details of the difficulties faced by an actual publisher even if it’s not the most representative of the industry And it looks like automated workflows is precisely an area where the developers want Typst to shine (see for example this blog post).

The overflow detection is indeed a problem: I don’t thing there’s a general way to get such warnings from Typst currently (see this issue). This will probably need a change in the compiler.

Regarding bibexport: using the new typst eval command (which replaces typst query) you can do:

typst eval --in document.typ 'query(cite).map(x => x.key).dedup()'

This will return the list of unique citation keys used in the document, as a JSON array.

For the article metadata, you probably have a template that accepts a list of author dictionaries as parameter. You can store this in a metadata element:

// In template.typ
#let template(authors, abstract, doc) = {
  [#metadata((authors: authors))<article-metadata>]
  doc
}

which authors use like this (using other fields but it should make no difference):

#import "journal.typ"

#show: journal.template.with(
  authors: (
  (
    name: "Johanna Swift",
    department: "Primary Logistics Department",
    institution: "Delivery Institute",
    city: "Berlin",
    country: "Germany",
    mail: "swift@delivery.de",
  ),
  (
    name: "Egon Stellaris",
    department: "Communications Group",
    institution: "Space Institute",
    city: "Florence",
    country: "Italy",
    mail: "stegonaris@space.it",
  ),
 ),
)

// Article content

Then running

typst eval --in document.typ 'query(<article-metadata>).first().value' --pretty`

will give you JSON you can use to check everything you need. In the above example this command returns:

{
  "authors": [
    {
      "name": "Johanna Swift",
      "department": "Primary Logistics Department",
      "institution": "Delivery Institute",
      "city": "Berlin",
      "country": "Germany",
      "mail": "swift@delivery.de"
    },
    {
      "name": "Egon Stellaris",
      "department": "Communications Group",
      "institution": "Space Institute",
      "city": "Florence",
      "country": "Italy",
      "mail": "stegonaris@space.it"
    }
  ]
}

(You could also do the validation directly in Typst but exporting the metadata to JSON is probably easier to integrate in your workflow.)

To check that the abstract can be converted to HTML you could try the actual compilation to HTML to see if you get an error (maybe with --input check-abstract=true on the command line, which your template could detect to produce only the abstract) but I don’t know if that covers everything you need. In any case HTML support is still experimental but improving fast and I think the Typst team has big ambitions for it.

[ensko]

@Kevin_McCurley

our LaTeX document class produces an external file when you run latex. … In typst I gather that is impossible to write an external file, so we would have to use so we’d have to use query on the document itself and then validate things in typst.

Since the recent Typst 0.15, you can also use bundle export, which actually lets you output additional files using asset():

#let template() = doc => context {
  if target() != "bundle" {
    // produce a regular output file
    return doc
  }

  // produce the main document and a metadata file
  document("main.pdf", doc)
  asset("metadata.json", json.encode(
    (a: 1, b: false)
  ))
}

#show: template()

...

Authors would compile as normal; when you run with bundle export, the PDF will land at <filename>/main.pdf instead of <filename>.pdf, and there will be a <filename>/metadata.json alongside it.

[gasche]

Two minor points:

1. I wrote a Typst template for the [acmsmall]{acmart} LaTeX configuration expected by some ACM conferences and journals, approximate-acmsmall. I encourage people to try to use it, and tell conference/journal editors about them. Currently it is possible to submit a Typst paper (with this template or another) to arXiv, asking for a PDF-only submission, but not possible to actually submit such a document to the ACM for publication, as the conference/journals I am familiar with (SIGPLAN-affiliated) ask for TeX sources to use in their own automated pipeline. I think that we need expressions of interest from prospective users to encourage editors to spend energy on Typst.

2. Regarding the question of having a Typst distribution: an alternative would be to have a small script to make a Typst document self-contained. The typst would fetch all external packages that the document relies on, possibly font files as well, and return a version of the document that builds from these local assets in a self-contained way, along with a metadata file describing where the packages are coming from and which Typst version is known to work for this self-contained document. The result would be highly reproducible.