Automatically including linked documents

Questions
, ,
1

It has probably been asked before, but I couldn’t find it. Is there a way to increase the maximum number of iterations from the default of 5 when using introspection ? 5 is a sensible limit on simple documents, but it can be limiting on more complex documents, especially with the new bundle feature.

My usecase :

I use the bundle feature to build a website, and also offer some pages as PDF. To avoid having to declare every pages in the bundle, I use a variant of the solution proposed in 8249

// template.typ
// Insert a link to target and requests the compilation of the document
#let inclink( target, src: auto, file-target: auto,
              path-cb: none, body, ..attrs
            ) = {
  src = if (src == auto) {
    if (target.split("/").last() == "") {
      target + "index.typ"
    } else if (target.endswith(regex("\.(html|pdf)"))) {
      target.trim(regex("\.(html|pdf)")) + ".typ"
    } else {
      target
    }
  } else { src }

  file-target = if (file-target == auto) {
    if (target.split("/").last() == "") {
      target + "index.html"
    } else {
      target
    }
  } else { file-target }
    
  if (path-cb != none) {
    [#metadata( ( target: target,
                  path-src: path-cb(src),
                  path-target: path-cb(file-target)
    ) ) <link-load>]
  }
  link(target, body, ..attrs)
}

// bundle.typ
#document("/index.html", include("/index.typ"))

#context {
  query(<link-load>)
    .dedup(key: a => a.value.path-target)
    .map(a => document(
      // parse the path representation to extract it as a string
      repr(a.value.path-target).slice(6, -2),
      include(a.value.path-src)
    ))
    .join()
}

// index.typ
#inclink("./page1/")[Page 1]    // Generates out/page1/index.html from src/page1/index.typ

This works properly, but it has the issue of consuming an iteration for every layer deep I need to find links. Every page is 2 click away from the homepage so that isn’t a problem in itself. The problem is that the deepest documents now only have 3 iterations to converge, which fails occasionally. The issue would be even worse on a more complex website or with more complex documents. In my case, increasing the limit to 6 would be sufficient.

I know the intended way to use the bundle feature is to declare every document explicitly and use labels to link to them, but I find that having a 1-1 correspondence to the filesystem is more convenient. It also makes it easier to export only parts of the site.

1 Like
2

I believe the layout iteration is not designed for that.

It’s better to generate the list of sources in a single interation.
For example, you can pass data from external systems to typst with sys.inputs / --input:

# list_sources.py
from pathlib import Path
import json

root_dir = Path(__file__).parent
sources = [p.relative_to(root_dir).as_posix() for p in root_dir.glob("**/*.typ")]
print(json.dumps(sources))

// main.typ
#let sources = json(bytes(sys.inputs.sources))
#for src in sources {
  document(
    if … { src.replace(".typ", "/index.html") } else { … },
    include src,
  )
}

typst compile --input "sources=$(python list_sources.py)" main.typ …

Directory walking · Issue #2123 · typst/typst · GitHub also has a few useful tips. Someone even came up with a way to list files by reading the .git/ folder.

3

The python solution doesn’t quite cut it because the files don’t match exactly 1/1. Some documents are split over multiple files, and others share source files (making both pdf and html).

I’ve figured out a way using command line queries.

# build.sh

filelist='[{"value": {"path-src": "path(\"/index.typ\")", "path-target": "path(\"/index.html\")"}}]'

until [ "$new_filelist" = '[]' ]; do
    prev_filelist="$filelist"
    new_filelist=$(for p in $(echo "$new_filelist" | jq '.[]."value"."path-src"'); do
                       p="${p%'\")"'}"
                       p="${p#'"path(\"/'}"
                       typst eval --features bundle,html --diagnostic-format short --target html --root ./ --in "./""$p" 'query(<link-load>)'
                   done | jq -s 'reduce .[] as $x ([]; . + $x)' | jq -c 'unique')
    new_filelist="$(jq -n --argjson A "$filelist" --argjson B "$new_filelist" '$B - $A' | jq -c 'unique')"
    filelist=$(jq -n --argjson A "$filelist" --argjson B "$new_filelist" '$A + $B' | jq -c 'unique')
done

echo $filelist > "filelist.json"

typst $1 --features bundle,html --format bundle --pretty bundle.typ out/

// bundle.typ
#{
  json("filelist.json")
    .dedup(key: a => a.value.path-target)
    .map(a => document(
      // parse the path representation to extract it as a string
      a.value.path-target.slice(6, -2),
      include(a.value.path-src.slice(6, -2))
    ))
    .join()
}

The shell script is pretty inefficient, because it calls typst eval on every file on every iteration. There is probably a way using jq to only inspect the files that are in new_filelist but not in prev_filelist. Or maybe I should use python instead of jq.
Each inclinked target file triggers exactly one typst eval call.

1 Like