Overriding template parameters; missing social convention or Typst design flaw?

[infomorphic-matti]

I believe I’ve come up with a useful solution to this problem, that’s granular, modular and yet reasonably user-friendly, that also avoids the issue of forcing people to fork to do more powerful customisation, based off of the idea of typed-tagless-final-form coding strategies (more specifically the idea of pluggable “interpreters” with an extensible “dsl”-ish thing of custom elements or pseudo-custom elements) and a little bit of looking at the source of the tidy typst package.

I actually describe two solutions, one suitable for typst in it’s current state, and one based off the same ideas but with careful usage of currently-hypothetical custom element types and probably custom fields (note that neither of these solutions can solve the recursion issue).

This post is split into 5, because of how long it is. (This is part 1 of 5, here’s hoping I don’t get flagged as spam…).

Hypothetical Future Solution

The hypothetical future solution is that packages, rather than merely defining custom types with basic fields, instead also define compositional steps in their final “show” function - to arbitrary degrees of nesting - as fields storing closures/lambdas, overwritable with set rules (or perhaps some new custom behaviour system if there was to be specific language support, which would in theory be able to actually replace show rules entirely…).

Then, with clever use of defaults to compose more complex behaviours out of less complex behaviours, each level of behaviour can be modified mostly independently, all the way up to the final show functionality. If other parameters (“value” parameters) were also modified to accept a closure returning their values, this could be extended all the way down to enable near-arbitrary calculation of the parameters also.

This sort of clever overriding - if combined with custom fields - would also allow users to modify and control the parametrisation of types too, if they modified the more granular behaviours in-sync to use their new mini-interfaces.

Basically, it enables precisely the degree of customisation that a user wants, with precisely the degree of modularity and precision needed, no more or no less. And it means that it’s possible to independently do modifications of individual behaviours as long as they don’t conflict (and even if they do it may be possible to write a third module to bridge over any modifications to the input or output parameters of modified functions). We could, for instance, imagine people distributing packages containing specific tweaks to other popular packages or templates, without having to fork them.

As for stuff like automatically generating content, you would probably want a form of @Mc-Zen’s solution, but with more granular content generator functions. Configuring basic parameters of your templates (e.g. title) becomes just another set rule (or many of them, in one-or-more modular/composible template-providable functions, similar to how we currently do it). You could also provide the traditional single-function option, as now things like behaviour are controlled granularly by set-rules if so desired anyway.

Potential Future Example

This may make more sense if I write out some basic examples as an illustration. They are only examples, but may illustrate what I mean when I say “build up behaviours” (in some sense we would be building up a DSL) .

They will use the pseudo type syntax that has been used earlier, though I will also add behaviour as an alias to field to differentiate the ones that are primarily intended for closure storage (they work fine with normal fields though, and indeed adding separate syntax would likely be undesirable since it’s likely that you’d want to be able to potentially vary the types or do something with auto and auto-replace it with the default). Also I use Rust a lot so some rusty syntax/styles might sneak in :p

#type axis {
  ...
  // Behaviour that makes the tick graphics from e.g. some internal function
  behaviour make_ticks: (self) => {... some content ...};
  // Behaviour that rotates them (the ticks created by make_ticks) as needed 
  // e.g. rotate y ticks by tau/4 radians. 
  behaviour rotate_ticks: (self, ticks) => {... some content ...};
  show it => { ... }; // `it` is `self`, essentially. 
}

#type graph {
  field page-title: error,
  field title: none,
  field x-bounds: (-1, 1),
  field y-bounds: (-1, 1),
  field resolutions: (100, 100),
  // Meant to be basic closure (x: float, y: float) -> bool
  field plot: (x, y) => false,
  /* More fields here*/,
  
  // Behaviour for "rendering" the plot into a resolutions[0] x resolutions[1] 
  // array of colours
  behaviour render: (self) => {
    let axis_diffs = (self.x-bounds, self.y-bounds).map(((start, end),) => end - start);
    let (x_celldiffs, y_celldiffs) = self.resolutions.zip(axis_diffs).map(((res, delta),) => delta/res);
    range(0, self.resolutions.at(0))
      .map(xidx => range(0, self.resolutions.at(1))
        .map(
          yidx => ((xidx + 0.5) * x_celldiffs, (yidx + 0.5) * y_celldiffs)
        ).map(self.plot).map(write => if write { black } else { white })
    )
  },
  
  // Get the "tick" coordinates for the axis (one of "x" or "y") to an array of the relevant coordinate
  behaviour ticks: (self, coord_axis) => { ... };  
 
  // Make the two "axes" from the ticks -> (x_axis_content|none, y_axis_content|none)
  behaviour make_axes: (self, x_ticks, y_ticks) => {
    // more axis special behaviour or wrapping goes here.
    let x_axis = [#axis(tick-coords: x_ticks)];
    let y_axis = [#axis(tick-coords: y_ticks)];
    (x_axis, y_axis)
  },

  // Embed graph content from output of render()
  // (self, render_output) -> content
  behaviour embed_graph_content: (self, render_output) => { ... },
  // combo behaviour for the graphs.
  // (self) -> content
  behaviour create_graph_content: (self) => { self.embed_graph_content(self.render()) },
  // Create the axes content in full.
  // (self) -> {x_axis_content, y_axis_content}
  behaviour create_axes(self) -> { 
    let x_ticks = self.ticks("x");
    let y_ticks = self.ticks("y");
    self.make_axes(x_ticks, y_ticks)
  },

  // in reality, you'd probably just use a basic parameter for this level of basic formatting but it's an example xD
  behaviour add_background: (self, content) => { ... some content ...}

  // fancy layout calc 1
  behaviour fancy_layout_calc_1: (self, graph_content, x_axis_content, y_axis_content) => { ... },
  // fancy layout calc 2
  behaviour fancy_layout_calc_2: (self, graph_content, x_axis_content, y_axis_content) => { ... },

  show self: self => {
    let (x_axis, y_axis) = self.create_axes();
    let graph_content = self.create_graph_content();
    let layout_info_1 = self.fancy_layout_calc_1(graph_content, x_axis_content, y_axis_content);
    let layout_info_2 = self.fancy_layout_calc_2(graph_content, x_axis_content, y_axis_content);
    self.add_background({
      ... more title stuff here with self.title or a custom rendering/layout/align function if not just a param. 
      place(dx: layout_info_1.graph, dy: layout_info_2.graph, graph_content);
      place(dx: layout_info_2.x-axis, dy: layout_info_1.x-axis, x_axis);
      place(dx: layout_info_1.y-axis, dy: layout_info_2.y-axis, y_axis);
    })
  }
}

// Module itself provides a function that does a bunch of set rules including for #graph.
// May also provide sub-functions that do some subset, and the user can always manually override
// individual parts. Usable with the standard .with and `show`-rule mechanisms
#let mainmodule.conf(page-title, body) = {
  #set graph(page-title: page-title)
};

// user wants to make graph have a higher resolution by default, and style text in the axes by default
#set graph(resolution: (200, 200))
#show axis: set text(weight: "bold")

// separate module wants to modify the graph to use png rendering, 
// overriding render to output png data, and create_content to use it rather than the normal format.
// can be used by standard `show:` method, and/or could provide a function that the user can manually 
// make a `set`-rule from.
// The module could also modify create_graph_content to make it ignore render() entirely.
let extension-module.make-graphs-png(body) = [
  #set graph(
    render: (self) => { ... fancy png generator ...}, 
    embed_graph_content: (self, render_output) => { ... modified for png output ... }
  )
  #body
]

// now produces an output with all the modifications (at least the 
// compatible ones without blatant conflict),  including the formatting on
// the axes, the increased default resolution, and the png version of the output. 
#graph( ... )

Here we note some important things:

• Almost all configuration - including of behavioural chunks - is done through set-rules, and/or direct specification (as they are fields). This makes it modular:
  • You could, for example, imagine custom “tweak” modules that apply highly precise set-rules, that could well be combined unless they change the actual interface of the functions (and even then they only affect other modules that do something similar).
  • Advanced configuration knobs for the implementing module could also be done in this way, e.g. if there was a template for a journal, but that journal had slightly different needed layouting logic for different “sections” or versions, the template could tweak the logic with as little or as much granularity as needed.
• Simpler properties remain trivial via set-rules.
• You may have noticed that in my definitions for “behaviours”, I rarely called out to other behaviours and stored their results for processing within the same behaviour, other than to provide more computational structure. This is deliberate:
  • By doing things this way, it makes it easier to - given the current behaviour - perform transformations on the inputs and outputs of that behaviour that preserve the interface, without needing to rewrite the internal computation structure. If there was or is a way to obtain the current “default/set” value of a set rule, then it would be fairly simple to write a new function that wraps it with some more custom logic, for simple-ish cases.
  • This way also helps separate behaviour from data (in a similar way to Rust traits) - in a conceptual sense, the “self” parameter is performing two functions:
    • Provide the values of normal fields.
    • Select the “implementation” of behaviours on request, enabling the indirection required to actually plug behaviours together even when they have been modified with set-rules. This should only be done for places where the rules logically combine into a larger chunk of computation that makes logical sense to replace at-once.
      • This is actually quite design-specific and it somewhat depends on degree of preference (and in the extreme case, you can modify the “last” value-producer and make it recalculate stuff from earlier using the embedded behaviours, the system is flexible enough to enable overrides producing almost any design).
      • The actual tradeoffs of this are not something I’ve gone into too much detail of but some of it involves deduplicating effort, but then by making more “nested” computations you can sort of build up a more nested set of levels of granularity for overrides, and the memoizing system of typst may make the potential performance advantage totally irrelevant.
      • It may help to conceptually imagine this implementation as both a “custom element type” with normal “fields”, combined with a nested structure of rust-like traits-and-impls which can be replaced in whole or in part by set-rules, and where there is a sort of “layered” structure of each “implementation” depending on the more granular “implementations” (but the user can also replace the “less granular” impls in a chunk). All leading up to a final “implementation” of a show-rule, which is the pseudotrait implemented by custom types more generally for custom rendering.

One Potential Drawback

The main potential drawback (other than the risk of conflicting packages etc. and maybe performance depending on impl details) I see of this is that of the issue of mandatory parameters/fields for custom types - for instance, if you had a type that required a title field, and no sensible default exists, you can currently only supply it as a positional argument and hence you can’t use set rules to configure it.

This means that either Typst would need to add mandatory named arguments, or you’d need to set a default that produces an error if unset (with manual checks, unless e.g. we added a special let func(a: error("optional fail info"), b: (:)) sort of thing, which would not be unreasonable. This could also be done by setting mandatory fields to be behaviours with a default closure that panics.

In terms of language-server-wise stuff, it may also be worth considering a way to “hide” custom fields if this system were to become popular for modularity purposes, or otherwise separate out “behaviour-like” fields from other fields either syntactically or otherwise for documentation clarity, even if they are the same thing internally.

[infomorphic-matti]

Solution in Typst (Current)

(Part 2 of 5)

The solution I’ve come up with (with influence from tagless final form and from tidy) for “current” typst, and usable in combination with the “hypothetical typst” solution presented above, involves using dictionaries to essentially emulate set-commands by acting as a sort of pseudo-tagless-final-form pseudo-“interpreter” for the “dsl” of the various internal operations to a typst package.

The Self-Interpreting Operation Dictionary - a Basic Tagless-Final-Form-Like System

To get an idea of the actual solution(s) in-full, I will first describe a single basic structure - what I will call the SIOD (Self-Interpreting Operation Dictionary). It looks something like the following (only in the most basic rendition, we get rid of the self-params in the body in a sec):

#let siod = (
  add: (self,) => {
    self.x + self.y
  }
  display_header: (self, title) => [=== #title],
  display_body: (self,) => [#(self.add)(self)],
  display: (self, title) => {
    (self.display_header)(self, title);
    (self.display_body)(self,)
  }
)

The functions here essentially provide a parameter for the SIOD dictionary itself to be passed in, allowing for indirection.self.x and self.y are mandatory parameters (you could also define them in the dictionary as “functions that panic” a-la x: (self, ) => { panic("missing mandatory definition of parameter x" }).

Consider then, if you wanted to modify the meaning of the add function to actually multiply stuff. You might do something like:

#let siod = siod + (add: (self,) => {self.x * self.y})

In this new “overridden” dictionary, when we call display, it will automatically use the new definition of add because of it’s internal indirection. In a more abstract sense, we’ve altered the “meaning” of the add function in the “DSL” that the dictionary defines and uses for its own definitions.

Similarly, if we wanted to add a new operation, say, multiply, and use that in display, we could do the following instead:

#let siod = siod + (
  display_body: (self,) => [#(self.multiply)(self)], 
  multiply: (self,) => (self.x * self.y)
);

This operation itself could be further overridden in future, affecting its “meaning” in any location where it has been used inside the “DSL”:

#let siod = siod + (multiply: (self,) => self.x * str(self.y))

To actually use the dictionary you would still need to define the x and y keys as well (or if you went for the option of making them functions that panic without definition, you’d need to define those :p).

“Compiling” the SIOD

It’s now time to get rid of those annoying self parameters in the bodies of functions (or indeed in the use of functions from the dictionary outside of the dictionary). We do this by an operation I will call “compiling”. The most basic version looks like the following:

#let compile(siod) = {
  let r = (:);
  for (k, v) in siod {
    if type(v) == function 
      r.insert(k, v.with(siod))
    } else {
      r.insert(k, v)
    }
  }
};

It essentially wraps the functions such that the self-parameter is provided automatically. I will describe some more advanced and useful forms later, but even this lets us do a lot.

For example, we can now simply “compile” a dictionary to access the functions in a way that lacks the first self param on every call - depending on the details of typst’s memoization, this should have minimal performance impact:

#let siod = (
  add: (self,) => {
    self.x + self.y
  }
  display_header: (self, title) => [=== #title],
  display_body: (self,) => [#(compile(self).add)()],
  display: (self, title) => {
    let self = compile(self);
    (self.display_header)(title);
    (self.display_body)()
  }
)

#let siod_with_nums = compile(siod + (x: 4, y: 10))
#let display = siod_with_nums.display
#display("This should be 14")

Recursion

Of course, shoving all of your structure into one flat dictionary would be incredibly… annoying, to put it mildly. As a starting point, we can therefore build a modified compile function that works with recursion. Something important to note here is that even in “inner” dictionaries, the functions must take the “outermost” dictionary as their first parameter, to be able to access all the functionalities of the modules.

Hence, the following modified compile function:

#let compile(siod, root: auto) = {
  let r = (:);
  let root = if root == auto { siod } else { root }; 

  for (k, v) in siod {
    if type(v) == function { 
      r.insert(k, v.with(root));
    } else if type(v) == dictionary {
      r.insert(k, compile(v, root: root)); 
    } else {
      r.insert(k, v);
    }
  };
  return r;
};

It can be used in the following way, noting that overriding something inside a sub-module is almost trivial by just rewriting the relevant sub-dictionary:

#let siod = (
  add: (self,) => {
    let self = compile(self);
    self.x + self.y
  },
  display: (
    header: (self, title) => [=== #title],
    body: (self,) => [#(self.add)(self,)],
    // use `new` cus it's rusty 
    new: (self, title) => {
      let self = compile(self);
      (self.display.header)(title);
      (self.display.body)()
    }
  ),
);

#{
  siod = siod + (add: (self, ) =>  self.x * self.y) + (x: 3, y: "trans");
  siod.display = siod.display + (header: (self, title) => [== #title]);
  siod = compile(siod);
  (siod.display.new)("hi") 
}

Advanced Compilation for More Structured Packages

In typst packages, typically, you’d use multiple files, or typst modules, for more complex systems. So far we’ve just shoved things into a giant dictionary, which is necessary as an endpoint, but I don’t think we’d want to write all our code that way.

Luckily, typst has a function that can turn modules into dictionaries (dictionary), and we can be smart about it as well during compilation, creating various potential mechanisms to allow people developing this way to keep some functions in the main module “not part of” the compiled SIOD, e.g. some sort of special variable name that if present prevents certain functions/vars being included in the dictionary (as well as one for dictionaries more generally that indicates things not to transform as part of the compilation process).

Note that it’s important to make this a denylist (rather than allowlist) to enable customisation with new functions.

This is a way to define the new compile function, along with a couple of utility functions and a “module/dictionary interface” to allow control over what parts of the module get embedded into the dictionary at all, and also to control which ones shouldn’t be transformed by the compilation process:

// Module functions to delete/not include when it first gets converted into a compiled SIOD.
#let _siod_module_exclude = (
  "compile",
  "module-convert",
  "make-filter-set"
);
// Dictionary keys that should not be transformed by compilation. Can also be used in normal dictionary defs.
#let _siod_dictionary_ignore = ();

// If something is a typst module, convert it into a dictionary for the SIOD system. 
// If you were to make a module encompassing this methodology, you may want to automatically
// exclude the module itself from the produced dictionaries.
#let module-convert(mod) = {
  if type(mod) != module { return mod; } else {
    let mod = dictionary(mod);
    if ("_siod_module_exclude" in mod) and type(mod.at("_siod_module_exclude")) == array {
      for exclude_key in mod._siod_module_exclude {
        let _ = mod.remove(exclude_key, default: none);
      }
      // remove this key as well as it's no longer a module
      let _ = mod.remove("_siod_module_exclude", default: none);
    };
    return mod;
  }
 };

// Make a dictionary with keys that are the elements of the array, mapped to `true`
#let make-filter-set(disallowed) = {
  let r = (:);
  for k in disallowed { r.insert(k, true); };
  return r;
};

// Modified compile function
#let compile(siod, root: auto) = {
  let r = (:);
  let root = if root == auto { siod } else { root };
  let siod = module-convert(siod);
  let filter-set = make-filter-set(
    siod.at("_siod_dictionary_ignore", default: ()) 
    + ("_siod_dictionary_ignore",)
  );

  for (k, v) in siod {
    if (type(v) == function) and (k not in filter-set) { 
      r.insert(k, v.with(root));
    } else if (type(v) == dictionary) and (k not in filter-set) {
      r.insert(k, compile(v, root: root)); 
    } else {
      r.insert(k, v);
    }
  };
  return r;
};

This can (assuming the above code is in siod.typ), then, be used by a package like follows (as an example - it may be more ergonomic to use an inner-module rather than composing a top-level dictionary by-hand):

In display.typ:

#import "siod.typ"
// This may be done automatically by an improved version
#let _siod_module_exclude = ("siod",); 

#let header(self, title) = [===#title]
#let body(self) = { 
  let self = siod.compile(self); 
  (self.modify-content)([unspecified body]) 
};
#let new(self, title) = {
  let self = siod.compile(self);
  (self.display.header)(title);
  (self.display.body)();
};

In main-pkg.typ:

#import "display.typ"
#import "siod.typ"
#let sym = (
  display: display,
  modify-content: (self, content) => rect(self.content-count * content, fill: red)
);

#let conf(content-count, sym: sym, content) = {
  let sym = siod.compile(sym + (content-count: content-count));
  show header: sym.display
};

In user.typ:

#import "main-pkg.typ"
#show: main-pkg.conf.with(
  4,
  sym: main-pkg.sym 
  + (modify-content: (self, content) => [#content; (cnt: #(self.content-count))])
);

Something important to note here, however, is that we’ve lost an ability by adding modules in the dictionary. In particular, while siod.compile converts everything to dictionaries, the actual non-compiled versions of it that you modify yourself contain modules.

Without converting these modules to dictionaries, it is impossible to modify any field that isn’t in a dictionary (in this example, that means all fields that are not in the root sym dictionary, where sym is short for symantic, from the same term in Tagless Final Form code).

This is not great. Even without this issue, though, overriding any more than one item at a time would require significant numbers of calls and just generally be an unwieldy mess - and it also doesn’t look much like the set rules (or show/set rules) we are familiar with.

Hence, the motivation for the next part - a proper overrides system capable of modifying an existing compiled construct and also un-compiled symantic dictionaries

A Proper Override System

An initial prototype of a proper overriding system may look something like the following:

// Override parts of `first` (a SIOD (or module) that could e.g. contain modules) 
// with `second`,  which is a nested dictionary/module structure in the 
// same format, that may be used to modify keys (supplying a dictionary or module will 
// apply the resulting item by-key, if the same component in `first` is also a dictionary or 
// module (which will be converted to a dictionary).
#let override-single(first, second) = {
  // handles edge-cases when passed a non-kv argument.
  // also means that modules are not converted into dictionaries unless by-key overrides will
  // actually happen.
  if (type(first) != dictionary and type(first) != module) 
    or (type(second) != dictionary and type(second) != module) { return second; };
  let first = module-convert(first);
  let second = module-convert(second);
  for (k, new-v) in second {
    // There is a debate to be had here as to if we should use `_siod_dictionary_ignore`, 
    // however, users are likely to want to be able to override individual components
    // even of un-compiled dictionaries (e.g. for parameter bundles)
    if (type(new-v) == dictionary or type(new-v) == module) and (k in first) {
      let old-v = first.at(k);
      if type(old-v) == dictionary or type(old-v) == module {
        old-v = override-single(old-v, new-v);
        first.insert(k, old-v);
      } else {
        // quick branch if old-v is not dict/module. Can just assign directly :)
        first.insert(k, new-v);
      };
    } else {
      first.insert(k, new-v);
    }
  }
  return first;
};

// Override the `first` with all the rest, applied in order of being provided.
#let override(first, ..rest) = {
  let first = first;
  for next in rest.pos() {
    first = override-single(first, next);
  }
  return first;
};

This little snippet of code can sensibly override - in a nested fashion - dictionaries and modules, applying it in-order to a sequence of overriding items. For example (not with an SIOD specifically):

#let data = (
  display: (
    content: [rect(width: 5pt, height: 5pt, stroke: red)]
    title: "Red Rectangle"
  )
  text-colour: rgb("55aa55")
  document-title: "Example Red Rect"
);
#let data = siod.override(
  data, 
  (text-font: "Droid Sans", text-colour: green),
  (
    display: (
      content: [rect(width: 20pt, height: 20pt), stroke: green]
    ),
    document-title: "Example Green Rect"
  ),
  (display: (title: "Green Rectangle"))
);

At the end of this, data looks roughly like:

#(
  display: (
    content: [rect(width: 20pt, height: 20pt), stroke: green],
    title: ("Green Rectangle")
  ),
  text-colour: green,
  text-font: "Droid Sans",
  document-title: "Example Green Rect"
);

This basic initial overriding system provides the root means to override nested SIOD configurations. However, it has the limitation of having a deep bracket nesting, especially when overriding deep styles (avoiding this requires something like ops.display.mechanism = siod.override(ops.display.mechanism, (/* your overrides here*/))), which is less ergonomic.

A second issue is that, when you are working with compiled SIODs - as you likely will be most of the time - this overriding mechanism is currently almost useless because all of the functions are actually using data from the “embedded” uncompiled SIOD to do everything - if you alter the “compiled” SIOD, it will make no difference to anything else calling the function you modified.

The third issue is a bit less obvious, but important for ergonomics. Unlike typst-native show-rules, which get access to the “item” that is created - as well as its parameters - and can transform it rather than duplicating all inner work of the existing function to recreate the original, the current system for overriding things has no way to obtain the value being overridden and transform it to create the new value. This is the issue we will solve first, as the other two are in the interaction between the compile system and the override system.

Fully Featured Override Functionality

Solving the third issue is comparatively trivial, as typst is a functional language. In particular, even within a dictionary, writing closures that return another closure is clean - it looks something like (key: (..) => (..) => output) - so we can simply modify the way structures are merged together in the override system to detect when a closure is present and call it on the old value to obtain the new value.

More particularly, we can generalise the overriding functions to enable customisation of how to process overrides, by passing a closure. This, then, results in the following improved override functions:

// Both of these can't simultaneously be module|dictionary
// That is handled by the main override function.
#let default-apply-override(first-elem, second-elem) = {
  if type(second-elem) == function {
    return second-elem(first-elem);
  } else {
    return second-elem;
  }
};

#let plain-apply-override(first-elem, second-elem) = second-elem;

// Override parts of `first` (a SIOD (or module) that could e.g. contain modules) 
// with `second`,  which is a nested dictionary/module structure in the 
// same format, that may be used to modify keys (supplying a dictionary or 
// module will apply the resulting item by-key, if the same component in `first` 
// is also a dictionary or module (which will be converted to a dictionary).
//
// If the second elements are functions, they will be passed the value in first 
// to generate the new value, unless a different application method (function 
// taking first-elem and second-elem as two arguments) is provided. If the key is 
// not present in `first`, then `auto` will be passed.
#let override-single(first, second, override-method: default-apply-override) = {
  // handles edge-cases when passed a non-kv argument.
  // also means that modules are not converted into dictionaries unless by-key overrides will
  // actually happen.
  if (type(first) != dictionary and type(first) != module) or (type(second) != dictionary and type(second) != module) {
      return override-method(first, second); 
  };
  let first = module-convert(first);
  let second = module-convert(second);
  for (k, new-v) in second {      
    let old-v = first.at(k, default: auto);
    // There is a debate to be had here as to if we should use `_siod_dictionary_ignore`, 
    // however, users are likely to want to be able to override individual components
    // even of un-compiled dictionaries (e.g. for parameter bundles)
    if (type(new-v) == dictionary or type(new-v) == module) and (type(old-v) == dictionary or type(old-v) == module) {
      old-v = override-single(old-v, new-v);
      first.insert(k, old-v);
    } else {
      old-v = override-method(old-v, new-v);
      first.insert(k, old-v);
    }
  }
  return first;
};

// Override the `first` with all the rest, applied in order of being provided, with
// dictionaries or modules being recursively applied to dictionaries or modules
// in the combination that is being-applied-to.
// 
// By default, if the `rest` contains a closure at a key, that closure is passed the 
// old value (auto if not present) to create the new value. This can be controlled 
// with the `override-method` parameter if so desired, though in most cases when 
// you want to override a function value, you can just ignore the closure parameters 
// and return it directly ^.^
#let override(first, override-method: default-apply-override, ..rest) = {
  let first = first;
  for next in rest.pos() {
    first = override-single(first, next, override-method: override-method);
  }
  return first;
};

With this, if you had a SIOD like:

#let siod = (
  display: (self, ..) => [my content]
);

And wanted to, say, wrap it in a rectangle, you could use the override dictionary:

#let override-dictionary = (
  display: (orig) => (..args) => rect(stroke: red, width: 100%, orig(..args))
);

If you wanted to add additional components that are SIOD-interpreted, you can just capture self again, like:

#let override-dictionary = (
  display: (o) => (self, ..args) => {(self.custom-module.header)(self); o(self, ..args) }
);

Similarly, you can capture and use and manipulate other arguments at-will, for instance you could reuse some sort of original parameter for an argument in your additions.

[infomorphic-matti]

Synthesising a Solution

(Part 3 of 5)

To finally and fully synthesise an actual solution to the remaining two problems - namely:

• That when you have a compiled SIOD (as you likely will most of the time), you can’t easily override stuff unless you also keep around a separate copy of the non-compiled SIOD (which would be annoying)
• That overriding deeply nested structures without duplicating the deep nesting currently requires a… significant depth of bracketing.

Because the intended way to use my solution is primarily via “compiled” SIODs, and we have a very convenient function (compile) to add things in-context to recursive SIOD components, we will solve both these problems at the same time. Furthermore, we will add some other utility modifications, that solve ergonomic problems that haven’t quite come up yet (e.g. automatically creating an argument for generic call-local overrides).

The core principle here is to recognise that the compile function is capturing context, including the “current” containing uncompiled SIOD for any given key. With a little extra information in the call indicating the keys leading from the “root” parameter to the current SIOD, a compiled SIOD can know how to wrap overriding dictionaries such that they will apply to it’s internal structures when applied on the root uncompiled SIOD.

This solves the nested structure nested specification problem. To solve the issue of needing to carry an uncompiled SIOD as a separate variable to override things, consider again that the compilation process essentially “embeds” the uncompiled SIOD in each function, so we can just define a custom function in each compiled SIOD that emits a new version of the compiled SIOD with the overrides applied (or, well, provide various options to control things like the context-presence, override locality, etc.).

compile Redux

To solve both of these problems, we can define a new compile function (along with some helper functions).

// Alias for `override` to prevent param conflicts
#let fn-override = override;

// wrap the arguments in a chain (array of strings `path`) of 
// dictionary keys (i.e. `(a: (b: (c: actual-item)))`). Returns an array, one entry
// per positional argument
#let wrap-in-dictionary(path: (), ..components) = {
  let wrapped = components.pos();
  let path-rev = path.rev();
  for path-component in path-rev {
    wrapped = wrapped.map((v) => ((path-component, v),).to-dict());
  };
  return wrapped;
};

// remove the given `name`-ed argument from the arguments, 
// returning a pair (arguments, extracted).
//
// deletes the named argument from the arguments. If there is
// no argument actually provided with the name, returns `none`
// as the extracted value. 
#let remove-argument-or(args, name) = {
  let (pos, named) = (args.pos(), args.named());
  let extracted = named.remove(name, default: none);
  (arguments(..pos, ..named), extracted)
};

#let default-overrides-extractor(args) = (args, ());
#let default-overrides-applicator(self, overrides, args, path-from-root) = {
  let localised-overrides = wrap-in-dictionary(path: path-from-root, ..overrides);
  let modified-self = override(self, override-method: default-apply-override, ..localised-overrides);
  (modified-self, args)
};

// Take an array of overrides, and then make them absolute according 
// to the localisation spec and "key path from root" required to 
// retrieve the dictionary the overrides should apply to.
// This returns an array of overrides made "absolute" (relative to the 
// "root" SIOD), according to the `localisation-spec` parameter, which
// has the following format:
//  * If `true`, the overrides are "relative" to the `sym` containing the 
//    called function, reducing verbosity in the common case.
//  * If `false`, the override dictionaries are applied directly to the root `sym`, 
//    so the paths are "relative" to the root `sym` (requiring more bracket
//    nesting for deep overrides). 
//  * If an array of strings, represents what "path" relative to the root SIOD
//    the overrides should be interpreted as starting at.
#let make-absolute-overrides(overrides, localisation-spec, path-from-root) = {
  if localisation-spec == true {
    wrap-in-dictionary(path: path-from-root, ..overrides)
  } else if localisation-spec == false {
    overrides
  } else if type(localisation-spec) == array {
    wrap-in-dictionary(path: localisation-spec, ..overrides)
  } else {
    panic("localisation-spec must be boolean or array of strings");
  };
}

// Parse the "override argument specification" into a pair of closures used to make a 
// "compiled" function have an automatically-managed override first.
// `spec` can look like the following:
// * `none` - no automatic override argument should be added to compiled functions
// * `"argument name"` - just a single override argument is to be injected, 
//     equivalent to the spec:
//     ```
//     (
//       override: "argument-name", 
//       local: none,
//       local-default: true, 
//       override-method: none,
//       override-method-default: default-apply-override
//     )
//     ```
// * A dictionary of the form:
//     ```
//     (
//       override: "override-sym",
//       local: none,
//       local-default: true,
//       override-method: none,
//       override-method-default: default-apply-override
//     )
//     ```:
//    * If any field is not present in the dictionary, then it's default is 
//      that specified above
//    * `override` if specified and not `none` must be a string. 
//      It is the argument name that will be used to identify the 
//      SIOD override a user wishes to apply when using a compiled 
//      function. Note that even if this is `none`, if `local` or 
//      `override-method` are not, they may still be removed 
//      from arguments.
//    * `local-default` if specified can be either a boolean or array, 
//      and is how the overrides should be interpreted if the user
//      does not explicitly specify or if there is no way for the user
//      to explicitly specify.
//    * `local` if unspecified or `none` means that `local-default` is 
//      always used, its value interpreted as described by 
//      the embed-override in `compile`. If it's a `str`, then it's 
//      a parameter name that, when present, controls 
//      localisation of the overrides.
//    * `override-method-default` is the override method that
//      should be used when the user doesn't or cannot specify.
//      It is interpreted as the same argument on the `override`
//      function is. 
//    * `override-method`, if not `none` and instead a `string`, is
//      the argument name that should be usable by a user
//      to change how the override, if any, happens.
//
// This function then returns a pair of closures 
// (the pair is so that the overrides modification is reusable)
// They look like the following:
// * The first returned closure is one that *takes* an `arguments`, and 
//   returns a pair `(arguments, overrides)`, where the returned `arguments` is
//   stripped of any override parameter (as specified), and the returned `overrides`
//   is an array of overrides (not localised). This will usually be zero or one
//   overrides. It's signature is basically `(arguments) => (arguments, overrides)`, 
//   note the lack of `..`.
// * The second returned closure is one that has the following signature:
//    `(self, overrides, arguments, path-from-root) => (self, arguments)`
//   This closure will remove the relevant `local`/`override-method` argument 
//    values from the arguments. It will - if there are any actual overrides - 
//    apply them to the *uncompiled* `self`, with appropriate 
//    localisation/methods. Even if there are no overrides (the array is
//    empty) it will still remove the arguments.
// If the spec is just `none`, then the first closure will naturally always return `()`.
// The second closure will act as if supplied the default `local-default` and 
// `override-method-default` values, and will not extract any parameters.
#let override-argument-specification-applicators(spec) = {
  // If the spec is `none`, then return some nice default functions
  if spec == none { return (default-overrides-extractor, default-overrides-applicator) };
  if type(spec) == str {
    spec = (
      override: spec,
    );
  }
  let override = spec.at("override", default: none);
  let local = spec.at("local", default: none);
  let local-default = spec.at("local-default", default: true);
  let override-method = spec.at("override-method", default: none);
  let override-method-default = spec.at("override-method-default", default: default-apply-override);
  
  let overrides-extractor = if override == none {
    default-overrides-extractor
  } else {
    (args) => {
      let (args, single-override) = remove-argument-or(args, override);
      if single-override == none {
        (args, ())
      } else {
        (args, (single-override,))
      }
    }
  };

  let remove-local-or-default(args) = if local == none {
    (args, local-default)
  } else {
    let (args, local-spec) = remove-argument-or(args, local);
    if local-spec == none {
      (args, local-default)
    } else {
      (args, local-spec)
    }
  };
  
  let remove-override-method-or-default(args) = if override-method == none {
    (args, override-method-default)
  } else {
    let (args, override-method-spec) = remove-argument-or(args, override-method);
    if override-method-spec == none {
      (args, override-method-default)
    } else {
      (args, override-method-spec)
    }
  };

  let overrides-applicator(sym, overrides, args, path-from-root) = {
    let (args, local-spec) = remove-local-or-default(args);
    let (args, override-method-spec) = remove-override-method-or-default(args);
    if overrides.len() == 0 { 
      (sym, args)
    } else {
      // Apply localisation first
      let absolute-overrides = make-absolute-overrides(overrides, local-spec, path-from-root);
      let overridden-sym = fn-override(sym, override-method: override-method-spec, ..absolute-overrides);
      (overridden-sym, args)
    }
  };
  (overrides-extractor, overrides-applicator)
};

// Turn your raw `SIOD`/`sym`-dictionary into a "compiled" SIOD, embedding the
// `self` parameter inside closures rather than needing to manually pass it.
// Provides numerous tools to define how this should happen and also to 
// can inject functions to make manipulating compiled SIODs intuitive.
//
// `embed-override: "override-sym"`, if not set to `none`, is a string name indicating 
// the name of the entry to insert into the compiled SIOD that can override entries 
// in that (a-la `override`), and re-compile. This inserted function has three named 
// parameters:
// * `recompile: true` - if true (the default), recompiles the overridden uncompiled SIOD
//    and returns that. Useful for convenience but if you are going to do large sequences of
//    of overrides at once it may be more performant to set this to `false` until
//    the last, and then use the normal `compile` function.
// * `local: true` - This can either be a boolean or a string array, interpreted
//    as per the `localisation-spec` parameter of `make-absolute-overrides`
//  * `override-method` - Identical (including the default value) to the `override` function.
//  * This function returns a compiled form of the *root* SIOD, not the nested one
//     i.e. you use it like `let sym = sym.display.override(/* overrides here */);`
//     NOT like `sym.display = sym.display.override(/* overrides here*/);`
// 
// `embed-override-argument: "override-sym"` is a means of making `compile` inject
//  some custom parameter parsing into compiled functions allowing users to
//  automatically override operations in the SIOD. The exact specification for 
//  such parameter(s) can be found in the `override-argument-specification-applicators`
//  function.
//
// `path-from-root` is an internal (recursion) parameter that shouldn't be touched 
// manually, but essentially represents the series of key retrievals needed
// to get to `sym` from `root` (in the case the `root: auto`, that is `()`, because
// `root == sym` in that case).
#let compile(
  sym, 
  root: auto, 
  embed-override: "override-sym",
  embed-override-argument: "override-sym",
  path-from-root: (),
) = {
  let r = (:);
  let root = if root == auto { sym } else { root };
  let sym = module-convert(sym);
  let filter-set = make-filter-set(
    sym.at("_siod_dictionary_ignore", default: ()) 
    + ("_siod_dictionary_ignore",)
  );
  let (overrides-extractor, overrides-applicator) = override-argument-specification-applicators(
    embed-override-argument
  );

  let function-wrapper = if embed-override-argument == none {
    // In the basic case, just use `.with` and avoid all transformations.
    (v) => v.with(root)
  } else {
    // In this case, we need to do the extractor/applicator mechanism
    (v) => (..args) => {
      let (args, overrides) = overrides-extractor(args);
      let (root-with-overrides, args) = overrides-applicator(root, overrides, args, path-from-root);
      v(root-with-overrides, ..args)
    }
  }; 

  for (k, v) in sym {
    if (type(v) == function) and (k not in filter-set) { 
      r.insert(k, function-wrapper(v));
    } else if (type(v) == dictionary) and (k not in filter-set) {
      let new-path = path-from-root + (k,);
      r.insert(k, compile(
        v, 
        root: root, 
        embed-override: embed-override,
        embed-override-argument: embed-override-argument,
        path-from-root: new-path, 
      )); 
    } else {
      r.insert(k, v);
    }
  };

  let override-function(
    recompile: true, 
    local: true, 
    override-method: default-apply-override, 
    ..overrides
  ) = {
    let overrides = overrides.pos();
    if overrides.len() == 0 and not recompile { return root; };
    if overrides.len() == 0 and recompile { 
      return compile(
        root, 
        embed-override: embed-override, 
        embed-override-argument: embed-override-argument
      ); 
    };
    let absolute-overrides = make-absolute-overrides(overrides, local, path-from-root);
    let new-root = override(root, override-method: override-method, ..absolute-overrides);
    if recompile { 
      compile(
        new-root, 
        embed-override: embed-override, 
        embed-override-argument: embed-override-argument
      ) 
    } else {
      new-root
    }
  }; 
  if embed-override != none {
    r.insert(embed-override, override-function);
  }

  return r;
};

// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this
// file, You can obtain one at https://mozilla.org/MPL/2.0/

The documentation in the code describes the capabilities of the new, much more powerful compile function. You could conceivably also modify it to allow customising the defaults on the embedded override-sym function (like you can with the argument thing), but really, this is already an extremely powerful system.

It means, for example, given some compiled SIOD sym, you can write code like:

// Override-and-Recompile the whole SIOD `sym`, locally to the `sym.display.title` sub-SIOD.
// Note that lack of need to 
#let sym = (sym.display.title.override-sym)((prefix: "HRT", add-to-listing: false))
// Invoke a function and change the meaning of some 
// internal operation on-demand to modify it.
#(sym.display.new)(
  [my content], 
  title: "Info", 
  override-sym: (
    add-to-listing: true, 
    write-box: (f) => (self, ..args) => { box(f(self, ..args), color: red, inset: 20%) }
  )
)

There is a further question, though, about how APIs in any larger module should take these SIODs as a parameter. Should they take them pre-compiled or un-compiled SIODs?

Pre-compiled SIODs have more intuitive functions for overriding and such, and a more ergonomic usage pattern. However, this would mean users needing to manually compile things first when calling functions initially, and it makes customisation within package functions themselves less ergonomic (though the functions for localisation may make this somewhat mitigable), especially since there is less control over things like the override-sym functionalities.

The solution is to embed a means of automatically detecting the compilation status of a passed in SIOD structure, and then go from there. Indeed, because compilation embeds the raw SIOD in all wrapped functions, a package or user of the package can e.g. customise their override functionality at-will and it won’t affect the implementations inside the raw SIOD at all because they resolve the methods entirely independently of the compiled form of the SIOD (even if they use the compile function per implementation, that’s unaffected by compiling the dictionary for external purposes).

Packages can provide compiled SIODs and as long as there is a way to retrieve the raw SIOD from it, users can also customise their “compile” parameters to whatever they want and the only thing needed to make it work is to embed access to the raw SIOD in a standard way for the package itself to set up compilation how it wants.

recompile & decompile

To enable the functionality above, we’ll want to define functions recompile and decompile (the code won’t be listed in this section, but instead in the “finished” module I embed later cus I want to :p).

decompile will - if it’s argument is a compiled SIOD - emit the corresponding raw SIOD (if the SIOD is raw it is just re-emitted).
recompile will - regardless of if it’s argument is “compiled” or not - re-compile it’s argument with some given compile parameters.

More broadly, we can embed compile info in a special key.

(edit: I couldn’t post the last two segments, so here are links to the markdown, and also to a raw upload of the typst module prototype: post 4, post 5, and the siod.typ prototype)

[fredguth]

This has been a fascinating discussion, and it really feels like the right moment to address these problems. The examples and back-and-forth here have been very helpful in clarifying the core issues.

What I’d like to add is a practical angle: what kind of community practices could we encourage so that templates become more reusable, and the experience of customizing one can transfer more easily to another?

Elembic might be one promising path—a sort of “framework” on top of Typst. Is this the way?

Perhaps we can also start simpler, by adopting a shared schema for common document types (papers, reports, theses), possibly leaning on something like CSL. Even without new language features, this could help us build conventions that make life easier today, while also providing a more stable contract so that Typst’s evolution doesn’t risk breaking every template that’s already been created.

If we agree on such a “contract,” we could even launch a small community effort to adapt existing templates to this approach—starting with one document type (papers would be the obvious candidate). What do you all think?

[PgBiel]

I’d like to add to @fredguth’s reply by expanding on elembic a bit (see full discussion: Elembic 1.0.0: Custom elements and types!).

As has been extensively discussed above by @Mc-Zen and others, I believe Typst’s own styling system will probably be the future of configuration of templates and packages, as it’s handy to set values applying to nested elements, without needing multiple levels of configuration. Something like:

#set template.appendix(supplement: [My Appendix])
#set template.figure-outline(use-boxes: false)
#show template.front-page: set template.text(sans-font: "Source Sans Pro")
#show template.theorem: emph

// instead of
#show: template.with(
  appendix: (supplement: [My Appendix]),
  figure-outline: (use-boxes: false),
  front-page: (text: (sans-font: "Source Sans Pro")),
  show_: (theorem: emph),
  // ...
)

For this reason, I created elembic which rewrites Typst’s styling system from scratch - show rules, set rules on functions (elements), typechecking - so we can experiment with creating our own settable properties and the like until this is implemented in the compiler itself (without using state). Following the same example, it allows for something that is fairly close:

#show: e.set_(template.appendix, supplement: [My Appendix])
#show: e.set_(template.figure-outline, use-boxes: false)
#show: e.filtered(template.front-page, e.set_(template.text, sans-font: "Source Sans Pro"))
#show: e.show_(template.theorem, emph)

For a full example on how to add settable properties to a template, you can check the handbook: Simple Thesis - The Elembic Handbook

The idea is, of course, still maturing. Someone pointed out that using set rules makes it harder to turn certain fields mandatory, requiring some duplication. There is some discussion on how to potentially improve this here, and contributions are welcome: Rewrite thesis template by siefkenj · Pull Request #69 · PgBiel/elembic · GitHub

All in all, I think elembic might be a great option to experiment with for templates today and gather feedback. But it must be noted that the end goal is to have its core features (notably custom setable and showable functions, as well as some form of typechecking) available in Typst itself, without needing a package.