Syntax overriding

General
, ,
1

(This is not a question so much as a discussion of what the idiomatic solution would be in future Typst versions, I assume at the very least that we would have custom elements and replace rules)

Suppose you want to add some extra content to each level 1 heading, content that isn’t semantically part of the heading itself, like a table of contents for that chapter.

We can already do this with a show rule but the extra content will be considered part of the heading which is bad.

In the future though, you could use a replace rule or (ignoring syntax convenience) use a custom element where the show rule outputs the real heading as well as the extra content.

By Replace Rule

If we only got replace rules and custom elements, then the first option would be clearly preferred: The source element is still a regular heading, so you can still use the usual syntax sugar for headings.

However this might lead to problems with the replace rule matching it’s own output.

There are a lot of ways this might not be a problem, but they have issues:
  • If replace rules don’t match on the original element if it is included in the output
  • Smuggle some data in one of the fields to allow the selectors to distinguish between elements created by syntax sugar and the ones that are the result of one of your show/replace rules.

But the first of these is unreliable in more complex scenarios and the second is just messy and may be impossible for some elements.

By Custom Element

If the issue of syntax is ignored though, then using a custom element is (in my opinion at least) the cleaner solution because there’s no issue of the show rule matching on it’s output.

Mini-conclusion

To solve the issues of the replace rules solution would require having a better way to distinguish between elements constructed with syntax sugar and other elements

To solve the issues of the custom elements solution would require a way to change what element the syntax sugar outputs.

2

Support for general user-defined syntax sugar would also solve the problem but I can’t see that actually happening.

A viable “way to distinguish between elements constructed with syntax sugar and other elements” might just be to have each element with syntax sugar have a completely different element (called the syntax sugar version of the element) that the syntax sugar actually outputs. This is then replaced with the actual e.g. heading by a builtin replace rule.

Edit: This also makes a lot of sense since the syntax sugar versions consistently don’t let you set all the fields that the raw element supports, which could be neatly reflected by the syntax sugar version of the element lacking those fields.
As for naming, I suggest adding a new module to std, std.sugar for accessing the syntax sugar versions of elements. This is better than e.g. heading.sugar because it avoids cluttering up the documentation for elements that have a syntax sugar version

We could also add an extra hidden field to elements with syntax sugar which can be retrieved using a builtin function but this just feels too magic and too messy.

3

In the future though, you could use a replace rule or (ignoring syntax convenience) use a custom element where the show rule outputs the real heading as well as the extra content.

I think this generally works for every case of adding additional content next to an element x without having it be part of x that I can think of today simply by writing a function for it, no custom should be needed (if the syntax inconvenience is ignored of course).

Explicit Syntax Sugar

I’ll explore the impact mostly within the proposed example, I think this can be generalized fairly well to other problems, but I’m interested in examples where my argument may not hold.

Like you mentioned, ideally we write = Foo and it generates a heading and the associated table of contents, both of which are distinct elements in the document, producing well tagged PDFs and introspection information.

If Typst provides sugar.heading which by default replaces to heading, then we could implement this by overriding that replace rule to also include the mini-toc, and it would generally work as long as you use = Foo to write your headings.

But what if you don’t? Perhaps you need something that the parser cannot handle well, or you have to set one of the fields you can’t set with the syntax sugar you’re now back to manually adding the extra content.

Perhaps this can be resolved with good API design, making the code that applies the additional content easy to use with both sugar.heading and heading. But at least on the surface this seems to put more cognitive load on the User to properly design the API this way. I think this is especially important, because Typst’s user base does not consist only of Programmers.

Re: Mini-Conclusion

I agree with the first paragraph, we should put work into distinguishing which kind of content can be matched again by a rule and which kind of content can’t because it is helpful for non-replacing rules too, some elements are more likely to run into this, like lists and enumerations.

Perhaps revoke rules play nicely into this, I could imagine that one may be able to revoke a rule within the rule itself like so (making up syntax here, we can bike shed later):

#show heading as heading-rule: it => {
    revoke heading-rule
    it
    mini-toc()
}

But maybe this has the same fundamental limitation that #show x: it => { show x: ...; it } has, where the inner rule has no effect due to realization being finished when the rule runs.

4

But what if you don’t? Perhaps you need something that the parser cannot handle well, or you have to set one of the fields you can’t set with the syntax sugar you’re now back to manually adding the extra content.

Perhaps this can be resolved with good API design, making the code that applies the additional content easy to use with both sugar.heading and heading.

Can’t you just put the transform logic in its own function?

But at least on the surface this seems to put more cognitive load on the User to properly design the API this way. I think this is especially important, because Typst’s user base does not consist only of Programmers.

Yeah maybe, but if they get this wrong to begin with it’s still trivial to refactor into being a separate function when they realise that would be helpful — no forsight/programmer instincts are required.

You may worry that once the transformation logic is in it’s own function that making it work for both sugar.heading and heading will cause extra headache, but in the simpler cases it would just work:

  • The fields of sugar.heading would be a subset of the fields of heading
  • There could (would) be a function sugar.to-normal which converts sugar elements to their non-sugar counterparts and leaves other elements unchanged.

The first point means that if the extra field (the one that the syntax sugar can’t handle) doesn’t need to be read by the transformation logic, then the function intended for sugar.heading would just work on a heading instead with no modification.

The second point means that if the transformation logic does need to read the extra field, you can just call sugar.to-normal on the input (unconditionally, since it leaves non-sugar elements alone) to make sure you’re getting a heading. This is obviously possible to forget, but there could probably be a compiler hint for when people make this mistake.

5

Re: Re: Mini-Conclusion

Using revoke like that is sometimes a footgun and so would probably be considered an antipattern.

In this case it’s fine, but it’s also going to prevent the rule from matching on child elements which the rule hasn’t been applied to yet, so it doesn’t serve the purpose of making sure the rule runs exactly once on each matching element.

Like I mentioned before, Typst already won’t match the same show rule multiple times on the same element, and this behaviour would probably carry over to replace rules, so these simple examples are ok.

With the talk of changing the semantics of show set rules to apply to all elements created in the matching show rule, we might get similar changes applied to show rules: A show rule never matches on an element created in a previous run of that show rule.
That would completely remove recursive show rule matches without blocking matches in child elements.
Whether true recursive show rules are useful enough to justify adding a rec keyword or similar to revert back to the old behaviour would need investigation — I’m tempted to think those cases can be covered more cleanly by just using a recursive function in the show rule.

If this change (fully removing recursive show rule matches) happened, then the naive replace heading approach would probably be fine, although note that the evaluation order is different than replace sugar.heading if further show/replace rules are applied to heading.