Help in reducing memory usage

[EduardoHitek]

Hello, I’d like some help reducing memory usage during PDF generation with a relatively large dataset. The idea is to use Typst to generate a report with up to 50k records, but with around 10k records, we’re already exceeding the 4GB RAM limit of our pod.
I’m passing the report input via a JSON file.
Is there any optimization I could apply to this template to reduce its memory usage?
Below is the base code of the template that generates the record tables.

#import "../../helpers/format-date.typ": get-formatted-date, format-datetime-to-short-date
#import "../../helpers/format-document.typ": format-account-code, format-document, translate-operation-type, format-amount-from-string
#import "../../helpers/theme.typ": border, font, get-brand-theme, semantic-colors, space
#import "../../components/display-content.typ": display-content
#import "../../components/section.typ": section
#import "statement-components/header.typ": header-content
#import "statement-components/footer.typ": footer-content
#import "statement-components/table-header-title.typ": table-header-title

#let bank-statement(
  created_at: none,
  owner-name: none,
  owner-document: none,
  start-date: none,
  end-date: none,
  account-code: none,
  is-ton: false,
  request-id: none,
  font-family: ("Inter", "Liberation Sans"),
  transactions: (),
  body,
) = {
  let theme = get-brand-theme(is-ton)

  set page(
    paper: "a4",
    margin: (
      top: if is-ton { 125pt } else { 100pt },
      bottom: 30pt,
      right: 30pt,
      left: 30pt,
    ),
    header: [#header-content(created_at, theme)],
  )

  set text(
    font: font-family,
  )

  set par(justify: false)

  section("Dados da Conta", font.size.s100)[
    #block(
      stroke: 0.5pt + semantic-colors.border.medium,
      radius: border.radius.medium,
      inset: space.medium,
    )[
      #grid(
        columns: (2.8fr, 1fr),
        align: (left, left),
        gutter: (space.small, 0pt),
        [#display-content("Nome", owner-name) ], [#display-content("Documento", format-document(owner-document))],
        [#display-content("Instituição", "Instituição")],

        grid(
          columns: (auto, auto),
          gutter: (space.xLarge1, 0pt),
          align: (left, left),
          display-content("Agência", "0001"), display-content("Conta", format-account-code(account-code)),
        ),
      )
    ]
  ]

  table(
    columns: (8%, 8%, 32%, 14%, 14%, 24%),
    align: (horizon + center, horizon + left, horizon + left, horizon + left, horizon + left, horizon + left),
    stroke: (x, y) => if y >= 2 and transactions.len() != 0 {
      (bottom: 0.5pt + semantic-colors.border.medium)
    },

    table.header(
      table.cell(colspan: 6, align: left, inset: (bottom: space.medium), stroke: none)[
        #text(
          size: font.size.s75,
          weight: font.weight.semibold,
          fill: semantic-colors.content.high,
        )[Período: #get-formatted-date(start-date) a #get-formatted-date(end-date)]
      ],
      [#table-header-title("DATA")],
      [#table-header-title("TIPO")],
      [#table-header-title("DESCRIÇÃO")],
      [#table-header-title("VALOR")],
      [#table-header-title("SALDO")],
      [#table-header-title("CONTRAPARTE")],
    ),

    ..(
      if transactions.len() == 0 {
        table.cell(colspan: 6, align: center, inset: (y: space.xLarge3), stroke: none)[
          #text(
            size: font.size.s75,
            fill: semantic-colors.content.high,
            weight: font.weight.semibold,
          )[Você não tem transações no período]
        ]
      } else {
        transactions
          .map(t => (
            text(
              size: font.size.s25,
              weight: font.weight.semibold,
              fill: semantic-colors.content.high,
            )[#format-datetime-to-short-date(t.created-at)],
            text(
              size: font.size.s25,
              weight: font.weight.semibold,
              fill: semantic-colors.content.high,
            )[#translate-operation-type(t.operation)],
            block(breakable: false, height: 20pt)[
              #text(size: font.size.s25, weight: font.weight.semibold, fill: semantic-colors.content.high)[#t.title]
              #if t.subtitle != "" {
                v(space.xSmall2, weak: true)
                text(size: font.size.s25, fill: semantic-colors.content.high)[#t.subtitle]
              }
            ],
            [#format-amount-from-string(
              t.amount,
              colored: true,
              with-sign: true,
              font-size: font.size.s25,
              font-weight: font.weight.semibold,
            )],
            [#format-amount-from-string(
              t.balance-after,
              colored: false,
              with-sign: true,
              font-size: font.size.s25,
              font-weight: font.weight.semibold,
            )],
            block(breakable: false)[
              #if t.counterparty.branch-code != "Desconhecido" or t.counterparty.account-code != "Desconhecido" {
                text(
                  size: font.size.s25,
                  weight: font.weight.semibold,
                  fill: semantic-colors.content.high,
                )[
                  #t.counterparty.institution-name
                ]
                v(space.xSmall2, weak: true)
                grid(
                  columns: (auto, auto, auto),
                  gutter: (space.xSmall2, 0pt),
                  align: (left + horizon),
                  [#text(
                      size: font.size.s25,
                      fill: semantic-colors.content.medium,
                    )[Ag:] #text(
                      size: font.size.s25,
                      fill: semantic-colors.content.high,
                    )[#t.counterparty.branch-code]],
                  text(size: font.size.s25, fill: semantic-colors.content.medium)[•],
                  [#text(
                      size: font.size.s25,
                      fill: semantic-colors.content.medium,
                    )[Cc:] #text(
                      size: font.size.s25,
                      fill: semantic-colors.content.high,
                    )[#t.counterparty.account-code]],
                )
              } else if t.recipient-name != "Desconhecido" {
                text(
                  size: font.size.s25,
                  fill: semantic-colors.content.high,
                  weight: font.weight.semibold,
                )[#t.recipient-name]
              } else {
                ""
              }

            ],
          ))
          .flatten()
      }
    ),
  )

  block(
    width: 100%,
    stroke: (top: border.width.small + theme.color.border.brand),
    inset: (top: space.medium, bottom: space.medium),
    breakable: false,
  )[
    #section("Informação do comprovante", font.size.s75)[
      #display-content("Código de autenticação", request-id)
    ]
  ]

  footer-content(theme)
}

[SillyFreak]

@EduardoHitek

#import "../../helpers/format-date.typ": get-formatted-date, format-datetime-to-short-date
#import "../../helpers/format-document.typ": format-account-code, format-document, translate-operation-type, format-amount-from-string
#import "../../helpers/theme.typ": border, font, get-brand-theme, semantic-colors, space
#import "../../components/display-content.typ": display-content
#import "../../components/section.typ": section
#import "statement-components/header.typ": header-content
#import "statement-components/footer.typ": footer-content
#import "statement-components/table-header-title.typ": table-header-title

Hi @EduardoHitek and welcome! Since your code doesn’t contain these parts (and also no example JSON file) we can’t try whether any changes affect memory usage.

Please don’t add these files but instead minimize your code to the extent possible while still showing the high memory usage, as recommended by the the question guidelines:

It would be great if you could post the smallest possible code samples that allow people to see what your problem is.

For reference, here is an older post about debugging performance problems:

How to optimize a `query` that leads to severe performance problems?

Hey I’m working with some fairly large documents (up to 1000 pages) where the headers must guide through the readers the sections, chapters and paragraphs. (Example: Section 3 · Chap. F · Par. 25 — Section 3 · Chapter G · Par. 2) Thanks to the support of three very helpful members here, I was able to implement a solution with query like so: How to get the number of the first and last paragraph on a page and show the result in the header? The main part to get the paragraph numbe…

The crucial point was that it contained code that was runnable out-of-the box and that we could experiment with.

---

I assume the size of the file comes either from a big transactions array, or from processing multiple bank-statements (you should include the code calling your template as well).

Instead of reading a JSON-file, you can include the test data directly in the code:

#let single-data = (created_at: ..., ...)
#let data = 100*(single-data,)

This creates a 100 element array of duplicated test bank statements. Please confirm that your test data shows the high memory usage, so that when we copy your code we can reproduce your problem directly.

[robmv]

This looks like a single large #table that current typst struggle with. I have been collecting tests and tricks to optimize this use case at Are there any best practices for documents with large tables?