Dealing with large files that have a lot of images?

Questions
1

I am working on a document that has an auto-generated appendix that contains a large number of large images. There are about 420 images and I have compressed these images down to about 1 MB each. Trying to include all of the images gives me a “memory allocation failed” message. I have tried reducing the number of images to experiment and I have some success, depending on what images I cut out, I can compile a document with anywhere between 50 and 200 images included (The maximum number of images appears unrelated to file size).

Is there any method or trick I can use to deal with this issue? Would it be best to just incrementally compile the document and stitch it together after the fact?

Thanks!

2 Likes
2

Except dividing the whole file into small parts in source code level, there are many ways:

  • typst compile xxx.typ --pages 1-90
  • typst compile xxx.typ -j 1
  • buy bigger memory cards
3 
typst compile xxx.typ --jobs 1

This is a valid command but it’s also not documented with typst --help. What is it supposed to accomplish?

1 Like
4

From typst-compile(1) — Arch manual pages

-j, --jobs=JOBS
Number of parallel jobs spawned during compilation. Defaults to number of CPUs. Setting it to 1 disables parallelism

So I don’t think it does really help with this problem.

1 Like
5

You could use -h on sub-commands too.

typst -h
typst c -h
typst c -j -h

I do not test a benchmark or profile to analyze memory usage, but multiple process instance run simultaneously will use more memory at first glance.

@scdyer Apart from real memory card hardware, u can also set virtual memory in a software way, that will use disk as memory.

6

Aha I see this parameter is documented under typst compile help.

7

You mean increase the amount of virtual memory available by increasing the size of the available swap(file).

1 Like
8

I was not aware I could break up compilation by pages. I will try that. The solution I ended up using was doing essentially that in a more roundabout way and merging the PDFs after.

1 Like
9

typst compile xxx.typ --pages 1-90

Okay so I actually just tried this and it does not resolve the problem. It still throws an out-of-memory error, even when I am excluding the Appendix pages that contain all of the images. I guess it still needs to load the images into memory and just does not print the pages to a file.

For those who are experiencing a similar problem, here is my hacky solution:
Seperate the auto-generated appendix into a different typst file and write the appendix-generating function in a way that it can take a numeric input to break up the output. For example my function looks like:

#let appendixtables(contents, start: 1, stop: 100000) = {...}

When I call this function I call it like so:

#appendixtables(yaml("MyInputFile.yml"),start: int(sys.inputs.start), stop: int(sys.inputs.stop))

And I build the appendix with a shell script that compiles the appendix file 30 times, incrementing the inputs by 10 each time to produce 30 unique 10-page documents that I have to then merge together. The shell command looks like this:
typst compile --input start=$start --input stop=$stop .\Appendix.typ Appendix\A_1-$j.pdf
This was in powershell but pretty easy to translate to BASH.

This is extremely hacky and if there is a better way to do this, I would love to hear it. This method also sacrifices including the correct page numbers, I just removed them since its an appendix and I don’t care. I am sure I could force the page numbers to be accurate but I don’t have the patience to make it work with the other packages that I am using. The other sacrifice is linking to any pages in the appendix is not possible and headers cannot be included in the table of contents

10

From what I gather you’re running on Windows. With a few minutes of work and zero cash outlay you could install a type 2 hypervisor such as VirtualBox, install an Ubuntu Linux guest VM into it, and in that VM install Rust, install Typst, and attempt to get your compile working there. You can experiment without risking Windows stability by, for example, increasing the Linux swap space to support more virtual memory for Typst.

Maybe your large project would compile on a machine with more RAM, but in today’s crazy market with soaring RAM and SSD prices that’s a purchase one would normally dismiss.