Typsastra may be something to consider before spending time with the Typst source code. The preview functionality is already customized to use less memory, as described here:
This does not affect memory use during the actual compilation though, so if you have an automated process that you aren’t looking at previews then it will not help.