I am using typst mainly on the Mac but have also installed it on an Ubuntu 24.04 LTS Desktop.
On the Mac --open uses open proper on Linux it shows me an Open With... selection (with the Document Viewer and Chromium as options.
I have set the default viewer to the Document Viewer and double clicking a PDF or using the commands open or xdg-open on the command line or in shell scripts (in a window) both open the PDF as required.
Do you mean that when you do typst compile file.typ --open in the terminal, you get a popup “Open with” dialog?
Does it work if you do typst compile file.typ --open xdg-open?
What does type xdg-open say?
To further investigate, you can run strace -f -o strace.log typst compile file.typ --open, and search for “xdg-open” in strace.log and see what’s there and shortly afterwards.