Since the documentation states that
Both
note()andnotefigure()accept atext-styleandpar-styleparameter…
using the function note() itself for the configuration seems to be your only option. Instead redefining it you can however just change the default value of text-style with with(). You will still have access to the other parameters and you can even override the text-style.
#let note = note.with(text-style: (size: 8pt))
Edit: This was also the answer by the author of the package to the same question in a GitHub issue, see Globally setting text characteristics for margin notes · Issue #5 · nleanba/typst-marginalia · GitHub.
A note regarding the compilation error. Set rules for custom elements/functions are not (yet) available. There is an experimental package called elembic that introduces custom elements and types. This does however not help you here since marginalia does not use this.