How should one quote video passages?

Questions
1

How could longer-form quotes be made more comfortable and clearer to understand? Screenplay packages like scriptie, its-scripted, screen-scripted and celluloid all appear to assume that

  1. each document refers to exactly one medium
  2. the play happens in the future, having no obvious mechanism to denote timestamps if it has already happened
  3. everything is continuous

These are suboptimal for art analyses and blog posts, where quotes are mixed-and-matched, then surrounded with a lot of contextualization/textual glue, reductively speaking.

Situation

I frequently interpret and analyze video and audio. Characteristics include:

  1. One quote refers to one medium (albeit the medium itself may reference other media).
  2. One quote has a start and an end that correspond to medium timestamps. Every quote should have at least one timestamp to facilitate verification.
  3. Multiple speakers may be involved, which can be textually identified. There is (at least) one exception: The only speaker is an unidentified narrator.
  4. Quotes may be long, sometimes even spanning several minutes.

What I do right now

Using Metropolis as an example:

#let qb(attribution, body, ..args) = quote(attribution: attribution, block: true, body, ..args)

#qb[@metropolis-1927[23:18 to 24:58]][
  / Joh: What were you doing in the machine halls, Freder?
  
  / Freder: I wanted to look into the faces
    of the people whose little children are my brothers, my sisters...
    
    Your magnificent city, father --
    and you the brain of this city --
    and all of us in the city's light -- --
    
    ...and where are the people, father,
    whose hands built your city -- -- --?

  / Joh: Where they belong...

  / Freder: ...where they belong...?

    ...In the depths...?

    What if one day those in the depths rise up against you?
]

#bibliography(bytes(`
metropolis-1927:
  type: video
  title: "Metropolis"
  date: "1927-01-10"
  runtime: "02:28:57"
`.text))

Playground
Playground839×596 34.8 KB

My gripes here:

  1. It is not possible to know when exactly each line is. (The entire section above is 1:40 long. I’d rather write the timestamp of each line once rather than having each verifying reader watch that duration.)
  2. The semantics are… off. In HTML export, this ends up as dt. We define the same terms multiple times here, this does not feel correct.
2

One idea I had was using WebVTT for input, which would also allow copying subtitles directly out of files if available locally (e.g. ffmpeg -loglevel quiet -i ... -map 0:s:0 -f webvtt -y /dev/stdout), but that is quite verbose for typing manually and leaves the formatting question unresolved.

3

I would try to cite the script, if it is available, then the timing is less important, but that does not work, if it is important.

I would probably make this into a table with:

Timecode | Speaker1 | Line

Timecode | Speaker1 | Line

Timecode | Speaker2 | Line

… that way you get the timing of each line.

If the speaker is not identified, then do the CC thing of [unknown], [disembodied voice], or something along those lines.

The best thing to do would be to make a simple python/js script to use ffmpeg for extracting the CC, and formatting/linting it into a csv, which you can then import as a table. Though this would probably suck still because there is no speaker for each line unless the speaker is not visible.

I suggest python or js because those are usually the ones with the most packages for this and are the best for LLM suggestions or full force vibe coding. You could honestly probably make a ML thing that detects the face of the speaker and assigns the subtitles to the speaker name, just verify the passage, but it would make bulk transcriptions easier.

I hope I was able to give you some ideas at least, but I am talking out of my .ass file.