Cinema and video

A cinema screen is a surface in the world playing a video. The server says where the screens are and what is on them; every client fetches and decodes the video itself.

What the server sends複製段落連結

Geometry and a timecode, on the existing voice status channel:

SSVC_SCREENS_V1
S <id> <world> <x> <y> <z> <facing> <width> <height> <playing> <position> <source>

A screen is flat and axis-aligned — one of the six block faces. Arbitrary angles would need a quaternion on the wire to serve builds nobody has asked for.

There is nowhere in that format to put a pixel, which is the strongest way to say the video is not coming through here.

The playhead is a position, not a clock複製段落連結

The snapshot carries where the playhead was, and the client adds the time since the message arrived. A fresh snapshot every five seconds corrects the drift.

Sending a live playhead instead would be a message per screen per frame, to say something every client could have worked out. A live source has no seekable timeline at all and is marked as such: asking a client to seek into one produces an error rather than a picture.

The client pipeline複製段落連結

fetcher (FFmpeg, -c copy) ──compressed──► reserve ──► decoder (FFmpeg) ──raw RGBA, loopback socket──► frame buffer ──► render thread ──► DynamicTexture
                                                             └──PCM, loopback socket──► sound buffer ──► positional speakers (the clock)

FFmpeg does the decoding, through the binary SocketSVC already downloads on demand for the soundboard and lends out as ffmpegBinary() — so the addon neither ships a copy nor fetches a second one. It is spawned with ProcessBuilder and never through a shell, so a URL from a server's config can never be read as a command.

Scaling and the frame cap happen in FFmpeg, not afterwards. A screen is a few blocks across; decoding 4K and throwing most of it away would cost the decode, the copy and the upload to show something nobody can see at that distance. Asking for the size actually drawn is the single biggest thing that makes this affordable. The default is 256×144 at 24fps.

The stream is fetched into a reserve first, and decoded from there. A video player's buffer holds the stream compressed — seconds of it are a few megabytes — and that is what this is: a second FFmpeg that only copies the stream as it arrives (-c copy, next to no CPU) into memory, up to 64 MB, as fast as the network gives it. The decoder reads from that reserve, so a network that hesitates for a few seconds costs nothing until the reserve runs dry, and playback starts as soon as two seconds are in hand. Decoded frames are enormous by comparison — 8 MB each at 1080p — so a buffer of those could only ever hold a fraction of a second at a sharp setting, and every hesitation longer than that was a stall. That was most of "a high resolution buffers all the time": the sharper the screen, the shorter the buffer, the more often it stalled. The reserve holds the same seconds at any size.

The frames arrive over a loopback socket, not over a pipe. They came over stdout once, and on Windows that was the rest of it: the anonymous pipe Java gives a child process holds four kilobytes, and measured on an ordinary machine it moved about 27 MB a second — three and a half frames of 1080p, or two thirds of 480p at 24 fps. The picture could not physically reach the game as fast as it was decoded, whatever the network did. The same frames over a loopback TCP socket moved at nearly a gigabyte a second. The sound has always travelled that way; the picture now does too — loopback only, one connection, closed with the stream.

The sound is the clock, and the picture follows it. Sound cannot be nudged — hold a frame back and there is a gap, drop one and there is a click — while a picture can be nudged all day without anybody noticing. So the sound plays at exactly the rate the sound card takes it, its position is measured back through the mixer's queue and the output device's buffer, and every video frame carries the moment it belongs to and is shown when the sound reaches that moment: the frame nearest the clock, looked up half a frame ahead so the picture sits centred on the sound rather than always a little behind it; a late frame is dropped, an early one held, and the last one stays up until the next arrives. It was the other way round once — the sound waited on the picture — which left the sound a steady tenth of a second ahead of the picture and turned every stumble in the decode into a hole or a click in the audio.

A screen nobody is near does not decode. Past 96 blocks its processes are stopped, and they start again inside 80 — two distances rather than one, because a single threshold means a screen that stops and starts every time somebody walks along its boundary. A screen across the map otherwise costs exactly as much as one you are standing in front of, and on a server with several screens that is most of the total cost.

The decoder is not allowed to take the whole machine. FFmpeg is held to two threads for anything up to 480p — which is what every screen gets by default — three at 720p and four at 1080p, never more than half the cores, and offered hardware decode (-hwaccel auto, which falls back to software on its own). Left alone it spreads across every core it can find — right for a transcoding job, wrong inside a game, because the cores it takes are the ones the render and client threads need. The fetcher costs next to nothing: it decodes nothing.

The sound is buffered too, and the film waits for both. It had no buffer of its own: frames went from the decoder straight to the mixer, so the only thing between a hiccup in the network and a hole in the audio was whatever FFmpeg happened to be holding — and a gap in sound is far more noticeable than a repeated video frame. It now fills the same way, with a decoder thread running ahead and a second thread draining it at the rate the film is heard. Starting on a full video buffer and an empty audio one is a film that begins in silence, so playback waits for both, and the progress bar shows the lesser of the two.

Sound is never skipped or held to catch up, the way a picture is: dropping 20 ms of audio is a click and holding it is a gap. When its own buffer runs dry the clock stops, the picture stops with it, and the film waits and refills — a pause rather than a skip. The whole buffer is under 80 KB — 20 ms of 48 kHz mono is under two kilobytes — so where the picture has to be capped by memory, this one simply holds the time it wants.

Decoded frames are buffered by memory, not by count. About two and a half seconds of picture at the default size, and never more than 48 MB — at 1080p that is six frames, and a screen that size was asked for on purpose. The decoded buffers no longer have to ride out the network, because the reserve does; they ride out the decode, which is local and fast. Filling them is what paces the decoder: it blocks on the socket while they are full, and reads on when a frame is shown.

The upload happens on the frame, not on the tick. This is the difference between a film and a slideshow: the client tick runs at 20 Hz, so uploading there caps every screen at 20 frames a second however fast it decoded — and a 24 fps source landing on a 20 Hz upload judders on every single frame, because the two rates do not divide. Reconciling what should be running stays on the tick, where starting and stopping processes belongs; frames reach the screen from the render slot. A texture may only be touched on that thread, and decoding must not happen there. It is built on DynamicTexture rather than raw GL calls — not the fastest possible path, and the one that works across six Minecraft versions and three loaders, which hand-written glTexSubImage2D does not.

The frame goes in as one copy. FFmpeg's rgba output is byte-for-byte the layout NativeImage.Format.RGBA stores, so the frame is memcpy'd straight into the image's own memory. It previously unpacked each frame into an int[] of ARGB on the decoder thread and wrote it back out one setPixel at a time on the render thread — about 166,000 per-pixel operations per frame, two thirds of them on the thread that draws the game, spent converting a thing into itself.

Sound複製段落連結

Screen audio is a separate stream, played through the local positional channels the addon API already offers, so it can be placed in the world with a direction. The video process decodes no audio at all — it would have nowhere to go.

Speakers take a cone, so standing in front of a stack is louder than standing behind it. Height is the honest limit: the HRTF is horizontal, so a speaker above the audience is placed correctly left-to-right and flat vertically.

Two jars, and what stays in the mod複製段落連結

The cinema is an addon, not part of SocketSVC. A player who never visits a server with screens should not be carrying a video decoder, and a server that does have screens should not need everybody to update the mod itself.

Nothing about the split is cosmetic: the client addon compiles against the client addon API and nothing else, and the server addon against com.socketsvc.api.addon and nothing else. Both are built inside those boundaries precisely so the API cannot quietly grow a hole for its own first addon.

Drawing it in the world複製段落連結

The quad is drawn through AddonContext.worldRenderer, which SocketSVC provides because an addon cannot do it itself: drawing in the world means a mixin into Minecraft's renderer, and mixins apply when classes are transformed — long before any addon jar has been opened.

SocketSVC's own hook is a mixin into LevelRenderer.submitEntities rather than a loader event. Fabric has an event that hands over a PoseStack and a SubmitNodeCollector; NeoForge and Forge have none that hands over a collector — their RenderLevelStageEvent gives the pose stack and nothing to submit to. All three loaders' events are built on that vanilla method, so injecting there is what lets one addon jar draw on all three.

It is drawn after the entities are submitted, so a player standing in front of a screen occludes it, and full-bright rather than lit by the world: a screen emits its own light, and a picture that went dark at night would be a screen that switched off.

Setting screens up複製段落連結

Put the server jar in SocketSVC's addons/ folder and its config beside it as addons/cinema.yml:

screens:
  lobby:
    world: world
    x: 10.5
    y: 66.0
    z: -20.5
    facing: north      # north, south, west, east, up, down
    width: 6
    height: 3.375
    source: "https://example.com/feature.mp4"
    live: false        # a stream with no seekable timeline
    playing: true
    duration-seconds: 0
    loop: false

The source has to be reachable by your players, not by the server: nothing on the server ever opens it. A file path, a LAN-only address or anything behind the server's own firewall shows nothing.

What a source can be複製段落連結

A page is resolved by yt-dlp on each client before anything is decoded. That matters because FFmpeg cannot open a watch URL at all — that address serves HTML, not video — and pointed at one it exits immediately with nothing useful to say, which looks exactly like a screen that is simply broken.

yt-dlp also reports the HTTP headers each stream has to be fetched with, and FFmpeg sends them. Most sites do not care, but Bilibili's CDN refuses (403) any request without the Referer and User-Agent the page was resolved with. Bilibili serves logged-out viewers at most 480p, and some videos are region locked outside mainland China — those fail with yt-dlp's own message.

Resolution happens client-side for the same reason the decoding does: the server never fetches what a screen points at, so it could not resolve one even if it wanted to, and a resolved URL is tied to the address that asked for it. A direct file or an HLS stream needs nothing but FFmpeg, so prefer one where you have the choice.

The tools fetch themselves複製段落連結

A player needs nothing installed. The first time a screen needs FFmpeg, or yt-dlp for a page, the addon asks SocketSVC for it and SocketSVC downloads it in the background into its own cache — the same binaries the soundboard uses, verified against their published checksums. Chat says once what is being downloaded and why, the screen shows Downloading FFmpeg 42% where the film will be, and playback starts the moment the download lands. Nothing to press, nothing to find.

What is still reported rather than fixed: a download that fails (the screen says so, and stopping and playing it again retries), and a machine no build of the tool exists for. /ssvc addon cinema shows both tools' state.

When a screen is blank複製段落連結

Every way this can fail looks identical from where the player is standing. /ssvc addon cinema in the client says which one it is:

Volume: 100%, quality: full, fps: 24 (auto)
Drawing: 177 frames/s, costing 0.30ms of each one
Decoding: 24 frames/s, 0/s never drawn
FFmpeg: ready
yt-dlp: ready
1 screen(s) declared by the server:
  lobby (world 1,109,18): FAILED: the source produced no video. The playhead was at 10s -
      if the file is shorter than that, there is nothing there to decode.
      https://samplelib.com/mp4/sample-5s.mp4

The rates are over the last few seconds, not since the game started — a cumulative count cannot tell a screen that is dropping frames now from one that dropped them all while it was starting, and those two want completely different fixes. Drawing is what the cinema costs of each frame; if that number is small and the game is still slow, the cinema is not what is slow.

A screen that cannot start is attempted once per source, not once per tick — otherwise a bad URL would spawn twenty processes a second. The reason is kept until the server changes what that screen shows or stops it, so fixing the config and reloading is what retries it.

Add cinema to addons.uses in config.yml as well, and players who do not have the client addon are offered it on their Addons tab instead of seeing a blank wall.

Building one without editing the file複製段落連結

Everything below writes straight back to addons/cinema.yml, so a cinema built in-game survives a restart.

/ssvc-server addon cinema list
/ssvc-server addon cinema create <id> [width] [height]     where you are standing, facing where you look
/ssvc-server addon cinema delete <id>                      also spelled remove, and also works as <id> delete
/ssvc-server addon cinema <id>                             what this screen is and what it is doing

Then, per screen:

A time is written however is easiest: seconds (90, 90.5), a clock (1:30, 1:02:03) or units (1h2m3s, 45m, 90s). So seek 1:02:03, seek 1h2m3s and seek 3723 all land in the same place, and so does play 1h.

All of it tab-completes — screens, verbs, facings and resolutions. A URL never does, because a console has nothing to guess one from.

Gated on socketsvc.admin.addon, which defaults to op.

/ssvc-server reload re-reads the file and leaves a screen that did not change still playing where it was. A screen whose source the file changed does restart, because it is no longer showing what it was showing.

Speakers複製段落連結

A screen with no speakers is heard from the picture itself. Give it some and the sound comes from where they are instead:

speakers:
  - x: 6.5
    y: 66.0
    z: -20.5
    facing: north
    range: 48
  - x: -5.5
    y: 66.0
    z: -20.5
    facing: north
    range: 48

Each one is a real positional, directional channel: full volume in front, quieter behind, fading out past its range. facing defaults to the screen's own and range to about eight times its width in blocks.

They are all fed from one decode. A stack either side of a screen sounds like one film from two places, not two films a few milliseconds apart — decoding per speaker would drift, and drift is audible as phasing long before it is audible as echo. Up to 16 per screen.

Adding or removing a speaker restarts that screen, because the decoder feeds a fixed set of channels. Moving, turning or re-ranging an existing one is applied live.

Resolution複製段落連結

resolution is what each client decodes at, up to 1920x1080. Left unset it is derived from the screen's size in blocks at 64 pixels per block — a 6-block screen lands on 384x216 — and a derived resolution is capped at 854x480. A wall-sized build would otherwise select 1080p on its own, which is 8 MB a frame on every viewer for a picture nobody asked to be that sharp. Anything above the cap is opted into.

Where the settings are複製段落連結

All three of the settings below are on the Addons tab, under the cinema's own switch: a volume slider, a quality choice, and a frame rate. They are the same settings the commands and the keys move, so changing one anywhere moves it everywhere — a slider on an open screen follows a volume key being held down.

The commands stay because a screen is not always where you are when a film is too loud.

Volume複製段落連結

A film is the one sound in the game somebody wants quieter without turning down the people talking over it, so the cinema has its own level, per player:

/ssvc addon cinema volume            what it is now
/ssvc addon cinema volume up | down  in steps of 10%
/ssvc addon cinema volume 60         0 to 200

It also claims two of SocketSVC's eight reserved key slots — Cinema volume up and Cinema volume down — bindable in the ordinary controls screen. Running out of slots is an ordinary thing to happen to somebody with several addons, so the command works either way.

A change is heard straight away, on a film that is already playing — within about a tenth of a second, however much of the film is buffered — and fades to the new level over a single 20 ms frame rather than jumping, so it does not click.

Applied to the samples rather than to the channel, because a local positional channel has a position, a range and a cone but no gain of its own. Levels above 100% are clipped rather than allowed to wrap — a sample past a short's range that is simply cast flips sign, which is heard as a crack rather than as distortion. The setting is saved in the addon's own data directory.

When your own machine is the problem複製段落連結

The server picks a resolution for everybody, and "everybody" includes whoever has the weakest machine in the room. Two client-side levers, both saved per player:

/ssvc addon cinema quality <full|high|medium|low>   a ceiling on the server's resolution
/ssvc addon cinema fps <auto|5-60>                  lock the decode rate

quality only ever reduces — a client cannot ask for more detail than the server is offering, because the server is the one that knows what the source can supply. Halving the resolution quarters the pixels.

fps is the blunter lever: halving the frame rate halves the decode, the copy and the upload together, in one step. It is the first thing to reach for on a machine that is struggling and the last thing anybody notices on a screen across a room.

Both restart what is playing, because a decoder is told its size and rate when it is spawned and cannot be told new ones. From the settings screen the restart waits until the value stops moving for 400 ms — dragging a frame-rate slider across its track would otherwise spawn a decoder per step and throw away all but the last.

Sync, in practice複製段落連結

The server holds a playhead and the moment it was true, and re-sends it every five seconds while anything is playing. Each client compares where its screen is against it, and catches up while the film plays rather than by restarting it.

The usual way a screen falls behind is its sound running out: on a slow connection the buffer empties, the film pauses while it refills, and the server's clock keeps going. So once a screen has been showing for two seconds:

The picture follows the sound, so it catches up with it. The pitch does not change: rather than resampling, which would make two percent faster a third of a semitone sharp, a few milliseconds of sound are dropped or repeated at a time, at the point where the waveform either side matches best, across a short crossfade. Between those joins every sample plays exactly as decoded, and a screen that is in step is never touched at all. A live source has no timecode to be out against and is never corrected.

Picture and sound, from one decoder複製段落連結

A screen decodes its picture and its sound with one FFmpeg process, from a single read of the source, so the two cannot disagree about where the film is. They used to be two processes, each opening the source for itself — and on a live stream those two could join at different points, or treat an ad break's splice differently, which is a desync that nothing afterwards can take back. Both outputs are also held to the source's own timestamps, so a film whose sound track starts a moment after its picture, or has a gap in it, stays in step wherever it is joined.

A source with no sound track, or sound and no picture, cannot be decoded that way — FFmpeg refuses an output with nothing to carry — so it is played with a decoder each instead, as before. That happens on its own; the log notes it once.

Live streams and ad breaks複製段落連結

• Every stream keeps a reserve — see the pipeline above. A live one starts as soon as 2 seconds are in hand — joining a live stream hands over a few seconds at once, so usually without a noticeable wait. If a bad connection runs the reserve dry, it pauses until a slightly bigger cushion has built up — 3.5 seconds, then 5, never more than 8 — and every minute of smooth playing gives a second of that back, so the stream drifts back towards live instead of staying behind for good. A film works the same way, except that it is fetched from where the server says it is: the fetcher seeks to the keyframe before that moment and keeps the film's own timestamps, and the decoder cuts at the moment itself, so the first frame is the one asked for and not the keyframe before it. A film in a playlist (HLS or DASH) is the one kind decoded straight off the network, because a playlist's clock may start over at a stitch and an exact cut cannot follow that.
• A Twitch channel is live on its own. source https://www.twitch.tv/<channel> marks it live without the live word; a past broadcast (twitch.tv/videos/...) or a clip is an ordinary film.
• Ad breaks keep the sound in step. Twitch splices ads in at their own resolution and sample rate; a live stream's sound is decoded so that the change of format cannot push it behind the picture.
• A dropped live stream reconnects. If a live screen's stream stops arriving for twenty seconds, or its decoder stops, it reconnects — looking the stream up afresh, since a Twitch stream address expires — after 2 seconds, then 4, 8 and so on up to 30 if it keeps dropping. A stream that ends for good (the channel goes offline) stops on the error yt-dlp gives.

/ssvc addon cinema shows each screen's sync — in step, or how far behind or ahead, the speed it is playing at, and how many times its sound ran out — and the game log notes every time a screen runs dry.

A restart is still there for a screen that is genuinely lost — somebody who joined mid-film, a decoder that stalled for a long while. FFmpeg cannot be told a new position while it runs, so a restart is visibly the film starting again, and three things stop it becoming a loop:

• The playhead is measured in decoded frames, not in elapsed time. A wall clock says where the film ought to be; it cannot tell that the decoder spent four seconds resolving a URL or that it is falling behind on a slow connection. The decoder is asked for a fixed frame rate, so counting what it produced is the only honest measure of how much film has gone past.
• The seek aims slightly ahead, by however long that screen took to start last time, so the first frame lands where the server says the film is then rather than where it was when we began.
• A twelve-second grace before a freshly started screen is judged for a restart, and a minute between one screen's restarts.

Resolved stream URLs are also cached for twenty minutes, so a restart does not re-run yt-dlp — which is the slowest part of starting and was most of what made a resync loop unbearable.

A film that stops while the server still says it is playing is given one more go after two seconds, with its address looked up afresh — a dropped connection or a signed address that expired mid-film looks exactly like the end of the film from the client, and the server cannot tell them apart either, since it never opens the file. A film that really has ended finds nothing at the playhead on the second go and says so; it is not tried a third time within ten minutes. A decoder reading straight off the web is also told to reconnect where it left off if the server drops it.

Limits worth knowing before you build a cinema複製段落連結

• A screen that is not playing says why, on itself. A dark panel with a line across the middle — Buffering 40%, Resolving..., Out of range, Unavailable, Paused, No source — and a progress bar under it where there is a real fraction to show. Buffering has one; resolving a URL does not, and a bar crawling along on no information is worse than no bar. The text scales from the screen's own size, so it reads the same on a two-block screen and one that fills a wall, and it is drawn over the picture as well as the panel — a film that stalls mid-play keeps its last frame, and the reason it stopped belongs on top of it.
• A screen with nothing playing draws as a dark panel, not as nothing. An invisible screen is indistinguishable from a screen nobody ever configured, and "off" versus "not there" is the first thing anybody wants to know.
• Only players with the cinema addon see anything. A client without it never asks SocketSVC for the screen snapshot at all. An addon is an offer, never a requirement, and there is deliberately no way for a server to insist on one.
• A screen is flat and axis-aligned. Curved and angled screens would need a quaternion on the wire to serve builds nobody has asked for.
• Vertical audio is flat. The HRTF is horizontal, so a speaker above the audience is placed correctly left-to-right and not vertically.
• Every viewer decodes independently. Fifty players watching one screen is fifty FFmpeg processes on fifty machines — which is the point, and also why the default is 256×144 at 24fps rather than something a laptop cannot afford.