A world is not only 3-D. It has panels drawn over the picture (a status panel, a map), surfaces in the world that show pictures (a mirror, a screen, a sign), words that appear when someone speaks, and materials painted from pictures. ConeScript gives all of these one type, the image, and this page defines it: what an image is, where images come from, what a pixel carries, and how images are laid over one another (compositing). The next two pages build on it: Layouts is the vocabulary for describing 2-D content, and Overlays, HUDs and speech is what a world draws over its visitors' pictures.
What an image is
An image is a rectangle of pixels: a width, a height, and at each pixel a few numbers. It is a value, like a number or a color. Everything 2-D in a world is an image or becomes one:
- the picture a camera makes of the world (each visitor's view is one);
- the picture in a mirror, a portal, a security-camera screen or a far-seeing stone;
- a status panel, a map, a dialogue, a speech bubble, once laid out;
- the pattern painted onto a material (see Textures);
- the shadow a light casts, which an engine keeps as an image of depth seen from the light.
Three sentences hold the whole model:
- Rendering a view gives an image. A camera looking at the world produces a picture.
- Laying out 2-D content gives an image. Boxes, words and shapes, once placed, produce a picture.
- Compositing images gives an image. Laying one picture over another, in an order, with a rule for how their pixels mix, produces a picture.
So a HUD (heads-up display: the panels drawn over a visitor's picture) is not a special kind of part. It is a 2-D layout rendered into an image and composited over the camera's image. The same layout could as well be put on a sign, a mirror, a wrist or a screen in the world: the layout does not know or care where its image ends up.
Terms used on this page. A texture is an image held on the graphics card where shaders can read it. A render target is a texture the graphics card draws into; engines call drawing a camera into one "render to texture". Compositing is combining images by laying them over each other. A pass is one sweep of the graphics card over an image (drawing the scene, blurring it, mapping its brightness). Tone mapping is the pass that turns the picture's light, which may be far brighter than a screen can show, into the colors a screen can show; before it the picture is in light values, after it in display colors.
How you write it
The type
image is a value type. A field may hold one, a function may return one, and a derive may define one from a source:
part lookingGlass, at: (...) // a wall mirror
part pane
field picture: image // what the glass shows: made by a source, below
Sources
A source makes an image from inputs, and makes a fresh one whenever an input changes. Wiring a source to the place its image is used is a subscription (Subscriptions): the image is defined by the source and is never set.
| Source | Makes an image from | Example |
|---|---|---|
render(camera: c, skip: [...]) | a camera's view of the world, leaving out the listed parts | the looking-glass's reflection; a portal's far side; a security camera |
layout(...) | a description of boxes, text, pictures and shapes (Layouts) | the HUD; a notice board; a dialogue |
paintedOnce(recipe, texel: d) | a recipe run once, at a resolution of d per pixel | a top-down map |
composite(base, over: [...], ...) | other images laid over a base | a visitor's finished picture |
| another part's image | the image itself, shared | gateCamera.picture on a screen |
part lookingGlass, at: (...)
part pane
field picture: image
derive(pane.picture, // a camera: the viewer's own, reflected about the glass,
render(camera: reflected(viewingCamera, about: pane), skip: [pane])) // drawn without the glass itself
derive(pane.look, screenMapped(pane.picture)) // shown where each pixel of the glass falls on screen
part gatehouse.screen, at: (...) // illustration
field picture: image
derive(picture, gateCamera.picture) // any image may feed any surface
part hud
field map: image = paintedOnce(topDown(land, walkway, brook, vegetation), texel: 2 m)
part noticeBoard, at: (...) // illustration
part board
field picture: image
derive(board.picture,
layout(text("Lanterns lit at dusk", size: 6 cm, style: styles.parchment)))
viewingCamera names the camera drawing the picture that is being made, not the visitor's being. That distinction matters as soon as one picture is seen inside another (below, "Per camera").
Compositing
composite(viewingCamera.picture, // the base: what the camera sees
over: [layer(hurtEdge.picture, blend: add), // a world's own layer
layer(hud.picture)], // blend defaults to over
order: [world, hud, browser]) // the browser's own controls are always last
layer(image, blend: b, opacity: k) is one image in a composite. Blends are over (the upper image's pixels cover the lower ones by their transparency), add (the upper image's light is added, as a glow is) and multiply (the lower image is darkened or tinted by the upper). browser in an order stands for the browser's own controls and may appear only last; a world that leaves it out has it put last anyway.
Effects on an image
2-D effects are functions from an image to an image: decals, ambient occlusion, fog, depth of field, temporal anti-aliasing, auto-exposure, bloom, tone mapping, color grading, a vignette and film grain. Written as functions:
derive(edge.picture, vignette(viewingCamera.picture, amount: 0.3, radius: 0.45, softness: 0.9)) // illustration derive(duskLook.picture, graded(viewingCamera.picture, lift: (0, 0, 0.02), gain: (1, 0.95, 0.9), saturation: 0.9)) // illustration
Bloom (bright light spilling into a soft glow), anti-aliasing (smoothing stair-stepped edges) and tone mapping are themselves image passes: a world's glow and smooth edges are images made from images, before any HUD is laid over them.
What it means
An image is a value; a source keeps it live
An image held in a field is a value like any other: it may be stored, passed to a function, put on a material. A live image (the HUD, a mirror, the minimap, a security-camera screen) is one whose definition is a source with inputs that change. The engine makes a fresh image when an input changes, as a spreadsheet recalculates a cell (Subscriptions). A mirror's inputs include the viewer's camera, so the mirror's image changes as the viewer moves; a map painted once has no changing inputs and is made once.
Because a source is a value definition, it follows the rules of value blocks (Blocks): making an image may not act on the world. A layout may describe a button; pressing the button is an event, and the action it performs runs in a handler, not in the layout.
Keep the description until the last moment
A layout or a piece of text is kept as a description (these words, this font, this size, this color) and turned into pixels only at the size it ends up on the screen. Walk up to a sign and its words stay sharp, because they are made again at the larger size, not stretched. This is the procedural bet applied to 2-D: send the recipe, make the pixels where they are needed. It also keeps a world small to download.
This applies in two places:
- Text. Each letter is kept as a multi-channel signed distance field (MSDF): instead of pixels, each texel holds distances to the letter's edges, from which an edge can be drawn one screen pixel wide at any size and angle.
- Pictures on surfaces. A mirror's picture is drawn only for the part of the screen the glass covers, from a camera cut to that rectangle, into a texture just big enough to hold it at the frame's own sharpness. A glass that covers only a few pixels of the screen need not get a new picture and may show the last it held.
The resolution of a live image is therefore not something content states: the engine chooses it from where the image ends up. What content may say instead is an open question (below).
Channels: what a pixel carries
| Channel | What it is for | Example |
|---|---|---|
| color | what the pixel shows | every image |
| transparency (alpha) | how much of what lies under it shows through: a panel of dark glass over the picture | a HUD's panels: dark glass |
| depth | how far each pixel is from the camera, so that a 2-D image composited with a 3-D picture can be hidden by what stands in front of it: words above a guard but behind a pillar | a view's picture keeps a depth texture beside its color; world text is tested against the scene's depth without writing it |
| perhaps more | the way a surface faces (normals), which part a pixel belongs to (ids, which the pointer could use) | not designed |
Color itself comes in two kinds. Before tone mapping an image holds light, which may exceed what a screen shows (so that light brighter than 1 survives to bloom). After tone mapping it holds display colors. A HUD is laid over the picture after tone mapping, so that its colors come out exactly as written. It follows that a layout composited over the picture is in display colors, while the same layout on a sign in the world is an image the sign's material reads, and is lit by the world's light like any surface.
Per camera
A mirror's picture depends on the camera looking at the mirror, so it is made once per camera that sees the glass, not once per visitor: each visitor sees their own reflection. A camera is not always a visitor's eye. Looking through a far-seeing stone, or at a security screen, the picture inside is drawn by another camera, and anything placed round "the viewer" in that picture (rain round the eye, the sky) is placed round that camera (Cameras and views). In practice:
- A picture inside a picture is drawn after the picture it shows, and only to a chosen depth: by default one view inside another. A view deeper than that is not drawn and shows what it last held, one frame behind.
- A surface is on its own drawing layer, which no picture draws, so it is left out of its own picture, and a portal seen through a portal shows what is behind it rather than a second picture.
- A picture is drawn only for a surface in front of the viewer and inside what the camera can see; a surface nobody sees costs nothing.
Compositing is the world's, within what the browser keeps
Which images are laid over which, in what order, blended how, and with which 2-D effects, is under the world's control. The browser keeps one thing for itself: its own controls are composited last and cannot be hidden or covered. A world that could hide the browser's controls could pretend to be the browser, and a visitor must always be able to leave, mute or block. That is the line between the world's composition and the browser's (Overlays, "Who owns which overlay").
The pointer travels back through the composition
Because a visitor's picture is a composition, a click lands on a pixel of the result, and the engine must work out which input image, and which pixel of it, that pixel came from. A click on an overlay maps to the layout element under it. A click on a sign in the world follows the ray from the eye to the sign's surface, then to the point in the sign's image, then to the element there. So every compositing step must be invertible: given a pixel of the result, which pixel of which input made it. This has to be designed in from the start. Overlays sets out the routing.
What an image cannot do
- It cannot act. An image is a value; making one never sets, starts, spawns, sends, says or plays.
- A world cannot reach the browser's own controls through compositing: they are not an image the world can read, cover or reorder.
- A world names sources and effects from the browser's vocabulary; it does not write graphics-card code (Packages).
How it connects
- Layouts: the
layout(...)source and its vocabulary. - Overlays, HUDs and speech: images composited over each visitor's picture; the pointer.
- Cameras and views: cameras as sources, mirrors, portals, far-seeing.
- Materials and Textures: materials that read images.
- Subscriptions: how a live image follows its inputs.
- Distribution: images are made on each visitor's machine; only acts cross the network.
Open questions
- Per camera or per visitor. A mirror's picture is made once per camera that sees the glass; a speech plate is in each visitor's picture. The language needs a way to say which, the same question as rain round each camera.
- The resolution of a live image. A far mirror needs few pixels, and a glass only a few pixels across needs no new picture. Whether content may state a resolution, or only a quality it requires, follows from the rule that content states requirements, not mechanisms; not settled.
- Which channels. Color, transparency and depth are needed; whether images also carry normals or part ids (for the pointer) is open.
- Depth in overlays. How a 2-D image composited over a 3-D picture is placed in depth (one depth for the whole image, or a depth per pixel) is not designed.