Building VoiceCraft: a voice-controlled RTS in Rust

What if you could play a real-time strategy game by speaking like a commander instead of clicking like an operator?

That question became VoiceCraft, a voice-controllable, StarCraft-style RTS prototype built in Rust. You can still use a mouse and keyboard, but the central experiment is different: describe the outcome you want, and let the game translate your words into action.

“Select the scouts, move north, and hold near the crystals.”

A command like that carries selection, movement, and intent in one sentence. VoiceCraft explores whether spoken language can become a useful layer above the traditional RTS interface—not a novelty bolted onto it, but a new way to manage attention.

Why voice belongs in an RTS

Real-time strategy games are full of decisions, but many of the actions between a decision and its result are mechanical. Select a group. Move the camera. Find a location. Click an ability. Return to the base. Repeat.

Voice can compress some of those steps. A player should be able to say:

• “Send two workers to the eastern resource field.”
• “Group these units as Alpha.”
• “Alpha, defend the northern entrance.”
• “Build another supply structure after this one finishes.”

The goal is not to remove mechanical skill or turn the game into an autopilot. It is to give the player another input channel. The mouse can handle spatial precision, the keyboard can handle immediate actions, and voice can express higher-level intent.

That combination is what makes the idea interesting.

The command loop

At a high level, every spoken order passes through a small pipeline:

speech → transcript → game command → validation → action → feedback

The transcript is only an intermediate format. VoiceCraft still needs to determine what the player meant, resolve references such as “these units” or “the north entrance,” check whether the order is valid, and turn it into deterministic game actions.

The command layer uses a constrained set of game concepts rather than allowing arbitrary code or unrestricted model output. A parsed order might look conceptually like this:

Command::Move {
    units: UnitSelector::Group("alpha"),
    destination: Location::Named("north_entrance"),
    stance: Stance::Defend,
}

Once language becomes a typed command, the simulation can treat voice input exactly like any other player input. That boundary is important: interpretation can be probabilistic, but the game state should not be.

Why Rust

Rust is a natural fit for the parts of this project that need to be fast, predictable, and safe under pressure.

An RTS simulation touches many systems at once: units, orders, pathfinding, resources, construction, combat, and the user interface. Rust's type system makes the boundaries between those systems explicit, while its performance leaves room for large unit counts and low-latency input processing.

It also encourages a useful architectural split:

• The simulation owns authoritative game state and advances in fixed steps.
• The command layer converts player intent into validated actions.
• The presentation layer renders the world and explains what the game understood.
• The voice layer listens and produces candidate commands without directly mutating the world.

Keeping those responsibilities separate makes voice control less magical—and much easier to test.

The hard part is not speech recognition

Turning audio into words is only the beginning. The more interesting problems appear after transcription.

Latency

An order that arrives two seconds late can be useless in the middle of a fight. Voice commands need to feel immediate, which means the pipeline must stream partial results, recognize common command shapes quickly, and avoid waiting for more language than it needs.

Ambiguity

“Move them over there” makes sense when a human can see the player's selection and cursor. The parser needs the same context. VoiceCraft therefore treats the current selection, camera position, recent commands, named groups, and visible landmarks as part of the command—not as hidden trivia.

Errors

In a strategy game, silently doing the wrong thing is worse than asking for clarification. A good voice interface should show what it heard and preview the interpreted order. High-impact or ambiguous actions can require a quick confirmation; obvious, reversible commands can execute immediately.

Feedback

Voice control needs visible feedback. Selected units should highlight, destinations should appear on the map, and the recognized command should be readable at a glance. The player must always know what the game is about to do.

Designing for both speed and accessibility

VoiceCraft started as a control experiment, but it also points toward a more flexible RTS interface.

Players who find rapid keyboard shortcuts difficult could use speech to reduce repetitive input. New players could express goals before memorizing every hotkey. Experienced players could combine voice with traditional controls, delegating routine actions while keeping direct control of precise micro.

The important design rule is that voice remains optional. A strong multimodal interface lets each input method do what it does best and allows the player to choose the mix.

What I am building toward

The first milestone is a small, complete match loop: gather resources, construct a base, produce units, and fight an opponent. Around that core, I am focusing on a compact voice vocabulary with enough context to feel natural.

The experiments I want to run next include:

• Naming and recalling control groups through conversation.
• Chaining short orders without introducing dangerous ambiguity.
• Measuring end-to-end command latency during combat.
• Supporting local speech recognition for privacy and offline play.
• Replaying typed commands to make simulation bugs reproducible.
• Testing which actions feel better by voice—and which should stay on the mouse and keyboard.

VoiceCraft is not trying to prove that every game should be voice-controlled. It is a way to explore a narrower, more useful question: can natural language make a complex game feel more direct without making it less precise?

That is the experiment. Now it is time to build the battlefield.