A Rhythm Game to Practice a Rhythm Game

I've been playing rhythm games since I was about 10. That process of learning new patterns and drilling them into my head until it clicks is most of the fun for me.

Fortnite Festival has no practice mode. Some charts are intense or long and there's no way to loop a section without playing the whole thing start to finish, which is a feature every rhythm game I've played in the past decade has had.

I know Rust and also use it in the microservices for Originoid; this felt like the right kind of problem, something I needed and something that would teach me, so I built a full practice client from scratch. Some of the behaviour differentiates from Festival itself since this was a personal project built around how I wanted to practice rather than a 1:1 recreation; most of the differences are exposed as configurable settings so the feel can be tuned to preference.

Architecture

The application is a game state machine where macroquad handles 3D rendering (the note highway) and egui handles menus and HUD as an immediate-mode GUI overlay; each frame polls input, advances game logic and renders while background tasks run on separate tokio/thread contexts communicating results back via shared state.

src/
├── audio/mod.rs         # SoLoud engine, multi-stem playback, position smoothing
├── chart/midi.rs        # MIDI parsing, note extraction, lift/chord detection
├── chart/tempo.rs       # Tempo mapping
├── config/settings.rs   # Persistent settings (JSON), keybinds, per-chart offsets
├── game/state.rs        # AppState enum (Loading, SongSelect, Playing, Paused, etc.)
├── game/session.rs      # Core gameplay: hit detection, sustains, overdrive, per-frame update
├── game/scoring.rs      # Points, combo multiplier, accuracy tracking
├── game/timing.rs       # Customisable hit windows
├── game/overdrive.rs    # Overdrive meter, activation, drain, phrase tracking
├── game/practice.rs     # Practice mode: sections, looping, count-in, auto-restart
├── game/replay.rs       # Input recording, replay database, seek/playback
├── input/keyboard.rs    # Key-to-action mapping, press/release edge detection
├── render/highway.rs    # 3D note highway, note/sustain rendering, smasher zone
├── render/hud.rs        # Score, combo, timing feedback, UR histogram, NPS graph
├── song/mod.rs          # Song loader: decrypt, parse, cache
├── song/stems.rs        # FFmpeg-based 5-stem extraction
└── ui/                  # Song select, settings, results, practice menu, replay dialog, etc.

The state machine progresses through Loading (settings, replay index) to SongSelect (track browser with search and instrument/difficulty selection) to LoadingSong (decrypt, parse chart, load stems) to Playing (active gameplay) which can transition to Paused, Practicing (looped subset with count-in and auto-restart) or Results (final score, accuracy, timing histogram, replay save); Results can then transition to WatchingReplay where recorded input events play back against the original chart.

Chart Source and Decryption

Fortnite stores its chart data as encrypted MIDI files; the how and where of acquiring them is something I will not be including. The encrypted charts use AES-256 in ECB mode; decryption strips PKCS7 padding and validates the output by checking for MThd (the standard MIDI magic bytes) at the start of the decrypted buffer. Once decrypted, the MIDI is parsed with the midly crate into instrument-specific note data for the selected difficulty.

The track catalog is persisted to disk as a JSON database; a background SyncManager thread handles refresh, reporting status via a SyncStatus enum (Idle, Fetching, Completed, Failed) so the song select screen can show sync progress without blocking the game loop.

MIDI Parsing

Each decrypted MIDI file contains instrument-specific tracks at four difficulty tiers where the MIDI note numbers map to fret positions:

Special markers encode gameplay mechanics beyond the basic fret notes: MIDI note 116 defines overdrive phrase boundaries where note-on starts a phrase and note-off ends it, base+6 through base+10 are lift notes and notes occurring on the same MIDI tick are detected as chords. Each parsed note is stored as a struct carrying its tick position, absolute time in milliseconds, fret number (0-4), sustain duration in ticks, milliseconds and beats, along with a NoteFlags bitfield using the bitflags crate for efficient per-note metadata:

pub struct Note {
    tick: u32,
    time_ms: f64,
    fret: u8,              // 0-4 (green through orange)
    sustain_ticks: u32,
    sustain_ms: f64,
    sustain_beats: f32,    // for display scaling
    flags: NoteFlags,      // TAP | OVERDRIVE | LIFT | CHORD
}

Practice Sections

Sections can be user-defined; you set a start tick, end tick and display name, and the practice mode uses those as its loop boundaries.

Song Loading Pipeline

The full pipeline from track selection to gameplay: the user picks a song, the client decrypts the chart and produces a standard MIDI file, the parser extracts notes for the selected instrument and difficulty while building the tempo map and detecting lifts, chords and overdrive phrases, stems are loaded from cache if they exist; the state machine then transitions to Playing or Practicing.

Input System

The input handler in input/keyboard.rs maps physical keys to a GameInput enum covering FretPress(0-4) for lane presses, FretRelease(0-4) for lift note detection, Overdrive, Pause, Restart, Confirm, Back and directional navigation. Default keybinds put the five frets on A/S/D/F/G with Space for overdrive, Escape for pause and R for restart; all keybinds are customisable and persist to settings.json.

The critical detail is edge detection: the handler compares the current frame's key state against the previous frame's state to emit press and release events once per transition, preventing held keys from generating repeated events. Lift notes depend on this since they trigger on fret release rather than press, meaning the input system needs to distinguish "key went from up to down this frame" from "key is still down from last frame" on every fret for every frame.

Hit Detection

Per frame, the engine in game/session.rs iterates through unprocessed notes in chronological order; for each note it checks whether the correct fret is active (pressed for standard notes, released for lift notes) and calculates delta = current_audio_position - note_time. If |delta| falls within the Perfect window the hit is rated at 1.0x, within the Good window it's 0.5x, beyond the window with no matching input the note registers as a miss. Both windows are customisable in settings. Each hit records its timing delta for the UR histogram; the engine then advances to the next unprocessed note.

Sustains activate when a note with sustain_ms > 0 is hit; points accumulate at 0.025 per millisecond for each frame the correct fret remains held, stopping when the player releases the fret or the sustain duration elapses. Chords require all frets in the chord to be held at the same time.

Scoring

Scoring is based on CHOpt's calculation, which is the standard the community uses for Fortnite Festival path optimisation and score verification.

The results screen shows timing metrics alongside the score: unstable rate is standard_deviation(all_timing_deltas) * 10 following the same formula osu! uses where lower values indicate more consistent timing, average offset is the mean of all deltas showing whether the player tends to hit early or late across the session; per-rating hit counts break down how many notes were Perfect, Good or Missed.

Audio Engine

The audio system in audio/mod.rs is built on SoLoud, a C++ audio engine accessed through Rust bindings, running in two modes depending on whether stems are available.

Multi-Stem Playback

The stems are MP4 containers with 10 audio tracks representing 5 instrument stem pairs; I extract them into separate WAV files using FFmpeg (which must be in PATH as an external tool). Detection via ffprobe handles two MP4 formats I've encountered: a single audio stream with 10+ channels where pan filters extract each pair, or 10 separate mono streams that get merged into stereo pairs. The output is PCM 16-bit WAV at 48kHz per stem:

A manifest JSON tracks which stems have been extracted along with a timestamp, so the client knows whether to re-extract after a song update. The stem extraction pipeline runs from the UI's stems dialog where you select the MP4 file via a native file picker (rfd crate) and ffprobe/ffmpeg handle the rest.

Stem Behaviour During Gameplay

The stem matching the player's selected instrument gets a 1.5x volume boost so you can hear your part above the mix; each stem tracks a state machine:

enum StemState {
    Normal,
    Cut,
    FadingIn { start, duration },
}

Miss a note and the player's stem transitions to Cut (muted); start hitting again and it transitions to FadingIn where volume ramps from zero back to normal over a configurable duration. The backing stem is never cut since it provides the base mix the player needs to stay oriented in the song. Playback speed (0.5x through 2.0x for practice mode) applies across all five stems.

Position Smoothing

SoLoud reports playback position with roughly 45ms granularity, which causes visible note stutter on the highway if I use the raw value for scroll position; the smoothing algorithm interpolates between updates by anchoring the SoLoud-reported position alongside a wall clock timestamp whenever a position jump (>0.1ms change from the last report) occurs, then estimating position on subsequent frames as anchor + (wall_clock - anchor_time) with the interpolation capped at 60ms to prevent overdrift during frame drops or GC pauses. A fallback wall-clock timer provides position tracking when no audio is loaded at all, so the chart can still scroll and gameplay functions in notes-only mode.

3D Highway Renderer

The highway in render/highway.rs uses macroquad's 3D API with a perspective camera positioned at (0, 5, -3) looking at (0, 0, 8) with a 45-degree FOV; the highway surface is a black plane with edge dividers and lane markers, the smasher zone (the target line where notes should be hit) sits at Z=2.0 in 3D space; each note's Z position is calculated as z = (time_until_note * track_speed * highway_length) / 1.5 so notes scroll toward the player along the Z axis at a rate determined by the configurable track speed.

Standard notes render as notes, lift notes as triangles (to distinguish "release this fret" from "press this fret" at a glance), sustains as coloured tails with configurable width extending behind their parent note and overdrive notes in a custom colour (default white). Lane colour presets ship with the client: Festival (all purple to match the in-game palette, the default), Festival Pro (green/red/yellow/blue/orange) and Monochrome.

HUD

The HUD in render/hud.rs is an egui layer rendered on top of the 3D highway via egui-macroquad using a custom dark theme, displaying the score counter, combo display, timing feedback text ("Perfect"/"Good"/"Miss") with a fade-out animation, a UR histogram (an osu!-style timing bar showing the spread of hit deltas), an NPS difficulty graph that splits the chart into 100 equal time segments and plots the notes-per-second for each segment to show intensity over time, the current section name during practice mode, the overdrive meter bar and a cached album art thumbnail.

Practice Mode

The practice system in game/practice.rs extends the gameplay engine with section-based looping; you can create sections with arbitrary start/end timestamps and optional per-section speed overrides, chain multiple sections together, or play from any seek position. Speed presets at 0.5x, 0.75x, 1.0x, 1.25x and 1.5x affect both audio playback rate and note scroll speed.

A configurable count-in plays the audio with no notes for a specified number of beats (default 4) before the section starts, giving you time to orient yourself before notes appear; auto-restart monitors the miss count during a section and loops back to the section start with the count-in when the threshold is exceeded. Custom sections persist to a file alongside the chart data so they survive between sessions.

Replay System

The replay system in game/replay.rs records every input event during gameplay - fret presses, fret releases, overdrive activations - with their timestamps relative to the chart start. After a session, you can save the replay with metadata (score, max combo, accuracy percentage, per-rating hit counts, timing statistics including UR and average offset) to a local replays directory as a replay index entry with a separate event file.

Playback feeds the recorded events back into the gameplay engine against the original chart, meaning the same hit detection, scoring and visual feedback run as they did during the original session. Seeking within a replay works by replaying all recorded events from the start up to the target timestamp to reconstruct the full game state (combo, score, overdrive meter, sustain states) at that point, which is straightforward since the event list is in chronological order.

Configuration

All settings persist to %APPDATA%/[client-name]/data/settings.json and include keybinds for all five frets plus overdrive, pause and restart; master volume (0.0-1.0); a global audio offset in milliseconds for system-level latency calibration; track speed presets controlling highway scroll rate; adjustable hit window timing; lane colour preset selection; highway render distance; note size and sustain tail width; overdrive colour; and UI toggles for timing feedback visibility, the UR histogram and the NPS graph.

Per-chart audio offsets are stored in {data_dir}/offsets/{shortname}.txt so songs that need individual calibration (due to audio encoding differences or stem sync issues) don't pollute the global offset value. The storage layout:

data/
├── settings.json      # Application settings and keybinds
├── offsets/           # Per-song audio offset calibration
├── cache/             # Charts, extracted stems, album art
└── replays/           # Replay index database and event files

Everything runs from local storage after initial setup. The binary is 11MB and I've never seen it use more than 500MB of RAM. I won't be making the client public.