The chore/reorg branch drifted too far to merge cleanly. Captures the
worthwhile ideas as actionable proposals: clientctx removal, lazy packet
priority ordering, SessionStage interface, semaphoreIndex race fix, and
several medium/low-priority restructuring items.
Add a JSON Format Support section to the README covering quests,
scenarios, and Hunting Road config. Add two missing CHANGELOG entries
for the rengoku .bin-first fix and the quest event board JSON fallback.
JSON-authored quests were serveable via MSG_SYS_GET_FILE but invisible
on the event quest board because loadQuestFile only read .bin files.
CompileQuestJSON already produces the uncompressed binary layout that
the event-board parser expects, so no decompression step is needed.
Also update rengoku priority tests to match the .bin-first convention
fixed in the previous commit.
All three binary loaders (quests, scenarios, rengoku) now follow the same
priority: .bin first, .json as fallback. The previous reversed order
(.json first) was confusing for operators with both files present.
Add jkrMagic and scenarioChunkSizeLimit named constants, and expand
comments throughout scenario_json.go to reflect confirmed client
behaviour: the 0x8000-byte per-chunk limit, which C0 metadata fields
the parser actually reads (m[0]–m[6] only), the signed-offset
encoding used for C1 m[8]–m[17], and per-byte roles in the 8-byte
sub-header. Inline magic literal replaced with the named constant.
Document client-side parser behaviour confirmed from mhfo-hd.dll:
- Container: 0x8000-byte per-chunk limit enforced by FUN_11525c60
- C0 metadata: client reads only m[0]–m[6]; m[7]–m[9] are not read,
so the constant-5 (m[8]) and the size-correlated m[9] are opaque to
the parser and can be set to any value
- C0 m[2]/m[5] role clarified: m[2]=0 is offset to str0 (no-op),
m[5]=str0_len is offset to str1 from strings section start
- C0 m[6] SceneRef stored as signed short; 0xFFFF = −1 for cat≠0
- C1 metadata: m[8]–m[17] are signed offsets — negative encodes
position from the post-0xFF dialog data section via ~value formula,
non-negative encodes position from the strings section start;
m[18–19] are read as individual bytes, not u16 pairs
Add TestScenarioRoundTrip_RealFiles covering 7 real scenario files
(cat=0/1/3, T101/T103, with and without chunk1). Tests skip gracefully
when game data is absent so CI stays green.
Decoded metadata structure from analysis of 145k+ real scenario files:
- Chunk0 m[0]/m[1] = CategoryID/MainID; m[5] = str0_len (offset to
str1); m[6] = MainID (cat=0) or 0xFFFF; m[8] = constant 5.
- Chunk1 m[9]=cumOff[2], m[10]=cumOff[1], m[14]=cumOff[3],
m[15]=total_string_bytes; m[11-13]/m[16-17] = dialog script offsets
beyond the 0xFF sentinel; m[20] = constant 5; m[21] ≈ data_size.
Update docs/scenario-format.md with full field tables for both chunks.
Closes#172. Scenario files in bin/scenarios/ can now be authored as
.json instead of .bin — the server compiles them to wire format on
load, falling back to .bin if no .json is present.
- Add ParseScenarioBinary / CompileScenarioJSON in scenario_json.go;
supports sub-header format (strings as UTF-8, metadata as base64),
inline format, and raw JKR blobs.
- Add PackSimple JKR type-3 (LZ77) compressor in jpk_compress.go,
ported from ReFrontier JPKEncodeLz.cs; round-trip tested against
UnpackSimple.
- Fix off-by-one in processDecode (jpk.go): last literal byte was
silently dropped for data that does not end on a back-reference.
- Wire loadScenarioBinary into handleMsgSysGetFile replacing the
inline os.ReadFile call; mirrors the existing loadQuestBinary pattern.
- Rewrite docs/scenario-format.md with full container/sub-header spec
and JSON schema examples.
The file documents project lineage and development phases, not just
a flat list of authors — HISTORY.md is a more accurate name and
follows common open-source convention.
nullcomp's passthrough path returns non-cmp-header data as-is without
error, which is correct for old uncompressed saves. However, a corrupt
backup slot containing garbage shorter than the minimum save layout
(100 bytes) would pass Decompress() and then panic in
updateStructWithSaveData() with a slice-bounds error at the name field
read (offset 88–100).
Add a minSaveSize check after each backup decompresses; skip the slot
if the result is too small. Also document the campaign system and the
fix in CHANGELOG under Unreleased.
Migration runner wraps each file in its own transaction via db.Begin().
Having an explicit BEGIN/END inside caused 'unexpected transaction status idle'.
Other migrations use no transaction wrapper.
Operators can now define Hunting Road configuration in a plain JSON file
(rengoku_data.json) instead of maintaining an opaque pre-encrypted binary.
The JSON is parsed, validated, assembled into the binary layout, and
ECD-encrypted at startup; rengoku_data.bin is still used as a fallback.
JSON schema covers both road modes (multi/solo) with typed floor and
spawn-table entries — floor number, spawn-table index, point multipliers,
and per-slot monster ID/variant/weighting fields. Out-of-range references
are caught at load time before any bytes are written.
Port ECD encryption/decryption from ReFrontier (C#) and FrontierTextHandler
(Python) into common/decryption. The cipher uses a 32-bit LCG key stream with
an 8-round Feistel-like nibble transformation and CFB chaining; all six key
sets are supported, key 4 being the default for all MHF files.
On startup, loadRengokuBinary now decrypts (ECD) and decompresses (JKR) the
binary to validate pointer bounds and entry counts, then logs a structured
summary (floor counts, spawn table counts, unique monster IDs). Failures are
non-fatal — the encrypted blob is still cached and served to clients unchanged,
preserving existing behaviour. Closes#173.
When primary savedata fails its SHA-256 integrity check, query
savedata_backups in recency order and return the first slot that
decompresses cleanly. Recovery is read-only — the next successful
Save() overwrites the primary with fresh data and a new hash,
self-healing the corruption transparently.
Closes#178
Adds human-readable JSON as an alternative quest format for bin/quests/.
The server tries .bin first (full backward compatibility), then falls back
to .json and compiles it to the MHF binary wire format on the fly.
JSON quests cover all documented fields: text (UTF-8 → Shift-JIS),
objectives (all 12 types), monster spawns, reward tables, supply box,
stages, rank requirements, variant flags, and forced equipment.
Also adds ParseQuestBinary for the reverse direction, enabling tools to
round-trip quests and verify that JSON-compiled output is bit-for-bit
identical to a hand-authored .bin for the same quest data.
49 tests: compiler, parser, 13 round-trip scenarios, and golden byte-
level assertions covering every section of the binary layout.
download-artifact picked up the Docker build provenance artifact
(Mezeporta~Erupe~39YWA8.dockerbuild) which fails to extract after
5 retries. Filter to only *-amd64 artifacts.
Promote [Unreleased] to [9.3.0] - 2026-03-19 in CHANGELOG and mark
the bookshelf save pointer fix as Done in improvements.md (it was
applied post-RC1 but the doc still said Pending).
MSG_MHF_GET_EXTRA_INFO (0xA6) and MSG_MHF_GET_COG_INFO (0xC3) had
Parse() returning NOT IMPLEMENTED. The dispatch loop treats any Parse
error as a hard drop — no ACK is ever sent, so the client waits
indefinitely and effectively soft-locks when entering the G-rank
Workshop or Master Felyne (Cog) screens.
Fix: parse AckHandle (the only field we can confirm from the protocol)
and respond with doAckBufFail so the client receives a well-formed
buf-type ACK with error code 1. The client's fail branch for these
requests exits cleanly without reading response fields, avoiding the
read-past-EOF crash that an empty success ACK would cause.
The full response format for both packets is still unknown; a complete
implementation requires further RE. The TODO comments mark the gap.
Fixes#180.
Add // stub: unimplemented to 70 empty game-feature handlers and
// stub: reserved to 56 protocol-reserved slots in handlers_reserve.go,
making them discoverable via grep. Add docs/unimplemented.md listing all
unimplemented handlers grouped by subsystem with descriptions.
Skip trials referencing monsters added after em106 (Odibatorasu, the
last F5 monster) and filter out item 7011 which does not exist before
G1, preventing client crashes on Forward.4/5 servers.
Also logs the pre-existing bookshelf save data pointer fix (already
applied during the savedata refactor) in the CHANGELOG.
All tracked items in both files have been resolved or accepted.
The content is superseded by CLAUDE.md architecture notes and commit
history; retaining fully-struck-through docs adds noise without value.
A second Ctrl+C/SIGINT while the 10-second shutdown countdown is
running now exits immediately instead of being silently dropped.
The signal channel is buffered to 2 so the second signal is captured,
and the countdown select exits via os.Exit(1) on receipt.
GetAllScores used SELECT * which broke when displayed_levels column was
added — now uses explicit column names. DisplayedAchievement handler
panicked on nil achievementService in empty-handler smoke tests — added
nil guard. Updated msg_build_test.go for renamed tactics point fields.
RE'd putAdd_ud_point (FUN_114fd490) and putAdd_ud_tactics_point
(FUN_114fe9c0) from the ZZ client DLL via Ghidra decompilation.
MsgMhfAddUdPoint fields: QuestPoints (sum of 11 category accumulators
earned per quest) and BonusPoints (kiju prayer song multiplier extra).
MsgMhfAddUdTacticsPoint fields: QuestID and TacticsPoints.
Adds diva_points table (migration 0009) for per-character per-event
point tracking, with UPSERT-based atomic accumulation in the handler.
RE'd putDisplayed_achievement from ZZ client DLL via Ghidra: the packet
sends opcode + 1 zero byte with no achievement ID, acting as a blanket
"I saw everything" signal.
Server changes:
- Track per-character last-displayed levels in new displayed_levels
column (migration 0008)
- GetAchievement compares current vs displayed levels per entry
- DisplayedAchievement snapshots current levels to clear notifications
- Repo, service, mock, and 3 new service tests
Protbot changes:
- New --action achievement: fetches achievements, shows rank-up markers,
sends DISPLAYED_ACHIEVEMENT, re-fetches to verify notifications clear
- Packet builders for GET/ADD/DISPLAYED_ACHIEVEMENT
Verify that SaveMercenary with rastaID=0 preserves both an existing
numeric rasta_id and a NULL rasta_id, preventing the silent data
corruption that broke game state saving.
Hardcoded counts (5) broke CI when migrations 0006 and 0007 were added.
Use readMigrations() to compute expected values so future migrations
don't require test updates.
Strengthen savedata persistence against corruption and race conditions:
- SHA-256 checksum: hash the decompressed blob on every save, store in
new savedata_hash column, verify on load to detect silent corruption.
Pre-existing characters with no hash are silently upgraded on next save.
- Atomic transactions: wrap character data + house data + hash + backup
into a single DB transaction via SaveCharacterDataAtomic, so a crash
mid-save never leaves partial state.
- Per-character save mutex: CharacterLocks (sync.Map of charID → Mutex)
serializes concurrent saves for the same character, preventing races
that could defeat corruption detection. Different characters remain
fully independent.
Migration 0008 adds the savedata_hash column to the characters table.
The G1-GG gacha code path (PR #150) included example data in comments
that used non-zero item_type/item_number/item_quantity on entry_type=100
rows. Users copying these values for ZZ servers caused client crashes
because the ZZ client interprets those fields as material requirements.
- Replace unclear PR #150 comment with explicit WARNING against using
G1-GG example values for ZZ servers
- Add gacha configuration guide header to GachaDemo.sql explaining the
3-table system, entry types, and item type codes
- Add Mega Potion example showing correct way to add custom items
- Reference Enumerations.md for the full item type list
Prevent savedata corruption and denial-of-service by adding four layers
of protection to the save pipeline:
- Bounded decompression (nullcomp.DecompressWithLimit): caps output size
to prevent OOM from crafted payloads that expand to exhaust memory
- Bounds-checked delta patching (deltacomp.ApplyDataDiffWithLimit):
validates offsets before writing, returns errors for negative offsets,
truncated patches, and oversized output; ApplyDataDiff now returns
original data on error instead of partial corruption
- Size limits on save handlers: rejects compressed payloads >512KB and
decompressed data >1MB before processing; applied to main savedata,
platedata, and platebox diff paths
- Rotating savedata backups: 3 slots per character with 30-minute
interval, snapshots the previous state before overwriting, backed by
new savedata_backups table (migration 0007)
New characters were missing their user_binary record, preventing them
from entering their house. Both sign server and API character creation
paths now insert the row. A backfill migration fixes existing databases.
The catch-up migration now creates the servers table if missing, like
sign_sessions. Startup cleanup queries in main.go use Exec instead of
MustExec so missing tables log warnings rather than panicking.
The catch-up migration assumed sign_sessions already existed, but
databases created from older schema dumps may not have this table.
Adding CREATE TABLE IF NOT EXISTS prevents the migration from failing
with "relation does not exist" on those databases.
Ghidra decompilation of mhfo-hd.dll shows all gacha packets use a
trailing uint8 for gacha type (0=normal, 1=stepup, 4=box). The server
ignores it since it derives the type from GachaID.
