-
Notifications
You must be signed in to change notification settings - Fork 390
-
Background
I have been gently exploring how hard it would be to split the game into multiple .pck files. There are a couple of reasons why one might want to do this:
Game size
The .pck file has ballooned to 93.8 MB. This is a practical issue because we use GitHub Pages for test builds, and GitHub Pages has a 1 GB soft limit on the size of the site, i.e. around 10 branches. I implemented deduplication for the Godot binary itself but this has relatively little impact.
The vast majority of the 93.8 MB is music. This makes sense because the pixel-art style compresses well, and scene files & code are small and compress well. But a typical bitrate for compressed music is 128 kbps, i.e. 16 kilobytes per second, and any given background music loop might be, say, 2 minutes long, which comes out to 1.83 MiB. But these music files change very rarely. So hypothetically we could split all .ogg and .mp3 resources to music.pck, and then deduplicate that file in the same way. This would allow us to have more simultaneous test builds published before hitting the size limit.
StoryQuests as DLC
Another way of looking at the size of the game is that around 2/3 comes from StoryQuests. Again, this is mostly music! As the number of StoryQuests in the tree increases, the size will steadily grow.
An alternative model would be to publish each StoryQuest as a separate .pck file, which would be downloaded on demand based on what the player chooses to play. We can't really version StoryQuests separately to the host game because the internal APIs of the game are not stable, but it's interesting to explore what it would look like anyway.
Exporting multiple .pcks at once
Godot has some documentation on this: https://docs.godotengine.org/en/stable/tutorials/export/exporting_pcks.html. However I have to say it's not that clear how to use it in our case. I would have expected to be able to define in (say) the Web export preset that the preset should export multiple .pck files, sliced in a defined way. However it appears that each export preset only produces one .pck file. So instead you are supposed to have multiple presets, for each platform ×ばつ slice of the game.
As far as I can tell this breaks the remote-deploy workflow, which I use for testing the game in a web browser. For that, you declare that the Web preset is runnable, and then you can click a button to export the web build, run a local web server, and open it in your browser. Super useful! But if additional presets need to be exported to have the full game, this can't work.
Maybe it's possible to write a plugin that hooks the export flow and runs additional exports as part of the process? Some automation would certainly be necessary to split 10+ StoryQuests out into 10+ .pck files because we can't manually manage this, especially as the number grows.
Also, if you define a preset that exports only certain resources, their dependencies are also included, meaning that each StoryQuest .pck would also include whatever elements of the base game are used. I think the workaround here would be to build each StoryQuest as a patch, where you tell Godot what other .pck files are guaranteed to have been loaded, and then the new .pck only contains changed files.
Loading extra .pcks on the web
The documentation shows how to load additional .pcks in GDScript. Fine. On the web there is the additional complication that files need to be downloaded on the browser side, and loaded into the fake filesystem that the WASM Godot binary can see.
I don't think this is especially difficult. After modifying the HTML shell a little to drive the "download and launch" API a bit more directly (which was necessary to add a click-to-play screen #2060) then downloading an additional .pck that will always be needed boils down to:
diff --git a/web/full-size.html b/web/full-size.html
index 0e184f9a7..56c97af12 100644
--- a/web/full-size.html
+++ b/web/full-size.html
@@ -237,6 +237,7 @@ const engine = new Engine(GODOT_CONFIG);
Promise.all([
engine.init(exe),
engine.preloadFile(pack, pack),
+ engine.preloadFile('music.pck', 'music.pck'),
]).then(() => {
return new Promise((resolve) => {
setStatusMode('ready');
and if one is needed dynamically, one can call JavaScript code from GDScript (we already do to use the URL hash as a scene switcher) so one could do that on-demand.
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 6 comments 6 replies
-
For now I will continue to play whack-a-mole with large music files:
When I have done this before, I have found unattributed third-party assets, so it is worth doing anyway!
Beta Was this translation helpful? Give feedback.
All reactions
-
Great research @wjt . So the conclusion for now is that it doesn't make a difference to split the music in its own .pck file? How far have you got with splitting each StoryQuest as a .pck DLC or patch? I personally like that idea the most, as it would allow growing the content created by learners. This kind of split by quest is something @JuanFdS recommended me at one point.
Beta Was this translation helpful? Give feedback.
All reactions
-
I will take another look at that when time permits. I think the way that would work best is to build each StoryQuest as a patch .pck. I'm not sure how to do it in a scalable way with 10+ quests in the tree but I guess it can be scripted.
The other annoying thing is that the built-in "exclude resources" feature works by listing out every file in the export config. So you can't say "Exclude everything in res://scenes/quests/story_quests", you (or rather the editor) recursively scans that folder and puts everything into the export preset. I think to do this properly will need an export plugin.
Beta Was this translation helpful? Give feedback.
All reactions
-
It would make a difference to the size of the web build to split the music into a separate pck file and deduplicate it. The problem is that I don't know how to do it in a clean way - if you export a scene that references a music file, then Godot follows that reference. AFAICT we would have to consider the game to be a "patch" on a pck of music.
Beta Was this translation helpful? Give feedback.
All reactions
-
I think the workaround here would be to build each StoryQuest as a patch, where you tell Godot what other .pck files are guaranteed to have been loaded, and then the new .pck only contains changed files.
My first attempt here was to define 2 new presets:
- "Web minus StoryQuests", a copy of the current web preset with these changes:
a. "Export all resources in the project except resources checked below" option set, and all resources in thestory_questsfolder selected - "Web only StoryQuests", a copy of the current web preset with these changes:
a. Target path changed tobuild/web/storyquests.html
b. "Export selected resources (and dependencies)" with all files in thestory_questsfolder checked (as above)
c. Under "Patching", the "Base Packs" option set tobuild/web/index.pck
I then exported the whole "Web minus StoryQuests" export, and just the pck file for "Web only StoryQuests".
I patched the web HTML shell to preload storyquests.pck in addition to index.pck:
diff --git a/web/full-size.html b/web/full-size.html index 0e184f9a..aa5f5bd9 100644 --- a/web/full-size.html +++ b/web/full-size.html @@ -237,6 +237,7 @@ const engine = new Engine(GODOT_CONFIG); Promise.all([ engine.init(exe), engine.preloadFile(pack, pack), + engine.preloadFile('storyquests.pck', 'storyquests.pck'), ]).then(() => { return new Promise((resolve) => { setStatusMode('ready');
and added the following in SceneSwitcher._ready (just a convenient place to run code early in the project startup):
var success := ProjectSettings.load_resource_pack(OS.get_executable_path().get_base_dir().path_join("storyquests.pck")) prints("hello", success)
The resulting storyquests.pck contained many .@@removal@@ files, which mean "this patch should delete this file". Specifically it deletes every other scene in the game, e.g. scenes/menus/title/title_screen.tscn.remap.@@removal@@, and so of course the game fails to load:
Path world_map/frays_end.tscn from URL hash #world_map/frays_end is not a scene; ignoring
index.js:467 ERROR: Cannot open file 'res://scenes/menus/title/title_screen.tscn'.
GDScript backtrace (most recent call first):
index.js:467 [0] change_to_file_with_transition (res://scenes/globals/scene_switcher/scene_switcher.gd:113)
index.js:467 [1] switch_to_intro (res://scenes/menus/splash/components/splash.gd:28)
This isn't what I expected but it does make sense. I told Godot that the project now only contains the scenes & resources in res://scenes/quests/story_quests, and that it is generating a patch from a build that contained many other scenes, so my instruction was indeed to delete these.
Next stop: skip the patch stuff, just export the story_quest folder & see how many duplicates are in there.
Beta Was this translation helpful? Give feedback.
All reactions
-
Turning off the patch setting did what I expected: it exported everything in story_quests, plus other files from elsewhere in the tree that are referred to by StoryQuests, including (among other things) assets, player scripts, etc. This is as expected.
Turning back on the patch setting and setting the export filter to "everything" does a better job. The resulting storyquests.pck file only contains stuff in scenes/quests/story_quests. This is 11 MB smaller - not surprising because the shared assets include our first-party music.
When loading a pack with ProjectSettings.load_resource_pack(), by default any paths in the new pack which are already in the resource filesystem are replaced. Setting the second argument to false disables this, which makes loading these two variants of the storyquests.pck equivalent.
So far in this experiment the storyquest.pck file is eagerly loaded, with engine.preloadFile() in the HTML shell. The next step is to try loading the StoryQuest bundle dynamically.
Beta Was this translation helpful? Give feedback.
All reactions
-
Simple enough:
diff --git a/scenes/globals/scene_switcher/scene_switcher.gd b/scenes/globals/scene_switcher/scene_switcher.gd
index 906e0fc7..d1535740 100644
--- a/scenes/globals/scene_switcher/scene_switcher.gd
+++ b/scenes/globals/scene_switcher/scene_switcher.gd
@@ -29,6 +29,29 @@ var _scene_rx := RegEx.create_from_string(
)
+func download_story_quests() -> void:
+ prints(Time.get_datetime_string_from_system(), "Fetching StoryQuest pack")
+ var request := HTTPRequest.new()
+ add_child(request)
+
+ request.download_file = OS.get_executable_path().get_base_dir().path_join("storyquests.pck")
+ request.download_chunk_size = 1024 * 1024
+ request.request_completed.connect(_on_request_completed.bind(request))
+
+ var error := request.request("http://localhost:8000/storyquests.pck")
+ if error != OK:
+ push_error("Oh no %s" % [error_string(error)])
+
+func _on_request_completed(result: int, response_code: int, headers: PackedStringArray, body: PackedByteArray, request: HTTPRequest) -> void:
+ prints(Time.get_datetime_string_from_system(), "Got response", result, response_code, headers, body.size())
+
+ var success := ProjectSettings.load_resource_pack(request.download_file, false)
+ prints("StoryQuest pack loaded", "successfully" if success else "unsuccessfully")
+
+ request.queue_free()
+
+
+
func _ready() -> void:
if OS.has_feature("web"):
_window = JavaScriptBridge.get_interface("window")
@@ -36,6 +59,9 @@ func _ready() -> void:
_on_hash_changed_ref = JavaScriptBridge.create_callback(_on_hash_changed)
_window.onhashchange = _on_hash_changed_ref
+ download_story_quests()
+
+
## On the web, load the world indicated by the URL hash, if any.
func _restore_from_hash() -> void:
This works. It's extremely slow, taking 7 seconds between "Fetching StoryQuest pack" and "Got response". The actual HTTP request from localhost takes under 100 ms (unsurprisingly) so it must all be in the cost of marshalling all that data across the WASM boundary. Running the same code in a desktop build is fast, and preloading pcks before starting the engine is also fast, so that supports this theory and tells us it is solvable. The JavaScript engine.preloadFile() method cannot be used after the engine is started but I see no reason why we could not do what it + engine.start() do internally:
for (const file of preloader.preloadedFiles) { me.rtenv['copyToFS'](file.path, file.buffer); }
So I won't bother going any further with optimising this. We can see that this can be done.
Beta Was this translation helpful? Give feedback.
All reactions
-
Oh in fact there is a documented engine.copyToFS method.
Beta Was this translation helpful? Give feedback.
All reactions
-
Zooming out a little bit... Let's ignore for now the GitHub Pages site size limit (this could be worked around by e.g. switching to a different hosting provider; this is a question of migration work + money spent rather than a fundamental limitation) and focus on the DLC StoryQuest piece.
From that perspective I don't think there is any advantage in splitting up the game as built from this repo into smaller pcks: it adds complexity and I don't see any reward for doing so.
It's more interesting when we think about out-of-tree StoryQuests. You could imagine having a way for a team to publish a .pck of their StoryQuest to a server somewhere, and then our upstream build of the game would be able to fetch and run quests from that server, removing the need to go through the pull-request flow at all. (Ignore for now the moderation challenges inherent in this user-generated content model.)
The big problem there is that StoryQuests are tightly coupled to the surrounding game. We saw this in #2032 most recently - every StoryQuest needed to be updated for the changes in the player scene. So any StoryQuest .pck file built prior to that change would fail to load in the current main branch. You could imagine adding version checking: we would introduce semantic versioning, bumping the major version whenever we break internal API, and refuse to load any StoryQuest targetting an older major version of the game. Unfortunately in practice we break the internal API every few weeks (in a small or large way) and so the effect would be that no out-of-tree quest could ever be loaded. This approach would work later in development when the game's internal API surface area has stabilised, but not now.
A different approach: add an export preset that only exports a single quest, plus its dependencies. Add an export plugin that changes the main scene to the first scene of that quest. Assume this can be done. (I quickly tried this and "Export selected resources (and dependencies)" is not sufficient, e.g. having ticked all of eldrune's resources it exports addons/dialogue_manager/dialogue_manager.gd but none of the other components of dialogue_manager so it doesn't work at all...)
Then the game would launch a quest by suspending itself and running a new instance of Godot with the StoryQuest's .pck as the main pack. When the quest is complete, the new instance of Godot quits, and the original instance picks back up, returning the player to Fray's End. I think this could be done. But I'm not entirely convinced it is worth prototyping. A much simpler version would be to load a JSON file from some server that gives URLs to StoryQuests' web builds, show them in the storybook, and tell the browser to switch to that URL when the player picks one.
Beta Was this translation helpful? Give feedback.
All reactions
-
Switching gear back to the size angle... From this angle, the point of splitting the build artifact in this repo up into 1 pck for the main game and 1 (or more!) pcks for StoryQuests is not to reduce the amount of data downloaded by players. The point is that, if the storyquests.pck file does not change between successive builds, then we could teach amalgamate-pages to deduplicate it between branches in the same way it now does for index.wasm, reducing the size of the GitHub Pages artifact, to keep us below the 1GB soft limit.
Alternatively you could imagine putting all the large music files into a music.pck file.
For this deduplication to work, the split-off .pck file would have to be byte-for-byte identical between successive builds of the project (unless the StoryQuests or music, respectively, have changed of course).
Any (non-patch) .pck you export contains project.binary (the binary representation of project.godot) at its root. We change this file during export to contain the output of git describe. Necessarily this changes each time you make a new commit. No matter for StoryQuests - we can add a custom feature flag like no_git_describe to the export preset, and change the plugin to not add this to project.godot if that flag is set. (Or export SQs as a patch -- because the two pcks would be built from the same tree, then the edits to the project settings would of course be the same in the two exports, so there is no diff, so there is no need to store a modified version in the patch pck.)
Only the project.binary from the main .pck is used. Good!
Then the question is: is Godot's export pipeline deterministic. Unfortunately the answer is no. I made 2 fresh checkouts of exactly the same commit (on my test branch) and opened each copy in the editor and a StoryQuest-only .pck. (Actually I exported a .zip file instead because then I can use diffoscope to compare them.) And there are in fact very slight differences in a few of the exported .scn (binary scene) files. I used Godot RE Tools to convert one such binary scene file back to plain text. The diff is in the order of the signal connections:
--- threadbare_tmp2/tmp.tscn 2026年04月01日 13:18:33.261179188 +0100 +++ threadbare_tmp/tmp.tscn 2026年04月01日 13:18:10.924901926 +0100 @@ -488,11 +488,11 @@ [node name="FireFly26" parent="." unique_id=1657380025 instance=ExtResource("5")] position = Vector2(963, 2556) -[connection signal="dialogo_terminado" from="quest_arbol/ArbolFantasma1" to="." method="reportar_arbol_contactado"] -[connection signal="dialogo_terminado" from="quest_arbol/ArbolFantasma1" to="." method="_on_arbol_fantasma_1_dialogo_terminado"] [connection signal="dialogo_terminado" from="quest_arbol/ArbolFantasma1" to="." method="_on_arbol_dialogo_terminado"] -[connection signal="dialogo_terminado2" from="quest_arbol2/ArbolFantasma2" to="quest_arbol2" method="_on_arbol_fantasma_2_dialogo_terminado_2"] +[connection signal="dialogo_terminado" from="quest_arbol/ArbolFantasma1" to="." method="_on_arbol_fantasma_1_dialogo_terminado"] +[connection signal="dialogo_terminado" from="quest_arbol/ArbolFantasma1" to="." method="reportar_arbol_contactado"] [connection signal="dialogo_terminado2" from="quest_arbol2/ArbolFantasma2" to="quest_arbol2" method="reportar_arbol_contactado2"] +[connection signal="dialogo_terminado2" from="quest_arbol2/ArbolFantasma2" to="quest_arbol2" method="_on_arbol_fantasma_2_dialogo_terminado_2"] [connection signal="dialogo_terminado3" from="quest_arbol3/ArbolFantasma3" to="quest_arbol3" method="reportar_arbol_contactado3"] [connection signal="dialogo_terminado3" from="quest_arbol3/ArbolFantasma3" to="quest_arbol3" method="_on_arbol_fantasma_3_dialogo_terminado_3"] [connection signal="body_entered" from="Area2D" to="Area2D" method="_on_Area2D_body_entered"]
At this point I'm going to give up on this angle of trying to deduplicate unmodified stuff between different branches by splitting up the game. A more fruitful angle might be to say: when building a PR, if the PR does not modify any files in scenes/quests/story_quests, use an alternative export preset that excludes those files from the build. We would only do this on the upstream repo.
Beta Was this translation helpful? Give feedback.
All reactions
-
Another size limit that may at some point become relevant: itch.io. A web game there has these limits:
- No single file can be more than 200 MB
- The whole project must be smaller than 500 MB
The approach of splitting out large music etc. to a separate pck seems relatively common to work around that first limit.
https://github.com/Motto73/TUASGuessr was an example I found where someone has split the game into 4 packs: 2 asset packs, one dlc pack, and the main game. The main game is a patch on top of the other three packs. They then have a custom HTML shell which hardcodes the approximate file sizes of each pck file:
And preloads each of the split packs into the virtual filesystem:
Then they have an autoload that loads the external packs:
I guess they carefully ensure that no assets from those packs are needed until that autoload has become _ready.
There seems no built-in way to automate the "export the project as multiple pcks" workflow. You have to manually, or in an external script, export the components of the project in the correct order. There's also no great way to add the file sizes for the split-off pcks dynamically in the HTML shell; or indeed to set some flag to tell the JavaScript code whether or not it needs to load the extra .pcks. You could automate this with external scripts, but I think this is excessive complexity when we depend so heavily on web builds for our education flows.
Beta Was this translation helpful? Give feedback.
All reactions
-
Sorry in advance for the side comment. I don't want to ruin this thread, which contains a lot of interesting facts.
On the StoryQuest submission (or cherry-picking) to upstream, I think that we should start considering the size of assets. In particular for the ones that weren't created by learners but are third-party free-licensed assets (which is the case of all music in existing SQs). Giving more priority to original content. I think that it can be a good learning in working with limits / constrains.
Beta Was this translation helpful? Give feedback.