Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Implement Gaussian Splats #1748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
azrogers wants to merge 32 commits into main
base: main
Choose a base branch
Loading
from gaussian-splats
Draft

Implement Gaussian Splats #1748

azrogers wants to merge 32 commits into main from gaussian-splats

Conversation

Copy link
Contributor

@azrogers azrogers commented Oct 14, 2025
edited
Loading

Closes #1670. This PR adds support for the KHR_gaussian_splatting extension in Cesium for Unreal (and, by virtue of CesiumGS/cesium-native#1262, SPZ as well). It does this using the Niagara particle system to handle rendering the splats from a custom NiagaraDataInterface. A subsystem is used to collect the splats into the singleton Niagara system.

This is very very experimental! Even when this gets merged I think we'll need to call it an experimental feature. But some things we should take care of before this makes it out of a draft PR:

  1. Sometimes the assertion ensure(GroupCount.X <= GRHIMaxDispatchThreadGroupsPerDimension.X); fails in RenderGraphUtils.h as a result of Niagara trying to generate too many sort keys at once. This is maybe an inevitable result of trying to pack so many splats into a single Niagara system, but packing them all into a single system is necessary so they sort together (else we'd get some awful looking seams between adjacent systems). I've implemented a really dirty system to try to mitigate this (trying to avoid resetting the system more often than every five ticks), but a better solution is almost certainly needed. Ideally we can find a way to avoid this assertion that doesn't involve splitting the scene into multiple Niagara systems.
  2. The Niagara system graph needs to be cleaned up. There's a lot of different code from a lot of different false-starts, rewrites, failed experiments, and other implementations I referenced (open-source implementations, license information for those implementations is included in the Niagara asset as a block comment). It would be good to streamline it into a finished implementation. Actually, it might be better just to do away with using the Niagara graph system entirely - just have the DataInterface provide a single function that takes in the index and matrices and returns all the splat information. That way we can have all of the compute shader code in the cpp file instead of putting it in a uasset that's opaque to git.
  3. Some splat tilesets aren't positioned correctly. Particularly, the Hines Rowing Center tileset from the Sandcastle hovers over the terrain. I had implemented a fix for this involving offsetting the position by the size of the bounding box, but I had to revert it because this example tileset from the other Sandcastle example is in the right position already, and would be offset into the wrong position. I'm not clear on what I'm missing here that has one working and one broken.
  4. Can the visual quality be improved? To me the sample datasets look blurry compared to the Sandcastle examples, but other splats loaded in other Gaussian Splatting projects in Unreal also look blurry. I don't know if we're just all doing it wrong or if this is a consequence of something Unreal is doing or what. It's also possible I'm just psyching myself out and it looks fine.
  5. Can the performance be improved? The shaders are very unoptimized. It's likely we could improve on some of them.
  6. (削除) I haven't yet tested it in a packaged build. It's likely the way I'm loading the Niagara system in the subsystem could break with a packaged build. (削除ここまで) ✅ Should be fixed in 1476ee2.
  7. The singleton actor holding the singleton Niagara system needs to be hidden in the editor. This shouldn't be difficult, I've just been skipping it for now because it's useful to be able to interact with the temp actor in the editor for debugging.
  8. This issue.
  9. (削除) Compute shader issue on Mac laptop. (削除ここまで) ✅ Fixed in 0695d2b.
  10. Floating point precision issues.

azrogers added 26 commits August 28, 2025 15:24
Copy link
Contributor

j9liu commented Oct 15, 2025

Someone actually tried packaging on this branch and is running into an issue, see #1749

Copy link
Contributor Author

A new issue to add to the list: when a Dynamic Pawn gets added to the level, the lighting on the splat goes haywire:
image

Copy link
Contributor Author

@david-lively reports this issue:

[UE] [2025年10月15日-16.55.01:627][ 59]LogCesium: Setting bounds: IsValid=true, Min=(X=357940124.922 Y=-120781445.651 Z=-123751071.001), Max=(X=357954552.308 Y=-120777170.055 Z=-123756954.477)
[UE] [2025年10月15日-16.55.01:627][ 59]LogCesium: Setting bounds: IsValid=true, Min=(X=357940124.922 Y=-120781445.651 Z=-123751071.001), Max=(X=357954552.308 Y=-120777170.055 Z=-123756954.477)
[UE] [2025年10月15日-16.55.01:627][ 59]LogCesium: Setting bounds: IsValid=true, Min=(X=357940124.922 Y=-120781445.651 Z=-123751071.001), Max=(X=357954552.308 Y=-120777170.055 Z=-123756954.477)
[UE] [2025年10月15日-16.55.01:627][ 59]LogCesium: Transform visible: false
validateComputeFunctionArguments:1175: failed assertion `Compute Function(Main_00005947_8a69e872): The pixel format (MTLPixelFormatR32Uint) of the texture (name:<null>) bound at index 5 is incompatible with the data type (MTLDataTypeInt) of the texture parameter (User_SplatInterface_SplatIndices [[texture(0)]]). MTLPixelFormatR32Uint is compatible with the data type(s) (
 uint
).'
validateComputeFunctionArguments:1175: failed assertion `Compute Function(Main_00005947_8a69e872): The pixel format (MTLPixelFormatR32Uint) of the texture (name:<null>) bound at index 5 is incompatible with the data type (MTLDataTypeInt) of the texture parameter (User_SplatInterface_SplatIndices [[texture(0)]]). MTLPixelFormatR32Uint is compatible with the data type(s) (
 uint
).'

Likely something to do with it running on the Apple GPU.

Copy link
Contributor Author

Gaussian Splats as depicted in the style of Tom Snyder:

squigglevision.mp4

Perhaps it goes without saying that there's some precision issues at play here. To some degree this problem is intractable with the requirements that a) everything happens on the GPU and b) everything must be sorted together. But there's definitely some things we can do to improve this I think. Probably if we rebased the Gaussian Splat Niagara system to be at the center of all currently loaded splats, it would help reduce precision issues instead of having it remain at (0, 0, 0) always.

Copy link
Member

kring commented Oct 17, 2025

@azrogers I've fixed the Android builds, though I'm not very satisfied with my understanding of what was going wrong. The fix was to remove the explicit reference to zlib::zlib in CesiumGltfReader's target_link_libraries. That was (apparently!) causing -isystem c:/android-ndk-r25b/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include to be added to the clang command-line in CesiumGltfReader, which was (apparently!) causing the wrong cmath or math.h or something to be included. Like I said, not an entirely satisfying explanation.

azrogers reacted with thumbs up emoji

Copy link

Particularly, the Hines Rowing Center tileset from the Sandcastle hovers over the terrain.

@azrogers, the Hines Rowing Center is using a WGS84 ellipsoid rather than Cesium World Terrain. Given this, it is known that it hovers over Cesium World Terrain. Here's a screenshot of it hovering in CesiumJS after the CWT terrain is selected.

image
azrogers reacted with thumbs up emoji

Copy link

weegeekps commented Oct 17, 2025
edited
Loading

Re: precision issues. Wow, yeah that's bad...

Probably if we rebased the Gaussian Splat Niagara system to be at the center of all currently loaded splats, it would help reduce precision issues instead of having it remain at (0, 0, 0) always.

This seems like the most reasonable solution given the situation.

EDIT: Although, this may be another instance where separate Niagara systems are warranted. That creates its own issues with sorting two splats on top of one another, but it might be an acceptable solution if precision continues to be an issue.

Copy link
Contributor Author

EDIT: Although, this may be another instance where separate Niagara systems are warranted. That creates its own issues with sorting two splats on top of one another, but it might be an acceptable solution if precision continues to be an issue.

We've been discussing this on the team. Splitting it up into multiple systems would cause some very obvious artifacts with two tilesets close by to each other, but should be fairly negligible with tilesets that are far away from each other - which is the most likely circumstance where the precision issues would manifest. The other circumstance would be if there are lots of tilesets, or a few very large tilesets, spanning a large distance - but you'd probably run into VRAM issues there before precision issues anyways.

Copy link

We've been discussing this on the team. Splitting it up into multiple systems would cause some very obvious artifacts with two tilesets close by to each other, but should be fairly negligible with tilesets that are far away from each other - which is the most likely circumstance where the precision issues would manifest. The other circumstance would be if there are lots of tilesets, or a few very large tilesets, spanning a large distance - but you'd probably run into VRAM issues there before precision issues anyways.

There are potentially changes that could be made on the tiler side to help with very large tilesets, allowing for the potential of better culling. It won't fix the precision issues, but it could help with any potential VRAM issues.

Let me know if you want to come to one of your stand ups and I'm willing to do so. Just need to know what day and the time.

Copy link

@weegeekps weegeekps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed verbally, this is looking good so far. My review for both this and the Cesium Native code has only been regarding implementation. On the Native side of things, the implementation is also looking good, but keep your eyes peeled for more changes regarding the rendering hints sometime soon.

The only major issue I've seen is that there appears to be some visual quality issues compared to CesiumJS. Per our discussion, it could be due to very minor differences in how we're doing the cut-off maths between the 3DGS shaders in CesiumJS and the shaders your PR.

azrogers reacted with thumbs up emoji
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@weegeekps weegeekps weegeekps left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support Gaussian splats glTF extension

AltStyle によって変換されたページ (->オリジナル) /