The Case of the Invisible Geometry: Debugging a Silent OpenGL Driver Validation Failure

How a samplerCube uniform guarded by if (false) made the entire scene vanish.

---

After a commit that added reflection probes and point shadow cubemaps to the main PBR shader, the editor scene view went black: no 3D geometry, no text on UI buttons, only particle emitters and UI button rectangles were visible. glDrawArrays executed for the right entities, the shader compiled without errors, and the framebuffer was configured correctly — yet every pixel was the exact clear color.

The root cause was a driver validation quirk: default.frag declared five samplerCube uniforms (a reflection probe and four point-shadow maps), but only bound real cubemap textures to their units when those features were actively used. When a scene had no reflection probe and no shadow-casting point lights, those texture units had no cubemap bound at all. The driver silently dropped every draw call against the main shader because the declared samplers had no valid backing texture — even though the code paths that would have actually sampled them were guarded by a uniform bool that was false.

The fix was to create a 1×1 black dummy cubemap at engine initialization and bind it to every samplerCube unit. Real cubemaps replace the binding when features are active.

---

Two screenshots from the editor told the same story:

1. Main menu scene: the UI canvas's button rectangles rendered in their button colors, but no text appeared on them. The 3D background props (BG Cube 1, BG Sphere, BG Teapot, etc.) listed in the hierarchy were completely invisible.
2. Castle throne room scene: a large 3D scene with 87 entities. Only the torch particle emitters were visible. No walls, no pillars, no floor, no text overlays.

The Tris: counter in the status bar read 0, but that turned out to be a red herring — the counter is only incremented inside Engine::Render(), which in editor mode just clears the back buffer to dark gray and hands off to ImGui. The actual scene rendering goes through Engine::RenderSceneToFBO(), which doesn't touch the counter. So Tris: 0 is expected in editor mode regardless of whether geometry is rendering.

The informative observation was what did render:

Everything that used mainShader was broken. Everything that used a different shader worked. That narrowed the problem to the shader program itself or to how it was being used.

---

Before we could see actual data, we burned through several plausible-sounding theories.

Hypothesis 1: mainShader failed to compile. Plausible because I had recently added an L2 spherical harmonics function and new uniforms for irradiance probes. The Shader class runs checkCompileErrors which prints to std::cerr. I redirected std::cout/std::cerr to a log file at startup (the engine links as WIN32 subsystem so stdout is discarded by default), and the log showed no shader errors. The shaders compiled and linked fine. Hypothesis ruled out.

Hypothesis 2: Alpha clipping. default.frag has an if (albedoAlpha.a < alphaThreshold) discard; test. If the texture format returned alpha = 0 for RGB textures in violation of the spec, every fragment would be discarded. I forced albedoAlpha.a = 1.0 at the top of the shader. No change. Ruled out.

Hypothesis 3: Depth test rejection. If the depth buffer was somehow pre-populated with 0.0, GL_LESS would reject every new fragment. Added a glDisable(GL_DEPTH_TEST) in the diagnostic frame and read the pre-draw depth buffer (came back as 1.0, correct). Ruled out.

Hypothesis 4: Lighting math producing NaN. If finalColor came out as NaN, some drivers drop the write. Added if (any(isnan(finalColor))) FragColor = vec4(1,0,1,1); magenta overrride — magenta never appeared, so no NaN. Ruled out.

Hypothesis 5: Viewport / matrix corruption. Logged camera position, view dimensions, and entity world matrices. All values were sensible: camera at (0, 2, 10), viewport 800×600 then resized to 1300×743, entity world matrices non-degenerate. Ruled out.

Each dead end was only dead in retrospect, but each one added diagnostic scaffolding that became load-bearing for the actual diagnosis.

---

The first practical blocker was simply seeing anything. Norlong is linked as a Windows GUI app (add_executable(Norlong WIN32 ...)), which means there's no inherited console. Any std::cout goes nowhere by default, and capturing it with Start-Process -RedirectStandardOutput didn't produce output either.

The fix was one of the smallest patches of this debug session:

EXPANDCOPY
cpp
// main.cpp
static std::ofstream engineLog("engine.log", std::ios::trunc);
std::cout.rdbuf(engineLog.rdbuf());
std::cerr.rdbuf(engineLog.rdbuf());

This redirects the standard streams into a file on disk. After rebuild, the engine wrote a readable log that confirmed shader compilation succeeded, scene loaded, and the main loop was running. That gave us a signal channel for everything that followed.

---

The key insight for actually finding the bug was to treat the fragment shader and framebuffer as an instrumentation surface. Two kinds of probes, composed:

CPU-side, per-frame: inside RenderSceneToFBO, before returning, blit the framebuffer color attachment back to CPU memory with glReadPixels and scan it for non-clear-color pixels. Log interesting aggregate stats (min/max R/G/B values, count of non-sky pixels, sample RGB triples) and a handful of OpenGL state variables (GL_DEPTH_WRITEMASK, GL_COLOR_WRITEMASK, GL_DEPTH_FUNC, current draw buffers):

EXPANDCOPY
cpp
std::vector<unsigned char> buf(W * H * 3);
glBindFramebuffer(GL_READ_FRAMEBUFFER, sceneViewFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0);
glReadPixels(0, 0, W, H, GL_RGB, GL_UNSIGNED_BYTE, buf.data());

int maxR = 0, diffFromSky = 0;
for (int i = 0; i < W * H; ++i) {
    int r = buf[i*3], g = buf[i*3+1], b = buf[i*3+2];
    if (r > maxR) maxR = r;
    if (r != 10 || g != 10 || b != 25) diffFromSky++;
}

That last check is the one that unlocked everything. The scene's clear color was (0.04, 0.04, 0.1) in linear space, which becomes (10, 10, 25) when written to an 8-bit framebuffer. Counting pixels that differ from exactly that value — not just pixels that are "bright" — distinguishes three very different failure modes:

• If fragments write (0,0,0) the count is high (the pixels are black, not sky).
• If fragments write finalColor that happens to be very small positive, the count is also nonzero.
• If fragments are never written at all, the count is 0 because every pixel is still the cleared value.

The diagnostic said diffFromSky = 0: every pixel was exactly the clear color. That is not "dark lighting". That is "no write happened". Fragments were either discarded via discard or never reached the output stage.

Shader-side, per-fragment: to distinguish which intermediate computation was breaking the pipeline, I temporarily replaced the final FragColor = vec4(finalColor, alpha) write with diagnostic variants and ran each one:

The critical clue: when the shader's output only depended on values the optimizer could compute without touching samplers, pixels appeared. When it depended on values that required the lighting path to run (which keeps the sampler-access code alive from a dead-code-elimination perspective), pixels did not appear.

That pattern — "the shader runs, writes sometimes work, and whether they work correlates with whether certain samplers are live" — is the fingerprint of a sampler-binding validation issue.

---

To isolate which sampler was the culprit, I checked out default.frag from commit 949d50b — a version from before the MRT/normal-output/reflection-probe refactor. It has a single out vec4 FragColor and no samplerCube reflectionProbe. With that shader, the scene rendered. Dim, because the ambient is low in the main menu, but rendered.

From there I ran a careful incremental merge: take the old working shader and add pieces of the new shader one at a time, re-running after each edit:

1. Add layout(location = 1) out vec4 NormalOutput; and write to it at the end → still works.

2. Add the samplerCube reflectionProbe uniform declaration (but no usage in main()) → still works.

3. Add the irradiance-probe SH uniforms (uniform vec3 shCoeffs[9], etc.) → still works.

4. Add the reflection-probe sampling block:

   EXPANDCOPY
   glsl
   if (hasReflectionProbe && metallic > 0.01) {
       ...
       vec3 envColor = textureLod(reflectionProbe, R, mip).rgb;
       Lo += envColor * F0 * ...;
   }

   → renders go black.

Commenting just that block back out restored rendering. The textureLod(reflectionProbe, ...) call was the discriminating line.

Crucially, hasReflectionProbe was false in this scene. The branch body never executed at runtime. But removing the code inside the branch fixed the problem. That could only mean one thing: the driver wasn't checking whether the sampler was accessed, it was checking whether it was declared and had a valid texture bound to its assigned unit.

---

default.frag declares several samplerCube uniforms:

EXPANDCOPY
glsl
uniform samplerCube pointShadowMap0;
uniform samplerCube pointShadowMap1;
uniform samplerCube pointShadowMap2;
uniform samplerCube pointShadowMap3;
uniform samplerCube reflectionProbe;

At engine initialization, the C++ side assigns these samplers to dedicated texture units:

EXPANDCOPY
cpp
mainShader->setInt("pointShadowMap0", 7);
mainShader->setInt("pointShadowMap1", 8);
mainShader->setInt("pointShadowMap2", 9);
mainShader->setInt("pointShadowMap3", 10);
// reflectionProbe was implicitly unit 15, set only inside the baking code

During normal scene rendering — say, a main menu with no active point shadows and no reflection probe — nothing ever binds a cubemap to units 7, 8, 9, 10, or 15. The C++ code only touches those units when a real point shadow or reflection probe is being rendered.

According to the OpenGL 3.3 core specification, this is not supposed to matter. Section 2.11.7 ("Shader Execution → Texture Access") defines incomplete-texture results: if no texture is bound to the unit a sampler reads from, the sampled value is (0, 0, 0, 1). Spec-compliant drivers handle this by returning that default value.

But some drivers — typically Windows OpenGL drivers bundled with certain GPU stacks — run a stricter validation pass at draw time: every samplerCube declared in the linked program must have a cubemap texture bound to its assigned unit, even if the shader's control flow would never actually sample it. When that validation fails, the draw is silently dropped. No GL_INVALID_OPERATION, no error callback — just no pixels.

That matches the symptoms exactly:

• The fragment shader executed as far as the GPU compiler could see it would (the CPU-side flag _dbgDrawn counted 4 draws per frame).
• Writes that used values the compiler could prove were independent of the guarded sampler access (constants, alpha, albedoAlpha) produced visible pixels, because the optimizer collapsed the shader to a short path that didn't keep the samplers live.
• Writes that depended on Lo or finalColor — which required the lighting loop, which contains the reflection-probe block — kept the sampler references live, triggered the draw-time validation, and resulted in a silently-dropped draw.

The old single-output shader worked coincidentally: it had the same samplerCube pointShadowMap0..3 declarations, but for some scenes those were bound by the point-shadow rendering pass earlier in the same frame, and the bindings persisted across state changes because nothing explicitly rebound unit 7-10 to anything else. The reflection-probe sampler was added after the old shader, so the coincidence broke.

This is exactly the class of bug that is very hard to trigger in CI or in validation layers for OpenGL, because the spec says it shouldn't happen. But it reliably happens on some production drivers, and you won't see a diagnostic message — just an empty frame.

---

Create a 1×1 black cubemap at engine startup and bind it to every samplerCube texture unit. Real cubemaps replace the binding when a feature needs them and fall back to the dummy otherwise:

EXPANDCOPY
cpp
glGenTextures(1, &dummyCubemap);
glBindTexture(GL_TEXTURE_CUBE_MAP, dummyCubemap);
unsigned char black[4] = {0, 0, 0, 255};
for (int face = 0; face < 6; ++face)
    glTexImage2D(GL_TEXTURE_CUBE_MAP_POSITIVE_X + face, 0, GL_RGBA, 1, 1, 0,
                 GL_RGBA, GL_UNSIGNED_BYTE, black);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_CUBE_MAP, GL_TEXTURE_WRAP_R, GL_CLAMP_TO_EDGE);

for (int unit : {7, 8, 9, 10, 15}) {
    glActiveTexture(GL_TEXTURE0 + unit);
    glBindTexture(GL_TEXTURE_CUBE_MAP, dummyCubemap);
}
glActiveTexture(GL_TEXTURE0);

Six glTexImage2D calls, five texture-unit bindings, one glGenTextures. Zero runtime cost in the draw path. The hasReflectionProbe uniform still gates the actual textureLod at runtime, so the dummy cubemap is only ever sampled when the shader would have sampled it anyway (which it doesn't, in scenes without probes), and when it is sampled, it contributes (0,0,0) which is the spec-correct default for an incomplete texture.

---

Always bind something to every sampler unit you declare. Relying on uniform-bool guards to prevent sampler access is legal by the spec but not portable in practice. Create a dummy texture per sampler type (sampler2D, samplerCube, sampler2DArray, whatever you use) and use it as a default binding. It costs nothing.

Use the framebuffer as a debugger. When output looks wrong, the fastest diagnostic is to temporarily replace the final color with something that isolates one intermediate value. Constants rule out the whole pipeline. Individual uniforms or samplers rule in/out specific inputs. Scaled intermediates reveal magnitude problems. Count pixels that differ from the clear color rather than pixels that are "visible" — black pixels are a valid output, and treating them as invisible hides information.

Trust bisecting more than reasoning. Several hypotheses above were sensible and ruled out quickly. The actual bug — a sampler validation pass that isn't in the spec — would not have been on any list of hypotheses to check. But a clean "works vs. doesn't work" bisect on the shader source converged on it in a few edits.

Redirect stdout/stderr to a log file in GUI apps from day one. Shader compile errors, GL debug messages, any std::cerr you write for diagnostics — none of them are visible in a WIN32-subsystem executable without either a console allocation or a file redirect. A three-line addition to main() would have surfaced all of this ten minutes earlier.

If you can, turn on GL_KHR_debug. Register a debug callback with glDebugMessageCallback in debug builds and log every error, performance warning, and undefined-behavior report the driver is willing to give you. The draw wasn't producing a GL_INVALID_OPERATION, so even this wouldn't have flagged the exact bug here, but in the general case it converts silent driver quirks into explicit log lines and is free instrumentation.

---

Commit: 2b5def6 of norlong — "Fix missing text/geometry in editor scene view".