Windows DirectComposition — Kernel-Mode Attack Surface Analysis

Abstract

This post documents the reconnaissance and static analysis phase of an attack surface investigation into the Windows DirectComposition (DComp) subsystem. The work was performed against a live Windows 10 x64 host (Build 19045.6466) as preparation for kernel-mode fuzzing of win32kbase.sys.

The goal was clear: extract System Service Numbers (SSNs) for the DComp channel syscalls, trace the full user-mode commit pipeline from application to kernel boundary, and understand how the kernel deserializes and dispatches command batches on the other side of that boundary. No exploitable vulnerability was found during this engagement — but the structural knowledge gained is a direct prerequisite for any meaningful fuzzing campaign against this subsystem.

Three primary activities are covered:

• Enumeration of NtDComposition* symbols from win32u.dll using WinDbg and extraction of exact SSN values from disassembled syscall stubs
• Static reverse-engineering of dcomp.dll via Ghidra tracing the full user-mode commit pipeline through three distinct phases
• Kernel-side analysis of NtDCompositionCommitChannel in win32kbase.sys — pointer probing, handle resolution, and batch dispatch logic

Methodology

The investigation was conducted in two parallel tracks. The first was dynamic analysis — attaching WinDbg to a live user-mode process and querying the win32u.dll export table. Since win32u.dll is the thin user-mode wrapper for all Win32k syscalls, its stubs contain the literal SSN values loaded into EAX immediately before the syscall instruction fires. Disassembling those stubs gives us the exact numbers needed to invoke the kernel directly.

The second track was static analysis via Ghidra. dcomp.dll was loaded with its PDB and the decompiler was used to trace the commit pipeline from the high-level CDevice::Commit entry point — the function called by every DirectComposition application to flush pending visual changes — all the way down to the NtDCompositionCommitChannel syscall stub. The kernel binary win32kbase.sys was then loaded separately to reverse the Ring 0 handler.

Syscall Enumeration — NtDComposition*

The x (Examine) command in WinDbg queries exported symbols by wildcard. Pointing it at win32u!NtDComposition* reveals every DComp syscall stub in the module — and there are a lot of them, confirming that the DComp attack surface is wide:

0:000> x win32u!NtDComposition*

From the full list, two functions were selected as primary targets based on their role in the commit pipeline — the function that initializes a channel and the function that sends data through it.

3.1 — NtDCompositionCommitChannel · SSN 0x111F

The syscall stub follows the standard x64 Windows convention. R10 preserves RCX because the syscall instruction will clobber it. The value loaded into EAX is the System Service Number — this is what we extract:

win32u!NtDCompositionCommitChannel:
00007ffa`71ff3410  4c8bd1           mov     r10,rcx
00007ffa`71ff3413  b81f110000       mov     eax,111Fh       ; ← SSN extracted here
00007ffa`71ff3418  f604250803fe7f01 test    byte ptr [SharedUserData+0x308],1
00007ffa`71ff3422  0f05             syscall
00007ffa`71ff3424  c3               ret

3.2 — NtDCompositionCreateChannel · SSN 0x1124

win32u!NtDCompositionCreateChannel:
00007ffa`71ff34b0  4c8bd1           mov     r10,rcx
00007ffa`71ff34b3  b824110000       mov     eax,1124h       ; ← SSN extracted here
00007ffa`71ff34b8  f604250803fe7f01 test    byte ptr [SharedUserData+0x308],1
00007ffa`71ff34c2  0f05             syscall
00007ffa`71ff34c4  c3               ret

3.3 — SSN Summary & Technical Observations

mov r10, rcx — Mandatory x64 Syscall Convention. This is an architectural requirement, not a DComp quirk. The syscall instruction overwrites RCX with the return address (for sysret), so the first argument — which the calling convention places in RCX — must be saved to R10 first. Every Win32k syscall stub begins this way.

Win32k SSN Range. Standard NT Executive syscalls occupy the 0x0000–0x01FF range. These IDs are in the 0x11xx block, confirming they route through the Win32k graphical subsystem dispatcher rather than the core NT executive. This is a different dispatch table, different filtering mechanisms, and a historically more vulnerable attack surface.

Practical note: The 0x11xx Win32k range is significant for EDR bypass research. Many kernel callback filters are registered for NT syscalls but have weaker coverage of Win32k-specific syscalls. Direct invocation at these SSNs can bypass some user-mode hook chains entirely.

Static Analysis — dcomp.dll Execution Flow

With the syscall targets identified, the next step was understanding the full call chain inside dcomp.dll. Every DirectComposition application calls IDCompositionDevice::Commit() to apply pending visual changes — but COM interfaces are thin wrappers. The real work happens in three phases inside the implementation:

Phase 1 — Entry Point: CDevice::Commit

CDevice::Commit is the transactional flush entry point. Before any kernel work happens, it performs several housekeeping operations:

• Synchronization — Acquires either a CRITICAL_SECTION via EnterCriticalSection or a vtable-dispatched lock at this+0x08, depending on whether a custom lock object exists at this+0x48.
• Dirty-object rebuild — Iterates a singly-linked list at this+0x1c8, calling CRebuildableObject::Rebuild on each pending visual update before committing.
• Virtual surface optimization — If g_VSAllowOptimizer is set, invokes CVirtualSurfaceOptimizer::OptimizeVirtualSurface to prune surfaces idle longer than 999ms and reclaim memory.
• Back-pressure handling — If the kernel batch queue is saturated, spins on CommitToKernel + WaitForCommitCompletionInternal until the queue drains.
• Telemetry — Under an SRW lock, invokes CTelemetryHelper::LogTelemetryLocked if 600 seconds have elapsed since the last telemetry flush.

long __thiscall DirectComposition::CDevice::Commit(CDevice *this, bool param_1, void *param_2)
{
    // Synchronization: custom lock at this+0x48, fallback to CRITICAL_SECTION at this+0x50
    if (*(longlong **)(this + 0x48) == (longlong *)0x0) {
        EnterCriticalSection((LPCRITICAL_SECTION)(this + 0x50));
    } else {
        cVar13 = (**(code **)(**(longlong **)(this + 0x48) + 8))();  // vtable acquire
        if (cVar13 != '\0') this[0x7c] = (CDevice)0x1;
        *(int *)(this + 0x78) = *(int *)(this + 0x78) + 1;
    }

    // Dirty-object rebuild loop — flushes pending visual changes
    this_02 = *(CRebuildableObject **)(this + 0x1c8);
    while (this_02 != (CRebuildableObject *)0x0) {
        uVar45 = CRebuildableObject::Rebuild(this_02);
        this_02 = *(CRebuildableObject **)(*(longlong *)(this + 0x1c8) + 8);
        if (-1 < (int)uVar45) {
            *(undefined8 *)(*(longlong *)(this + 0x1c8) + 8) = 0;
        }
    }

    // Virtual surface optimizer (idle > 999ms → candidate for removal)
    if (g_VSAllowOptimizer != false) {
        UVar19 = GetTickCount64();
        // CVirtualSurfaceOptimizer::OptimizeVirtualSurface(...)
    }

    // Back-pressure: spin until kernel batch queue drains
    if (cVar13 != '\0') {
        do {
            iVar18 = CommitToKernel(this, (longlong *)p_Var29, 0, in_R9);
            if (iVar18 < 0) break;
            lVar16 = WaitForCommitCompletionInternal(this, false);
            if (lVar16 < 0) break;
            bVar47 = ShouldBlockForExcessivePending(this);
        } while (bVar47);
    }

    // Telemetry (SRW lock, 600 000ms throttle)
    AcquireSRWLockExclusive((PSRWLOCK)(this + 0x2b0));
    if (*(GetTickCount64()) > *(ulonglong *)this_00) {
        CTelemetryHelper::LogTelemetryLocked(this_00);
        *(ULONGLONG *)this_00 = UVar19 + 600000;
    }
    ReleaseSRWLockExclusive((PSRWLOCK)(this + 0x2b0));

    // Release synchronization
    if (*(longlong **)(this + 0x48) != (longlong *)0x0) {
        (**(code **)(**(longlong **)(this + 0x48) + 0x18))();  // vtable release
    } else {
        LeaveCriticalSection((LPCRITICAL_SECTION)(this + 0x50));
    }
}

Phase 2 — Batch Preparation: CDevice::CommitToKernel

CommitToKernel validates the batch before it crosses the privilege boundary. The most interesting part for fuzzing purposes is the integrity check — if the committed batch size doesn't match the expected counter, the process calls FailFastForKernelBatchFailure rather than propagating the mismatch. This is a deliberate hardening decision to prevent corrupted batches from reaching the kernel.

Key field offsets documented from the Ghidra analysis:

int __thiscall DirectComposition::CDevice::CommitToKernel(
    CDevice *this, longlong *param_1, undefined1 param_2, undefined8 param_3)
{
    // Pre-flush resource manager if present
    if (*(longlong **)(this + 0x48) != (longlong *)0x0)
        (**(code **)(**(longlong **)(this + 0x48) + 0x18))();

    if (*(int *)(this + 0xa0) != 0) {
        // Batch validation: compare committed range to expected size
        iVar1 = (**(code **)(**(longlong **)(this + 0x88) + 0x10))(
                    *(longlong **)(this + 0x88),
                    *(int *)(this + 0x98) - *(int *)(this + 0x9c),  // actual size
                    &local_24, local_res20);

        // INTEGRITY CHECK — mismatch → FailFast (non-recoverable)
        if (local_24 != *(uint *)(this + 0xa0))
            FailFastForKernelBatchFailure(this, local_24);

        // Post-validate cleanup
        if (*(void **)(this + 0xa8) != (void *)0x0) {
            DefaultHeap::Free(*(void **)(this + 0xa8));
            *(undefined8 *)(this + 0xa8) = 0;
        }
        *(undefined4 *)(this + 0xa0) = 0;
        *(undefined4 *)(this + 0x9c) = *(undefined4 *)(this + 0x98);

        if (local_res20[0] != '\0')
            CheckForDelayedDestructionObjects(this);
    }

    // Commit to kernel channel — vtable +0x18 on KernelChannel at this+0x88
    iVar1 = (**(code **)(**(longlong **)(this + 0x88) + 0x18))(
                *(longlong **)(this + 0x88), &local_20, local_28, ...);

    if (-1 < iVar1) {
        *(int *)(this + 0x188) = local_20 + 1;  // increment commit sequence
        *(undefined8 *)(this + 0x268) = 0;
    }
    return iVar1;
}

Phase 3 — Kernel Bridge: KernelChannel::CommitChannel

This is the last user-mode function. It retrieves the channel handle from this+0x8 (assigned during NtDCompositionCreateChannel), calls the syscall, and translates the kernel's NTSTATUS response into a COM HRESULT.

ulonglong __thiscall DirectComposition::KernelChannel::CommitChannel(
    KernelChannel *this, longlong param_1)
{
    // Cross the privilege boundary — this+8 is the channel handle
    uVar2 = NtDCompositionCommitChannel(*(undefined4 *)(this + 8));
    uVar1 = (uint)uVar2;

    if (uVar1 == 0) return uVar2;  // S_OK fast-path

    // NTSTATUS → HRESULT translation table
    if ((int)uVar1 < -0x3fffffe8) {
        if (uVar1 == 0xc0000017) return 0x8007000e;  // STATUS_NO_MEMORY → E_OUTOFMEMORY
        if (uVar1 == 0x803e0006) return 0x88980800;  // DXGI-specific
        if (uVar1 == 0xc0000001) return 0x80004005;  // STATUS_UNSUCCESSFUL → E_FAIL
        if (uVar1 == 0xc0000002) return 0x80004001;  // STATUS_NOT_IMPLEMENTED → E_NOTIMPL
        if (uVar1 == 0xc0000008) return 0x80070006;  // STATUS_INVALID_HANDLE → E_HANDLE
        if (uVar1 != 0xc000000d) goto LAB_fallback;
    } else {
        if (uVar1 == 0xc0000022) return 0x80070005;  // STATUS_ACCESS_DENIED → E_ACCESSDENIED
        if (uVar1 == 0xc0000024) return 0x80070006;  // STATUS_OBJECT_TYPE_MISMATCH → E_HANDLE
        if (uVar1 != 0xc00000bb) goto LAB_fallback;
    }
    return 0x80070057;  // E_INVALIDARG

LAB_fallback:
    return (ulonglong)(uVar1 | 0x10000000);  // generic facility-code remapping
}

The NTSTATUS-to-HRESULT mapping table is worth studying carefully. The 0x10000000 OR fallback on the last line is particularly notable — it silently remaps any unrecognized NTSTATUS to a generic facility code, potentially swallowing unexpected error conditions from the kernel.

Kernel-Side Analysis — NtDCompositionCommitChannel

The kernel handler for this syscall is located at offset 0x5eb90 in win32kbase.sys, export index 1367. This is where Ring 0 takes control.

5.1 — User-Mode Pointer Validation

The first thing the kernel does is probe every user-supplied pointer against MmUserProbeAddress. This is mandatory defensive coding for any kernel function that accepts user-mode memory references — attempting to dereference a pointer above MmUserProbeAddress without probing first would be a kernel vulnerability itself.

The probe pattern works by substituting the pointer with MmUserProbeAddress if it falls outside user space, then performing a read — the structured exception handler fires on an invalid user address, cleanly returning STATUS_ACCESS_VIOLATION rather than crashing the kernel.

// Probe optional output pointer param_2
if (param_2 != (undefined4 *)0x0) {
    puVar6 = param_2;
    if (*(undefined4 **)MmUserProbeAddress_exref <= param_2)
        puVar6 = *(undefined4 **)MmUserProbeAddress_exref;
    *puVar6 = *puVar6;  // SEH fires here on invalid user address
}

// Probe mandatory output pointer param_3
puVar7 = param_3;
if (*(undefined1 **)MmUserProbeAddress_exref <= param_3)
    puVar7 = *(undefined1 **)MmUserProbeAddress_exref;
*puVar7 = *puVar7;

5.2 — Process and Channel Object Lookup

After pointer validation, the kernel resolves the calling process's Win32k state via PsGetCurrentProcessWin32Process(), then looks up the channel object in a generic table keyed by the handle passed from user space.

This lookup is lock-protected: KeEnterCriticalRegion disables normal APC delivery, and ExAcquireResourceExclusiveLite takes exclusive ownership of the channel table lock. A missing handle returns STATUS_ACCESS_DENIED (0xC0000022) rather than a NULL dereference.

lVar4 = PsGetCurrentProcessWin32Process();
plVar9 = *(longlong **)(lVar4 + 0x100);  // Win32 process state

if (plVar9 == (longlong *)0x0) {
    uVar3 = 0xc0000001;  // STATUS_UNSUCCESSFUL — no Win32 state initialised
} else {
    // Acquire exclusive lock on channel table
    KeEnterCriticalRegion();
    ExAcquireResourceExclusiveLite(plVar9[1], 1);

    // Look up channel handle in generic table
    local_48 = param_1;
    lVar4 = RtlLookupElementGenericTable(*plVar9, &local_48);
    if (lVar4 != 0) plVar10 = *(longlong **)(lVar4 + 8);

    if (plVar10 == (longlong *)0x0) {
        uVar3 = 0xc0000022;  // STATUS_ACCESS_DENIED — handle not found
    } else {
        LOCK();
        *(int *)(plVar10 + 1) = (int)plVar10[1] + 1;  // AddRef channel object
        UNLOCK();
    }

    ExReleaseResourceLite(plVar9[1]);
    KeLeaveCriticalRegion();
}

5.3 — Batch Deserialization Loop

The core of the handler. Once the channel object is resolved and ref-counted, the kernel enters a loop that dequeues and dispatches command batches until the channel's ring buffer is empty. Each iteration calls FUN_1c005ee20 (the batch dispatcher, documented in 5.4) and passes the dequeued batch to FUN_1c005f594 for compositor dispatch.

do {
    local_60 = 0;
    local_58 = 0;
    // Dequeue one batch from channel ring buffer
    cVar1 = FUN_1c005ee20(plVar9, &local_60, &local_58, param_4 != 0);
    if (local_60 != 0) {
        // Dispatch batch record to compositor
        FUN_1c005f594(plVar9, local_60, local_58,
                      *(byte *)(plVar9 + 6) & 1, lVar4);
    }
} while (cVar1 == '\0');  // '\0' means "more batches pending"

// Write output parameters back to user-mode pointers
local_78 = *(int *)((longlong)plVar9 + 0x204) != 0;  // has_pending_work flag
local_74 = (undefined4)plVar9[0x2f];                  // commit sequence number
uVar3 = 0;  // STATUS_SUCCESS

5.4 — Batch Command Dispatcher: FUN_1c005ee20

This is the most complex function in the kernel-side analysis. It contains the fast path / full path branch logic, the command serializer chain, and ETW telemetry hooks.

Fast Path vs Full Path. The fast path is taken when all dirty-state flags at offsets 0x30, 0xf0, and the linked lists at 0x1a8–0x250 are clear. If clean and param_4 == 0, the function simply wakes the compositor event at this+0x168 and returns — no batch record is constructed.

Command Serializer Chain. The full path chains through a sequence of specialized serializers, each responsible for a specific resource type:

Special Command Records. Two fixed-format records are emitted at the end of the serializer chain under specific conditions:

• TYPE_0x34 — Control block record, emitted if this+0x2e8 is non-null. Serializes 9 DWORD fields from offsets 0x2f0–0x310.
• TYPE_0x0C — Visibility change record, emitted if this+0x250 is 1 or 2. Signals window show/hide state to the compositor.

Conclusions & Next Steps

The reconnaissance phase is complete. No exploitable vulnerability was identified — but that is not the conclusion of the research, it is the baseline. The structural understanding built here is the prerequisite for any meaningful fuzzing. You cannot fuzz what you don't understand.

What was established:

• SSNs for NtDCompositionCommitChannel (0x111F) and NtDCompositionCreateChannel (0x1124) extracted and verified on Build 19045.6466
• Full user-mode commit pipeline traced through all three phases with critical vtable offsets and field offsets documented
• Kernel handler NtDCompositionCommitChannel partially reversed — pointer probing, handle lookup, and batch dispatch logic fully understood
• Batch dispatcher FUN_1c005ee20 annotated with fast path, full path, command serializer chain, and ETW hooks identified

Planned next steps, in order:

1. Direct Syscall Harness — Build a harness invoking NtDCompositionCreateChannel then NtDCompositionCommitChannel directly, bypassing dcomp.dll entirely and with it, the user-mode batch size integrity check.
2. Malformed Batch Fuzzing — Target edge cases in FUN_1c005ee20: mismatched batch size counters, invalid command type IDs (especially 0x34 and 0x0C), and overlapping cross-adapter surface entries.
3. Full PDB Reverse of win32kbase.sys — Load with symbols into Ghidra to fully reverse FUN_1c005f594 (batch compositor dispatch) and FUN_1c005f78c (batch record allocator).
4. Live Kernel Instrumentation — WinDbg hardware breakpoint on FUN_1c005ee20 to capture live batch structure layouts during normal DWM operation.
5. Error Path Audit — Specifically audit the NTSTATUS | 0x10000000 fallback mapping. Error codes suppressed here may indicate kernel code paths that were not expected to be reachable from user mode.

// Research conducted on isolated hardware. No production systems involved. For educational and defensive security research purposes only.