Nirvana v2: Electric Boogaloo

Direct syscalls are usually explained as an entry-point problem.

That framing is useful, but incomplete. The familiar story is that an endpoint product hooks an exported ntdll function, so an implant skips the export body, loads the syscall number itself, executes syscall, and avoids the user-mode hook.

That is the right place to start.

It is not the right place to stop.

The more interesting question shows up after the kernel has already done the work:

Where did user mode resume?

That one question is the center of this project. If the kernel just serviced NtProtectVirtualMemory, did execution return to the post-syscall address inside the real NtProtectVirtualMemory stub? Or did it return somewhere else: a private assembly thunk (a small hand-written code stub), a dynamically generated stub, or the syscall instruction borrowed from an unrelated ntdll export?

That is return provenance. “Provenance” just means origin, and here the origin we care about is the post-syscall return address. Once a process can answer that question for the syscalls it cares about, direct syscall detection becomes much sharper than “did execution enter this function?”

Nirvana v2: Electric Boogaloo is my native x64 implementation of that idea. The name is a nod to the older Nirvana-style process instrumentation callback technique, but the goal here is narrower and more concrete: use the callback path to capture the previous user PC after a syscall, recover the syscall number from kernel-maintained thread metadata, and compare the two.

The useful invariant is simple:

syscall number ownership must match return-address ownership

If syscall ID 0x0050 belongs to ntdll.dll!NtProtectVirtualMemory, then a normal exported call should return through the post-syscall address owned by NtProtectVirtualMemory. If it returns through NtQuerySystemTime, that is not a normal exported call. It is a syscall-gadget mismatch.

The implementation repository is here:

https://github.com/CameronBabcock/DirectSyscallDetector

Research notes: the raw WinDbg, IDA, kernel-symbol, and harness notes behind this post are published with the project as ResearchNotes.md. This article is the polished narrative; the notes preserve the evidence trail.

The Short Version

The detector has four moving parts:

1. Build a live syscall catalog from the loaded ntdll.dll and win32u.dll.
2. Install a process instrumentation callback with NtSetInformationProcess(ProcessInstrumentationCallback).
3. Capture the previous user PC from callback-entry R10, plus stack arguments 5 and 6 from the saved user stack.
4. Have a monitor thread query NtQueryInformationThread(ThreadLastSystemCall) for the syscall number and first argument, then classify the event.

Two bypass shapes matter.

A raw direct syscall executes its own syscall instruction from memory that is not one of the cataloged Windows syscall stubs.

A gadgeted syscall, sometimes called an indirect syscall, is more slippery. It reuses a real syscall instruction inside ntdll, but loads the wrong syscall number before jumping there.

Return provenance catches both because it does not merely ask whether the syscall instruction lived inside ntdll. It asks whether the specific syscall number and the specific post-syscall return address belong to the same exported stub.

Applying that invariant gives a compact set of verdicts:

Recovered syscall ID	Previous PC after syscall	Verdict
Known ID	Expected return for that same ID	Clean
Known ID	Not in any cataloged stub	Raw direct syscall
Known ID	In a cataloged stub for a different syscall ID	Syscall-return mismatch
Unknown ID	Any return address	Unknown syscall

The Visual Studio demo exercises the cases I care about:

Probe	What happens	Expected classification
Baseline ntdll!NtAllocateVirtualMemory	Calls the real exported native API	Clean
Raw MASM NtAllocateVirtualMemory	Executes syscall from project-owned assembly	Raw direct syscall
Raw MASM NtProtectVirtualMemory	Executes another syscall from project-owned assembly	Raw direct syscall
Gadgeted NtProtectVirtualMemory	Loads the NtProtectVirtualMemory syscall number, then jumps to a syscall instruction inside ntdll!NtQuerySystemTime	Syscall-return mismatch

The final output is intentionally plain:

[baseline ntdll!NtAllocateVirtualMemory] clean
  syscall: 0x0018 (ntdll.dll!NtAllocateVirtualMemory)
  previous pc owner: ntdll.dll!NtAllocateVirtualMemory

[raw MASM NtAllocateVirtualMemory] raw direct syscall
  syscall: 0x0018 (ntdll.dll!NtAllocateVirtualMemory)
  previous pc: 0x00007FF7A6D9970B

[ntdll syscall gadget for NtProtectVirtualMemory] syscall-return mismatch
  syscall: 0x0050 (ntdll.dll!NtProtectVirtualMemory)
  previous pc owner: ntdll.dll!NtQuerySystemTime

The last case is the whole project in one sentence:

This was NtProtectVirtualMemory, but it returned through NtQuerySystemTime.

That is a stronger statement than “the return address was in ntdll.” It tells you which syscall ran, which stub owned the return, and why those facts do not agree.

Why Entry-Point Thinking Is Not Enough

A lot of user-mode API monitoring starts with a reasonable question:

Did execution enter ntdll!NtAllocateVirtualMemory?

Direct syscalls are designed to make that question boring. They skip the export entry point on purpose.

So a better monitor might ask:

Did the syscall instruction live somewhere inside ntdll?

That catches raw syscalls from arbitrary executable memory, but it still misses the more interesting trick. A gadgeted syscall can borrow a clean syscall instruction from a legitimate ntdll stub while loading a different syscall number.

This project asks a narrower question:

The kernel recorded syscall ID N.
Which export owns syscall ID N?
Which export owns the address the thread returned to?
Do those owners match?

That is why the gadget probe gets caught. The return PC is inside ntdll.dll, but it is inside the wrong syscall stub.

The Stub Gives Us An Anchor

On native x64 Windows, user mode normally enters the kernel through small syscall stubs exported by ntdll.dll. GUI-related syscalls commonly route through win32u.dll. The exact bytes vary by build and patch state, but the normal logical shape is familiar:

mov     r10, rcx
mov     eax, ServiceNumber
syscall
ret

Two details make this usable as a detection anchor.

First, the syscall number lives in EAX. Those numbers are not stable across Windows builds, so a serious detector should not ship a hard-coded table and pretend it describes every machine.

Second, the expected user-mode return address is the instruction immediately after syscall. Since syscall is two bytes, the detector stores:

ExpectedReturnAddress = SyscallInstructionAddress + 2

That syscall + 2 address becomes the return anchor for a specific exported stub.

Building A Live Syscall Catalog

The catalog is deliberately built at runtime from the process’s own loaded modules:

ntdll.dll
win32u.dll

That choice keeps the detector honest. It observes the machine it is running on instead of trusting a stale syscall-number spreadsheet.

For each module, the catalog builder walks the PE export directory, skips forwarded exports, scans candidate functions for mov eax, imm32 followed by syscall, and records:

syscall number
module name
export/function name
function address
syscall instruction address
expected return address
generated PHNT-backed signature metadata

The catalog then exposes two indexes:

syscall number          -> one or more exported syscall stubs
expected return address -> owning exported syscall stub

That split is the entire point. In a clean call, both indexes point to the same owner. In a gadgeted call, they split.

For example:

SystemCallNumber:
  0x0050 -> ntdll.dll!NtProtectVirtualMemory

PreviousProgramCounter:
  ntdll.dll!NtQuerySystemTime + 0x14

Verdict:
  syscall-return mismatch

The signature printer follows the recovered syscall ID, not the previous PC. If a gadget returns through NtQuerySystemTime, but the kernel says the syscall was NtProtectVirtualMemory, then the right arguments to print are NtProtectVirtualMemory arguments.

That seems obvious once stated, but it is an easy place to get the tooling subtly wrong.

Keeping The Metadata Honest With PHNT

I wanted the output to be useful without pretending to recover more than the detector actually sees.

The project bundles a copy of PHNT, the Process Hacker native API headers that declare the full Nt* syscall prototypes Windows does not document publicly. A generator turns those declarations into a compile-checked signature registry.

The generated factories look like this:

using TFunction = std::remove_pointer_t<decltype(&::NtAllocateVirtualMemory)>;
return MakeSyscallSignature<TFunction>(
    "NtAllocateVirtualMemory",
    std::array<std::string_view, 6>{
        "ProcessHandle",
        "BaseAddress",
        "ZeroBits",
        "RegionSize",
        "AllocationType",
        "PageProtection",
    },
    std::array<std::string_view, 6>{
        "HANDLE",
        "PVOID*",
        "ULONG_PTR",
        "PSIZE_T",
        "ULONG",
        "ULONG",
    });

The decltype(&::NtAllocateVirtualMemory) part is not decoration. It makes the generated file compile against PHNT. If the registry drifts away from the declarations available to the build, the compiler catches it instead of letting the detector print stale metadata.

At runtime, the path is:

ThreadLastSystemCall says syscall ID 0x18
  -> catalog resolves 0x18 to NtAllocateVirtualMemory
  -> generated registry provides NtAllocateVirtualMemory metadata
  -> printer emits captured argument names, types, and values

The current demo prints only values it actually captures:

NtAllocateVirtualMemory:
  arg1 ProcessHandle
  arg5 AllocationType
  arg6 PageProtection

NtProtectVirtualMemory:
  arg1 ProcessHandle
  arg5 OldProtection

Arguments 2 through 4 are intentionally absent. The detector does not fabricate them to make the output look complete.

The Kernel Handoff I Needed To Trust

The technique depends on one private user-mode callback path, so I did not want to treat it like folklore. Before trusting the detector, I wanted to know exactly what the kernel was handing to user mode.

The callback is installed with:

NtSetInformationProcess(
    NtCurrentProcess(),
    ProcessInstrumentationCallback,
    ...);

On the tested Windows 26100 x64 target, ProcessInstrumentationCallback is process information class 0x28 / decimal 40.

The kernel stores the callback in the process object:

_KPROCESS+0x168 InstrumentationCallback

The researched path is:

NtSetInformationProcess
  -> case ProcessInstrumentationCallback / 0x28
  -> validate callback target with MmValidateUserCallTarget
  -> lock the process
  -> write _KPROCESS.InstrumentationCallback

That validation detail affects the implementation. The callback thunk should live in a real loaded module, meaning image-backed code, rather than freshly allocated anonymous memory. The exception is if you deliberately register dynamic code as a valid Control Flow Guard target.

This is one of those places where reverse engineering changes the engineering shape. The result is not just “there is a callback.” It is “there is a callback, and the callback target has to satisfy the kernel’s user-call-target validation rules.”

The Discovery That Made The Demo Work

The most important kernel function in the notes is:

nt!KiSetupForInstrumentationReturn

The observed x64 logic is roughly:

mov rax, gs:188h
mov rdx, [rax+0B8h]
mov r8,  [rdx+168h]       ; _KPROCESS.InstrumentationCallback
test r8, r8
jnz  callback_present
ret

callback_present:
cmp word ptr [rcx+170h], 33h
jnz ret
mov rax, [rcx+168h]       ; old trap-frame Rip
mov [rcx+58h], rax        ; trap-frame R10 = old Rip
mov [rcx+168h], r8        ; trap-frame Rip = callback
ret

RCX is the trap-frame pointer.

The relevant offsets are:

_KTRAP_FRAME+0x058 R10
_KTRAP_FRAME+0x168 Rip
_KTRAP_FRAME+0x170 SegCs
_KTRAP_FRAME+0x180 Rsp

The kernel checks SegCs == 0x33, the user x64 code segment, then rewrites the trap frame:

old Rip -> trap-frame R10
callback -> trap-frame Rip

So callback entry looks like this:

RIP = callback
R10 = previous user PC
RSP = original user RSP
RAX = return value / existing trap-frame RAX

This is the subtle part:

on this x64 path, callback-entry R10 is the previous PC. RAX is not.

That discovery changed the project from “interesting crashy harness” into a working detector. My early research harness treated RAX as the previous PC and fell over. Once the callback used R10, the probe completed cleanly.

The notes also checked the TEB instrumentation fields:

TEB+0x2d0 InstrumentationCallbackSp
TEB+0x2d8 InstrumentationCallbackPreviousPc
TEB+0x2e0 InstrumentationCallbackPreviousSp

Those fields exist, but in the tested path they were not the source of the previous PC. The authoritative handoff was trap-frame R10.

Why The Callback Starts In Assembly

Once the previous PC arrives in R10, the callback has a very strict job: save first, think later.

Normal C++ is the wrong first instruction. The compiler owns volatile registers, and a C++ prologue or call setup can destroy the evidence before the detector has a chance to publish it.

So the callback starts as a save-first assembly thunk:

pushfq
push rax
push rcx
push rdx
push rbx
push rbp
push rsi
push rdi
push r8
push r9
push r10
push r11
push r12
push r13
push r14
push r15

; publish evidence to a fixed shared record

pop r15
pop r14
pop r13
pop r12
pop r11
pop r10
pop r9
pop r8
pop rdi
pop rsi
pop rbp
pop rbx
pop rdx
pop rcx
pop rax
popfq

mov r11, r10
jmp r11

The final callback follows that pattern. It saves state, filters to the armed worker thread, publishes the evidence into a fixed callback slot, waits for the monitor to acknowledge it, restores state, and jumps back to the previous PC.

It does not call native APIs. It does not allocate. It does not format strings. It does not take C++ locks.

The rule is:

the callback captures; the monitor interprets

Keeping that boundary clean is what makes the demo understandable. The callback is tiny and hostile-environment friendly. The monitor thread can afford to do the heavier work.

The Other Half: ThreadLastSystemCall

Capturing the previous PC tells us where user mode is about to resume. It does not, by itself, tell us which syscall the kernel just serviced.

For that, the detector uses NtQueryInformationThread(ThreadLastSystemCall) from a different thread.

On x64, kernel syscall entry first reconstructs the first argument register state:

nt!KiSystemCall64:
swapgs
...
mov rcx, r10

Later, the user syscall dispatcher stores metadata in the current thread:

nt!KiSystemServiceUser+0xc6:
mov [rbx+88h], rcx     ; _KTHREAD.FirstArgument
mov [rbx+80h], eax     ; _KTHREAD.SystemCallNumber
mov [rbx+90h], rsp     ; _KTHREAD.TrapFrame

The relevant researched offsets were:

_KTHREAD+0x080 SystemCallNumber
_KTHREAD+0x088 FirstArgument
_KTHREAD+0x090 TrapFrame
_KTHREAD+0x184 State
_KTHREAD+0x232 PreviousMode

NtQueryInformationThread(ThreadLastSystemCall) reaches that data through:

NtQueryInformationThread
  -> ThreadLastSystemCall case 21
  -> PspQueryLastCallThread
  -> _KTHREAD.SystemCallNumber
  -> _KTHREAD.FirstArgument

There is an important catch. The researched PspQueryLastCallThread path:

rejects the current thread
requires KTHREAD.State == Waiting
requires PreviousMode == UserMode
checks that ContextSwitches remained stable
returns FirstArgument and SystemCallNumber

So ThreadLastSystemCall is not a magical in-callback escape hatch.

If the instrumentation callback tries to call NtQueryInformationThread on itself, two bad things happen:

1. PspQueryLastCallThread rejects the current thread.
2. The query syscall would overwrite the caller’s own latest syscall metadata anyway.

The final demo uses a monitor thread for that reason. The callback pauses the target in user mode long enough for the monitor to query the worker thread from the outside.

That is the handoff:

callback thread: captures previous PC and stack snapshot
monitor thread: queries syscall ID and classifies the event

Honest Argument Capture

This is one of the places where the output intentionally stays modest.

The allocation harness tried to recover a direct NtAllocateVirtualMemory call:

NtStatus status = DirectNtAllocateVirtualMemory(
    (HANDLE)-1,
    &base,
    0,
    &region_size,
    MEM_RESERVE | MEM_COMMIT,
    PAGE_READWRITE);

The callback could recover:

previous PC from callback-entry R10
syscall number from bytes near the direct stub
return status from RAX
stack arg 5 from saved user RSP
stack arg 6 from saved user RSP

It could not reliably recover:

arg1 from callback-entry RCX
arg2 from callback-entry RDX
arg3 from callback-entry R8
arg4 from callback-entry R9

The observed callback-entry state for the first four volatile argument registers did not match the original user-mode function arguments. The caller home-space slots were not useful either.

So the detector prints what it actually knows:

ThreadLastSystemCall:
  first argument

callback stack snapshot:
  fifth argument
  sixth argument

For NtAllocateVirtualMemory, that is enough to show:

ProcessHandle(HANDLE)=0xFFFFFFFFFFFFFFFF,
AllocationType(ULONG)=0x00003000,
PageProtection(ULONG)=0x00000004

0x3000 is MEM_RESERVE | MEM_COMMIT. 0x4 is PAGE_READWRITE.

That is useful telemetry. It is not a full argument trace, and the detector should not pretend otherwise.

The Callback Ring

The callback and monitor need a tiny event transport that does not make the callback path complicated.

The final project uses a small global callback ring:

constexpr std::size_t CallbackSlotCount{ 16 };

struct alignas(64) CallbackSlot
{
    volatile LONG State{};
    ULONG Reserved0{};
    volatile std::uint64_t ThreadId{};
    volatile std::uint64_t PreviousProgramCounter{};
    volatile std::uint64_t StackPointer{};
    volatile std::uint64_t StackArgument5{};
    volatile std::uint64_t StackArgument6{};
    volatile std::uint64_t Reserved1{};
    volatile std::uint64_t Reserved2{};
};

The state machine is intentionally tiny:

Free -> Writing -> Ready -> Free

The MASM callback:

1. claims a free slot with an interlocked compare-exchange;
2. writes thread ID, previous PC, stack pointer, stack argument 5, and stack argument 6;
3. marks the slot ready;
4. waits until the monitor clears it.

The monitor:

1. waits for a ready slot;
2. suspends or opens the worker thread as needed;
3. calls NtQueryInformationThread(ThreadLastSystemCall);
4. copies the callback record;
5. clears the slot;
6. classifies the event.

This is still a demo transport, not production telemetry infrastructure. A production sensor would need better answers for lost events, callback ownership, timeout behavior, cross-process deployment, telemetry export, protected process boundaries, and overhead.

The ring exists to keep the callback path small and make the concurrency boundary explicit.

Classification: Where The Signal Lands

Once the monitor has both data sources, classification is direct:

observedEntry = catalog.FindPrimaryBySyscallNumber(SystemCallNumber)
returnOwner   = catalog.FindByReturnAddress(PreviousProgramCounter)

if query failed:
    QueryFailed
else if observedEntry is null:
    UnknownSyscall
else if returnOwner is null:
    RawDirectSyscall
else if returnOwner.Number != observedEntry.Number:
    SyscallGadgetMismatch
else:
    Clean

The important part is that the classifier catches two different bypass shapes for two different reasons.

Raw Direct Syscall

The demo’s raw MASM helper does this:

mov r10, rcx
mov eax, dword ptr [g_DsdNtAllocateVirtualMemorySyscall]
syscall
ret

The syscall number is valid because it came from the live catalog. The return address is suspicious because it points back into the demo executable’s assembly helper instead of a cataloged Windows syscall stub.

observedEntry: NtAllocateVirtualMemory
returnOwner:   none
verdict:       raw direct syscall

Syscall Gadget Mismatch

The gadget helper does this:

mov r10, rcx
mov eax, dword ptr [g_DsdNtProtectVirtualMemorySyscall]
mov r11, qword ptr [g_DsdGadgetSyscallInstruction]
jmp r11

g_DsdGadgetSyscallInstruction points to the syscall instruction inside a different ntdll stub, such as NtQuerySystemTime.

The syscall ID is NtProtectVirtualMemory. The return PC owner is NtQuerySystemTime.

observedEntry: NtProtectVirtualMemory
returnOwner:   NtQuerySystemTime
verdict:       syscall-return mismatch

This is the key difference from broad “return address is inside ntdll” checks. The detector is not suspicious of ntdll by itself. It is checking whether the right ntdll stub owns the return.

From Detection To Tripwire

The demo classifies events and prints them, but the same primitive can support a more active tripwire model.

At callback time, the process has three useful facts:

what syscall just completed
where user mode is about to resume
what status value is currently in RAX

Those facts are enough to separate passive telemetry from policy. The monitor can decide that a known-good return path is normal evidence, an unknown return path is worth logging, and a raw or gadgeted path is a tripwire event.

The signal is powerful because it is not tied to a single user-mode API hook. A module can avoid ntdll!NtProtectVirtualMemory and still leave a post-syscall return address. A gadget can borrow a clean syscall instruction and still return through the wrong stub. The tripwire watches the return provenance after the kernel has already done the work.

That same signal plays out differently across process types:

Anti-cheat or anti-tamper module in a game process

• Signal: unfamiliar module-owned return paths around memory-query, protection, or handle syscalls
• Response: record, raise local confidence, or move the protected component into a safer mode
• Why it matters: detects inspection behavior without relying only on hooked API entry points

Offensive malware, anti-EDR, or anti-AV research in an authorized lab

• Signal: raw or gadgeted syscall activity near sensitive allocation, protection, section, thread, or handle operations
• Response: self-unload, reduce capability, delay, or emit operator telemetry
• Why it matters: gives an agent a way to notice that its assumptions about the process changed

EDR canary process or protected sensor component

• Signal: unexpected direct-syscall provenance inside a process that should have boring syscall paths
• Response: generate tamper telemetry, snapshot context, or correlate with image-load and memory events
• Why it matters: converts a low-level primitive into a higher-confidence tamper signal

Productized endpoint sensor

• Signal: known syscall ID with mismatched return owner, plus suspicious module provenance
• Response: enrich the event with module identity, loaded-image history, and memory provenance
• Why it matters: helps distinguish noisy native API usage from deliberate syscall-path manipulation

The return-value angle should be treated carefully. On the researched x64 path, callback-entry RAX contains the syscall return status. The current project preserves it and returns to the previous PC. It is observe-only.

A policy-aware variant could choose to preserve, normalize, or fail a result before resuming user mode, but that belongs in controlled research or productized sensor logic, not in an indiscriminate demo.

The public point is not “here is a trick to interfere with a named product.” The useful idea is:

post-syscall return provenance can become a process-local tripwire

For defensive engineering, that can mean agent self-protection, canary processes, or tamper-aware telemetry. For CNO and pentest agent work, it can mean authorized self-awareness: unload if the environment starts behaving in a way the agent has never seen before, or report that an anti-cheat, anti-EDR, anti-AV, or other inspection component appears to be present.

Evidence Trail

Here is the compact claim-to-evidence map from the notes and final project:

Claim	Evidence
ProcessInstrumentationCallback is set through NtSetInformationProcess, class 0x28 / decimal 40	IDA switch case and live NtSetInformationProcess breakpoint
The callback is stored in _KPROCESS.InstrumentationCallback	WinDbg dt nt!_KPROCESS Instrumentation* and live store at NtSetInformationProcess+0x2330
x64 previous PC is delivered in callback-entry R10	KiSetupForInstrumentationReturn disassembly, trap-frame before/after, callback-entry registers
RAX is not previous PC	Trap-frame and callback-entry values; allocation probe showed RAX as syscall return status
TEB instrumentation previous-PC fields were not the source in this path	TEB field dump at callback entry
ThreadLastSystemCall reaches PspQueryLastCallThread	IDA NtQueryInformationThread case 21 and WinDbg disassembly
PspQueryLastCallThread reads _KTHREAD.FirstArgument and _KTHREAD.SystemCallNumber	WinDbg disassembly and _KTHREAD offsets
Cross-thread blocked syscall metadata can be queried	Direct NtWaitForSingleObject harness matched event handle and syscall number
Direct allocation callback can recover previous PC, return status, and stack args 5/6	Direct NtAllocateVirtualMemory harness output
First four arguments are not reliable at callback entry	Allocation harness register and home-space results
Final detector catches clean, raw, and gadget cases	Visual Studio demo verdict checks

The full raw evidence is in ResearchNotes.md.

Reproducing The Project

The solution has two projects:

DirectSyscallDetectorLib
  static library
  C++23 / latest MSVC mode
  MASM enabled
  PHNT vendored under third_party/phnt

DirectSyscallDetectorDemo
  console app
  links the library
  runs the four probes

The intended verification commands are:

& 'C:\Program Files\Microsoft Visual Studio\18\Insiders\MSBuild\Current\Bin\amd64\MSBuild.exe' `
  .\DirectSyscallDetector.sln /m /p:Configuration=Debug /p:Platform=x64

& 'C:\Program Files\Microsoft Visual Studio\18\Insiders\MSBuild\Current\Bin\amd64\MSBuild.exe' `
  .\DirectSyscallDetector.sln /m /p:Configuration=Release /p:Platform=x64

.\x64\Release\DirectSyscallDetectorDemo.exe

The demo returns nonzero if an expected verdict fails:

expect baseline ntdll allocate: pass
expect raw allocate: pass
expect raw protect: pass
expect ntdll gadget protect: pass

Productization Notes

This is an in-process research demo, not an injectable EDR sensor.

That distinction matters. The signal is useful, but a production design has more work to do:

• decide how to install or broker callbacks across many target processes;
• handle processes where another component already owns the instrumentation callback;
• avoid unacceptable overhead in syscall-heavy workloads;
• design backpressure and lost-event behavior;
• export telemetry without calling complex APIs inside the callback;
• account for PPL and protected-process boundaries;
• correlate with image loads, memory provenance, ETW, call stacks, and suspicious allocation/protection behavior;
• track Windows-build differences in private callback and ThreadLastSystemCall behavior.

There are also research questions worth keeping open:

• What happens when an attacker borrows a same-ID alias stub?
• How should a sensor treat patched or remapped syscall stubs?
• How often do legitimate components use syscall gadgets or custom syscall thunks?
• Which syscall families produce the best detection value relative to callback overhead?
• How stable is this path across Windows versions and processor modes?

Those are not reasons to ignore the signal. They are the engineering questions that turn a sharp primitive into a deployable detection.

The Mental Model

The whole detector compresses down to this:

1. Catalog every live syscall stub in ntdll.dll and win32u.dll.
2. Store syscall number -> syscall metadata.
3. Store syscall + 2 return address -> stub owner.
4. Install a process instrumentation callback.
5. Capture callback-entry R10 as previous PC.
6. Capture stack args 5/6 from saved user RSP.
7. Query ThreadLastSystemCall from a monitor thread for syscall ID and arg1.
8. Compare syscall-ID ownership to return-address ownership.

The recovered syscall ID tells you what actually ran.

The return PC tells you whether the path was normal.

Together, those facts give the detector a precise, testable conclusion grounded in the kernel handoff instead of a brittle entry-point assumption.

Closing

I built this because I wanted a direct-syscall detector that did not stop at “hook the native API harder” or “scan for suspicious syscall stubs.” Those ideas have value, but they miss the part of the story the kernel gives us for free: after the syscall completes, Windows has to resume user mode somewhere.

Runtime cataloging, process instrumentation callbacks, ThreadLastSystemCall, and PHNT-backed signatures fit together into one clean model:

after the kernel did the work, where did the thread come back?

For raw and gadgeted direct syscalls, that question carries a lot of signal.

The best part is that the detection sentence stays human-readable. Not “untrusted control transfer near native transition.” Not “suspicious syscall-path condition.” Just:

This was NtProtectVirtualMemory, but it returned through NtQuerySystemTime.

That is the kind of evidence I want from a sensor: specific enough for an engineer, clear enough for an analyst, and grounded enough that the next debugging question is obvious.

If you work on EDR, CNO, pentest agent work, endpoint security, Windows internals, or detection engineering and this kind of research is relevant to your team, I would be happy to talk.

• Email: Cameron@CameronBabcock.net
• LinkedIn: https://www.linkedin.com/in/cameronbabcock/