Upon reading the title of this article, one might pose the initial question: what would an ARM-based operating system do with an x86 instruction? Or a chunk of x86 instructions? Or an entire x86 binary? Windows 10, for example, does this by taking a set of x86 instructions below:
push ebp
mov ebp,esp
pop ebp
nop
jmp ntdll_775d0000!LdrInitializeThunk
And translating it to the following:
str wfp,[x28,#-4]! // push ebp
mov wfp,w28 // mov ebp,esp
ldr wfp,[x28],#4 // pop ebp
add w9,w9,#0x83,lsl #0xC
add w9,w9,#0x1FE
bl 00000000`03109aa8 // (get jump function address)
br xip1 // jmp ntdll_775d0000!LdrInitializeThunk
First off, ARM and x86 are completely different architectures. An ARM processor is incapable of executing x86 code and the hardware provides no means to do so. This leaves the task up to software developers to facilitate it themselves.
Microsoft's recent version of Windows 10 for ARM-based processors assumes such a task, by simulating an x86 processor entirely in userland. An emulator module (xtajit.dll) employs a form of just-in-time (JIT) translation to convert x86 code to ARM (shown above) within a loop, as the x86 process is executing. On each pass, a chunk of x86 code is translated to ARM, and the translation is executed.
All of this, as you might have guessed, can make the experience of running x86 programs a comparatively slow experience. However, a cache of already-translated code (located in C:\Windows\XtaCache) eliminates much of the overhead. A compiler (xtac.exe) and background caching service (XtaCache) handle full binary translation and caching. Hybrid binaries (located in C:\Windows\SyChpe32) containing x86-to-ARM stubs also help to reduce overhead.
In this article, I present what I believe are five key features of x86 emulation, concluding with an example of the raw opcode translation procedure.
CHPE DLLs
A peculiar system directory exists on Windows 10 for ARM. Located at C:\Windows\SyChpe32, this folder holds a small set of DLL files with the same names as the most frequently used libraries on Windows: ntdll.dll, kernel32.dll, advapi32.dll, etc. Several interesting characteristics are observable in these types of files, including a mixture of x86 and ARM functions and the presence of a strange section name in the header.
.hexpthk
When we first begin investigating a CHPE file (ntdll.dll), we discover a new section type in the header:
000001e0: 00 00 00 00 00 00 00 00 2e 68 65 78 70 74 68 6b .........hexpthk
000001f0: e0 53 00 00 00 10 00 00 00 54 00 00 00 04 00 00 .S.......T......
00000200: 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00 60 ............ ..`
00000210: 2e 74 65 78 74 00 00 00 c2 5d 17 00 00 70 00 00 .text....]...p..
.hexpthk possibly stands for Hybrid Executable Push Thunk. The primary purpose of the 1300+ functions in this section is to provide a set of x86 jump stubs for the library's native ARM APIs. This eliminates the need for JIT translation or XTA cache file access, thus reducing a bit of overhead.
One of the stubs in this section:
ntdll_773e0000!EXP+#RtlCreateUserProcess:
773e2520 8bff mov edi,edi
773e2522 55 push ebp
773e2523 8bec mov ebp,esp
773e2525 5d pop ebp
773e2526 90 nop
773e2527 e9a4620d00 jmp ntdll_773e0000!#RtlCreateUserProcess (774b87d0)
Which jumps to:
ntdll_773e0000!#RtlCreateUserProcess:
00000000`774b87d0 29ba7bfd stp wfp,wlr,[sp,#-0x30]!
00000000`774b87d4 910003fd mov fp,sp
00000000`774b87d8 52800028 mov w8,#1
...
00000000`774b8810 94000004 bl ntdll_773e0000!#RtlCreateUserProcessEx (00000000`774b8820)
00000000`774b8814 28c67bfd ldp wfp,wlr,[sp],#0x30
00000000`774b8818 d65f03c0 ret
push_thunk and pop_thunk
This leads us to another collection of special CHPE functions (in the .text section): the push_thunk and the pop_thunk. These attempt to fetch the translated function from the JIT cache buffer or XTA cache file. Ntdll.dll contains a total of 282 push_thunk and 1829 pop_thunk functions. Below is a disassembly snippet of #NtAllocateVirtualMemory$push_thunk:
ntdll_773e0000!#NtAllocateVirtualMemory$push_thunk:
00000000`77525630 a9bb53f3 stp x19,x20,[sp,#-0x50]!
...
00000000`77525674 900002aa adrp x10,ntdll_773e0000!_os_wowa64_dispatch_call (00000000`77579000)
00000000`77525678 b9400150 ldr wip0,[x10]
00000000`7752567c b0000188 adrp x8,ntdll_773e0000!NtAccessCheck (00000000`77556000)
00000000`77525680 11060109 add w9,w8,#0x180
00000000`77525684 d63f0200 blr xip0
00000000`77525688 d503201f nop
00000000`7752568c 37f80085 tbnz x5,#0x1F,ntdll_773e0000!#NtAllocateVirtualMemory$push_thunk+0x6c (00000000`7752569c)
00000000`77525690 900002b0 adrp xip0,ntdll_773e0000!_os_wowa64_dispatch_call (00000000`77579000)
00000000`77525694 b9402210 ldr wip0,[xip0,#0x20]
00000000`77525698 d63f0200 blr xip0
00000000`7752569c b9400ba8 ldr w8,[fp,#8]
...
00000000`775256c4 d65f03c0 ret
XTA cache (.jc) files
We notice a number of XTA cache (.jc) files in the C:\Windows\XtaCache directory whose filenames correspond to several common DLLs:
ACWINRT.DLL.8CB3A2AB47A53C8A2B0154CD9DCFBAB3.323EB21400CAFAA0582C5009A75869C1.mp.1.jc
APPHELP.DLL.95D0BAD69AB6222384AF242317B56149.DF78C05157184445FAFBB6CE538964A5.mp.1.jc
BCRYPTPRIMITIVES.DLL.013B0F82E47EF5A5FCF5A8526296185B.EF8F27FC52B74ACB9AC3BC89FA928CE1.mp.1.jc
CFGMGR32.DLL.5D4A7278FAF3DBBDBF2DDB915FCAFAB5.F945F9A53B0D60BD9E78255821B9CF3C.mp.1.jc
CMD.EXE.82E4EB063821519C52FE2943A99BF400.1D875ED0DCA07279C17651C0E6C07D68.mp.1.jc
COMBASE.DLL.4F0AC5980CD35BD10454EEE975995AC5.27FF208B6378EEC51D8E744BAF092F94.mp.1.jc
CRYPT32.DLL.C5DAC220D689F508F0E17E2A66CA3F1A.320EA93B91430A11C07BCF035279C389.mp.1.jc
CRYPTBASE.DLL.16CA22D6D04B8B1996C66338057F9FBF.385D1574AF589DD7DBE99097ADFA13A7.mp.1.jc
DEVOBJ.DLL.C27E6011E5F5F63CCBE4C1EDDE39AB82.8F5C03EA8E04FE31C38209807883CEFF.mp.1.jc
KERNEL32.DLL.834794B203AF374A9362DB4ED6A773E1.DDEEC7202859318755E42D071DBE493B.mp.1.jc
KERNELBASE.DLL.FD274E0B2908F4427DD5E105D6793B46.DB966B70C90268F5B3A22AF2FFD62FB9.mp.1.jc
MMDEVAPI.DLL.4095F18EA4B2E4980A31F9CFF10ACB18.F3FFB7D241CFAF41DBF7A8EB1FA2F14C.mp.1.jc
MSASN1.DLL.592DF648596867992875D9F422985BE2.C91F12FC8B8289618C00A6A0B4CB63F6.mp.1.jc
MSVCP_WIN.DLL.7A9A3D8939BA215908F2D277F6036868.059202EDB6FBBF9C5D3CEA601F657783.mp.1.jc
MSVCRT.DLL.D751099CF1900AB3B0B21169E3ACAEA1.D32B8BAE88BDFA4E405C93DE74A58C58.mp.1.jc
NTDLL.DLL.25E64FDB4C4531A2DA3649AF45082708.A8B2D871AD511568138A61A746F3477E.mp.1.jc
SECHOST.DLL.B4519BA08D878093477A68086F2C73E9.6B8F8A4E1C74BF41ECB65F5BC09C99AE.mp.1.jc
SSPICLI.DLL.8A3767BF366F5B126EEE0A97F09F3821.177E02DDE1B9F225A2E156445C8C26C6.mp.1.jc
UCRTBASE.DLL.FDEE79B96ED912C4BECA94A2153A8DA6.2B41C9D2592D756F6DA93B24EBBAB8BB.mp.1.jc
WINBRAND.DLL.0B7D029DE53AA3517BAAE21FFF2958A2.080F7128AEA24497BE12FC799A471BCA.mp.1.jc
WINTRUST.DLL.36F3317E5E83FB1D21A49E8598B82079.1A4137576B1F56F2356651AA77E6C162.mp.1.jc
WLDP.DLL.52F5CBD679E57BDD90519D96701B1362.42321841F03DF5F906568D1916ABD9FC.mp.1.jc
.jc files have a header:
0:000:x86> db 07640000
07640000 58 54 41 43 13 00 00 00-00 00 00 00 48 0f 01 00 XTAC........H...
07640010 1f 01 00 00 38 00 00 00-12 00 00 00 40 18 01 00 ....8.......@...
07640020 64 00 00 00 50 00 00 00-50 00 00 00 50 a1 00 00 d...P...P...P...
07640030 d4 a1 00 00 ec 0e 01 00-4e 00 54 00 44 00 4c 00 ........N.T.D.L.
07640040 4c 00 2e 00 44 00 4c 00-4c 00 00 00 00 00 00 00 L...D.L.L.......
07640050 42 4c 43 4b e4 0e 01 00-00 00 00 00 00 00 00 00 BLCK............
07640060 bf 39 03 d5 c0 03 5f d6-00 00 40 79 fd ff ff 17 .9...._...@y....
07640070 20 00 40 79 fb ff ff 17-c0 00 40 79 f9 ff ff 17 .@y......@y....
.jc header format
Below is the format for the cache file header:
XTAC header
0x00 DWORD 'XTAC'
0x04 DWORD Always 0x13
0x08 DWORD BOOL? Most files set to 0, some are 1
0x0c DWORD Offset of JC address pair table
0x10 DWORD Number of JC address pairs (two DWORDs = 8 bytes per pair)
0x14 DWORD Offset of module name
0x18 DWORD Length of module name (in bytes)
0x1c DWORD Offset of module NT path name (usually at very end of file)
0x20 DWORD Length of module NT path name (in bytes)
0x24 DWORD Offset of BLCK stubs
0x28 DWORD ?
0x2c DWORD Size of BLCK stubs (always 0xa150)
0x30 DWORD ?
0x34 DWORD ?
0x38 WCHAR[] usually start of module name
XTA cache (.jc) files are composed of a JC address pair table and translated x86-to-ARM code. Entries in this pointer table each contain two relative virtual addresses: the RVA of the original function in the x86 binary and the RVA of the translated function in the cache file, in that order.
Pointer table entries do not always point to the start of a function. They often point to return addresses within a function, ie. the address of an instruction immediately following a call instruction. Hence multiple address pairs often exist for a single function.
Below is a translation of __mainCRTStartup from CMD.EXE.82E4EB063821519C52FE2943A99BF400.1D875ED0DCA07279C17651C0E6C07D68.mp.1.jc:
// __mainCRTStartup - translated ARM code from XTA cache file for C:\Windows\SysWOW64\cmd.exe
1bebc: 528b49bb mov w27, #0x5a4d // 'MZ'
1bec0: 51405926 sub w6, w9, #0x16, lsl #12
1bec4: 512374c6 sub w6, w6, #0x8dd
1bec8: 48dffcc7 ldarh w7, [x6]
1becc: 6b1b00e2 subs w2, w7, w27
1bed0: 54000761 b.ne 0x1bfbc // b.any
(...snip...)
Original (x86) __mainCRTStartup at 0x4168dd (base address 0x400000 + 0x168dd):
0x004168dd b84d5a0000 mov eax, 0x5a4d // 'MZ'
0x004168e2 663905000040. cmp word [0x400000], ax // [0x400000:2]=0xffff
0x004168e9 7555 jne 0x416940
(...snip...)
Notice the equivalent "mov eax, 0x5a4d" and "mov w27, #0x5a4d" instructions. The indirect memory access in the cmp instruction ends up being split into four ARM instructions, while the relative branch jne instruction is easily translated into a b.ne. With the cache file accessible, the emulator can then fetch a translated function if needed.
So, the final question remains: how does the emulator (xtajit.dll) actually perform the translation?
XTA JIT (xtajit.dll)
It all begins in BTCpuSimulate, the beating heart of the x86 emulator. However, emulation doesn't start right away when an x86 process begins. The emulation module, xtajit.dll, has not yet been loaded. Windows automatically loads the native ARM ntdll.dll (C:\Windows\System32\ntdll.dll), the CHPE ntdll.dll (C:\Windows\SyChpe32\ntdll.dll), and the wow64 DLLs required for emulation. Before emulation begins, the xtajit function BTCpuProcessInit is called:
NTSTATUS BTCpuProcessInit(PWCSTR wImageName, PVOID pCpuThreadSize)
{
if (!NT_SUCCESS(GetProcessorPowerInformation()))
return;
InitializeCacheDatabase(); // sub_18000E770
/*...*/
// Gets Wow64InfoPtr from TEB
TEB teb = GetCurrentTeb();
TEB32 teb32 = teb + teb->Peb32Offset;
if (teb == teb32->SubSystemTib)
wow64InfoPtr = teb32->TlsSlots[10]; // Wow64InfoPtr
else
wow64InfoPtr = teb->TlsSlots[10]; // Wow64InfoPtr
wow64InfoPtr->CpuFlags |= 2; // WOW64_CPUFLAGS_SOFTWARE
/*...*/
// Done
return STATUS_SUCCESS;
}
Then the emulation starts in the BtCpuSimulate loop:
void BTCpuSimulate()
{
// Calls RtlWow64GetCurrentCpuArea (sub_180015530)
PWOW64_CPU_AREA cpuArea = GetWow64ArmCpuArea();
while (1)
{
/* This function is in the wow64 module. Cross-process items might
include calls to BTCpuNotifyMemory functions. Among those are:
BTCpuNotifyMemoryAlloc, BTCpuNotifyMemoryFree,
BTCpuNotifyMemoryProtect, BTCpuNotifyMemoryDirty */
Wow64ProcessPendingCrossProcessItems();
// Inside the CPU emulator
GetCurrentTeb()->TlsSlots[2] = TRUE;
if (gWow64CpuInfo->ProcessInitComplete)
dmb_ish(); // Wait for all memory accesses to finish
// Updates x86 registers in WOW64 ARM context structure (sub_180015038)
CpupSwitchToX86(
cpuArea->Wow64ArmContext, // Destination
cpuArea->Wow64Context, // Source
);
// Emulates/executes x86 instruction (sub_1800215d0)
EmulateX86Function(
gWow64CpuInfo->BTProperties, // Initialized in BTCpuProcessInit
cpuArea->Wow64ArmContext // Same as 1st argument to CpupSwitchToX86
);
}
}
Registers for x86 are updated in CpupSwitchToX86, where a WOW64_CONTEXT structure inside a WOW64_CPU_AREA structure is updated with the newest values of each register from a source WOW64_CONTEXT. But the function perhaps deserving the most attention in this article is a function I have named EmulateX86Function.
EmulateX86Function
EmulateX86Function, not surprisingly, is a very large and very complex function, from which flow several more very large and very complex functions. Not surprisingly, because one would generally assume that converting between two completely different CPU architectures involves quite a bit of work. X86 instructions must be taken apart and analyzed, deconstructed, and interpreted as ARM instructions. This is all done through the cooperation of several routines, all stemming from a root function: EmulateX86Function. Below is a (truncated) reverse engineering of this function:
// sub_1800215d0
void EmulateX86Function(PVOID btProperties, PWOW64_ARM_CONTEXT wow64ArmContext)
{
PVOID jitCacheInfo = wow64ArmContext->JitCacheInfo;
do
{
DWORD eip = wow64ArmContext->Eip;
DWORD bitShift = jitCacheInfo->BitShift;
DWORD mask = jitCacheInfo->Mask;
PVOID funcTbl = jitCacheInfo->FunctionTable;
DWORD offset = (eip >> bitShift) & mask;
PVOID funcEntry = funcTbl + offset;
PVOID funcEntryRes = NULL;
int counter = 0;
do
{
DWORD btFuncOff = funcEntry->BinaryTranslatedFunc;
if (btFunc != NULL)
{
PVOID origFunc = funcEntry->OriginalFunc;
funcEntry++;
if (origFunc == eip)
{
funcEntryRes = funcEntry - 1;
break;
}
}
else
{
funcEntryRes = NULL;
break;
}
} while (++counter < 32);
if (funcEntryRes != NULL)
{
PVOID dispatchInfo = btProperties->DispatchInfo;
DWORD wordSize = dispatchInfo->WordSize;
PVOID pJitCacheInfo = dispatchInfo->JitCacheInfo;
// Offset of the dispatch routine in the JIT-cache buffer
DWORD offset = dispatchInfo->Offset;
// Pointer to the JIT-cache buffer in memory
PVOID pJitCache = pJitCacheInfo->pJitCache; // Offset 0x20 (32-bit)
PVOID dispatcher = pJitCache + offset;
/* Example: EXP+LdrInitializeThunk (x86) is the first function.
* Once BTDispatchRoutine returns, wow64ArmContext->Eip becomes
* LdrInitializeThunk (ARM64). */
BTDispatchRoutine(wow64ArmContext->Context, wow64ArmContext,
dispatcher);
}
} while (wow64ArmContext->ExecutionFlag != 1);
BTCpuSimulateComplete(btProperties, wow64ArmContext, bUnknown1, bUnknown2,
bUnknown3, bUnknown4, bUnknown5);
}
So the emulation begins, but which function or functions does it hand off the translation to and where?
Binary JIT Translation
Before I answer this question right away, let’s take a look at a translation of LdrInitializeThunk (the very first x86 function executed in an x86 process):
// Original x86 function
ntdll_775d0000!EXP+#LdrInitializeThunk:
775d1760 8bff mov edi,edi
775d1762 55 push ebp
775d1763 8bec mov ebp,esp
775d1765 5d pop ebp
775d1766 90 nop
775d1767 e9f4310800 jmp ntdll_775d0000!LdrInitializeThunk (77654960)
// JIT translation
00000000`0310a168 b81fcf9d str wfp,[x28,#-4]! // push ebp
00000000`0310a16c 2a1c03fd mov wfp,w28 // mov ebp,esp
00000000`0310a170 b840479d ldr wfp,[x28],#4 // pop ebp
00000000`0310a174 11420d29 add w9,w9,#0x83,lsl #0xC
00000000`0310a178 1107f929 add w9,w9,#0x1FE
00000000`0310a17c 97fffe4b bl 00000000`03109aa8 // (get jump function address)
00000000`0310a180 d61f0220 br xip1 // jmp ntdll_775d0000!LdrInitializeThunk
To answer the question, observe the following call flow which illustrates the process behind this translation:
BTCpuSimulate
└─ EmulateX86Function
├─ sub_180022c60
│ └─ sub_180022d20
│ │
│ ├─ sub_1800255b8
│ │ └─ BTXtaDeconstructOpcode
│ │
│ └─ sub_180023e10
│ ├─ BTXtaCreatePushPop // writes "str wfp,[x28,#-4]!" (@180042e04) push ebp
│ ├─ sub_180041fc0
│ │ └─ sub_180043e30 // writes "mov wfp,w28" (@180043eb4) mov ebp,esp
│ ├─ BTXtaCreatePushPop // writes "ldr wfp,[x28],#4" (@180042d1c) pop ebp
│ ├─ sub_180040070
│ │ └─ sub_18003fd30
│ │ └─ sub_1800473f8 // writes "add w9,w9,#0x83,lsl #0xC" (@18004756c) jmp ...
│ │ // writes "add w9,w9,#0x200" (@18004757c) jmp contd...
│ └─ sub_18003fd30
│ └─ sub_180040070
│ └─ sub_180046550 // writes "bl 00000000`03109aa8" (@180046580) jmp end...
└─ sub_180020928
└─ sub_180021008
└─ memcpy
Translation begins at BTCpuSimulate, and ends at a call to memcpy in sub_180021008. Many, many different functions are provided for translation. Sets of functions are designated for groups of similar x86 instructions. Together, they all perform the same general task:
1. Deconstruct an x86 instruction's bytes into a struct (BTXtaDeconstructOpcode)
2. Create an ARM instruction from the deconstructed data (BTXtaCreatePushPop)
3. Save the converted ARM instruction to a buffer (sub_180021008)
4. Repeat the above four steps for each instruction in a function
5. Save the converted function to the JIT-cache buffer
As I mentioned earlier, x86 to ARM translation is quite complex. The reason for xtajit.dll's relatively large size is partly due to the overall size of the x86 instruction set, and the need to emulate it in an ARM-based environment. Instructions are organized into groups: push and pop, for example, use some of the same functions during translation. In the end, the resulting instructions are saved to the JIT-cache buffer.
An x86 instruction's bytes are deconstructed in the following function:
// sub_180025b80
INT BTXtaDeconstructOpcode(PXTA_INFO xtaInfo, PXTA_INSTRUCTION xtaInstruction)
{
if (xtaInstruction->dword_12 >= 15)
goto loc_18005523c;
offset = xtaInfo->InstructionOffset; // From start of function (0xA4)
address = xtaInfo->InstructionAddr; // Absolute address of instruction
BYTE opcode = *((PBYTE)(address + offset)); // Instruction opcode
xtaInfo->InstructionOffset += 1; // Move to next byte
if (opcode > 0xFF)
goto loc_180025e18;
switch (opcode)
{
// push opcodes
case 0x50:
case 0x51:
case 0x52:
case 0x53:
case 0x54:
case 0x55: // push ebp
case 0x56:
case 0x57:
x8 = xtaInstruction->dword_1D;
x12 = opcode & 7;
if (xtaInstruction->dword_1D & 1 == 0)
goto loc_18002db6c;
xtaInstruction->dword_19 = 4;
xtaInstruction->dword_1C = 0x14;
xtaInstruction->TransferRegister = opcode & 7;
xtaInstruction->dword_24 |= (1 << (opcode & 7)) | 0x10;
xtaInstruction->GenerateArm64Func = BTXtaCreatePushPop;
xtaInstruction->dword_28 |= 0x10;
xtaInstruction->OtherFunc = sub_1800399b0;
xtaInstruction->dword_8 |= 1;
break;
// mov r32,r/m32
case 0x8b:
// ...
break;
}
return 0;
}
And translated below:
#define XTAR_X28 28
#define XTAI_PUSH_POP 0xb81fcc00
// sub_180042bb0
VOID BTXtaCreatePushPop(PXTA_DATA xtaData, PXTA_INSTRUCTION xtaInstruction,
PJIT_CACHE_FUNCTION jitCacheFunc)
{
// ...
if (/* do some checks... */)
{
// ...
// loc_180042de8
jitCacheFunc->Buffer[0] = XTAI_PUSH_POP | (XTAR_X28 << 5) |
gXtaContextRegs[xtaInstruction->TransferRegister + 0x6e];
}
// ...
}
The final bytes are then written to a buffer. This buffer is later copied to the JIT-cache buffer with memcpy (see call graph above) and finally executed.
So, we finally witness the actual translation, and very little of what’s involved is surprising. Given that the constituent parts of an instruction are organized into bytes and bit fields, translation simply becomes a matter of deconstructing and reconstructing the parts for a different architecture. But if we again recall this flowchart:
BTCpuSimulate
└─ EmulateX86Function
├─ sub_180022c60
│ └─ sub_180022d20
│ │
│ ├─ sub_1800255b8
│ │ └─ BTXtaDeconstructOpcode
│ │
│ └─ sub_180023e10
│ ├─ BTXtaCreatePushPop // writes "str wfp,[x28,#-4]!" (@180042e04) push ebp
│ ├─ sub_180041fc0
│ │ └─ sub_180043e30 // writes "mov wfp,w28" (@180043eb4) mov ebp,esp
│ ├─ BTXtaCreatePushPop // writes "ldr wfp,[x28],#4" (@180042d1c) pop ebp
│ ├─ sub_180040070
│ │ └─ sub_18003fd30
│ │ └─ sub_1800473f8 // writes "add w9,w9,#0x83,lsl #0xC" (@18004756c) jmp ...
│ │ // writes "add w9,w9,#0x200" (@18004757c) jmp contd...
│ └─ sub_18003fd30
│ └─ sub_180040070
│ └─ sub_180046550 // writes "bl 00000000`03109aa8" (@180046580) jmp end...
└─ sub_180020928
└─ sub_180021008
└─ memcpy
This entire procedure is all for merely four instructions (push ebp; mov ebp, esp; pop ebp; jmp). Imagine the overhead if this were to be performed for a full x86 binary.
Which brings us back to the use of XTA cache files: how are they created in the first place? This is done through independent background caching and compilation with the help of the caching service.
XTA Caching Service (XtaCache)
XtaCache is fairly small. Its purpose is to listen for module load notifications and to start the compiler if needed. Notifications are sent by the emulator (xtajit.dll) when a module is loaded. If necessary, the service will create a new XTA cache file in C:\Windows\XtaCache and start the compiler (xtac.exe). The compiler then performs a partial or full translation of the module's executable code. Once finished, it notifies the service, whereupon XtaCache saves the changes to the XTA cache file and closes all open file handles.
xtajit → XtaCache
Communication between xtajit and XtaCache is achieved using NtAlpcSendWaitReceivePort. Below is the call flow showing inter-process communication between the emulator and caching service:
BTCpuNotifyMapViewOfSection → sub_180014A80 → sub_180015AB0 → sub_180015C28 → sub_180015D80 → sub_180019010 → NtAlpcSendWaitReceivePort}
BTCpuNotifyMapViewOfSection is called every time a module is loaded (since NtMapViewOfSection is called every time a module is loaded). Eventually it passes a module file handle to NtAlpcSendWaitReceivePort, which sends the message to the compiler, xtac.exe.
XtaCache → xtac
Once XtaCache receives a module file handle from the emulation module, it must determine whether or not it the x86 module file needs to be compiled and written to a cache file. The following is the call flow leading to the execution of the compiler (xtac.exe):
1. NtAlpcSendWaitReceivePort receives module file handle from xtajit.dll
2. NtCreateFile opens module, creates a SHA256 hash from file name and PE header data
3. TpAllocWork creates callback function which will start compiler (xtac.exe)
4. Callback function formats XTA cache file name, and creates XTA cache file (NtCreateFile)
5. NtCreateSection creates section handle to be passed to compiler in a command line
6. CreateProcessAsUserW starts compiler with command line: xtac.exe -p [section handle]
The following function is responsible for receiving and responding to an ALPC message:
NTSTATUS ProcessIncomingAlpcMessage(PALPC_CONTEXT pAlpcContext,
PORT_MESSAGE msg, PALPC_MESSAGE_ATTRIBUTES msgAttr)
{
/*...*/
if (msg->TotalLength >= 80 && msg->MsgType == 2)
{
if (msgAttr->ValidAttributes & 0x10000000 != 0)
{
xtaCacheMsgAttr =
(PXTACACHE_MESSAGE_ATTRIBUTES)AlpcGetMessageAttribute(
msgAttr, 0x10000000);
if (xtaCacheMsgAttr->MsgType == 0xA1)
{
/* This will create the XTA cache file and start the compiler.
* hModuleNew is assigned a duplicate handle of hModule. */
result = PrepareJitCacheFileCompilation(
pAlpcContext->qword_0, unknown_0->qword_28,
xtaCacheMsgAttr->hModule, unknown_2, &hModuleNew);
if (NT_SUCCESS(result))
{
/*...*/
if (hModuleNew != 0)
{
xtaCacheMsgAttr =
(PXTACACHE_MESSAGE_ATTRIBUTES)AlpcGetMessageAttribute(
&newMsgAttr, 0x10000000);
xtaCacheMsgAttr->hModule = hModuleNew;
xtaCacheMsgAttr->MsgType = 12;
}
xtaCacheMsg->Result = STATUS_SUCCESS;
}
}
}
}
/*...*/
// Send response to xtajit emulator
NTSTATUS result = NtAlpcSendWaitReceivePort(
pAlpcContext->hAlpcPort, 0x10000, msg, newMsgAttr, NULL, NULL, NULL, 0);
// Close the module file handle
xtaCacheMsgAttr = (PXTACACHE_MESSAGE_ATTRIBUTES)AlpcGetMessageAttribute(
msgAttr, 0x10000000);
NtClose(xtaCacheMsgAttr->hModule);
return result;
}
Eventually a response is sent to the emulator (xtajit.dll) notifying it of the result, including any errors or anomalies in the message and message attributes.
Conclusion
It is worth noting that x86 emulation on ARM is a relatively new feature and is still in its infancy. Microsoft is one of the first to incorporate such a feature in its latest operating system. Supporting x86 emulation on ARM is a topic that is of increasing concern, as ARM processors become more and more ubiquitous and the concomitant demand for x86 emulation grows.
As traditionally x86- and x64-based operating systems attempt to migrate over to an ARM-based architecture, however, the drawbacks to emulation become more noticeable. Whereas x86 on x64 or ARM32 on ARM64 is a non-trivial matter of switching contexts and whatnot, x86 on ARM requires full-on CPU simulation in user mode – all in a single, infinite loop. The overhead encountered during such a process could be immense, depending on the size of the executing binary and the availability of cache files.
A gradually diminishing amount of free disk space could also become a concern, due to a growing cache file directory as the number of x86 binaries executed over time increases. Much remains to be seen as to what direction Microsoft and other software publishers will take in regards to x86 emulation. No doubt there are plenty of opportunities for innovation and improvement upon current techniques, and plenty to watch out for in future updates.
Resources
Below is a list of the materials used during research:
- Raspberry Pi 3 Model B+ (https://www.raspberrypi.org/products/raspberry-pi-3-model-b/)
- WOA Deployer for Raspberry Pi 3 (https://github.com/WOA-Project/WOA-Deployer-Rpi)
- Debugging Tools for Windows (https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools)
- IDA Pro 7.2 (https://www.hex-rays.com/products/ida/support/download.shtml)
- PuTTY 0.71 and PSFTP (https://www.chiark.greenend.org.uk/~sgtatham/putty/)
- Sysinternals Suite for ARM64 (https://live.sysinternals.com/ARM64/)
- WoW64 internals ...re-discovering Heaven's Gate on ARM (https://wbenny.github.io/2018/11/04/wow64-internals.html)
- ARM Instruction Set Reference Guide (https://static.docs.arm.com/100076/0100/arm_instruction_set_reference_guide_100076_0100_00_en.pdf)
- A64 Instruction Set Architecture (https://static.docs.arm.com/ddi0596/a/DDI_0596_ARM_a64_instruction_set_architecture.pdf)
