Bug #6507

i386 makecontext(3c) needs to 16-byte align the stack

Added by Josef Sipek almost 3 years ago. Updated over 2 years ago.

Status:ClosedStart date:2015-12-11
Priority:NormalDue date:
Assignee:Josef Sipek% Done:

100%

Category:lib - userland libraries
Target version:-
Difficulty:Medium Tags:needs-triage

Description

Even though the i386 ABI specifies that the required stack alignment is 4 bytes, both gcc and cc assume a 16-byte alignment to facilitate SSE operations many of which require 16-byte alignment. (Note that the stack pointer must be 16-byte aligned just before the call instruction pushes the return address. Since stacks grow downward, this results in esp ending with 0xc just before the first instruction of the function executes.)

libc already uses 16-byte alignment for stacks created for threads (via thr_create, pthread_create, and the main thread). This was fixed in:

commit ebe15f48e9897d68d978938414a5c16cb0ceb049
Author: Roger A. Faulkner <Roger.Faulkner@Sun.COM>
Date:   Tue Sep 22 09:27:56 2009 -0700

    6881217 32bit stack frames should be aligned on 16-byte boundaries (for sse2 code)

Unfortunately, makecontext does stack setup on its own, and therefore it did not benefit from the above change.

In the wild, a mis-aligned stack will show up as a SIGSEGV on a memory referencing SSE instruction - even though the memory referenced is mapped and readable/writable. For example, I've seen PowerDNS recursor SIGSEGV in:

_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0xa: movl   0x8(%ebp),%ebx
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0xd: movdqu (%eax),%xmm0
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0x11:movl   0x10(%ebp),%eax
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0x14:movaps %xmm0,-0x18(%ebp) <=== HERE
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0x18:negl   %eax
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0x1a:pushl  %eax
_ZNKSt15_Deque_iteratorIcRcPcEmiEi+0x1b:leal   -0x18(%ebp),%eax
> <ebp::whatis
941e670 is in [ heap ] [823e000,9917000)
> <ebp-0x18::dump
           0 1 2 3  4 5 6 7 \/ 9 a b  c d e f  01234567v9abcdef
941e650:  405ef7fe 01000000 80128afe 04d8f6fe  @^..............

So, the memory operand to movaps is effectively 0x941e658. That is a problem because the Intel architecture manuals state:

When the source or destination operand is a memory operand, the operand must be aligned on a 16-byte (128-bit version) or 32-byte (VEX.256 encoded version) boundary or a general-protection exception (#GP) will be generated.

This matches what we see - %trapno is 0xd which is:

#define T_GPFLT         0xd     /* #gp  general protection fault        */

The attached test program demonstrates makecontext setting up insufficiently aligned stacks:

$ ./a.out 
 0: ss_sp = 8061150 ; gregs[UESP] = 806154c
 1: ss_sp = 8061151 ; gregs[UESP] = 806154c
 2: ss_sp = 8061152 ; gregs[UESP] = 806154c
 3: ss_sp = 8061153 ; gregs[UESP] = 806154c
 4: ss_sp = 8061154 ; gregs[UESP] = 8061550
 5: ss_sp = 8061155 ; gregs[UESP] = 8061550
 6: ss_sp = 8061156 ; gregs[UESP] = 8061550
 7: ss_sp = 8061157 ; gregs[UESP] = 8061550
 8: ss_sp = 8061158 ; gregs[UESP] = 8061554
 9: ss_sp = 8061159 ; gregs[UESP] = 8061554
10: ss_sp = 806115a ; gregs[UESP] = 8061554
11: ss_sp = 806115b ; gregs[UESP] = 8061554
12: ss_sp = 806115c ; gregs[UESP] = 8061558
13: ss_sp = 806115d ; gregs[UESP] = 8061558
14: ss_sp = 806115e ; gregs[UESP] = 8061558
15: ss_sp = 806115f ; gregs[UESP] = 8061558

c.c Magnifier - makecontext alignment tester (485 Bytes) Josef Sipek, 2015-12-11 05:05 PM

Makefile - test 2 makefile (237 Bytes) Josef Sipek, 2015-12-17 02:57 PM

asm.S Magnifier - test 2 asm (133 Bytes) Josef Sipek, 2015-12-17 02:57 PM

stacker.c Magnifier - test 2 C (1.38 KB) Josef Sipek, 2015-12-17 02:57 PM

History

#1 Updated by Josef Sipek almost 3 years ago

Here is a more comprehensive test.

Without the fix:

$ gmake
gcc -c -D_ASM -o asm.o asm.S
gcc -O1 -fno-inline -fno-omit-frame-pointer -c -o stacker.o stacker.c
gcc -o stacker asm.o stacker.o
$ ./stacker
tid=1 stack test:             
main      :47: esp:08039adc ebp:08039d08
d         :29: esp:08039acc ebp:08039ad8
c         :28: esp:08039abc ebp:08039ac8
b         :27: esp:08039aac ebp:08039ab8
a         :17: esp:08039a9c ebp:08039aa8
main      :49: esp:08039adc ebp:08039d08
testing different uc.uc_stack.ss_sp alignment...
argc=0 ss_sp = 8061650 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061651 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061652 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061653 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061654 ; gregs[UESP] = 8065250
argc=0 ss_sp = 8061655 ; gregs[UESP] = 8065250
argc=0 ss_sp = 8061656 ; gregs[UESP] = 8065250
argc=0 ss_sp = 8061657 ; gregs[UESP] = 8065250
argc=0 ss_sp = 8061658 ; gregs[UESP] = 8065254
argc=0 ss_sp = 8061659 ; gregs[UESP] = 8065254
argc=0 ss_sp = 806165a ; gregs[UESP] = 8065254
argc=0 ss_sp = 806165b ; gregs[UESP] = 8065254
argc=0 ss_sp = 806165c ; gregs[UESP] = 8065258
argc=0 ss_sp = 806165d ; gregs[UESP] = 8065258
argc=0 ss_sp = 806165e ; gregs[UESP] = 8065258
argc=0 ss_sp = 806165f ; gregs[UESP] = 8065258
testing different argc values...
argc=7 ss_sp = 806165f ; gregs[UESP] = 806523c
argc=6 ss_sp = 806165f ; gregs[UESP] = 8065240
argc=5 ss_sp = 806165f ; gregs[UESP] = 8065244
argc=4 ss_sp = 806165f ; gregs[UESP] = 8065248
argc=3 ss_sp = 806165f ; gregs[UESP] = 806524c
argc=2 ss_sp = 806165f ; gregs[UESP] = 8065250
argc=1 ss_sp = 806165f ; gregs[UESP] = 8065254
argc=0 ss_sp = 806165f ; gregs[UESP] = 8065258
custom stack via {get,make,set}context test:
d         :29: esp:08065238 ebp:08065244
c         :28: esp:08065228 ebp:08065234
b         :27: esp:08065218 ebp:08065224
a         :17: esp:08065208 ebp:08065214
exiting...

With the fix:

$ ./stacker                                                                  
tid=1 stack test:
main      :47: esp:08044f0c ebp:08045138
d         :29: esp:08044efc ebp:08044f08
c         :28: esp:08044eec ebp:08044ef8
b         :27: esp:08044edc ebp:08044ee8
a         :17: esp:08044ecc ebp:08044ed8
main      :49: esp:08044f0c ebp:08045138
testing different uc.uc_stack.ss_sp alignment...
argc=0 ss_sp = 8061650 ; gregs[UESP] = 806523c
argc=0 ss_sp = 8061651 ; gregs[UESP] = 806523c
argc=0 ss_sp = 8061652 ; gregs[UESP] = 806523c
argc=0 ss_sp = 8061653 ; gregs[UESP] = 806523c
argc=0 ss_sp = 8061654 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061655 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061656 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061657 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061658 ; gregs[UESP] = 806524c
argc=0 ss_sp = 8061659 ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165a ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165b ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165c ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165d ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165e ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165f ; gregs[UESP] = 806524c
testing different argc values...
argc=7 ss_sp = 806165f ; gregs[UESP] = 806522c
argc=6 ss_sp = 806165f ; gregs[UESP] = 806523c
argc=5 ss_sp = 806165f ; gregs[UESP] = 806523c
argc=4 ss_sp = 806165f ; gregs[UESP] = 806523c
argc=3 ss_sp = 806165f ; gregs[UESP] = 806523c
argc=2 ss_sp = 806165f ; gregs[UESP] = 806524c
argc=1 ss_sp = 806165f ; gregs[UESP] = 806524c
argc=0 ss_sp = 806165f ; gregs[UESP] = 806524c
custom stack via {get,make,set}context test:
d         :29: esp:0806522c ebp:08065238
c         :28: esp:0806521c ebp:08065228
b         :27: esp:0806520c ebp:08065218
a         :17: esp:080651fc ebp:08065208
exiting...

#2 Updated by Josef Sipek almost 3 years ago

  • Description updated (diff)

#3 Updated by Electric Monk over 2 years ago

  • % Done changed from 70 to 100
  • Status changed from New to Closed

git commit ceef08daa722b3a411ef838c03fb2fe6ada2f884

commit  ceef08daa722b3a411ef838c03fb2fe6ada2f884
Author: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Date:   2016-01-05T22:18:25.000Z

    6507 i386 makecontext(3c) needs to 16-byte align the stack
    Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom