X.Org Security Advisory, June 20th, 2006
setuid return value check problems on Linux systems

Overview

A lack of checks for setuid() failures when invoked by a privileged
process (e.g., X server, xdm, xterm, if installed setuid or setgid)
may cause the process to execute certain privileged operations
(file access) as root while it was intended to be executed with a
less privileged effective user ID, on systems where setuid() called
by root can fail.  This can be used by a malicious local user to
overwrite files and possibly elevate privileges in some corner
cases.

Vulnerability details

In Linux 2.6, it is possible that setuid(user_uid). can fail even
when invoked from a process running as root.

This is because there is a 'maximum processes' ulimit, which is
honoured by setuid(), seteuid(), and setgid().
These functions may fail because of this ulimit; if the return
value is not checked, then code which is assumed to be running
unprivileged, may in fact be running with uid 0.

Since ulimits on maximum processes are set by the kernel by default,
any Linux 2.6 system is affected by default..

Affected versions

X.Org versions 6.7.0 to 7.1 inclusive are vulnerable on systems
where setuid() called by root may fail. Older X11R6 versions are
probably affected also, but are not supported by X.Org.

Fix

Apply one of the following patches:

X.Org 6.8.2 
http://www.freedesktop.org/releases/X11R6.8.2/patches/
MD5 (xorg-68x-setuid.patch) = 0ce4435659d13cb75e409e92639f22eb
SHA1 (xorg-68x-setuid.patch) = d00815d19152da84de6677fcae04e6d96ee5db70

X.Org 6.9.0
http://www.freedesktop.org/releases/X11R6.9.0/patches/
MD5 (x11r6.9.0-setuid.diff) = 8e95fc06109d44ac280431d9cd8b41c9
SHA1 (x11r6.9.0-setuid.diff) = e576d725dd5f8d6c70df4b024adeecc5f7f90dc6

X.Org 7.0
http://www.freedesktop.org/releases/X11R7.0/patches/
MD5 (x11r7.0-setuid.diff) = a336e7e01a0876ec182c90277ab3e6fe
SHA1 (x11r7.0-setuid.diff) = 16a6a1c4a3527390caf53a45f4718ef378c90c14

X.Org 7.1
http://www.freedesktop.org/releases/X11R7.1/patches/
MD5 (libX11-1.0.1-setuid.diff) = 4b14554b64e4a8b1ec3c2b85cb5199b6
SHA1 (libX11-1.0.1-setuid.diff) = 6e2b6a43d394a474b8b731abb8d811625845421c

MD5 (xtrans-1.0.0-setuid.diff) = a3704e53fae7249379d842f6e626423a
SHA1 (xtrans-1.0.0-setuid.diff) = 82b913fe5ec96fd55afb8356ae338b90ed0f179b

MD5 (xorg-xserver-1.1.0-setuid.diff) = bd7f9871a9142197b8f45ad09969c6c5
SHA1 (xorg-xserver-1.1.0-setuid.diff) = e72b50c6434d429abaf0c13d9e78e1d467579fe9

MD5 (xdm-1.0.4-setuid.diff) = 24d467822a4dbf2f536ee419e0322f2d
SHA1 (xdm-1.0.4-setuid.diff) = 5b33a136ceffd40230fb65bf3cc635f8fc84e279

MD5 (xf86dga-1.0.1-setuid.diff) = 2a07eebe5796a86f307f9c1a3d0a2fa0
SHA1 (xf86dga-1.0.1-setuid.diff) = 4f184e186b280792878ec9118181067de7339f96

MD5 (xinit-1.0.2-setuid.diff) = 1377016ad0dd0e127419e4452d66a8ef
SHA1 (xinit-1.0.2-setuid.diff) = 816fa2fea8dbc1479ed594dace6281538de5e0ad

MD5 (xload-1.0.1-setuid.diff) = 9813ecc6d82157d1e5d19cf265af6ff9
SHA1 (xload-1.0.1-setuid.diff) = b14a6f911c2043052aa5006f3146fc5534705c2f

Thanks

This class of setuid() problems was first discovered by Roman
Veretelnikov in Vixie cron. 
Dirk Mueller and Marcus Meissner  provided a detailed analysis of the
issue affecting the X.Org source.