Some googling later it seem to be a BUG with the kernel. The fix, suggested to either update the kernel, recompile the kernel or add some kernel parameters in GRUB to fix the issue. I decided to go with the kernel parameters because this was a production server and the downtime window was very slim.
The fix,
- vi /etc/boot/grub.conf
- Add to the end of the kernel line. “clock=pit noapic nolapic”
- reboot and check time. “watch date”
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/md7
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
title Red Hat Linux (2.4.9-e.3smp)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3smp ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3smp.img
title Red Hat Linux Advanced Server (2.4.9-e.3)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3 ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3.img
Source: Rexiology::WorkSource: Linux Kernel Parameters