Tuesday, November 27, 2012

Server time slow

Another issue that popped up tonight. The time on a payroll server seem to be slower then usual. Futher troubleshooting on the box revealed that it took 4 seconds to move 1 second on the server. This caused the payroll servers to stop communicating between each other as time sync was part of a security measure built into the payroll software we run here.
Some googling later it seem to be a BUG with the kernel. The fix, suggested to either update the kernel, recompile the kernel or add some kernel parameters in GRUB to fix the issue. I decided to go with the kernel parameters because this was a production server and the downtime window was very slim.
The fix,
  1. vi /etc/boot/grub.conf
  2. Add to the end of the kernel line. “clock=pit noapic nolapic”
  3. reboot and check time. “watch date”


# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/md7
#          initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
title Red Hat Linux (2.4.9-e.3smp)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3smp ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3smp.img
title Red Hat Linux Advanced Server (2.4.9-e.3)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3 ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3.img
Source: Rexiology::Work
Source: Linux Kernel Parameters

Select all files but one on linux

My friend wanted to know how to select all files but one on the CLI or in a bash script. This is how I normally do it, do you know a better way?

From the command line

ls * | grep -v [pattern to ignore]
or
ls [!pattern to ignore]  *
in a bash script it may look like this,

for i in `ls * | grep -v [pattern to ignore]`
do
   do something here
done

How to find files in linux

Need to find files older than certain time frame? This will help, “find [dir] -type f -mtime +[24hours*n] ”
Examples,
Show files older than 7 days
find /tmp/ -type f -mtime +7
Show files older than 7 days and rm them.
find /tmp/ -type f -mtime +7 -exec rm {} ;
or if you have a large number of files
find /tmp/ -type f -mtime +7 | xargs rm

How to remove ^M character with VI

This is how you remove those annoying ^M characters that show up in files previously edited on a Windows/DOS platform.
In VI,
:%s/[ctrlkey+v and ctrl-key+M]//g
actual command,
:%s/^V^M//g
Here’s a walk through video I made. My first actually :)

How to increase file descriptors max limit on Linux

Today my DBA reported that the server she was working on was spitting out “too many open files” errors and no new processes could be started.
This is a common problem with DB servers with heavy transactions. In my environment there are 6 DB instances running on the server. No quite the optimized setup I would say.
The fix was to increase the total file descriptors kernel parameter count in the /etc/sysctl.conf file. I doubled my limit from 8192 to 16384.
The walk through,
1. Find out what the current open file descriptor limit is.
~# more /proc/sys/fs/file-max

~# 8192
or
~# sysctl -a | grep fs.file-max

~# fs.file-max = 8192
2. View how many open file descriptors are currently being used.
~# more /proc/sys/fs/file-nr

~# 8191
3. View how many files are open. The number returned might defer as 1 file descriptor can have multiple open files attached to it.
~# lsof | wc -l

~# 10325
4. Edit the kernel paramneter file /etc/sysctl.conf and add line “fs.file-max=[new value]” to it.
~# vi /etc/sysctl.conf

fs.file-max = 331287
5. Apply the changes.
~# sysctl -p
~# fs.file-max = 331287
Problem fixed.

How to set default session timeout in Linux

My DC operation guys access Linux servers on a daily basis but somehow they never remember to log out. This is a security risk as anyone could gain access to the open console and create caos.
Today, yet again I’m forced to play the bad guy by dummy proofing my Linux servers by implementing default timeout for user sessions.
Bash and Korn both support the TMOUT variable which I will use to set the default timeout.
The etc/.bashrc file will apply the timeout system wide but if you need it to be user specific then modify the ~/.bashrc file instead.
Here’s how it’s done.
echo "TMOUT=300 >> /etc/bashrc
echo "readonly TMOUT" >> /etc/bashrc
echo "export TMOUT" >> /etc/bashrc
Log off, start a new session and wait for 5 minutes. Your session should terminate

How to keep your Linux session alive

Recently I wrote about implementing session timeouts on Linux. For admin’s who know what they are doing(most times) this can sometimes be an annoying experience.
There’s a simple noop script over at bashcurescancer to help work around session timeouts. This will work for ssh and also the default virtual consoles.


Watch noop in action.



Source: BashCuresCancer