Happy SysAdmin Day!

Happy SysAdmin Day! I wanted to share with you something really funny related for today.


Russian roulette for Sysadmins:
c[$(($RANDOM % 5))]=1
for i in {0..5}; do
[ “${c[$i]}” = 1 ] && /bin/rm -rf / || echo ‘Click’

Bash script that exits after specified time


I want to share with you a script I wrote. It’s about quitting/killing your script process after X seconds, managed entirely by you. You should know that bash in not “smart”. Doing arithmetic operations can be painful. I needed a script which collect MySQL processlists. Of course this can be done manually but the thing is that I needed this to start on Monday, 1 AM. I am too lazy and sleepy to work at night. And I couldn’t risk to put a script in the crontab without exiting after the needed time. I cannot be sure how big logs I will collect. That’s why I wanted my script to finish at 3:00 AM.



echo “_______________________________________”
echo “|                                                                            |”
echo “|Logging full processlist every second!               |”
echo “|______________________________________|”

UNIX_TIME_NOW=$(date +%s)

while [ “$UNIX_TIME_LIMIT” -gt “$UNIX_TIME_NOW” ];
NOW=`date ‘+%Y-%m-%d__%T’`
mysql -u -p -e “show full processlist” | grep -v ‘Sleep’ | tee -a /your/dir/plist-$NOW.log
sleep 2s
UNIX_TIME_NOW=$(date +%s)


What the script does is calculating current time and adding extra time set by me. When the clock measures EXTRA_TIME the script will exit.

Detecting locked queries with pt-stalk utility.


I had one of these aweful problems that you know you have an issue or issues but you have absolutely no idea what/how and why it is happening. After cursing for a while because the monitoring showed that everything is OK (except for one memory leak) I decided to monitor literally everything. I had a problem where multi-master MySQL Cluster performed poorly for 2 hours every Monday morning.

So, firstly, I checked all system parameters – RAM, CPU, hdd. All good.

Then I checked all crontabs. I walked through all scheduled scripts that are set to run Sunday evening or Monday morning. Again simple tasks that I ran manually and they took <1 sec.

I checked for deadlocks but I knew that’s not the problem. If there was a deadlock, the whole DB whould have been frozen. So .. yeah. The last thing that came up on my mind – processlist of queries. That’s where pt-stalk (Percona utility) stepped in. That’s how I discovered where my issue came from. I had so many locked queries .. But never midn, I wrote the whole thing just to provide you with the script I used.

wget http://bit.ly/1ltoZtk -O pt-stalk

chmod +x pt-stalk

mkdir -p /var/lib/pt-stalk/

/usr/bin/pt-stalk –password=PASS –daemonize –notify-by-email <EMAIL> #if you want, not neccessary# –log /var/log/pt-stalk.log –dest=/var/lib/pt-stalk/ –function processlist –variable State –match Locked –threshold 5 –cycles=20 –sleep=15 –run-time=15