Download and analyze log file automatically via scp, expect and php

Recently I want to test new features of EasyHA Monitoring Tool on log file analysis. The log file is stored on Linux server. Unfortunately I am not permitted to change any file on the server. I am permitted to read the file, of cause, I can download them from other Linux server.

The log file analysis is scheduled. It is launched automatically every morning. To download the log file, I have to input the password. How can I do this?

Step 1: Install Expect

Provided the log files is stored on server A, and I want to download it from server B. First I install expect on server B. Expect is a tool for automating interactive applications. Expect allows you to pass the password for the Linux login account from the program, instead of entering the password on the terminal.

The Linux distribution on the server B is centOS. So I just run the following command as root.

# yum install expect

And then run “ls /usr/bin/expect”, check whether the file exists.

Step 2: Build Expect Script to Download File

The script downloadfile.sh is listed below:

#!/usr/bin/expect

set timeout 10

set host [lindex $argv 0]

set username [lindex $argv 1]

set password [lindex $argv 2]

set src_file [lindex $argv 3]

set dest_file [lindex $argv 4]

set port [lindex $argv 5]

spawn scp -P $port $username@$host:$src_file $dest_file

expect {

  "(yes/no)?"

  {

  send "yes\n"

  expect "*assword:" { send "$password\n"}

  }

  "*assword:"

  {

     send "$password\n"}

  }

expect "100%"

expect eof

To test the script, please invoke the script like this:

$ expect downloadfile.sh  10.22.21.180 jboss thepwd /jboss/ log/cls-stat.log /home/lion 22

Step 3: Create Bash Shell Script to Download File and Analyze.

The script sample code:

expect downloadfile.sh  10.22.21.180 jboss thepwd /jboss/ log/cls-stat.log /home/lion 22

php runtime-exception-analyze.php /home/lion/css-stat.log

Note: To call expect script, you should use expect, other than sh,  the following command is incorrect.

$ sh downloadfile.sh  10.22.21.180 jboss thepwd /jboss/ log/cls-stat.log /home/lion 22

The script is named analyze.sh.

Step 4: Schedule Tasks on Linux Using Crontab.

To edit the list of cronjobs you can run:

$ crontab -e

This will open a the default editor to let us manipulate the crontab. If you save and exit the editor, all your cronjobs are saved into crontab. Cronjobs are written in the following format:

0 1 * * * sh /home/lion/ analyze.sh > /home/lion/analyze.log

The script will be executed every day 1AM.

Contact me

If you have any question or suggestion, please contact me. williams.voon # gmail.