Automatic

Expect is a natural and intuitive automation scripting language that operates in much the same way humans do when interacting with a system. You type in commands and expect a certain response to your command. When you receive the expected response, you enter another command and so on. Expect works in the same way, except you have to provide the script with commands and expected responses to those commands. Basically, you have to script out the entire two-way "conversation."

You can think of an Expect script as a dialog script written for two actors: a sender and a receiver. One of the more popular activities to automate is an SSH session between two hosts, in which one host is the sender (local host) and the other is the receiver (remote host). Being able to emulate every keystroke and create a true interactive session between two systems via a script is an exciting proposition.

Expect Setup

Most Linux distributions include Expect [1] as part of the available and installable software packages. In other words, you won't have to download and install from source code. Use your system's package manager to download and install Expect and any required dependencies or associated packages. For example:

$ sudo yum install expect

or

$ sudo apt-get install expect

Once you have Expect installed, you can begin writing scripts.

Creating an Interactive SSH Session

You must provide both sides of the conversation in your script because you're setting up an interactive system. Look at a few essential items before diving right into a script.

To make an Expect script executable as a standalone program, you must do two things: Make the script executable, and supply the path to the script for expect. The path on my system is: /usr/bin/expect; therefore, enter that path on the first line of your script with a preceding "shebang" (#!):

#!/usr/bin/expect -f

The -f switch tells Expect that it is reading commands from a file.

The spawn command spawns or launches an external command for you. In this case, ssh to a remote host (aspen):

spawn ssh aspen

Be sure to change the host aspen to the name of your remote host. When you SSH to a remote system, you are prompted for a password, and this password prompt is what you "expect" from the remote system; therefore, you enter that expected response:

expect "password: "

From the local side, you have to enter your password at the password prompt. To send anything to the remote system, it must be included in double quotes and must include a hard return (\r). Change PASSWORD to your password:

send "PASSWORD\r"

Again, you have to enter the expected response from the remote system, which in this case is a user prompt ($).

expect "$ "

Now that you're logged in to the remote system, you can begin your interactive session on that remote host as follows:

send "ps -ef |grep apache\r"

Output will appear as STDOUT. After the command has executed, you're returned to a prompt, so tell the Expect script that bit of information:

expect "$ "

Finally, send the exit command to the remote system to log out. Don't forget that hard return (\r):

send "exit\r"

The script in its entirety looks as follows:

#!/usr/bin/expect -f
spawn ssh aspen
expect "password: "
send "PASSWORD\r"
expect "$ "
send "ps -ef |grep apache\r"
expect "$ "
send "exit\r"

Change permissions on the script so that it is executable; for example,

$ chmod 755 script.sh

and try it for yourself.

Expect Caveats

If your script hangs and doesn't continue, try the command manually yourself and look for the response. If the remote system drops you to a prompt as its final act, then place that in your script (e.g., expect "$ "). Be sure you have entered the hard return (\r) inside the closing quotation mark in your send line. You might also find that your system needs two backslashes on the send line for a hard return (\r).

Sometimes Expect scripts execute too fast, and you won't see your expected response. If that happens, place a sleep command and a number of seconds for the command preceding it to wait for a response, or your data might be ignored.

For example, if you connect to a remote system and there's a delay in creating that connection, your script will continue to execute and fail because it sends commands before the remote system has time to respond.

You have to think about network delays, shell responses, and system timing when scripting in Expect. Like any scripting language, Expect has its quirks, but you'll find that it's an easy way to automate those repetitious keystrokes and procedures. The time you spend debugging your scripts is well worth the effort.

Autoexpect

Of course, some lazy system administrators take lazy to a higher level and even cheat at writing Expect scripts by invoking a shell "watcher" or recorder script named Autoexpect [2]. Once invoked, Autoexpect watches your every keystroke and records it to a file named, script.exp by default. You'll almost certainly have to edit and prune this script to achieve your desired results; however, it can save hours of script debugging to have an almost complete script from which to work.

If you simply run a freshly created Autoexpect script, it will likely fail because, if you issued a command that answers your request by displaying information to the screen, the script picks up that answer, too, and copies it into the script file.

For example, if during your Autoexpect session, you type, ls, the result of that command appears in your script.exp file as well. After you've created a few Expect scripts by hand, you'll appreciate the cleanup editing you have to do in an Autoexpect-created script. To install Autoexpect, issue the command:

$ sudo apt-get install expect-dev

Likely, you'll require many more dependencies for this feature, so prepare yourself for a slight delay while everything installs.

Interactive SSH Session

After installing Autoexpect and all of its required packages, you're ready to create Expect scripts automatically by stepping through the procedures you want to automate. Using the above example, SSH to a remote system and run a

ps -ef |grep apache

command and then log out. Invoking Autoexpect is easy:

$ Autoexpect
Autoexpect started, file is script.exp
$

Although it looks as if nothing has happened or is happening, every keystroke you type will be recorded into script.exp. Every STDOUT response you receive will also be copied into that same file. Your entire session is recorded – but not just recorded, it is also formatted in Expect script style. To stop recording keystrokes to your script, press Ctrl+D on your keyboard to stop Autoexpect and copy the buffer to your file.

The complete transcription of this simple procedure is very long and includes a lot of commentary from the author, Don Libes (Listing 1). You can see that you have a lot of cleanup to do before you distill this transcript to its essential parts. Autoexpect also changes permissions on the script.exp file so that it is executable.

01 #!/usr/bin/expect -f
02 #
03 # This Expect script was generated by Autoexpect on Thu Oct 11 15:53:18 2012
04 # Expect and Autoexpect were both written by Don Libes, NIST.
05 #
06 # Note that Autoexpect does not guarantee a working script.  It
07 # necessarily has to guess about certain things.  Two reasons a script
08 # might fail are:
09 #
10 # 1) timing - A surprising number of programs (rn, ksh, zsh, telnet,
11 # etc.) and devices discard or ignore keystrokes that arrive "too
12 # quickly" after prompts.  If you find your new script hanging up at
13 # one spot, try adding a short sleep just before the previous send.
14 # Setting "force_conservative" to 1 (see below) makes Expect do this
15 # automatically - pausing briefly before sending each character.  This
16 # pacifies every program I know of.  The -c flag makes the script do
17 # this in the first place.  The -C flag allows you to define a
18 # character to toggle this mode off and on.
19 set force_conservative 0  ;# set to 1 to force conservative mode even if
20                           ;# script wasn't run conservatively originally
21 if {$force_conservative} {
22         set send_slow {1 .1}
23         proc send {ignore arg} {
24                 sleep .1
25                 exp_send -s -- $arg
26         }
27 }
28 #
29 # 2) differing output - Some programs produce different output each time
30 # they run.  The "date" command is an obvious example.  Another is
31 # ftp, if it produces throughput statistics at the end of a file
32 # transfer.  If this causes a problem, delete these patterns or replace
33 # them with wildcards.  An alternative is to use the -p flag (for
34 # "prompt") which makes Expect only look for the last line of output
35 # (i.e., the prompt).  The -P flag allows you to define a character to
36 # toggle this mode off and on.
37 #
38 # Read the man page for more info.
39 #
40 # -Don
41 set timeout -1
42 spawn $env(SHELL)
43 match_max 100000
44 expect -exact "]0;khess@trapper: ~khess@trapper:~\$ "
45 send -- "ssh aspen\r"
46 expect -exact "ssh aspen\r
47 khess@aspen's password: "
48 send -- "PASSWORD\r"
49 expect -exact "\r
50 Linux aspen 2.6.32-43-server #97-Ubuntu SMP Wed Sep 5 16:56:41 UTC 2012 x86_64 GNU/Linux\r
51 Ubuntu 10.04.4 LTS\r
52 \r
53 Welcome to the Ubuntu Server!\r
54  * Documentation:  http://www.ubuntu.com/server/doc\r
55 \r
56   System information as of Thu Oct 11 15:55:28 CDT 2012\r
57 \r
58   System load:  1.09               Temperature:         40 C\r
59   Usage of /:   1.0% of 454.22GB   Processes:           168\r
60   Memory usage: 22%                Users logged in:     1\r
61   Swap usage:   0%                 IP address for eth0: 192.168.1.250\r
62 \r
63   Graph this data and manage this system at https://landscape.canonical.com/\r
64 \r
65 7 packages can be updated.\r
66 7 updates are security updates.\r
67 \r
68 New release 'precise' available.\r
69 Run 'do-release-upgrade' to upgrade to it.\r
70 \r
71 *** System restart required ***\r
72 Last login: Thu Oct 11 15:53:41 2012 from trapper\r\r
73 ]0;khess@aspen: ~khess@aspen:~\$ "
74 send -- "ps -ef|grep apache\r"
75 expect -exact "ps -ef|grep apache\r
76 www-data   555 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
77 www-data   556 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
78 www-data   557 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
79 www-data   558 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
80 www-data   559 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
81 khess    21504 21433  0 15:55 pts/1    00:00:00 grep apache\r
82 root     23171     1  0 Sep27 ?        00:00:28 /usr/sbin/apache2 -k start\r
83 ]0;khess@aspen: ~khess@aspen:~\$ "
84 send -- "exit\r"
85 expect -exact "exit\r
86 logout\r
87 Connection to aspen closed.\r\r
88 ]0;khess@trapper: ~khess@trapper:~\$ "
89 send -- "^D"
90 expect eof
91 khess@trapper:~$

The parts you needed for this script to execute correctly are shown in Listing 2 in my cleaned up version. You can see that complex prompts, such as

01 #!/usr/bin/expect -f
02 set force_conservative 0  ;# set to 1 to force conservative mode even if
03                           ;# script wasn't run conservatively originally
04 if {$force_conservative} {
05         set send_slow {1 .1}
06         proc send {ignore arg} {
07                 sleep .1
08                 exp_send -s -- $arg
09         }
10 }
11 set timeout -1
12 spawn $env(SHELL)
13 match_max 100000
14 expect -exact "$ "
15 send -- "ssh aspen\r"
16 expect -exact "password: "
17 send -- "PASSWORD\r"
18 expect -exact "$ "
19 send -- "ps -ef|grep apache\r"
20 expect -exact "$ "
21 send -- "exit\r"
22 expect -exact "$ "

expect -exact "exit\r
logout\r
Connection to aspen closed.\r\r
]0;khess@trapper: ~khess@trapper:~\$ "

have been shortened significantly to the following command:

expect -exact "$ "

The prompt still works because Expect looks for the last few characters in an expect line and not the entire string. You could shorten the line that expects the password prompt from:

expect -exact "password: "

to

expect -exact ": "

A word of caution against shortening your expect lines too much – the script becomes more difficult, not easier, to read and interpret. In the future, when you try to figure out what's going on, you might not see that ": " is a password prompt. Unless you are diligent at including comments in your scripts, you might spend hours debugging this shortened version.

Summary

To be perfectly honest, I only use Autoexpect when building an Expect draft script. To sit down and attempt writing Expect line-by-line just isn't appealing after being seduced and ruined by the ease of removing unwanted lines from an Autoexpect-created script [3].

Autoexpect makes using Expect fun and more intuitive by letting you perform a procedure one time instead of many. After discovering and using Autoexpect, my Expect scripting creation time and debug time has been cut by at least two-thirds. I suspect you'll have much the same return on your time when you begin to automate common tasks with Autoexpect.