What to Expect

As I've shown in previous columns, Bash has a number of mechanisms for automating interactive processes. For example, the input redirection feature called a "here document" can place standard input for a command within a script, as shown in Listing 1.

01 for jobname in $jobs; do
02 for molecule in $mlist; do
03
04 g09 <<END >$molecule.log
05 @$jobname
06
07 $charge $spin
08 @$molecule.xyz
09
10 END
11
12 done
13 done

The Gaussian 09 (g09) computational chemistry simulation package uses a small input that tells it what type of predictions are desired and which molecule or reaction should be studied. A specific input file can obviously be created for a specific job, but placing commands like these into a script allows you to run a series of jobs on each molecule in a set in an automated way. This approach is clearly easier and more efficient than setting up tens or hundreds of input files manually. And, it makes running a new simulation on an existing molecule set trivial.

However, as a general solution, the here document technique has several disadvantages. First, not every interactive prompt will accept input from scripts in this way. For example, many password prompts and the key exchange prompt on an initial SSH connection are configured to accept input only from the terminal.

Second, when using this technique, it is not easy for the script to process the output from the command and make decisions based on it. More importantly, whereas the responses to multiple commands can be included before the end of the here document, the script cannot examine the responses to individual prompts to respond appropriately to them on the fly.

The Expect facility is a scripting environment designed specifically for automating true interactive processes. Expect scripts are examples of "chat scripts," which consist of a series of expected prompt-response pairs. Expect was written by Don Libes and is included with Ubuntu Unix. You can find Expect resources online [1] [2].

Introducing Expect

The best introduction to Expect is a simple script that automates the initial SSH connection to a host specified as the script's argument (Listing 2).

01 #!/usr/bin/expect
02
03 set host [lindex $argv 0]
04 set timeout 5
05 spawn /usr/bin/ssh aefrisch@$host
06 expect  "(yes/no)? " {
07   send "yes\r"
08   }
09 expect "password:"
10 send "secret\r"
11 expect "$ "
12 interact

The standalone version of Expect is automatically combined with a scripting language called Tcl (pronounced "tickle"). The first line in this example is the Tcl construct for assigning a value to a variable – here the variable host. The form [lindex $argv n] returns the nth argument to the script (numbering starts at 0), so the first command places the first script argument into the variable host. The second command assigns the value 5 to the variable timeout, which is a built-in Expect variable that sets a limit on how long an expect statement will examine the output (see below).

The remainder of the script is pure Expect, and it illustrates the three most important Expect constructs:

• spawn – Start a command and set up a conversation with that process.
• expect – Examine the command output for a string or expression; wait until it appears or the timeout period expires. Anything that appears in the output before the specified item is ignored, so you only have to look for the last part of what you want. An expect statement times out if it fails to find the specified item within the seconds corresponding to the value of the timeout variable. The default is 10 seconds, and a value of -1 disables timeouts.
• send – Send a string to the process started with spawn.

The script conversation (line number) in Listing 2 is:

• (5) spawn – Attempt to connect to the remote host with ssh.
• (6) expect – Wait for "(yes/no)? " in the output. This is the end of the several-line output about host key exchange.
• (7) send – Send the string "yes" plus a carriage return (\r) if "(yes/no)? " is encountered.
• (9) expect – Wait for "password:" in the output. This is the password prompt.
• (10) send – Send the password string plus carriage return.
• (11) expect – Wait for a dollar sign, the shell prompt.
• (12) interact – Transfer input control to the terminal, allowing the user to enter commands.

Notice the difference between first two expect statements. In the first one, the send command is subordinate to the expect command because it is enclosed in curly braces. The send command is run only if the expect command locates the specified item in the command output. If it times out, the send command is skipped. In contrast, the second expect and send commands are independent of one another. If the expect command fails to find a password prompt within the timeout period, it will end, but the send command will execute anyway.

An expect statement has the form:

expect {
  item1 {
    statements
    }
  item2 {
    statements
    }
  ...
  }

The curly braces immediately after the expect command are only needed when more than one search item is specified. I'll show examples of multi-item expect statements later in this column. Items are searched for in the order they appear in the statement, but the first matched item wins, and the statements following it are executed. Normally, the expect statement ends at this point.

Initializing a VM in the Cloud

Next, I will examine the script used to initialize a virtual machine in the Amazon Cloud. The purpose of this script is to set up passwordless SSH connections to the VM for both the default user, named ec2-user, and root. It sends the current user's public key to the remote host and adds it to the ~/.ssh/authorized_keys file for both remote users. It also changes the SSH configuration for the remote root user, as well as the options for the remote sshd server, restarting the latter afterwards. Once these activities are completed, neither the ssh nor scp command will require the use of the instance key or a password, and the current user will be allowed to run remote commands as root using ssh.

This script is called ec2_ssh_init, and Listing 3 shows its initial section.

01 #!/usr/bin/expect
02
03 set h [lindex $argv 0]
04 spawn /usr/bin/scp -i /home/aefrisch/.ec2/AEF.pem /home/aefrisch/.ssh/id_rsa.pub ec2-user@${h}:
06 expect "(yes/no)? "
07 send "yes\r"
08 expect "known hosts"
09 close
10 sleep 10

The script begins by saving its first argument into the variable h. It then starts an scp command and connects to it; the scp command uses the EC2 key file for authentication via the -i option. Next, the script looks for the host key exchange prompt, answers "yes" to it, and then waits for the response. When the script receives a response, it closes the connection and then waits for 10 seconds to allow the scp command plenty of time to complete.

The remainder of the script (Listing 4) runs a series of commands on the remote system by spawning an ssh command.

01 spawn /usr/bin/ssh -i /home/aefrisch/.ec2/AEF.pem ec2-user@$h
02 expect "$"
03 send "cat id_rsa.pub >> .ssh/authorized_keys; rm id_rsa.pub\r"
04 expect "$"
05 send "sudo tcsh\r"
06 expect "#"
07 send "cp .ssh/authorized_keys ~root/.ssh\r"
08 expect "#"
09 send "cd /etc/ssh; cp sshd_config{,.0}\r"
10 expect "#"
11 send "cat sshd_config.0 | sed -e 's/forced-commands-only/yes/' > sshd_config\r"
12 expect "#"
13 send "/etc/init.d/sshd restart\r"
14 expect "#"
15 send "exit\r"
16 expect "$"
17 send "logout\r"
18 close

All of the expect commands look for shell prompts, for either the ec2-user or root account. First, the script adds the key file it copied previously to the authorized_keys file for ec2-user on the remote host. It then starts a root shell via sudo. As root, the script copies ec2-user's authorized_keys file to the one for the root account. Then, it changes to the SSH server configuration directory, edits the master copy of the server configuration file (via sed), installs it as the active configuration file, and then restarts the sshd daemon.

The remainder of the script terminates the root shell, logs out from the SSH session, and then terminates the spawn command.

Communicating with a Device

Expect can be used to communicate with devices as well as processes. Listing 5 communicates with an atomic clock over its serial port to retrieve the current time.

01 #!/usr/bin/expect
02
03 set clock /dev/ttyS0
04 spawn -open [open $clock r+]
05
06 # set line characteristics for talking to clock
07 stty ispeed 300 ospeed 300 parenb -parodd cs7 hupcl -cstopb cread clocal -icrnl -opost -isig -icanon -iexten -echo -noflsh < $clock
11
12 send "o"
13 expect -re "."
14 send "\r"
15 expect -re "."
16 expect -re "(................*)"
17 exit

This script begins by setting the variable clock to the device file corresponding to the serial port where the clock is connected. It then begins a conversation with the device with the spawn command.

The stty command sets the serial line characteristics required for communicating with the device according to the manufacturer's specifications.

The actual data retrieval happens via two send-expect pairs; in this case, the script plays the part of the initiator, and the device is the responder. The script begins by sending the character "o"; the device will respond with a character. The -re option to the expect command says that the specified item is a regular expression rather than a literal string. The script then sends a carriage return, which tells the device to send the time. The time is returned in the form of an initial character that is not part of the time, followed by 16 or more characters. To capture the latter, the expect statement again uses regular expressions.

The first expression consists of a period, which will match the unwanted initial character. The second consists of 16 periods and an asterisk enclosed in parentheses. The latter will match any 16 or more characters. The parentheses are not literal characters to be matched but rather define this part of the regular expression as an entity that can be referenced later. The script ends at this point, but it automatically returns the most recent expression match to whatever process called it. In this case, the time string from the device will be returned.

Making Any Command Work Like top

The next script I cover, loop (Listing 6), lets any command function like the top command: The command is rerun periodically until a key is pressed. The beginning of the script processes the script's argument (if present) and sets the output refresh interval (which is also the Expect timeout period).

01 #!/usr/bin/expect
02
03 set cmd [lindex $argv 0]
04 if {"$cmd" == ""} {
05    send "Usage: loop command \[refresh\]\n"
06    exit
07    }
08 set tout [lindex $argv 1]
09 if {"$tout" == ""} {set tout 5}
10 set timeout $tout

This script illustrates a method for checking for the presence of required arguments via the Tcl if statement. The send command is used to print an error message when the first required script argument – the command to be run – is omitted. Before any spawn command, the expect and send commands interact with the script's standard input and output (respectively), so the send command can be used for messages from the script.

The script's second argument can be used optionally to specify the refresh interval for the specified command; by default, it is 5 seconds. The Expect timeout variable is set to this value.

The remainder of the script (Listing 7) runs the command and waits for a termination command, when the user presses any key.

01 set done 0
02 stty -echo
03 while {$done == 0} {
04    system /usr/bin/clear
05    system $cmd
06    stty raw
07    expect -re "." {set done 1 }
08    stty -raw
09    } # end while
10 stty echo
11 exit

The variable done is set to 0, and character echoing is disabled before the Tcl while loop. The while loop repeats as long as done's value is 0. The loop body runs two external Unix commands via the Expect system command. Unlike spawn, system runs the specified command without attaching to its input and output.

In the case of this script, the entire conversation takes place via the script's default I/O streams. Note that the square brackets within the error message (indicating that the second script argument is optional) must be quoted with backslashes (Listing 6) because square brackets have a special meaning to Tcl.

Once the user-specified command is run, the tty is set in raw mode. Thus, user input is processed character by character, rather than as individual lines (i.e., waiting for a carriage return). The expect statement looks for a regular expression consisting of any one character. When a character is received – because the user pressed a key – done is set to 1 and the script will terminate at the next iteration of the while loop.

If the expect statement times out, the value of done remains unchanged. Raw mode is then turned off again to avoid interference with the command output display, and the next while loop iteration begins. After the while loop ends, terminal echoing is restored and the script exits.

Expecting More than One Thing

Listing 8 shows another version of the ssh script I described previously. In this case, I have combined all of the various expect statements into a single statement, which also functions like a loop.

01 #!/usr/bin/expect
02
03 set host [lindex $argv 0]
04 set timeout 2
05 set user [lindex $argv 1]
06 if {"$user" == ""} { set user $env(USER) }
07 spawn /usr/bin/ssh $user@$host
08 expect  {
09   "(yes/no)? " {
10      send "yes\r"
11      exp_continue
12      }
13
14   "word:" {
15      send "whatever\r"
16      send_tty "Key exchange needed with $host\n"
17      exp_continue
18      }
19
20   -re "($|%|>) " { interact }
21
22   timeout {
23      send_tty "No response from $host\n"
24      exit
25      }
26   }

The script begins as usual by saving its arguments into variables (host and user) and setting the timeout. It also illustrates the use of a shell environment variable within an Expect script.

The form $env(NAME) returns the value of the environment variable NAME. In this script, I avoid hard-wiring the username into the spawn command and use the USER environment variable as a default remote username. In this way, the script is usable by an user and not just me. The expect command has four search items:

• "(yes/no)? " – The host key exchange prompt. If found, the response "yes" is sent, and the expect statement continues via exp_continue, instead of terminating as it would by default.
• "word:" – The password prompt, which the script answers with the password. The script next prints a message to the screen indicating that user key exchange has not yet been complete for this remote host. The command send_tty always sends output to the screen regardless of where standard output is currently directed. The expect statement again continues after executing the preceding statements because exp_continue is included.
• "($|%|>) " – A regular expression corresponding to various versions of the shell command prompt. When found, the script runs the interact command, which allows the user to take control of the SSH session. The expect command will end at this point.
• timeout – This keyword indicates that the Expect timeout period has expired. In this case, it usually means the remote host has not responded. The script prints a message on the screen indicating this, and both the expect statement and the script terminate at that point.

Automating Root Operations Securely

Although the preceding Expect script is quite useful, it does have one security problem: the presence of the user's password. This is not a good idea in general, and at the very least, the script file would need to be stored in a secure location and properly protected.

The next script overcomes this problem. It illustrates obtaining a password once from the user at the beginning of the script execution and looping over a list of items within an Expect script. Listing 9 shows the first part of the script, which processes the command arguments: the item to copy to remote hosts and the remote user to specify in the scp command (defaulting to root). It also sets the Expect timeout period. The script next reads an external file named xferhosts (in the current directory) to obtain the lists of hosts to which to copy the item. You can use this as a recipe for reading items from an external file via Tcl:

01 #!/usr/bin/expect
02
03 set xfer [lindex $argv 0]
04 set user [lindex $argv 1]
05 if {"$user" == ""} { set user "root"}
06 if {"$xfer" == ""} {
07    send_user "Usage: xfer item \[user\]\n"
08    exit
09    }
10 set timeout 10

set file [open "xferhosts" r]
set hlist [split [read $file] "\n"]
close $file

Next, the script prompts for the remote password, an operation that is especially important when the remote user is root. Note the use of the echo keyword to the tty command to enable/disable character echoing to the screen:

stty -echo
send_tty "_"
expect_tty "\n"
set pwd [string trimright          "$expect_out(buffer)" "\n"]
stty echo
if {"$pwd" == "" } { exit }

The set command that defines the pwd variable uses a Tcl construct to retrieve the contents of the Expect buffer.

The latter consists of all of the characters it received from the previous match through the current match. In this example, it's what the user entered at the password prompt. The script removes the newline from the end of that data interpreted as a character string.

Listing 10 shows the loop over the list of hosts from the file. The Tcl foreach command runs over each item in turn, and the if statement ensures that the hostname is not null.

01 foreach h $hlist {
02 if {"$h" != ""} {
03    spawn scp -r $xfer $user@$h:/tmp
04    expect {
05       "(yes/no)? " {
06          send "yes\r"
07          exp_continue
08          }
09
10       "word:" {
11          send "$pwd\r"
12          exp_continue
13          }
14
15       -re "." { exp_continue }
16
17       timeout { break }
18       }
19    }
20 }

The expect statement is similar to that in the previous script. It handles the host key exchange and password prompt if they occur, and it also terminates the current loop iteration if the timeout period expires via the break command. The other item in the expect statement is a regular expression corresponding to any single character.

The scp command sends output as it runs, and as long as characters are received, you know the command is still running. The regular expression will be matched, and the exp_continue command will run, which also has the effect of resetting the timeout counter. When no character is received for the length of the timeout period, then the loop iteration ends.

A Prompt with a Timeout

The Bash shell provides a way to solicit user input via the read command and save it in a variable, as in this example:

read -p "Prompt string: "          answer < /dev/tty

Such a prompt will wait forever until the user enters a response. Sometimes, you want the prompt to timeout if the user does not answer. With Expect, you can create a prompt with a built-in timeout period. The script to do so (Listing 11) begins by processing its arguments.

01 #!/usr/bin/expect
02
03 set prompt [lindex $argv 0]
04 set response [lindex $argv 1]
05 set tout [lindex $argv 2]
06 if {"$prompt" == ""} {
07   set prompt "Enter response"
08   }
09 if {"$tout" == ""} {set tout 5}

The default prompt is "Enter response", and the default timeout period is 5 seconds. The following code shows the part of the script that displays the prompt string and processes the user's response (if any):

send_tty "$prompt: "
set timeout $tout
set response ""
expect "\n" {
   set response [string trimright       "$expect_out(buffer)" "\n"]
   }
send "$response"

The script displays the prompt string on the screen, sets the timeout value, and assigns a default value to the variable response. The expect statement changes the value of response to any input that the user enters. The script ends by returning the value of the response to its caller. Listing 12 shows an example.

01 #!/bin/bash
02
03 # Example use of timed prompt script:
04 # The default response is '"'red'"'
05 color=`./timed_prompt "What is your favorite color? " red 2`
06 echo "Your color is $color"
07 echo
08 color=`./timed_prompt "What is your favorite color? " red 20`
09 echo "Your color is $color"

The following lines are the output from the script when a response is given to the second, but not the first, prompt:

$ ./use_prompt
What is your favorite color?
Your color is red
What is your favorite color? green
Your color is green
$

The script displays the default color (red) in the first message and the chosen color in the second.

I hope you have enjoyed this look at Expect. I am sure you will find many uses for this excellent automation tool for interactive processes.