Exploring Expect: A Tcl-based Toolkit for Automating Interactive Programs

713 Pages·1994·4.28 MB·English

by  Don Libes

Exploring Expect
Don Libes
Beijing ⋅ Cambridge ⋅ Farnham ⋅ Köln ⋅ Sebastopol ⋅ Tokyo
DEDICATION
To Sue
A Note Regarding Supplemental
Files
Supplemental files and examples for this book can be found at
http://examples.oreilly.com/9781565920903/. Please use a standard desktop web
browser to access these files, as they may not be accessible from all ereader
devices.
All code files or examples referenced in the book will be available online. For
physical books that ship with an accompanying disc, whenever possible, we’ve
posted all CD/DVD content. Note that while we provide as much of the media
content as we are able via free download, we are sometimes limited by licensing
restrictions. Please direct any questions or concerns to [email protected].
Preface
This book is a tutorial for Expect, a software suite for automating interactive tools.
Expect has turned out to be very popular. This is good and bad. It’s good because
people are able to do things more easily than before and in some cases do things
that they would never have even tried. Expect is not simply another language. It is a
completely new type of tool that addresses problems that were not even recognized
as problems in the past.
The bad news is that since writing Expect, I’ve been plagued by people asking me
questions about it. While it was not originally intended to be a rich or complex
piece of software, it has admittedly become more sophisticated as various features
have been added. But more importantly, the nature of automating interactive
programs involves dealing with issues that are quite unlike the issues involved in
traditional programming.
While Expect comes with a “man page” (which is quite a misnomer at 25 pages), I
continue to get requests for information that does not properly belong there. Often
the requests are for examples. Sometimes they are simply for advice. This book is
an attempt to write down all of these things and to describe everything in the man
page in a tutorial fashion.
This book draws upon the thousands of Expect applications that people have
described to me, the common and not-so-common problems that people have
discussed with me—and explanations of the limitations—what you can’t do with
Expect.
I pull no punches. Expect is not meant to do everything for everyone, and I am
quite frank about discussing its limitations. While some may represent my own
limitations, others mirror my beliefs about how UNIX should work rather than
what can be accomplished. Indeed, I have resisted requests for enhancements that
do not add anything specifically useful to the original intent of Expect or that can
already be done more easily by another program. Expect is not yet-another
kitchen-sink language.
I am convinced[1] that Expect is very easy to use for the majority of applications
that users put it to. And I am convinced that many people can learn to do useful
Expect scripting in an hour or two.
Nevertheless, I recognize that the language is substantial. Using it is one thing.
Mastering it is quite another. Just reading through the 500+ pages in this book may
be an onerous task for some. However, as I said above, I believe that much of the
reason for the length of this book is due to the unique nature of automating
interactive programs. No matter how you accomplish it—whether using Expect, a
commercial product, or a home-grown set of kludges—automating interactive
programs is a task that is full of surprises. And while the examples in this book are
specific to Expect, the knowledge you gain from them can be taken and applied to
other interaction automation tools.
Indeed, Expect represents only the tip of the iceberg in the field of interactive
automation. Already, GUI automators are on the market. Eventually, hypermedia
automators will make their debut, combining simulations of human voice, images,
and all sorts of other sensory data.
Expect—Why another tool?
I initially did not view Expect as something that would last very long. It struck me
as solving a very simple problem but not in the best way possible. Indeed, I
originally wrote Expect as an experiment to demonstrate the need for a general
way of handling interaction automation. I expected that, having shown the utility of
it, the popular shells of the day would all soon incorporate these functions,
allowing Expect-like things to be accomplished from the shells without requiring
the use of another tool.
But to my surprise that has not been the case. Some shells (e.g., Korn shell, Z
shell) provide co-processes but offer no access to pseudo-terminals and no in-line
stream pattern matching. Most shells don’t even provide co-processes. Thus, Expect
remains very important to shell programmers.
Expect is also useful with environments that have limited or baffling Expect-like
functions. For example, Emacs, a popular editor, has actually had the ability to do
Expect-like processing for years. However, Emacs is a fairly unusual programming
environment and few people do real Emacs programming. Perl is another unusual
programming environment, again, with its own Expect-like functionality; however,
the implementation is difficult to use and many Perl programmers find it easier to
call Expect scripts for these tasks. Perl is also a large language, which makes its use
for Expect-like programming all the more formidable for the casual user.[2]
By comparison, Expect is simple. It really only does one thing. But it does it very
well. Everything in Expect is optimized to help you automate interactive programs.
That doesn’t mean it can’t do other things. These other things simply aren’t the
focus of Expect. You can use Expect to work with non-interactive applications.
Expect rests on top of Tcl which provides a very pleasant environment in which to
work. I have implemented and used a number of complex software packages that
use Tcl as a scripting language, and I look forward to using any other application
that does similarly.
Tcl—A Little History
Underneath Expect is Tcl, a small and simple language designed to be embedded in
applications. Much of the reason Expect appears so coherent and well thought out
is actually due to Tcl.
Tcl is a tour de force. It is powerful yet elegant, drawing a fine line between
primitives and extensibility, and between simplicity and overkill. Tcl allowed me to
concentrate on the application requirements of Expect. Tcl will allow you to call
and mix the Expect primitives in all sorts of interesting ways to control your
applications.
The intent behind Tcl matches Expect’s philosophy perfectly. Expect doesn’t need a
specialized language. Any generic but extensible control language would have
sufficed. However, at the time that I was thinking about writing Expect, no such
extensible control language existed. I was irritated at the thought of having to
create a language specifically for such a simple application. There was simply no
justification for designing yet another language.
I had been thinking about writing an Expect-like program after helping Scott
Paisley write a program to automate the initial login and command in a telnet
session. The program understood only a few simple commands. For instance, find
waited for a single fixed string to arrive. Every session ended in a permanent sort
of interact with a single telnet process. There were no variables and no flow
control commands such as if. And the program used pipes instead of ptys. The
program solved our immediate problem, but it had a lot of special-case coding and
I thought it could be generalized. This was in 1987.
For a long time I considered borrowing a shell, integrating the Expect primitives
into it, and then re-releasing it. However, shells are not intended to be used this
way and I did not have any interest in maintaining a shell once I had stuck my
hands in it. At that time, shells were renowned for being messy beasts. For
example, the C shell was well known for not having consistent and robust parsing.
And the Bourne shell had stimulated the Obfuscated C Code Contest—a contest
that actually celebrates and revels in torturous code. I wanted something else.
I was fortunate to be able to attend the 1990 Winter USENIX Conference in
Washington DC (January 22-26). I had been thinking about writing something like
Expect for several years, and I decided to go there and ask some wizards about
what they did for portable language facilities. To my delight, there was a talk at the
conference addressing that very topic.
John Ousterhout, a professor at the University of California at Berkeley, had
designed a language for embedding into applications—my very need! By the
middle of the talk, I knew I wanted to try using it. At the end of talk, when he said
it was freely available, I swooned.
Four days later, I had a copy of Tcl. I couldn’t believe it. It was not only everything
John had promised, it was also easy to use and well documented. By February 7
(eight days after first downloading Tcl), I had a working, albeit primitive, Expect. It
had the core of what Expect has today: send, expect, spawn, interact, plus a
logging function. At that time, the compiled Tcl code was about 48K, while Expect
added on another 12K. The idea of a control language being larger than the
application seemed peculiar, but it worked too well to go back to the old ways of
designing ad hoc interfaces.
Even with minimal functionality, it clearly suggested some interesting uses and I
thought that it might be nice to inform others. I had a USENIX Call-For-Papers
hanging next to my workstation. It said:
“The final deadline for receipt of submissions is February 7, 1990. Abstracts received after this deadline will
not be considered.”
It was February 8, one day after the deadline. I sat down, banged out the requested
extended abstract, and sent it in on the 9th.[3] It was accepted with top marks by all
the referees and appeared at the very next conference. No person was more
astonished by this rapid publication describing a Tcl-based application than John.
(John published another Tcl-related paper at the 1991 Winter USENIX Conference
—leading to Tcl papers at three consecutive USENIX conferences and a high
profile very early in its life.)
During 1990 and 1991, my local ftp server distributed over 2500 copies of Tcl, all
for the purpose of running Expect. I like to think that Expect was a catalyst in the
success of Tcl, but even without Expect, Tcl would have caught on eventually. Tcl
is now used by thousands of applications and millions of users. Some of the Tcl
extensions (Tk, in particular) stand on their own, and like Expect, allow people to
do things more easily than before and in many cases things that they would never
even have tried.
Since writing Expect, I’ve used Tcl to build several other applications and
extensions. While none of them are as general purpose as Expect, the results are
wonderful. Even for building specialized tools, Tcl is a joy to work with. I hope
never to go back to yacc and lex again.
While the focus here is Expect, not Tcl, I believe that this book is worth reading
even if you are learning Tcl for other reasons—perhaps contemplating putting it to
use in your own applications. Expect is a good example of what can be
accomplished with Tcl. Plus there are many techniques—such as the debugger and
the signal handler—that while invented for Expect, can be applied to just about any
other Tcl application. By reusing my efforts (and in some cases, avoiding my
mistakes), it will be that much easier when designing your own Tcl-based
applications.
Acknowledgments
I owe a large amount to the many people that used Expect and gave me excellent
feedback. There are literally hundreds of people who made contributions. Many
suggestions, although cavalierly made, developed into important aspects of Expect.
Some people donated sizable chunks of code while yet others debated with me on
philosophical aspects.
Thanks to Scott Paisley who initially sowed the seeds for the idea and then ended
up listening to me rave on about it for several years before I took any action. Scott
also wrote the dialback script that appears in Chapter 1 (p. 4).
I would also like to thank (in no particular order) John Conti, Mike Gourlay, Frank
Terhaar-Yonkers, Jerry Friesen, Brian Fitzgerald, Mark Diekhans, Kevin Kenny,
Rob Nagler, Terrence Brannon, Ken Manheimer, Steve Clark, Keith Eberhardt,
Brian Woodson, Pete Termaat, Sandy Ressler, Hal Peterson, Mike O’Dell, Steve
Legowik, Scott Hess, Mark Diekhans, David Vezie, Keith Hanlan, Rick Lyons, Jeff
Okamoto, Tom Tromey, Phil Shepherd, Michael Grant, Bob Proulx, Marty
Leisner, Ting Leung, Karl Lehenbauer, Przemek Klosowski, Bill Tierney, Steve
Summit, Rainer Wilcke, Alon Albert, Jay Schmidgall, Corey Satten, Konrad
Haedener, Tor Lillqvist, Pasi Kaara, Kartik Subbarao, Steve Ray, Karl Vogel, Seth
Perlman, Dave Mielke, Brian Bebeau, Bob Bagwill, Dan Bernstein, Pascal Meheut,
Rusty Wilson, Jonathan Kamens, Bill Houle, Todd Richmond, Mark Weissman,
Chris Matheus, Lou Salkind, Bud Bach, R.K. Lloyd, Chip Rosenthal, David D
‘Zoo’ Zuhn, John Rouillard, and numerous others whose names litter my log books
and change files. Of special note are Rob Savoye, who automated the Expect
configuration procedure; the people who wrote Autoconf on which Rob’s work
rests; Henry Spencer, who wrote the regular expression engine used by Expect and
Tcl; and Arnold Robbins, who wrote the POSIX date formatter.
And thanks to John Ousterhout for Tcl and for Tk. It is difficult to describe the
amount of work that he has put into making them possible, making them so usable,
and then making them freely available. John is a renaissance computer scientist. He
programs with great style—his code is a pleasure to read, and it is documented in a
thoughtful and readable fashion. Tcl is a great accomplishment, yet it is only one of
many for John. John is a continuing source of excellent ideas followed up by
gargantuan amounts of effort.
A significant amount of support for the development of Expect was provided by
the National Institute of Standards and Technology (NIST). The development of
Expect fit in well with the charter of NIST, an agency of the U.S. Department of
Commerce, whose goal is, after all, to promote commerce and assist industry in the
development of technology. Even though Expect was never specifically funded as a
separate project at NIST, I was permitted to work on Expect to the extent that it