Test iOS Apps with UI Automation: Bug Hunting Made Easy

217 Pages·2013·18.44 MB·English

by  Jonathan Penn

Early Praise for Test iOS Apps with UI Automation
Jonathan Penn is the perfect person to teach you how to test your iOS apps with
UI Automation. He is passionate about testing and has a lot of experience both
in iOS development and in the JavaScript required to make your tests sing. This
book is filled with techniques you’ll use immediately to make your iOS apps more
robust.
➤ Daniel Steinberg, Dim Sum Thinking
Automated testing is essential to delivering quality software. In this book Jonathan
lays out the details of how to use the UI Automation tools to build a test suite
that will keep your app in tip-top shape.
➤ Bill Dudney
Automated testing is a technique that every developer can benefit from. It’s a dry
topic, but Jonathan is able to bring clarity, comedy, and context to a very useful
though poorly documented tool.
➤ Josh Smith
Web-to-mobile converts often find it challenging to bring their automated-testing
habits with them, due in part to the fact that touch gestures and native UI widgets
are much harder to expose to automated testing than HTTP requests and HTML
output. Apple’s UI Automation is a big step in the right direction in this regard,
but it takes the guidance of a pioneer like Jonathan Penn to assemble a full
repertoire of developer tools to extend the reach of Apple’s tools and achieve a
high level of code confidence.
➤ Chris Adamson
My iOS development knowledge has been raised a notch by reading this book. It’s
great that I could take my existing JavaScript skills and learn how to apply them
to the field of iOS testing. I’ll definitely be using these techniques on the next app
I build!
➤ Stephen Orr, lead developer, Made Media
Being a big advocate of automated acceptance testing myself, in the past few years
UI Automation has become my weapon of choice for iOS applications. Jonathan
Penn is an absolute authority on the topic and his work has made my life easier
too many times to count. Now whenever people ask for advice on the topic, I can
just point them to this book, and it will answer all their questions and then some.
➤ Alexander Repty
Jonathan’s book is the best I’ve read about building great automated tests for
your iOS applications. The book has clear and comprehensive examples that help
you understand how to write great tests for your own projects. Not only does he
show you how to write solid tests; he shares best practices and techniques to
maintain a test suite as it grows. If you want to go beyond unit tests and automate
your app end-to-end, this book will get you started.
➤ Shiney Rossi, senior mobile engineer, Nest Labs
Jonathan Penn succeeds at opening up the world of UI Automation testing to
everyone with his new book, Test iOS Apps with UI Automation. From acceptance
testing to performance testing, Test iOS Apps covers all the steps to go from zero
to a full suite of automated tests that will help make your apps better. Sit down
and enjoy this engaging book to learn how to automate everything!
➤ Conrad Stoll, software engineer, Mutual Mobile
Test iOS Apps with UI Automation
Bug Hunting Made Easy
Jonathan Penn
The Pragmatic Bookshelf
Dallas, Texas • Raleigh, North Carolina
Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and The Pragmatic
Programmers, LLC was aware of a trademark claim, the designations have been printed in
initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer,
Pragmatic Programming, Pragmatic Bookshelf, PragProg and the linking g device are trade-
marks of The Pragmatic Programmers, LLC.
Every precaution was taken in the preparation of this book. However, the publisher assumes
no responsibility for errors or omissions, or for damages that may result from the use of
information (including program listings) contained herein.
Our Pragmatic courses, workshops, and other products can help you and your team create
better software and have more fun. For more information, as well as the latest Pragmatic
titles, please visit us at http://pragprog.com.
The team that produced this book includes:
Brian P. Hogan (editor)
Potomac Indexing, LLC (indexer)
Candace Cunningham (copyeditor)
David J Kelly (typesetter)
Janet Furlow (producer)
Juliet Benda (rights)
Ellie Callahan (support)
Copyright © 2013 The Pragmatic Programmers, LLC.
All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form, or by any means, electronic, mechanical, photocopying,
recording, or otherwise, without the prior consent of the publisher.
Printed in the United States of America.
ISBN-13: 978-1-937785-52-9
Encoded using the finest acid-free high-entropy binary digits.
Book version: P1.0—August 2013
Contents
Acknowledgments . . . . . . . . . . . vii
Introduction . . . . . . . . . . . . . ix
1. UI Automation Overview . . . . . . . . . . 1
1.1 Capturing Our First Script from the Simulator 2
1.2 Finding Our Way around UI Automation 6
2. Testing Behavior with UI Automation . . . . . . 11
2.1 Talking to the UI through JavaScript 12
2.2 Reporting Errors 17
2.3 Verifying that the Test Does What It Says 19
3. Building a Test Suite . . . . . . . . . . 23
3.1 Testing with a Modal Alert View 23
3.2 Importing Script Files at Runtime 27
3.3 Testing Results from a Live API 29
3.4 Grouping Test Steps and Their Output 32
4. Organizing Test Code . . . . . . . . . . 41
4.1 Starting a JavaScript Toolbox 41
4.2 Describing the App with Screen Objects 48
4.3 Reusing a Generic Screen Prototype 52
4.4 Converting Our Test Suite to Screen Objects 54
5. Maps, Gestures, and Alerts . . . . . . . . . 59
5.1 Exploring the Map 59
5.2 Identifying Elements with Accessibility APIs 64
5.3 Testing with Gestures 67
5.4 Advanced Alert-Handling 72
•
Contents vi
6. Strategies for Testing Universal Apps . . . . . . 79
6.1 Universalizing an Application 79
6.2 Finding Elements in the New Idiom 81
6.3 Building an iPad Test Suite with Reusable Pieces 84
6.4 Searching the Element Tree with Predicates 91
6.5 Advanced Predicate Usage and Beyond 98
7. Automating Performance Tests . . . . . . . . 101
7.1 Setting Up Custom Instruments Templates 101
7.2 Capturing Steps to Reproduce a Memory Leak 103
7.3 Triggering Simulator Memory Warnings with
AppleScript 109
7.4 Stress Testing 114
8. Setting Up Application Data . . . . . . . . 121
8.1 Seeding Data in Xcode with Application Data Packages 121
8.2 Seeding Data Dynamically with a Factory 128
8.3 Choose Your Own Adventure with Environment
Variables 135
8.4 Hiding Test-Setup Code from Release 139
9. Stubbing External Services . . . . . . . . . 147
9.1 Choosing a Geographical Location 147
9.2 Faking a Network-Service API 150
9.3 Wrapping Service APIs in a Facade 156
9.4 Stubbing a Facade with Data in the App Bundle 159
10. Command-Line Workflow . . . . . . . . . 165
10.1 Practicing with the Raw Commands 165
10.2 Automating the Build-and-Run Process with Rake 172
10.3 Reading Environment Variables from Script Files 178
10.4 Resetting the Simulator to Test Permissions 181
10.5 Running Tests on an Attached Device 185
11. Third-Party Tools and Beyond . . . . . . . . 191
11.1 Enhancing UI Automation 191
11.2 Testing outside the Box 194
A1. Bibliography . . . . . . . . . . . . 199
Index . . . . . . . . . . . . . . 201
Acknowledgments
First, I want to thank my inner circle of authors, who encouraged me to go
through the pain to write a book in the first place. Daniel Steinberg, Bill
Dudney, Joshua Smith, and Jason Gilmore—thank you. I would have been
lost without your example and your terrifying stories.
Thanks to all who submitted feedback and errata throughout the beta process,
and specifically those who slogged through the tech reviews and took the time
to write up the awesome feedback: Chris Adamson, Heath Borders, Jayme
Deffenbaugh, Jason Gilmore, Jeff Holland, Ben Lachman, Kurt Landrus,
Kevin Munc, Mark Norgren, Stephen Orr, Julián Romero, Shiney Rossi, Joshua
Smith, Daniel Steinberg, Conrad Stoll, Elizabeth Taylor, TJ Usiyan, and Alex
Vollmer.
Thanks to CocoaConf for giving me all those opportunities to practice the
material in this book—over and over.
Thanks to the team at The Pragmatic Programmers for the resources they
provided and for letting me prove myself. Special thanks to my editor, Brian
Hogan, for wisely convincing me to scrap the first draft of the book and for
fielding my incessant questions.
To my parents, who fed my famished curiosity. To my daughter, Niah, who
thinks I work at a coffee shop for a living. To my son, Ian, who thinks I know
what I want to do when I grow up. And to my partner, Colleen. She put up
with my swinging moods and sleepless nights and surely shed more sweat
than I did.
For great justice.
report erratum • discuss
Introduction
We iOS developers have a lot on our minds. We want to build useful and bug-
free software for our customers while keeping up with Apple’s fast pace.
Software development is fraught with trade-offs and, unfortunately, testing
our software is often traded away in the crunch time before a release date.
So what’s the best way to hunt for bugs in our apps? We spend a lot of our
own time manually launching and walking through the features one by one,
tapping, swiping…over and over again. This book helps us find a better way.
What Can We Do About It?
Nothing will replace the spot-checking ingenuity of a human tester, but we
can certainly automate the important common tasks and free ourselves up
to focus on other things. We want to use automated tests to raise confidence
while we keep forging ahead and to give us useful information when something
goes wrong.
In this book, we’re going to focus on testing by scripting interactions through
the user interface. This is known as full stack or integration testing in some
circles. We’re launching the whole app, tapping and gesturing, waiting for
animations, and reacting to results from the screen.
We’re going to be strategic with how we apply these tests. Automation testing
is a powerful way to smoke out bugs, but it’s not without its limitations. These
kinds of tests are slow and it’s not feasible to test every edge case using this
technique. We’re not going to cover effective lower-level testing strategies such
as unit tests—for more information about that, you’d want to read Graham
Lee’s book Test-Driven iOS Development [Lee12], or Daniel Steinberg’s book
Test Driving iOS Development With Kiwi [Ste12]. Here, we’re looking to test
deep slices of the application while answering the question "Did we wire the
components correctly?"
We have two ultimate goals with these tests. First, we want to verify correct
behavior with acceptance tests that list the steps a user would take and the
report erratum • discuss