Author: Alan Page

As I type this, I realize I have a mildly embarrassing gap in my testing career. I don’t recall ever writing any automation for mobile applications.

Maybe that’s not exactly true – nearly 20 years ago, I wrote some automation in C++ to test Windows CE, and some applications that ran on it…and technically that was an OS used to power Windows Phone for a while (among other devices) – so I guess the gap is that I have never played with Appium. We used Appium when I worked on MS Teams for (some?) mobile automation, but I wasn’t directly involved.

Today is my chance.

TestProject Experiment Number Two

After my success playing with recorded test automation (I really want to call this recorded test assistance) for web, I decided to try the same thing using TestProject to test a mobile application.

Getting Started – The Setup

I was mentally prepared to install the Android SDK (or at least ADB), and spend a bunch of time getting a connection set up, but holy crap – was it always this easy?

I already have the TestProject agent running on my computer (for running the web tests), so all I had to do was select Android as my target, and everything else just worked.

Overall, the setup wizard is extremely easy to follow, and worked amazingly well. My phone was already in developer mode, so all I had to do was configure USB debugging, (optionally) disable a few animation features (for testability), and the TestProject web app connected to my phone and showed a mirror on screen in seconds. I thought for sure that I’d spend an hour or more getting this going, but I was wrong.

Granted – I don’t have much experience here other than half a dozen sessions debugging over ADB, so this sort of thing may be the norm now, so all I can say is setting up a remote development session can’t get much better than this.

The wizard is pretty much the same as the web wizard with one cool addition. Once the connection is configured, I can browse the device packages to choose which one I want to test:

I decided it would be fun to play with some automation for the Twitter app. There’s an option to reset the app before starting, and there’s even a default step added to reset, but for my experiment, I deleted that step.

As in my tests on angryweasel.com, I’m just going to record selecting the top 4 elements from right to left and see what happens.

Note that the steps here are able to pull more friendly names from the package, so the test steps are pretty readable as is.

Creating Tests

Using the same test strategy as I did with web tests, I then added validation to ensure that the correct screen showed in the right place.

For each screen, I just used the “Validations” design element to confirm the existence of a particular control. There are probably better ways to do this, but for my quick test, it’s fine.

Similar to what I wrote for web tests, I have some actions with validation.

The Element Locator

If you’ve written automation in Appium or Selenium (or other frameworks that rely on IDs, XPaths, etc.) you are already familiar with the game we play to find elements.

One of the other handy features of TestProject is a simple, but useful element locator that makes it easy to discover if the element you’re working with is currently visible. In the example below, I’ve clicked the Find button in the test designer to show me the Twitter Home button.

Pretty cool!

As a nice aside, notice that the screenshot above (unlike the other screenshots), my phone screen fills the emulator (vs. the other screenshots where it fills 70% or so of the emulator screen). The TestProject team fixed the issue practically moments after I reported it.

A Few Words About Automation

Let me be the first to point out that this isn’t particularly great automation, and that the validation I’m doing probably isn’t what needs to be tested at the UI level for Twitter.

When you write UI automation, use it to find bugs that can only be found at the UI level. If it’s a special visual effect that can only be detected via human eyes, then use human eyes to validate the effects.

Of equal importance is your strategy. Don’t start by deciding what to automate. Instead, start with what you want to test – and then use automation to assist your testing. I’ll refer again to my article on the TestProject site that expands on these ideas in more depth.

All that said, it seems to me that I could very well use TestProject for proper targeted UI tests where they were needed, and that it would serve me well.

Observations

I’m still surprised by how easy it was to get up and running testing a mobile app, and the test steps were pretty straightforward and intuitive.

One of the areas where mobile automation really helps is in making sure applications work across a variety of devices. I only have one phone, but TestProject does support working with multiple connected devices to a single desktop. You just connect a USB Hub with a suite of devices, and you have an instant test lab. It’s also worth mentioning that an account can have multiple agents assigned to it. This enables executing tests not only on your desktop (and devices connected to it), but also on remote agents (and devices connected to them) in different locations anywhere in the world. It could be fun to see if TestProject could target a big virtual and real mobile test farm like BitBar, but there’s plenty of flexibility and potential right out of the box.

As with the web tests, I’d love to see a way to view / edit the underlying code, but my curiosity was less with mobile apps – perhaps because my knowledge of Appium is much less than my knowledge of Selenium.

Overall, it seems like a pretty usable and robust choice for most mobile automation tasks.

One more investigation to go where I’ll see how TestProject does with some coded tests.

Share this:

Like this:

Jerry Weinberg passed away earlier this week, and I’ve taken some time to re-read parts of many of his books, where I’ve been constantly reminded of his leadership knowledge. I grabbed this quote from Becoming a Technical Leader to share“People don’t become leaders because they never fail. They become leaders because of the way they respond to failure.”

I spent a chunk of my early career in globalization testing (which reminds me that my last wikipedia edit was to fix this section of the software testing article). Netflix, in yet another valuable post, describe their approach to pseudo-localization.

Share this:

Like this:

I posted this other places, but forgot to put it here. If you, or someone you know is interested an capable of embracing and driving Modern Testing principles with a fantastic group of developers, producers and other technical staff, and are in (or open to moving to) Austin, Texas, check out or share this position on my team.

Share this:

Like this:

I wrote an article on the perils of Test Automation (Something’s Rotten with Test Automation) for TestProject last year. I’ve had a bit of time since then to look into their product since then (currently in beta), and thought it was worth sharing a bit about it, as I think it’s a pretty good product.

This post is a walk-through of my experience creating a simple web test. I plan to write a few follow-up posts to cover mobile device testing and finally a bit on my experiences playing with their Java SDK.

Please note: I am accepting a small stipend from TestProject for posting this, but the article ideas and content are entirely mine, and I would not write this if I didn’t endorse the product.

What is TestProject?

On the surface, TestProject is yet-another web and mobile automation tool. It supports recorded tests (but better than any test recording frameworks I’ve used before), as well as coded tests, which can be uploaded and run from the TestProject web site.

Yet Another Automation Framework?

We’ve had a recent resurgence of web automation frameworks, but for reasons I’ll describe in this post – as well as in the next two follow-ups, TestProject has a lot of promise, and a lot of potential. It encapsulates Selenium and Appium (no separate download needed), will eventually support at least five programming languages, and makes it extremely simple to share and schedule tests.

Better yet, the price is right. For most users, it’s FREE. The free package includes 5 agents, SDK support, 30 days of history, and some other goodies. For those who need the Professional version, it’s $8 a month per agent – which is still ridiculously affordable.

But…The Weasel Doesn’t Like UI Automation

If you’ve read my posts, seen my talks, or followed me on Twitter, you’ve likely seen me speak out against UI automation. To be clear, I’ve spoken out (and will speak out) against bad UI automation, and against trying to do too much testing in the UI.

A fewtargeted tests that verify critical parts of the UI are not just something I support, but a pretty damn good idea. I think TestProject is a good solution for this task for many teams.

Recorded Tests – A Walkthrough

I have always been (very) leery of recorded tests, but somewhere when I wasn’t paying attention, the tools got pretty good – or at least much better. It’s still possible to create some pretty bad tests via recording tools, but I’ve learned that at least it’s possible now to create reasonable tests with recording tools.

To try things out, I made up a (slightly convoluted) test scenario where I wanted to make sure that all of the menu pages on angryweasel.com went to the right page.

My “test plan” is to click each link, and make sure it goes to where it’s supposed to go (probably by verifying an expected element exists).

Step 1 – Create a new project

Step 2 – Create a Test

At this stage, I could go straight into “Design” mode – but we’ll go here later after we get the basics done via the recorder.

The TestProject Agent

One thing I really like about TestProject is the TestProject agent – it’s cross-platform (I’m testing it now on my Ubuntu system), and it encapsulates Selenium or Appium as needed for the target platform. Once the agent is up and running, the TestProject web site will see it’s running and connect.

When the recorder page comes up, it looks (something) like this:

I moved the recorder window so I could see the menu items, and then clicked them from right to left. That gives me a not-very-readable output like this:

I can play this back, and it runs well. But at this point, there are two BIG problems. The first issue is that there’s no validation. We’ll add that in a few minutes, but first, I’d like to document the “Click A <Link>” names and make them a little more debuggable.

My preference would be to rename these, but that’s not an option yet in TestProject – but to aid in debugging, I can at least add a comment.

One cool thing to note here is that I can view and edit the current selector, and select by ID, Classname, Tagname, Name or (the default) Xpath. Consistently named labels are essential for writing maintainable automation, so this is something that I think will be pretty handy.

I won’t show it here, but I ended up changing the selection method for each validation, and all worked exactly as I expected.

Validation

I found creating the validation step to be pretty straightforward too. Just create a new test step, and have it verify that a specific element contains specific text. For those of us who have written coded web automation, it’s pretty easy to picture what Selenium code is running under the hood. I also think that this is indirectly a pretty reasonable way for non-coders to learn how Selenium works. Probably not TestProject’s intent, but I think there’s potential there.

I can run my tests from here, and the green bar on the left side shows that all of my test steps passed.

From here, I created a Job on the TestProject site, set the job to run on Chrome and Firefox, and added this test to the job.

Test Results

Not super exciting results, but certainlynot bad results for 30 minutes of “test development” with a new tool.

Parameters

I mentioned above that I gave every action a one second timeout (note that it doesn’t wait or sleep a second – one second is just the length of time I’ve instructed the framework to wait for an element.

What happens if I decide the timeout should be 2 seconds -or half a second. It’s preferrable to change things like this in one place, and fortunately, TestProject allows you to add parameters. I added one called my_timeout, and now, if I ever need to change it, I can change the timeout in once place.

Failures

When you write tests, it’s a good idea to test what happens when they fail. I changed the “Validate “About Angry Weasel” test to look for the test “About Angry easel”. As expected, the test failed, and gave me reasonably actionable information to debug the failure from.

Final Notes (for now)

Overall, I didn’t find much to hate. Some things I’d like to see in future updates include:

Ability to rename steps for easier debugging

There are a lot of “are you sure you want to do this” type dialogs that I already feel a need to suppress

Given that I can mentally “see” what the underlying code looks like, it would be nice to generate code as well for more subtle tweaks.

As I said in the intro, I have a few more posts planned to walk through more of what I learn while learning the tool, but happy to hear your feedback on the tool or on my approach in using it if you have them.

Share this:

Like this:

A few episodes ago on AB Testing, we talked about the Theory of Constraints and related topics. A lot of us frequently see “QA” or “Test” as a bottleneck, even on Agile teams – where several changes may be waiting for signoff from the designated quality or testing specialist before being deployed or marked as done.

If you listen to the podcast, you know I’m against this level of specialization. I’m all for people who have specialization – but things fall apart if they’re the only one who can do a “thing”. I like to imagine the extreme case of specialization where one person just writes design docs, one person writes code, but only internal functions; because another person writes external APIs, and another person tweaks all of the code for performance, and another person writes test cases, and another person runs them; unless they’re automated, then it’s another person…

At this point some of you are laughing because this sounds ridiculous, but others are thinking, “that’s pretty close to my current team – wtf?”

Today, I made a quick (or at least I planned for it to be quick) run to Home Depot to pick up a part. I knew exactly where the part was (aisle 40 – don’t ask how I knew that), so I jogged back, grabbed the part, and went to the front. I jumped in the self checkout line, scanned the part, shoved my credit card in the slot, and got ready to grab my receipt.

But the price was wrong. Or it was right, technically, but I was charged twice. Luckily, there was a cashier right there, so I explained that I scanned once, but was charged twice.

She stared for a moment, and said, “I can’t fix that. You’ll need to go to customer service.” Apparently, I needed the specialist. Or another specialist at least.

I went down to the customer service desk, where I was behind a customer with a cart full of stuff and a lot of questions. I waited 10 minutes for my meeting with the specialist, but it wasn’t moving very quickly, so I walked out of the main store area to the Returns counter to ask the returns specialist about it (the Returns counter is past the check out area to make (I assume) it easier to return stuff without entering the store.

Eventually, the Returns Specialist was able to help me, but he had a very difficult time. Apparently, he knew how to do returns, but since I wasn’t technically returning anything he thought it seemed like a problem that Customer Service should help with.

I try to be a quick thinker, and since my quick trip to the store was pushing twenty minutes in-store time, I grabbed my receipt and part, jogged back to aisle 40 to grab a duplicate part, returned to the counter with both, and I was able to return the item I didn’t want, and all was well.

On one hand, this is just another story of customer service gone south, but at the core, I see it as a flaw of too much specialization – and I think it’s important that we all recognize when we see these things around us. If your software team is ever blocked by a specialist – whether it’s waiting for a tester, waiting for a decision, or even waiting for a deployment, there’s a lesson to be learned, and a potential improvement to be made.

Like this:

I’ve been reading Bad Blood (currently about 30% of the way through). So far, it’s a pretty damning investigative review of Theranos, but I’m still waiting for the plot twist :}. Wikipedia tells me that Jennifer Lawrence will star in the film adaptation.

This (yet another) article on the failure of open plan offices is making the rounds. Buried in the end of the article is this quote:

“And probably least bad is small team rooms of fewer than ten people, preferably fewer than six. I’ve sat together with really small teams before and that’s been OK. Some people who don’t like the open office at all might even still enjoy this configuration.”

This setup (a “team room”) has been, in my experience the hands down winner of the most productive work environment I’ve ever worked in – and I hate that it’s lumped into the valid arguments that go against barnyard style work seating.

In case you were under a rock and missed it, this link on Testing in Production the Netflix Way includes both a summary, a transcript, and a video of Nora Jones (I don’t think this is the first time I’ve linked to one of her talks). If you do anything with services or the web, you should check it out.By the way – the tech world needs more twenty-minute talks. The hour long presentation should be the exception, and not the norm. Someone remind me to write a full blog on this someday.

Share this:

Like this:

I finished off a few books while flying this week, including Lean Change Management, which is one of those books that makes you want to read a bunch more books. It’s full of models and references, and, given that so much of my role is about leading change and helping others lead change, is probably something I’ll keep close at hand.

Ronan Mehigan sent me a link to an article on Specialists vs. Generalists. I like the article, but it also reminds me why I like the notion of Generalizing Specialists and Specializing Generalists so much.

Maybe I’m late to the party, but I just discovered dev.to. I’ve found a nice handful of articles there, including the following bullet…

Given the state of US politics and media, I generally defer to politifact.com or reuters.com for a (hopefully) more neutral view, but part of my daily reading continues to be the wonderful daily summary on What The Fuck Just Happened Today

Share this:

Like this:

I try to remember that a big part of my job is growing leaders – and that I need to always try to improve as a leader myself. Most times when I need a little inspiration for reflection, I turn to Simon Sinek, and re-read Together is Better. Here’s one of the quotes that caught my eye this week.When we tell people to do their jobs, we get workers. WHen we trust people to get the job done, we get leaders.

It’s double-quote Friday. Harlan Ellison passed away earlier this month. He was a fantastic, and creative writer, with a lot of passion. One quote of his that has entered my mind a lot as I watch various bits of US politics is this one.“You are not entitled to your opinion. You are entitled to your informed opinion. No one is entitled to be ignorant.”

In addition to these weekly posts, I write a lot. Internal newsletters, articles for others (and in the past, books and book contributions). Writing has been a big part of my growth, and this article on why writing is important is worth a read for any lifelong learner.

Finally, the FIFA World Cup has been hugely entertaining this year, and barring a continued run of form for France (I’m cheering for France, so they’re bound to lose), we may have a first time champion. Even more interesting, this is the first time EVER where one of Germany, Brazil, or Argentina are not in the semi-final. (Wikipedia link)