Electronics and programming interspersed at various levels of difficulty.

Thursday, July 18, 2013

Android game automation - part 1

First: this is borderline immoral so don't ask for any source code or help.

My friend got me into a repetitive Android game that I will not name here. Basically it's a different kind of Farmville (I assume) that requires you to mindlessly click 'animals' to 'farm' money from them. On top of that you have to also activate two type of farms in order to feed the animals and evolve them. Feeding is not a requirement, so it will only be done in the second iteration of this automation.

As a rule of thumb any task that takes you at least 5 minutes every day for a year should be automated if it could be done in less than 20 hours.

I was opening the app about 1-3 times a day, clicking on each animal and closing it. Even though it's mindless and almost fun it still took about 3 minutes each: 1 minute the log-in and load time, 1 minute the mindless task, 1 minute reorganizing stuff to have it perform better. There is clearly room for improvement because now the 'robot' logs in every 5 minutes, does the job and gets out. Initial estimation shows it earns about 80% more virtual money than I was able (wanting) to.

Organization

The playing area is composed of several rectangles that you have to swipe through. Above you see only one are but my current field is about 5x5 this.

The small houses with the gold floating on top of them should be clicked every 5 minutes or so.

The animal in the top right part should be clicked evey 2 hours or so.

The green farm bottom center should be activated every 10 hours or so by clicking on the Activate button a few times and then on the farm itself.

Reverse-engineering

The app is a crappy port, probably from iOS, so most if its part is native. It also uses a custom game engine that I haven't been able to find any info on. The game logic itself is stored in some binary files that I assume to be some recompiled Lua scripts.
Seeing that this would take too long I went down the brute-force approach. It would also work if the app were to be updated, which in this case happens at around two weeks. That's right, they are pushing 80-150MB every two weeks to every device in the user base!

Idea

I envisioned the initial implementation to just follow my moves. It is close to impossible to find a macro recorder for Android so I rolled my own.

Since the app was native it was not possible to click by finding out objects inside the Window Manager, it must be done via the low-level [hardware] event manager.

Just by name we can figure out that event6 events are pushed by the touchscreen, event1 is the power key, event2 is the magnetometer, event3 the accelerometer, event4 the light sensor, event5 the headphone jack detection and event0 the remaining hardware keys (volume buttons, AFAIK).

Recording the events

it's just a matter of running getevent either on the device itself or via adb. Something like "getevent >events.out". The output should be similar to:

You might spot a problem here that I did not recognize in the beginning. Anyway, onward to

Playing back the events:

During some kind of twisted design decision it was decided that the "sendevent" command has a different format than what "getevent" provides. Luckily I found a blog post that had an almost working solution written in Python that did everything:

The problems

Everything went nice but for a recorded session lasting about 30 seconds the script took about 15 minutes to replay.

Also, all the micro-swipes were being converted to clicks and getting lost and all the swipe inertia was being lost because the speed was much lower. You normally don't realize all this stuff happening behind the scenes this when using a capacitive touchscreen.

The first issue could be dealt with by doing a few optimizations. You can see from the listing above that for every touch event there is are a ton of accelerometer events. A simple grep takes care of that.

Also, sending events from the PC to the phone is really slow so it can be solved like this:

Not all events are needed so some of them were stripped, I forgot which, but probably touch area and several of the touch confirmations. Here's a draft I had in the folder, ending with a parsed getevent.

Multi-touch devices use the following Linux input events:ABS_MT_POSITION_X: (REQUIRED) Reports the X coordinate of the tool.ABS_MT_POSITION_Y: (REQUIRED) Reports the Y coordinate of the tool.ABS_MT_PRESSURE: (optional) Reports the physical pressure applied to the tip of the tool or the signal strength of the touch contact.ABS_MT_TOUCH_MAJOR: (optional) Reports the cross-sectional area of the touch contact, or the length of the longer dimension of the touch contact.ABS_MT_TOUCH_MINOR: (optional) Reports the length of the shorter dimension of the touch contact. This axis should not be used if ABS_MT_TOUCH_MAJOR is reporting an area measurement.ABS_MT_WIDTH_MAJOR: (optional) Reports the cross-sectional area of the tool itself, or the length of the longer dimension of the tool itself. This axis should not be used if the dimensions of the tool itself are unknown.ABS_MT_WIDTH_MINOR: (optional) Reports the length of the shorter dimension of the tool itself. This axis should not be used if ABS_MT_WIDTH_MAJOR is reporting an area measurement or if the dimensions of the tool itself are unknown.ABS_MT_ORIENTATION: (optional) Reports the orientation of the tool.ABS_MT_DISTANCE: (optional) Reports the distance of the tool from the surface of the touch device.ABS_MT_TOOL_TYPE: (optional) Reports the tool type as MT_TOOL_FINGER or MT_TOOL_PEN.ABS_MT_TRACKING_ID: (optional) Reports the tracking id of the tool. The tracking id is an arbitrary non-negative integer that is used to identify and track each tool independently when multiple tools are active. For example, when multiple fingers are touching the device, each finger should be assigned a distinct tracking id that is used as long as the finger remains in contact. Tracking ids may be reused when their associated tools move out of range.ABS_MT_SLOT: (optional) Reports the slot id of the tool, when using the Linux multi-touch protocol 'B'. Refer to the Linux multi-touch protocol documentation for more details.BTN_TOUCH: (REQUIRED) Indicates whether the tool is touching the device.BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_BACK, BTN_SIDE, BTN_FORWARD, BTN_EXTRA, BTN_STYLUS, BTN_STYLUS2: (optional) Reports button states.BTN_TOOL_FINGER, BTN_TOOL_PEN, BTN_TOOL_RUBBER, BTN_TOOL_BRUSH, BTN_TOOL_PENCIL, BTN_TOOL_AIRBRUSH, BTN_TOOL_MOUSE, BTN_TOOL_LENS, BTN_TOOL_DOUBLETAP, BTN_TOOL_TRIPLETAP, BTN_TOOL_QUADTAP: (optional) Reports the tool type.[ 71418.045740] EV_ABS ABS_MT_TRACKING_ID 0000215d[ 71418.045789] EV_ABS ABS_MT_TOUCH_MAJOR 00000019[ 71418.045813] EV_ABS ABS_MT_POSITION_X 0000026a[ 71418.045837] EV_ABS ABS_MT_POSITION_Y 0000017a[ 71418.045861] EV_SYN SYN_MT_REPORT 00000000[ 71418.045882] EV_SYN SYN_REPORT 00000000[ 71418.056783] EV_ABS ABS_MT_TRACKING_ID 0000215d[ 71418.056838] EV_ABS ABS_MT_TOUCH_MAJOR 00000019[ 71418.056862] EV_ABS ABS_MT_POSITION_X 0000026a[ 71418.056885] EV_ABS ABS_MT_POSITION_Y 0000017a[ 71418.056908] EV_SYN SYN_MT_REPORT 00000000[ 71418.056929] EV_SYN SYN_REPORT 00000000[ 71418.067805] EV_ABS ABS_MT_TRACKING_ID 0000215d

All these changes improved the time from 15 minutes down to 5 minutes, but it was not good enough for 'production'.

But the most important problem of this approach was that it had no provision for application layout update and required a new recording with each game field change (i.e. moving animals around). Recording including parsing and everything took around 2-5 minutes per session and was probably required every two days.