Testing Vs. Monitoring Mobile App Performance
Over the past few months I’ve interacted with dozens of mobile developers. In doing so I noticed that the difference between testing mobile apps and monitoring mobile apps is often not clear to developers. Both are important to include in the app development process but serve different purposes.
Below is a list of differences between the two and their unique benefits.
1. Development/QA vs. Production
Performance testing can be done in development environments using powerful tools like Apple’s Instruments which simulate any device, connection type, etc and provide a high degree of detail to help fix performance problems. Performance testing can also be done in QA environments using live testers or using scripts that run on a myriad of devices.
On the other hand performance monitoring involves watching production users in the wild as they use an app, using low-overhead, always-on instrumentation.
Testing helps catch performance issues before a new version is released to users, while monitoring helps catch performance issues missed in testing.
2. Synthetic Testing vs. Real User Monitoring
Testing app performance using scripts is known as synthetic testing. Scripts can simulate common use cases, or randomized interactions.
Real user monitoring as the name implies involves watching performance on handsets as they are used by live users.
The benefit of synthetic testing is that it can catch glaring performance issues on popular handset configurations. The downside of synthetic testing is that it only can test a small subset of the configurations users will run an app on. Also the handsets used do not represent the actual levels of available resources; synthetic testing can not simulate the actual memory, network bandwidth, or CPU cycles available on phones, given that unknown other apps might be hogging these resources. Plus it can inaccurately simulate the data loads of real users.
Moreover in the case of Android phones the operating system might be modified by handset manufacturer in a way that affects available resources and is often not represented by synthetic tests.
3. Estimated Performance vs. Actual Performance
When testing performance in QA with live users the test pool might not be representative of live users in production. Possible differences include:
- Handset quality used by testers might be better or worst than actual
- Volume of user data downloaded by testers might be significantly different than actual
- Connection speeds of testers might be much faster or slower than actual
In addition there is a question of statistical significance of the results from the pool of testers. For example, an app might only have a handful of testers in a strategic geography, say China, and might get performance estimates that are wildly different than actual.
Nonetheless estimating performance using testers is useful as an early warning mechanism for new app versions, and monitoring real user performance is useful as a way of confirming estimates.
Summary
Mobile app development can benefit from both testing and monitoring mobile app performance. In the future you can expect that the two could be tied together, with accurate production performance data from past app versions enhancing performance estimates from development and QA environments. For example past out-of-memory characteristics could be used to predict if a new app version, tested in QA, might cause a high level of out-of-memory instances once released to production.
