No Country for Indie Developers: A Study of Google Play's Closed Testing Requirements for New Personal Developer Accounts
In November 2023, Google Play enforced new "closed testing" requirements targeting personal developer accounts. These mandates require developers to recruit 20 testers who must remain opted-in for 14 consecutive days before an app can be published. This created a sudden, significant barrier to entry for independent (indie) developers who lack the resources of large corporate teams. Put yourself in the position of an indie developer.
Put yourself in the position of an indie developer. You found a niche problem, spent months in building a viable prototype, and want to release your product to start attracting real users.
The current policy wants you to find 20 individuals willing to test (interact) your app for 14 days continuously. As per the documentation, you can reach out to your friends, family, colleagues, or social media forums to find those individuals.
Google’s reasoning is that the developers must utilize the 14-days timeframe to improve their app by collecting feedback from real users. Once the requirements are satisfied, then you are able to publish the app on Play Store.
- Influence of the policy on an online discussion forum 'Reddit': We conducted a mixed-method qualitative study to map the impact of these requirements. In particular,
- We extracted and analyzed discussion threads from developer communities on Reddit (e.g., r/androiddev, r/FlutterDev, and the newly emerged r/AndroidClosedTesting).
- We designed and executed a survey with 14 indie Android developers (recruited from Reddit and local tech incubators) to validate the themes found in the online discussions.
- Impact of the policy: We identified how developers perceived and addressed the requirements of the policy.
- Thematic Analysis: We analyzed 897 comments and found that developers perceive the requirements as "discriminatory" and favored corporate accounts. Several developers reported their logistic frustration in finding "willing" testers for their apps; some argued that "their friends are not their target users". We also found several online testing hubs/communities on platforms like Fiverr, potentially remarked as "scams" and/or "black market" for app testing. The general negative comments against the policy argued that the requirements do not enforce strict app testing practices among developers; rather shifts the burden of testing from the app store to individual developers.
- Developer survey analysis: further confirmed by findings of Reddit analysis. They pressed on the logistical challenges in finding the right users and the incentives they can offer without getting scammed. If any of the testers miss a day (during the 14-days testing phase), the entire testing process fails to meet the policy requirements.
- Sample Bias: The Reddit analysis may suffer from an "echo chamber" effect, where negative sentiment is amplified. Majority of the comments in our sample were filled with negative sentiments.
- Demographics: The survey sample size (14 developers) is small, though we mitigated this by ensuring geographic diversity (including developers from India, Pakistan, Portugal, and Jordan)
- Longitudinal Quality Study: Investigating if the current testing requirements actually reduce the number of low-quality apps over time.
- Intermediary Ethics: Studying the rise of third-party "testing farms" (like those on Fiverr) and their potential to exploit small creators.
- Qualitative Research Methods: Applied Grounded Theory, Axial Coding, and Memoing to extract structured insights from unstructured text.
- Statistical Analysis: Calculated Fleiss’ Kappa to validate inter-rater reliability during data annotation.
- Ecosystem Dynamics: Learned how platform policies directly manipulate software supply chains and developer migration (e.g., developers threatening to move to iOS or Web).