Scripted test cases only find the bugs you already imagined. Exploratory testing finds the rest, but only when it has structure. This guide shows you how to run exploratory testing with a crowd of real testers, using charters, time-boxed sessions and debriefs that turn curiosity into a ranked list of fixes.
Key takeaways
- Exploratory testing works best with a written charter for every session, not a vague “go break it” brief.
- Time-box sessions to 60 to 90 minutes and require notes, evidence and a short debrief.
- Spread charters across devices, markets and user personas so the crowd covers different risk, not the same happy path.
- Measure results by unique, reproduced, high-severity issues, not by raw bug count.
- Feed confirmed findings back into regression suites so each cycle makes the next one smarter.
What exploratory testing is (and what it is not)
Exploratory testing is simultaneous learning, test design and execution. The tester studies the product, forms a hypothesis about where it might fail, and then tests that hypothesis right away. As a result, each step informs the next one.
However, exploratory does not mean unplanned. Good sessions have a mission, a time limit and a clear output. Without those, you get “monkey testing”: lots of taps, few insights and reports nobody can reproduce.
It also is not a replacement for scripted checks. Scripts confirm known behavior. Exploration hunts for unknown behavior. Therefore, mature teams run both, and they use exploration to decide which new scripts are worth writing.
Why the crowd changes the equation
An in-house tester explores with one mental model and a handful of devices. A crowd brings many mental models at once. For example, a tester in São Paulo pays with Pix on a Samsung Galaxy A54, while another in Jakarta uses a Xiaomi Redmi Note 13 on a congested Telkomsel connection. Because their contexts differ, they trip over different bugs.
In addition, crowd testers are new to your product. They have no “curse of knowledge,” so they try paths your team stopped noticing months ago.
Writing exploratory testing charters that focus the crowd
A charter is a one-to-three sentence mission for a single session. A widely used template is:
Explore [target area] with [resources, data or conditions] to discover [type of risk or information].
Here are charters that work well in crowd cycles:
- Explore the checkout flow with saved cards and an expired card to discover payment error handling gaps.
- Explore account sign-up with right-to-left languages (Arabic, Hebrew) to discover layout and truncation issues.
- Explore the offline mode with airplane mode toggled mid-upload to discover data loss or duplicate records.
- Explore push notification deep links on iOS 18 and Android 15 to discover routing failures from a cold start.
- Explore the profile editor with VoiceOver and TalkBack enabled to discover unlabeled controls and focus traps.
Charter quality checklist
Before you publish charters to testers, check each one against these questions:
- Is the target narrow enough to explore in one session? “The whole app” is too broad.
- Does it name a condition or resource, such as a device, network, locale, account type or data set?
- Does it state the risk you care about, so the tester knows what “interesting” means?
- Is it different from every other charter in the cycle? Overlap wastes testing hours.
- Can a tester finish it without internal knowledge they do not have?
Mapping charters to a coverage grid
Next, assign charters across a grid so you avoid blind spots. Put features on one axis and contexts on the other. Contexts include device tier, OS version, market, network and persona. Consequently, each cell gets an owner instead of every tester drifting to the login screen.
Running time-boxed exploratory testing sessions
Session-based test management gives exploratory testing its discipline. Each session is an uninterrupted block, usually 60 to 90 minutes, tied to one charter. During the session, the tester records what they did, what they saw and what they suspect.
A sample 24-hour crowd cycle
The table below shows how a typical cycle can flow on TestGate, where cycles kick off in under 24 hours.
| Time | Activity | Owner |
|---|---|---|
| Hour 0 | Build, release notes and 12 to 20 charters shared | Your team |
| Hours 1 to 3 | Testers matched by device, market and skill; NDA and access confirmed | Test lead |
| Hours 3 to 14 | Time-boxed sessions run in parallel across time zones | Testers |
| Hours 6 to 18 | Rolling triage: reproduce, merge duplicates, set severity | Test lead |
| Hours 18 to 22 | Session debriefs and coverage gap review | Test lead and your team |
| Hour 24 | Summary report; confirmed bugs synced to Jira or Linear | Platform |
What testers should capture during a session
Ask for structured session notes, not a stream of consciousness. Specifically, each session sheet should include:
- Charter and setup: device model, OS build, app version, network and locale.
- Test notes: short time-stamped entries of what was tried and why.
- Bugs: each with steps, expected and actual result, plus a screen recording.
- Issues and questions: confusing behavior that may or may not be a defect.
- Time split: rough percentages for testing, bug investigation and setup.
The time split matters more than it looks. For example, if a session spent 50% on setup, your test environment or instructions need work. That is a process bug, and it costs every tester in the cycle.
Exploratory testing heuristics the best testers use
Heuristics are quick prompts that help testers decide what to try next. They are especially useful for new crowd testers, because they give a shared vocabulary. Share a short list with every cycle.
- Interrupt it: incoming calls, lock screen, app switch, low battery mode and permission revocation mid-flow.
- Starve it: no network, 2G or throttled 3G, full storage and denied camera or location access.
- Repeat it: double-tap the pay button, go back and resubmit, refresh during a transaction.
- Stretch it: very long names, emoji, accented characters, zero, negative numbers and dates around midnight or DST changes.
- Change it: rotate the device, change system font size to the largest setting, toggle dark mode, switch language.
- Follow the money: refunds, partial payments, currency switching, coupons combined with gift cards.
These prompts consistently surface high-value bugs. For instance, a double-tap on “Place order” over a slow connection is a classic source of duplicate charges. Similarly, a German translation that is 35% longer than English often breaks a button label on a 5.4-inch iPhone 13 mini.
Debriefing: turning sessions into decisions
A session is not done until it is debriefed. The debrief is a 10 to 15 minute review between the tester and the test lead. Its goal is to extract what the notes do not say outright.
The PROOF debrief agenda
Many teams use a simple agenda summarized as PROOF:
- Past: What happened during the session?
- Results: What bugs and issues did you find?
- Obstacles: What slowed you down or blocked you?
- Outlook: What still needs testing in this area?
- Feelings: How confident are you about the quality of this area?
The “Outlook” and “Feelings” answers are gold. A tester who says “I only scratched the surface of refunds” tells you exactly where to aim the next charter. As a result, debriefs turn exploratory testing into a feedback loop rather than a one-off event.
The triage gate between sessions and your backlog
Raw crowd output is noisy. Therefore, on TestGate every report passes a triage gate before it reaches your backlog. A test lead reproduces the issue, merges duplicates and sets severity. Only then does it sync to Jira, Azure DevOps, Linear, GitHub or GitLab. For more on that step, read our guide to bug triage at scale.
Measuring exploratory testing results
Bug count is the most common metric and the least useful. A cycle with 200 cosmetic duplicates is worse than one with 15 reproduced, high-severity defects. Instead, track metrics that reflect risk reduction.
| Metric | What it tells you | Healthy signal |
|---|---|---|
| Unique confirmed bugs per session | Efficiency of charters and testers | Stable or rising across cycles |
| Share of high and critical severity | Whether testers hunt real risk | Meaningful share, not just cosmetics |
| Duplicate rate | Charter overlap | Falling as charters sharpen |
| Charter coverage | Feature and context cells explored | All priority cells covered |
| Escaped defects in the area | Real-world effectiveness | Falling after each release |
| Setup time share | Environment and brief quality | Low and shrinking |
For example, imagine a cycle of 12 testers running 24 sessions. If it produces 30 confirmed bugs and 6 are high severity, that is a useful baseline. Next cycle, sharpen the charters that found nothing and retire the ones that keep finding duplicates.
Common exploratory testing mistakes to avoid
Even experienced teams fall into the same traps. Watch for these:
- One giant charter: “Test the new release” produces shallow, overlapping sessions.
- No build notes: testers waste time rediscovering known issues. Always share a known-issues list.
- Happy-path devices only: if every tester uses a flagship iPhone, you miss the budget Android phones many of your users own.
- Skipping debriefs: you lose the context that explains why an area feels fragile.
- Rewarding volume: paying or praising by bug count invites trivial reports. Reward severity and clarity instead.
- Not closing the loop: confirmed bugs that never become regression checks will return.
When to schedule exploratory cycles
Exploration pays off most at moments of change. Good triggers include a major feature launch, a redesign, a new market or payment method, a large dependency upgrade, and the first build on a new OS beta. In addition, a short exploratory pass right before release catches the integration bugs that unit tests never see.
If you are new to crowd cycles, start with our walkthrough on running your first crowdtest cycle in 24 hours. Because exploration depends on context, give testers the same release notes and test accounts your own team uses. In addition, tell them which areas are out of scope, so no session is wasted on a feature that is still behind a flag. Then, pair it with TestGate’s exploratory testing service for charter design and test lead support.
Frequently asked questions
What is a test charter in exploratory testing?
A test charter is a short mission statement for one session. It names the area to explore, the resources or data to use, and the kind of risk you want to uncover, so the tester has focus without a rigid script.
How long should an exploratory testing session be?
Most teams use sessions of 60 to 90 minutes. That is long enough to get deep into a feature, yet short enough that the tester stays sharp and the notes stay detailed.
Can exploratory testing replace scripted regression tests?
No. Exploratory testing finds unknown risks, while scripted and automated regression checks confirm that known behavior still works. You get the best coverage when you run both.
How many testers do I need for a crowd exploratory cycle?
For a single release, 8 to 15 testers spread across your key devices and markets is a solid starting point. Add more when you need wider device, language or payment coverage.
Run your next exploratory cycle with TestGate
Send us your build and your riskiest feature, and we will turn it into charters, sessions and triaged results within a day: talk to us.
